SysAdmin TnT
System administration tips ‘n’ tricks
Python 3 on CentOS 7
Without impacting your native Python installation, you can install Python 3 along side of it using the following method.
Installing Python 3
Step 1: If you can, take a snapshot of your VM before proceeding.
Step 2: Run the following commands:
$ sudo yum update
$ sudo yum install @development
$ sudo yum install -y zlib-devel openssl-devel sqlite-devel bzip2-devel xz-libs
$ VER=3.6.5
$ PYVER="Python-${VER}"
$ wget http://www.python.org/ftp/python/${VER}/${PYVER}.tar.xz
$ xz -d ./${PYVER}.tar.xz
$ tar -xvf ./${PYVER}.tar
$ cd ${PYVER}
$ ./configure
$ make
$ sudo make altinstall
To see what you’ve done:
$ which python3 /usr/local/bin/python3 $ which python /usr/bin/python $ which python3.6 /usr/local/bin/python3.6
You can create symlinks to safely reference items of interest:
$ ls -al /usr/local/bin/ total 24836 drwxr-xr-x. 2 root root 196 May 17 17:38 . drwxr-xr-x. 12 root root 131 Apr 11 00:59 .. -rwxr-xr-x. 1 root root 101 May 17 16:38 2to3-3.6 -rwxr-xr-x. 1 root root 242 May 17 16:38 easy_install-3.6 -rwxr-xr-x. 1 root root 99 May 17 16:38 idle3.6 lrwxrwxrwx. 1 root root 21 May 17 17:38 pip3 -> /usr/local/bin/pip3.6 -rwxr-xr-x. 1 root root 214 May 17 16:38 pip3.6 -rwxr-xr-x. 1 root root 84 May 17 16:38 pydoc3.6 lrwxrwxrwx. 1 root root 24 May 17 17:31 python3 -> /usr/local/bin/python3.6 -rwxr-xr-x. 2 root root 12699000 May 17 16:38 python3.6 -rwxr-xr-x. 2 root root 12699000 May 17 16:38 python3.6m -rwxr-xr-x. 1 root root 3117 May 17 16:38 python3.6m-config -rwxr-xr-x. 1 root root 441 May 17 16:38 pyvenv-3.6
Python 3 and pip
Issues like missing modules may arise. They’ll look something like the following:
$ python3 ./main.py Traceback (most recent call last): File "./main.py", line 8, in <module> from lxml.etree import ET ModuleNotFoundError: No module named 'lxml'
Can pip help us?
$ which pip3 /usr/local/bin/pip3
Looks like I have a pip3. Let’s try…
$ pip3 install lxml -bash: /usr/local/bin/pip3: /usr/local/bin/python3.6: bad interpreter: No such file or directory
Hmmm…that does NOT look good. Do I have python3.6?
$ which python3.6 /usr/bin/python3.6
Sure do, but it’s in a location that pip3 is paying no attention to. As you can see from the output, pip3 is unusable. Just running pip3 results in bad interpreter: No such file or directory. We need to fix pip or fix the part where it’s expecting python3.6 to be in /usr/local/bin/python3.6
Easiest thing to try is to update the operating system’s hash table where it stores its mappings of programs to their locations so it doesn’t have to search every time. Sometimes those mappings can get out of sync, especially if a location of a program changes.
$ hash hits command 1 /usr/bin/sudo 2 /usr/local/bin/pip3 2 /usr/bin/python3 2 /usr/bin/ls $ hash -r $ hash hash: hash table empty
Maybe that will help pip3 find what it needs? Um, no. Not one bit.
$ pip3 install lxml -bash: /usr/local/bin/pip3: /usr/local/bin/python3.6: bad interpreter: No such file or directory
Why is pip3 expecting python3.6 to be in that location when it clearly isn’t?
After fixing my symlink (not because it would help with my current issue but because it’s good to fix things), I returned my focus to pip3. I would like to uninstall and reinstall it. If it is being managed by yum, it can easily be uninstalled by calling yum remove.
$ sudo yum remove python3-pip Loaded plugins: fastestmirror No Match for argument: python3-pip No Packages marked for removal
But, it isn’t. How did pip3 get on this machine? Can python3 run pip as a module?
$ python3 -m pip /usr/local/bin/python3: No module named pip
-m module-name
Searches sys.path for the named module and runs the corresponding .py file as a script.
Nope. Okay, enough spinning my wheels. The setup I have isn’t making much sense. How is this supposed to happen? How is pip supposed to be installed? Maybe I can do what’s supposed to be done (or by trying something else somebody else claims to have worked for them) and things will just work?
$ python3 -m ensurepip --user
Looking in links: /tmp/tmp41dea8fo
Requirement already satisfied: setuptools in /usr/lib/python3.6/site-packages (39.2.0)
Collecting pip
Installing collected packages: pip
Successfully installed pip-18.1
$ python3 -m pip
Usage:
/usr/local/bin/python3 -m pip <command> [options]
Commands:
install Install packages.
download Download packages.
uninstall Uninstall packages.
freeze Output installed packages in requirements format.
list List installed packages.
show Show information about installed packages.
check Verify installed packages have compatible dependencies.
config Manage local and global configuration.
search Search PyPI for packages.
wheel Build wheels from your requirements.
hash Compute hashes of package archives.
completion A helper command used for command completion.
help Show help for commands.
General Options:
-h, --help Show help.
--isolated Run pip in an isolated mode, ignoring environment variables and user configuration.
-v, --verbose Give more output. Option is additive, and can be used up to 3 times.
-V, --version Show version and exit.
-q, --quiet Give less output. Option is additive, and can be used up to 3 times (corresponding to
WARNING, ERROR, and CRITICAL logging levels).
--log <path> Path to a verbose appending log.
--proxy <proxy> Specify a proxy in the form [user:passwd@]proxy.server:port.
--retries <retries> Maximum number of retries each connection should attempt (default 5 times).
--timeout <sec> Set the socket timeout (default 15 seconds).
--exists-action <action> Default action when a path already exists: (s)witch, (i)gnore, (w)ipe, (b)ackup, (a)bort).
--trusted-host <hostname> Mark this host as trusted, even though it does not have valid or any HTTPS.
--cert <path> Path to alternate CA bundle.
--client-cert <path> Path to SSL client certificate, a single file containing the private key and the
certificate in PEM format.
--cache-dir <dir> Store the cache data in <dir>.
--no-cache-dir Disable the cache.
--disable-pip-version-check
Don't periodically check PyPI to determine whether a new version of pip is available for
download. Implied with --no-index.
--no-color Suppress colored output
Yep!
Can I do what I had originally set out to do?
$ python3 -m pip install --user lxml Collecting lxml Using cached https://files.pythonhosted.org/packages/dd/ba/a0e6866057fc0bbd17192925c1d63a3b85cf522965de9bc02364d08e5b84/lxml-4.5.0-cp36-cp36m-manylinux1_x86_64.whl Installing collected packages: lxml Successfully installed lxml-4.5.0
Yep!
Resizing a Guest’s Disk (KVM) – Extending
Resizing the disk on a KVM system is rather straight-forward, but it sure does require that you get the steps right!
The Steps
- Remove snapshots
- Stop the VM
- Enlarge the disk itself
- In GUI, find your VM. Choose Hardware, then click on the disk and then Resize Disk
- From CLI, run one of the following:
qemu-img resize <disk> +<size>G- Example:
qemu-img resize /vm/images/158/vm-158-disk-1.qcow2 +32Gto increase image size by 32GB
- Example:
qm resize <vmid> <disk> +<size>G- Example:
qm resize 158 /vm/images/158/vm-158-disk-1.qcow2 +32Gto increase VM #158’s disk by 32GB
- Example:
- Start the VM and resize the partition then LVM
- Run fdisk inside your guest and remove the partition that you want to extend
[root@centos7 ~]# lsblk | grep disk sda 8:0 0 64G 0 disk [root@centos7 ~]# fdisk /dev/sda Welcome to fdisk (util-linux 2.23.2). Changes will remain in memory only, until you decide to write them. Be careful before using the write command. Command (m for help): p Disk /dev/sda: 68.7 GB, 68719476736 bytes, 134217728 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk label type: dos Disk identifier: 0x000be40f Device Boot Start End Blocks Id System /dev/sda1 * 2048 2099199 1048576 83 Linux /dev/sda2 2099200 67108863 32504832 8e Linux LVM Command (m for help): d Partition number (1,2, default 2): 2 Partition 2 is deleted Command (m for help): p Disk /dev/sda: 68.7 GB, 68719476736 bytes, 134217728 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk label type: dos Disk identifier: 0x000be40f Device Boot Start End Blocks Id System /dev/sda1 * 2048 2099199 1048576 83 Linux Command (m for help): n Partition type: p primary (1 primary, 0 extended, 3 free) e extended Select (default p): p Partition number (2-4, default 2): First sector (2099200-134217727, default 2099200): Using default value 2099200 Last sector, +sectors or +size{K,M,G} (2099200-134217727, default 134217727): Using default value 134217727 Partition 2 of type Linux and of size 63 GiB is set Command (m for help): p Disk /dev/sda: 68.7 GB, 68719476736 bytes, 134217728 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk label type: dos Disk identifier: 0x000be40f Device Boot Start End Blocks Id System /dev/sda1 * 2048 2099199 1048576 83 Linux /dev/sda2 2099200 134217727 66059264 83 Linux Command (m for help): t Partition number (1,2, default 2): 2 Hex code (type L to list all codes): 8e Changed type of partition 'Linux' to 'Linux LVM' Command (m for help): p Disk /dev/sda: 68.7 GB, 68719476736 bytes, 134217728 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk label type: dos Disk identifier: 0x000be40f Device Boot Start End Blocks Id System /dev/sda1 * 2048 2099199 1048576 83 Linux /dev/sda2 2099200 134217727 66059264 8e Linux LVM Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. WARNING: Re-reading the partition table failed with error 16: Device or resource busy. The kernel still uses the old table. The new table will be used at the next reboot or after you run partprobe(8) or kpartx(8) Syncing disks. [root@centos7 ~]# - Reboot the VM
- Resize the LVM physical volume
[root@centos7 ~]# pvdisplay --- Physical volume --- PV Name /dev/sda2 VG Name cl_vm-07 PV Size <31.00 GiB / not usable 3.00 MiB Allocatable yes PE Size 4.00 MiB Total PE 7935 Free PE 1 Allocated PE 7934 PV UUID 8dCBQY-gbR5-xy4k-U0Ap-fDfv-4idB-J3rD5V [root@centos7 ~]# pvresize /dev/sda2 Physical volume "/dev/sda2" changed 1 physical volume(s) resized / 0 physical volume(s) not resized [root@centos7 ~]# pvdisplay --- Physical volume --- PV Name /dev/sda2 VG Name cl_vm-07 PV Size <63.00 GiB / not usable 2.00 MiB Allocatable yes PE Size 4.00 MiB Total PE 16127 Free PE 8193 Allocated PE 7934 PV UUID 8dCBQY-gbR5-xy4k-U0Ap-fDfv-4idB-J3rD5V
- Resize the logical volume
[root@pete-dev-centos7 ~]# lvdisplay --- Logical volume --- LV Path /dev/cl_vm-07/swap LV Name swap VG Name cl_vm-07 LV UUID 6E7b0t-0q8s-Bm6m-LqRI-jwfY-GQcr-eMmERY LV Write Access read/write LV Creation host, time vm-07, 2017-05-23 08:22:40 -0400 LV Status available # open 2 LV Size 2.00 GiB Current LE 512 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 8192 Block device 253:1 --- Logical volume --- LV Path /dev/cl_vm-07/root LV Name root VG Name cl_vm-07 LV UUID aeVwKb-Th7A-al03-aKAq-HONA-mciK-Nsew02 LV Write Access read/write LV Creation host, time vm-07, 2017-05-23 08:22:40 -0400 LV Status available # open 1 LV Size 28.99 GiB Current LE 7422 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 8192 Block device 253:0 [root@centos7 ~]# lvresize /dev/cl_vm-07/root -l +8193 Size of logical volume cl_vm-07/root changed from 28.99 GiB (7422 extents) to <61.00 GiB (15615 extents). Logical volume cl_vm-07/root successfully resized. ### NOTE: ### You can also use a one-liner: ### lvextend /dev/cl_vm-07/root /dev/sda2 ### Without any options, it will use the maximum size in that group [root@centos7 ~]# lvdisplay --- Logical volume --- LV Path /dev/cl_vm-07/swap LV Name swap VG Name cl_vm-07 LV UUID 6E7b0t-0q8s-Bm6m-LqRI-jwfY-GQcr-eMmERY LV Write Access read/write LV Creation host, time vm-07, 2017-05-23 08:22:40 -0400 LV Status available # open 2 LV Size 2.00 GiB Current LE 512 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 8192 Block device 253:1 --- Logical volume --- LV Path /dev/cl_vm-07/root LV Name root VG Name cl_vm-07 LV UUID aeVwKb-Th7A-al03-aKAq-HONA-mciK-Nsew02 LV Write Access read/write LV Creation host, time vm-07, 2017-05-23 08:22:40 -0400 LV Status available # open 1 LV Size <61.00 GiB Current LE 15615 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 8192 Block device 253:0
- Grow the filesystem
### NOTE: ### In Centos 7 default filesystem is xfs. ### Also noteworthy: xfs file system supports extend not reduce ### If you want to resize the filesystem use xfs_growfs rather than resize2fs. [root@centos7 ~]# xfs_growfs /dev/vm-07/root meta-data=/dev/mapper/vm--07-root isize=512 agcount=4, agsize=1900032 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=0 spinodes=0 data = bsize=4096 blocks=7600128, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0 ftype=1 log =internal bsize=4096 blocks=3711, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 data blocks changed from 7600128 to 15989760 ### Note: For ext4 filesystem use [root@centos7 ~]#resize2fs /dev/vm-07/root [root@centos7 ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/vm--07-root 61G 23G 39G 37% / devtmpfs 909M 0 909M 0% /dev tmpfs 920M 0 920M 0% /dev/shm tmpfs 920M 8.4M 912M 1% /run tmpfs 920M 0 920M 0% /sys/fs/cgroup /dev/sda1 1014M 279M 736M 28% /boot tmpfs 184M 0 184M 0% /run/user/1002 tmpfs 184M 0 184M 0% /run/user/1001
- Run fdisk inside your guest and remove the partition that you want to extend
Other Useful References
nfsd on Linux
Background
The Network File System (NFS, referring to the nfsd daemon, not the general overarching concept of network file systems) is handy when you need to access a large, centralized file store from one or more servers throughout your LAN. NFS over WAN would take more consideration and more in-depth investigate as to what your “lowest common denominator” would be (i.e. your weakest link may be beyond your control).
On a LAN, NFS works well. It’s responsive, easy to set up, and robust in a variety of environments.
The setup involves a client accesses the data made available by another machine, the server.
Overview
If I may borrow this well-written excerpt from this Wikipedia article:
- The server implements NFS daemon processes, running by default as
nfsd, to make its data generically available to clients. - The server administrator determines what to make available, exporting the names and parameters of directories, typically using the
/etc/exportsconfiguration file and theexportfscommand. - The server security-administration ensures that it can recognize and approve validated clients.
- The server network configuration ensures that appropriate clients can negotiate with it through any firewall system.
- The client machine requests access to exported data, typically by issuing a
mountcommand. (The client asks the server (rpcbind) which port the NFS server is using, the client connects to the NFS server (nfsd), nfsd passes the request to mountd) - If all goes well, users on the client machine can then view and interact with mounted filesystems on the server within the parameters permitted.
Note that automation of the NFS mounting process may take place — perhaps using /etc/fstab and/or automounting facilities.
Quick GuideS
CentOS 7
A very helpful guide for NFS on CentOS 7 can be found here.
More References
How NFS is Implemented in the Linux Kernel
RFC Documents
Cisco AnyConnect and Host-Only Connectivity in VirtualBox
Sometimes I desire to have a VM on my local machine, which is running Max OS X 10.10. I prefer to be able to use SCP to transfer files from my host to my guest, usually some flavor of Linux.
It seems pretty straight-forward to set up a host only connection. There are articles all over describing exactly how it can be done. Here’s one and here’s another.
Under normal conditions, this works flawlessly. However, at work, we use Cisco AnyConnect (here is some marketing material if you’re interested), which hijacks my routing table and sends everything to the watchful eye of our network admins…or at least through their gateways and hops how they see fit. This means that even a local “host-only” route is over-written by whatever they have dictated. Here is my “host-only” route being sent out my VPN tunnel:
192.168.56 link#10 UCS 0 0 utun0
This means that if you want to have a local VM with which to play and experiment, connecting to it isn’t all that straight-forward. First, you have to disconnect from your VPN (AnyConnect > Disconnect). You’ll see that your VirtualBox local host-only route is gone completely:
$ netstat -nr Routing tables Internet: Destination Gateway Flags Refs Use Netif Expire default 10.65.10.1 UGSc 53 315 en3 10.65.10/23 link#5 UCS 3 0 en3 10.65.10.1 4c:4e:35:77:77:41 UHLWIir 54 0 en3 600 10.65.10.97 127.0.0.1 UHS 0 0 lo0 10.65.10.99 68:5b:35:77:77:85 UHLWI 0 0 en3 644 10.65.11.255 ff:ff:ff:ff:ff:ff UHLWbI 0 3 en3 127 127.0.0.1 UCS 0 0 lo0 127.0.0.1 127.0.0.1 UH 71 839273 lo0 169.254 link#5 UCS 0 0 en3
The VirtualBox host-only interface may have an IP address, both on the host and the guest, but its route is not re-added to my routing table when I disconnect from the VPN. I have to go into VirtualBox > Preferences > Network > Host-only Network, remove the host-only interface (vboxnet0) and re-add it. Once I do this, its route is added to my routing table and I can connect to my VM guest!
$ netstat -nr Routing tables Internet: Destination Gateway Flags Refs Use Netif Expire default 10.65.10.1 UGSc 65 315 en3 10.65.10/23 link#5 UCS 3 0 en3 10.65.10.1 4c:4e:35:77:77:41 UHLWIir 66 0 en3 527 10.65.10.97 127.0.0.1 UHS 0 0 lo0 10.65.10.99 68:5b:35:77:77:85 UHLWI 0 0 en3 571 10.65.11.255 ff:ff:ff:ff:ff:ff UHLWbI 0 6 en3 127 127.0.0.1 UCS 0 0 lo0 127.0.0.1 127.0.0.1 UH 73 840630 lo0 169.254 link#5 UCS 0 0 en3 192.168.56 link#11 UC 2 0 vboxnet 192.168.56.255 ff:ff:ff:ff:ff:ff UHLWbI 0 1 vboxnet
Windows 10 Connecting to Mac OS X (Yosemite 10.10.5) SMB Share
Problem:
I have a machine running Windows 10 Professional and a Mac Mini running Mac OS X (Yosemite 10.10.5). I can’t connect to my Mac’s SMB share from Win10.
Resolved:
Yes
Takeaway:
On Yosemite 10.10.5, be sure to allow Windows File Sharing
Log Location:
Mac – Console.app – Unable to find exact log other than to go to All Messages – search for ntlmv
Windows – Didn’t access logs, used secpol.msc, not sure I had to
How I Stumbled Upon the Resolution:
First Stop: apple.com
This post seems to be about OD users versus Local users. Not sure if that’s me or not so let’s read on.
Second Stop: macwindows.com
After making the security changes recommended above, I rebooted and tried again. Nearly the same error message but it’s changed slightly:
From:
digest-service: digest-request: od failed with 2 proto=ntlmv2
digest-service: digest-request: guest failed with -1561745590 proto=ntlmv2
To:
digest-service: digest-request: od failed with 2 proto=ntlmv1-with-v2-session
digest-service: digest-request: guest failed with -1561745590 proto=ntlmv1-with-v2-session
Then:
Changed back to “Send NTLMv2 response only” according to this (I reboot whenever in doubt so I’ve just gotten somewhat used to it with some things). Well, now we’re back to the original failure message. Since I didn’t attack this from the same direction, it’s good to be able to come back to a base. Moving on…
So, then:
I got this from here:
On the Yosemite (10.10) side: Apple Menu > System Preferences > Sharing Left pane checkbox for File Sharing > Right side click Options Checkbox for Share Files And Folders Using SMB Windows File Sharing, check the accounts you want to be able to access files Done
I then logged in with the username that showed in that list and…I’m in!
Failed WordPress Update – Round 1
I recently attempted to update my WordPress installation. The goal was to update the website to the latest WordPress, 4.3.1 from 4.2.4. After clicking update, in less than a split-second, the site returned ERROR 500 and that’s it. The logs are sporadically placed so I had trouble finding them. I don’t know if CentOS likes dropping logs closer to the codebase’s location rather than in /var/log or if I inadvertently did that somehow but whatever the reason, I’m reconfiguring my server to drop its logs into /var/log.
When I try to start it by hand, all I get is a FAILED after two warnings. That’s it:
[root@cvi log]# service httpd restart Stopping httpd: [FAILED] Starting httpd: [Tue Nov 17 18:27:27 2015] [warn] module php5_module is already loaded, skipping [Tue Nov 17 18:27:27 2015] [warn] module proxy_ajp_module is already loaded, skipping [FAILED]
My config has the following:
<IfModule !worker.c> LoadModule php5_module modules/libphp5.so </IfModule> <IfModule worker.c> LoadModule php5_module modules/libphp5-zts.so </IfModule>
Looking up the httpd.conf manual and reading more about the IfModule sections, it seems like a legit configuration so I’m going to leave it. Something else is causing my web server to fail to start. I have yet to find a cause for the “FAILED” start and had to roll back my public website to an earlier snapshot…which HostGator accomplished for me without any issues. Thanks, HostGator! Whew! That saved my week.
Now it’s time to reconfigure my logging so I can track this down when it happens again.