Working From Home

I was perusing StackOverflow and came across a post containing tips and suggestions for working from home from experienced remote workers.  As I was reading through it, I remembered that I had a long run at working from home myself.  I once worked with a partner as consultants and we did this without an office.  He was usually on the road and I was usually rolling out of bed and planting myself in front of my computer.

IMPORTANT: I’m not an occupational therapist or a trained professional, this is only my opinion based on my experience.

Following their lead, I’ll start with location.

Location

I recently got a door on my office and it has made all the difference in the world.  I am sure my wife doesn’t fully appreciate them because I can lock them from the inside, but for me and my concentration, they have been key.  To build on what others are saying in the article, I do agree that having a different or isolated space is important.  If you have to work at the kitchen table in a room with constant distractions, more power to you.  Some people can, I’m not here to say it doesn’t work for you, but it doesn’t work for me.

In addition to a space, have proper lighting (I prefer natural light to artificial), room to spread out, a comfy chair, and a reliable headset for calls and to block out background noise.

Schedule

Your daily schedule should mimic what you would do if you were in the office or if your employer had business or operating hours.  If being home automatically signs you up for a collection of responsibilities and you can’t hold normal business hours at home, then make arrangements with your manager so that they are aware what hours will be your most productive and which hours will probably be chopped full of kid duties or whatever may block you from being productive.  Kids are a common cause for interrupted time but there are other things that may split up your day.  During your productive time, mentally prepare to focus on your work – get your tasks done and move on!

Daily Routine

Having food on hand prevents you from having to spend too much time trying to figure out what you’re doing for lunch.  Your home office may still be located in an area where there are lunch opportunities outside of the home that are faster than your at-home options.  Again, the decision is yours.  I recommend coming up with a solution that works 80% of the time by default – when you don’t have time to think about it, 80% of the time you can rely on a small list of options to meet your needs.  You don’t want to be opening the fridge every day asking yourself, “What do I feel like eating today?”  Personally, I’m pretty bad at answering that question myself so PB&J or cold-cuts will solve that issue 50% of the time.  I also know that leftovers will answer the question 40% of the time.  The other 10% of the time, I’ll take the opportunity to go for a walk and grab a burger at the corner.  I have 30+ restaurants within a 1-mile radius of my home office so I have options!  You may or may not.

The Great Outdoors

It’s hard to remember to get up and get out and do something so set a timer.  Buy an eff timer or a piece of software that will remind you to get up!  It’s important.  Don’t ignore our great outdoors!

Communication

Keep it simple.  I don’t like having 6 or 8 different channels for the 3 or 4 teams or groups with whom I have to communicate.  Use a chat solution that will notify you if you are mentioned if the noise gets to be too much but having to check a half dozen channels every so often is, in my opinion, is more disruptive than just having three or four large channels.  Once you decide on a solution, don’t hesitate to over communicate.  Not every conversation needs a fly gif from gliffy – just state your purpose and move on.  Not every comment requires a comment in return – just give it a thumbs up and move on.  You could spend all day in Teams adding fancy responses and trying to make everybody laugh every time something is said but that gets old after a while and isn’t sustainable.  Plus, it begs the question, “Do you have enough to do?”

Tools of the Remote Trade

Software selections have to be decided by, a.) your employer, or b.) your group.  The selection of software that may or may not work for you is beyond the scope of this post.  However, I will say that I do think that you need to invest in the following:

  1. a desk or desk-like surface – you’re going to regret sitting on your couch day in and day out
  2. proper lighting – staying in the dark is fine but eventually, you’ll want to reduce the contrast between your surroundings and your monitor – for your eyes’ sake
  3. proper chair – if it’s too hard, get a cushion; if it’s too soft, get lumbar support; I like “chairs” that can help build your core but you’ll want to stand, sit, work your core all in moderation
  4. organization – make sure you can find what you need – pencils, files, highlighters
  5. refreshments – if it’s coffee, invest in a solution that works for you.  I like the aero press and am okay making a cup every two or three hours
  6. snacks – I can get by with peanuts and gum so I have a good stock of each
  7. screens in your window – maybe the weather is absolutely perfect some days and you want to breathe fresh air.  Keep your cats in and the bugs out by making sure your screens are in good shape
  8. reliable computer – having to deal with equipment issues every week is a nightmare.  Failing systems will cause you to miss meetings or worse, cause your colleagues to spend their time repeatedly helping your figure out if your mic is working today
  9. external hard drive – computers fail, back up your data
  10. good keyboard, mouse, and external monitor – don’t punish yourself by forcing your fingers to use the keyboard on the new Macbook – what a nightmare!  Get a nice external keyboard and a second monitor…and all the dongles (thanks, @AppleCares!)

 

Extending KVM Disk

I’ve been a fan of virtual machines from early on, like KVM for instance. A greatly respected ex-coworker recommended Proxmox, which was the front end to a Type [1.5] hypervisor. He nonchalantly try to sell a few us on it from time to time and I think I failed to make note of how awesome it really was.

I use it daily and am sometimes asked to extend a user’s VM host’s disk. Usually, it to support development (storing test data) and not because of laziness or ignorance (needing more space for crap because they’re not sure how to keep it in check).

The process for increasing the size of your KVM disk is as follows.

Prerequisites

  • KVM host
  • KVM guest
  • root access via command line
  • VM ID
  • Know your LVM
    • Group name; use vgdisplay
    • Logical volume name: use lvdisplay
    • Is it ext4 or xfs

Procedure

  1. Shutdown your guest
    shutdown -hP now
  2. Make a copy of your VM’s current disk
    cp /var/lib/images/vm-117.qcow2 /var/lib/images/vm-117.qcow2.orig
  3. Increase the size of the disk image (CLI or GUI)
    qm resize <vmid> <disk> +<size>G
    qemu-img resize ...
    Example:
    qm resize 117 /dev/sda +100%FREE
  4. Convert VMDK to raw, if applicable
    qemu-img convert -O raw vm-117.qcow2 vm-117.raw
  5. Start guest
  6. On guest, enlarge the partition using a partition tool (Proxmox already has parted installed, you can use that if you prefer). If your disk already has a swap (id 82) partition, you will want to delete it and add it back to the end of the disk.  See this guide or the last step in this guide.
    fdisk /dev/sda
    Create a new partition to add to the new volume group
    type: primary
    start:<default value>
    end:<default value>
    type:8e
    write changes: w
  7. Reboot or rescan partitions on guest
  8. Initialize a disk or partition for use by LVM
    pvcreate /dev/sdX#
  9. Verify your physical volumes
    pvdisplay
  10. Add the partition to the partition map.  Confirm your partition is present using
    cat /proc/partitions
  11. If your partition isn’t there, reboot or rescan
    partx /dev/sdX#
  12. Get the name of your volume group
    vgdisplay
  13. Extend your volume group by adding your new partition to it
    vgextend <vg_name> <partition>
  14. Get the name of your logical volume root
    lvscan
  15. Extend the logical volume (-r means you don’t have to run resize2fs separately)
    lvextend -r -l +100%FREE /dev/<vg_name>/<lv_root>
    Example:
    lvextend -l +100%FREE /dev/mapper/centos-root
  16. Resize the filesystem (if you did not do a -r above).
    Determine your filesystem:
    mount | column -t
    for ext4, use resize2fs /dev/mapper/centos-root (or value from mount command above)
    for xfs, use xfs_growfs /dev/mapper/centos-root (or value from mount command above)
  17. Confirm larger size

 

The Boot Process

I boot systems multiple times a day. Sometimes, I have to pay close attention to how they are booting or why they are not booting up. Being familiar with the boot process is rather important. Here it is in a nutshell.

  • Power supply sends signal to the motherboard and other system components
  • Processor is hard-coded to know where to find BIOS (in system BIOS ROM – normally at location FFFF0h, right at the end of system memory)
  • BIOS runs tests, called POST (Power On Self Test).
  • After the test, results of POST are compared with data in CMOS chip. Typically one beep means “all is well” while other sequences of beeps indicate “all is not well”. The sequences of beeps are used for troubleshooting and are typically specific to the motherboard in question. Check with the manufacturer for POST beep codes.
  • Next, the video card’s BIOS is loaded and run. Other BIOSes are found and executed. The order depends on timing, mostly?
  • BIOS displays its startup screen and performs additional tests such as memory count-up tests and an inventory of hardware. Settings for memory and hard drives are set at this time. Messages are typically displayed to the screen for logical devices it finds as well as Plug and Play devices.
  • BIOS looks for boot devices according to boot order and a master boot record (at cylinder 0, head 0, sector 1)
  • The master boot record points to the boot sector where the kernel can be found
  • Entry point for kernel is located and the kernel is loaded into RAM. BIOS hands off to the kernel at this point.
  • Operating system continues to load according to instructions and parameters passed to kernel

Thanks to flint.cs.yale.edu for their wonderful write-up with links to more in-depth articles.


Modifying initramfs

BACKGROUND

Booting Linux involves various components at several different stages.  The following summary was taken from here.

First is BIOS.  After loading date, time, and important peripherals from CMOS, then the storage devices are probed.  When the first hard disk and its geometry are recognized, the system control passes from BIOS to the boot loader.

The Master Boot Record (first 512 bytes) on the disk is where you will find the boot loader.  The commands executed by the boot loader determine the rest of the boot process.  Control is passed to the actual operating system, like the Linux kernel.

The kernel and an initial RAM-based file system is loaded into memory.  The initramfs contains a small executable called init that handles the mounting of the real root file system.

The init program handles mounting the proper root file system.  If it is successful, initramfs is cleaned and the init program on the root file system is executed.

The init process handles the actual booting of the system through different levels.  I don’t want to summarize the following information because every bit of it is important for the purpose of the document (again, taken directly from here):

initramfs

 

initramfs is a small cpio archive that the kernel can load to a RAM disk. It provides a minimal Linux environment that enables the execution of programs before the actual root file system is mounted. This minimal Linux environment is loaded into memory by BIOS routines and does not have specific hardware requirements other than sufficient memory. initramfs must always provide an executable named init that should execute the actual init program on the root file system for the boot process to proceed.

 

Before the actual root file system can be mounted and the actual operating system can be started, the kernel needs the corresponding drivers to access the device on which the root file system is located. These drivers may include special drivers for certain kinds of hard drives or even network drivers to access a network file system. The needed modules for the root file system may be loaded by init on initramfs. After the modules are loaded, udev provides the initramfs with the needed devices. initramfs is available during the entire boot process. This makes it possible to handle all device events generated during boot.

Problem

So, my issue was trying to get a system image onto a system using Clonezilla.  This is fairly straight-forward as long as you don’t have any devices present that the Clonezilla’s Linux kernel isn’t prepared to recognize.  Otherwise, during boot, you will be dropped to a shell in the initramfs itself.  The system I was attempting to restore had a QLogic card which Clonezilla wasn’t prepared to handle.

Solution

Add the required firmware to the initramfs so it’s available for the module to load.

Process

Since Initramfs is essentially a concatenation of gzipped cpio archives which are extracted into a ramdisk and used as an early userspace by the Linux kernel, the only thing we need to do is to concatenate another file onto the archive.

The Q&A presented here opened my eyes to how easy this can be done.  This quote specifically cleared up any confusion I had:

Debian Installer’s initrd.gz is in fact a single gzipped cpio archive containing all the files the installer needs at boot time. By simply appending another gzipped cpio archive – containing the firmware files we are missing – we get the show on the road!

This means, all I had to do was:

    1. Extract the Clonezilla ISO
      mount -t iso9660 -o loop ./clonezilla-live.iso myiso
      # copy contents from read-only directory to a writable one
      cp -a myiso myiso_write
      cd myiso_write
    2. Concatenate my firmware onto the initramfs contained within the ISO
      echo '/lib/firmware/ql2500_fw.bin' | cpio -H newc -o | gzip >> live/initrd.img
    3. Repackage the ISO
      genisoimage -A 'My Clonezilla Installer' -f -r -hide-rr-moved -hide-joliet-trans-tbl \
           -J -l -allow-limited-size -b syslinux/isolinux.bin -c syslinux/boot.cat -no-emul-boot \
           -boot-load-size 4 -boot-info-table -eltorito-alt-boot -efi-boot boot/grub/efiboot.img \
           -no-emul-boot ./ > ../clonezilla-live-PJA.iso
    4. Boot from ISO
    5. Restore system

I had the easy task of just adding another file to the cpio archive.  If you need to modify existing files or structure or just unpack it in full for some other reason, you’ll need to consult this link – Modifying the Clonezilla initrd.

The file initrd.img from the Clonezilla live is not a ext2 file system, it’s cpio format. Therefore you can not mount it, instead you have to do something like this:

The initrd.img maybe in gzip format, or in xz format. You can use command “file initrd.img” to know the foramt.

(1) mkdir ~/tmp/initrd; cd ~/tmp/initrd

(2) for gzip format, run: zcat initrd.img | cpio -idm
    for xz format, run: xzcat initrd.img | cpio -idm

Then you can edit the files in ~/tmp/initrd. After that, you can use the following command to pack it as new initrd.img:

(3) cd ~/tmp/initrd

(4) For gzip format, run: find . | cpio --quiet -o -H newc | gzip -9 > ../initrd.img
    For xz format, run: find . | cpio --quiet -o -H newc | xz -c -9 --check=crc32 >  ../initrd.img

Then the new one is in ~/tmp/initrd.img

Configuring DD-WRT For My Environment

Wireless

Basic Setup Guides:

Kong

Knowledge Base:

Question: Is it better to have one SSID on 5GHz and another on 2.4GHz, or the same SSID for 5GHz and 2.4GHz?

One Answer: “I use same SSID on both bands. I set the TX Power higher on the 5Ghz so clients with 5Ghz capabilities will choose it.”

Actually: “TX increases noise = diminished signal quality.  TX can increase heat.  People often LOWER TX to increase throughput and stability.  The only reason I could see to raise the TX power is if you are using external antennae that have a long enough cable run as to cause a signal loss over the cable run.  The optimal TX-power is the LOWEST that’s still adequate for your needs. Better to reduce electrosmog than to increase it.  As for safety, more power = more heat = more potential for damage. Also, more power = more distortion of the signal = more packet loss.”

Current Setting:

Network Mode: Mixed (I have a mix of old and new clients)
WPA Algorithm: TKIP+AES
TX Power: default TX

 

DD-WRT on Netgear WNDR4500

Client OS: macOS Sierra 10.12.6

This router has been having strange issues since the day I got it.  It was given to me by a friend who installed dd-wrt on it.  I configured it for basic Internet connectivity with an Ethernet hand-off from my ISP but after a few days, it failed.  I couldn’t put my finger on it the first time it “flaked out” and I decided at that moment to just swap it out with a different router rather than troubleshoot the issue.  The issue wasn’t very straight-forward.  I would lose pings to it after a few minutes and I recall back then that I thought there was a memory leak that may have been causing it to reboot itself.  Why a memory leak?  Well, the ping round-trip times would gradually increase and sometimes spike to 2000ms+.  Sure, that doesn’t indicate a memory leak but I’m just saying, that’s what it felt like in the moment and I’m only mentioning this because the point is, it wasn’t something that just jumped out at me.

Then, I took it over to a friend’s house to swap their ancient WRT54G router out with a N900 router to see if they could get better coverage.  While onsite, this router did nothing that I asked it to do.  I couldn’t see any indication that it was even in working condition so for the second time, I just bagged it.

Now, it’s a rainy weekend and I have begun cleaning up my office some and this damn router is still sitting around and I’m still trying to figure out what to do with it.  This is the final straw.  If I can’t get some life out of it, it’s going in the trash (well, freecycle, actually).  Either way, I’m getting rid of it!

Let’s try just plugging it in.  After about a minute, I get a blinking green power LED.  I actually found this post from kb.netgear.com so yeah, sure, let’s do the necessary due diligence and see if I can reset it and gain connectivity to it.  I hit the reset button for 10+ seconds and for 30+ seconds and the light says blinking green.  Hmmm.  That doesn’t help at all.  Next, I did the reset where you pull the power, push in the reset button, then reattach power and that had the same results – no IP address and a blinking power light.

I decided to assign my NIC a static 192.168.1/24 IP address (Open System Preferences > go to Network > select Ethernet > under Configure IPv4 choose “Manually” and enter in the following information: IP Address = 192.168.1.111 Subnet Mask = 255.255.255.0 Router = <leave blank>).  Voilà!  I can ping 192.168.1.1.  I am not so sure this is going to work for everybody in a situation like this but it works for me!

cichlid-2:~ pja$ ping 192.168.1.1
64 bytes from 192.168.1.1: icmp_seq=140 ttl=100 time=0.345 ms
64 bytes from 192.168.1.1: icmp_seq=141 ttl=100 time=0.303 ms

Can I get to the web GUI?  Let’s try it out and see.  No luck on port 80 and 443.  I suppose I’ll port scan it to see if any services at all are up and running and bound to a port.  There is a built-in Network Utility that has a port scanner capability.  Open Spotlight > type in “Network Utility” > click on the Port Scan tab and enter in 192.168.1.1 as the target (Internet or IP address).  Click Scan.  This will take some time.  In fact, it takes a long, long time.  I specified a port range of 1-1024 (the standard reserved ports) and it still took an incredibly long time.  I decreased the range to 1-22 just to see what a response looked like and I still couldn’t get a response.  Decreasing this further to 1-2 and the utility still failed to give me any results.

I decided to try another tool.  I use home brew on my Mac.  I found a pretty handy script that gives you a rudimentary command line interface. brew install netcat and we’re ready.  Let’s scan ports 1 to 2048 on 192.168.1.1.  That looks something like this:

for PORT in {1..2048}; do netcat -vnz -w 1 192.168.1.1 $PORT; done

At least I can see what’s going on.  With the Network Utility port scanner, I couldn’t.  I could dig into that more to see if there was a way but netcat takes less time to get set up than all that other Apple stuff.

And we wait…

Most importantly, I need port 69 to respond if we’re going to have any chance to bring this thing back from the dead.  After about 15 minutes, I took the output from netcat, copied it to a text file, and grepped for anything that wasn’t “Operation timed out”.  No hits were returned.

Next, since this had dd-wrt on it before, I decided to do the 30/30/30 reset to it.  This may have been a mistake.  I found the reset instructions on this page, but this page says don’t do it on newer routers.  I don’t know what “newer” means but the alternate instructions mention the WPS button.  This router has a WPS button so maybe this router is a “newer” router.  Did I destroy it by using the 30/30/30 method?  I don’t know!  But what I found next suggests that I did NOT!

After the 30/30/30, I got more lights on the front of the router, like a solid green power LED, a green 2.4GHz LED, a 5GHz LED, and an LED on the switchport into which my laptop is plugged.  This is looking good.  I ran another netcat to see what that produced and the test happened at a much faster rate!  It took less than a minute to complete.  Wow!  What a difference.  Searching for ports that responded resulted in a familiar looking list:

cichlid-2:Downloads pja$ grep -v 'timed out' ./Netgear\ Port\ Scan\ after\ reset.txt | grep -v 'refused'
192.168.1.1 23 (telnet) open
192.168.1.1 53 (domain) open
192.168.1.1 80 (http) open

I opened a browser and went to http://192.168.1.1 and found that this router is using a beta from 2015.  I’m guessing that beta version had issues.  I’m okay trying a beta from 2017.  This one from Oct 10 to be precise.

At this point, it looks like I have a working router.  This escapade didn’t turn out to be exactly what I thought it would be.  I thought I’d be jtagging some stuff but I didn’t have to.  I’ve never jtagged anything and wanted to be sure I documented the entire process but, alas, there is nothing to document.  Just the basic troubleshooting steps I took to get to this point.

Next, configuring the dd-wrt router for my needs.

Setting Up a New Service in SystemD on CentOS 7

I have a custom service that I need to start on reboot on a CentOS system running systemd.  Following the guidelines posted here, I was able to do it myself in my environment.

  1. Create the following file: /etc/systemd/system/tcp-server.service
  2. Put the following contents in the file:
  3. [Unit]
    Description=tcp-server for hhreplay Service
    After=network.target
    [Service]
    Type=simple
    User=root
    Environment="OPTIONS=--logfile /tmp/tcp-server.log"
    ExecStart=/home/jenkins/tcp-server.py $OPTIONS
    Restart=on-abort[Install]
    WantedBy=multi-user.target
  4. Chown your log file: chown root:root /tmp/tcp-server.log
  5. Reload systemd: sudo systemctl daemon-reload
  6. Start your service to ensure it’s a functional service file: systemctl start tcp-server
  7. Check status: systemctl status tcp-server
  8. If it looks good, then set it to start at boot: systemctl enable tcp-server --now

MAN PAGE

In-depth information can be found in the systemd.service man page.  Read up on options such as Type, RemainAfterExit, ExecStart, WantedBy, ExecStop , Environment, etc.

Special Considerations

  1. In some distributions you are required to have a ExecStart in your service file.  This can be set to /bin/true if you don’t explicitly need something there.
  2. RemainAfterExit=true can be added to trick systemd into believing that the service is running.  This is helpful when you don’t have a ExecStart and still need a ExecStop to run after the service, or another service, shutsdown.
  3. DefaultDependencies=no means ignore all dependencies and run “first” on start and “last” on stop.  If you have Before or After clauses, they will still be honored.

Additional References

0pointer.netSystemD for Administrators Part XXI
freedesktop.orgsystemd System and Service Manager