Grep Basics – But, Are You Sure You Want to Use Grep?

Keep in mind: many things you want to do in grep you can do in awk.  Or, maybe there is just a better way.  I learned grep early on and have been forcing it to do things it wasn’t really designed to do, so here I will actually be listing solutions using other tools – tools that may be better designed for the task at hand.

Display Line Number of a Match

awk:

awk '/textstring/ {print FNR}' textfile

sed:

sed -n '/pattern/=' filename

Logical Volume Manager (LVM) on Linux

xfs_repair on LVM

Search for LVM information on all available hard drives and create all the needed entries in /dev so you can actually use them.

vgscan -v --mknodes

All recognizable LVM information were created under /dev/mapper  You’ll probably see something like this:

$ ls -al /dev/mapper
total 0
drwxr-xr-x. 2 root root 100 Apr 13 13:49 .
drwxr-xr-x. 20 root root 3100 Apr 13 13:49 ..
lrwxrwxrwx. 1 root root 7 Apr 13 13:49 centos-root -> ../dm-0
lrwxrwxrwx. 1 root root 7 Apr 13 13:49 centos-swap -> ../dm-1
crw-------. 1 root root 10, 236 Apr 13 13:49 control

The volume named centos-root is the logical volume that is mounted as the root partition.

Activate your volumes:

vgchange -a y

Repair the XFS filesystem:

xfs_repair /dev/mapper/centos-root

You may receive an error stating there is an unwritten (dirty, corrupt) log.  As this note from RedHat states, you may have data loss if you clear the log.

The xfs_repair utility cannot repair an XFS file system with a dirty log. To clear the log, mount and unmount the XFS file system. If the log is corrupt and cannot be replayed, use the -L option (“force log zeroing”) to clear the log, that is, xfs_repair -L /dev/device. Be aware that this may result in further corruption or data loss.

If you can’t mount the volume and clear your log, you will have to use -L and risk data loss.

xfs_repair messageI haven’t experienced any serious data loss after using -L, or if I did, it wasn’t apparent to me.  ** Use at your own risk **

Debian Networking

Using a Bridge

$ cat /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
iface eno1 inet manual

auto vmbr0
iface vmbr0 inet static
  address 172.16.16.111
  netmask 255.255.255.0
  gateway 172.16.16.1
  dns-domain mydomain.local
  dns-nameservers 8.8.4.4 8.8.8.8
  bridge_ports eno1
  bridge_stp off
  bridge_fd 0

Out-of-Band Management

IPMI

…For Fun and Profit!

Users
# Let's look at what we currently have before adding anything
$ ipmitool user list
ID Name              Callin Link Auth   IPMI Msg   Channel Priv Limit
1                    true   false       false      Unknown (0x00)
2 ADMIN              true   false       false      Unknown (0x00)
...

$ ipmitool user set name 3 devopsuser
$ ipmitool user list
ID  Name	     Callin  Link Auth	IPMI Msg   Channel Priv Limit
1                    true    false      false      Unknown (0x00)
2   ADMIN            true    false      false      Unknown (0x00)
3   devopsuser       true    false      false      Unknown (0x00)
...

$ ipmitool user set password 3
Password for user 3: 
Password for user 3: 
Set User Password command successful (user 3)
$ ipmitool channel setaccess 1 3 link=on ipmi=on callin=on privilege=4
Set User Access (channel 1 id 2) successful.
$ ipmitool user enable 3
 

Serial

In my lab, I began adopting serial as an alternative to IPMI and standard Layer3 management interfaces.  Serial comes with its own set of problems.  Mostly, redirecting to serial.  There are several things to be cognizant of.

  1. The stage(s) of a system’s boot procedure that are responsible for output redirection
  2. The inputs and outputs of a system
  3. The behavior of a system when more than one redirection is possible

References

Thomas-Krenn.com

System Administration Links

chroot

An excellent guide from Bart Simons

Classics

hunter2

Support and Help

Password Recovery

[CentOS] Reset Root Password (single user mode)

If you don’t remember the root password, or you just want to boot in the most minimal way, you can add init=/bin/bash to your linux line to bypass the regular init sequence and just drop to a shell. The line should look like

linux /vmlinuz-xxx root=xxx ro init=/bin/bash

Whatever is after vmlinuz and root (where the xxxs are) leave as-is.  You can remove everything else on that line except for the leading word linux, vmlinuz, root and ro.  Add init=/bin/bash

The system ought to boot and present a bash shell running as user root.  If your system has full-disk encryption, you will need to enter the encryption password.

At this command line, run the following commands:

mount -o remount,rw /
mount /proc

Then you can view and modify the user database. The main user database file is /etc/passwd. It contains user names (for both physical users and system accounts), but passwords are in a different file /etc/shadow. If you’ve forgotten a password, you can’t recover it, all you can do is change it.

If you want to change the password for an account, run

passwd root

to change user root’s password.

If you see a token manipulation error then make sure you booted into an environment that supports read/write (use rw in linux line or remount as rw).

Once you’re finished, you can reboot.

umount /
reboot

[CentOS] Reset Root Password (live CD chroot mode)

  1. Boot into your live environment
  2. Make directory on which to mount your partition
    mkdir ./mnt || /bin/true
  3. Identify your disks using fdisk
    fdisk -l
  4. Look for the bootable partition of type Linux
  5. If your partition is a standard Linux partition:
    mount /dev/sda1 ./mnt
  6. If your partition is a Linux LVM:
    pvscan
    vgscan
    vgchange -a y
    lvscan
    mount /dev/sda1 ./mnt
    mount --bind /dev ./mnt/dev<
    mount --bind /proc ./mnt/proc
    mount --bind /sys ./mnt/sys
    
  7. Change root into your mounted directory
    chroot ./mnt /bin/bash
  8. Type passwd and enter a new password
  9. Type reboot to reboot your system
Provisioning

Foreman – PXEless Bare Metal & Virtual Machine Provisioning

Volume Management

[LVM] Working With LVM Volumes

 

Git Stuff

Storing Screts

“Why should I worry? Our GitHub repository is private, and only our team can access the secrets!”

— Safelyl Storing Secrets in Git
EmbeddedArtistry.com

Even if you don’t want to concern yourself with it, consider it a challenge to keep your secrets secret.


Tips & Tricks

Tell git which key to use

Same way you tell any SSH session which key to use.

# vi ~/.ssh/config

Host github.com
    Hostname %h
    User git
    IdentityFile ~/.ssh/github_id_rsa

Workflow

Update Local with Remote

Use git pull --rebase

Rebase Local Master with Remote Master
git stash                               # optional
git checkout master
git fetch origin
git rebase origin/master

IPTables – Log Everything

First, this allows all traffic in and logs those requests.

Secondly, the final rules could probably be tweaked.  I may do another post to examine this set very, very closely.

NOTE: if you’re making changes from a remote location, make sure you tell your firewall to let you in.  If you already have rules in place, this will do no harm.  If you have no rules in place, this will hopefully make it clear to you and your system that SSH needs to be let through.  The -s <start-ip> <end-ip> is optional.

iptables -A INPUT -s <start-ip> <end-ip> -p tcp --dport 22 -j ACCEPT

View current distro info (if you’re not sure):

root@aptly:~# lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 7.8 (wheezy)
Release: 7.8
Codename: wheezy

Based on the distro info, do we know where iptables configs are being stored?  From experience, I would suspect /etc/sysconfig (RHEL flavors) or /etc/iptables (Debian flavors) but neither directory exists in my case.  Taking a look at iptables -L -n, there are no rules loaded.  So, maybe I’ll just create my own and load them at boot.  Also, take a look at netstat -tlnp to see what is currently listening and be sure not to block anything that you know you need.

# netstat -tlnp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State      PID/Program name
tcp        0      0 0.0.0.0:111   0.0.0.0:*       LISTEN     1642/rpcbind
tcp        0      0 0.0.0.0:22    0.0.0.0:*       LISTEN     2519/sshd
tcp        0      0 127.0.0.1:25  0.0.0.0:*       LISTEN     2471/exim4
tcp        0      0 0.0.0.0:58397 0.0.0.0:*       LISTEN     1673/rpc.statd
tcp        0      0 0.0.0.0:10050 0.0.0.0:*       LISTEN     2357/zabbix_agentd
tcp6       0      0 :::111        :::*            LISTEN     1642/rpcbind
tcp6       0      0 :::22         :::*            LISTEN     2519/sshd
tcp6       0      0 ::1:25        :::*            LISTEN     2471/exim4
tcp6       0      0 :::36063      :::*            LISTEN     1673/rpc.statd
tcp6       0      0 :::10050      :::*            LISTEN     2357/zabbix_agentd

Now, let’s begin…

Start by making a place for your rules.

mkdir /etc/iptables
vi /etc/iptables/iptables.ipv4

Add the following, or something like it:

# INPUT
iptables -N LOGGING

# Allows all loopback (lo0) traffic and drop all traffic to 127/8 that doesn't use lo0
iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT ! -i lo -d 127.0.0.0/8 -j REJECT

# Accepts all established inbound connections
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j LOGGING

# Allows SSH connections
# The --dport number is the same as in /etc/ssh/sshd_config
iptables -A INPUT -p tcp -m state --state NEW --dport 22 -j LOGGING

# Allow ping
# note that blocking other types of icmp packets is considered a bad idea by some
# remove -m icmp --icmp-type 8 from this line to allow all kinds of icmp:
# https://security.stackexchange.com/questions/22711
iptables -A INPUT -p icmp -m icmp --icmp-type 8 -j LOGGING

# log iptables denied calls (access via 'dmesg' command)
iptables -A INPUT -j LOGGING

iptables -A LOGGING -m limit --limit 4/min -j LOG --log-prefix "iptables-accepted: " --log-level 4
iptables -A LOGGING -j ACCEPT

# OUTPUT
# Allows all outbound traffic
iptables -A OUTPUT -m state --state NEW,RELATED,ESTABLISHED -j ACCEPT
iptables -A OUTPUT -j ACCEPT

# Reject all other inbound - default deny unless explicitly allowed policy:
iptables -A INPUT -j REJECT

# iptables -A FORWARD -i eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -j REJECT

Save your changes. Note, the commands in this file are to be *run* on the command line, NOT *loaded* by iptables. So…load this series of iptables commands:

bash /etc/iptables/iptables.ipv4

Assuming you did not lose contact with your host, check the state of your firewall, now.

# iptables -L -n
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:22
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
REJECT all -- 0.0.0.0/0 127.0.0.0/8 reject-with icmp-port-unreachable
LOGGING all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
LOGGING tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:22
LOGGING icmp -- 0.0.0.0/0 0.0.0.0/0 icmptype 8
LOGGING all -- 0.0.0.0/0 0.0.0.0/0
REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable

Chain FORWARD (policy ACCEPT)
target prot opt source destination
REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable

Chain OUTPUT (policy ACCEPT)
target prot opt source destination
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 state NEW,RELATED,ESTABLISHED
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0

Chain LOGGING (5 references)
target prot opt source destination
LOG all -- 0.0.0.0/0 0.0.0.0/0 limit: avg 4/min burst 5 LOG flags 0 level 4 prefix "iptables-accepted: "
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0

I will note here first that as it stands now, this may not be the most efficient use of the rules.  I doesn’t seem to be the most succinct BUT for now, it’s okay.

Save the current rules to an IPTables config.

iptables-save > /etc/iptables/rules.v4

This is what the rules.v4 file contains:

# Generated by iptables-save v1.4.14 on Thu Mar 12 10:04:00 2020
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:LOGGING - [0:0]
-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -d 127.0.0.0/8 ! -i lo -j REJECT --reject-with icmp-port-unreachable
-A INPUT -m state --state RELATED,ESTABLISHED -j LOGGING
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j LOGGING
-A INPUT -p icmp -m icmp --icmp-type 8 -j LOGGING
-A INPUT -j LOGGING
-A INPUT -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -j REJECT --reject-with icmp-port-unreachable
-A OUTPUT -m state --state NEW,RELATED,ESTABLISHED -j ACCEPT
-A OUTPUT -j ACCEPT
-A LOGGING -m limit --limit 4/min -j LOG --log-prefix "iptables-accepted: "
-A LOGGING -j ACCEPT
COMMIT
# Completed on Thu Mar 12 10:04:00 2020

Now…normally, I would follow methods that are more generally acceptable for making sure these rules are loaded each reboot (see iptables-persistent).  However, the system on which I’m setting this up is a system that is being retired.  The system is providing legacy services and cannot be updated under any circumstances.  This exercise was begun to simply determine what services or servers are accessing this server so I don’t want to do anything too intrusive, like updating packages or even installing packages, for that matter.  Therefore, I will load these rules at boot time via cron.

@reboot /sbin/iptables-restore < /etc/iptables/rules.v4

One more test:

# iptables -F
# iptables-restore /etc/iptables/rules.v4
# iptables -L -n

And now reboot to test it all.

I’m seeing some ntp that I didn’t account for:

Mar 12 10:19:25 aptly kernel: [ 6.316149] iptables-accepted: IN=eth0 OUT= MAC=9a:6c:12:75:a3:20:00:1c:73:8d:05:d7:08:00 SRC=88.99.76.254 DST=10.61.75.72 LEN=76 TOS=0x18 PREC=0x00 TTL=50 ID=28740 DF PROTO=UDP SPT=123 DPT=123 LEN=56

What the heck is 88.99.76.254?  Seems legit, according to this post, but it did remind me that I ought to update my NTP servers to point to our internal ones.