Linux on Benny Simmonds

Default programs based on MIME type with XDG

Tue, 01 Feb 2022 18:12:29 +1100

From an ancient post I previously did…I need to refresh my mind on this topic often enough thought it worthy of breaking it out.

How Linux systems figure out what program should open a file

Programs that handle arbitrary files (e.g. web browsers, irc clients, file managers) delegate to a general purpose resource handler. XDG MIME Applications is the ubiquitous option here, and is not only an implementation, but a full blown specification.

Debian install notes

Sun, 11 Jul 2021 11:55:08 +0000

I’m moving lots of my machines back to Debian in the yearning for rock solid stability and a complete distribution that just works. Debian was the first GNU/Linux I ever used as a teenager back in late 90s and I’ve fond memories of it.

Starting with the official amd64 install guide.

Preparation

Make USB key

I quickly discovered I want the image that includes proprietary firmware, so I can get my (nasty) wifi card working as quickly as possible.

Gentoo Linux

Mon, 22 Feb 2021 21:29:55 +0000

The next step in my minimalist computing journey.

Enter Gentoo, my first source based GNU/Linux distro. Pre-packaged binaries, which is the approach most other (binary based) distros take, must often cater for the lowest common denominator to ensure packages can run on lots of differing setups out in the wild. On a source based distro, I can articulate my specific needs (USE flags on Gentoo) to finely tune the binaries to my system. For example, as I plan to steer clear of software like systemd, kde and gnome, I can ensure support for these packages is NOT built into other program binaries I build for my system.

KVM virtualisation

Sat, 31 Oct 2020 20:56:23 +0000

The Kernel Virtual Machine is a hypervisor for Linux on hardware with virtualization extensions (Intel VT or AMD-V). It is deployed as a loadable kernel modules, kvm.ko, and either kvm-intel.ko or kvm-amd.ko.

The KVM Debian Wiki rocks, and provides details on the basics including a great performance tuning section.

Install

Easy instructions to get QEMU/KVM and virt-manager up and running on Arch.

Kubernetes

Tue, 18 Aug 2020 20:11:30 +0000

The name Kubernetes originates from Greek, meaning helmsman or pilot.

Terminology

k8s is two concepts; the control plane and nodes.

Installing Arch Linux on the Pinebook Pro

Mon, 27 Jul 2020 20:27:23 +0000

Updated 2022-04-11: Installed a minimal version of Manjaro, a SLICK flavour of Arch

The pinebook pro is a beautiful 64-bit ARM based laptop, that reminds me of the form factor of a modern macbook air, shipping with a premium magnesium alloy shell, 64GB eMMC and a 10,000 mAH battery. All this for $200.

As a NIX machine, I’ve found Manjaro to be delightful. I have dreams of one day installing OpenBSD.

Digital Signatures

Fri, 24 May 2019 21:17:01 +0000

The sequence of tasks undertaken that make digital signatures possible. This does have a slight XML flavour to it.

A digital signature is a mathematical scheme for verifying the authenticity of digital messages.

The concept of digital signature completely hinges on assymetric cryptography (such as DSA or RSA).

To validate a signature

First the message can be normalised, and in the case of XML will use something like the “Exclusive XML Canonicalization” (XML-C14N), so we’re comparing apples with apples. This will disgard things like usage of white space.
Using the normalised representation, compute a hash (e.g. SHA1) of the timestamp (contained WS-Security header) and entire message payload (the SOAP body).
Using the public key from the partner organisation certificate, RSA decrypt the hash computed by partner organisation.
If the two hashes are identical, we know the message has not been tampered with.
(optional) Validate the timestamp (TTL) defined by partner organisation (typically 7 minutes from the original transmission time by the sender). To mitigate possible damage caused by replay attacks.

To create a signature

Wraps the response message in a SOAP envelope, which includes some WS-Security related headers including a timestamp.
The timestamp is set to a configurable number of minutes (e.g. 10 minutes) in the future.
Normalises the message using the “Exclusive XML Canonicalization” (XML-C14N)
Using the normalised message form, compute a (e.g. SHA1) hash of the timestamp (WS-Security header) and entire response message payload (e.g. the SOAP body).
Uses the private key of signing certificate, RSA signs the computed hash, and stores the result in the relevant security header (the SignatureValue header).
The message is then delivered to partner organisation.

For the above to work, there needs to be some established agreement as to the specific cipher suites and canonicalisation method used. This is all

Arch Linux

Sat, 06 Apr 2019 11:32:10 +1100

After witnessing insane minimalism paired with a tiler (tiling window manager), knew it was my time to take the pilgrimage to Arch Linux.

Some characteristics that make Arch unique:

The Arch Way embody the principles behind Arch Linux; simplicity, modernity, pragmatism, user centrality and versatility.
Forces one to build the system up by hand.
This encourages you to question the role of each component of the system, and available options to satisfy that component (e.g. the terminal emulator).
The result is a highly tailored and minimal system that meets precisely your needs.
Practical and pragmatic documentation. The Arch Wiki is the gold standard when it comes to documentation.
The Arch User Repository (AUR) is a treasure chest of pre-packaged useful recent software. Somehow every program I’ve ever needed has been available on AUR.
Rolling upgrades.

Arch was born in 2001, when Canadian programmer Judd Vinet, inspired by the elegance of systems such as Slackware and the BSD’s, set out to build his own distro based on a similar ethos. The first formal release, 0.1, dropped on March 11, 2002.

LCA 2018

Sun, 28 Jan 2018 17:09:01 +0000

I’m a convert. This conference was incredible, with passionate presenters, excellent content with an amazing inclusive and friendly community vibe. Most sessions are available on YouTube.

Things I am inspired to do post LCA 2018:

Setup ctags with Vim. Think ‘go to definition’ for Vim, with support for 40+ langs. Specifically Exuberant Ctags and unite-tag.
Play with more kernel syscalls (like ptrace).
Signal processing and logic analyser lab and kit. More notes.
Resume the nand2tetris journey, part 2.
Use IRC more. Some CLI options Irssi BitchX WeeChat.
And keybase. Imagine a Slack for the whole world, except end-to-end encrypted across all your devices. Or a Team Dropbox where the server can’t leak your files or be hacked.
Always support and subscribe to lwn.net.
Look into supporting the Software Conservancy Foundation (SCF)
Pick up some new langs; go and rust seem to be current shiny.
I met a number of kernel developers. Find out a bit more about what they do through the offical MAINTAINERS file.
Yubikey or Nitrokey, start using. Open source physical USB device to enable secure encryption and signing of data. The secret keys are always stored inside the Nitrokey which protects against malware (such as computer viruses) and attackers. A user-chosen PIN and a tamper-proof smart card protect the Nitrokey in case of loss and theft.
Linux Device Drivers book now maintained by Jonathan Corbet (executive editor at lwn.net), start reading, and create drivers, the gateway drug to the kernel.
Time to refresh handle, bio, and CV.
Checkout TaskWarrior, a CLI for managing tasks.
Consider using more compact unique IDs such as Twitters snowflake e.g. 12738165059
Use more state machines.
Study The Architecture of Open Source Applications.

Some of the sessions I got to attend.

LXC (Lexy)

Thu, 18 Jan 2018 20:56:01 +0000

Born in 2008, LXC (pronouced lexy) is a userspace interface for the Linux kernel containment features, that enables the creation and management of application containers.

LXC leverages a number of kernel features to contain processes:

Kernel namespaces (ipc, uts, mount, pid, network and user)
Apparmor and SELinux profiles
Seccomp policies
Chroots (using pivot_root)
Kernel capabilities
CGroups (control groups)

LXC containers are often considered as something in between a chroot and a full blown VM. The goal of LXC is to create an environment as close as possible to a standard Linux installation, without the need for a separate kernel.

Ansible

Sun, 31 Dec 2017 18:09:01 +0000

Ansible is a radically simple IT automation platform that makes your applications and systems easier to deploy. Avoid writing scripts or custom code to deploy and update your applications. Automate in a language that approaches plain English, using SSH, with no agents to install on remote systems.

With Ansible and its amazing module ecosystem, you describe what needs to be accomplished (i.e. declarative), rather than describing how to accomplish each step (i.e. imperative).

Vagrant

Sat, 30 Dec 2017 20:09:01 +0000

When something becomes messy in the software industry, we have the habit of tucking the problems away behind abstractions. To me Vagrant is simply an abstraction layer for working with a range of hypervisors (libvirt, VirtualBox, VMWare, Hyper-V, Docker and more), in a reliable and consistent manner.

By articulating VM’s as a simple text based (Ruby) configuration file, a Vagrantfile, achieve greater simplicity, portability across systems, and consistency which could theoretically be used with any virtual environment running on top of any other system.

Linux Storage and File Systems

Mon, 12 Jun 2017 20:59:20 +0000

Partitioning

Two popular partition schemes are used in the wild, MBR and GPT.

MBR

MBR, or Master Boot Record, often associated with BIOS, was introduced in 1983 with IBM PC DOS 2.0, is a special boot sector located at the beginning of a drive. This sector contains a boot loader (e.g GRUB), and details about the logical partitions. MBR supports drives upto 2TiB, and up to 4 primary partitions.

# fdisk /dev/vda
Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x9228f9b7.

Command (m for help): m
Command action
   a   toggle a bootable flag
   b   edit bsd disklabel
   c   toggle the dos compatibility flag
   d   delete a partition
   g   create a new empty GPT partition table
   G   create an IRIX (SGI) partition table
   l   list known partition types
   m   print this menu
   n   add a new partition
   o   create a new empty DOS partition table
   p   print the partition table
   q   quit without saving changes
   s   create a new empty Sun disklabel
   t   change a partition's system id
   u   change display/entry units
   v   verify the partition table
   w   write table to disk and exit
   x   extra functionality (experts only)

Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p):
Using default response p
Partition number (1-4, default 1):
First sector (2048-2097151, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-2097151, default 2097151): +500M
Partition 1 of type Linux and of size 500 MiB is set

Command (m for help): l
 0  Empty           24  NEC DOS         81  Minix / old Lin bf  Solaris
 1  FAT12           27  Hidden NTFS Win 82  Linux swap / So c1  DRDOS/sec (FAT-
 2  XENIX root      39  Plan 9          83  Linux           c4  DRDOS/sec (FAT-
 3  XENIX usr       3c  PartitionMagic  84  OS/2 hidden C:  c6  DRDOS/sec (FAT-
 4  FAT16 <32M      40  Venix 80286     85  Linux extended  c7  Syrinx
 5  Extended        41  PPC PReP Boot   86  NTFS volume set da  Non-FS data
 6  FAT16           42  SFS             87  NTFS volume set db  CP/M / CTOS / .
 7  HPFS/NTFS/exFAT 4d  QNX4.x          88  Linux plaintext de  Dell Utility
 8  AIX             4e  QNX4.x 2nd part 8e  Linux LVM       df  BootIt
 9  AIX bootable    4f  QNX4.x 3rd part 93  Amoeba          e1  DOS access
 a  OS/2 Boot Manag 50  OnTrack DM      94  Amoeba BBT      e3  DOS R/O
 b  W95 FAT32       51  OnTrack DM6 Aux 9f  BSD/OS          e4  SpeedStor
 c  W95 FAT32 (LBA) 52  CP/M            a0  IBM Thinkpad hi eb  BeOS fs
 e  W95 FAT16 (LBA) 53  OnTrack DM6 Aux a5  FreeBSD         ee  GPT
 f  W95 Ext'd (LBA) 54  OnTrackDM6      a6  OpenBSD         ef  EFI (FAT-12/16/
10  OPUS            55  EZ-Drive        a7  NeXTSTEP        f0  Linux/PA-RISC b
11  Hidden FAT12    56  Golden Bow      a8  Darwin UFS      f1  SpeedStor
12  Compaq diagnost 5c  Priam Edisk     a9  NetBSD          f4  SpeedStor
14  Hidden FAT16 <3 61  SpeedStor       ab  Darwin boot     f2  DOS secondary
16  Hidden FAT16    63  GNU HURD or Sys af  HFS / HFS+      fb  VMware VMFS
17  Hidden HPFS/NTF 64  Novell Netware  b7  BSDI fs         fc  VMware VMKCORE
18  AST SmartSleep  65  Novell Netware  b8  BSDI swap       fd  Linux raid auto
1b  Hidden W95 FAT3 70  DiskSecure Mult bb  Boot Wizard hid fe  LANstep
1c  Hidden W95 FAT3 75  PC/IX           be  Solaris boot    ff  BBT
1e  Hidden W95 FAT1 80  Old Minix

Command (m for help): t
Selected partition 1
Hex code (type L to list all codes): 83
Changed type of partition 'Linux' to 'Linux'

Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.

The partition is now available as a block device, below we now see /dev/vda1:

systemd

Sat, 20 May 2017 20:52:20 +0000

Cheatsheet

==========================================          ==========================================
             INFORMATION                                         LOGS                         
==========================================          ==========================================
                                                                                              
systemctl                                           journalctl                                
    list-dependencies                                   -u <name>  by service                 
    list-sockets                                        -f         follow live log            
    list-jobs                                           -k         kernel messages            
    list-unit-files                                     -k         kernel messages            
    list-units                                          -r         reverse (newest first)     
    get-default                                         -o json    format output              
    --type=service --all                                -b -1      from previous boot         
                                                        -n 10      limit output lines (tail)  
==========================================              -S,--since e.g "2012-10-30 18:17:16"  
             SERVICES                                   -U,--until e.g. "10 min ago"          
==========================================                                                    
                                                                                              
systemctl                                           ==========================================
    stop <name>                                                  SYSTEM STATES                
    start <name>                                    ==========================================
    restart <name>                                                                            
    reload <name>                                   systemctl                                 
    status <name>                                       reboot       reboot.target            
    enable <name>                                       poweroff     poweroff.target          
    disable <name>                                      emergency    emergency.target         
    show <name>                                         default      default.target           
    -H <host> status network

Red Hat Certified System Administrator 8 (RHCSA)

Wed, 03 May 2017 20:51:43 +0000

I’m a software engineer not a sysadmin. In fact I know I’m a bad one. The RHCSA is a rote, mechanical process. It did force me to discover discover several gaps in my GNU/Linux knowledge so overall happy about that.

Essential Tools

Documentation

The ability to understand a program using local documentation resources; man, info, /usr/share/doc, within the RPM package.

Tmux Quick Reference

Mon, 17 Apr 2017 20:17:10 +0000

Kudos to afair for putting together this neat Tmux Cheat Sheet, which I’m addicted to at the moment.

==========================================          ==========================================
             TMUX COMMAND                                        WINDOW (TAB)
==========================================          ==========================================
                                                                                              
List    tmux ls                                     List         ^b w
New          -s <session>                           Create       ^b c
Attach       att -t <session>                       Rename       ^b , <name>
Rename       rename-session -t <old> <new>          Last         ^b l               (lower-L)
Kill         kill-session -t <session>              Close        ^b &
                                                                                              
==========================================          Goto #       ^b <0-9>
             CONTROLS                               Next         ^b n
==========================================          Previous     ^b p
                                                    Choose       ^b w <name>
Detach       ^b d
List         ^b =                                   ==========================================
Buffer       ^b <PgUpDn>                                         PANE (SPLIT WINDOW)
Command      ^b : <command>                         ==========================================
                                                                                              
Copy         ^b [ ... <space> ... <enter>           Show #       ^b q
 Moving         vim/emacs key bindings              Split Horiz  ^b "                --------
 Start          <space>                             Split Vert   ^b %                   |
 Copy           <enter>                             Pane->Window ^b !
Paste        ^b ]                                   Kill         ^b x
                                                                                              
==========================================          Reorganize   ^b <space>
             SESSION (Set of Windows)               Expand       ^b <alt><arrow>
==========================================          Resize       ^b ^<arrow>
                                                    Resize x n   ^b <n> <arrow>
New          ^b :new     ^b :new -s <name>
Rename       ^b $                                   Select       ^b <arrow>
List         ^b s                                   Previous     ^b {
Next         ^b (                                   Next         ^b }
Previous     ^b )                                   Switch       ^b o                  other
                                                    Swap         ^b ^o
                                                    Last         ^b ;

Linux cheatsheet

Sat, 18 Feb 2017 21:06:01 +1100

A survey of the standard and high quality programs that feature in most Unix based distributions, with the GNU variants being my favourite. The bash shell is a great way of interfacing and orchestrating these beautifully crafted programs. As a starting point, I’ve listed each program offered by the GNU Core Utilities and util-linux umbrella projects; considered the de facto standard on most distributions.

Quick Reference

General

Command	What is does
`apropos compress`	Show commands that relate to a keyword
`man -t ascii \| ps2pdf - > ascii.pdf`	Make a PDF of a man page
`which command`	Full path of command
`time command`	Show execution time of a given command
`time cat`	Start stopwatch, ^d to stop
`cat file.txt \| xclip -selection clipboard`	Copy to clipboard
`nohup ./script.sh &`	Keep program running after leaving SSH session (see bash post if input needed)
`timeout 20s ./script.sh`	Run script.sh for 20 seconds only
`while true; do timeout 30m ./script.sh; done`	Restart a program every 30 minutes

System Information

Command	What is does
`uname -a`	Show kernel version and system architecture
`head -n1 /etc/issue`	Show name and version of distribution
`cat /proc/partitions`	Show all partitions registered on the system
`grep MemTotal /proc/meminfo`	Show RAM total seen by the system
`grep "model name" /proc/cpuinfo`	Show CPU(s) info
`lspci -tv`	Show PCI info
`lsusb -tv`	Show USB info
`mount \| column -t`	List mounted filesystems on the system (and align output)
`grep -F capacity: /proc/acpi/battery/BAT0/info`	Show state of cells in laptop battery
`dmidecode -q \| less`	Display SMBIOS/DMI information
`smartctl -A /dev/sda \| grep Power_On_Hours`	How long has this disk (system) been powered on in total
`hdparm -i /dev/sda`	Show info about disk sda
`hdparm -tT /dev/sda`	Do a read speed test on disk sda
`badblocks -s /dev/sda`	Test for unreadable blocks on disk sda

Command	What is does
`cd -`	Go previous directory
`cd`	Go home
`(cd dir123 && pwd)`	Jump into a directory, run a command there, and return to origin
`pushd .`	Put cwd on stack, so you can `popd` back to it

File Searching

Command	What is does
`alias l='ls -l --color=auto'`	Quick listing
`ls -lrt`	List long by date
`ls -lS`	List long by size
`ls /usr/bin \| pr -T9 -W$COLUMNS`	Print in 9 columns to width of terminal
`find -name '*.[ch] \| xargs grep -E 'foo'`	Search for ‘foo’ in all `.c` and `.h` files in cwd and below
`find -type f -print0 \| xargs -r0 grep -F 'example'`	Search all regular files for ’example’
`find -maxdepth 1 -type f \| xargs grep -F 'example'`	As above, but don’t recurse
`find -maxdepth 1 -type d \| while read dir; do echo $dir; echo somecmd; done`	Wash each result over multiple commands
`find -type f ! -perm -444`	Find files not readable by all
`find -type d ! -perm -111`	Find dirs not accessable by all
`find . -size 30c`	By file size (30 bytes)
`find . -name "*.gz" -delete`	Delete all gz files
`locate -r 'file[^/]*\.txt`	Search cached index for names
`look <keyword>`	Search English dictionary with a given prefix keyword
`grep --color reference /usr/share/dict/words`	Highlight occurances of regex against English dictionary
`readlink -f file.txt`	Full path of file
`namei -l /bin/bash`	Drills through directories and links showing permission mask all the way down

Archiving and Compression

Command	What is does
gpg -c file	Encrypt file
gpg file.gpg	Decrypt file
tar -c dir/ \| bzip2 > dir.tar.bz2	Make compressed archive of dir
bzip2 -dc dir.tar.bz2 \| tar -x	Extract archive
tar -c dir/ \| gzip \| gpg -c \| ssh user@remote ‘dd of=dir.tar.gz.gpg’	Make encrypted archive of `dir` on remote machine
find dir/ -name ‘*.txt’ \| tar -c –files-from=- \| bzip2 > dir_txt.tar.bz2	Make archive of subset of `dir` and below
find dir/ -name ‘*.txt’ \| xargs cp -a –target-directory=dir_txt/ –parents	Make copy of subset of `dir` and below
( tar -c /dir/to/copy ) \| ( cd /where/to/ && tar -x -p )	Copy (with permissions) copy/ dir to /where/to/ dir
( cd /dir/to/copy && tar -c . ) \| ( cd /where/to/ && tar -x -p )	Copy (with permissions) contents of copy/ dir to /where/to/
( tar -c /dir/to/copy ) \| ssh -C user@remote ‘cd /where/to/ && tar -x -p’	Copy (with permissions) copy/ dir to remote:/where/to/ dir
dd bs=1M if=/dev/sda \| gzip \| ssh user@remote ‘dd of=sda.gz’	Backup harddisk to remote machine

Networking

Command	What is does
`ethtool eth0`	Show status of ethernet interface eth0
`ethtool --change eth0 autoneg off speed 100 duplex full`	Manually set ethernet interface speed
`iw dev wlan0 link`	Show link status of wireless interface wlan0
`iw dev wlan0 set bitrates legacy-2.4 1`	Manually set wireless interface speed
`iw dev wlan0 scan`	List wireless networks in range
`ip link show`	List network interfaces
`ip link set dev eth0 name wan`	Rename interface eth0 to wan
`ip link set dev eth0 up`	Bring interface eth0 up (or down)
`ip addr show`	List addresses for interfaces
`ip addr add 1.2.3.4/24 brd + dev eth0`	Add (or del) ip and mask (255.255.255.0)
`ip route show`	List routing table
`ip route add default via 1.2.3.254`	Set default gateway to 1.2.3.254
`ss -tupl`	List internet services on a system
`ss -tup`	List active connections to/from system
`host bencode.net`	Lookup DNS ip address for name or vice versa
`hostname -i`	Lookup local ip address (equivalent to host `hostname`)
`whois bencode.net`	Lookup whois info for hostname or ip address
`mtr google.com`	Nice trace route

Text Manipulation

Command	What is does
`sed 's/string1/string2/g'`	Replace string1 with string2
`sed 's/$.*$1/\12/g'`	Modify anystring1 to anystring2
`sed '/^ #/d; /^ $/d'`	Remove comments and blank lines
`sed ':a; /\\$/N; s/\\\n//; ta'`	Concatenate lines with trailing \
`sed 's/[ \t]*$//'`	Remove trailing spaces from lines
`seq 10 \| sed "s/^/ /; s/ *$.\{7,\}$/\1/"`	Right align numbers
`seq 10 \| sed p \| paste - -`	Duplicate a column
`sed -n '1000{p;q}'`	Print 1000th line
`sed -n '10,20p;20q'`	Print lines 10 to 20
`sed -n 's/.<title>$.$<\/title>.*/\1/ip;T;q'`	Extract title from HTML web page
`sed -i 42d ~/.ssh/known_hosts`	Delete a particular line
`sort -t. -k1,1n -k2,2n -k3,3n -k4,4n`	Sort IPV4 ip addresses
`echo 'Test' \| tr '[:lower:]' '[:upper:]'`	Case conversion
`tr -dc '[:print:]' < /dev/urandom`	Filter non printable characters
`tr -s '[:blank:]' '\t' </proc/diskstats \| cut -f4`	cut fields separated by blanks
`history \| wc -l`	Count lines
`seq 10 \| paste -s -d ' '`	Concatenate and separate line items to a single line
`sort -u file1 file2`	Union of unsorted files
`sort file1 file2 \| uniq -d`	Intersection of unsorted files
`sort file1 file1 file2 \| uniq -u`	Difference of unsorted files
`sort file1 file2 \| uniq -u`	Symmetric Difference of unsorted files
`join -t'\0' -a1 -a2 file1 file2`	Union of sorted files
`join -t'\0' file1 file2`	Intersection of sorted files
`join -t'\0' -v2 file1 file2`	Difference of sorted files
`join -t'\0' -v1 -v2 file1 file2`	Symmetric Difference of sorted files
`shuf file1`	Randomise lines in a file
`comm file1 file2`	Combine lines from two sorted files

Set Operations

Command	What is does
`sort -u file1 file2`	Union of unsorted files
`sort file1 file2 \| uniq -d`	Intersection of unsorted files
`sort file1 file1 file2 \| uniq -u`	Difference of unsorted files
`sort file1 file2 \| uniq -u`	Symmetric Difference of unsorted files
`join -t'\0' -a1 -a2 file1 file2`	Union of sorted files
`join -t'\0' file1 file2`	Intersection of sorted files
`join -t'\0' -v2 file1 file2`	Difference of sorted files
`join -t'\0' -v1 -v2 file1 file2`	Symmetric Difference of sorted files

Windows Networking

Command	What is does
`smbtree`	Find windows machines. See also findsmb
`nmblookup -A 1.2.3.4`	Find the windows (netbios) name associated with ip address
`smbclient -L windows_box`	List shares on windows machine or samba server
`mount -t smbfs -o fmask=666,guest //windows_box/share /mnt/share`	Mount a windows share
`echo 'message' \| smbclient -M windows_box`	Send popup to windows machine

Monitoring and Debugging

Command	What is does
`tail -f /var/log/messages`	Monitor messages in a log file
`strace -c ls >/dev/null`	Summarise/profile system calls made by command
`strace -f -e open ls >/dev/null`	List system calls made by command
`strace -f -e trace=write -e write=1,2 ls >/dev/null`	Monitor what’s written to stdout and stderr
`ltrace -f -e getenv ls >/dev/null`	List library calls made by command
`lsof -p $$`	List paths that process id has open
`lsof ~`	List processes that have specified path open
`tcpdump not port 22`	Show network traffic except ssh. See also tcpdump_not_me
`ps -e -o pid,args --forest`	List processes in a hierarchy
`ps -e -o pcpu,cpu,nice,state,cputime,args --sort pcpu \| sed '/^ 0.0 /d'`	List processes by % cpu usage
`ps -e -orss=,args= \| sort -b -k1,1n \| pr -TW$COLUMNS`	List processes by mem (KB) usage. See also ps_mem.py
`ps -C firefox-bin -L -o pid,tid,pcpu,state`	List all threads for a particular process
`ps -p 1,$$ -o etime=`	List elapsed wall time for particular process IDs
`watch -n.1 pstree -Uacp $$`	Display a changing process subtree
`last reboot`	Show system reboot history
`free -m`	Show amount of (remaining) RAM (-m displays in MB)
`watch -n.1 'cat /proc/interrupts'`	Watch changeable data continuously
`udevadm monitor`	Monitor udev events to help configure rules
`ulimit -Sv 1000`	Limit memory usage for following commands to 1MiB
`fuser -k 8000/tcp`	Kill the program using port 8000
`lsof -p 123,789 -u 1234,abe`	All files used by PID 123 or 789, or by user abe or UID 1234
`kill -HUP $(lsof -t /home/foo/file)`	SIGHUP the processes using /home/foo/file
`cat /dev/urandom \| base64 \| pv -lbri2 > /dev/null`	Monitor progress of output

Disk Space

Command	What is does
`ls -lSr`	Show files by size, biggest last
`du -s * \| sort -k1,1rn \| head`	Show top disk uses in current dir
`du -hs /home/* \| sort -k1,1h`	Sort paths by easy to interpret disk usage
`df -h`	Show free space on mounted filesystems
`df -i`	Show free inodes on mounted filesystems
`fdisk -l`	Show disks partitions sizes and types (run as root)
`rpm -q -a --qf '%10{SIZE}\t%{NAME}\n' \| sort -k1,1n`	List all packages by installed size (Bytes) on rpm distros
`dpkg-query -W -f='${Installed-Size;10}\t${Package}\n' \| sort -k1,1n`	List all packages by installed size (KBytes) on deb distros
`dd bs=1 seek=2TB if=/dev/null of=ext3.test`	Create a large test file (taking no space)
`> file`	truncate data of file or create an empty file

CD/DVD

Command	What is does
`gzip < /dev/cdrom > cdrom.iso.gz`	Save copy of data cdrom
`mkisofs -V LABEL -r dir \| gzip > cdrom.iso.gz`	Create cdrom image from contents of dir
`mount -o loop cdrom.iso /mnt/dir`	Mount the cdrom image at /mnt/dir (read only)
`wodim dev=/dev/cdrom blank=fast`	Clear a CDRW
`gzip -dc cdrom.iso.gz \| wodim -tao dev=/dev/cdrom -v -data -`	Burn cdrom image
`cdparanoia -B`	Rip audio tracks from CD to wav files in current dir
`wodim -v dev=/dev/sr0 -audio -pad *.wav`	Make audio CD from all wavs in current dir
`oggenc --tracknum=$track track.cdda.wav -o track.ogg`	Make ogg file from wav file
`for i in *.mp3; do mpg123 --rate 44100 --stereo --buffer 3072 --resync -w "$(basename $i .mp3).wav" $i; done`	Decode mp3 files to 16-bit, stereo, 44.1 kHz waves
for i in *.mp3; do lame --decode $i ``basename $i .mp3``.wav; done	Decode mp3 files to 16-bit, stereo, 44.1 kHz waves
`normalize -m *.wav`	Normalise levels in wavs, mix mode is loud as possible

Locales

Command	What is does
`printf "%'d\n" 1234`	Print number with thousands grouping appropriate to locale
`BLOCK_SIZE=\'1 ls -l`	Use locale thousands grouping in ls. See also l
`echo "I live in$(locale territory)"`	Extract info from locale database
`LANG=en_IE.utf8 locale int_prefix`	Lookup locale info for specific country. See also ccodes
`locale -kc $(locale \| sed -n 's/$LC_.\{4,\}$=.*/\1/p') \| less`	List fields available in locale database

Dates and Times

Command	What is does
`cal -3`	Display a calendar
`cal 9 1752`	Display a calendar for a particular month year
`date -d fri`	What date is it this friday
`[ $(date -d '12:00 today +1 day' +%d) = '01' ] \|\| exit`	exit a script unless it’s the last day of the month
`date --date='25 Dec' +%A`	What day does xmas fall on, this year
`date --date='@2147483647'`	Convert seconds since the epoch (1970-01-01 UTC) to date
`TZ='America/Los_Angeles' date`	What time is it on west coast of US (use tzselect to find TZ)
`date --date='TZ="America/Los_Angeles" 09:00 next Fri'`	What’s the local time for 9AM next Friday on west coast US

Images

Most of these rely on the imagemagick cli programs.

Linux Ops Guide

Mon, 01 Aug 2016 20:09:01 +0000

Here I aim to cover a set of common administration tasks. Things like, the hostname, system logs, what users are currently logged in, physical devices that are connected, logical volumes, file system and inode allocation, attached network interfaces and their addressing, processes and daemons currently running, kernel verison, local users and groups, installed packages, remote mounts, network shares, system uptime, bread and butter OS stats (CPU, IO, network, memory).

Booting

shutdown -r +5 System going down for a reboot  #wall broadcast msg
shutdown -c  #cancel reboot
shutdown -r 00:00  #schedule for midnight
shutdown -h +5  #halt system in 5 mins
shutdown -h now

Alternatively, just use systemd:

Secure SHell (SSH)

Mon, 01 Aug 2016 20:09:01 +0000

The OpenSSH secure shell, ssh, provides the necessary client/server security plumbing, to allow shell execution on a remote machine. ssh can be used interactively, as per a normal shell, or to run one off commands, for example:

$ ssh ben@wookie.local uname -a
ben@wookie.local's password:
Linux wookie.local 3.14-1-686-pae #1 SMP Debian 3.14.7-1 (2014-06-16) i686 GNU/Linux

Hot tip: the w command is gem for showing users currently logged in

$ w -f
USER     TTY      FROM             LOGIN@   IDLE   JCPU   PCPU WHAT
ben      tty2     :0               17:05   10:36m 16:53  50.97s /usr/bin/evince /home/ben/podcasts/redhat/Docs/RH124-RHEL7.pdf

User ben logged into virtual console 2 (tty2) via a graphical login (:0) at about 5PM. OK, lets access a couple of the virtual terminals:

bash

Sun, 01 May 2016 01:01:01 +0000

Bash is a Unix shell written by Brian Fox in 1989 for the GNU Project as a free replacement for the Bourne shell. To this day, Bash remains one of the most powerful and ubiquitous scripting tools on the planet.

Kudos to Denys Dovhan and his awesome Bash handbook. The most digestable, and enjoyable method I’ve found to groking bash. <3

thinkfan

Sat, 19 Mar 2016 17:39:10 +0000

I run Debian on a ThinkPad T420s. ThinkPad’s have a great Linux compatibility story, and the T420s is no exception. The cooling fan situation seemed in need of some attention though, with fans running at high RPM most of the time. Not only was this really noisy, hit the battery quite hard too.

Enter thinkfan, a simple fan control program.

Works with any linux hwmon driver, especially with thinkpad_acpi. It is designed to eat as little CPU power as possible.

awk

Sun, 17 Jan 2016 08:14:01 +0000

After recently getting my hands dirty with the wonderful [sed]({% post_url 2015-09-15-sed %}), the time has come for me to step it up with awk.

Incepted in 1977 at Bell Labs, AWK is an interpreted programming language designed for text processing, and is a defacto part of most nix based distributions.

The original version of UNIX awk was designed and implemented by Alfred Aho, Peter Weinberger, and Brian Kernighan (notice the first letter of their surnames together) of Bell Laboratories. Brian Kernighan continues to maintain and enhance it. Paul Rubin and Jay Fenlason, of the Free Software Foundation, wrote gawk, to be compatible with the original version of awk distributed in Seventh Edition UNIX.

GNU Stream Editor (sed)

Tue, 15 Sep 2015 19:48:01 +0000

sed has rocked my world.

sed (stream editor) isn’t an interactive text editor. Instead, it is used to filter text, i.e., it takes text input, performs some operation (or set of operations) on it, and outputs the modified text. sed is typically used for extracting part of a file using pattern matching or substituting multiple occurrences of a string within a file.

Basic syntax

sed ' [RANGE] COMMANDS ' [INPUTFILE]

If no INPUTFILE is specified, sed filters the contents of standard input.

NeoMutt setup (2022 edition)

Sat, 18 Aug 2012 21:03:40 +1000

Intro

If you haven’t heard of mutt you’re missing out.

Mutt is a small but very powerful text-based mail client for UNIX operating systems

It’s a command line based mail reader (MUA). The neomutt project is a contemporary fork of the original mutt with added features, patch sets and fixes.

I started seriously using mutt as my full-time mail client in 2012. I found its simplicity refreshing. It quickly became apparent how much useless baggage modern graphical mail clients (e.g., thunderbird, outlook, etc) actually bog you down with, not to mention the harrowing way they actually encode mail. Its taken me a long time to appreciate this:

Command	What is does
`sed 's/string1/string2/g'`	Replace string1 with string2
`sed 's/\(.*\)1/\12/g'`	Modify anystring1 to anystring2
`sed '/^ #/d; /^ $/d'`	Remove comments and blank lines
`sed ':a; /\\$/N; s/\\\n//; ta'`	Concatenate lines with trailing \
`sed 's/[ \t]*$//'`	Remove trailing spaces from lines
`seq 10 \| sed "s/^/ /; s/ *\(.\{7,\}\)/\1/"`	Right align numbers
`seq 10 \| sed p \| paste - -`	Duplicate a column
`sed -n '1000{p;q}'`	Print 1000th line
`sed -n '10,20p;20q'`	Print lines 10 to 20
`sed -n 's/.<title>\(.\)<\/title>.*/\1/ip;T;q'`	Extract title from HTML web page
`sed -i 42d ~/.ssh/known_hosts`	Delete a particular line
`sort -t. -k1,1n -k2,2n -k3,3n -k4,4n`	Sort IPV4 ip addresses
`echo 'Test' \| tr '[:lower:]' '[:upper:]'`	Case conversion
`tr -dc '[:print:]' < /dev/urandom`	Filter non printable characters
`tr -s '[:blank:]' '\t' </proc/diskstats \| cut -f4`	cut fields separated by blanks
`history \| wc -l`	Count lines
`seq 10 \| paste -s -d ' '`	Concatenate and separate line items to a single line
`sort -u file1 file2`	Union of unsorted files
`sort file1 file2 \| uniq -d`	Intersection of unsorted files
`sort file1 file1 file2 \| uniq -u`	Difference of unsorted files
`sort file1 file2 \| uniq -u`	Symmetric Difference of unsorted files
`join -t'\0' -a1 -a2 file1 file2`	Union of sorted files
`join -t'\0' file1 file2`	Intersection of sorted files
`join -t'\0' -v2 file1 file2`	Difference of sorted files
`join -t'\0' -v1 -v2 file1 file2`	Symmetric Difference of sorted files
`shuf file1`	Randomise lines in a file
`comm file1 file2`	Combine lines from two sorted files