Linux cheatsheet

A survey of the standard and high quality programs that feature in most Unix based distributions, with the GNU variants being my favourite. The bash shell is a great way of interfacing and orchestrating these beautifully crafted programs. As a starting point, I’ve listed each program offered by the GNU Core Utilities and util-linux umbrella projects; considered the de facto standard on most distributions. Quick Reference General System Information Directory Navigation File Searching Archiving and Compression Networking Text Manipulation Set Operations Windows Networking Monitoring and Debugging Disk Space CD/DVD Locales Dates and Times Images Finding Documentation Manual Pages Appropriate Commands whatis GNU Info Entry /usr/share/doc Documentation RPM bundled documentation Examples grep cut sort tr wc tar rsync sed awk ssh (Secure Shell) wget BFL of Common Programs Resources Quick Reference General Command What is does apropos compress Show commands that relate to a keyword man -t ascii | ps2pdf - > ascii.pdf Make a PDF of a man page which command Full path of command time command Show execution time of a given command time cat Start stopwatch, ^d to stop cat file.txt | xclip -selection clipboard Copy to clipboard nohup ./script.sh & Keep program running after leaving SSH session (see bash post if input needed) timeout 20s ./script.sh Run script.sh for 20 seconds only while true; do timeout 30m ./script.sh; done Restart a program every 30 minutes System Information Command What is does uname -a Show kernel version and system architecture head -n1 /etc/issue Show name and version of distribution cat /proc/partitions Show all partitions registered on the system grep MemTotal /proc/meminfo Show RAM total seen by the system grep "model name" /proc/cpuinfo Show CPU(s) info lspci -tv Show PCI info lsusb -tv Show USB info mount | column -t List mounted filesystems on the system (and align output) grep -F capacity: /proc/acpi/battery/BAT0/info Show state of cells in laptop battery dmidecode -q | less Display SMBIOS/DMI information smartctl -A /dev/sda | grep Power_On_Hours How long has this disk (system) been powered on in total hdparm -i /dev/sda Show info about disk sda hdparm -tT /dev/sda Do a read speed test on disk sda badblocks -s /dev/sda Test for unreadable blocks on disk sda Directory Navigation Command What is does cd - Go previous directory cd Go home (cd dir123 && pwd) Jump into a directory, run a command there, and return to origin pushd . Put cwd on stack, so you can popd back to it File Searching Command What is does alias l='ls -l --color=auto' Quick listing ls -lrt List long by date ls -lS List long by size ls /usr/bin | pr -T9 -W$COLUMNS Print in 9 columns to width of terminal find -name '*.[ch] | xargs grep -E 'foo' Search for ‘foo’ in all .c and .h files in cwd and below find -type f -print0 | xargs -r0 grep -F 'example' Search all regular files for ’example’ find -maxdepth 1 -type f | xargs grep -F 'example' As above, but don’t recurse find -maxdepth 1 -type d | while read dir; do echo $dir; echo somecmd; done Wash each result over multiple commands find -type f ! -perm -444 Find files not readable by all find -type d ! -perm -111 Find dirs not accessable by all find . -size 30c By file size (30 bytes) find . -name "*.gz" -delete Delete all gz files locate -r 'file[^/]*\.txt Search cached index for names look <keyword> Search English dictionary with a given prefix keyword grep --color reference /usr/share/dict/words Highlight occurances of regex against English dictionary readlink -f file.txt Full path of file namei -l /bin/bash Drills through directories and links showing permission mask all the way down Archiving and Compression Command What is does gpg -c file Encrypt file gpg file.gpg Decrypt file tar -c dir/ | bzip2 > dir.tar.bz2 Make compressed archive of dir bzip2 -dc dir.tar.bz2 | tar -x Extract archive tar -c dir/ | gzip | gpg -c | ssh user@remote ‘dd of=dir.tar.gz.gpg’ Make encrypted archive of dir on remote machine find dir/ -name ‘*.txt’ | tar -c –files-from=- | bzip2 > dir_txt.tar.bz2 Make archive of subset of dir and below find dir/ -name ‘*.txt’ | xargs cp -a –target-directory=dir_txt/ –parents Make copy of subset of dir and below ( tar -c /dir/to/copy ) | ( cd /where/to/ && tar -x -p ) Copy (with permissions) copy/ dir to /where/to/ dir ( cd /dir/to/copy && tar -c . ) | ( cd /where/to/ && tar -x -p ) Copy (with permissions) contents of copy/ dir to /where/to/ ( tar -c /dir/to/copy ) | ssh -C user@remote ‘cd /where/to/ && tar -x -p’ Copy (with permissions) copy/ dir to remote:/where/to/ dir dd bs=1M if=/dev/sda | gzip | ssh user@remote ‘dd of=sda.gz’ Backup harddisk to remote machine Networking Command What is does ethtool eth0 Show status of ethernet interface eth0 ethtool --change eth0 autoneg off speed 100 duplex full Manually set ethernet interface speed iw dev wlan0 link Show link status of wireless interface wlan0 iw dev wlan0 set bitrates legacy-2.4 1 Manually set wireless interface speed iw dev wlan0 scan List wireless networks in range ip link show List network interfaces ip link set dev eth0 name wan Rename interface eth0 to wan ip link set dev eth0 up Bring interface eth0 up (or down) ip addr show List addresses for interfaces ip addr add 1.2.3.4/24 brd + dev eth0 Add (or del) ip and mask (255.255.255.0) ip route show List routing table ip route add default via 1.2.3.254 Set default gateway to 1.2.3.254 ss -tupl List internet services on a system ss -tup List active connections to/from system host bencode.net Lookup DNS ip address for name or vice versa hostname -i Lookup local ip address (equivalent to host hostname) whois bencode.net Lookup whois info for hostname or ip address mtr google.com Nice trace route Text Manipulation Command What is does sed 's/string1/string2/g' Replace string1 with string2 sed 's/\(.*\)1/\12/g' Modify anystring1 to anystring2 sed '/^ *#/d; /^ *$/d' Remove comments and blank lines sed ':a; /\\$/N; s/\\\n//; ta' Concatenate lines with trailing \ sed 's/[ \t]*$//' Remove trailing spaces from lines seq 10 | sed "s/^/ /; s/ *\(.\{7,\}\)/\1/" Right align numbers seq 10 | sed p | paste - - Duplicate a column sed -n '1000{p;q}' Print 1000th line sed -n '10,20p;20q' Print lines 10 to 20 sed -n 's/.*<title>\(.*\)<\/title>.*/\1/ip;T;q' Extract title from HTML web page sed -i 42d ~/.ssh/known_hosts Delete a particular line sort -t. -k1,1n -k2,2n -k3,3n -k4,4n Sort IPV4 ip addresses echo 'Test' | tr '[:lower:]' '[:upper:]' Case conversion tr -dc '[:print:]' < /dev/urandom Filter non printable characters tr -s '[:blank:]' '\t' </proc/diskstats | cut -f4 cut fields separated by blanks history | wc -l Count lines seq 10 | paste -s -d ' ' Concatenate and separate line items to a single line sort -u file1 file2 Union of unsorted files sort file1 file2 | uniq -d Intersection of unsorted files sort file1 file1 file2 | uniq -u Difference of unsorted files sort file1 file2 | uniq -u Symmetric Difference of unsorted files join -t'\0' -a1 -a2 file1 file2 Union of sorted files join -t'\0' file1 file2 Intersection of sorted files join -t'\0' -v2 file1 file2 Difference of sorted files join -t'\0' -v1 -v2 file1 file2 Symmetric Difference of sorted files shuf file1 Randomise lines in a file comm file1 file2 Combine lines from two sorted files Set Operations Command What is does sort -u file1 file2 Union of unsorted files sort file1 file2 | uniq -d Intersection of unsorted files sort file1 file1 file2 | uniq -u Difference of unsorted files sort file1 file2 | uniq -u Symmetric Difference of unsorted files join -t'\0' -a1 -a2 file1 file2 Union of sorted files join -t'\0' file1 file2 Intersection of sorted files join -t'\0' -v2 file1 file2 Difference of sorted files join -t'\0' -v1 -v2 file1 file2 Symmetric Difference of sorted files Windows Networking Command What is does smbtree Find windows machines. See also findsmb nmblookup -A 1.2.3.4 Find the windows (netbios) name associated with ip address smbclient -L windows_box List shares on windows machine or samba server mount -t smbfs -o fmask=666,guest //windows_box/share /mnt/share Mount a windows share echo 'message' | smbclient -M windows_box Send popup to windows machine Monitoring and Debugging Command What is does tail -f /var/log/messages Monitor messages in a log file strace -c ls >/dev/null Summarise/profile system calls made by command strace -f -e open ls >/dev/null List system calls made by command strace -f -e trace=write -e write=1,2 ls >/dev/null Monitor what’s written to stdout and stderr ltrace -f -e getenv ls >/dev/null List library calls made by command lsof -p $$ List paths that process id has open lsof ~ List processes that have specified path open tcpdump not port 22 Show network traffic except ssh. See also tcpdump_not_me ps -e -o pid,args --forest List processes in a hierarchy ps -e -o pcpu,cpu,nice,state,cputime,args --sort pcpu | sed '/^ 0.0 /d' List processes by % cpu usage ps -e -orss=,args= | sort -b -k1,1n | pr -TW$COLUMNS List processes by mem (KB) usage. See also ps_mem.py ps -C firefox-bin -L -o pid,tid,pcpu,state List all threads for a particular process ps -p 1,$$ -o etime= List elapsed wall time for particular process IDs watch -n.1 pstree -Uacp $$ Display a changing process subtree last reboot Show system reboot history free -m Show amount of (remaining) RAM (-m displays in MB) watch -n.1 'cat /proc/interrupts' Watch changeable data continuously udevadm monitor Monitor udev events to help configure rules ulimit -Sv 1000 Limit memory usage for following commands to 1MiB fuser -k 8000/tcp Kill the program using port 8000 lsof -p 123,789 -u 1234,abe All files used by PID 123 or 789, or by user abe or UID 1234 kill -HUP $(lsof -t /home/foo/file) SIGHUP the processes using /home/foo/file cat /dev/urandom | base64 | pv -lbri2 > /dev/null Monitor progress of output Disk Space Command What is does ls -lSr Show files by size, biggest last du -s * | sort -k1,1rn | head Show top disk uses in current dir du -hs /home/* | sort -k1,1h Sort paths by easy to interpret disk usage df -h Show free space on mounted filesystems df -i Show free inodes on mounted filesystems fdisk -l Show disks partitions sizes and types (run as root) rpm -q -a --qf '%10{SIZE}\t%{NAME}\n' | sort -k1,1n List all packages by installed size (Bytes) on rpm distros dpkg-query -W -f='${Installed-Size;10}\t${Package}\n' | sort -k1,1n List all packages by installed size (KBytes) on deb distros dd bs=1 seek=2TB if=/dev/null of=ext3.test Create a large test file (taking no space) > file truncate data of file or create an empty file CD/DVD Command What is does gzip < /dev/cdrom > cdrom.iso.gz Save copy of data cdrom mkisofs -V LABEL -r dir | gzip > cdrom.iso.gz Create cdrom image from contents of dir mount -o loop cdrom.iso /mnt/dir Mount the cdrom image at /mnt/dir (read only) wodim dev=/dev/cdrom blank=fast Clear a CDRW gzip -dc cdrom.iso.gz | wodim -tao dev=/dev/cdrom -v -data - Burn cdrom image cdparanoia -B Rip audio tracks from CD to wav files in current dir wodim -v dev=/dev/sr0 -audio -pad *.wav Make audio CD from all wavs in current dir oggenc --tracknum=$track track.cdda.wav -o track.ogg Make ogg file from wav file for i in *.mp3; do mpg123 --rate 44100 --stereo --buffer 3072 --resync -w "$(basename $i .mp3).wav" $i; done Decode mp3 files to 16-bit, stereo, 44.1 kHz waves for i in *.mp3; do lame --decode $i ``basename $i .mp3``.wav; done Decode mp3 files to 16-bit, stereo, 44.1 kHz waves normalize -m *.wav Normalise levels in wavs, mix mode is loud as possible Locales Command What is does printf "%'d\n" 1234 Print number with thousands grouping appropriate to locale BLOCK_SIZE=\'1 ls -l Use locale thousands grouping in ls. See also l echo "I live in$(locale territory)" Extract info from locale database LANG=en_IE.utf8 locale int_prefix Lookup locale info for specific country. See also ccodes locale -kc $(locale | sed -n 's/\(LC_.\{4,\}\)=.*/\1/p') | less List fields available in locale database Dates and Times Command What is does cal -3 Display a calendar cal 9 1752 Display a calendar for a particular month year date -d fri What date is it this friday [ $(date -d '12:00 today +1 day' +%d) = '01' ] || exit exit a script unless it’s the last day of the month date --date='25 Dec' +%A What day does xmas fall on, this year date --date='@2147483647' Convert seconds since the epoch (1970-01-01 UTC) to date TZ='America/Los_Angeles' date What time is it on west coast of US (use tzselect to find TZ) date --date='TZ="America/Los_Angeles" 09:00 next Fri' What’s the local time for 9AM next Friday on west coast US Images Most of these rely on the imagemagick cli programs. ...

February 18, 2017 · 25 min

C

Update: 2022-04-26: Overhauling to be simplier and bsd/clang friendly. Rules of engagement Style The language Basic Types Integer Types Real and Complex Floating Types void Memory Access Operators Strings The Pre Processor Macros Patterns vtables person_s.h print_vtable.h print_vtable.c main.c Libraries C standard library C POSIX library Unit Testing Sample C code Rules of engagement How I program C by Eskil Steenberg Notes on Programming in C by Rob Pike The Ten Commandments for C Programmers by Henry Spencer Style suckless FreeBSD man 9 style The language Basic Types Integer Types C99 with stdbool.h introduced boolean type _Bool (1 byte), and macros true (1) and false (0). ...

November 26, 2016 · 9 min

Shay Banon on Innovation

Some great wisdom imparted by Shay Banon (creator of Elasticsearch). Be bored. To promote innovation. Open source and building a critical mass. Be committed. You need to care for it now; IRC, forums, negative feedback, bugs. The benevolent dicator model (BDM) versus the democracy with a leader model. Be distributed. Choose a useful (and normally hard) problem to solve. Best way to learn is to actually build something. Double down on passion. Recruit passionate people (e.g. story about the guy who paid for Elastic training off his own back). Videos are for collaboration. Simple things, should be simple. Not easy though. Hierarchy matters. Humans can’t make decisions in groups. IT is obsessed with whats next. You only need to look backwards for answers.

November 22, 2016 · 1 min

make

A small make orientation guide. make is a versatile task runner, its core competency is in creating files from other files Make essentials Equal signs Built-in variables Phony targets C specifics Custom variables Implicit variables Example Makefiles Make essentials make generates files from other files, using recipes, the syntax is as follows. Please note, thanks to POSIX standardisation the recipe MUST be indented with a tab (not spaces): target_file: prerequisite_file1 prerequisite_file2 shell command to build target_file (MUST be indented with tabs, not spaces) another shell command (these commands are called the "recipe") Unless you specify otherwise, Make assumes that the target (target_file above) and prerequisites (prerequisite_file1 and prerequisite_file2) are actual files or directories. You can ask Make to build a target from the command line like this: ...

October 9, 2016 · 7 min

Linux Ops Guide

Here I aim to cover a set of common administration tasks. Things like, the hostname, system logs, what users are currently logged in, physical devices that are connected, logical volumes, file system and inode allocation, attached network interfaces and their addressing, processes and daemons currently running, kernel verison, local users and groups, installed packages, remote mounts, network shares, system uptime, bread and butter OS stats (CPU, IO, network, memory). Booting shutdown -r +5 System going down for a reboot #wall broadcast msg shutdown -c #cancel reboot shutdown -r 00:00 #schedule for midnight shutdown -h +5 #halt system in 5 mins shutdown -h now Alternatively, just use systemd: ...

August 1, 2016 · 12 min

Secure SHell (SSH)

The OpenSSH secure shell, ssh, provides the necessary client/server security plumbing, to allow shell execution on a remote machine. ssh can be used interactively, as per a normal shell, or to run one off commands, for example: $ ssh ben@wookie.local uname -a ben@wookie.local's password: Linux wookie.local 3.14-1-686-pae #1 SMP Debian 3.14.7-1 (2014-06-16) i686 GNU/Linux Hot tip: the w command is gem for showing users currently logged in $ w -f USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT ben tty2 :0 17:05 10:36m 16:53 50.97s /usr/bin/evince /home/ben/podcasts/redhat/Docs/RH124-RHEL7.pdf User ben logged into virtual console 2 (tty2) via a graphical login (:0) at about 5PM. OK, lets access a couple of the virtual terminals: ...

August 1, 2016 · 6 min

Architecture

A collection of software concepts I plan to apply to some up coming projects. Some fundamental philosophies: Automation everywhere. A clean (agnostic) contract with the underlying operating system, promoting portability between execution environments. Can scale without major changes to tooling, architecture or development. Smallest possible delta between development and production, enabling continuous integration. Deployment Processes are first class citizens. Execute the application as one or more stateless processes. Model process types explicitly, e.g. HTTP requests might be handled by a web process, while long running backend tasks by a worker process. Always rely on the operating systems user-space process manager (e.g. systemd, Upstart) to manage output streams, respond to crashed processes and handle user initiated restarts and shutdowns. Concurrency; Physical distribution and clean contract with operating system, e.g. containers (e.g. docker). Versioning; the ability to deploy and hotswap versions side by side. Pluggable; ability to snap modules into architecture (punch through all layers), see attached resources under backing services. Store all runtime configuration as environmental variables. They are a language and OS agnostic standard, and unlike other config options such as Java System Properties, are not accidentially added into the source code repo. The array of process types and number of processes of each type is known as the process formation. ...

July 13, 2016 · 3 min

soapUI mock bug

Today I stumbled onto interesting soapUI quirk, involving a combination of mock services, SOAP 1.2 and multipart message definitions. In essence, the soapUI mock service will always return an HTTP 500, with the following response: {% highlight xml %} <soap:Envelope xmlns:soap=“http://www.w3.org/2003/05/soap-envelope"> soap:Body soap:Fault soap:Code soap:ValueServer</soap:Value> </soap:Code> soap:Reason <soap:Text xml:lang=“en”>Missing operation for soapAction [http://services.net.bencode/wsdl/2016/06/01/retrievecoolnesslevelrequest] and body element [{http://services.net.bencode/wsdl/2016/06/01}retrieveCoolnessLevelRequest] with SOAP Version [SOAP 1.2]</soap:Text> </soap:Reason> </soap:Fault> </soap:Body> </soap:Envelope> {% endhighlight %} Sigh. Thankfully some legend known as fyerf posted a solution on the smartbear community forums. ...

June 1, 2016 · 3 min

DIY Computer Part 4 Machine Language

A continuation of my participation in the amazing Nand2Tetris course, by Noam Nisan and Shimon Schocken, now running on Coursera. In this course you will build a modern computer system, from the ground up. We’ll take you from constructing elementary logic gates all the way through creating a fully functioning general purpose computer. In the process, you will learn how really computers work, and how they are designed. If interested, see prior posts: ...

May 23, 2016 · 7 min

bash

Bash is a Unix shell written by Brian Fox in 1989 for the GNU Project as a free replacement for the Bourne shell. To this day, Bash remains one of the most powerful and ubiquitous scripting tools on the planet. Contents Useful Shortcuts Initialisation Shell Grammar Variables Local variables Environment variables Positional arguments Expansions Brace expansion Command substitution Arithmetic expansion Double and single quotes Stream Redirection here documents Arrays Conditions if statements case statements Loops For Loops Select Loops While Loops Until Loops Functions Coprocesses Builtins Bash Recipes Top 6 largest things in the current directory Display the 23rd line of /etc/passwd Filter the first column from process status Delete Subversion scrap files Move shell scripts and mark them as executable Pattern matching Scan code base against list of patterns Rename Multiple Files Run a command every time a file is modified Keep a program running after leaving SSH session Simple menu and functions Complete example Resources Kudos to Denys Dovhan and his awesome Bash handbook. The most digestable, and enjoyable method I’ve found to groking bash. <3 ...

May 1, 2016 · 16 min