Linux

Ansible is a radically simple IT automation platform that makes your applications and systems easier to deploy. Avoid writing scripts or custom code to deploy and update your applications. Automate in a language that approaches plain English, using SSH, with no agents to install on remote systems. With Ansible and its amazing module ecosystem, you describe what needs to be accomplished (i.e. declarative), rather than describing how to accomplish each step (i.e. imperative). ...

When something becomes messy in the software industry, we have the habit of tucking the problems away behind abstractions. To me Vagrant is simply an abstraction layer for working with a range of hypervisors (libvirt, VirtualBox, VMWare, Hyper-V, Docker and more), in a reliable and consistent manner. By articulating VM’s as a simple text based (Ruby) configuration file, a Vagrantfile, achieve greater simplicity, portability across systems, and consistency which could theoretically be used with any virtual environment running on top of any other system. ...

Partitioning Two popular partition schemes are used in the wild, MBR and GPT. MBR MBR, or Master Boot Record, often associated with BIOS, was introduced in 1983 with IBM PC DOS 2.0, is a special boot sector located at the beginning of a drive. This sector contains a boot loader (e.g GRUB), and details about the logical partitions. MBR supports drives upto 2TiB, and up to 4 primary partitions. # fdisk /dev/vda Welcome to fdisk (util-linux 2.23.2). Changes will remain in memory only, until you decide to write them. Be careful before using the write command. Device does not contain a recognized partition table Building a new DOS disklabel with disk identifier 0x9228f9b7. Command (m for help): m Command action a toggle a bootable flag b edit bsd disklabel c toggle the dos compatibility flag d delete a partition g create a new empty GPT partition table G create an IRIX (SGI) partition table l list known partition types m print this menu n add a new partition o create a new empty DOS partition table p print the partition table q quit without saving changes s create a new empty Sun disklabel t change a partition's system id u change display/entry units v verify the partition table w write table to disk and exit x extra functionality (experts only) Command (m for help): n Partition type: p primary (0 primary, 0 extended, 4 free) e extended Select (default p): Using default response p Partition number (1-4, default 1): First sector (2048-2097151, default 2048): Using default value 2048 Last sector, +sectors or +size{K,M,G} (2048-2097151, default 2097151): +500M Partition 1 of type Linux and of size 500 MiB is set Command (m for help): l 0 Empty 24 NEC DOS 81 Minix / old Lin bf Solaris 1 FAT12 27 Hidden NTFS Win 82 Linux swap / So c1 DRDOS/sec (FAT- 2 XENIX root 39 Plan 9 83 Linux c4 DRDOS/sec (FAT- 3 XENIX usr 3c PartitionMagic 84 OS/2 hidden C: c6 DRDOS/sec (FAT- 4 FAT16 <32M 40 Venix 80286 85 Linux extended c7 Syrinx 5 Extended 41 PPC PReP Boot 86 NTFS volume set da Non-FS data 6 FAT16 42 SFS 87 NTFS volume set db CP/M / CTOS / . 7 HPFS/NTFS/exFAT 4d QNX4.x 88 Linux plaintext de Dell Utility 8 AIX 4e QNX4.x 2nd part 8e Linux LVM df BootIt 9 AIX bootable 4f QNX4.x 3rd part 93 Amoeba e1 DOS access a OS/2 Boot Manag 50 OnTrack DM 94 Amoeba BBT e3 DOS R/O b W95 FAT32 51 OnTrack DM6 Aux 9f BSD/OS e4 SpeedStor c W95 FAT32 (LBA) 52 CP/M a0 IBM Thinkpad hi eb BeOS fs e W95 FAT16 (LBA) 53 OnTrack DM6 Aux a5 FreeBSD ee GPT f W95 Ext'd (LBA) 54 OnTrackDM6 a6 OpenBSD ef EFI (FAT-12/16/ 10 OPUS 55 EZ-Drive a7 NeXTSTEP f0 Linux/PA-RISC b 11 Hidden FAT12 56 Golden Bow a8 Darwin UFS f1 SpeedStor 12 Compaq diagnost 5c Priam Edisk a9 NetBSD f4 SpeedStor 14 Hidden FAT16 <3 61 SpeedStor ab Darwin boot f2 DOS secondary 16 Hidden FAT16 63 GNU HURD or Sys af HFS / HFS+ fb VMware VMFS 17 Hidden HPFS/NTF 64 Novell Netware b7 BSDI fs fc VMware VMKCORE 18 AST SmartSleep 65 Novell Netware b8 BSDI swap fd Linux raid auto 1b Hidden W95 FAT3 70 DiskSecure Mult bb Boot Wizard hid fe LANstep 1c Hidden W95 FAT3 75 PC/IX be Solaris boot ff BBT 1e Hidden W95 FAT1 80 Old Minix Command (m for help): t Selected partition 1 Hex code (type L to list all codes): 83 Changed type of partition 'Linux' to 'Linux' Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks. The partition is now available as a block device, below we now see /dev/vda1: ...

Cheatsheet ========================================== ========================================== INFORMATION LOGS ========================================== ========================================== systemctl journalctl list-dependencies -u <name> by service list-sockets -f follow live log list-jobs -k kernel messages list-unit-files -k kernel messages list-units -r reverse (newest first) get-default -o json format output --type=service --all -b -1 from previous boot -n 10 limit output lines (tail) ========================================== -S,--since e.g "2012-10-30 18:17:16" SERVICES -U,--until e.g. "10 min ago" ========================================== systemctl ========================================== stop <name> SYSTEM STATES start <name> ========================================== restart <name> reload <name> systemctl status <name> reboot reboot.target enable <name> poweroff poweroff.target disable <name> emergency emergency.target show <name> default default.target -H <host> status network

I’m a software engineer not a sysadmin. In fact I know I’m a bad one. The RHCSA is a rote, mechanical process. It did force me to discover discover several gaps in my GNU/Linux knowledge so overall happy about that. Essential Tools Documentation man /usr/share/doc RPM bundled documentation General Searching Techniques Shell history Globbing I/O Redirection and Pipes Essential File Management Linux file system layout Finding Files locate find Archiving and compression with tar Hard and Soft Links Working with text Regular expressions grep sed and awk Connecting a Linux host Consoles Terminals and TTYs Switch Users (su) sudo SSH Managing users and groups Creating users User properties User configuration files Creating and managing groups Managing password properties Managing Permissions Changing file ownership Managing basic permissions Understanding umask (user mask) Special permissions suid sgid Sticky bit Understanding ACLs Configuring Networking Network device naming Managing runtime network configuration with ip Storing network configuration persistently nmcli Routing and DNS Managing Processes Shell jobs ps Memory usage CPU load System activity with top Sending signals to processes Priority and niceness tuned profiles Managing Software RPM and yum rpm queries yum Groups Repositories Modules and Application Streams Red Hat Subscription Manager Systemd Scheduling Tasks cron at systemd Timers tmpfiles.d Logging Rsyslog Systemd Journal Logrotate Managing Storage Disk layout Creating partitions GPT partitions with parted MBR partitions with fdisk File System Choices XFS Ext4 Mounting Persistent block device naming systemd mounts Swap Advanced Storage LVM Stratis VDO LUKS Advanced tasks Kernel management Boot procedure GRUB Systemd targets Essential troubleshooting Changing root password Managing network services SSH httpd (apache) SELinux Context Labels Booleans File context labels SELinux logs SELinux troubleshooting Firewalling with firewalld Automating installs Time services Remote file systems NFS CIFS with Samba Automount Containers Autostarting non-root containers with systemd user units Dont forget list Exam shakedown Linux Gems Essential Tools Documentation The ability to understand a program using local documentation resources; man, info, /usr/share/doc, within the RPM package. ...

Kudos to afair for putting together this neat Tmux Cheat Sheet, which I’m addicted to at the moment. ========================================== ========================================== TMUX COMMAND WINDOW (TAB) ========================================== ========================================== List tmux ls List ^b w New -s <session> Create ^b c Attach att -t <session> Rename ^b , <name> Rename rename-session -t <old> <new> Last ^b l (lower-L) Kill kill-session -t <session> Close ^b & ========================================== Goto # ^b <0-9> CONTROLS Next ^b n ========================================== Previous ^b p Choose ^b w <name> Detach ^b d List ^b = ========================================== Buffer ^b <PgUpDn> PANE (SPLIT WINDOW) Command ^b : <command> ========================================== Copy ^b [ ... <space> ... <enter> Show # ^b q Moving vim/emacs key bindings Split Horiz ^b " -------- Start <space> Split Vert ^b % | Copy <enter> Pane->Window ^b ! Paste ^b ] Kill ^b x ========================================== Reorganize ^b <space> SESSION (Set of Windows) Expand ^b <alt><arrow> ========================================== Resize ^b ^<arrow> Resize x n ^b <n> <arrow> New ^b :new ^b :new -s <name> Rename ^b $ Select ^b <arrow> List ^b s Previous ^b { Next ^b ( Next ^b } Previous ^b ) Switch ^b o other Swap ^b ^o Last ^b ;

A survey of the standard and high quality programs that feature in most Unix based distributions, with the GNU variants being my favourite. The bash shell is a great way of interfacing and orchestrating these beautifully crafted programs. As a starting point, I’ve listed each program offered by the GNU Core Utilities and util-linux umbrella projects; considered the de facto standard on most distributions. Quick Reference General System Information Directory Navigation File Searching Archiving and Compression Networking Text Manipulation Set Operations Windows Networking Monitoring and Debugging Disk Space CD/DVD Locales Dates and Times Images Finding Documentation Manual Pages Appropriate Commands whatis GNU Info Entry /usr/share/doc Documentation RPM bundled documentation Examples grep cut sort tr wc tar rsync sed awk ssh (Secure Shell) wget BFL of Common Programs Resources Quick Reference General Command What is does apropos compress Show commands that relate to a keyword man -t ascii | ps2pdf - > ascii.pdf Make a PDF of a man page which command Full path of command time command Show execution time of a given command time cat Start stopwatch, ^d to stop cat file.txt | xclip -selection clipboard Copy to clipboard nohup ./script.sh & Keep program running after leaving SSH session (see bash post if input needed) timeout 20s ./script.sh Run script.sh for 20 seconds only while true; do timeout 30m ./script.sh; done Restart a program every 30 minutes System Information Command What is does uname -a Show kernel version and system architecture head -n1 /etc/issue Show name and version of distribution cat /proc/partitions Show all partitions registered on the system grep MemTotal /proc/meminfo Show RAM total seen by the system grep "model name" /proc/cpuinfo Show CPU(s) info lspci -tv Show PCI info lsusb -tv Show USB info mount | column -t List mounted filesystems on the system (and align output) grep -F capacity: /proc/acpi/battery/BAT0/info Show state of cells in laptop battery dmidecode -q | less Display SMBIOS/DMI information smartctl -A /dev/sda | grep Power_On_Hours How long has this disk (system) been powered on in total hdparm -i /dev/sda Show info about disk sda hdparm -tT /dev/sda Do a read speed test on disk sda badblocks -s /dev/sda Test for unreadable blocks on disk sda Directory Navigation Command What is does cd - Go previous directory cd Go home (cd dir123 && pwd) Jump into a directory, run a command there, and return to origin pushd . Put cwd on stack, so you can popd back to it File Searching Command What is does alias l='ls -l --color=auto' Quick listing ls -lrt List long by date ls -lS List long by size ls /usr/bin | pr -T9 -W$COLUMNS Print in 9 columns to width of terminal find -name '*.[ch] | xargs grep -E 'foo' Search for ‘foo’ in all .c and .h files in cwd and below find -type f -print0 | xargs -r0 grep -F 'example' Search all regular files for ’example’ find -maxdepth 1 -type f | xargs grep -F 'example' As above, but don’t recurse find -maxdepth 1 -type d | while read dir; do echo $dir; echo somecmd; done Wash each result over multiple commands find -type f ! -perm -444 Find files not readable by all find -type d ! -perm -111 Find dirs not accessable by all find . -size 30c By file size (30 bytes) find . -name "*.gz" -delete Delete all gz files locate -r 'file[^/]*\.txt Search cached index for names look <keyword> Search English dictionary with a given prefix keyword grep --color reference /usr/share/dict/words Highlight occurances of regex against English dictionary readlink -f file.txt Full path of file namei -l /bin/bash Drills through directories and links showing permission mask all the way down Archiving and Compression Command What is does gpg -c file Encrypt file gpg file.gpg Decrypt file tar -c dir/ | bzip2 > dir.tar.bz2 Make compressed archive of dir bzip2 -dc dir.tar.bz2 | tar -x Extract archive tar -c dir/ | gzip | gpg -c | ssh user@remote ‘dd of=dir.tar.gz.gpg’ Make encrypted archive of dir on remote machine find dir/ -name ‘*.txt’ | tar -c –files-from=- | bzip2 > dir_txt.tar.bz2 Make archive of subset of dir and below find dir/ -name ‘*.txt’ | xargs cp -a –target-directory=dir_txt/ –parents Make copy of subset of dir and below ( tar -c /dir/to/copy ) | ( cd /where/to/ && tar -x -p ) Copy (with permissions) copy/ dir to /where/to/ dir ( cd /dir/to/copy && tar -c . ) | ( cd /where/to/ && tar -x -p ) Copy (with permissions) contents of copy/ dir to /where/to/ ( tar -c /dir/to/copy ) | ssh -C user@remote ‘cd /where/to/ && tar -x -p’ Copy (with permissions) copy/ dir to remote:/where/to/ dir dd bs=1M if=/dev/sda | gzip | ssh user@remote ‘dd of=sda.gz’ Backup harddisk to remote machine Networking Command What is does ethtool eth0 Show status of ethernet interface eth0 ethtool --change eth0 autoneg off speed 100 duplex full Manually set ethernet interface speed iw dev wlan0 link Show link status of wireless interface wlan0 iw dev wlan0 set bitrates legacy-2.4 1 Manually set wireless interface speed iw dev wlan0 scan List wireless networks in range ip link show List network interfaces ip link set dev eth0 name wan Rename interface eth0 to wan ip link set dev eth0 up Bring interface eth0 up (or down) ip addr show List addresses for interfaces ip addr add 1.2.3.4/24 brd + dev eth0 Add (or del) ip and mask (255.255.255.0) ip route show List routing table ip route add default via 1.2.3.254 Set default gateway to 1.2.3.254 ss -tupl List internet services on a system ss -tup List active connections to/from system host bencode.net Lookup DNS ip address for name or vice versa hostname -i Lookup local ip address (equivalent to host hostname) whois bencode.net Lookup whois info for hostname or ip address mtr google.com Nice trace route Text Manipulation Command What is does sed 's/string1/string2/g' Replace string1 with string2 sed 's/\(.*\)1/\12/g' Modify anystring1 to anystring2 sed '/^ *#/d; /^ *$/d' Remove comments and blank lines sed ':a; /\\$/N; s/\\\n//; ta' Concatenate lines with trailing \ sed 's/[ \t]*$//' Remove trailing spaces from lines seq 10 | sed "s/^/ /; s/ *\(.\{7,\}\)/\1/" Right align numbers seq 10 | sed p | paste - - Duplicate a column sed -n '1000{p;q}' Print 1000th line sed -n '10,20p;20q' Print lines 10 to 20 sed -n 's/.*<title>\(.*\)<\/title>.*/\1/ip;T;q' Extract title from HTML web page sed -i 42d ~/.ssh/known_hosts Delete a particular line sort -t. -k1,1n -k2,2n -k3,3n -k4,4n Sort IPV4 ip addresses echo 'Test' | tr '[:lower:]' '[:upper:]' Case conversion tr -dc '[:print:]' < /dev/urandom Filter non printable characters tr -s '[:blank:]' '\t' </proc/diskstats | cut -f4 cut fields separated by blanks history | wc -l Count lines seq 10 | paste -s -d ' ' Concatenate and separate line items to a single line sort -u file1 file2 Union of unsorted files sort file1 file2 | uniq -d Intersection of unsorted files sort file1 file1 file2 | uniq -u Difference of unsorted files sort file1 file2 | uniq -u Symmetric Difference of unsorted files join -t'\0' -a1 -a2 file1 file2 Union of sorted files join -t'\0' file1 file2 Intersection of sorted files join -t'\0' -v2 file1 file2 Difference of sorted files join -t'\0' -v1 -v2 file1 file2 Symmetric Difference of sorted files shuf file1 Randomise lines in a file comm file1 file2 Combine lines from two sorted files Set Operations Command What is does sort -u file1 file2 Union of unsorted files sort file1 file2 | uniq -d Intersection of unsorted files sort file1 file1 file2 | uniq -u Difference of unsorted files sort file1 file2 | uniq -u Symmetric Difference of unsorted files join -t'\0' -a1 -a2 file1 file2 Union of sorted files join -t'\0' file1 file2 Intersection of sorted files join -t'\0' -v2 file1 file2 Difference of sorted files join -t'\0' -v1 -v2 file1 file2 Symmetric Difference of sorted files Windows Networking Command What is does smbtree Find windows machines. See also findsmb nmblookup -A 1.2.3.4 Find the windows (netbios) name associated with ip address smbclient -L windows_box List shares on windows machine or samba server mount -t smbfs -o fmask=666,guest //windows_box/share /mnt/share Mount a windows share echo 'message' | smbclient -M windows_box Send popup to windows machine Monitoring and Debugging Command What is does tail -f /var/log/messages Monitor messages in a log file strace -c ls >/dev/null Summarise/profile system calls made by command strace -f -e open ls >/dev/null List system calls made by command strace -f -e trace=write -e write=1,2 ls >/dev/null Monitor what’s written to stdout and stderr ltrace -f -e getenv ls >/dev/null List library calls made by command lsof -p $$ List paths that process id has open lsof ~ List processes that have specified path open tcpdump not port 22 Show network traffic except ssh. See also tcpdump_not_me ps -e -o pid,args --forest List processes in a hierarchy ps -e -o pcpu,cpu,nice,state,cputime,args --sort pcpu | sed '/^ 0.0 /d' List processes by % cpu usage ps -e -orss=,args= | sort -b -k1,1n | pr -TW$COLUMNS List processes by mem (KB) usage. See also ps_mem.py ps -C firefox-bin -L -o pid,tid,pcpu,state List all threads for a particular process ps -p 1,$$ -o etime= List elapsed wall time for particular process IDs watch -n.1 pstree -Uacp $$ Display a changing process subtree last reboot Show system reboot history free -m Show amount of (remaining) RAM (-m displays in MB) watch -n.1 'cat /proc/interrupts' Watch changeable data continuously udevadm monitor Monitor udev events to help configure rules ulimit -Sv 1000 Limit memory usage for following commands to 1MiB fuser -k 8000/tcp Kill the program using port 8000 lsof -p 123,789 -u 1234,abe All files used by PID 123 or 789, or by user abe or UID 1234 kill -HUP $(lsof -t /home/foo/file) SIGHUP the processes using /home/foo/file cat /dev/urandom | base64 | pv -lbri2 > /dev/null Monitor progress of output Disk Space Command What is does ls -lSr Show files by size, biggest last du -s * | sort -k1,1rn | head Show top disk uses in current dir du -hs /home/* | sort -k1,1h Sort paths by easy to interpret disk usage df -h Show free space on mounted filesystems df -i Show free inodes on mounted filesystems fdisk -l Show disks partitions sizes and types (run as root) rpm -q -a --qf '%10{SIZE}\t%{NAME}\n' | sort -k1,1n List all packages by installed size (Bytes) on rpm distros dpkg-query -W -f='${Installed-Size;10}\t${Package}\n' | sort -k1,1n List all packages by installed size (KBytes) on deb distros dd bs=1 seek=2TB if=/dev/null of=ext3.test Create a large test file (taking no space) > file truncate data of file or create an empty file CD/DVD Command What is does gzip < /dev/cdrom > cdrom.iso.gz Save copy of data cdrom mkisofs -V LABEL -r dir | gzip > cdrom.iso.gz Create cdrom image from contents of dir mount -o loop cdrom.iso /mnt/dir Mount the cdrom image at /mnt/dir (read only) wodim dev=/dev/cdrom blank=fast Clear a CDRW gzip -dc cdrom.iso.gz | wodim -tao dev=/dev/cdrom -v -data - Burn cdrom image cdparanoia -B Rip audio tracks from CD to wav files in current dir wodim -v dev=/dev/sr0 -audio -pad *.wav Make audio CD from all wavs in current dir oggenc --tracknum=$track track.cdda.wav -o track.ogg Make ogg file from wav file for i in *.mp3; do mpg123 --rate 44100 --stereo --buffer 3072 --resync -w "$(basename $i .mp3).wav" $i; done Decode mp3 files to 16-bit, stereo, 44.1 kHz waves for i in *.mp3; do lame --decode $i ``basename $i .mp3``.wav; done Decode mp3 files to 16-bit, stereo, 44.1 kHz waves normalize -m *.wav Normalise levels in wavs, mix mode is loud as possible Locales Command What is does printf "%'d\n" 1234 Print number with thousands grouping appropriate to locale BLOCK_SIZE=\'1 ls -l Use locale thousands grouping in ls. See also l echo "I live in$(locale territory)" Extract info from locale database LANG=en_IE.utf8 locale int_prefix Lookup locale info for specific country. See also ccodes locale -kc $(locale | sed -n 's/\(LC_.\{4,\}\)=.*/\1/p') | less List fields available in locale database Dates and Times Command What is does cal -3 Display a calendar cal 9 1752 Display a calendar for a particular month year date -d fri What date is it this friday [ $(date -d '12:00 today +1 day' +%d) = '01' ] || exit exit a script unless it’s the last day of the month date --date='25 Dec' +%A What day does xmas fall on, this year date --date='@2147483647' Convert seconds since the epoch (1970-01-01 UTC) to date TZ='America/Los_Angeles' date What time is it on west coast of US (use tzselect to find TZ) date --date='TZ="America/Los_Angeles" 09:00 next Fri' What’s the local time for 9AM next Friday on west coast US Images Most of these rely on the imagemagick cli programs. ...

Here I aim to cover a set of common administration tasks. Things like, the hostname, system logs, what users are currently logged in, physical devices that are connected, logical volumes, file system and inode allocation, attached network interfaces and their addressing, processes and daemons currently running, kernel verison, local users and groups, installed packages, remote mounts, network shares, system uptime, bread and butter OS stats (CPU, IO, network, memory). Booting shutdown -r +5 System going down for a reboot #wall broadcast msg shutdown -c #cancel reboot shutdown -r 00:00 #schedule for midnight shutdown -h +5 #halt system in 5 mins shutdown -h now Alternatively, just use systemd: ...

The OpenSSH secure shell, ssh, provides the necessary client/server security plumbing, to allow shell execution on a remote machine. ssh can be used interactively, as per a normal shell, or to run one off commands, for example: $ ssh ben@wookie.local uname -a ben@wookie.local's password: Linux wookie.local 3.14-1-686-pae #1 SMP Debian 3.14.7-1 (2014-06-16) i686 GNU/Linux Hot tip: the w command is gem for showing users currently logged in $ w -f USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT ben tty2 :0 17:05 10:36m 16:53 50.97s /usr/bin/evince /home/ben/podcasts/redhat/Docs/RH124-RHEL7.pdf User ben logged into virtual console 2 (tty2) via a graphical login (:0) at about 5PM. OK, lets access a couple of the virtual terminals: ...

Bash is a Unix shell written by Brian Fox in 1989 for the GNU Project as a free replacement for the Bourne shell. To this day, Bash remains one of the most powerful and ubiquitous scripting tools on the planet. Contents Useful Shortcuts Initialisation Shell Grammar Variables Local variables Environment variables Positional arguments Expansions Brace expansion Command substitution Arithmetic expansion Double and single quotes Stream Redirection here documents Arrays Conditions if statements case statements Loops For Loops Select Loops While Loops Until Loops Functions Coprocesses Builtins Bash Recipes Top 6 largest things in the current directory Display the 23rd line of /etc/passwd Filter the first column from process status Delete Subversion scrap files Move shell scripts and mark them as executable Pattern matching Scan code base against list of patterns Rename Multiple Files Run a command every time a file is modified Keep a program running after leaving SSH session Simple menu and functions Complete example Resources Kudos to Denys Dovhan and his awesome Bash handbook. The most digestable, and enjoyable method I’ve found to groking bash. <3 ...