Bash is a Unix shell written by Brian Fox in 1989 for the GNU Project as a free replacement for the Bourne shell. To this day, Bash remains one of the most powerful and ubiquitous scripting tools on the planet.

Contents⌗

• Useful Shortcuts
• Initialisation
• Shell Grammar
  • Variables
    • Local variables
    • Environment variables
    • Positional arguments
  • Expansions
    • Brace expansion
    • Command substitution
    • Arithmetic expansion
    • Double and single quotes
  • Stream Redirection
    • here documents
  • Arrays
  • Conditions
    • if statements
    • case statements
• Loops
  • For Loops
  • Select Loops
  • While Loops
  • Until Loops
• Functions
• Coprocesses
• Builtins
• Bash Recipes
  • Top 6 largest things in the current directory
  • Display the 23rd line of /etc/passwd
  • Filter the first column from process status
  • Delete Subversion scrap files
  • Move shell scripts and mark them as executable
  • Pattern matching
  • Scan code base against list of patterns
  • Rename Multiple Files
  • Run a command every time a file is modified
  • Keep a program running after leaving SSH session
  • Simple menu and functions
  • Complete example
• Resources

Kudos to Denys Dovhan and his awesome Bash handbook. The most digestable, and enjoyable method I’ve found to groking bash. <3

Useful Shortcuts⌗

See man readline

Movement

• Ctrl + A: move cursor to beginning of line
• Ctrl + E: move cursor to end of line
• Ctrl + L: clear screen
• Alt + F: Move cursor forward one word on the current line
• Alt + B: Move cursor backward one word on the current line

Editing

• Ctrl + U: delete from cursor to beginning of line
• Ctrl + K: delete from cursor to end of line
• Ctrl + W: delete whole word before the cursor
• Ctrl + H: delete last character (backspace)
• Ctrl + T: transfer (swap) the last two characters before the cursor
• Esc+T: transfer (swap) the last two words before the cursor

Process

• Ctrl + C: kill whatever is interactively running
• Ctrl + D: exit the current session
• Ctrl + Z: puts whatever is running into a suspended background process, use fg to restore it.

History

• Ctrl + R: search through previous commands
• Ctrl + P: previous command (from history)
• Ctrl + N: next command (from history)
• !!: the last command (handy when you forget to sudo, sudo !!)
• !1050: run command 1050 (as journalled by history)

Initialisation⌗

Bash provides several config files, that can be used when a fresh bash instance is created. Such as:

• ~/.bash_profile user login-shell config
• ~/.profile user login-shell config
• ~/.bashrc user interactive (sub) shell config
• /etc/bash_profile system-side login shell config
• /etc/profile system-wide login-shell config
• /etc/bashrc system-wide interactive (sub) shell config

A single .bashrc soon become bloated and convoluted. A nifty trick is to break things up into several smaller configs (.bashrc), and “including” them with the source command.

First somewhere to house the individual configs:

mkdir ~/.bashrc.d
chmod 700 ~/.bashrc.d

Then in the .bashrc or .bash_profile include all child configs:

for file in ~/.bashrc.d/*.bashrc;
do
  source "$file"
done

Then rip out chunks into ~/.bashrc.d/myfile.bashrc. Ensure they all have execution rights:

chmod +x ~/.bashrc.d/*.bashrc

Shell Grammar⌗

Variables⌗

Local variables⌗

A local variable can be declared using = sign (no spaces) and its value can be retrieved using the $ sign.

os="linux"
echo $os
unset os

We can also declare a variable local to a single function using the local keyword.

local local_var="NAND gate"

Environment variables⌗

Environment variables are variables accessible to anything running in current shell session. They are created just like local variables, but using the keyword export instead.

export GLOBAL_VAR="guten tag"

Bash comes with a bunch of reserved variables, here’s some handy ones:

Some like to make their prompt stand out from the noise (e.g. by making it bright green). Add to ~/.bash_profile:

export PS1="\[$(tput bold)\]\[$(tput setaf 2)\][\u@\h \W]\\$ \[$(tput sgr0)\]"

Or goes nuts and colorise each component:

export PS1="\[$(tput bold)\]\[$(tput setaf 1)\][\[$(tput setaf 3)\]\u\[$(tput setaf 2)\]@\[$(tput setaf 4)\]\h \[$(tput setaf 5)\]\W\[$(tput setaf 1)\]]\[$(tput setaf 7)\]\\$ \[$(tput sgr0)\]"

Positional arguments⌗

Defined within the context of a function.

• $0: Name of script.
• $1 to $9: The parameter list elements from 1 to 9.
• ${10} to ${N}: The parameter list elements from 10 to N.
• $* or $@: All positional parameters except $0.
• $#: The number of parameters, not counting $0.
• $FUNCNAME: The function name.

Expansions⌗

Brace expansion⌗

Using a pair curly braces { and }, can be used to generate strings or ranges of numbers.

echo beg{i,a,u}n # begin began begun
echo {e..a} # e d c b a

echo {0..5} # 0 1 2 3 4 5
echo {00..8..2} # 00 02 04 06 08

Command substitution⌗

Stores the result of an evaluation into a variable, or passes the result along for another evaluation. Done by enclosing the expression either in backticks `, or within $().

kernel=$(uname -r)
#or
kernel=`uname -r`
echo $kernel #4.4.6-301.fc23.x86_64

Arithmetic expansion⌗

Bash eats arithmetic for breakfast. Syntax is similar to the command substitution capture group $(), but doubles up on the braces $(()).

bug=$(( ((10 + 5*3) - 7) / 2 ))
echo $bug # 9
echo \$(( bug \* 1000 )) #9000

Double and single quotes⌗

Unlike single quotes, with double quotes, variables and command substitutions are expanded automatically.

echo "Your home: $HOME" # Your home: /home/vimjock
echo 'Your home: $HOME' # Your home: \$HOME

If a variable contains whitespace, take care to expand it in double quotes, which will preserve the literal value of all characters.

WEIRD="A string that is just trouble"
echo $WEIRD   # A string that is just trouble
echo "$WEIRD" # A string that is just trouble

Words of the form $'string' expands, with backslash-escaped characters replaced as specified by the ANSI C standard (such as \n for newline, \t for horizontal tab, \u3b2 for unicode character 3b2, and so on).`

Stream Redirection⌗

Bash views the outside world (input and output) as streams of data. The brilliant thing about streams, like water streams, is that their flow can be channeled in and out of many upstream and/or downstream programs, creating complex results.

0 | stdin | The standard input. 1 | stdout | The standard output. 2 | stderr | The errors output.

Redirection operators for controlling the flow of streams:

The order of redirections is important.

ls > dirlist 2>&1

Directs both stdout and stderr to file dirlist. While:

ls 2>&1 > dirlist

Directs only stdout to the file dirlist.

Another example:

join <(sort file1.txt) <(sort file2.txt)

This will bind the outputs of two sort commands, as input arguments one and two of the join command:

Bash redirection can integrate with logical devices:

/dev/fd/fd | File descriptor fd is duplicated. /dev/stdin | File descriptor 0 is duplicated. /dev/stdout | File descriptor 1 is duplicated. /dev/stderr | File descriptor 2 is duplicated. /dev/tcp/host/port | Open the corresponding TCP socket. /dev/udp/host/port | Open the corresponding UDP socket.

Often when running text searches as a low privileged user, you will encounter permission and other errors, like this:

$ grep ben /etc/*
grep: abrt: Is a directory
grep: audisp: Permission denied
grep: audit: Permission denied
...

Lets redirect them to /dev/null:

$ grep ben /etc/* 2> /dev/null
/etc/group:wheel:x:10:ben
/etc/group:users:x:100:ben
/etc/group:ben:x:1000:ben
/etc/group:postgres:x:26:root,ben
...

Nice and clean.

here documents⌗

here documents are a bit rad:

$ python - <<"XXXX"
> foo=15
> print "Magical number of foo is %i.\n" %(foo,)
> XXXX
Magical number of foo is 15.

Another:

sudo tee /etc/systemd/system/docker.service.d/http-proxy.conf << END
[Service]
Environment="HTTP_PROXY=http://proxy.bencode.net:8080"
Environment="HTTPS_PROXY=http://proxy.bencode.net:8080"
END

Arrays⌗

langs[0]=c
langs[1]=java
langs[2]=go

Or as a single compound assignment:

langs=(c java go)

To refer individual elements:

echo ${langs[1]} # java
echo ${langs[*]} # c java go
echo \${langs[@]} # c java go

The @ operator (unlike *) can honor whitespace.

langs=(c java go "visual basic")
printf "+ %s\n" "\${langs[@]}"

# c

# java

# go

# visual basic

Slicing:

langs=(c java go "visual basic")
echo \${langs[@]:1:2}

# java go

Adding:

langs=(c awk)
echo \${langs[@]}

# c awk

langs=(java "${langs[@]}" sql bash)
echo ${langs[@]}

# java c awk sql bash

Removing:

langs=(ruby python "visual basic" perl)
unset langs[2]
echo \${langs[@]}

# ruby python perl

Looping:

for lang in ${langs[@]}; do echo "$lang is nifty"; done

# ruby is nifty

# python is nifty

# perl is nifty

Conditions⌗

Expression is enclosed in double squares [[ ]]. Expressions can be daisy chained using the && and/or || operators.

File system expressions:

Shell variables:

Strings:

Arithmetic:

if statements⌗

With single and multi line variants:

# single line

if [[ 100 -eq 100 ]]; then echo "one hungey"; else echo "no hungey"; fi

# multi line

if [[ "drpepper" == "drpepper" ]]; then
echo "one hungey"
else
echo "no hungey"
fi

# if else

if [[ -e main.c ]]; then
echo "found main"
elif [[ $(date +%A) == "Sunday" ]]; then
echo "day of rest"
else
echo "no dice"
fi

In some instances, such as pattern matching, omit double quotes:

if [[ $file == *.o ]]; then echo "its an ELF"; fi

case statements⌗

| to delimit multiple patterns,) to terminate the pattern list, * as default catch all pattern, and ;; to divide each block.

file=h

if [[ -n $file ]]; then

case \$file in
"c"|"h")
echo "my precious source code"
;;
"o")
echo "silly object file, nuke it"
;;
\*)
echo "something else"
;;
esac
else
echo "not found bra";
fi

Loops⌗

Bash comes with C-like looping; for, for in, select, while and until loops. In addition bash also provides builtin break and continue commands, for manipulating the flow of loops.

For Loops⌗

The super handy for.

for hero in linus stallman ritchie kernighan pike fox
echo \$hero
done

#linus
#stallman
#ritchie
#kernighan
#pike
#fox

Single line syntax:

for i in {1..5}; do echo \$i; done

#1
#2
#3
#4
#5

And lastly, the classical for:

for (( i = 0; i < 5; i++ )); do
echo \$i;
done

#0
#1
#2
#3
#4

Move shell scripts from one location, to another and change their permissions along the way.

for FN in $HOME/*.sh; do
  mv "$FN" "$HOME/scripts"
  chmod +x "$HOME/scripts/\${FN}"
done

Select Loops⌗

Useful for creating menus. The list of expanded words from a list is printed out, each preceded by a number. The PS3 prompt is then displayed and a line is read from standard input.

#!/bin/bash
PS3="Please choose an environment: "
select ENV in dev tst ppd prd
do
echo -n "Enter build version to deploy: " && read BUILD
case $ENV in
    dev) echo "./doinst.bash $BUILD noddy" ;;
tst) echo "./doinst.bash $BUILD tulip" ;;
    ppd) echo "./doinst.bash $BUILD woody" ;;
prd) echo "./doinst.bash \$BUILD chipper" ;;
esac
break;
done

Here’s what this does:

$ ./select.bash
1) dev
2) tst
3) ppd
4) prd
Please choose an environment: 2
Enter build version to deploy: 1.6
./doinst.bash 1.6 tulip

While Loops⌗

The awesome keeps on coming.

x=0
while [[ $x -lt 5 ]]; do
echo $(( x * x ))
  x=$(( x + 1 ))
done

#0
#1
#4
#9
#16

Until Loops⌗

Opposite of while; keeping looping if the condition is false.

Functions⌗

A shell function stores a series of commands for later execution.

cool_func() {
echo "be cool"
}

cool_func

When a function is executed, the arguments to the function be the positional parameters during its execution. When the function is complete, these values are restored to the values they had prior to the functions execution. The function can return a result using an exit code.

get_day() {
day=\$(date +%A)

if [[ -n $1 ]]; then
echo "g'day $1, its $day"
else
echo "g'day cobber"
fi

return 0
}

get_day Benjamin # g'day Benjamin, its Wednesday
get_day # g'day cobber

Coprocesses⌗

Builtins⌗

bash, :, ., [, alias, bg, bind, break, builtin, caller, cd, command, compgen, complete, compopt, continue, declare, dirs, disown, echo, enable, eval, exec, exit, export, false, fc, fg, getopts, hash, help, history, jobs, kill, let, local, logout, mapfile, popd, printf, pushd, pwd, read, readonly, return, set, shift, shopt, source, suspend, test, times, trap, true, type, typeset, ulimit, umask, unalias, unset, wait

Bash Recipes⌗

Top 6 largest things in the current directory⌗

du -hxs * | sort -hr | head -6
234M    code
30M     cygwin
6.8M    datatsudio
5.0M    c-projects
4.9M    scripts

Display the 23rd line of /etc/passwd⌗

head -n 23 /etc/passwd | tail -n 1
sed -n ' 23 p ' /etc/passwd
awk ' NR == 23 { print $0 } ' < /etc/passwd

Filter the first column from process status⌗

awk ' { print $1 } ' <(ps -aux)

Delete Subversion scrap files⌗

Delete files that report a status of ?

$ svn status bnc
M    bnc/amin.c
?    bnc/dmin.c
?    bnc/mdiv.tmp
A    bnc/optrn.c
M    bnc/optson.c
?    bnc/prtbout.4161
?    bnc/rideaslist.odt

A possible solution using a while loop and a combination of grep, cut and rm:

svn status bnc | grep '^?' | cut -c8- | while read FILE; do echo "$FN"; rm -rf "$FN"; done

Alternatively the read statement can be used to do the parsing:

svn status bnc | \
while read TAG FN
do
if [[ $TAG == \? ]]
then
echo $FN
    rm -rf "$FN"
fi
done

Move shell scripts and mark them as executable⌗

for FN in $HOME/*.sh; do
  mv "$FN" "$HOME/scripts"
  chmod +x "$HOME/scripts/\${FN}"
done

Pattern matching⌗

text=DOOM89761234rocks

if [[ $text =~ ([[:alpha:]]*)[[:digit:]]+([[:alpha:]]*) ]]; then
  echo "${BASH_REMATCH[1]} \${BASH_REMATCH[2]}";
else
echo "wat";
fi

Output

DOOM rocks

Scan code base against list of patterns⌗

Given a list of patterns, scan a code base for them, and report a total of how many hits there were for each.

cut -d $'\t' -f 2 keywords.txt | while read KEYWORD; do COUNT=$(grep -rnwo ~/code/git/das/src --include _.java --include _.jsp -e \""$KEYWORD"\" | wc -l); if [[ $COUNT -gt 0 ]]; then echo "$KEYWORD $COUNT"; fi; done

#AUTHOR 8
#ENV 19

#

Rename Multiple Files⌗

Given a bunch of files named in the form game.of.thrones.s04e10.hdtv.x264.mp4, each contained within their own subdirectory. First I wanted to remove the all subdirs, flattening out the tree, and then finish up by renaming each file to the simplier form GOT.S04E10.mp4.

Using find, locate all subdirectories, moving any contents they may have back into the parent directory. The -exec switch has its limitations, such as with logical && operators, which requires a real shell (sh).

find . -name "Game*" -type d -exec sh -c 'cd "{}" && mv * ../' \;

The subdirectories can be disposed of:

find . -name "Game*" -type d -exec rm -rf "{}" \;

Now the renaming with sed. By leveraging capture groups (donuts) in sed (eg \1), can pick out match results of interest and jam them into the substitution result. While here takes advantage of the casing functionality by decorating the capture group with either a \U for uppercasing or \L for lower.

for f in *.*; do mv "$f" "GOT.`echo $f | sed -rn ' s/.*([sS][0-9]{2}[eE][0-9]{2}).*(\..{3})/\U\1\L\2/p '`"; done

Run a command every time a file is modified⌗

while inotifywait -e close_write report.tex
do
  make
done

Keep a program running after leaving SSH session⌗

If no input is required:

nohup ./script.sh &

Otherwise:

./script.sh
<provide input as needed>
<Ctrl-Z>            # sleep the process
jobs -l             # figure out the job id
disown -h <jobid>   # disown the job
bg -h <jobid>       # continue running

Simple menu and functions⌗

me="$(basename "$(test -L "$0" && readlink "$0" || echo "$0")")"
u="$USER"
stow=/usr/bin/stow

menu()
{
	echo "usage:   " $me "[OPTION]"
	echo " "
	echo "init:    Install the basics (git/yay)"
	echo "dots:    Get dots from github (into '~/dots' folder)"
	echo "stow:    Restore home stow from dots repo"
	echo "unstow:  Cleanup home stow from dots repo"
	echo "apps:    Use 'yay' to install all programs"
	echo "dwm:     Clones dwm repo and applies patches"
	echo " "
}

init()
{
	cd /tmp/
	curl -LO https://aur.archlinux.org/cgit/aur.git/snapshot/yay.tar.gz
	tar xvzf yay.tar.gz
	cd yay
	makepkg -sci
	sudo pacman -S --needed git
}

dwm()
{
	cd ~
	rm -r dwm
	git clone git://git.suckless.org/dwm
	cd dwm
	curl -LO https://dwm.suckless.org/patches/center/dwm-center-6.1.diff:
}

apps()
{
	test -f ~/dots/restore/applist && yay -S --needed - < ~/dots/restore/applist || echo "Do dots & stow first dude!"
}

dots()
{
	cd ~
	git clone git@github.com:bm4cs/dots.git
}

stow()
{
	cd ~/dots/stow-home
	for d in *; do $stow -t ~ $d; done

	#Setup ROOT stow files
	#cd ~/dots/stow_root; for d in *; do sudo stow -t / $d; done
}

unstow()
{
	cd ~/dots/stow-home
	for d in *; do
		$stow -D -t ~ $d || true
	done
}

restow()
{
    if [ -n "$1" ]; then
	    cd ~/dots/stow-home/
        $stow -D -t ~ $1 || true
	    $stow -t ~ $1
    fi
}



if [ -n "$1" ]; then
	$1 ${@:2}
else
	menu
fi

Complete example⌗

dpkgArch="$(dpkg --print-architecture)" \
  && case "${dpkgArch##*-}" in \
    amd64) ARCH='x64';; \
    ppc64el) ARCH='ppc64le';; \
    s390x) ARCH='s390x';; \
    arm64) ARCH='arm64';; \
    armhf) ARCH='armv7l';; \
    i386) ARCH='x86';; \
    *) echo "unsupported architecture"; exit 1 ;; \
  esac \
  # gpg keys listed at https://github.com/nodejs/node#release-keys
  && set -ex \
  && for key in \
    4ED778F539E3634C779C87C6D7062848A1AB005C \
    94AE36675C464D64BAFA68DD7434390BDBE9B9C5 \
    74F12602B6F1C4E913FAA37AD3A89613643B6201 \
    71DCFD284A79C3B38668286BC97EC7A07EDE3FC1 \
    8FCCA13FEF1D0C2E91008E09770F7A9A5AE15600 \
    C4F0DFFF4E8C1A8236409D08E73BC641CC11F4C8 \
    C82FA3AE1CBEDC6BE46B9360C43CEC45C17AB93C \
    DD8F2338BAE7501E3DD5AC78C273792F7D83545D \
    A48C2BEE680E841632CD4E44F07496B3EB3C1762 \
    108F52B48DB57BB0CC439B2997B01419BD92F80A \
    B9E2F5981AA6E0CD28160D9FF13993A75599653C \
  ; do \
      gpg --batch --keyserver hkps://keys.openpgp.org --recv-keys "$key" || \
      gpg --batch --keyserver keyserver.ubuntu.com --recv-keys "$key" ; \
  done \
  && curl -fsSLO --compressed "https://nodejs.org/dist/v$NODE_VERSION/node-v$NODE_VERSION-linux-$ARCH.tar.xz" \
  && curl -fsSLO --compressed "https://nodejs.org/dist/v$NODE_VERSION/SHASUMS256.txt.asc" \
  && gpg --batch --decrypt --output SHASUMS256.txt SHASUMS256.txt.asc \
  && grep " node-v$NODE_VERSION-linux-$ARCH.tar.xz\$" SHASUMS256.txt | sha256sum -c - \
  && tar -xJf "node-v$NODE_VERSION-linux-$ARCH.tar.xz" -C /usr/local --strip-components=1 --no-same-owner \
  && rm "node-v$NODE_VERSION-linux-$ARCH.tar.xz" SHASUMS256.txt.asc SHASUMS256.txt \
  && ln -s /usr/local/bin/node /usr/local/bin/nodejs \
  # smoke tests
  && node --version \
  && npm --version

Resources⌗

• Awesome Bash - A curated list of delightful Bash resources
• Denys Dovhans Bash handbook
• Bashful - A collection of libraries to simplify writing Bash scripts