A survey of the standard and high quality programs that feature in most Unix based distributions, with the GNU variants being my favourite. The bash shell is a great way of interfacing and orchestrating these beautifully crafted programs. As a starting point, I’ve listed each program offered by the GNU Core Utilities and util-linux umbrella projects; considered the de facto standard on most distributions.

• Quick Reference
  • General
  • System Information
  • Directory Navigation
  • File Searching
  • Archiving and Compression
  • Networking
  • Text Manipulation
  • Set Operations
  • Windows Networking
  • Monitoring and Debugging
  • Disk Space
  • CD/DVD
  • Locales
  • Dates and Times
  • Images
• Finding Documentation
  • Manual Pages
  • Appropriate Commands
  • whatis
  • GNU Info Entry
  • /usr/share/doc Documentation
  • RPM bundled documentation
• Examples
  • grep
  • cut
  • sort
  • tr
  • wc
  • tar
  • rsync
  • sed
  • awk
  • ssh (Secure Shell)
  • wget
• BFL of Common Programs
• Resources

Quick Reference

General

System Information

Directory Navigation

File Searching

Archiving and Compression

Networking

Text Manipulation

Set Operations

Windows Networking

Monitoring and Debugging

Disk Space

CD/DVD

Locales

Dates and Times

Images

Most of these rely on the imagemagick cli programs.

identify foo.jpg | Show meta including resolution

Finding Documentation

Manual Pages

The infamous manual (man) page documentation system. Man pages are organised by the following sections:

An explicit section can be requested. For the man page relating to the file format of /etc/passwd

man 5 passwd

The -k switch is great for searching across man’s treasure chest of documentation. For example, say you want to set the system time, but have no idea what program to use to achieve this. Use the -k switch to scan documentation for time.

$ man -k time
ac (1)               - print statistics about users connect time
adjtime (3)          - correct the time to synchronize the system clock
adjtimex (2)         - tune kernel clock
after (n)            - Execute a command after a time delay
aio_suspend (3)      - wait for asynchronous I/O operation or timeout
asctime (3)          - transform date and time to broken-down time or ASCII
asctime (3p)         - convert date and time to a string

The lions share of search results seems to come from section 2 and 3 (C kernel and library calls). Focusing on the task at hand, administering the system time, lets filter results to man sections 1 (user commands) and 8 (system management commands).

$ man -k time | grep -Pe '.*\([1,8]\).*'
ac (1)               - print statistics about users connect time
booleans (8)         - Policy booleans enable runtime customization of SELinux policy
ccrewrite (1)        - Rewrite CLR assemblies for runtime code contract verification.
chrt (1)             - manipulate the real-time attributes of a process
date (1)             - print or set the system date and time
dnssec-settime (8)   - Set the key timing metadata for a DNSSEC key
jack_showtime (1)    - The JACK Audio Connection Kit example client

The date program looks perfect.

Appropriate Commands

Basically an equivalent to the man -k switch for searching.

$ apropos clock
adjtime (3)          - correct the time to synchronize the system clock
adjtimex (2)         - tune kernel clock
alarm (2)            - set an alarm clock for delivery of a signal
clock (3)            - determine processor time

whatis

For a very brief overview of a man page matching a keyword.

$ whatis vim
vim (1)              - Vi IMproved, a programmers text editor

GNU Info Entry

A purpose built documentation system from GNU, info features hyperlinks (prefixed with *), aimed at dealing with larger documentation sets than man.

info goes against the grain in terms of keyboard navigation. Its odd. Page up, page down, enter to follow a link, and l to go back,

Some keys for driving info:

• n next node
• p previous node
• u parent node
• t top node
• home end pgup pgdn scroll content
• l go back
• q quit
• H keyboard shortcuts cheatsheet

Searching info:

$ info --apropos=tee
"(coreutils)tee invocation" -- tee
"(libc)Control Functions" -- feupdateenv
"(gawk)Tee Program" -- 'tee' utility

And then info gawk tee for example to pull up the third result.

/usr/share/doc Documentation

A gold mine of documents and sample configuration files. Usually for distributions that are not considered core, and don’t offer man or info pages.

RPM bundled documentation

$ rpm -qd tmux
/usr/share/doc/tmux/CHANGES
/usr/share/doc/tmux/FAQ
/usr/share/doc/tmux/TODO
/usr/share/man/man1/tmux.1.gz

Examples

grep

grep prints lines that contain a match for a pattern.

Useful modes:

• -r or -R for recursive
• -n show line number
• -w match the whole word
• -v invert match (i.e. blacklist)
• -l just give the file name of matching files
• -i case insensative
• -P Perl style regular expressions

Recursively search all files from the current directory, containing Romero, including the line number where they are found:

{% highlight bash %} $ grep -rnw . -e ‘Romero’ Binary file ./datatsudio/.metadata/.plugins/seg0/c530.dat matches ./files/diff/heros_new:4:Romero,John,671028 ./files/heros:7:Romero,John,671028 {% endhighlight %}

The --include and --exclude are very useful for filtering target files, and the amount of work grep needs to do. Exclude *.dat binary files from the above example:

{% highlight bash %} $ grep -rnw . -e ‘Romero’ –exclude ‘*.dat’ ./grep/diff/heros_new:4:Romero,John,671028 ./grep/heros:7:Romero,John,671028 {% endhighlight %}

Perl patterns:

{% highlight bash %} $ echo “2016-10-13” | grep -Pe ‘\d{4}-\d{2}-\d{2}’ 2016-10-13 {% endhighlight %}

Color highlight numeric 0 to 5:

{% highlight bash %} $ echo “2016-10-13” | grep –color ‘[0-5]’ 2016-10-13 {% endhighlight %}

Overall total of how many times an expression matches:

{% highlight bash %} $ grep -rnwo . –include *.bash –include *.sh -e ‘BASH_REMATCH’ | wc -l 12 {% endhighlight %}

cut

Removes portions from each line of input. By default will use standard input, when no FILE specified, or when FILE is -.

Select the first field for the colon delimitered file /etc/passwd.

$ cut -d : -f 1 /etc/passwd
LocalService
NetworkService
Guest
SYSTEM

Hack just the date portion (chars 1-10) off the front of logs, and show the unique dates:

$ cut -c1-10 dircdds.log | grep -Pe '\d{4}-\d{2}-\d{2}' | sort -h | uniq
2016-03-21
2016-03-22
2016-03-24
2016-03-29
2016-04-21

sort

By default will sort in dictionary order.

$ cut -d : -f 3 /etc/passwd | sort
0
1
1000
1001
107
11
113
12

Useful sort modes:

• -h human numeric (e.g. 2K 3G)
• -n numeric
• -r reverse
• -R random
• -u unique

tr

For translating (e.g. uppercasing, stripping, truncating, etc) text.

Convert lower case characters to upper.

$ echo "Linus Torvalds" | tr [:lower:] [:upper:]
LINUS TORVALDS

Make all lower case characters o:

$ echo "Linus Torvalds" | tr [:lower:] o
Loooo Tooooooo

Replace the range of characters a to o, with @:

$ echo "Linus Torvalds" | tr a-o @
L@@us T@rv@@@s

wc

Count aggregates of the contents of a file.

By default will show counts of lines, words and bytes.

$ wc pthreads.make
7  29 252 pthreads.make

Useful counts:

• -l, --lines newlines
• -w, --words words
• -c, --bytes bytes
• -m, --chars characters

Just show the number of lines:

$ wc -l pthreads.make
7  pthreads.make

Pipe support just works:

$ cat 2016-05-01-bash.markdown | wc -l
1100

tar

The rock solid archiving tool that you can always lean on.

Create an archive of all of the /etc directory:

tar -cvf etcy.tar /etc 2> /dev/null

• -c create mode
• -v verbose list each file that gets processed
• -f the tar file being delt with

Same, with compression:

tar -czf etcy.tar.gz /etc 2> /dev/null

• -z (gzip) or -j (bzip2) compression

Example compression sizes:

 28M etcy.tar
4.4M etcy.tar.bz2
5.6M etcy.tar.gz

Whats in this tarball? -t or --list has answers:

tar -tf etcy.tar
etc/
etc/idmapd.conf
etc/openldap/
etc/openldap/ldap.conf
...

Unpack the entire tar:

tar -xf etcy.tar

Unpack specific things:

tar -xf etcy.tar etc/openldap/ldap.conf

Results in:

.
├── etc
│   └── openldap
│       └── ldap.conf
├── etcy.tar
├── etcy.tar.bz2
└── etcy.tar.gz

rsync

The smart file copier; only transfers blocks that are needed, on the fly compression.

In its simplist form, copy a file locally:

rsync etcy.tar /mnt/sdd5/backups/

Some optional switches:

• -v verbose
• -h human friendly (29,242,419 bytes becomes 29.24M)
• --progress show progress during transfer
• -z compression

Put a file onto a remote server:

rsync etcy.tar iris.local:/home/ben/

• -a archive mode for presevation of symlinks, devices, attributes, permissions.
• -u update mode, skips files that are newer on the target
• -b backup
• -e remote shell to use (e.g. -e ssh)
• --delete remove files/dirs in the destination, that arent in the source

Complete example:

rsync --progress -avhe ssh Fedora* schnerg@192.168.1.111:/raid1/sdd1/Software/Big/Linux/Fedora

Only get diffs, do multiple times for dodgy downloads:

rsync -P rsync://rsync.server.com/path/to/file file

Restrict flow rate:

rsync --bwlimit=1m fromfile tofile

Mirror web site (with compression and encryption):

rsync -az -e ssh --delete ~/public_html/ remote.com:'~/public_html'

Synchronise current dir with remote dir:

rsync -auz -e ssh remote:/dir/ . && rsync -auz -e ssh . remote:/dir/

sed

For more, see my post on [sed]({% post_url 2015-09-15-sed %}).

awk

Given a longform (-l) list of files and directories, filter only those starting with “pki” and ending with “.jar”, outputting only the shortname.

$ ls -l | awk 'match($10, /^pki.*\.jar$/) { print $10 }'
pki_jcsi_2.1.2.jar
pki_jcsi_base_2.1.2.jar
pki_jcsi_provider_2.1.2.jar
pki_jcsi_smime_2.1.6.jar

For a deeper survey of awk see my [post]({% post_url 2016-01-17-awk %}).

ssh (Secure Shell)

ssh $USER@$HOST command | Run command on $HOST as $USER ssh -f -Y $USER@$HOSTNAME xeyes| Run GUI command on $HOSTNAME as $USERscp -p -r $USER@$HOST: file dir/| Copy with permissions to $USER’s home directory on $HOSTscp -c arcfour $USER@$LANHOST: bigfile| Use faster crypto for local LANssh -g -L 8080:localhost:80 root@$HOST | Forward connections to $HOSTNAME:8080 out to $HOST:80 ssh -R 1434:imap:143 root@$HOST| Forward connections from $HOST:1434 in to imap:143ssh-copy-id $USER@$HOST | Install public key for $USER@$HOST for password-less log in

wget

Download local browsable verison of a webpage:

(cd dir/ && wget -nd -pHEKk http://www.bencode.net)

Continue downloading a partial download:

wget -c http://www.site.org/large.iso

Download specific types (e.g. png) of files:

wget -r -nd -np -l1 -A '*.png' http://www.slashgot.org

Pipe and process output:

wget -q -O- http://www.slashdot.org | grep 'a href' | head

Update a local copy of a site:

wget --mirror http://www.slashdot.org

Schedule a download in the future:

echo 'wget http://www.lobste.rs' | at 21:00

BFL of Common Programs

An overview of common programs that generally exist on nix based systems.

Resources

• The Linux Command Line
• Pádraig Brady’s Reference Guide