Linux Utilities


A survey of the standard and high quality programs that feature in most Unix based distributions, with the GNU variants being my favourite. The bash shell is a great way of interfacing and orchestrating these beautifully crafted programs. As a starting point, I’ve listed out each utility offered by the amazing GNU Core Utilities and util-linux umbrella projects; considered defacto standard on most distributions.

Contents

Command Description
addpart tell the kernel about the existence of a partition
agetty alternative Linux getty
arch print machine hardware name
awk pattern scanning and processing language
base32 base32 encode/decode data and print to standard output
base64 base64 encode/decode data and print to standard output
basename strip directory and suffix from filenames
blkdiscard discard sectors on a device
blkid locate/print block device attributes
blockdev call block device ioctls from the command line
cal display a calendar
cat concatenate files and print on the standard output
cfdisk display or manipulate a disk partition table
chcon change file SELinux security context
chcpu configure CPUs
chfn change your finger information
chgrp change group ownership
chmod change file mode bits
chown change file owner and group
chroot run command or interactive shell with special root directory
chrt manipulate the real-time attributes of a process
chsh change your login shell
cksum checksum and count the bytes in a file
col filter reverse line feeds from input
colcrt filter nroff output for CRT previewing
colrm remove columns from a file
column columnate lists
comm compare two sorted files line by line
cp copy files and directories
csplit split a file into sections determined by context lines
ctrlaltdel set the function of the Ctrl-Alt-Del combination
cut remove sections from each line of files
date print or set the system date and time
dd convert and copy a file
delpart tell the kernel to forget about a partition
df report file system disk space usage
dir list directory contents
dircolors color setup for ls
dirname strip last component from file name
dmesg print or control the kernel ring buffer
du estimate file space usage
echo display a line of text
eject eject removable media
env run a program in a modified environment
expand convert tabs to spaces
expr evaluate expressions
factor factor numbers
fallocate preallocate or deallocate space to a file
false do nothing, unsuccessfully
fdformat low-level format a floppy disk
fdisk manipulate disk partition table
findfs find a filesystem by label or UUID
findmnt find a filesystem
flock manage locks from shell scripts
fmt simple optimal text formatter
fold wrap each input line to fit in specified width
fsck check and repair a Linux filesystem
fsck.cramfs fsck compressed ROM file system
fsck.minix check consistency of Minix filesystem
fsfreeze suspend access to a filesystem (Ext3/4, ReiserFS, JFS, XFS)
fstrim discard unused blocks on a mounted filesystem
getopt parse command options (enhanced)
grep print lines matching a pattern
groups print the groups a user is in
head output the first part of files
hexdump display file contents in hexadecimal, decimal, octal, or ascii
hostid print the numeric identifier for the current host
hostname show or set the system’s host name
hwclock read or set the hardware clock (RTC)
id print real and effective user and group IDs
install copy files and set attributes
ionice set or get process I/O scheduling class and priority
ipcmk make various IPC resources
ipcrm remove certain IPC resources
ipcs show information on IPC facilities
isosize output the length of an iso9660 filesystem
join join lines of two files on a common field
kill terminate a process
kill terminate a process
last show a listing of last logged in users
ldattach attach a line discipline to a serial line
line TODO
link call the link function to create a link to a file
ln make links between files
logger enter messages into the system log
login begin session on the system
logname print user’s login name
look display lines beginning with a given string
losetup set up and control loop devices
ls list directory contents
lsblk list block devices
lscpu display information about the CPU architecture
lslocks list local system locks
lslogins display information about known users in the system
lsof list open files
mcookie generate magic cookies for xauth
md5sum compute and check MD5 message digest
mesg display (or do not display) messages from other users
mkdir make directories
mkfifo make FIFOs (named pipes)
mkfs build a Linux filesystem
mkfs.bfs make an SCO bfs filesystem
mkfs.cramfs make compressed ROM file system
mkfs.minix make a Minix filesystem
mknod make block or character special files
mkswap set up a Linux swap area
mktemp create a temporary file or directory
more file perusal filter for crt viewing
mount mount a filesystem
mountpoint see if a directory or file is a mountpoint
mv move (rename) files
namei follow a pathname until a terminal point is found
newgrp log in to a new group
nice run a program with modified scheduling priority
nl number lines of files
nohup run a command immune to hangups, with output to a non-tty
nologin politely refuse a login
nproc print the number of processing units available
nsenter run program with namespaces of other processes
numfmt Convert numbers from/to human-readable strings
od dump files in octal and other formats
partx tell the kernel about the presence and numbering of on-disk partitions
paste merge lines of files
pathchk check whether file names are valid or portable
pg is a pager, allows viewing one page at a time
pivot_root change the root filesystem
pr convert text files for printing
printenv print all or part of environment
printf format and print data
prlimit get and set process resource limits
ps report a snapshot of the current processes
ptx produce a permuted index of file contents
pwd print name of current/working directory
raw bind a Linux raw character device
readlink print resolved symbolic links or canonical file names
readprofile read kernel profiling information
realpath print the resolved path
rename rename files
renice alter priority of running processes
reset terminal initialization
resizepart tell the kernel about the new size of a partition
rev reverse lines characterwise
rm remove files or directories
rmdir remove empty directories
runcon run command with specified SELinux security context
runuser run a command with substitute user and group ID
script make typescript of terminal session
scriptreplay play back typescripts, using timing information
sed stream editor for filtering and transforming text
seq print a sequence of numbers
setarch change reported architecture in new program environment and set personality flags
setpriv run a program with different Linux privilege settings
setsid run a program in a new session
setterm set terminal attributes
sfdisk display or manipulate a disk partition table
sha1sum compute and check SHA1 message digest
sha2 message digests
shred overwrite a file to hide its contents, and optionally delete it
shuf generate random permutations
sleep delay for a specified amount of time
sort sort lines of text files
split split a file into pieces
stat display file or file system status
stdbuf Run COMMAND, with modified buffering operations for its standard streams.
stty change and print terminal line settings
su run a command with substitute user and group ID
sulogin single-user login
sum checksum and count the blocks in a file
swaplabel print or change the label or UUID of a swap area
swapoff enable/disable devices and files for paging and swapping
swapon enable/disable devices and files for paging and swapping
switch_root switch to another filesystem as the root of the mount tree
sync Synchronize cached writes to persistent storage
tac concatenate and print files in reverse
tail output the last part of files
tailf follow the growth of a log file
taskset set or retrieve a process’s CPU affinity
tcpdump dump traffic on a network
tee read from standard input and write to standard output and files
test check file types and compare values
timeout run a command with a time limit
touch change file timestamps
tr translate or delete characters
true do nothing, successfully
truncate shrink or extend the size of a file to the specified size
tsort perform topological sort
tty print the file name of the terminal connected to standard input
tunelp set various parameters for the lp (printer) device
ul do underlining
umount unmount file systems
uname print system information
unexpand convert spaces to tabs
uniq report or omit repeated lines
unlink call the unlink function to remove the specified file
unshare run program with some namespaces unshared from parent
uptime Tell how long the system has been running.
users print the user names of users currently logged in to the current host
utmpdump dump UTMP and WTMP files in raw format
uuidgen create a new UUID value
vdir list directory contents
vipw edit the password, group, shadow-password or shadow-group file
w Show who is logged on and what they are doing.
wall write a message to all users
wc print newline, word, and byte counts for each file
wdctl show hardware watchdog status
whereis locate the binary, source, and manual page files for a command
who show who is logged on
whoami print effective userid
wipefs wipe a signature from a device
write write to another user
yes output a string repeatedly until killed

Documentation

Manual Pages

The infamous manual (man) page documentation system. Man pages are organised by the following sections:

Section Name Description
1 User commands (Programs) Commands that can be executed by the user from within a shell.
2 System calls Functions which wrap operations performed by the kernel.
3 Library calls Library functions excluding the system call wrappers (Most of the libc functions).
4 Special files (devices) Files found in /dev which allow to access to devices through the kernel.
5 File formats and configuration files Various human-readable file formats and configuration files.
6 Games Games and funny little programs available on the system.
7 Overview, conventions, and miscellaneous Various topics, conventions and protocols, character set standards, the standard filesystem layout, and miscellaneous other things.
8 System management commands Commands like mount(8), many of which only root can execute.

An explicit section can be requested. For the man page relating to the file format of /etc/passwd

man 5 passwd

The -k switch is great for searching across man’s treasure chest of documentation. For example, say you want to set the system time, but have no idea what program to use to achieve this. Use the -k switch to scan documentation for time.

$ man -k time
ac (1)               - print statistics about users connect time
adjtime (3)          - correct the time to synchronize the system clock
adjtimex (2)         - tune kernel clock
after (n)            - Execute a command after a time delay
aio_suspend (3)      - wait for asynchronous I/O operation or timeout
asctime (3)          - transform date and time to broken-down time or ASCII
asctime (3p)         - convert date and time to a string

The lions share of search results seems to come from section 2 and 3 (C kernel and library calls). Focusing on the task at hand, administering the system time, lets filter results to man sections 1 (user commands) and 8 (system management commands).

$ man -k time | grep -Pe '.*\([1,8]\).*'
ac (1)               - print statistics about users connect time
booleans (8)         - Policy booleans enable runtime customization of SELinux policy
ccrewrite (1)        - Rewrite CLR assemblies for runtime code contract verification.
chrt (1)             - manipulate the real-time attributes of a process
date (1)             - print or set the system date and time
dnssec-settime (8)   - Set the key timing metadata for a DNSSEC key
jack_showtime (1)    - The JACK Audio Connection Kit example client

The date program looks perfect.

Appropriate Commands

Basically an equivalent to the man -k switch for searching.

$ apropos clock
adjtime (3)          - correct the time to synchronize the system clock
adjtimex (2)         - tune kernel clock
alarm (2)            - set an alarm clock for delivery of a signal
clock (3)            - determine processor time

whatis

For a very brief overview of a man page matching a keyword.

$ whatis vim
vim (1)              - Vi IMproved, a programmers text editor

GNU Info Entry

A purpose built documentation system from GNU, info features hyperlinks (prefixed with *), aimed at dealing with larger documentation sets than man.

info goes against the grain in terms of keyboard navigation. Its odd. Page up, page down, enter to follow a link, and l to go back,

Some keys for driving info:

  • n next node
  • p previous node
  • u parent node
  • t top node
  • home end pgup pgdn scroll content
  • l go back
  • q quit
  • H keyboard shortcuts cheatsheet

Searching info:

$ info --apropos=tee
"(coreutils)tee invocation" -- tee
"(libc)Control Functions" -- feupdateenv
"(gawk)Tee Program" -- 'tee' utility

And then info gawk tee for example to pull up the third result.

/usr/share/doc Documentation

A gold mine of documents and sample configuration files. Usually for distributions that are not considered core, and don’t offer man or info pages. ### RPM bundled documentation

$ rpm -qd tmux
/usr/share/doc/tmux/CHANGES
/usr/share/doc/tmux/FAQ
/usr/share/doc/tmux/TODO
/usr/share/man/man1/tmux.1.gz

Examples

grep

grep prints lines that contain a match for a pattern.

Useful modes:

  • -r or -R for recursive
  • -n show line number
  • -w match the whole word
  • -v invert match (i.e. blacklist)
  • -l just give the file name of matching files
  • -i case insensative
  • -P Perl style regular expressions

Recursively search all files from the current directory, containing Romero, including the line number where they are found:

$ grep -rnw . -e 'Romero'
Binary file ./datatsudio/.metadata/.plugins/seg0/c530.dat matches
./files/diff/heros_new:4:Romero,John,671028
./files/heros:7:Romero,John,671028

The --include and --exclude are very useful for filtering target files, and the amount of work grep needs to do. Exclude *.dat binary files from the above example:

$ grep -rnw . -e 'Romero' --exclude '*.dat'
./grep/diff/heros_new:4:Romero,John,671028
./grep/heros:7:Romero,John,671028

Perl patterns:

$ echo "2016-10-13" | grep -Pe '\d{4}-\d{2}-\d{2}'
2016-10-13

Color highlight numeric 0 to 5:

$ echo "2016-10-13" | grep --color '[0-5]'
2016-10-13

Overall total of how many times an expression matches:

$ grep -rnwo . --include \*.bash --include \*.sh -e 'BASH_REMATCH' | wc -l
12

diff

patch

shuf

cut

Removes portions from each line of input. By default will use standard input, when no FILE specified, or when FILE is -.

Select the first field for the colon delimitered file /etc/passwd.

$ cut -d : -f 1 /etc/passwd
LocalService
NetworkService
Guest
SYSTEM

Hack just the date portion (chars 1-10) off the front of logs, and show the unique dates:

$ cut -c1-10 dircdds.log | grep -Pe '\d{4}-\d{2}-\d{2}' | sort -h | uniq
2016-03-21
2016-03-22
2016-03-24
2016-03-29
2016-04-21

nl

tail

uniq

sort

By default will sort in dictionary order.

$ cut -d : -f 3 /etc/passwd | sort
0
1
1000
1001
107
11
113
12

Useful sort modes:

  • -h human numeric (e.g. 2K 3G)
  • -n numeric
  • -r reverse
  • -R random
  • -u unique

tr

For translating (e.g. uppercasing, stripping, truncating, etc) text.

Convert lower case characters to upper.

$ echo "Linus Torvalds" | tr [:lower:] [:upper:]
LINUS TORVALDS

Make all lower case characters o:

$ echo "Linus Torvalds" | tr [:lower:] o
Loooo Tooooooo

Replace the range of characters a to o, with @:

$ echo "Linus Torvalds" | tr a-o @
L@@us T@rv@@@s

wc

Count aggregates of the contents of a file.

By default will show counts of lines, words and bytes.

$ wc pthreads.make 
7  29 252 pthreads.make

Useful counts:

  • -l, --lines newlines
  • -w, --words words
  • -c, --bytes bytes
  • -m, --chars characters

Just show the number of lines:

$ wc -l pthreads.make 
7  pthreads.make

Pipe support just works:

$ cat 2016-05-01-bash.markdown | wc -l
1100

tar

The rock solid archiving tool that you can always lean on.

Create an archive of all of the /etc directory:

tar -cvf etcy.tar /etc 2> /dev/null
  • -c create mode
  • -v verbose list each file that gets processed
  • -f the tar file being delt with

Same, with compression:

tar -czf etcy.tar.gz /etc 2> /dev/null
  • -z (gzip) or -j (bzip2) compression

Example compression sizes:

 28M etcy.tar
4.4M etcy.tar.bz2
5.6M etcy.tar.gz

Whats in this tarball? -t or --list has answers:

tar -tf etcy.tar
etc/
etc/idmapd.conf
etc/openldap/
etc/openldap/ldap.conf
...

Unpack the entire tar:

tar -xf etcy.tar

Unpack specific things:

tar -xf etcy.tar etc/openldap/ldap.conf

Results in:

.
├── etc
│   └── openldap
│       └── ldap.conf
├── etcy.tar
├── etcy.tar.bz2
└── etcy.tar.gz

rsync

The smart file copier; only transfers blocks that are needed, on the fly compression.

In its simplist form, copy a file locally:

rsync etcy.tar /mnt/sdd5/backups/

Some optional switches:

  • -v verbose
  • -h human friendly (29,242,419 bytes becomes 29.24M)
  • --progress show progress during transfer
  • -z compression

Put a file onto a remote server:

rsync etcy.tar iris.local:/home/ben/
  • -a archive mode for presevation of symlinks, devices, attributes, permissions.
  • -u update mode, skips files that are newer on the target
  • -b backup
  • -e remote shell to use (e.g. -e ssh)
  • --delete remove files/dirs in the destination, that arent in the source

sed

For more, see my post on sed.

awk

Given a longform (-l) list of files and directories, filter only those starting with “pki” and ending with “.jar”, outputting only the shortname.

$ ls -l | awk 'match($10, /^pki.*\.jar$/) { print $10 }'
pki_jcsi_2.1.2.jar
pki_jcsi_base_2.1.2.jar
pki_jcsi_provider_2.1.2.jar
pki_jcsi_smime_2.1.6.jar

For a deeper survey of awk see my post.