coreutils


A survey of the standard and high quality programs that feature in most Unix based distributions, with the GNU variants being my favourite. The bash shell is a great way of interfacing and orchestrating these beautifully crafted programs.

Bash is a Unix shell written by Brian Fox in 1989 for the GNU Project as a free replacement for the Bourne shell. To this day, Bash remains one of the most powerful and ubiquitous scripting tools on the planet.

Contents

Command | Category | Description — | — | —

arch System Context arch (1) - print machine hardware name (same as uname -m)
awk File Output pattern scanning and processing language
base32 File Output base32 (1) - base32 encode/decode data and print to standard output
base64 File Output base64 (1) - base64 encode/decode data and print to standard output
basename File Name Manipulation basename (1) - strip directory and suffix from filenames
cat File Output cat (1p) - concatenate and print files
chcon SELinux chcon (1) - change file SELinux security context
chgrp File Attributes chgrp (1p) - change the file group ownership
chmod File Attributes chmod (3p) - change mode of a file relative to directory file descriptor
chown File Attributes chown (3p) - change owner and group of a file relative to directory file descriptor
chroot Modified Invocation chroot (2) - change root directory
cksum File Summary cksum (1p) - write file checksums and sizes
comm Sort Operations comm (1p) - select or reject lines common to two files
cp Basic cp (1p) - copy files
csplit File (partial) Output csplit (1p) - split files based on context
cut Fields Operations cut (1p) - cut out selected fields of each line of a file
date System Context date (1p) - write the date and time
dd Basic dd (1p) - convert and copy a file
df Disk df (1p) - report free disk space
dir Directory Listing dir (1) - list directory contents
dircolors Directory Listing dircolors (1) - color setup for ls
dirname File Name Manipulation dirname (3p) - report the parent directory name of a file pathname
du Disk du (1p) - estimate file space usage
echo Printing Text echo (1p) - write arguments to standard output
env Modified Invocation env (1p) - set the environment for command invocation
expand Characters Operations expand (1p) - convert tabs to spaces
expr Conditions expr (1p) - evaluate arguments as an expression
factor Numeric Operations factor (1) - factor numbers
false Conditions false (1p) - return false value
fmt File Formatting fmt (1) - simple optimal text formatter
fold File Formatting fold (1p) - filter for folding lines
grep File Output print lines matching a pattern
groups User Information groups (1) - print the groups a user is in
head File (partial) Output head (1p) - copy the first part of files
hostid System Context hostid (1) - print the numeric identifier for the current host
hostname System Context hostname (7) - hostname resolution description
id User Information id (1p) - return user identity
install Basic install (1) - copy files and set attributes
join Fields Operations join (1p) - relational database operator
kill Process Control kill (3p) - send a signal to a process or a group of processes
link Special File Types link (3p) - link one file to another file relative to two directory file descriptors
ln Special File Types ln (1p) - link files
logname User Information logname (1p) - return the user’s login name
ls Directory Listing ls (1p) - list directory contents
lsof System Context list open files
md5sum File Summary md5sum (1) - compute and check MD5 message digest
mkdir Special File Types mkdir (3p) - make a directory relative to directory file descriptor
mkfifo Special File Types mkfifo (3p) - make a FIFO special file relative to directory file descriptor
mknod Special File Types mknod (3p) - make directory, special file, or regular file
mktemp File Name Manipulation mktemp (3) - make a unique temporary filename
mv Basic mv (1p) - move files
nice Modified Invocation nice (3p) - change the nice value of a process
nl File Output nl (1p) - line numbering filter
nohup Modified Invocation nohup (1p) - invoke a utility immune to hangups
nproc System Context nproc (1) - print the number of processing units available
numfmt Numeric Operations numfmt (1) - Convert numbers from/to human-readable strings
od File Output od (1p) - dump files in various formats
paste Fields Operations paste (1p) - merge corresponding or subsequent lines of files
pathchk File Name Manipulation pathchk (1p) - check pathnames
pr File Formatting pr (1p) - print files
printenv Working Context printenv (1) - print all or part of environment
printf Printing Text printf (3p) - print formatted output
ps System Context report a snapshot of the current processes
ptx Sort Operations ptx (1) - produce a permuted index of file contents
pwd Working Context pwd (1) - print name of current/working directory
readlink Special File Types readlink (1) - print resolved symbolic links or canonical file names
realpath File Name Manipulation realpath (1) - print the resolved path
rm Basic rm (1) - remove files or directories
rmdir Special File Types rmdir (1) - remove empty directories
runcon SELinux runcon (1) - run command with specified SELinux security context
sed File Output stream editor for filtering and transforming text
seq Numeric Operations seq (1) - print a sequence of numbers
sha1sum File Summary sha1sum (1) - compute and check SHA1 message digest
sha2 File Summary  
shred Basic shred (1) - overwrite a file to hide its contents, and optionally delete it
shuf Sort Operations shuf (1) - generate random permutations
sleep Delaying sleep (1) - delay for a specified amount of time
sort Sort Operations sort (1) - sort lines of text files
split File (partial) Output split (1) - split a file into pieces
stat Disk stat (1) - display file or file system status
stdbuf Modified Invocation stdbuf (1) - Run COMMAND, with modified buffering operations for its standard streams.
stty Working Context stty (1) - change and print terminal line settings
sum File Summary sum (1) - checksum and count the blocks in a file
sync Disk sync (1) - Synchronize cached writes to persistent storage
tac File Output tac (1) - concatenate and print files in reverse
tail File (partial) Output tail (1) - output the last part of files
tcpdump Network dump traffic on a network
tee Redirection tee (1) - read from standard input and write to standard output and files
test Conditions test (1) - check file types and compare values
timeout Modified Invocation timeout (1) - run a command with a time limit
touch File Attributes touch (1) - change file timestamps
tr Characters Operations tr (1) - translate or delete characters
true Conditions true (1) - do nothing, successfully
truncate Disk truncate (1) - shrink or extend the size of a file to the specified size
tsort Sort Operations tsort (1) - perform topological sort
tty Working Context tty (1) - print the file name of the terminal connected to standard input
uname System Context uname (1) - print system information
unexpand Characters Operations unexpand (1) - convert spaces to tabs
uniq Sort Operations uniq (1) - report or omit repeated lines
unlink Special File Types unlink (1) - call the unlink function to remove the specified file
uptime System Context uptime (1) - Tell how long the system has been running.
users User Information users (1) - print the user names of users currently logged in to the current host
vdir Directory Listing vdir (1) - list directory contents
wc File Summary wc (1) - print newline, word, and byte counts for each file
who User Information who (1) - show who is logged on
whoami User Information whoami (1) - print effective userid
yes Printing Text yes (1) - output a string repeatedly until killed

TODO: Need to replace descriptions above with the following section 1s cat (1) - concatenate files and print on the standard output chgrp (1) - change group ownership chmod (1) - change file mode bits chown (1) - change file owner and group chroot (1) - run command or interactive shell with special root directory cksum (1) - checksum and count the bytes in a file comm (1) - compare two sorted files line by line cp (1) - copy files and directories csplit (1) - split a file into sections determined by context lines cut (1) - remove sections from each line of files date (1) - print or set the system date and time dd (1) - convert and copy a file df (1) - report file system disk space usage dirname (1) - strip last component from file name du (1) - estimate file space usage echo (1) - display a line of text env (1) - run a program in a modified environment expand (1) - convert tabs to spaces expr (1) - evaluate expressions false (1) - do nothing, unsuccessfully fold (1) - wrap each input line to fit in specified width HEAD (1) - Simple command line user agent head (1) - output the first part of files hostname (1) - show or set the system’s host name id (1) - print real and effective user and group IDs join (1) - join lines of two files on a common field kill (1) - terminate a process link (1) - call the link function to create a link to a file ln (1) - make links between files logname (1) - print user’s login name ls (1) - list directory contents mkdir (1) - make directories mkfifo (1) - make FIFOs (named pipes) mknod (1) - make block or character special files mktemp (1) - create a temporary file or directory mv (1) - move (rename) files nice (1) - run a program with modified scheduling priority nl (1) - number lines of files nohup (1) - run a command immune to hangups, with output to a non-tty od (1) - dump files in octal and other formats paste (1) - merge lines of files pathchk (1) - check whether file names are valid or portable pr (1) - convert text files for printing printf (1) - format and print data

Help

There so many beautiful elegant little programs waiting to help you, it can be overwhelming knowing where to start, let alone recalling subtle options and quirks they have. Luckily there is a tomb of quality documentation at your finger tips. If there is only one thing you can remember, it should be this, finding help.

Manual Pages

The infamous manual (man) page, documentation system. Man pages are organised by the following convention:

Section Name Description
1 User commands (Programs) Commands that can be executed by the user from within a shell.
2 System calls Functions which wrap operations performed by the kernel.
3 Library calls Library functions excluding the system call wrappers (Most of the libc functions).
4 Special files (devices) Files found in /dev which allow to access to devices through the kernel.
5 File formats and configuration files Various human-readable file formats and configuration files.
6 Games Games and funny little programs available on the system.
7 Overview, conventions, and miscellaneous Various topics, conventions and protocols, character set standards, the standard filesystem layout, and miscellaneous other things.
8 System management commands Commands like mount(8), many of which only root can execute.

An explicit section can be requested. For the man page relating to the file format of /etc/passwd

man 5 passwd

The -k switch is great for searching across man’s treasure chest of documentation. For example, say you want to set the system time, but have no idea what program to use to achieve this. Use the -k switch to scan documentation for time.

$ man -k time
RESET (7)            - restore the value of a run-time parameter to the default value
SET (7)              - change a run-time parameter
ac (1)               - print statistics about users connect time
adjtime (3)          - correct the time to synchronize the system clock
adjtimex (2)         - tune kernel clock
after (n)            - Execute a command after a time delay
aio_suspend (3)      - wait for asynchronous I/O operation or timeout
asctime (3)          - transform date and time to broken-down time or ASCII
asctime (3p)         - convert date and time to a string
...

The lions share of search results seems to come from section 2 and 3 (C kernel and library calls). Focusing on the task at hand, administering the system time, lets filter results to man sections 1 (user commands) and 8 (system management commands).

$ man -k time | grep -Pe '.*\([1,8]\).*'
ac (1)               - print statistics about users connect time
booleans (8)         - Policy booleans enable runtime customization of SELinux policy
ccrewrite (1)        - Rewrite CLR assemblies for runtime code contract verification.
chrt (1)             - manipulate the real-time attributes of a process
date (1)             - print or set the system date and time
dnssec-settime (8)   - Set the key timing metadata for a DNSSEC key
jack_showtime (1)    - The JACK Audio Connection Kit example client
kbdrate (8)          - reset the keyboard repeat rate and delay time

...

The date program looks perfect.

Appropriate Commands

whatis

For a very brief overview of a man page matching a keyword.

$ whatis vim
vim (1)              - Vi IMproved, a programmers text editor

GNU Info Entry

A purpose built documentation system from GNU, info features hyperlinks (prefixed with *), aimed at dealing with larger documentation sets than man.

info goes against the grain in terms of keyboard navigation. Its odd. Page up, page down, enter to follow a link, and l to go back,

/usr/share/doc Documentation

grep

grep prints lines that contain a match for a pattern.

Useful modes:

  • -r or -R for recursive
  • -n show line number
  • -w match the whole word
  • -v invert match (i.e. blacklist)
  • -l just give the file name of matching files
  • -i case insensative
  • -P Perl style regular expressions

Recursively search all files from the current directory, containing Romero, including the line number where they are found:

$ grep -rnw . -e 'Romero'
Binary file ./datatsudio/.metadata/.plugins/seg0/c530.dat matches
./files/diff/heros_new:4:Romero,John,671028
./files/heros:7:Romero,John,671028

The --include and --exclude are very useful for filtering target files, and the amount of work grep needs to do. Exclude *.dat binary files from the above example:

$ grep -rnw . -e 'Romero' --exclude '*.dat'
./grep/diff/heros_new:4:Romero,John,671028
./grep/heros:7:Romero,John,671028

Perl patterns:

$ echo "2016-10-13" | grep -Pe '\d{4}-\d{2}-\d{2}'
2016-10-13

Color highlight numeric 0 to 5:

$ echo "2016-10-13" | grep --color '[0-5]'
2016-10-13

Overall total of how many times an expression matches:

$ grep -rnwo . --include \*.bash --include \*.sh -e 'BASH_REMATCH' | wc -l
12

diff

patch

shuf

cut

Removes portions from each line of input. By default will use standard input, when no FILE specified, or when FILE is -.

Select the first field for the colon delimitered file /etc/passwd.

$ cut -d : -f 1 /etc/passwd
LocalService
NetworkService
Guest
SYSTEM

Hack just the date portion (chars 1-10) off the front of logs, and show the unique dates:

$ cut -c1-10 dircdds.log | grep -Pe '\d{4}-\d{2}-\d{2}' | sort -h | uniq
2016-03-21
2016-03-22
2016-03-24
2016-03-29
2016-04-21

nl

tail

uniq

sort

By default will sort in dictionary order.

$ cut -d : -f 3 /etc/passwd | sort
0
1
1000
1001
107
11
113
12

Useful sort modes:

  • -h human numeric (e.g. 2K 3G)
  • -n numeric
  • -r reverse
  • -R random
  • -u unique

tr

For translating (e.g. uppercasing, stripping, truncating, etc) text.

Convert lower case characters to upper.

$ echo "Linus Torvalds" | tr [:lower:] [:upper:]
LINUS TORVALDS

Make all lower case characters o:

$ echo "Linus Torvalds" | tr [:lower:] o
Loooo Tooooooo

Replace the range of characters a to o, with @:

$ echo "Linus Torvalds" | tr a-o @
L@@us T@rv@@@s

wc

Count aggregates of the contents of a file.

By default will show counts of lines, words and bytes.

$ wc pthreads.make 
7  29 252 pthreads.make

Useful counts:

  • -l, --lines newlines
  • -w, --words words
  • -c, --bytes bytes
  • -m, --chars characters

Just show the number of lines:

$ wc -l pthreads.make 
7  pthreads.make

Pipe support just works:

$ cat 2016-05-01-bash.markdown | wc -l
1100

tar

The rock solid archiving tool that you can always lean on.

Create an archive of all of the /etc directory:

tar -cvf etcy.tar /etc 2> /dev/null
  • -c create mode
  • -v verbose list each file that gets processed
  • -f the tar file being delt with

Same, with compression:

tar -czf etcy.tar.gz /etc 2> /dev/null
  • -z (gzip) or -j (bzip2) compression

Example compression sizes:

 28M etcy.tar
4.4M etcy.tar.bz2
5.6M etcy.tar.gz

Whats in this tarball? -t or --list has answers:

tar -tf etcy.tar
etc/
etc/idmapd.conf
etc/openldap/
etc/openldap/ldap.conf
...

Unpack the entire tar:

tar -xf etcy.tar

Unpack specific things:

tar -xf etcy.tar etc/openldap/ldap.conf

Results in:

.
├── etc
│   └── openldap
│       └── ldap.conf
├── etcy.tar
├── etcy.tar.bz2
└── etcy.tar.gz

rsync

The smart file copier; only transfers blocks that are needed, on the fly compression.

In its simplist form, copy a file locally:

rsync etcy.tar /mnt/sdd5/backups/

Some optional switches:

  • -v verbose
  • -h human friendly (29,242,419 bytes becomes 29.24M)
  • --progress show progress during transfer
  • -z compression

Put a file onto a remote server:

rsync etcy.tar iris.local:/home/ben/
  • -a archive mode for presevation of symlinks, devices, attributes, permissions.
  • -u update mode, skips files that are newer on the target
  • -b backup
  • -e remote shell to use (e.g. -e ssh)
  • --delete remove files/dirs in the destination, that arent in the source

sed

For more, see my post on sed.

awk

Given a longform (-l) list of files and directories, filter only those starting with “pki” and ending with “.jar”, outputting only the shortname.

$ ls -l | awk 'match($10, /^pki.*\.jar$/) { print $10 }'
pki_jcsi_2.1.2.jar
pki_jcsi_base_2.1.2.jar
pki_jcsi_provider_2.1.2.jar
pki_jcsi_smime_2.1.6.jar

For a deeper survey of awk see my post.