My preparation notes for the RHCSA 8 exam.

Essential Tools⌗

Documentation⌗

The ability to understand a program using local documentation resources; man, info, /usr/share/doc, within the RPM package.

man⌗

To browse man pages for a keyword use -k, e.g. scan documentation for all things relating to password:

man -k password

Alternatively:

mandb
apropos passwd

Specific sections with man, refer to different topics, e.g. section 5 is about config files, so man 5 passwd would bring up the documentation on /etc/passwd.

1 = user commands 5 = configuration files 7 = broad topics such as background 8 = sys admin

man -k user | grep 8 | grep create

/usr/share/doc⌗

A gold mine of documents and sample configuration files. Usually for distributions that are not considered core, and don’t offer man or info pages.

RPM bundled documentation⌗ 

$ rpm -qd tmux
/usr/share/doc/tmux/CHANGES
/usr/share/doc/tmux/FAQ
/usr/share/doc/tmux/TODO
/usr/share/man/man1/tmux.1.gz

General Searching Techniques⌗

General search engine:

$ updatedb
$ locate passwd

Search path for passwd:

$ which passwd
/usr/bin/passwd

Search one-line man page descriptions:

$ whatis passwd
passwd (1)           - update user's authentication tokens
sslpasswd (1ssl)     - compute password hashes
passwd (5)           - password file

Find binaries and man pages for ls:

$ whereis -bm ls
ls: /usr/bin/ls /usr/share/man/man1/ls.1.gz /usr/share/man/man1/ls.1p.gz

Shell history⌗

  • history dump history, by default the last 1000 commands
  • ctrl+r to search backwards through history for pattern
  • history -c clear history (in-memory only)
  • history -w write history
  • !32 run history event 32 (again)

Globbing⌗

aka using wildcards see man 7 glob

  • ls host* zero or more chars
  • ls ?ost any single char
  • ls [hm]ost groups of chars
  • ls [!hm]ost negated groups of chars
  • ls [0-9][0-9]script multiple groups of restricted chars

I/O Redirection and Pipes⌗

  • < stdin from a file or another programs stdout
  • > stdout to new file (overwrite if exists)
  • >> stdout to file (appending if exists)
  • 2> stderr redirection
  • 2>&1 stderr to stdout (useful for piping stderr, as pipes only work with stdout)
  • | pipe stdout from one program to stdin of another (pipes only support stdout to stdin communication, i.e. not stderr)

Essential File Management⌗

Linux file system layout⌗

See man hier and man file-hierarchy

Big hitters:

  • Boot partition: /boot/ and /efi/
  • System configuration: /etc/
  • Scripts and binaries: /bin/, /sbin/, /usr/sbin/ now all link back to /usr/bin/
  • Shared libraries: /lib/, /lib64/ link to /usr/lib/ and /usr/lib64/ respectively
  • Virtual kernel file system: /proc/ such as /proc/meminfo
  • Persistent variable data: /var/ such as /var/cache/, /var/log/, /var/tmp

Finding Files⌗

locate⌗ 

$ updatedb
$ locate passwd

find⌗

Basic examples:

$ find / -size +100M -exec ls -l {} \;
$ find /etc -name motd  #named motd
$ find /etc -user schnerg  #owned by user shnerg
$ find / -mtime 3  #modified in last 3 days
$ find / -mtime +3  #not within the last 3 days

$ id ben
uid=1000(ben) gid=1000(ben) groups=1000(ben),1004(finance)
$ find / -uid 1000

$ find / -user ben -type f  #filter by files
$ find / -user ben -type f -exec cp {} /home/mary \; #execute a shell command against each result file `{}`

Archiving and compression with tar⌗

Creating archives:

tar cvf foo.tar directory1 file1 file2
tar czvf foo.tar.gz directory1 file1 file2 #with gzip
tar cjvf foo.tar.bz directory1 file1 file2 #with bzip

List contents (without extraction):

tar tvf foo.tar

Extract them:

tar xvf foo.tar
tar xzvf foo.tar.gz
tar xjvf foo.tar.bz

Extract from a base directory:

tar xvf foo.tar -C /

Diffencing an archives contents with an existing exploded structure:

$ tar -dzvf dir1-v2.tar.gz
directory1/
directory1/file4
directory1/wookie4
tar: directory1/wookie4: Warning: Cannot stat: No such file or directory
directory1/wookie3
tar: directory1/wookie3: Warning: Cannot stat: No such file or directory
directory1/file3
directory1/imp1
directory1/imp1: Mod time differs
directory1/imp1: Size differs
directory1/imp2
directory1/file1
directory1/file2

Compression:

gzip file1
gzip -d file1
bzip2 file1

Listing compression stats on a compressed file:

$ gzip -l hello1.gz
         compressed        uncompressed  ratio uncompressed_name
                 83                  62   6.5% hello1

Archiving with star.

star -c -f=foo.tar directory1 hello1 hello2
star -cz -f=foo.tar.gz directory1 hello1 hello2 #with compression

Listing

$ star -t -f=foo.tar
directory1/
directory1/file4
directory1/file3
directory1/imp1
directory1/imp2
directory1/file1
directory1/file2
hello1
hello2

Extract a specific (hello1) file from the archive:

star -x -f=foo.tar hello1

Soft links, or symbolic links (symlinks), are simply pointers to other files. Symlinks can span multiple file systems. Permissions on symlinks aren’t real. The underlying permissions of the target file is what gets applied. They can easily be created with ln like so:

ln -s /etc/motd ~/motd

Hard links are links to a specific inode (shown with ls -i) on the file system. Due to this coupling, cannot span different file systems or devices.

$ ls -l
drwxrwxr-x. 2 ben ben 4096 May 13 20:42 directory1
-rw-rw-r--. 1 ben ben    0 May 13 20:53 hello1
lrwxrwxrwx. 1 ben ben    9 May 14 17:31 motd -> /etc/motd

$ ln hello1 hello1-hardlink
$ ls -l
drwxrwxr-x. 2 ben ben 4096 May 13 20:42 directory1
-rw-rw-r--. 2 ben ben    0 May 13 20:53 hello1
-rw-rw-r--. 2 ben ben    0 May 13 20:53 hello1-hardlink
lrwxrwxrwx. 1 ben ben    9 May 14 17:31 motd -> /etc/motd

In the ls long listing output, take note of the 2nd column, which represents the count of references to the same inode, which increases after creating a hard link. Some properties of hard links:

  • Hard links will always report the same metadata such as permission bits, modification timestamps, etc
  • inode reference counts will increase for each hard link.
  • Removal of the target file or hard link will not result in broken links, as they both physically reference the same inode.

Working with text⌗

Regular expressions⌗

See man 7 regex

  • . any single character

  • ? one or more

  • * zero or more

  • cat for concatenation, commonly used to dump contents to stdout

  • tac concatenation in reverse order

  • cut parses fields based on simple delimiter cut -d : -f 1 /etc/passwd cuts the first field in /etc/passwd based on a colon delimiter

  • sort can sort alphabetically or numerically e.g. cut -d : -f 3 /etc/passwd | sort -n

  • head first n lines

  • tail last n lines

  • tr translator e.g. lower to upper casing cut -d : -f 1 /etc/passwd | tr [a-z] [A-Z]

grep⌗

The pinnacle of text processing, handed down by god himself.

grep '^#' /etc/sysconfig/sshd

Noteworthy:

  • always place regex between single quotes to avoid ambiguity of globbing
  • use -e to specify multiple expressions e.g. man -k password | grep -e '1' -e '8'
  • -B will provide n lines of before context e.g. -B 5 shows preceding 5 lines of each match
  • -v to inverse (e.g. things not comments grep -v '^#')
  • -i case insensitive
  • [^linux] negate characters, this will match against any characters that are not ‘l’, ‘i’, ‘n’, ‘u’ or ‘x’.
  • -E extended regular expression support

sed and awk⌗

Powerful, line oriented text editors and full blown text based languages in their own right.

awk -F : '/anna/ { print $4 }' /etc/passwd

sed is a stream based (i.e. non-interactive) editor.

Print line 5 (-n will suppress auto printing of pattern space):

sed -n 5p /etc/passwd

Change user bill to william (-i is in-place mode and will mutate the target file, use without -i to test and write out to stdout first):

sed -i s/bill/william/g /etc/passwd

Delete line 4 (using -e editor mode):

sed -i -e '4d' /etc/passwd

Connecting a Linux host⌗

Consoles Terminals and TTYs⌗

A console is the environment which a user is presented with (e.g. graphical or textual)

A terminal an envionment opened on a console that provides access to a shell.

Graphical environments are optional in Linux. To make multiple consoles possible, has the concept of a virtual terminal aka a TTY (short for TeleTYpewriter).

Every terminal is associated with a device /dev/tty1 to /dev/tty6.

This interestly also applies to terminal emulators that are launched a graphical environment such as GNOME, /dev/pts/1, /dev/pts/2 and so on. Use the tty program to output the connected TTY.

The shortcut Alt+F1-6 (or the chvt program) will jump you between TTY1 through to TTY6:

  • TTY1 graphical login
  • TTY2 graphical console
  • TTY3 graphical session
  • TTY4-6 non-graphical consoles

Switch Users (su)⌗

When creating a shell, its environment dictates much of its behavior.

su by default will create a sub shell, that will simply use the existing environment. The bashrc file is used to bootstrap a sub shell.

This often is not wanted. More useful is to create fresh the environment of the target user.

This is known as a login shell, and can be obtained by passing a bare - (dash), -l or --login to the su command. The profile file is used to bootstrap a login shell.

su - shnerg
su -l shnerg
su --login shnerg

/etc/profile is the global shell configuration, and applies to all users login shells.

A login shell (.bash_profile) vs interactive shell (.bashrc).

sudo⌗

sudo executes a command as another user, without requiring use of a login shell.

sudo uses a pluggable based policy, /etc/sudoers by default, to determine what users can do.

/etc/sudoers should never be edited directly, but instead using the visudo command.

The %wheel rule is commonly (lazily?) used to grant users sudo access, by putting them in the wheel group.

SSH⌗

Remote encrypted access, using OpenSSH server daemon.

systemctl status sshd

SSH supports authentication via a simple username and password, but also using an asymetric keypair.

ssh-keygen -t dsa

Managing users and groups⌗

Broadly, there are users for services, humans and root.

Conventions for UID (see /etc/login.defs):

  • < 201: privilaged users
  • 201 - 999: system accounts
  • 1000 - 60000: average joe users

Humans don’t always need to interact directly with a Linux host, for example a web or email server. If this is the case, their default shell should be changed from /bin/bash to /sbin/nologin

Creating users⌗

useradd shnerg will register a new local user account on the system. This involves:

  • create entry in /etc/passwd
  • create entry in /etc/shadow
  • create home directory /home/shnerg
  • create user specific bash initialisation scripts .bash_profile, .bashrc and .bash_logout

The /etc/skel/ directory provides the skeleton scripts and files to be copied into new users home directories.

To remove a user and their home directory, use the -r option, and -f even if the user is logged in.

userdel -rf shnerg

User properties⌗

User objects are made up of many attributes, shown by usermod --help

  • -c an arbitrary annotation such as a role (GECOS field)
  • -d home dir path
  • -e point in time to disable the user
  • -g -u gid uid
  • -G groups
  • -s default shell such as /bin/bash, /sbin/nologin
  • -R location to chroot the user into, interesting!
  • -L -U lock unlock

User configuration files⌗

  • /etc/default/useradd default new user properties
  • /etc/login.defs more default new user properties (if conflicts, takes precedence)
  • /etc/skel/ cloned to new user home directories
  • /etc/passwd user database, all properties of users are encoded here
  • /etc/shadow user password storage and properties, the format of an entry: login:encrypted-password:password-changed-date:min-age:max-age:warning-days:inactive-days:user-expiry-date. Use passwd -S shnerg to display password props for a user.
  • /etc/group all groups

Creating and managing groups⌗

groupadd, groupdel and groupmod

The most common property is the gid

Ways to add users to a group:

  • vi /etc/group
  • vigr for vi with group validation
  • usermod -aG shnerg people

Use getent group finance to validate a group exists, and id <user> to validate the group memberships the user has.

Some facinating (to me anyway) group management programs include newgrp to switch the primary group for the current session, and sg to execute a command as a different group.

Managing password properties⌗

Programs to be across: passwd, chage

  • passwd -S mike displays all password related props
  • echo password | passwd --stdin to set password programmatically (by default will interactively prompt)
  • default password attributes are controlled by /etc/login.defs

Managing Permissions⌗

File permissions are applied at 3 levels; the user, the group and others. Each can read, write and/or execute.

A sample file permission bitmap could be -rwxrw-rw-. The first bit - a dash indicates its a plain old file (there are several types, such as l for symlink, d a directory, …).

Then follows the user, group and others bits.

note: Linux uses a simplistic exit on match algorithm. If the user matches and has no permissions, Linux will not bother evaluating the group or others permission bits (even if they would grant access!).

Changing file ownership⌗

  • chown change owner, can take the names of the login and group like so chown anna:sales sales. Either the user or group can be omitted to not change its existing value.
  • chgrp will change only the group ownership. Its redundant these days with the powers that chown has.

Managing basic permissions⌗

Linux supports three levels of permissions:, known affectionately as UGO (user/group/others).

Permission Octal File Dir
read 4 open list
write 2 modify create/delete
execute 1 run cd

chmod supports symbolic and octal variations of permissions. Some symbolic examples:

In octal notation, set read/write/execute for the user, read/write for the group and just read for others:

chmod 764 afile

In symbolic notation, set user bits to read/execute, remove the write permission for the group (leaving other permission in tact), and add execute permission for others:

chmod u=rx,g-w,o+x afile

More examples:

chmod u+x file1
chmod g-rw file1
chmod o+wx file1

Perhaps the most useful form, apply execute permission to user, group and others:

chmod +x file1

To navigate directory structure, requires execute permission on the directory. Execute bits could be set on directories, but not files, to allow a browsable tree, using chmod with the X (big x) modifier.

When creating new files, the default owner and group will be that of the user (e.g. ben). newgrp finance will default the group to finance.

Default permissions are applied with umask.

groupadd finance #add group
getent group #verify
usermod -G finance amy #add user to group
mkdir /home/finance #create a dir
chown :finance /home/finance #change its group
chmod -R o-rwx g+rw /home/finance #remove other perms and +rw group perms
exit #logout user to reload groups

Recursively setting execute on directories only:

chmod ugo-x -R finance #strip execute on everything
chmod ug+X -R finance #user and group directory exec bit only

To apply the permission bits to all specify a (as opposed to the usual u, g and o):

chmod a+r file1

Understanding umask (user mask)⌗

Simply put is a bit mask.

This mask is applied to the system wide defaults 666 for files, and 777 for directories.

$ umask
0022

Breaking down each bit:

  • The first 0 will not apply any mask to the special bits (suid/guid/sticky bit)
  • The second 0 will apply no mask to the owner
  • The third 2 will mask/strip out (think subtract) write permission (2 in octal) for the group
  • The forth bit 2 will mask/strip out write for others

In practice umask values of 0, 2 and 7 are used:

  • 0 means 6 for files, and 7 for directories
  • 2 means 4 for files and 5 for directories
  • 7 means 0 for files and 0 for directories

The base /etc/bashrc and /etc/profile bash environment bootstrapping files contain entries for setting up default umask values.

Special permissions⌗

Permission Octal File Dir
suid 4 run as owner -
sgid 2 run as group inherit group owner
sticky 1 - only delete if owner

suid⌗

The running of processes as their original owner. Impersonation if you will. Known as suid. Take for example the /usr/bin/passwd program:

-rwsr-xr-x.   1 root root       27872 Feb  5  2016 passwd
   ^

Note the s (suid) bit. While passwd is owned and grouped by root, its runnable by average joe users under roots context, as if being run by the real root user.

Can be set with chmod:

chmod u+s file1
chmod 4500 file1
chmod 2500 file1
chmod 6444 file

sgid⌗

Very useful for defining a group owner that gets inherited within a directory tree.

  • Imagine a /data/sales/ dir.
  • If a user mike creates a file (or dir) within /data/sales/ the user will be set to mike, and the group also set to mike.
  • In a group environment, such as /data/sales/, it would be more useful if the group was set to the sales group
  • The sgid special bit will propagate the group owner to new files or directories and is set with chmod g+s /data/sales/ or chmod 2770 /data/sales/

Sticky bit⌗

Prevents the removal of files and/or directories unless that user is the owner. To set:

chmod +t mydir
chmod 1777 mydir

+t sets the sticky bit:

drw-rw---T. 1 ben  finance    0 May 14 19:06 mydir
         ^

Understanding ACLs⌗

Several years later, in addition to the special bits, ACL (access control list) support was introduced to the kernel.

ACLs offer a few benefits over the simple UGO system:

  • more granular inheritable permission chains on specific directories
  • multiple owners

Scenario, under /data/ exists accounting/ (owned by root:accounting) and sales/ (owned by root:sales).

We want to grant the sales group rx permission to /data/accounting/.

This is not possible with the simple UGO model.

For directories:

  • setfacl -R -m g:sales:rx accounting to set ACLs on existing files and directories
  • setfacl -m d:g:sales:rx accounting to set the default ACL on new objects that are created
  • getfacl accounting to view ACLs

For files:

  • setfacl -m u:george:r myfile

To remove ACLs involves using the dash -

For example setfacl -m d:o::- secret-dir will strip all ACLs for others. Interestingly this (i.e. no permissions for others) will propagate down the tree to any new objects created within secret-dir, awesome!

ls will tack a + symbol to the end of the permission breakdown (e.g. drwxrwxr-x+), to indicate an ACL exists.

Configuring Networking⌗

Network device naming⌗

  • BIOS naming based on hardware properties such as em[1-N] for embedded NICs, p[slot_number]p[port_number]
  • udev naming ethX
  • Physical naming similar to BIOS naming with more variations
  • Logical naming such as vlan or alias
  • To get classical ethX naming, use biosdevname=0 and net.ifnames=0 GRUB boot options

Managing runtime network configuration with ip⌗

ip is useful for showing live networking state.

ip addr help ip addr add 10.0.0.10/24 dev enpls0 ip link show

Storing network configuration persistently⌗

Persistent network configuration is stored in /etc/sysconfig/network-scripts/, each NIC device is represented e.g. ifcfg-enp1s0

The Network Manager service is responsible for managing these network interface configs. An NM configuration is called a connection.

Frontends to the core NetworkManager service, include nmcli and nmtui. They depend on the NetworkManager daemon to be running.

nmcli⌗

man nmcli-examples (example 10)

Bash tab completion rocks for CLI’s like nmcli, check its installed with rpm -qa bash-completion. With nmcli go nuts with double tabbing which will even sensibly dump out specific interface names, to figure out all the options it needs.

Commonly used options:

  • con-name for the profile label
  • ipv4.method for static vs DHCP
  • ipv4.addresses
  • ipv4.dns
  • ipv4.gateway
  • autoconnect

Hot tip: always specify a CIDR style subnet mask, as the default is 32!

To add a new connection:

nmcli connection add con-name limeleaf ifname enp1s0 type ethernet ip4.addresses 192.168.4.210/24 ipv4.gateway 192.168.4.2 ip4.addresses 1.2.3.4/24 ipv4.dns 8.8.8.8

To activate a connection profile (this will re-parse configuration even if the same connection is already active):

nmcli connection up enp1s0-profile

Verify connection status:

nmcli connection show

Modify an existing connection profile to define the DNS:

nmcli connection modify enp1s0-profile ipv4.dns 8.8.8.8

nmcli in the above will update /etc/resolv.conf

nmcli also features an interacive edit mode nmcli connection edit simoid-enp1s0, which will display a shell nmcli>

Routing and DNS⌗

ip route show ip route del default via 192.168.4.1 ip route add default via 192.168.4.2

Using nmcli to set persistent routes (default gateway):

nmcli connection edit simoid-enp1s0
nmcli> set ipv4.gatway 192.168.122.1
nmcli> save
nmcli> quit
nmcli connection up simoid-enp1s0

Setting the hostname on RHEL is done with hostnamectl:

  • hostnamectl status
  • hostnamectl set-hostname host14.bencode.net

Managing Processes⌗

  • In Linux everything is a process (including threads). Threads cannot be individually managed.
  • All processes are assigned a PID
  • Mother hening chores includes setting their scheduling priority and sending signals

Shell jobs⌗

The concept of foreground and background shell processes.

Normally when running a shell command interactively, it is blocking (synchronous) with stdout and stdin wired to the terminal.

Trailing the command with an ampersand &, will unhook stdin and stdout, assign it a job number, and let it continue processing.

  • Example sleep 100 & will output the assigned job number and pid e.g. [3] 2970 = job 3, pid 2970
  • jobs will list all background jobs
  • fg 3 will foreground job 3
  • To background an active shell process Ctrl-Z to stop the job, and simply bg to background it.

ps⌗

The way god reports on processes.

  • ps supports both BSD (naked options) and sys-v (hyphened options) styles, ps -L completely different meaning to ps L
  • ps aux overview of all processes
  • ps -fax process tree
  • ps -fU benjamin all processes owned by a user
  • ps -f --forest -C sshd show process tree only for the sshd process
  • ps L show all format specifiers available
  • ps -eo pid,ppid,user,cmd list processes using specific format specifiers

Memory usage⌗

  • Linux tries to cache files for provide a fast experience. Often as a result, memory appears to over-saturated.
  • Swap provides a virtual (fake) memory address space, backing the memory by (much slower) disk if needed.

Use free to report on the memory situation e.g. free -m show memory units in mebibytes:

  • free truly un-utilised memory
  • available memory be used by buffers or cache that can be liberated immediately
  • If free memory is low and swap used, indicates the server is under memory pressure and could use more RAM

CPU load⌗

Processes as placed into a run queue, which the kernel scheduler uses to allocate processes to CPU cores.

  • uptime to show load averages over 1, 5 and 15 minute spans
  • Load average is the average count of processes that are in a runnable or uninteruptable state.
  • lscpu for CPU meta, including number of CPU’s, sockets, cores per socket and threads per core.
  • uptime load is not normalised by the number of CPU cores (i.e. 1 on single core = 100%, but on a 4 core CPU = 25% load)

System activity with top⌗

Keyboard options:

  • f select display fields
  • M,P,T sort on memory use, CPU or time
  • W save display settings
  • 1 show individual CPU cores
  • k to kill a PID
  • r to set nice level on a PID

Interpreting top by line:

  • 1 is just uptime
  • 2 is processes by categories: stopped = ctrl-z, zombie child processes that have lost their parent process and have become unmanagable.
  • 3 is CPU stats: us user space, sy system space, ni processes with changed niceness, id idle time, wa blocked on I/O, hi hardware interupts, st stolen time (zen virtualisation)
  • 4 for memory stats:

Sending signals to processes⌗

Signals are a way of communicating with processes, even if they’re busily working away.

  • man 7 signal describes the classical signals such as SIGHUP (1), SIGKILL (9) and SIGTERM (15).
  • Signal handling very much depends on the program. Example, nginx will gracefully reparse config if it receives a SIGHUP with terminating active connections.
  • kill is used to send a signal to a PID
  • killall to send signals to all processes that match a search expression (e.g. killall -SIGTERM 'dd' to send SIGTERM to all dd processes)
  • pkill will send a signal based on a the text pattern of a several process attributes (e.g. pkill -signal 15 -U bob send SIGTERM to all of bob’s processes, pkill -signal 1 sshd send SIGHUP to the sshd process).

Priority and niceness⌗

In a nutshell, the amount of priority the process scheduler will give to a process.

  • Nice values range from -20 to 19 (the lower the more priority, the higher the nicer a process is consider toward other processes)
  • Users can make their processes nicer (lower scheduler priority), but not more aggressive (i.e. higher priority)
  • Use the nice and renice commands to alter the priority of non-realtime processes
  • nice will spawn new processes with a nice preset e.g. nice -n -5 dd if=/dev/zero of=/dev/null
  • renice will alter the niceness of an existing process e.g. renice -n 10 -p 34627

In top:

  • the PR column is priority, the lower the higher priority. Priority rt or realtime is a special case, and is the supreme priority.
  • NI is nice level (-20 most aggressive, 19 nicest)

tuned profiles⌗

tuned is a system performance optimiser service.

  • Make sure its running systemctl status tuned
  • tuned-adm is the CLI
  • tuned-adm list show available profiles
  • tuned-adm profile powersave to set the powersave profile
  • tuned-adm active show current profile

Managing Software⌗

RPM and yum⌗

RPM remains the package format of choice for hat-based distros. RPM facts:

  • its from the 90’s
  • its an archive packed by cpio includes a manifest, and list of dependencies
  • they can include scripts
  • RHEL 8 has the concept of protected base packages that can’t be removed (such as vi)

yum was built to be a friendly package frontend:

  • yum search nmap
  • yum install nmap
  • yum remove
  • yum update update all packages
  • yum update kernel update just the kernel package
  • yum provides */sepolicy a deeper search that scans files within each package
  • yum info nmap show the package manifest
  • yum list all
  • yum list installed
  • yum history list of recent package activity
  • yum history undo 4 undo transaction 4 in the above history list

Cool tip yumdownloader (in the yum-utils package) will download RPM to file system for inspection.

rpm queries⌗

With yum, the older rpm CLI is used less directly these days. However RPMs are still managed by the same underlying accounting database as forever, which the rpm CLI exposes.

This is useful for querying, such as the specific files installed as part of a package, and so on.

  • rpm -qf /usr/bin/awk which package installed this file?
  • rpm -ql tmux list each file installed by the tmux package
  • rpm -qc openssh-server list the configuration files for a package
  • rpm -qp --scripts foo.rpm review the scriptlets (pre-install, post-install) of a standalone RPM

yum Groups⌗

Chunks up software into broad categories.

  • yum group list
  • yum group list hidden
  • yum group info "System Tools"
  • yum group install --with-optional "Directory Client"

Repositories⌗

New in RHEL 8 are AppStreams.

Defined by /etc/yum.repos.d/.

appstream.repo:

[appstream]
name=appstream
baseurl=file:///repo/AppStream
gpgcheck=0

base.repo:

[base]
name=base
baseurl=file:///repo/BaseOS
gpgcheck=0

To verify run yum repolist

Modules and Application Streams⌗

New in RHEL 8, appstreams separate user (i.e. application) packages from core system (i.e. base) packages.

  • Application Streams come as either traditional RPMs or the new module format.
  • Modules (ex: php) themselves can in-turn contain streams (ex php:7.1, php:7.2).
  • Enabling a module stream (php:7.1) opens up access to its packages
  • Modules can have profiles (e.g. a minimal, devel)
  • Module streams support upgraded and downgrading between each other (php:7.1 > php:7.3 or php:8.0 > php:7.1)

Managed with yum:

  • yum module list
  • yum module provides httpd show the module that provides a paricular package
  • yum module info php specific module info
  • yum module info --profile php show the profiles of a specific module
  • yum module list php to list available modules
  • yum module install php:7.3 or yum install @php:7.3 will enable and install specific module stream
  • yum module install php:7.3/devel to install the module using a specific profile
  • yum module enable php:7.1 enables the module stream, without installing

Updates between module streams just works:

  • yum module install php:7.1
  • some time later yum module install php:7.3

Beware yum update will use enabled module streams (e.g. php:7.1 will not automatically be upgraded to php:7.3)

Red Hat Subscription Manager⌗

The RHEL repositories require an active subscription.

  • subscription-manager register
  • subscription-manager attach --auto

Systemd⌗

The init system. The kernel hands over to it, when its ready to bootstrap user space.

  • Managed items are called units (services, mounts, timers, sockets etc)
  • systemctl is the management CLI
  • systemctl -t help list of supported unit types
  • systemctl list-unit-files list each unit, its definition file and status
  • systemctl enabled vsftpd enable (auto start) service
  • systemctl start vsftpd start the service process

Modifying service configuration (see man systemd.service):

  • Default unit files: /usr/lib/systemd/system/
  • Custom unit files: /etc/systemd/system/
  • Runtime generated unit files: /run/systemd/
  • systemctl cat rsyslog.service dump unit configuration
  • systemctl edit unit.service will create overlay in /etc/systemd/system
  • systemctl show to dump available parameters that can be used in unit configs
  • systemctl daemon-reload after modifying unit files, often is necessary

When editing an overlay, can just add the extra options as they will be additive to the existing base configuration:

[Service]
Restart=on-failure
RestartSec=60

Scheduling Tasks⌗

cron⌗

cron, the classical scheduling daemon.

  • has no stdout
  • crontab -e to create user specific job
  • /etc/cron.d/ to create system wide job
  • /etc/cron.{hourly,daily,weekly,monthly} managed by anacron, for regular script execution
  • /etc/crontab (deprecated) was once used to configure jobs. crontab remains useful for specifying the environment for cron such as the SHELL.

cron time specification (man 5 crontab) example */10 4 11 12 1-5:

  • */10 every 10 minutes
  • 4 only on hour 4
  • 11 only on day 11
  • 12 only on month 12
  • 1-5 only on day of week 1-5

Example, write hello to syslog on minute 57, hour 20:

  • crontab -e
  • 57 20 * * * logger hello

anacron, runs commands periodically.

  • Unlike cron doesn’t assume the machine is running all day everyday.
  • Configured by /etc/anacrontab

at⌗

at unlike cron is used for one-off jobs.

  • make sure atd daemon is running
  • provide its own interactive shell to take job specifications
  • atq to list at queue
  • atrm to remove at job

Example:

  • at teatime
  • logger have a cup of tea

systemd Timers⌗

cron is still the gold standard, however this is still a viable option.

  • man 5 systemd.timer and man 5 systemd.time for time specification
  • ls /usr/lib/systemd/system/*.timer to list timers
[Unit]
Description=Discard unused blocks once a week
Documentation=man:fstrim
ConditionVirtualization=!container

[Timer]
OnCalendar=weekly
AccuracySec=1h
Persistent=true
RandomizedDelaySec=6000

[Install]
WantedBy=timers.target

Managing temporary files⌗

A common way to manage (create, delete) temporary files. See man tmpfiles.d.

  • /usr/lib/tmpfiles.d/ setting files
  • For example /usr/lib/tmpfiles.d/tmp.conf contain settings for automatic tmp files cleanup
  • systemd-tmpfiles-clean.timer unit can be configured to automatically clean up temporary files (by triggering systemd-tmpfiles-clean.service which in turn runs systemd-tmpfiles --clean).
  • If you want to make modifications, copy conf file from /usr/lib/tmpfiles.d/ to /etc/tmpfiles.d/ and edit it there.
  • Run systemd-tmpfiles --clean /etc/tmpfiles.d/tmp.conf manually to parse and test configuration changes.
  • To register a new custom tmpfiles configuration systemd-tmpfiles --create /etc/tmpfiles.d/foo.conf

Logging⌗

Rsyslog⌗

The rocket-fast Syslog Server

rsyslogd is the defacto syslogd used by most distros. It monitors configurable sources (e.g. /dev/log) and writes to configurable sinks (e.g. in /var/log/)

  • A daemon managed by the rsyslogd.service unit
  • Configured by /etc/rsyslog.conf
  • Snap-in configs in /etc/rsyslog.d/
  • Each logger rule line is made up of 3 elements; facility ({auth,authpriv,cron,daemon,kern,lpr,mail,mark,news,security,syslog,user,uucp,local{0-7}}), severity ({debug,info,notice,warn,err,crit,alert,emerg,panic}) and an action (regular file, database table, remote machine, a tty, discard, and more).
  • For services that don’t have a specific facilty, use local{0-7}
  • You can use the logger CLI to write messages to rsyslogd manually

Sample rules from /etc/rsyslog.conf:

*.info;mail.none;authpriv.none;cron.none       /var/log/messages

Log everything INFO or higher, except mail/authpriv/cron to /var/log/messages

mail.*                                         -/var/log/maillog

Notice the - before the filename. This tells rsyslog to buffer writes.

Systemd Journal⌗

Being systemd, it invented its own logger, called journald. By default journald is in-memory, but it sinks logs to /dev/log which rsyslog listens to.

While rsyslogd, depending on it config, will likely perist journald logs to /var/log/.

  • By default writes journal to /run/log/journal, which is cleared across reboots.
  • It’s possible to persist the systemd journal logs. mkdir /var/log/journal/ and restart the journald.service unit.
  • Update /etc/systemd/journald.conf, set Storage to one of {persistent,volatile,auto} (auto will use /var/log/journal/ only if it exists)
  • Journal logs are propagated to rsyslogd using the imjournal input module
  • The journalctl CLI is the frontend for querying journal logs.
  • Use tab completion to build out filters, such as journalctl UNIT=dbus.service

Logrotate⌗

Used to roll up (rotate) logs.

  • Its started through cron.daily
  • Configured by /etc/logrorate.conf or /etc/logrotate.d/

Managing Storage⌗

Disk layout⌗

This is driven by the underlying management scheme; either BIOS based or UEFI based.

BIOS, designed in the early 80’s, uses a MBR (master boot record) to define the partition layout of the system. With 512 bytes to store boot information, and 64 bytes for partition layout, can support upto 4 partitions with a max size of 2TiB. The 4 partition limitation was later overcome, by leveraging logical partitions contained within an extended partition.

UEFI (Universal Extended Firmware Interface), uses GPT (GUID partition table), supports upto 128 partitions.

Useful commands:

  • lsblk lists out all block devices attached to a system
  • parted is the preferred partition management program
  • All block devices are represented in /dev/ e.g. /dev/vda1
  • /proc/partitions

Creating partitions⌗

GPT partitions with parted⌗

parted is now the defacto utility to be used, however fdisk and gdisk remain available.

  • parted /dev/sdb to get started on a block device
  • print to list partition table
  • mklabel msdos|gpt to define the partition type (MBR or GPT)
  • mkpart [part-type] name fs-type start end
    • part-type (optional) of primary, logical or extended only applies to MBR
    • name a mandatory label
    • fs-type an irrelevant file system piece of metadata (does NOT actually layer a file system onto the volume)
    • start end the locations starting from the beginning of the block device to apply the partition
  • udevadm settle to flush changes
  • cat /proc/partitions to verify

MBR partitions with fdisk⌗

Good old fdisk. Sanely defaults when working with MBR partitions, e.g. if 3 primary partitions exist, knows that only 4 primaries can exist, so defaults the next partition to type extended, which you can then fill up with logical partitions.

If the block device is in use, fdisk will be unable to write the partition table. Run partprobe if this is the case.

Logical partitions get named sequentially within the extended partition, in which they live. If a logical partition is removed, other higher logical partitions will decrement. For example:

  • vdb4 is an extended
  • vdb5 is logical
  • vdb6 is logical
  • vdb5 gets removed
  • vdb6 becomes vdb5 (breaking any fstab entries dependent on the block device name)

File System Choices⌗

XFS⌗

The default.

  • Fast
  • CoW (copy on write) to guarantee data integrity; before writing a file to disk, the original is preserved elsewhere making it possible to revert to it’s previous state
  • Size can be increased, but not decreased
  • mkfs.xfs /dev/vdb2
  • xfs_admin to manage properties of an XFS file system, such as defining a label
  • xfsdump for creating backups, including XFS specific attributes
    • Example: xfsdump -I 0 -f /backupfiles/data.xfsdump /data creates a full backup of the contents of /data
  • xfsrestore to restore these backups:
    • Example: xfsrestore -f /backupfiles/data.xfsdump /data
  • xfsrepair is used to repair broken XFS file systems

Ext4⌗

The old (v6 and before) default.

  • Backward compatible with Ext2
  • Uses a journal to guarantee data integrity
  • Size can be increased and decreased (after growing partitions with parted, use resize2fs <device-name>)
  • mkfs.ext4 /dev/vdb3
  • tune2fs to manage Ext4, like labelling

Mounting⌗

The act of attaching a block device to sub-branch within the / file system tree.

  • /etc/fstab (fs table) is used to persistently mount volumes
  • In the post systemd world, fstab nowdays is simply a frontend to systemd mounts (using the systemd-fstab-generator code generator)
  • After modifying fstab ensure you refresh systemd with systemctl daemon-reload
  • To auto mount, unmounted volumes in fstab mount -a
  • When unmounting, you may get a target is busy response. Use the awesome lsof /mnt to track down processes currently using the mountpoint.

Persistent block device naming⌗

Block device names, like /dev/sdb are not guaranteed to be reissued consistently (particularly in cloud environments), or if partitions get reorganised.

Other identifiers include UUID, labels and device paths, which are all represented under /dev/disk/.

  • blkid shows the UUID and LABEL id’s assigned to each block device
  • To mount based on a UUID:
    • Use blkid to find the uuid for the block device
    • In fstab replace the block device name with UUID=22c2d576-0ec2-4ded-8392-fb17a795fb42
  • To mount based on a label:
    • Use tune2fs -L or xfs_admin -L to set labels on Ext4 or XFS
    • In fstab replace the block device name with LABEL=foo

systemd mounts⌗

systemd manages all persistent mounts, even those done using fstab

  • Hand crafting .mount files, allows you to directly define a systemd mount
  • This provides finer control over when a mount is required (unlike fstab which is simply done at startup)
  • /usr/lib/systemd/system/tmp.mount provides a great example (disabled by default)
  • Convention is for system RPM packages to use /usr/lib/systemd/system/ and user defined mounts /etc/systemd/system/
  • The name of the .mount file is important! It must match the path of the mount point e.g. foo.mount mounts to /foo. foo-bar.mount mounts to /foo/bar

Swap⌗

RAM emulated on disk.

  • All Linux systems should have some swap space.
  • Can exist on a block device, including on a swap file
  • Using parted ensure to set the type to linux-swap, or type 82 in fdisk
  • Use mkswap to initialise the swap FS.
  • Activate it with swapon
  • Use free -m to show available swap space

Advanced Storage⌗

LVM⌗

LVM (Logical Volume Manager) is a higher level abstraction of storage, with rich features; resizing, snapshots.

It works by abstracting physical volumes (PVs) from logical logical volumes (LVs), through a volume group (VG). For example a 20GiB LV can span two 10GiB PV’s.

Device Mapper is the

The LVM tango:

  • Create partitions as type lvm (set n lvm on in parted, or type 8e in fdisk)
  • Create PV with pvcreate /dev/sdb1
  • Verify with pvs
  • Create VG with vgcreate vgdata /dev/sdb1
  • Verify with vgs
  • Create LV with lvcreate -n lvdata -L 1G vgdata
  • Verify with lvs
  • Apply a file system mkfs.xfs /dev/vgdata/lvdata
  • Register LV in fstab (no need for labels or uuid based mounting, as LVM is device independent)

Growing LVM depends on how far down the LV/VG/PV stack the space shortage goes.

Growth tango:

  • Is there enough space in the volume group with vgs
  • No? vgextend
  • Extend the logical volume with lvextend -r -L +1G to grow the volume file system its hosting
  • If you forget the -r switch its over to you based on what FS you’re dealing with
    • Ext4 use e2resize
    • XFS use xfs _growfs

Stratis⌗

Stratis is RedHat’s solution to Btrfs and ZFS, implemented in user space, to better support cloud and containerised environments.

  • Built on top of raw block devices (including LVM). No partitions.
  • Features include; thin provisioning, snapshots, cache tier, programmatic API, monitoring and repair
  • Creates a /dev/stratis/my-pool/ for each pool, full of links to actual devices
  • XFS is put on a volume on top of the pool
  • Each pool can contain one or more file systems
  • df doesnt work, as stratis volumes are thin provisioned
  • instead use the stratis related utilities, such as stratis [blockdev|filesystem|pool]

Creating a new pool:

  • yum install stratis-cli stratisd
  • systemctl enable --now stratisd
  • wipefs -a /dev/vdb to clear any existing partition tables that may exist
  • stratis pool create mypool /dev/vdb - partitions NOT supported, at least 1GiB
  • stratis filesystem create mypool myfs1 plops on XFS
  • stratis filesystem list mypool

Mounting the pool:

  • mkdir /myfs1
  • mount /dev/stratis/mypool/myfs1 /myfs1
  • stratis pool list
  • stratis filesystem list
  • stratis blockdev list mypool
  • blkid to find UUID of stratis volume, then mount in fstab as normal

Snapshots:

  • An independent file system that can be mounted
  • Needs at least 0.5GiB to store the XFS journal
  • The snapshot is not linked to it origin in anyway
  • To create one stratis filesystem snapshot mypool myfs1 myfs1-snapshot
  • To revert to a snapshot:
    • umount /myfs1
    • stratis filesystem destroy mypool myfs1
    • stratis filesystem snapshot mypool myfs1-snapshot myfs1

VDO⌗

Virtual Data Optimiser (VDO): focuses on storage data in the most efficient way, with the concept of deduplicated and compressed storage pools.

  • Used mainly in cloud and containerised environments.
  • Like stratis provides thin-provisioned storage.
    • For VMs and containers, set the logical size 10x the physical size.
    • For object storage, go 3x.
  • Must be at least 4GiB.
  • Can be created on a block device OR a partition.
  • df doesnt work here either :( instead use vdostats --human-readable
  • yum install vdo kmod-kvdo
  • vdo create --name=vdo1 --device=/dev/vdb --vdoLogicalSize=1T
  • mkfs.xfs -K /dev/mapper/vdo1 -K (do NOT attempt to discard blocks) is handy speed up hack, when working with thin-provisioned storage
  • In fstab include the x-systemd.requires=vdo.service, and the discard mount options
  • Remember /usr/share/doc/vdo/examples has great systemd mount templates

Troubleshooting:

  • When creating may get the error vdo: ERROR - Found existing signature on /dev/vdb at offset 512. This is a safety check, telling you the volume looks to already be initialised, as possibly used. If its an old partition and you’re happy to replace it, run wipefs --all --force /dev/vdb

LUKS⌗

The Linux Unified Key Setup (LUKS) is a disk encryption specification created by Clemens Fruhwirth in 2004.

LUKS is a standard on-disk format. This facilitates compatibility and interoperability between programs, and assures programs implement password management in a secure manner.

Two default programs are provided, cryptsetup a dm-crypt reference implementation, and luksmeta for storing metadata in a LUKSv1 header.

To setup a LUKS encrytped volume:

  1. Create partition with parted
  2. Format the LUKS device cryptsetup luksFormat <partition-device-name>
  3. Bind LUKS volume as a device mapper name cryptsetup luksOpen <partition-device-name> <device-mapper-name>
  4. Format the LUKS volume mkfs.xfs <device-mapper-name>
  5. Mount the device mapper name (ex /dev/mapper/<device-mapper-name>)

The /etc/crypttab and /etc/fstab files can be used to automate steps 3 and 4.

An example crypttab entry (see man crypttab for more). The third param sets the password (if empty, or set to none or -, the password must be interactively entered during system boot):

myluksvolume    /dev/sda5    none

Advanced tasks⌗

Kernel management⌗

Use modprobe <module-name> to manually load a kernel module, modprobe -r <module-name to unload, and modinfo <module-name> to list parameters that a module supports.

Module params can be edited under /etc/modprobe.d/

The /proc directory provides a UI to the kernel.

  • Pid directories map to each running process, providing metadata about each
  • Status files such as /proc/partitions
  • Tunable kernel parameters are managed under /proc/sys. To temporarily update echo new value into the /proc/sys file of interest such as echo 1 > /proc/sys/net/ipv4/ip_forward. Once happy persist the configuration using /etc/sysctl.conf.
  • sysctl -a to dump all kernel tunables

Install new kernel with yum update kernel or yum install kernel (both have the same effect).

Boot procedure⌗

GRUB⌗

GRUB2 runtime parameters, are editing before bootstrapping the system. When booting, the GRUB bootloader will display boot entries:

  • e to edit. The linux line is most interesting, responsible for booting the kernel. Options are tacked on the back of this line, and can be freely edited here. When ready boot the system with the desired parameters.
    • rhgb redhat graphical boot
    • quiet quiet boot
  • c for command interface. help for lits of supported commands. esc will return you out to boot menu.

To persist GRUB2 boot parameters:

  1. Edit the /etc/default/grub file.
  2. Compile changes to grub.cfg using either:
  • For BIOS systems: grub2-mkconfig -o /boot/grub2/grub.cfg
  • For UEFI systems: grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg

Systemd targets⌗

A target is just a group of unit files. Isolatable targets define a desired final state, such as emergency.target, rescue.target, multi-user.target and graphical.target.

Disabling and enabling services, links them into their desired targets. This is dictated by their WantedBy setting within the unit file, as shown by systemctl cat httpd

[Install]
WantedBy=multi-user.target

The default target and their wants are defined in /etc/systemd/system/, as symlinks. This directory is just a big bag of symlinks.

To boot into a specific target:

  • GRUB2 boot prompt systemd.unit=rescue.target
  • On a running system systemctl isloate xxx.target

systemctl list-dependencies visually lays out the hierarchy of targets and units.

systemctl get-default and set-default can be used to, you guessed it, set the default target.

Essential troubleshooting⌗

Depending how deep the problem is. Common solutions include:

  • Tweaking GRUB2 kernel args
    • rd.break to pause in initramfs just after loading the kernel
    • setting the init system to init=/bin/bash
  • Then setout to reduce the service footprint using systemd.unit=emergency.target and systemd.unit=rescue.target.

Changing root password⌗

Edit GRUB2 entry while booting, add rd.break to end of linux kernel line, once in a shell, remount sysroot as rw, update the root password and flag to selinux this is cool:

mount -o remount,rw /sysroot
chroot /sysroot
echo secret | passwd --stdin root
touch /.autorelabel
^d ^d

Bash scripting⌗

Only the most basic skills needed.

Managing network services⌗

SSH⌗

  • ssh-keygen creates a new keypair
  • ssh-copy-id copies public key to target host
  • ssh-agent /bin/bash caches private key passphrase in shell
  • ssh-add adds passphrase to cache

Configuration:

  • server = /etc/ssh/sshd_config
  • client = /etc/ssh/ssh_config

Key settings: Port, PermitRootLogin, AllowUsers, PubkeyAuthentication, PasswordAuthentication, X11Forwarding

Remote file management:

  • scp local-file user@remote-host:/path
  • sftp FTP over SSH
  • rsync
    • -r recurse entire tree
    • -l sync symlinks
    • -p preserve symlinks
    • -n dry run
    • -a archive mode (same as -rlptgoD)
    • -A archive mode and syncs ACLs
    • -X sync SELinux context labels

httpd⌗

SELinux⌗

Network security⌗

Automating installs⌗

Configuring time services⌗

Access remote file systems⌗

Containers⌗

Rando cool⌗

  • ctrl+l = clear terminal
  • ls -d don’t show contents of directories
  • \ls un-alias a command, by preceding it with a backslash \
  • alias to display evaluated bash aliases
  • tar command options are not prefixed with a hypen - (BSD compat)
  • tac is the inverse program of cat
  • chvt jumps between TTY e.g. chvt 3
  • ssh-keygen supports a number of ciphers, set using [-t dsa | ecdsa | ed25519 | rsa], RSA by default
  • yum history full journal of package installs
  • yumdownloader is included in yum-utils lets you download packages to local file system
  • run-parts which comes as part of the cron ecosystem, is a script that runs all executables in a directory.
  • A sector is 512 bytes.
  • The act of creating a file system is referred to as making as opposed to formatting on Linux
  • man test for a quickref of bash evaluations possible