My preparation notes for the RHCSA 8 exam.

Essential Tools

Documentation

The ability to understand a program using local documentation resources; man, info, /usr/share/doc, within the RPM package.

man

To browse man pages for a keyword use -k, e.g. scan documentation for all things relating to password:

man -k password

Alternatively:

mandb
apropos passwd

Specific sections with man, refer to different topics, e.g. section 5 is about config files, so man 5 passwd would bring up the documentation on /etc/passwd.

1 = user commands 5 = configuration files 7 = broad topics such as background 8 = sys admin

man -k user | grep 8 | grep create

/usr/share/doc

A gold mine of documents and sample configuration files. Usually for distributions that are not considered core, and don’t offer man or info pages.

RPM bundled documentation

$ rpm -qd tmux
/usr/share/doc/tmux/CHANGES
/usr/share/doc/tmux/FAQ
/usr/share/doc/tmux/TODO
/usr/share/man/man1/tmux.1.gz

General Searching Techniques

General search engine:

$ updatedb
$ locate passwd

Search path for passwd:

$ which passwd
/usr/bin/passwd

Search one-line man page descriptions:

$ whatis passwd
passwd (1)           - update user's authentication tokens
sslpasswd (1ssl)     - compute password hashes
passwd (5)           - password file

Find binaries and man pages for ls:

$ whereis -bm ls
ls: /usr/bin/ls /usr/share/man/man1/ls.1.gz /usr/share/man/man1/ls.1p.gz

Shell history

  • history dump history, by default the last 1000 commands
  • ctrl+r to search backwards through history for pattern
  • history -c clear history (in-memory only)
  • history -w write history
  • !32 run history event 32 (again)

Globbing

aka using wildcards see man 7 glob

  • ls host* zero or more chars
  • ls ?ost any single char
  • ls [hm]ost groups of chars
  • ls [!hm]ost negated groups of chars
  • ls [0-9][0-9]script multiple groups of restricted chars

I/O Redirection and Pipes

  • < stdin from a file or another programs stdout
  • > stdout to new file (overwrite if exists)
  • >> stdout to file (appending if exists)
  • 2> stderr redirection
  • 2>&1 stderr to stdout (useful for piping stderr, as pipes only work with stdout)
  • | pipe stdout from one program to stdin of another (pipes only support stdout to stdin communication, i.e. not stderr)

Essential File Management

Linux file system layout

See man hier and man file-hierarchy

Big hitters:

  • Boot partition: /boot/ and /efi/
  • System configuration: /etc/
  • Scripts and binaries: /bin/, /sbin/, /usr/sbin/ now all link back to /usr/bin/
  • Shared libraries: /lib/, /lib64/ link to /usr/lib/ and /usr/lib64/ respectively
  • Virtual kernel file system: /proc/ such as /proc/meminfo
  • Persistent variable data: /var/ such as /var/cache/, /var/log/, /var/tmp

Finding Files

locate

$ updatedb
$ locate passwd

find

Basic examples:

$ find / -size +100M -exec ls -l {} \;
$ find /etc -name motd  #named motd
$ find /etc -user schnerg  #owned by user shnerg
$ find / -mtime 3  #modified in last 3 days
$ find / -mtime +3  #not within the last 3 days

$ id ben
uid=1000(ben) gid=1000(ben) groups=1000(ben),1004(finance)
$ find / -uid 1000

$ find / -user ben -type f  #filter by files
$ find / -user ben -type f -exec cp {} /home/mary \; #execute a shell command against each result file `{}`

Archiving and compression with tar

Creating archives:

tar cvf foo.tar directory1 file1 file2
tar czvf foo.tar.gz directory1 file1 file2 #with gzip
tar cjvf foo.tar.bz directory1 file1 file2 #with bzip

List contents (without extraction):

tar tvf foo.tar

Extract them:

tar xvf foo.tar
tar xzvf foo.tar.gz
tar xjvf foo.tar.bz

Extract from a base directory:

tar xvf foo.tar -C /

Diffencing an archives contents with an existing exploded structure:

$ tar -dzvf dir1-v2.tar.gz
directory1/
directory1/file4
directory1/wookie4
tar: directory1/wookie4: Warning: Cannot stat: No such file or directory
directory1/wookie3
tar: directory1/wookie3: Warning: Cannot stat: No such file or directory
directory1/file3
directory1/imp1
directory1/imp1: Mod time differs
directory1/imp1: Size differs
directory1/imp2
directory1/file1
directory1/file2

Compression:

gzip file1
gzip -d file1
bzip2 file1

Listing compression stats on a compressed file:

$ gzip -l hello1.gz
         compressed        uncompressed  ratio uncompressed_name
                 83                  62   6.5% hello1

Archiving with star.

star -c -f=foo.tar directory1 hello1 hello2
star -cz -f=foo.tar.gz directory1 hello1 hello2 #with compression

Listing

$ star -t -f=foo.tar
directory1/
directory1/file4
directory1/file3
directory1/imp1
directory1/imp2
directory1/file1
directory1/file2
hello1
hello2

Extract a specific (hello1) file from the archive:

star -x -f=foo.tar hello1

Soft links, or symbolic links (symlinks), are simply pointers to other files. Symlinks can span multiple file systems. Permissions on symlinks aren’t real. The underlying permissions of the target file is what gets applied. They can easily be created with ln like so:

ln -s /etc/motd ~/motd

Hard links are links to a specific inode (shown with ls -i) on the file system. Due to this coupling, cannot span different file systems or devices.

$ ls -l
drwxrwxr-x. 2 ben ben 4096 May 13 20:42 directory1
-rw-rw-r--. 1 ben ben    0 May 13 20:53 hello1
lrwxrwxrwx. 1 ben ben    9 May 14 17:31 motd -> /etc/motd

$ ln hello1 hello1-hardlink
$ ls -l
drwxrwxr-x. 2 ben ben 4096 May 13 20:42 directory1
-rw-rw-r--. 2 ben ben    0 May 13 20:53 hello1
-rw-rw-r--. 2 ben ben    0 May 13 20:53 hello1-hardlink
lrwxrwxrwx. 1 ben ben    9 May 14 17:31 motd -> /etc/motd

In the ls long listing output, take note of the 2nd column, which represents the count of references to the same inode, which increases after creating a hard link. Some properties of hard links:

  • Hard links will always report the same metadata such as permission bits, modification timestamps, etc
  • inode reference counts will increase for each hard link.
  • Removal of the target file or hard link will not result in broken links, as they both physically reference the same inode.

Working with text

Regular expressions

See man 7 regex

  • . any single character

  • ? one or more

  • * zero or more

  • cat for concatenation, commonly used to dump contents to stdout

  • tac concatenation in reverse order

  • cut parses fields based on simple delimiter cut -d : -f 1 /etc/passwd cuts the first field in /etc/passwd based on a colon delimiter

  • sort can sort alphabetically or numerically e.g. cut -d : -f 3 /etc/passwd | sort -n

  • head first n lines

  • tail last n lines

  • tr translator e.g. lower to upper casing cut -d : -f 1 /etc/passwd | tr [a-z] [A-Z]

grep

The pinnacle of text processing, handed down by god himself.

grep '^#' /etc/sysconfig/sshd

Noteworthy:

  • always place regex between single quotes to avoid ambiguity of globbing
  • use -e to specify multiple expressions e.g. man -k password | grep -e '1' -e '8'
  • -B will provide n lines of before context e.g. -B 5 shows preceding 5 lines of each match
  • -v to inverse (e.g. things not comments grep -v '^#')
  • -i case insensitive
  • [^linux] negate characters, this will match against any characters that are not ‘l’, ‘i’, ‘n’, ‘u’ or ‘x’.
  • -E extended regular expression support

sed and awk

Powerful, line oriented text editors and full blown text based languages in their own right.

awk -F : '/anna/ { print $4 }' /etc/passwd

sed is a stream based (i.e. non-interactive) editor.

Print line 5 (-n will suppress auto printing of pattern space):

sed -n 5p /etc/passwd

Change user bill to william (-i is in-place mode and will mutate the target file, use without -i to test and write out to stdout first):

sed -i s/bill/william/g /etc/passwd

Delete line 4 (using -e editor mode):

sed -i -e '4d' /etc/passwd

Connecting a Linux host

Consoles Terminals and TTYs

A console is the environment which a user is presented with (e.g. graphical or textual)

A terminal an envionment opened on a console that provides access to a shell.

Graphical environments are optional in Linux. To make multiple consoles possible, has the concept of a virtual terminal aka a TTY (short for TeleTYpewriter).

Every terminal is associated with a device /dev/tty1 to /dev/tty6.

This interestly also applies to terminal emulators that are launched a graphical environment such as GNOME, /dev/pts/1, /dev/pts/2 and so on. Use the tty program to output the connected TTY.

The shortcut Alt+F1-6 (or the chvt program) will jump you between TTY1 through to TTY6:

  • TTY1 graphical login
  • TTY2 graphical console
  • TTY3 graphical session
  • TTY4-6 non-graphical consoles

Switch Users (su)

When creating a shell, its environment dictates much of its behavior.

su by default will create a sub shell, that will simply use the existing environment. The bashrc file is used to bootstrap a sub shell.

This often is not wanted. More useful is to create fresh the environment of the target user.

This is known as a login shell, and can be obtained by passing a bare - (dash), -l or --login to the su command. The profile file is used to bootstrap a login shell.

su - shnerg
su -l shnerg
su --login shnerg

/etc/profile is the global shell configuration, and applies to all users login shells.

A login shell (.bash_profile) vs interactive shell (.bashrc).

sudo

sudo executes a command as another user, without requiring use of a login shell.

sudo uses a pluggable based policy, /etc/sudoers by default, to determine what users can do.

/etc/sudoers should never be edited directly, but instead using the visudo command.

The %wheel rule is commonly (lazily?) used to grant users sudo access, by putting them in the wheel group.

SSH

Remote encrypted access, using OpenSSH server daemon.

systemctl status sshd

SSH supports authentication via a simple username and password, but also using an asymetric keypair.

ssh-keygen -t dsa

Managing users and groups

Broadly, there are users for services, humans and root.

Conventions for UID (see /etc/login.defs):

  • < 201: privilaged users
  • 201 - 999: system accounts
  • 1000 - 60000: average joe users

Humans don’t always need to interact directly with a Linux host, for example a web or email server. If this is the case, their default shell should be changed from /bin/bash to /sbin/nologin

Creating users

useradd shnerg will register a new local user account on the system. This involves:

  • create entry in /etc/passwd
  • create entry in /etc/shadow
  • create home directory /home/shnerg
  • create user specific bash initialisation scripts .bash_profile, .bashrc and .bash_logout

The /etc/skel/ directory provides the skeleton scripts and files to be copied into new users home directories.

To remove a user and their home directory, use the -r option, and -f even if the user is logged in.

userdel -rf shnerg

User properties

User objects are made up of many attributes, shown by usermod --help

  • -c an arbitrary annotation such as a role (GECOS field)
  • -d home dir path
  • -e point in time to disable the user
  • -g -u gid uid
  • -G groups
  • -s default shell such as /bin/bash, /sbin/nologin
  • -R location to chroot the user into, interesting!
  • -L -U lock unlock

User configuration files

  • /etc/default/useradd default new user properties
  • /etc/login.defs more default new user properties (if conflicts, takes precedence)
  • /etc/skel/ cloned to new user home directories
  • /etc/passwd user database, all properties of users are encoded here
  • /etc/shadow user password storage and properties, the format of an entry: login:encrypted-password:password-changed-date:min-age:max-age:warning-days:inactive-days:user-expiry-date. Use passwd -S shnerg to display password props for a user.
  • /etc/group all groups

Creating and managing groups

groupadd, groupdel and groupmod

The most common property is the gid

Ways to add users to a group:

  • vi /etc/group
  • vigr for vi with group validation
  • usermod -aG shnerg people

Use getent group finance to validate a group exists, and id <user> to validate the group memberships the user has.

Some facinating (to me anyway) group management programs include newgrp to switch the primary group for the current session, and sg to execute a command as a different group.

Managing password properties

Programs to be across: passwd, chage

  • passwd -S mike displays all password related props
  • echo password | passwd --stdin to set password programmatically (by default will interactively prompt)
  • default password attributes are controlled by /etc/login.defs

Managing Permissions

File permissions are applied at 3 levels; the user, the group and others. Each can read, write and/or execute.

A sample file permission bitmap could be -rwxrw-rw-. The first bit - a dash indicates its a plain old file (there are several types, such as l for symlink, d a directory, …).

Then follows the user, group and others bits.

note: Linux uses a simplistic exit on match algorithm. If the user matches and has no permissions, Linux will not bother evaluating the group or others permission bits (even if they would grant access!).

Changing file ownership

  • chown change owner, can take the names of the login and group like so chown anna:sales sales. Either the user or group can be omitted to not change its existing value.
  • chgrp will change only the group ownership. Its redundant these days with the powers that chown has.

Managing basic permissions

Linux supports three levels of permissions:, known affectionately as UGO (user/group/others).

Permission Octal File Dir
read 4 open list
write 2 modify create/delete
execute 1 run cd

chmod supports symbolic and octal variations of permissions. Some symbolic examples:

In octal notation, set read/write/execute for the user, read/write for the group and just read for others:

chmod 764 afile

In symbolic notation, set user bits to read/execute, remove the write permission for the group (leaving other permission in tact), and add execute permission for others:

chmod u=rx,g-w,o+x afile

More examples:

chmod u+x file1
chmod g-rw file1
chmod o+wx file1

Perhaps the most useful form, apply execute permission to user, group and others:

chmod +x file1

To navigate directory structure, requires execute permission on the directory. Execute bits could be set on directories, but not files, to allow a browsable tree, using chmod with the X (big x) modifier.

When creating new files, the default owner and group will be that of the user (e.g. ben). newgrp finance will default the group to finance.

Default permissions are applied with umask.

groupadd finance #add group
getent group #verify
usermod -G finance amy #add user to group
mkdir /home/finance #create a dir
chown :finance /home/finance #change its group
chmod -R o-rwx g+rw /home/finance #remove other perms and +rw group perms
exit #logout user to reload groups

Recursively setting execute on directories only:

chmod ugo-x -R finance #strip execute on everything
chmod ug+X -R finance #user and group directory exec bit only

To apply the permission bits to all specify a (as opposed to the usual u, g and o):

chmod a+r file1

Understanding umask (user mask)

Simply put is a bit mask.

This mask is applied to the system wide defaults 666 for files, and 777 for directories.

$ umask
0022

Breaking down each bit:

  • The first 0 will not apply any mask to the special bits (suid/guid/sticky bit)
  • The second 0 will apply no mask to the owner
  • The third 2 will mask/strip out (think subtract) write permission (2 in octal) for the group
  • The forth bit 2 will mask/strip out write for others

In practice umask values of 0, 2 and 7 are used:

  • 0 means 6 for files, and 7 for directories
  • 2 means 4 for files and 5 for directories
  • 7 means 0 for files and 0 for directories

The base /etc/bashrc and /etc/profile bash environment bootstrapping files contain entries for setting up default umask values.

Special permissions

Permission Octal File Dir
suid 4 run as owner -
sgid 2 run as group inherit group owner
sticky 1 - only delete if owner

suid

The running of processes as their original owner. Impersonation if you will. Known as suid. Take for example the /usr/bin/passwd program:

-rwsr-xr-x.   1 root root       27872 Feb  5  2016 passwd
   ^

Note the s (suid) bit. While passwd is owned and grouped by root, its runnable by average joe users under roots context, as if being run by the real root user.

Can be set with chmod:

chmod u+s file1
chmod 4500 file1
chmod 2500 file1
chmod 6444 file

sgid

Very useful for defining a group owner that gets inherited within a directory tree.

  • Imagine a /data/sales/ dir.
  • If a user mike creates a file (or dir) within /data/sales/ the user will be set to mike, and the group also set to mike.
  • In a group environment, such as /data/sales/, it would be more useful if the group was set to the sales group
  • The sgid special bit will propagate the group owner to new files or directories and is set with chmod g+s /data/sales/ or chmod 2770 /data/sales/

Sticky bit

Prevents the removal of files and/or directories unless that user is the owner. To set:

chmod +t mydir
chmod 1777 mydir

+t sets the sticky bit:

drw-rw---T. 1 ben  finance    0 May 14 19:06 mydir
         ^

Understanding ACLs

Several years later, in addition to the special bits, ACL (access control list) support was introduced to the kernel.

ACLs offer a few benefits over the simple UGO system:

  • more granular inheritable permission chains on specific directories
  • multiple owners

Scenario, under /data/ exists accounting/ (owned by root:accounting) and sales/ (owned by root:sales).

We want to grant the sales group rx permission to /data/accounting/.

This is not possible with the simple UGO model.

For directories:

  • setfacl -R -m g:sales:rx accounting to set ACLs on existing files and directories
  • setfacl -m d:g:sales:rx accounting to set the default ACL on new objects that are created
  • getfacl accounting to view ACLs

For files:

  • setfacl -m u:george:r myfile

To remove ACLs involves using the dash -

For example setfacl -m d:o::- secret-dir will strip all ACLs for others. Interestingly this (i.e. no permissions for others) will propagate down the tree to any new objects created within secret-dir, awesome!

ls will tack a + symbol to the end of the permission breakdown (e.g. drwxrwxr-x+), to indicate an ACL exists.

Configuring Networking

Network device naming

  • BIOS naming based on hardware properties such as em[1-N] for embedded NICs, p[slot_number]p[port_number]
  • udev naming ethX
  • Physical naming similar to BIOS naming with more variations
  • Logical naming such as vlan or alias
  • To get classical ethX naming, use biosdevname=0 and net.ifnames=0 GRUB boot options

Managing runtime network configuration with ip

ip is useful for showing live networking state.

ip addr help ip addr add 10.0.0.10/24 dev enpls0 ip link show

Storing network configuration persistently

Persistent network configuration is stored in /etc/sysconfig/network-scripts/, each NIC device is represented e.g. ifcfg-enp1s0

The Network Manager service is responsible for managing these network interface configs. An NM configuration is called a connection.

Frontends to the core NetworkManager service, include nmcli and nmtui. They depend on the NetworkManager daemon to be running.

nmcli

man nmcli-examples (example 10)

Bash tab completion rocks for CLI’s like nmcli, check its installed with rpm -qa bash-completion. With nmcli go nuts with double tabbing which will even sensibly dump out specific interface names, to figure out all the options it needs.

Commonly used options:

  • con-name for the profile label
  • ipv4.method for static vs DHCP
  • ipv4.addresses
  • ipv4.dns
  • ipv4.gateway
  • autoconnect

Hot tip: always specify a CIDR style subnet mask, as the default is 32!

To add a new connection:

nmcli connection add con-name limeleaf ifname enp1s0 type ethernet ip4.addresses 192.168.4.210/24 ipv4.gateway 192.168.4.2 ip4.addresses 1.2.3.4/24 ipv4.dns 8.8.8.8

To activate a connection profile (this will re-parse configuration even if the same connection is already active):

nmcli connection up enp1s0-profile

Verify connection status:

nmcli connection show

Modify an existing connection profile to define the DNS:

nmcli connection modify enp1s0-profile ipv4.dns 8.8.8.8

nmcli in the above will update /etc/resolv.conf

nmcli also features an interacive edit mode nmcli connection edit simoid-enp1s0, which will display a shell nmcli>

Routing and DNS

ip route show ip route del default via 192.168.4.1 ip route add default via 192.168.4.2

Using nmcli to set persistent routes (default gateway):

nmcli connection edit simoid-enp1s0
nmcli> set ipv4.gatway 192.168.122.1
nmcli> save
nmcli> quit
nmcli connection up simoid-enp1s0

Setting the hostname on RHEL is done with hostnamectl:

  • hostnamectl status
  • hostnamectl set-hostname host14.bencode.net

Managing Processes

  • In Linux everything is a process (including threads). Threads cannot be individually managed.
  • All processes are assigned a PID
  • Mother hening chores includes setting their scheduling priority and sending signals

Shell jobs

The concept of foreground and background shell processes.

Normally when running a shell command interactively, it is blocking (synchronous) with stdout and stdin wired to the terminal.

Trailing the command with an ampersand &, will unhook stdin and stdout, assign it a job number, and let it continue processing.

  • Example sleep 100 & will output the assigned job number and pid e.g. [3] 2970 = job 3, pid 2970
  • jobs will list all background jobs
  • fg 3 will foreground job 3
  • To background an active shell process Ctrl-Z to stop the job, and simply bg to background it.

ps

The way god reports on processes.

  • ps supports both BSD (naked options) and sys-v (hyphened options) styles, ps -L completely different meaning to ps L
  • ps aux overview of all processes
  • ps -fax process tree
  • ps -fU benjamin all processes owned by a user
  • ps -f --forest -C sshd show process tree only for the sshd process
  • ps L show all format specifiers available
  • ps -eo pid,ppid,user,cmd list processes using specific format specifiers

Memory usage

  • Linux tries to cache files for provide a fast experience. Often as a result, memory appears to over-saturated.
  • Swap provides a virtual (fake) memory address space, backing the memory by (much slower) disk if needed.

Use free to report on the memory situation e.g. free -m show memory units in mebibytes:

  • free truly un-utilised memory
  • available memory be used by buffers or cache that can be liberated immediately
  • If free memory is low and swap used, indicates the server is under memory pressure and could use more RAM

CPU load

Processes as placed into a run queue, which the kernel scheduler uses to allocate processes to CPU cores.

  • uptime to show load averages over 1, 5 and 15 minute spans
  • Load average is the average count of processes that are in a runnable or uninteruptable state.
  • lscpu for CPU meta, including number of CPU’s, sockets, cores per socket and threads per core.
  • uptime load is not normalised by the number of CPU cores (i.e. 1 on single core = 100%, but on a 4 core CPU = 25% load)

System activity with top

Keyboard options:

  • f select display fields
  • M,P,T sort on memory use, CPU or time
  • W save display settings
  • 1 show individual CPU cores
  • k to kill a PID
  • r to set nice level on a PID

Interpreting top by line:

  • 1 is just uptime
  • 2 is processes by categories: stopped = ctrl-z, zombie child processes that have lost their parent process and have become unmanagable.
  • 3 is CPU stats: us user space, sy system space, ni processes with changed niceness, id idle time, wa blocked on I/O, hi hardware interupts, st stolen time (zen virtualisation)
  • 4 for memory stats:

Sending signals to processes

Signals are a way of communicating with processes, even if they’re busily working away.

  • man 7 signal describes the classical signals such as SIGHUP (1), SIGKILL (9) and SIGTERM (15).
  • Signal handling very much depends on the program. Example, nginx will gracefully reparse config if it receives a SIGHUP with terminating active connections.
  • kill is used to send a signal to a PID
  • killall to send signals to all processes that match a search expression (e.g. killall -SIGTERM 'dd' to send SIGTERM to all dd processes)
  • pkill will send a signal based on a the text pattern of a several process attributes (e.g. pkill -signal 15 -U bob send SIGTERM to all of bob’s processes, pkill -signal 1 sshd send SIGHUP to the sshd process).

Priority and niceness

In a nutshell, the amount of priority the process scheduler will give to a process.

  • Nice values range from -20 to 19 (the lower the more priority, the higher the nicer a process is consider toward other processes)
  • Users can make their processes nicer (lower scheduler priority), but not more aggressive (i.e. higher priority)
  • Use the nice and renice commands to alter the priority of non-realtime processes
  • nice will spawn new processes with a nice preset e.g. nice -n -5 dd if=/dev/zero of=/dev/null
  • renice will alter the niceness of an existing process e.g. renice -n 10 -p 34627

In top:

  • the PR column is priority, the lower the higher priority. Priority rt or realtime is a special case, and is the supreme priority.
  • NI is nice level (-20 most aggressive, 19 nicest)

tuned profiles

tuned is a system performance optimiser service.

  • Make sure its running systemctl status tuned
  • tuned-adm is the CLI
  • tuned-adm list show available profiles
  • tuned-adm profile powersave to set the powersave profile
  • tuned-adm active show current profile

Managing Software

RPM and yum

RPM remains the package format of choice for hat-based distros. RPM facts:

  • its from the 90’s
  • its an archive packed by cpio includes a manifest, and list of dependencies
  • they can include scripts
  • RHEL 8 has the concept of protected base packages that can’t be removed (such as vi)

yum was built to be a friendly package frontend:

  • yum search nmap
  • yum install nmap
  • yum remove
  • yum update update all packages
  • yum update kernel update just the kernel package
  • yum provides */sepolicy a deeper search that scans files within each package
  • yum info nmap show the package manifest
  • yum list all
  • yum list installed
  • yum history list of recent package activity
  • yum history undo 4 undo transaction 4 in the above history list

Cool tip yumdownloader (in the yum-utils package) will download RPM to file system for inspection.

rpm queries

With yum, the older rpm CLI is used less directly these days. However RPMs are still managed by the same underlying accounting database as forever, which the rpm CLI exposes.

This is useful for querying, such as the specific files installed as part of a package, and so on.

  • rpm -qf /usr/bin/awk which package installed this file?
  • rpm -ql tmux list each file installed by the tmux package
  • rpm -qc openssh-server list the configuration files for a package
  • rpm -qp --scripts foo.rpm review the scriptlets (pre-install, post-install) of a standalone RPM

yum Groups

Chunks up software into broad categories.

  • yum group list
  • yum group list hidden
  • yum group info "System Tools"
  • yum group install --with-optional "Directory Client"

Repositories

New in RHEL 8 are AppStreams.

Defined by /etc/yum.repos.d/.

appstream.repo:

[appstream]
name=appstream
baseurl=file:///repo/AppStream
gpgcheck=0

base.repo:

[base]
name=base
baseurl=file:///repo/BaseOS
gpgcheck=0

To verify run yum repolist

Modules and Application Streams

New in RHEL 8, appstreams separate user (i.e. application) packages from core system (i.e. base) packages.

  • Application Streams come as either traditional RPMs or the new module format.
  • Modules (ex: php) themselves can in-turn contain streams (ex php:7.1, php:7.2).
  • Enabling a module stream (php:7.1) opens up access to its packages
  • Modules can have profiles (e.g. a minimal, devel)
  • Module streams support upgraded and downgrading between each other (php:7.1 > php:7.3 or php:8.0 > php:7.1)

Managed with yum:

  • yum module list
  • yum module provides httpd show the module that provides a paricular package
  • yum module info php specific module info
  • yum module info --profile php show the profiles of a specific module
  • yum module list php to list available modules
  • yum module install php:7.3 or yum install @php:7.3 will enable and install specific module stream
  • yum module install php:7.3/devel to install the module using a specific profile
  • yum module enable php:7.1 enables the module stream, without installing

Updates between module streams just works:

  • yum module install php:7.1
  • some time later yum module install php:7.3

Beware yum update will use enabled module streams (e.g. php:7.1 will not automatically be upgraded to php:7.3)

Red Hat Subscription Manager

The RHEL repositories require an active subscription.

  • subscription-manager register
  • subscription-manager attach --auto

Systemd

The init system. The kernel hands over to it, when its ready to bootstrap user space.

  • Managed items are called units (services, mounts, timers, sockets etc)
  • systemctl is the management CLI
  • systemctl -t help list of supported unit types
  • systemctl list-unit-files list each unit, its definition file and status
  • systemctl enabled vsftpd enable (auto start) service
  • systemctl start vsftpd start the service process

Modifying service configuration (see man systemd.service):

  • Default unit files: /usr/lib/systemd/system/
  • Custom unit files: /etc/systemd/system/
  • Runtime generated unit files: /run/systemd/
  • systemctl cat rsyslog.service dump unit configuration
  • systemctl edit unit.service will create overlay in /etc/systemd/system
  • systemctl show to dump available parameters that can be used in unit configs
  • systemctl daemon-reload after modifying unit files, often is necessary

When editing an overlay, can just add the extra options as they will be additive to the existing base configuration:

[Service]
Restart=on-failure
RestartSec=60

Scheduling Tasks

cron

cron, the classical scheduling daemon.

  • has no stdout
  • crontab -e to create user specific job
  • /etc/cron.d/ to create system wide job
  • /etc/cron.{hourly,daily,weekly,monthly} managed by anacron, for regular script execution
  • /etc/crontab (deprecated) was once used to configure jobs. crontab remains useful for specifying the environment for cron such as the SHELL.

cron time specification (man 5 crontab) example */10 4 11 12 1-5:

  • */10 every 10 minutes
  • 4 only on hour 4
  • 11 only on day 11
  • 12 only on month 12
  • 1-5 only on day of week 1-5

Example, write hello to syslog on minute 57, hour 20:

  • crontab -e
  • 57 20 * * * logger hello

anacron, runs commands periodically.

  • Unlike cron doesn’t assume the machine is running all day everyday.
  • Configured by /etc/anacrontab

at

at unlike cron is used for one-off jobs.

  • make sure atd daemon is running
  • provide its own interactive shell to take job specifications
  • atq to list at queue
  • atrm to remove at job

Example:

  • at teatime
  • logger have a cup of tea

systemd Timers

cron is still the gold standard, however this is still a viable option.

  • man 5 systemd.timer and man 5 systemd.time for time specification
  • ls /usr/lib/systemd/system/*.timer to list timers
[Unit]
Description=Discard unused blocks once a week
Documentation=man:fstrim
ConditionVirtualization=!container

[Timer]
OnCalendar=weekly
AccuracySec=1h
Persistent=true
RandomizedDelaySec=6000

[Install]
WantedBy=timers.target

Managing temporary files

A common way to manage (create, delete) temporary files. See man tmpfiles.d.

  • /usr/lib/tmpfiles.d/ setting files
  • For example /usr/lib/tmpfiles.d/tmp.conf contain settings for automatic tmp files cleanup
  • systemd-tmpfiles-clean.timer unit can be configured to automatically clean up temporary files (by triggering systemd-tmpfiles-clean.service which in turn runs systemd-tmpfiles --clean).
  • If you want to make modifications, copy conf file from /usr/lib/tmpfiles.d/ to /etc/tmpfiles.d/ and edit it there.
  • Run systemd-tmpfiles --clean /etc/tmpfiles.d/tmp.conf manually to parse and test configuration changes.
  • To register a new custom tmpfiles configuration systemd-tmpfiles --create /etc/tmpfiles.d/foo.conf

Logging

Rsyslog

The rocket-fast Syslog Server

rsyslogd is the defacto syslogd used by most distros. It monitors configurable sources (e.g. /dev/log) and writes to configurable sinks (e.g. in /var/log/)

  • A daemon managed by the rsyslogd.service unit
  • Configured by /etc/rsyslog.conf
  • Snap-in configs in /etc/rsyslog.d/
  • Each logger rule line is made up of 3 elements; facility ({auth,authpriv,cron,daemon,kern,lpr,mail,mark,news,security,syslog,user,uucp,local{0-7}}), severity ({debug,info,notice,warn,err,crit,alert,emerg,panic}) and an action (regular file, database table, remote machine, a tty, discard, and more).
  • For services that don’t have a specific facilty, use local{0-7}
  • You can use the logger CLI to write messages to rsyslogd manually

Sample rules from /etc/rsyslog.conf:

*.info;mail.none;authpriv.none;cron.none       /var/log/messages

Log everything INFO or higher, except mail/authpriv/cron to /var/log/messages

mail.*                                         -/var/log/maillog

Notice the - before the filename. This tells rsyslog to buffer writes.

Systemd Journal

Being systemd, it invented its own logger, called journald. By default journald is in-memory, but it sinks logs to /dev/log which rsyslog listens to.

While rsyslogd, depending on it config, will likely perist journald logs to /var/log/.

  • By default writes journal to /run/log/journal, which is cleared across reboots.
  • It’s possible to persist the systemd journal logs. mkdir /var/log/journal/ and restart the journald.service unit.
  • Update /etc/systemd/journald.conf, set Storage to one of {persistent,volatile,auto} (auto will use /var/log/journal/ only if it exists)
  • Journal logs are propagated to rsyslogd using the imjournal input module
  • The journalctl CLI is the frontend for querying journal logs.
  • Use tab completion to build out filters, such as journalctl UNIT=dbus.service

Logrotate

Used to roll up (rotate) logs.

  • Its started through cron.daily
  • Configured by /etc/logrorate.conf or /etc/logrotate.d/

Managing Storage

Disk layout

This is driven by the underlying management scheme; either BIOS based or UEFI based.

BIOS, designed in the early 80’s, uses a MBR (master boot record) to define the partition layout of the system. With a cap of 64 bytes, can support upto 4 partitions. The 4 partition limitation was later overcome, by leveraging logical partitions within an extended partition.

UEFI (Universal Extended Firmware Interface), uses GPT (GUID partition table), supports upto 128 partitions.

Useful commands:

  • lsblk dumps all block devices attached to a system
  • parted is the preferred partition management program
  • All block devices are represented in /dev/ e.g. /dev/vda1
  • /proc/partitions

Rando cool

  • ctrl+l = clear terminal
  • ls -d don’t show contents of directories
  • \ls un-alias a command, by preceding it with a backslash \
  • alias to display evaluated bash aliases
  • tar command options are not prefixed with a hypen -
  • tac is the inverse program of cat
  • chvt jumps between TTY e.g. chvt 3
  • ssh-keygen supports a number of ciphers, set using [-t dsa | ecdsa | ed25519 | rsa], RSA by default
  • yum history full journal of package installs
  • yumdownloader is included in yum-utils lets you download packages to local file system
  • run-parts which comes as part of the cron ecosystem, is a script that runs all executables in a directory.