Here I aim to cover a set of common administration tasks. Things like, the hostname, system logs, what users are currently logged in, physical devices that are connected, logical volumes, file system and inode allocation, attached network interfaces and their addressing, processes and daemons currently running, kernel verison, local users and groups, installed packages, remote mounts, network shares, system uptime, bread and butter OS stats (CPU, IO, network, memory).

Booting⌗

shutdown -r +5 System going down for a reboot  #wall broadcast msg
shutdown -c  #cancel reboot
shutdown -r 00:00  #schedule for midnight
shutdown -h +5  #halt system in 5 mins
shutdown -h now

Alternatively, just use systemd:

systemctl halt
systemctl shutdown
systemctl poweroff:29

Runlevels (legacy):

init 0  #shutdown
init 6  #reboot

Targets⌗

A systemd target is simply a collection of units. Several types of units are possible systemctl -t help:

service
socket
busname
target
device
mount
automount
swap
timer
path
slice
scope

Unit configuration files live in /usr/lib/systemd/system e.g /usr/lib/systemd/system/sshd.service, and define the service unit; pre and post execution commands, core runtime command, dependencies on other targets and/or units, targets that include this unit (e.g. multi-user.target) and so on. Most common targets:

multi-user.target - a multi user, text based computing environment
graphical.target -
emergency.target - root shell, read-only file system
rescue.target - a bare bones troubleshooting environment

The isolate command allows switching between targets (not all targets support “isolation”):

systemctl isolate multi-user.target

The default target can be altered with set-default (symlink housekeeping):

systemctl set-default graphical.target

The target used at boot time can be specified by altering the GRUB bootloader. Interupt the boot sequence, and use e to modify the GRUB script, find the kernel init string (starting with linux16) and add, for example, systemd.unit=rescue.target to the end. C X to continue boot.

Interrupting Boot⌗

By appending rd.break to the kernel init string in the GRUB boot loader (press esc to present the boot menu, select the target kernel, and then e to modify it), will inject us into the initramfs emergency mode shell; a barebones mini environment. C X to continue boot. /sysroot contains the eventual root mount point, that initramfs has prepared. Since we have interupted the init process, /sysroot has been “re-rooted” yet.

mount -oremount, rw /sysroot   #get r/w perms
chroot /sysroot   #chroot jail
passwd root   #yup
touch /.autorelabel   #selinux relabelling during next boot
exit   #exit chroot jail
exit   #exit initramfs shell

SELinux contexts will be lost. Some options include creating .autorelabel in root (i.e. touch /.autorelabel). SELinux will relabel everything.

Process Management⌗

Listing⌗

ps aux | gnome  #manually grep ps output
pgrep gnome  #grep for processes
pgrep gnome -l  #show process names
pgrep -u ben -l vi  #processes by user
pgrep -v -u root  #v flag inverts, so all processes not owned by root

Killing⌗

pkill httpd   #kill 15 all processes that grep to httpd
kill -l   #list signal table
pkill -SIGTERM httpd   #explicit signal

Important signals:

1 SIGHUP hang up, similar to closing a terminal window
2 SIGINT interupt, similar to ^C
3 SIGQUIT, request process to quit
9 SIGKILL, brutally murder the process immediately
15 SIGTERM, gracefully terminate
18 SIGCONT, continue a stopped process
19 SIGSTOP, suspend process
20 SIGTSTP, optional suspend

Kill by TTY (terminal):

$ w
03:03:21 up 41 min,  4 users,  load average: 0.05, 0.04, 0.05
USER     TTY      FROM             LOGIN@   IDLE   JCPU   PCPU WHAT
ben      :0       :0               02:26   ?xdm?  36.10s  0.11s gdm-session-worker [pam/gdm-password]
ben      pts/0    :0               02:26    1.00s  0.09s  2.15s /usr/libexec/gnome-terminal-server
john     pts/2    localhost        03:02    9.00s  0.05s  0.03s vim test
$ pkill -t pts/2
$ pkill -u john sshd

Jobs and Suspending⌗

Work can be put to the background by adding an ampersand & to the end
Job ids should always be prefixed with a percent %, to make it clear a job is being referred to.

$ (while true; do echo -n “my program” » ~/output.txt; sleep 1; done) & [1] 5398

$ jobs [1]+ Running ( while true; do echo -n “my program” » ~/output.txt; sleep 1; done ) &

$ kill -SIGSTOP %1 $ jobs [1]+ Stopped ( while true; do echo -n “my program” » ~/output.txt; sleep 1; done ) &

$ kill -SIGCONT %1 $ jobs [1]+ Running ( while true; do echo -n “my program” » ~/output.txt; sleep 1; done ) &

$ kill 15 %1 $ jobs [1]+ Terminated ( while true; do echo -n “my program” » ~/output.txt; sleep 1; done )

Priority and Nice⌗

Nice levels range from -20 to 19, -20 representing the most favourable, and 19 the least favourable.

$ ps aux   #BSD style, processes for all users, is user oriented format, a = BSD tsyle, x = all processes, u = user oriented format
$ ps axo pid,comm,nice   #the PID, command and nice level
$ ps -u root   #POSIX style all procs owned by root

Nice experiment:

$ dd if=/dev/zero of=~/tmp/bigfile bs=1M count=1024
$ time nice -n 19 tar -cvf bigfile.tar bigfile
bigfile

real  0m3.022s
user  0m0.018s
sys 0m1.185s

# time nice -n -20 tar -cvf bigfile.tar bigfile
bigfile

real  0m2.909s
user  0m0.018s
sys 0m1.295s

Nice experiment two:

Start Apache with systemd:

# systemctl start httpd
# ps axo comm,pid,nice | grep httpd
httpd            3874   0
httpd            3940   0
httpd            3941   0
httpd            3942   0
httpd            3943   0
httpd            3944   0

Kill processes and restart them with nice:

# systemctl stop httpd
# nice -n 10 httpd
# ps axo comm,pid,nice | grep httpd
httpd            4344  10
httpd            4345  10
httpd            4346  10
httpd            4347  10
httpd            4348  10
httpd            4349  10

Renice:

# renice -n 3 $(pgrep httpd)
4344 (process ID) old priority 10, new priority 3
4345 (process ID) old priority 10, new priority 3
4346 (process ID) old priority 10, new priority 3
4347 (process ID) old priority 10, new priority 3
4348 (process ID) old priority 10, new priority 3
4349 (process ID) old priority 10, new priority 3

# ps axo comm,pid,nice | grep httpd
httpd            4344   3
httpd            4345   3
httpd            4346   3
httpd            4347   3
httpd            4348   3
httpd            4349   3

Load Averages and Activity⌗

The w program, not only shows users that are currently logged into the system, but CPU load averages across 1, 5 and 15 minute time spans.

$ w
21:46:11 up  1:56,  1 user,  load average: 0.43, 0.53, 0.75
USER     TTY        LOGIN@   IDLE   JCPU   PCPU WHAT
ben      tty2      19:49    1:56m 51:16   1.02s /opt/google/chrome/chrome

A personal favourite when it comes to process monitoring, is top, a curses based CLI that dynamically refreshes based on activity taking place. Useful shortcuts:

m toggle memory display modes in the HUD
t toggle tasks display modes in the HUD
l toggle uptime (first line) display in the HUD
V forest view (parent/child)
H thread view (as opposed to process view)
B bold key fields
k kill
r renice
z toggle color/mono display
L locate/search

Launch options:

top -n 2 start, refresh twice, then terminate.
top -d 2 start, setting the refresh polling interval to 2 seconds

System Logging⌗

Traditionally was powered by the rsyslogd daemon (with logs typically stored in /var/log), however with RHEL 7, systemd’s log subsystem journald has been included.

journald by default temporarily stores its state in /run/log/journal, which is not peristent across system reboots. To change this default behaviour, /etc/systemd/journald.conf and set Storage=persistent. Then a reload on the systemd daemon. Logs will now be stored in /var/log/journal.

logrotate is

Users⌗

Files that relate to local security⌗

/etc/passwd⌗

The local /etc/passwd (“pass wood”) user database, defines all local accounts by name, an x for the password (which lives over in /etc/shadow), uid and gid, friendly description, home directly and the default shell (e.g. /sbin/nologin for service accounts, interactive shells such as /bin/bash for humans).

$ tail -n 2 /etc/passwd
ben:x:1000:1000:Ben Simmonds:/home/ben:/bin/bash
tomcat:x:91:91:Apache Tomcat:/usr/share/tomcat:/bin/nologin

/etc/shadow⌗

The password hashes /etc/shadow:

$ sudo tail -n 2 /etc/shadow
ben:$6$dGHFbUzk$6wMNejH9ue/n6riVfyDm8KgLCZQ5FWpzujU0yu6FICg09ihRvRlFrwONcTIuJfuA4ZdbjX4d3litJ9PcjDTYg.:16871:0:99999:7:::
tomcat:!!:16782::::::

/etc/group⌗

Defines all user to group mappings.

$ tail -n 4 /etc/group
ben:x:1000:ben
lirc:x:977:
tomcat:x:91:
postgres:x:26:root,ben

/etc/skel⌗

The skeleton directory /etc/skel is useful for define the default “blue print” of files and/or directories, which will be automatically cloned into new users home directories.

Live user information⌗

What user am I logged on as, and what is the assigned group id?⌗

$ whoami
ben

$ id
uid=1000(ben) gid=1000(ben) groups=1000(ben),10(wheel),26(postgres),100(users),974(pkg-build) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

What users are on this host?⌗

$ w
 20:39:41 up 48 days,  5:34,  3 users,  load average: 0.00, 0.11, 0.22
USER     TTY        LOGIN@   IDLE   JCPU   PCPU WHAT
ben      tty2      14Aug16 48days  2:17m 23.63s /opt/google/chrome/chrome
ben      tty3      20:37    1:47   0.03s  0.03s -bash
george   pts/12    20:39    5.00s  0.01s  0.01s -bash

Alternatively who gives us a more succinct output:

$ who
ben      tty2         2016-08-14 15:08 (:0)
ben      tty3         2016-10-01 20:37
george   pts/12       2016-10-01 20:39 (127.0.0.1)

Local user management⌗

New account⌗

useradd -m -d /home/schnerg -u 1501 -g 66 -s /bin/bash

Set a password (`/etc/shadow`)⌗

passwd schnerg

Lock an account⌗

usermod -L schnerg

Interestingly the lock, will prefix the hash in /etc/shadow with a bang !:

schnerg:!$6$dGHFbUzk$6wMNejH9ue/n6riVfyDm8KgLCZQ5FWpzujU0yu6FICg09ihRvRlFrwONcTIuJfuA4ZdbjX4d3litJ9PcjDTYg.:16871:0:99999:7:::

Unlock the account⌗

usermod -U schnerg

Remove the user⌗

userdel schnerg

Useful options are --remove-all-files and --backup.

Bulk user creation⌗

User accounts can be created in bulk using the newusers program, which takes in an input of passwd formatted (e.g. colon separated) lines.

Transfer all ownership for userX to userY⌗

Transfer file ownership to another user:

find / -uid 1003 -exec chown -v 1010:1010 {} \;

Processes⌗

Process ID (or pid) 1 is always the init system (e.g. sysv, upstart, systemd). The mother process the kernel hands over to initialise userspace.

The niceness of a process, indicates how “nice” a process is to other processes. The higher, the nicer (i.e. low priority). The lower, the more anti-social (think loney child) to fellow processes (i.e. high priority).

To communicate with processes, Linux supports both POSIX standard signals and real-time signals.

A signal is an asynchronous notification sent to a process to notify it of an event.

Signals are supported by system calls; kill(2) (signals target process/es), killpg(2) (signals all members of a process group) and tgkill(2) (signals a thread within a process).

For a listing of signals kill -l or man 7 signal:

Signal	Value	Action	Comment
SIGHUP	1	Term	Hangup detected on controlling terminal or death of controlling process
SIGINT	2	Term	Interrupt from keyboard
SIGQUIT	3	Core	Quit from keyboard
SIGILL	4	Core	Illegal Instruction
SIGABRT	6	Core	Abort signal from abort(3)
SIGFPE	8	Core	Floating point exception
SIGKILL	9	Term	Kill signal
SIGSEGV	11	Core	Invalid memory reference
SIGPIPE	13	Term	Broken pipe: write to pipe with no readers
SIGALRM	14	Term	Timer signal from alarm(2)
SIGTERM	15	Term	Termination signal
SIGUSR1	30,10,16	Term	User-defined signal 1
SIGUSR2	31,12,17	Term	User-defined signal 2
SIGCHLD	20,17,18	Ign	Child stopped or terminated
SIGCONT	19,18,25	Cont	Continue if stopped
SIGSTOP	17,19,23	Stop	Stop process
SIGTSTP	18,20,24	Stop	Stop typed at terminal
SIGTTIN	21,21,26	Stop	Terminal input for background process
SIGTTOU	22,22,27	Stop	Terminal output for background process

What processes are running, and how expensive are they?

$ top
top - 20:48:29 up 48 days,  5:43,  3 users,  load average: 0.26, 0.21, 0.20
Tasks: 330 total,   1 running, 329 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1.3 us,  0.3 sy,  0.0 ni, 98.2 id,  0.2 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  7997192 total,   320028 free,  5251400 used,  2425764 buff/cache
KiB Swap: 10235900 total,  8807296 free,  1428604 used.  1358536 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 1715 ben       20   0 1900716 193724  35100 S   3.7  2.4  50:36.03 gnome-shell
 1592 ben       20   0  802420  98724  75620 S   3.3  1.2  39:44.34 Xorg
 2946 ben       20   0  870620  49236  16468 S   3.0  0.6   2:10.96 gnome-terminal
27607 ben       20   0 1257740 104748  42508 S   2.7  1.3   0:06.78 sublime_text
 5714 ben       20   0 6432148 821476  22692 S   0.7 10.3  19:05.09 java
    1 root      20   0  196972   6196   4108 S   0.0  0.1   0:05.43 systemd

Search for processes by name

$ ps -ef | grep -e 'spotify'
ben      12561     1  0 Sep25 tty2     00:02:16 /usr/lib64/spotify-client/spotify
ben      12564 12561  0 Sep25 tty2     00:00:00 /usr/lib64/spotify-client/spotify --type=zygote
ben      12577 12561  0 Sep25 tty2     00:01:14 /usr/lib64/spotify-client/spotify --type=gpu-process --channel=123455.0.371806952
ben      12602 12564  0 Sep25 tty2     00:04:53 /usr/lib64/spotify-client/spotify --type=renderer --disable-pinch

Kill parent spotify process (PID 12561)

sending it the aggressive (murder the siht out of) SIGKILL (9) signal, as opposed to the default SIGTERM (15) signal. For a cheatsheet of available signals kill -l.

Kill (brutally murder) the parent spotify process (and hence it children) by sending it SIGKILL (9):

$ kill -s 9 12561

Kill a process by name with killall:

$ sudo killall cupsd

Kill processes based on name or other attributes with pkill, for example, kill all processes started by user ben:

$ sudo pkill -u ben

You may want to verify the processes identified by the pkill filter you define, before pulling the trigger, thats where pgrep comes in:

$ pgrep -u ben
631
1207
1568
1576
1586
1590
1592
1610
...

Network⌗

What network activity is taking place?

netstat is a swiss army knife networking. Some useful switches:

-t TCP
-u UDP
-p show the process attached to ports
-l listening ports
-n show them numerically (i.e. 80 not http)

Using all the above (unless run as superuser, PID/program information is masked for processes not owned):

$ sudo netstat -tupln
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:33541         0.0.0.0:*               LISTEN      2338/python2
tcp        0      0 127.0.0.1:42381         0.0.0.0:*               LISTEN      24420/python2
tcp        0      0 192.168.122.1:53        0.0.0.0:*               LISTEN      1272/dnsmasq
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1057/sshd
tcp        0      0 127.0.0.1:631           0.0.0.0:*               LISTEN      1905/cupsd
tcp        0      0 127.0.0.1:42331         0.0.0.0:*               LISTEN      3451/python2
tcp6       0      0 :::3689                 :::*                    LISTEN      16582/rhythmbox
tcp6       0      0 :::22                   :::*                    LISTEN      1057/sshd
tcp6       0      0 ::1:631                 :::*                    LISTEN      1905/cupsd
udp        0      0 127.0.0.1:323           0.0.0.0:*                           893/chronyd
udp        0      0 0.0.0.0:41597           0.0.0.0:*                           914/avahi-daemon: r
udp        0      0 0.0.0.0:5353            0.0.0.0:*                           4549/chrome
udp        0      0 0.0.0.0:5353            0.0.0.0:*                           4549/chrome
udp        0      0 0.0.0.0:5353            0.0.0.0:*                           914/avahi-daemon: r
udp        0      0 0.0.0.0:15068           0.0.0.0:*                           24947/dhclient
udp        0      0 192.168.122.1:53        0.0.0.0:*                           1272/dnsmasq
udp        0      0 0.0.0.0:67              0.0.0.0:*                           1272/dnsmasq
udp        0      0 0.0.0.0:68              0.0.0.0:*                           24947/dhclient
udp6       0      0 ::1:323                 :::*                                893/chronyd
udp6       0      0 :::46187                :::*                                914/avahi-daemon: r
udp6       0      0 :::5353                 :::*                                914/avahi-daemon: r
udp6       0      0 :::31564                :::*                                24947/dhclient

File System⌗

Everything in UNIX is a file. Whether it be a regular file, directory, block special file, character special file, a library, a stream or a network file (Internet socket, NFS file or UNIX domain socket) to name a few.

The lsof utility reports on file system activity being managed by the kernel. Kudos to tutorialLinux for these lsof tips, and the numerous others.

Which processes have this file open?

lsof /usr/lib64/libpthread-2.23.so

Which files does process X have open?

lsof -p 1
lsof -p $(pgrep systemd | head -n 1)

Where is the binary for this process?

lsof -p 24855 | grep bin

Which shared libraries is this program using?

lsof -p 24855 | grep .so

Where is this process logging to?

lsof -p 24855 | grep log

Which processes are using a particular library (e.g. a lib with vunerabilities)?

lsof grep libfoo.so

Which files does user XYZ have open?

lsof -u schnerg
lsof -u schnerg -i #network only

Which process is listening on port 5432 (or using protocol y)?

lsof -i :5432
lsof -i tcp

Linux Ops Guide