Systemd and SysVinit#

Unix-based systems used SysV-style start-stop-scripts to control services and so did most Linux distributions. Systemd has become the default for most Linux distributions since 2011, and it is the default for most recent versions of Ubuntu. This chapter covers primarly systemd or how to convert to systemd as well as how to use systemd to manage services.

The original SysVinit was a replacement for initscripts and was used on most Linux distributions. Systemd was started by Lennart Poettering as a replacement for initd and SysVinit to manage services like on other operating systems as Microsoft Windows for example. With a dependency graph systemd can make your system more reliable and stable as you define the dependencies between services and the order in which they are started.

Available unit types#

Systemd defines every service as a unit. A unit is a collection of configuration files and a set of binaries that are used to start and stop the service. The unit name is the name of the service. The example below shows how to list all available units on your system as they depend on the systemd version on your system.

$ systemctl -t help
Available unit types:
service
mount
swap
socket
target
device
automount
timer
path
slice
scope

By knowing the unit types you can list all available units of that type as we can see below where the command systemctl --type=service lists all available services.

$ systemctl --type=service
  UNIT                        LOAD   ACTIVE SUB     DESCRIPTION
  abrt-journal-core.service   loaded active running Creates ABRT problems from …
  abrt-oops.service           loaded active running ABRT kernel log watcher
  abrt-xorg.service           loaded active running ABRT Xorg log watcher
  abrtd.service               loaded active running ABRT Automated Bug Reportin…
  accounts-daemon.service     loaded active running Accounts Service
  alsa-state.service          loaded active running Manage Sound Card State (re…
  atd.service                 loaded active running Deferred execution scheduler
  auditd.service              loaded active running Security Auditing Service
  avahi-daemon.service        loaded active running Avahi mDNS/DNS-SD Stack

Systemd also can act like a scheduler and run services on a specific time schedule, but also verifies if all requirements are satisfied before execution the schedule. In a later section we will see how to use systemd to manage timers.

$ systemctl --type=timer
  UNIT                        LOAD   ACTIVE SUB     DESCRIPTION
  dnf-makecache.timer         loaded active waiting dnf makecache --timer
  fstrim.timer                loaded active waiting Discard unused blocks once …
  logrotate.timer             loaded active waiting Daily rotation of log files
  mlocate-updatedb.timer      loaded active waiting Updates mlocate database ev…
  rdiff-backup.timer          loaded active running Execute rdiff-backup
  snapper-cleanup.timer       loaded active waiting Daily Cleanup of Snapper Sn…
  snapper-timeline.timer      loaded active waiting Timeline of Snapper Snapsho…
  sysstat-collect.timer       loaded active waiting Run system activity account…
  sysstat-summary.timer       loaded active waiting Generate summary of yesterd…
  systemd-tmpfiles-clean.tim… loaded active waiting Daily Cleanup of Temporary …
  unbound-anchor.timer        loaded active waiting daily update of the root tr…

Unit type mount is another example and can be created for mounting filesystems, but systemd itself generates during the boot procedure of the system all required unit files based on the content of /etc/fstab. This makes the transition smooth for a lot people working with Linux.

$ systemctl --type=mount
  UNIT                        LOAD   ACTIVE SUB     DESCRIPTION
  -.mount                     loaded active mounted Root Mount
  boot-efi.mount              loaded active mounted /boot/efi
  boot.mount                  loaded active mounted /boot
  dev-hugepages.mount         loaded active mounted Huge Pages File System
  dev-mqueue.mount            loaded active mounted POSIX Message Queue File Sy…
  home.mount                  loaded active mounted /home
  run-user-1000-doc.mount     loaded active mounted /run/user/1000/doc
  run-user-1000-gvfs.mount    loaded active mounted /run/user/1000/gvfs
  run-user-1000.mount         loaded active mounted /run/user/1000
  srv.mount                   loaded active mounted /srv
  sys-fs-fuse-connections.mo… loaded active mounted FUSE Control File System
  sys-kernel-config.mount     loaded active mounted Kernel Configuration File S…
  sys-kernel-debug.mount      loaded active mounted Kernel Debug File System
  sys-kernel-tracing.mount    loaded active mounted Kernel Trace File System
  tmp.mount                   loaded active mounted Temporary Directory /tmp
  var-lib-docker.mount        loaded active mounted /var/lib/docker
  var-lib-libvirt.mount       loaded active mounted /var/lib/libvirt
  var-lib-nfs-rpc_pipefs.mou… loaded active mounted RPC Pipe File System

Managing services#

One of the benefits of systemd is that it can monitor the state of services and make sure that they are running as expected. This is done by using the systemctl command. In the section Available unit types it was shown how to list all services on the system. The example below shows how to request the status of a service, but also if it is enabled or not to be started on boot.

$ systemctl status sshd.service
● sshd.service - OpenSSH server daemon
    Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled; vendor preset: disabled)
    Active: active (running) since Sun 2022-04-17 09:36:55 CEST; 20s ago
      Docs: man:sshd(8)
            man:sshd_config(5)
  Main PID: 663271 (sshd)
      Tasks: 1 (limit: 18476)
    Memory: 2.1M
        CPU: 14ms
    CGroup: /system.slice/sshd.service
            └─663271 "sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups"

apr 17 09:36:55 server01 systemd[1]: Starting OpenSSH server daemon...
apr 17 09:36:55 server01 sshd[663271]: Server listening on 0.0.0.0 port 22.
apr 17 09:36:55 server01 sshd[663271]: Server listening on :: port 22.
apr 17 09:36:55 server01 systemd[1]: Started OpenSSH server daemon.

Note

By default systemd will assume the unit type service if no unit type is specified. In the case for the example above systemctl status sshd.service and systemctl status sshd are the same.

Enabling the sshd service with command systemctl enable <service name> will tell systemd to start the service when the system boots, but the service isn’t automatically started. To start the service we need to use the systemctl start sshd command.

$ systemctl enable sshd

Disabling the sshd service with command systemctl diable <service name> has the same behavior as systemctl enable <service name> and will only make sure the services isn’t started when the system boots.

$ systemctl disable sshd

Note

The --now option is used to start or stop the service immediately after enabling or disabling it. This option is available since Red Hat Enterprise Linux 7.2 and CentOS 7.2. Older versions require two commands to both enable and start a service.

Starting the sshd service with command systemctl start <service name> will start the service immediately. The service doesn’t have to be enabled for this and when the system system reboots the service will be stopped automatically, but not started if it isn’t enabled.

$ systemctl start sshd

The same rules as with starting a service also apply to stopping a service. This makes troubleshooting easier as it is easy to see if the service is running or not and should be running or not.

$ systemctl stop sshd

Like most SysVinit scripts systemd also has the option to restart a service by stopping and starting if for you. This is done by using the systemctl restart <service name> command.

$ systemctl restart sshd
$ systemctl reload sshd

With the command systemctl is-active <service name> we can check if the service is running or not as systemd will return active if the service is running and inactive if it isn’t. This make scripting and automation easier.

$ systemctl is-active sshd
active

The same rules apply to verify if a service is enabled or not as well with the command systemctl is-enabled <service name>.

$ systemctl is-enabled sshd
enabled

List active services#

$ systemctl list-units --type=service
  UNIT                        LOAD   ACTIVE SUB     DESCRIPTION
  abrt-journal-core.service   loaded active running Creates ABRT problems from …
  abrt-oops.service           loaded active running ABRT kernel log watcher
  abrt-xorg.service           loaded active running ABRT Xorg log watcher
  abrtd.service               loaded active running ABRT Automated Bug Reportin…
  accounts-daemon.service     loaded active running Accounts Service
  alsa-state.service          loaded active running Manage Sound Card State (re…
  atd.service                 loaded active running Deferred execution scheduler
  auditd.service              loaded active running Security Auditing Service
  avahi-daemon.service        loaded active running Avahi mDNS/DNS-SD Stack
$ systemctl list-units --type=service --all
  UNIT                        LOAD   ACTIVE SUB     DESCRIPTION
  abrt-journal-core.service loaded    active   running Creates ABRT problems fr…
  abrt-oops.service         loaded    active   running ABRT kernel log watcher
  abrt-vmcore.service       loaded    inactive dead    Harvest vmcores for ABRT
  abrt-xorg.service         loaded    active   running ABRT Xorg log watcher
  abrtd.service             loaded    active   running ABRT Automated Bug Repor…
  accounts-daemon.service   loaded    active   running Accounts Service
  alsa-restore.service      loaded    inactive dead    Save/Restore Sound Card …
  alsa-state.service        loaded    active   running Manage Sound Card State …
● apparmor.service          not-found inactive dead    apparmor.service
  atd.service               loaded    active   running Deferred execution sched…
  auditd.service            loaded    active   running Security Auditing Service
  auth-rpcgss-module.servi… loaded    inactive dead    Kernel Module supporting…
● auto-cpufreq.service      not-found inactive dead    auto-cpufreq.service
● autofs.service            not-found inactive dead    autofs.service
  avahi-daemon.service      loaded    active   running Avahi mDNS/DNS-SD Stack

List enabled services#

$ systemctl list-unit-files --type=service
  UNIT FILE                                          STATE           VENDOR PRESET
  abrt-journal-core.service                          enabled         enabled
  abrt-oops.service                                  enabled         enabled
  abrt-pstoreoops.service                            disabled        disabled
  abrt-vmcore.service                                enabled         enabled
  abrt-xorg.service                                  enabled         enabled
  abrtd.service                                      enabled         enabled
  accounts-daemon.service                            enabled         enabled
  alsa-restore.service                               static          -
  alsa-state.service                                 static          -
  anaconda-direct.service                            static          -
  anaconda-fips.service                              static          -
  anaconda-nm-config.service                         static          -
  anaconda-noshell.service                           static          -
  anaconda-pre.service                               static          -
  [email protected]                            static          -
  anaconda-sshd.service                              static          -
  [email protected]                             static          -
  anaconda.service                                   static          -
  arp-ethers.service                                 disabled        disabled
  atd.service                                        enabled         enabled
  auditd.service                                     enabled         enabled
  auth-rpcgss-module.service                         static          -
  [email protected]                                    alias           -
  avahi-daemon.service                               enabled         enabled
$ systemctl --failed --type=service
  UNIT                         LOAD   ACTIVE SUB    DESCRIPTION

Dependencies#

$ systemctl list-dependencies sshd.service
sshd.service
● ├─system.slice
○ ├─sshd-keygen.target
○ │ ├─[email protected]
○ │ ├─[email protected]
○ │ └─[email protected]
● └─sysinit.target
●   ├─dev-hugepages.mount
●   ├─dev-mqueue.mount
●   ├─dracut-shutdown.service
○   ├─fedora-import-state.service
○   ├─iscsi.service
●   ├─kmod-static-nodes.service
○   ├─ldconfig.service
○   ├─lvm2-lvmetad.socket
$ systemctl list-dependencies sshd.service --reverse
sshd.service

Masking services#

With systemd it is possible to define the state of a service by enabling or disabling it. It still allows the service to be started or enabled even. Another option exists and systemd allows a service to be masked. This means that the service is a symbolic link to /dev/null and stops systemd from starting it.

$ systemctl mask iptables
Created symlink from /etc/systemd/system/iptables.service to /dev/null.

Masked services can also be unmasked so they can be enabled and started again.

$ systemctl unmask iptables

Note

The main purpose of masking services is to prevent administrative errors when competing services like iptables and firewalld are installed on the same system.

Managing bootlevels#

$ systemctl list-dependencies graphical.target | grep target
graphical.target
● └─multi-user.target
●   ├─basic.target
●   │ ├─paths.target
●   │ ├─slices.target
●   │ ├─sockets.target
●   │ ├─sysinit.target
●   │ │ ├─cryptsetup.target
●   │ │ ├─local-fs.target
●   │ │ ├─swap.target
●   │ │ └─veritysetup.target
●   │ └─timers.target
●   ├─getty.target
●   ├─nfs-client.target
●   │ └─remote-fs-pre.target
●   └─remote-fs.target
●     └─nfs-client.target
●       └─remote-fs-pre.target
$ systemctl list-units --type=target --all
$ systemctl list-unit-files --type=target --all
$ systemctl isolate graphical.target
$ systemctl get-default
graphical.target
$ systemctl set-default multi-user.target

To be used with Grub:

systemd.unit=rescue.target (emergency.target)

Managing temporary files#

A generic mechanism for creating, adjusting, and deleting temporary files is provided by systemd-tmpfiles. It is mainly used in combinantion of volatile and temporary files and directories located under /run/, /tmp/, and /var/tmp/ for example. A standard set of rules is shipped with the systemd-tmpfiles package, but it can be customized to suit your needs by adding additional configuration files under /etc/tmpfiles.d/. A full list of locations with configuration files is shown in the table below.

Location

Description

/etc/tmpfiles.d/*.conf

Manual configuration

/run/tmpfiles.d/*.conf

Generated configuration

/usr/lib/tmpfiles.d/*.conf

System configuration

In unit file /usr/lib/systemd/system/systemd-tmpfiles-clean.service, shown below, it is defined which command is runned to clean up the temporary files. Secondly, it is defined when the unit file has to be executed and that is right after both dependencies local-fs and time-set are met, but also just before the system shutdown. If required the service can also be executed manually by running systemctl start systemd-tmpfiles-clean.service.

#  SPDX-License-Identifier: LGPL-2.1-or-later
#
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

[Unit]
Description=Cleanup of Temporary Directories
Documentation=man:tmpfiles.d(5) man:systemd-tmpfiles(8)
DefaultDependencies=no
Conflicts=shutdown.target initrd-switch-root.service
After=local-fs.target time-set.target
Before=shutdown.target

[Service]
Type=oneshot
ExecStart=systemd-tmpfiles --clean
SuccessExitStatus=DATAERR
IOSchedulingClass=idle

The previous unit file describes the service, but is only executed during the boot and shutdown process. Running it regulary is done by the timer unit file /usr/lib/systemd/system/systemd-tmpfiles-clean.timer. This unit file is defined to run the service every day and at least 15 minutes after the system has been booted. This to not affect the system performance when the system is booted.

#
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

[Unit]
Description=Daily Cleanup of Temporary Directories
Documentation=man:tmpfiles.d(5) man:systemd-tmpfiles(8)
ConditionPathExists=!/etc/initrd-release

[Timer]
OnBootSec=15min
OnUnitActiveSec=1d

Creating a new configuration file /etc/tmpfiles.d/local.conf with the examples below and running command systemd-tmpfiles --create to create the directory defined in the configuration file. It also cleans up the defined folder after 10 days.

d       /var/local/appdump 2750 apprun appgrp 10d -

With option d the systemd-tmpfiles --clean triggers the removal of all files passed the 10 day mark, but command systemd-tmpfiles --remove doesn’t remove anything. If the configuration file is changed to start with D the files are still removed after the 10 days, but also all files are removed if the command systemd-tmpfiles --remove is run.

D       /var/local/appdump 2750 apprun appgrp 10d -

Another example is to create symbolic links and directories in /run/ with the example below.

L       /run/mtab  -    root   root   - /proc/self/mounts
D       /run/myapp 2750 apprun appgrp - -

More options do exist, but are not shown here and can be found with man tmpfiles.d.