Packaging Services With Systemd
This is going to be of a rambling post as I try to find my way to best configure a long-running service that’s managed with Systemd. The motivation here is to install the service on a Linux machine, then configure it to run using Systemd such that:
- It launches on startup
- It restarts when the launch fails with a non-zero error code
- It runs as a non-root user
- It can be configured via environment variables loaded from an external file
We’ll go through each step piece-by-piece, as each one builds on the next, and a typical service may not need each of these properties.
The service in question is also packaged as an RPM but this is out of scope for this post.
The Package
You’ll need to start with a Systemd service definition file. A very basic one is below:
# my-service.service
[Unit]
Description=My Service
[Service]
Type=simple
ExecStart=/usr/local/bin/my-service
[Install]
WantedBy=multi-user.target
This will package a service which will launch the binary /usr/local/bin/my-service when started. This is a “simple” service in that Systemd will assume the service is started as soon as the executable is launch. This is made explicit using Type=simple but this line is optional.
Installing The System To Launch On Startup
To start your service, and configure it to start on system startup, use the following commands:
# Necessary if changing definition files, such as during upgrades
systemctl daemon-reload
systemctl enable my-service
systemctl start my-service
You can view the logs using journalctl:
journalctl -u my-service
Restart On Failure
Going back to our configuration, should you want to automatically restart the service if it crashes, you can add a Restart=on-failure directive.
# my-service.service
[Unit]
Description=My Service
[Service]
Type=simple
ExecStart=/usr/local/bin/my-service
Restart=on-failure
[Install]
WantedBy=multi-user.target
Setting Restart=on-failure will configure Systemd to try an restart the service if the executable returns a non-zero error code. For a long running service, this is perfect, as such an error code will usually indicate that the service has crashed.
There’s also a RestartSec= config which can also be used to pause between restart attempts. The default is 100 msec which might be a little short for miss-configured services, but I’d imagine if a service crash is encountered, you’d want this as low as you can have it.
It looks like Systemd will detect when a crash loop happens and will cease restarting the service after 5 attempts within a limited timespan. It would be nice if there is a way to launch the service, then apply the Restart policy after a brief period of time. This will avoid the crash loop altogether, which is likely to happen when the service is miss-configured. There is a watchdog feature that I haven’t looked into yet, but it may be what I’m looking for.
Running As A Non-Root User
Systemd launches services as root, which may not be what you want. You can configure a different user to run the service as by using the User= field:
# my-service.service
[Unit]
Description=My Service
[Service]
Type=simple
ExecStart=/usr/local/bin/my-service
User=myuser
Restart=on-failure
[Install]
WantedBy=multi-user.target
The user myuser will need to exist prior to the service starting, so you’ll need to make sure you create it when your package is installed. Obviously you’d want to do this only if the user doesn’t exist, which I didn’t know of a way to do other than grep the /etc/passwd file. A quick ask on ChatGCP yielded this command using getent:
getent passwd myuser > /dev/null || useradd -r -s /sbin/nologin myuser
External Environment Config File
Systemd supports the Environment= directive which lets you set environment variables. This will work if you have maybe a handful of variables, but if you’ve got many, the directive EnvironmentFile= can be used.
# my-service.service
[Unit]
Description=My Service
[Service]
Type=simple
EnvironmentFile=/etc/my-service.env
ExecStart=/usr/local/bin/my-service
Restart=on-failure
[Install]
WantedBy=multi-user.target
This takes the location of a typical env-file, which will be used to configure the service.
I should note that this works if you’re not expecting root to change values in env-file. This is the case for what I’m working on, however there is an alternative process for those that want to distribute their service to others, and allow them to setup overrides without changing any of the files that come from the package. I’ve yet to try it so I can’t say whether it would work, but I may make a follow-up post when I do.
Anyway, that’s all for now. I’m sure more will come up as I go further down this rabbit hole.