[lug] Systemd unit file experiments

[Rob Nagler]

Since Pandora's Can is opened, I thought I'd mention something I just
learned today through some empirical research.

Here are the unit files for apache and nginx:

*# /usr/lib/systemd/system/httpd.service*
[Unit]
Description=The Apache HTTP Server
After=network.target remote-fs.target nss-lookup.target
Documentation=man:httpd(8)
Documentation=man:apachectl(8)

[Service]
Type=notify
EnvironmentFile=/etc/sysconfig/httpd
ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND
ExecReload=/usr/sbin/httpd $OPTIONS -k graceful
ExecStop=/bin/kill -WINCH ${MAINPID}
# We want systemd to give httpd some time to finish gracefully, but still
want
# it to kill httpd after TimeoutStopSec if something went wrong during the
# graceful stop. Normally, Systemd sends SIGTERM signal right after the
# ExecStop, which would kill httpd. We are sending useless SIGCONT here to
give
# httpd time to finish.
KillSignal=SIGCONT
PrivateTmp=true

[Install]
WantedBy=multi-user.target


*# /usr/lib/systemd/system/nginx.service*
[Unit]
Description=The nginx HTTP and reverse proxy server
After=network.target remote-fs.target nss-lookup.target

[Service]
Type=forking
PIDFile=/run/nginx.pid
# Nginx will fail to start if /run/nginx.pid already exists but has the
wrong
# SELinux context. This might happen when running `nginx -t` from the
cmdline.
# https://bugzilla.redhat.com/show_bug.cgi?id=1268621
ExecStartPre=/usr/bin/rm -f /run/nginx.pid
ExecStartPre=/usr/sbin/nginx -t
ExecStart=/usr/sbin/nginx
ExecReload=/bin/kill -s HUP $MAINPID
KillSignal=SIGQUIT
TimeoutStopSec=5
KillMode=process
PrivateTmp=true

[Install]
WantedBy=multi-user.target

We use nginx as a reverse proxy and apache for mod_perl/middleware. I'm
uncontainerizing our mod_perl services so I need to switch from docker unit
files to directly invoking httpd. It's pretty interesting how different
these two files are, right down to the descriptions, even though they are
supposed to be interchangeable components, that is, you can use httpd as
reverse proxy just as well as nginx. Both can fork and detach, but the
systemd files are oddly very different.

The biggest question above is type=simple, notify vs forking. Here's a nice
article that explains why you want to avoid forking:
https://www.lucas-nussbaum.net/blog/?p=877  I think httpd almost got it
right, but it probably should be "simple" unless mod_systemd is loaded,
which manages sd_notify. mod_systemd was introduced in httpd 2.5, and
CentOS 7 uses httpd 2.4, and hasn't back ported mod_systemd afaict.
However, the article fails to mention PIDFile, which nginx correctly uses.

Aside. Why would I have bothered looking at this? We use httpd as
middleware on a multi-tenant (app) system so there are multiple instances
of httpd running so we have to generate systemd unit files for each app.

This is an area where systemd unit files are not as good as service
scripts. We have a single service script that starts all our services
through parameterization. You can't do that with unit files, which instead
need to be generated. As you can see, both the nginx and httpd unit files
are "implicitly coupled" to a lot of behavior about how these services
start and stop. For example, nginx -t is a good ExecStartPre value, and
httpd should have this, too, and it sort of does in that it uses graceful
for Reload which does -t implicitly. It's interesting that nginx doesn't do
nginx -s reload, which is the correct, explicitly coupled way to reload
nginx, because ExectStartPre is not called on reload.

You may say this doesn't matter, but it turns out it does as this comment
<https://serverfault.com/a/700754> mentions (but didn't get as many votes,
unfortunately). There's a bug in the sysv init scripts, just like there's a
bug in the systemd unit file. You have to use reload, because it runs
configtest and then sends the signal. If you just send the signal, nginx
will ignore invalid changes, which led to some confusion the post
<https://serverfault.com/questions/378581/nginx-config-reload-without-downtime>.
Turns out httpd has the same behavior (surprise!) so the httpd unit's
ExecReload is correct.

Back to the main theme: type=simple. Neither nginx nor httpd are tightly
integrated with systemd in CentOS 7 so they should definitely not be
notify. Forking is dangerous, because if the port is in use (say you
started httpd accidentally), nginx will die after the fork unless you set
PIDFile, which nginx correctly does. Don't do this with dockerized
services, however, because the PID will be 1.

I had mentioned that I don't want to learn systemd unit file syntax, and
the careful reader will notice that nginx references $MAINPID and httpd
specifies ${MAINPID}. I'd argue $MAINPID is better, because $MAINPID does
not contain space. I like the explanation in this post
<https://stackoverflow.com/a/37865527>. The example in the man page
<https://www.freedesktop.org/software/systemd/man/systemd.service.html#Command%20lines>
with echo's is really confusing. Exercise for the reader, what does this
output?

ExecStart=/bin/echo "$ONE $TWO $THREE"


Hint: nothing. :)

This is what I've ended up for our app units:

[Unit]
Description={{ app }}
After=network.target remote-fs.target nss-lookup.target

[Service]
Type=forking
PIDFile=/srv/{{ app }}/httpd.pid
ExecStart=/srv/{{ app }}/start
ExecReload=/srv/{{ app }}/reload
ExecStop=/srv/{{ app }}/stop
KillMode=process
# SIGCONT is ignored by httpd to allow ExecStop to do its job.
KillSignal=SIGCONT
# Second signal is SIGKILL, which kills all processes in the group.
TimeoutStopSec=5

[Install]
WantedBy=multi-user.target

The commands start, stop, and reload should be obvious from the above
discussion.  This post is also in our wiki
<https://github.com/radiasoft/devops/wiki/SystemdUnitFiles>. I'll expand on
it there.

Rob
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lug.boulder.co.us/pipermail/lug/attachments/20180331/7f4680e7/attachment.html>