apache2 / httpd graceful/reload failures on Ubuntu 21.04

Fri Aug 27 09:26:30 UTC 2021

Hi,

I've been experiencing a failed apache2 service on Ubuntu 21.04 when
performing a reload using the `systemctl reload apache2` command. The
command does not always fail, but seems to be failing more often as
the number of vhosts increases (currently ca 120). My

The `systemctl reload apache2` command exits without error, but the
service ends up in a failed state. Running `systemctl start apache2`
after this failure starts the service without issues.
I had taken to do running `systemctl reload apache2; systemctl status
apache2` to validate that I have a running service, but this would
report "success" even when the service is "failed".

Expecting some timing issue, I increased the "RestartSec" systemd
parameter to 500ms using
`/etc/systemd/system/apache2.service.d/override.conf`
[Service]
RestartSec=500ms

This has not fixed the issue either.

Testing the reload using `apachectl -k graceful` can also trigger the
"failed" state of the process.

The consistent error is with the persistence of shared memory
segments. The indicated error is incorrect, there's plenty of space on
the filesystem. The configuration has been kept as close as possible
to the default Ubuntu config.

My gut feeling is some weird interaction between graceful and systemd
as seen in the logs. The RestartSec change not solving the problem
kind of goes against that.

Any help appreciated! Thanks, Bernard Spil.

$ df -h /var/run/apache2/
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           1.6G  6.6M  1.6G   1% /run

$ ls mods-enabled/*.load | sed 's/mods-enabled\///;s/\.load//'
access_compat
alias
auth_mellon
authn_core
authn_file
authz_core
authz_host
authz_user
brotli
deflate
dir
env
filter
headers
http2
lbmethod_byrequests
mime
mpm_event
negotiation
proxy
proxy_balancer
proxy_http
proxy_http2
proxy_wstunnel
remoteip
reqtimeout
rewrite
setenvif
slotmem_shm
socache_shmcb
ssl
status

/var/log/apache2/error.log:
[Fri Aug 27 00:00:18.881934 2021] [mpm_event:notice] [pid 138928:tid
140168396681856] AH00493: SIGUSR1 received.  Doing graceful restart
[Fri Aug 27 00:00:19.155640 2021] [slotmem_shm:error] [pid 138928:tid
140168396681856] (28)No space left on device: AH02611: create:
apr_shm_create(/var/run/apache2/slotmem-shm-pd38fd8d0_acc_example_org_6.shm)
failed
[Fri Aug 27 00:00:19.155679 2021] [:emerg] [pid 138928:tid
140168396681856] AH00020: Configuration Failed, exiting
[Fri Aug 27 00:05:01.645184 2021] [core:warn] [pid 166984:tid
140602886914688] AH00098: pid file /var/run/apache2/apache2.pid
overwritten -- Unclean shutdown of previous Apache run?
[Fri Aug 27 00:05:01.656692 2021] [mpm_event:notice] [pid 166984:tid
140602886914688] AH00489: Apache/2.4.46 (Ubuntu) OpenSSL/1.1.1j
configured -- resuming normal operations
[Fri Aug 27 00:05:01.656748 2021] [core:notice] [pid 166984:tid
140602886914688] AH00094: Command line: '/usr/sbin/apache2'

journalctl:
Aug 27 02:00:13 web01.example.org systemd[1]: Starting Rotate log files...
Aug 27 02:00:18 web01.example.org systemd[1]: Reloading The Apache HTTP Server.
Aug 27 02:00:18 web01.example.org systemd[1]: Reloaded The Apache HTTP Server.
Aug 27 02:00:18 web01.example.org systemd[1]: logrotate.service: Succeeded.
Aug 27 02:00:18 web01.example.org systemd[1]: Finished Rotate log files.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Main
process exited, code=exited, status=1/FAILURE
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164444 (apache2) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 166489 (apache2) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164446 (apache2) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164448 (apache2) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164450 (apache2) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164452 (apache2) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164454 (apache2) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164456 (apache2) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164458 (apache2) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164459 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164460 (apache2) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164461 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164462 (apache2) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164463 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164464 (apache2) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164465 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164466 (apache2) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164467 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164468 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164469 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164470 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164471 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164472 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164473 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164474 (apache2) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164475 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164476 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164477 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164478 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164479 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164480 (apache2) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164481 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164482 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164483 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164484 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164485 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164486 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164487 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164488 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164489 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164490 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164491 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164490 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164491 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164492 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164493 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164494 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164495 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164496 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164497 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164498 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164499 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164500 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164501 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164502 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164503 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164504 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164505 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164506 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164507 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164508 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164509 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164510 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164511 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164512 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164513 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164514 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164515 (apache2) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164516 (apache2) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164517 (apache2) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164518 (apache2) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164519 (apache2) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164520 (apache2) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164521 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164522 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164523 (apache2) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164524 (apache2) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164525 (apache2) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164526 (apache2) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164527 (apache2) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164528 (apache2) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164529 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164530 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164531 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164532 (apache2) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164533 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164534 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164535 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164536 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164537 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164538 (apache2) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164539 (apache2) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164540 (apache2) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164573 (apache2) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing
process 164606 (n/a) with signal SIGKILL.
Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Failed
with result 'exit-code'.
Aug 27 02:05:01 web01.example.org CRON[166975]:
pam_unix(cron:session): session opened for user root by (uid=0)
Aug 27 02:05:01 web01.example.org CRON[166976]: (root) CMD
(/root/bin/apache-workaround)
Aug 27 02:05:01 web01.example.org systemd[1]: Starting The Apache HTTP Server...
Aug 27 02:05:01 web01.example.org systemd[1]: Started The Apache HTTP Server.

When running a vanilla `apachectl -k graceful`:
Aug 27 10:47:18 web01.sias.intra.lighting.com sudo[176897]: sysadmin:
TTY=pts/0 ; PWD=/etc/apache2 ; USER=root ; COMMAND=/usr/sbin/apachectl
-k graceful
Aug 27 10:47:18 web01.sias.intra.lighting.com sudo[176897]:
pam_unix(sudo:session): session opened for user root by
sysadmin(uid=1000)
Aug 27 10:47:18 web01.sias.intra.lighting.com sudo[176897]:
pam_unix(sudo:session): session closed for user root
Aug 27 10:47:19 web01.sias.intra.lighting.com systemd[1]:
apache2.service: Main process exited, code=exited, status=1/FAILURE
Aug 27 10:47:19 web01.sias.intra.lighting.com systemd[1]:
apache2.service: Killing process 173239 (apache2) with signal SIGKILL.
Aug 27 10:47:19 web01.sias.intra.lighting.com systemd[1]:
apache2.service: Killing process 174924 (apache2) with signal SIGKILL.
...snip...
Aug 27 10:47:19 web01.sias.intra.lighting.com systemd[1]:
apache2.service: Killing process 176797 (apache2) with signal SIGKILL.
Aug 27 10:47:19 web01.sias.intra.lighting.com systemd[1]:
apache2.service: Killing process 176838 (n/a) with signal SIGKILL.
Aug 27 10:47:19 web01.sias.intra.lighting.com systemd[1]:
apache2.service: Failed with result 'exit-code'.
Aug 27 10:47:20 web01.sias.intra.lighting.com sudo[176903]: sysadmin:
TTY=pts/0 ; PWD=/etc/apache2 ; USER=root ; COMMAND=/usr/bin/systemctl
status apache2
Aug 27 10:47:20 web01.sias.intra.lighting.com sudo[176903]:
pam_unix(sudo:session): session opened for user root by
sysadmin(uid=1000)
Aug 27 10:47:20 web01.sias.intra.lighting.com sudo[176903]:
pam_unix(sudo:session): session closed for user root
Aug 27 10:47:34 web01.sias.intra.lighting.com sudo[176906]: sysadmin:
TTY=pts/0 ; PWD=/etc/apache2 ; USER=root ; COMMAND=/usr/bin/systemctl
start apache2
Aug 27 10:47:34 web01.sias.intra.lighting.com sudo[176906]:
pam_unix(sudo:session): session opened for user root by
sysadmin(uid=1000)
Aug 27 10:47:34 web01.sias.intra.lighting.com systemd[1]: Starting The
Apache HTTP Server...
Aug 27 10:47:34 web01.sias.intra.lighting.com systemd[1]: Started The
Apache HTTP Server.