Seen one of these? Stopped a container, would not restart, got this “Can not set cookie dm_task_set_cookie failed” error. After trying to manually remove the /dev/mapper/docker-* file systems, I got a semaphore error.
So check your semaphores and see if you need to increase.
ipcs -u
—— Messages Status ——–
allocated queues = 0
used headers = 0
used space = 0 bytes
—— Shared Memory Status ——–
segments allocated 0
pages allocated 0
pages resident 0
pages swapped 0
Swap performance: 0 attempts 0 successes
—— Semaphore Status ——–
used arrays = 128
allocated semaphores = 128
And this is it, I had used all 128 arrays…. Increasing all the values didn’t really hurt I don’t think, but I could have just increased the number of arrays.
ipcs -ls
—— Semaphore Limits ——–
max number of arrays = 256
max semaphores per array = 500
max semaphores system wide = 64000
max ops per semop call = 200
semaphore max value = 32767
Increased in /etc/sysctl.conf
#kernel.sem=250 32000 100 128 – old value only number I really needed to increase was the 128 to 256.
kernel.sem=500 64000 200 256
This solved the issue for me. Course you could do the windows thing and reboot…. Here is the journalctl -xe partial output.
— Subject: Unit docker.service has failed
— Defined-By: systemd
— Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
—
— Unit docker.service has failed.
—
— The result is failed.
Jun 27 22:04:29 xxxxx.redhawk.org systemd[1]: Dependency failed for Run docker-cleanup every hour.
— Subject: Unit docker-cleanup.timer has failed
— Defined-By: systemd
— Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
—
— Unit docker-cleanup.timer has failed.
—
— The result is dependency.
Jun 27 22:04:29 xxxxx.redhawk.org systemd[1]: Job docker-cleanup.timer/start failed with result ‘d
Jun 27 22:04:29 xxxxx.redhawk.org systemd[1]: Unit docker.service entered failed state.
Jun 27 22:04:29 xxxxx.redhawk.org systemd[1]: docker.service failed.
Jun 27 22:04:29 xxxxx.redhawk.org polkitd[639]: Unregistered Authentication Agent for unix-process
[user@xxxxx ~]#