Orchestrator fails to start after reboot using SystemD

by Matthias Crauwels on Apr 5, 2020 12:00:00 AM

While testing in an orchestrator lab I saw that none of my Orchestrator on-raft nodes were coming online after a reboot. This is the status report from SystemD.

$ sudo systemctl status orchestrator
 * orchestrator.service - orchestrator: MySQL replication management and visualization
  Loaded: loaded (/etc/systemd/system/orchestrator.service; enabled; vendor preset: disabled)
  Active: failed (Result: exit-code) since Fri 2020-04-03 09:30:05 UTC; 30s ago
  Docs: https://github.com/github/orchestrator
  Main PID: 957 (code=exited, status=1/FAILURE)
 
 Apr 03 09:30:05 orchestrator-1 systemd[1]: Started orchestrator: MySQL replication management and visualization.
 Apr 03 09:30:05 orchestrator-1 orchestrator[957]: 2020-04-03 09:30:05 ERROR dial tcp 127.0.0.1:3306: connect: connection refused
 Apr 03 09:30:05 orchestrator-1 orchestrator[957]: 2020-04-03 09:30:05 FATAL dial tcp 127.0.0.1:3306: connect: connection refused
 Apr 03 09:30:05 orchestrator-1 systemd[1]: orchestrator.service: main process exited, code=exited, status=1/FAILURE
 Apr 03 09:30:05 orchestrator-1 systemd[1]: Unit orchestrator.service entered failed state.
 Apr 03 09:30:05 orchestrator-1 systemd[1]: orchestrator.service failed.

In this cluster, I'm using a MySQL backend for my Orchestrator instances. It seems that Orchestrator tried to start BEFORE my MySQL service was available. I did research on how I could set dependencies for the services in SystemD. I came across this article in Fedora Magazine. In our case, the Wants relationship is not good, as this only guarantees that both services will be started, but we need MySQL to be available before Orchestrator can start and we don't want to place a hard requirement on MySQL as Orchestrator works fine with SQLite. This made me choose for the Before of After relationships. Since we are configuring Orchestrator and not MySQL we will update the Orchestrator service config. This means Orchestrator needs to start After MySQL is started. SystemD offers a convenient way to configure overrides by using systemctl edit <service>. When you have no overrides configured yet, it will open an editor with an empty file. Dependencies are configured in the [Unit] section of the service definition. The following example shows how we can define our orchestrator service to be started after the mysqld service.

$ sudo systemctl edit orchestrator
 <add this content>
 [Unit]
 After=mysqld.service

Note: You can read more about the dependency configuration in this manual. Let's test our changes by rebooting the machine.

$ sudo reboot
 ...
 $ sudo systemctl status orchestrator
 * orchestrator.service - orchestrator: MySQL replication management and visualization
  Loaded: loaded (/etc/systemd/system/orchestrator.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/orchestrator.service.d
  +- override.conf
  Active: active (running) since Fri 2020-04-03 09:36:32 UTC; 52s ago
  Docs: https://github.com/github/orchestrator
  Main PID: 1483 (orchestrator)
  CGroup: /system.slice/orchestrator.service
  +- 1483 /usr/local/orchestrator/orchestrator http
 
 You can now see my reboot Orchestrator started up fine.

To end this problem, I went looking in the Orchestrator issue tracker and I found this issue where the same problem is described. I have submitted a pull-request and when accepted this issue will be addressed. Until then, if you have not found the issue, I hope this blog has saves you troubleshooting time looking for a solution.

No Comments Yet

Let us know what you think

See all

Share this

Orchestrator fails to start after reboot using SystemD

Share this

No Comments Yet

Subscribe by email