doc/design-systemd.rst - ganeti - Git at Google

 ===================
 Systemd integration
 ===================

 .. contents:: :depth: 4

 This design document outlines the implementation of native systemd
 support in Ganeti by providing systemd unit files. It also briefly
 discusses the possibility of supporting socket activation.


 Current state and shortcomings
 ==============================

 Ganeti currently ships an example init script, compatible with Debian
 (and derivatives) and RedHat/Fedora (and derivatives). The initscript
 treats the whole Ganeti system as a single service wrt. starting and
 stopping (but allows starting/stopping/restarting individual daemons).

 The initscript is aided by ``daemon-util``, which takes care of correctly
 ordering the startup/shutdown of daemons using an explicit order.

 Finally, process supervision is achieved by (optionally) running
 ``ganeti-watcher`` via cron every 5 minutes. ``ganeti-watcher`` will - among
 other things - try to start services that should be running but are not.

 The example initscript currently shipped with Ganeti will work with
 systemd's LSB compatibility wrappers out of the box, however there are
 a number of areas where we can benefit from providing native systemd
 unit files:

   - systemd is the `de-facto choice`_ of almost all major Linux
     distributions. Since it offers a stable API for service control,
     providing our own systemd unit files means that Ganeti will run
     out-of-the-box and in a predictable way in all distributions using
     systemd.

   - systemd performs constant process supervision with immediate
     service restarts and configurable back-off. Ganeti currently offers
     supervision only via ganeti-watcher, running via cron in 5-minute
     intervals and unconditionally starting missing daemons even if they
     have been manually stopped.

   - systemd offers `socket activation`_ support, which may be of
     interest for use at least with masterd, luxid and noded. Socket
     activation offers two main advantages: no explicit service
     dependencies or ordering needs to be defined as services will be
     activated when needed; and seamless restarts / upgrades are possible
     without rejecting new client connections.

   - systemd offers a number of `security features`_, primarily using
     the Linux kernel's namespace support, which may be of interest to
     better restrict daemons running as root (noded and mond).

 .. _de-facto choice: https://en.wikipedia.org/wiki/Systemd#Adoption
 .. _socket activation: http://0pointer.de/blog/projects/socket-activation.html
 .. _security features: http://0pointer.de/blog/projects/security.html

 Proposed changes
 ================

 We propose to extend Ganeti to natively support systemd, in addition to
 shipping the init-script as is. This requires the addition of systemd
 unit files, as well as some changes in daemon-util and ganeti-watcher to
 use ``systemctl`` on systems where Ganeti is managed by systemd.

 systemd unit files
 ------------------

 Systemd uses unit files to store information about a service, device,
 mount point, or other resource it controls. Each unit file contains
 exactly one unit definition, consisting of a ``Unit`` an (optional)
 ``Install`` section and an (optional) type-specific section (e.g.
 ``Service``). Unit files are dropped in pre-determined locations in the
 system, where systemd is configured to read them from. Systemd allows
 complete or partial overrides of the unit files, using overlay
 directories. For more information, see `systemd.unit(5)`_.

 .. _systemd.unit(5): http://www.freedesktop.org/software/systemd/man/systemd.unit.html

 We will create one systemd `service unit`_ per daemon (masterd, noded,
 mond, luxid, confd, rapi) and an additional oneshot service for
 ensure-dirs (``ganeti-common.service``). All services will ``Require``
 ``ganeti-common.service``, which will thus run exactly once per
 transaction (regardless of starting one or all daemons).

 .. _service unit: http://www.freedesktop.org/software/systemd/man/systemd.service.html

 All daemons will run in the foreground (already implemented by the
 ``-f`` flag), directly supervised by systemd, using
 ``Restart=on-failure`` in the respective units. Master role units will
 also treat ``EXIT_NOTMASTER`` as a successful exit and not trigger
 restarts. Additionally, systemd's conditional directives will be used to
 avoid starting daemons when they will certainly fail (e.g. because of
 missing configuration).

 Apart from the individual daemon units, we will also provide three
 `target units`_ as synchronization points:

   - ``ganeti-node.target``: Regular node/master candidate functionality,
     including ``ganeti-noded.service``, ``ganeti-mond.service`` and
     ``ganeti-confd.service``.

   - ``ganeti-master.target``: Master node functionality, including
     ``ganeti-masterd.service``, ``ganeti-luxid.service`` and
     ``ganeti-rapi.service``.

   - ``ganeti.target``: A "meta-target" depending on
     ``ganeti-node.target`` and ``ganti-master.target``.
     ``ganeti.target`` itself will be ``WantedBy`` ``multi-user.target``,
     so that Ganeti starts automatically on boot.

 .. _target units: http://www.freedesktop.org/software/systemd/man/systemd.target.html

 To allow starting/stopping/restarting the different roles, all units
 will include a ``PartOf`` directive referencing their direct ancestor
 target. In this way ``systemctl restart ganeti-node.target`` or ``systemctl
 restart ganeti.target`` will work as expected, i.e. restart only the node
 daemons or all daemons respectively.

 The full dependency tree is as follows:

 ::

     ganeti.target
     ├─ganeti-master.target
     │ ├─ganeti-luxid.service
     │ │ └─ganeti-common.service
     │ ├─ganeti-masterd.service
     │ │ └─ganeti-common.service
     │ └─ganeti-rapi.service
     │   └─ganeti-common.service
     └─ganeti-node.target
       ├─ganeti-confd.service
       │ └─ganeti-common.service
       ├─ganeti-mond.service
       │ └─ganeti-common.service
       └─ganeti-noded.service
         └─ganeti-common.service

 Installation
 ~~~~~~~~~~~~
 The systemd unit files will be built from templates under
 doc/examples/systemd, much like what is currently done for the
 initscript. They will not be installed with ``make install``, but left
 up to the distribution packagers to ship them at the appropriate
 locations.

 SysV compatibility
 ~~~~~~~~~~~~~~~~~~
 Systemd automatically creates a service for each SysV initscript on the
 system, appending ``.service`` to the initscript name, except if a
 service with the given name already exists. In our case however, the
 initscript's functionality is implemented by ``ganeti.target``.

 Systemd provides the ability to *mask* a given service, rendering it
 unusable, but in the case of SysV services this also results in
 failure to use tools like ``invoke-rc.d`` or ``service``. Thus we have
 to ship a ``ganeti.service`` (calling ``/bin/true``) of type
 ``oneshot``, that depends on ``ganeti.target`` for these tools to
 continue working as expected.  ``ganeti.target`` on the other hand will
 be marked as ``PartOf = ganeti.service`` for stop and restart to be
 propagated to the whole service.

 The ``ganeti.service`` unit will not be marked to be enabled by systemd
 (i.e. will not be started at boot), but will be available for manual
 invocation and only be used for compatibility purposes.

 Changes to daemon-util
 ----------------------

 ``daemon-util`` is used wherever daemon control is required:

   - In the sample initscript, to start and stop all daemons.
   - In ``ganeti.backend`` to start the master daemons on master failover and
     to stop confd when leaving the cluster.
   - In ``ganeti.bootstrap``, to start the daemons on cluster initialization.
   - In ``ganeti.cli``, to control the daemon run state during certain
     operations (e.g. renew-crypto).

 Currently, ``daemon-util`` uses two auxiliary tools for managing daemons
 ``start-stop-daemon`` and ``daemon``, in this order of preference.  In
 order not to confuse systemd in its process supervision, ``daemon-util``
 will have to be modified to start and stop the daemons via ``systemctl``
 in preference to ``start-stop-daemon`` and ``daemon``. This
 will require a basic check against run-time environment integrity:

   - Make sure that ``systemd`` runs as PID 1, which is a `simple
     check`_ against the existence of ``/run/systemd/system``.
   - Make sure ``systemd`` knows how to handle Ganeti natively. This can
     be a check against the ``LoadState`` of the ``ganeti.target`` unit.

 Unless both of these checks pass, ``daemon-util`` will fall back to its
 current behavior.

 .. _simple check: http://www.freedesktop.org/software/systemd/man/sd_booted.html

 Changes to ganeti-watcher
 -------------------------

 Since the daemon process supervision will be systemd's responsibility,
 the watcher must detect systemd's presence and not attempt to start any
 missing services. Again, systemd can be detected by the existence of
 ``/run/systemd/system``.

 Future work
 ===========

 Socket activation
 -----------------

 Systemd offers support for `socket activation`_. A daemon supporting
 socket-based activation, can inherit its listening socket(s) by systemd.
 This in turn means that the socket can be created and bound by systemd
 during early boot and it can be used to provide implicit startup
 ordering; as soon as a client connects to the listening socket, the
 respective service (and all its dependencies) will be started and the
 client will wait until its connection is accepted.

 Also, because the socket remains bound even if the service is
 restarting, new client connections will never be rejected, making
 service restarts and upgrades seamless.

 Socket activation support is trivial to implement (see
 `sd_listen_fds(3)`_) and relies on information passed by systemd via
 environment variables to the started processes.

 .. _sd_listen_fds(3): http://www.freedesktop.org/software/systemd/man/sd_listen_fds.html

 .. vim: set textwidth=72 :
 .. Local Variables:
 .. mode: rst
 .. fill-column: 72
 .. End:
	===================
	Systemd integration
	===================

	.. contents:: :depth: 4

	This design document outlines the implementation of native systemd
	support in Ganeti by providing systemd unit files. It also briefly
	discusses the possibility of supporting socket activation.


	Current state and shortcomings
	==============================

	Ganeti currently ships an example init script, compatible with Debian
	(and derivatives) and RedHat/Fedora (and derivatives). The initscript
	treats the whole Ganeti system as a single service wrt. starting and
	stopping (but allows starting/stopping/restarting individual daemons).

	The initscript is aided by ``daemon-util``, which takes care of correctly
	ordering the startup/shutdown of daemons using an explicit order.

	Finally, process supervision is achieved by (optionally) running
	``ganeti-watcher`` via cron every 5 minutes. ``ganeti-watcher`` will - among
	other things - try to start services that should be running but are not.

	The example initscript currently shipped with Ganeti will work with
	systemd's LSB compatibility wrappers out of the box, however there are
	a number of areas where we can benefit from providing native systemd
	unit files:

	- systemd is the `de-facto choice`_ of almost all major Linux
	distributions. Since it offers a stable API for service control,
	providing our own systemd unit files means that Ganeti will run
	out-of-the-box and in a predictable way in all distributions using
	systemd.

	- systemd performs constant process supervision with immediate
	service restarts and configurable back-off. Ganeti currently offers
	supervision only via ganeti-watcher, running via cron in 5-minute
	intervals and unconditionally starting missing daemons even if they
	have been manually stopped.

	- systemd offers `socket activation`_ support, which may be of
	interest for use at least with masterd, luxid and noded. Socket
	activation offers two main advantages: no explicit service
	dependencies or ordering needs to be defined as services will be
	activated when needed; and seamless restarts / upgrades are possible
	without rejecting new client connections.

	- systemd offers a number of `security features`_, primarily using
	the Linux kernel's namespace support, which may be of interest to
	better restrict daemons running as root (noded and mond).

	.. _de-facto choice: https://en.wikipedia.org/wiki/Systemd#Adoption
	.. _socket activation: http://0pointer.de/blog/projects/socket-activation.html
	.. _security features: http://0pointer.de/blog/projects/security.html

	Proposed changes
	================

	We propose to extend Ganeti to natively support systemd, in addition to
	shipping the init-script as is. This requires the addition of systemd
	unit files, as well as some changes in daemon-util and ganeti-watcher to
	use ``systemctl`` on systems where Ganeti is managed by systemd.

	systemd unit files
	------------------

	Systemd uses unit files to store information about a service, device,
	mount point, or other resource it controls. Each unit file contains
	exactly one unit definition, consisting of a ``Unit`` an (optional)
	``Install`` section and an (optional) type-specific section (e.g.
	``Service``). Unit files are dropped in pre-determined locations in the
	system, where systemd is configured to read them from. Systemd allows
	complete or partial overrides of the unit files, using overlay
	directories. For more information, see `systemd.unit(5)`_.

	.. _systemd.unit(5): http://www.freedesktop.org/software/systemd/man/systemd.unit.html

	We will create one systemd `service unit`_ per daemon (masterd, noded,
	mond, luxid, confd, rapi) and an additional oneshot service for
	ensure-dirs (``ganeti-common.service``). All services will ``Require``
	``ganeti-common.service``, which will thus run exactly once per
	transaction (regardless of starting one or all daemons).

	.. _service unit: http://www.freedesktop.org/software/systemd/man/systemd.service.html

	All daemons will run in the foreground (already implemented by the
	``-f`` flag), directly supervised by systemd, using
	``Restart=on-failure`` in the respective units. Master role units will
	also treat ``EXIT_NOTMASTER`` as a successful exit and not trigger
	restarts. Additionally, systemd's conditional directives will be used to
	avoid starting daemons when they will certainly fail (e.g. because of
	missing configuration).

	Apart from the individual daemon units, we will also provide three
	`target units`_ as synchronization points:

	- ``ganeti-node.target``: Regular node/master candidate functionality,
	including ``ganeti-noded.service``, ``ganeti-mond.service`` and
	``ganeti-confd.service``.

	- ``ganeti-master.target``: Master node functionality, including
	``ganeti-masterd.service``, ``ganeti-luxid.service`` and
	``ganeti-rapi.service``.

	- ``ganeti.target``: A "meta-target" depending on
	``ganeti-node.target`` and ``ganti-master.target``.
	``ganeti.target`` itself will be ``WantedBy`` ``multi-user.target``,
	so that Ganeti starts automatically on boot.

	.. _target units: http://www.freedesktop.org/software/systemd/man/systemd.target.html

	To allow starting/stopping/restarting the different roles, all units
	will include a ``PartOf`` directive referencing their direct ancestor
	target. In this way ``systemctl restart ganeti-node.target`` or ``systemctl
	restart ganeti.target`` will work as expected, i.e. restart only the node
	daemons or all daemons respectively.

	The full dependency tree is as follows:

	::

	ganeti.target
	├─ganeti-master.target
	│ ├─ganeti-luxid.service
	│ │ └─ganeti-common.service
	│ ├─ganeti-masterd.service
	│ │ └─ganeti-common.service
	│ └─ganeti-rapi.service
	│ └─ganeti-common.service
	└─ganeti-node.target
	├─ganeti-confd.service
	│ └─ganeti-common.service
	├─ganeti-mond.service
	│ └─ganeti-common.service
	└─ganeti-noded.service
	└─ganeti-common.service

	Installation
	~~~~~~~~~~~~
	The systemd unit files will be built from templates under
	doc/examples/systemd, much like what is currently done for the
	initscript. They will not be installed with ``make install``, but left
	up to the distribution packagers to ship them at the appropriate
	locations.

	SysV compatibility
	~~~~~~~~~~~~~~~~~~
	Systemd automatically creates a service for each SysV initscript on the
	system, appending ``.service`` to the initscript name, except if a
	service with the given name already exists. In our case however, the
	initscript's functionality is implemented by ``ganeti.target``.

	Systemd provides the ability to mask a given service, rendering it
	unusable, but in the case of SysV services this also results in
	failure to use tools like ``invoke-rc.d`` or ``service``. Thus we have
	to ship a ``ganeti.service`` (calling ``/bin/true``) of type
	``oneshot``, that depends on ``ganeti.target`` for these tools to
	continue working as expected. ``ganeti.target`` on the other hand will
	be marked as ``PartOf = ganeti.service`` for stop and restart to be
	propagated to the whole service.

	The ``ganeti.service`` unit will not be marked to be enabled by systemd
	(i.e. will not be started at boot), but will be available for manual
	invocation and only be used for compatibility purposes.

	Changes to daemon-util
	----------------------

	``daemon-util`` is used wherever daemon control is required:

	- In the sample initscript, to start and stop all daemons.
	- In ``ganeti.backend`` to start the master daemons on master failover and
	to stop confd when leaving the cluster.
	- In ``ganeti.bootstrap``, to start the daemons on cluster initialization.
	- In ``ganeti.cli``, to control the daemon run state during certain
	operations (e.g. renew-crypto).

	Currently, ``daemon-util`` uses two auxiliary tools for managing daemons
	``start-stop-daemon`` and ``daemon``, in this order of preference. In
	order not to confuse systemd in its process supervision, ``daemon-util``
	will have to be modified to start and stop the daemons via ``systemctl``
	in preference to ``start-stop-daemon`` and ``daemon``. This
	will require a basic check against run-time environment integrity:

	- Make sure that ``systemd`` runs as PID 1, which is a `simple
	check`_ against the existence of ``/run/systemd/system``.
	- Make sure ``systemd`` knows how to handle Ganeti natively. This can
	be a check against the ``LoadState`` of the ``ganeti.target`` unit.

	Unless both of these checks pass, ``daemon-util`` will fall back to its
	current behavior.

	.. _simple check: http://www.freedesktop.org/software/systemd/man/sd_booted.html

	Changes to ganeti-watcher
	-------------------------

	Since the daemon process supervision will be systemd's responsibility,
	the watcher must detect systemd's presence and not attempt to start any
	missing services. Again, systemd can be detected by the existence of
	``/run/systemd/system``.

	Future work
	===========

	Socket activation
	-----------------

	Systemd offers support for `socket activation`_. A daemon supporting
	socket-based activation, can inherit its listening socket(s) by systemd.
	This in turn means that the socket can be created and bound by systemd
	during early boot and it can be used to provide implicit startup
	ordering; as soon as a client connects to the listening socket, the
	respective service (and all its dependencies) will be started and the
	client will wait until its connection is accepted.

	Also, because the socket remains bound even if the service is
	restarting, new client connections will never be rejected, making
	service restarts and upgrades seamless.

	Socket activation support is trivial to implement (see
	`sd_listen_fds(3)`_) and relies on information passed by systemd via
	environment variables to the started processes.

	.. _sd_listen_fds(3): http://www.freedesktop.org/software/systemd/man/sd_listen_fds.html

	.. vim: set textwidth=72 :
	.. Local Variables:
	.. mode: rst
	.. fill-column: 72
	.. End: