doc/design-kvmd.rst

==========
KVM daemon
==========

.. toctree::
   :maxdepth: 2

This design document describes the KVM daemon, which is responsible for
determining whether a given KVM instance was shutdown by an
administrator or a user.


Current state and shortcomings
==============================

This design document describes the KVM daemon which addresses the KVM
side of the user-initiated shutdown problem introduced in
:doc:`design-internal-shutdown`.  We are also interested in keeping this
functionality optional.  That is, an administrator does not necessarily
have to run the KVM daemon if either he is running Xen or even, if he
is running KVM, he is not interested in instance shutdown detection.
This requirement is important because it means the KVM daemon should
be a modular component in the overall Ganeti design, i.e., it should
be easy to enable and disable it.

Proposed changes
================

The instance shutdown feature for KVM requires listening on events from
the Qemu Machine Protocol (QMP) Unix socket, which is created together
with a KVM instance.  A QMP socket typically looks like
``/var/run/ganeti/kvm-hypervisor/ctrl/<instance>.qmp`` and implements
the QMP protocol.  This is a bidirectional protocol that allows Ganeti
to send commands, such as, system powerdown, as well as, receive events,
such as, the powerdown and shutdown events.

Listening in on these events allows Ganeti to determine whether a given
KVM instance was shutdown by an administrator, either through
``gnt-instance stop|remove <instance>`` or ``kill -KILL
<instance-pid>``, or by a user, through ``poweroff`` from inside the
instance.  Upon an administrator powerdown, the QMP protocol sends two
events, namely, a powerdown event and a shutdown event, whereas upon a
user shutdown only the shutdown event is sent.  This is enough to
distinguish between an administrator and a user shutdown.  However,
there is one limitation, which is, ``kill -TERM <instance-pid>``.  Even
though this is an action performed by the administrator, it will be
considered a user shutdown by the approach described in this document.

Several design strategies were considered.  Most of these strategies
consisted of spawning some process listening on the QMP socket when a
KVM instance is created.  However, having a listener process per KVM
instance is not scalable.  Therefore, a different strategy is proposed,
namely, having a single process, called the KVM daemon, listening on the
QMP sockets of all KVM instances within a node.  That also means there
is an instance of the KVM daemon on each node.

In order to implement the KVM daemon, two problems need to be addressed,
namely, how the KVM daemon knows when to open a connection to a given
QMP socket and how the KVM daemon communicates with Ganeti whether a
given instance was shutdown by an administrator or a user.

QMP connections management
--------------------------

As mentioned before, the QMP sockets reside in the KVM control
directory, which is usually located under
``/var/run/ganeti/kvm-hypervisor/ctrl/``.  When a KVM instance is
created, a new QMP socket for this instance is also created in this
directory.

In order to simplify the design of the KVM daemon, instead of having
Ganeti communicate to this daemon through a pipe or socket the creation
of a new KVM instance, and thus a new QMP socket, this daemon will
monitor the KVM control directory using ``inotify``.  As a result, the
daemon is not only able to deal with KVM instances being created and
removed, but also capable of overcoming other problematic situations
concerning the filesystem, such as, the case when the KVM control
directory does not exist because, for example, Ganeti was not yet
started, or the KVM control directory was removed, for example, as a
result of a Ganeti reinstallation.

Shutdown detection
------------------

As mentioned before, the KVM daemon is responsible for opening a
connection to the QMP socket of a given instance and listening in on the
shutdown and powerdown events, which allow the KVM daemon to determine
whether the instance stopped because of an administrator or user
shutdown.  Once the instance is stopped, the KVM daemon needs to
communicate to Ganeti whether the user was responsible for shutting down
the instance.

In order to achieve this, the KVM daemon writes an empty file, called
the shutdown file, in the KVM control directory with a name similar to
the QMP socket file but with the extension ``.qmp`` replaced with
``.shutdown``.  The presence of this file indicates that the shutdown
was initiated by a user, whereas the absence of this file indicates that
the shutdown was caused by an administrator.  This strategy also handles
crashes and signals, such as, ``SIGKILL``, to be handled correctly,
given that in these cases the KVM daemon never receives the powerdown
and shutdown events and, therefore, never creates the shutdown file.

KVM daemon launch
-----------------

With the above issues addressed, a question remains as to when the KVM
daemon should be started.  The KVM daemon is different from other Ganeti
daemons, which start together with the Ganeti service, because the KVM
daemon is optional, given that it is specific to KVM and should not be
run on installations containing only Xen, and, even in a KVM
installation, the user might still choose not to enable it.  And finally
because the KVM daemon is not really necessary until the first KVM
instance is started.  For these reasons, the KVM daemon is started from
within Ganeti when a KVM instance is started.  And the job process
spawned by the node daemon is responsible for starting the KVM daemon.

Given the current design of Ganeti, in which the node daemon spawns a
job process to handle the creation of the instance, when launching the
KVM daemon it is necessary to first check whether an instance of this
daemon is already running and, if this is not the case, then the KVM
daemon can be safely started.

Design alternatives
===================

At first, it might seem natural to include the instance shutdown
detection for KVM in the node daemon.  After all, the node daemon is
already responsible for managing instances, for example, starting and
stopping an instance.  Nevertheless, the node daemon is more complicated
than it might seem at first.

The node daemon is composed of the main loop, which runs in the main
thread and is responsible for receiving requests and spawning jobs for
handling these requests, and the jobs, which are independent processes
spawned for executing the actual tasks, such as, creating an instance.

Including instance shutdown detection in the node daemon is not viable
because adding it to the main loop would cause KVM specific code to
taint the generality of the node daemon.  In order to add it to the job
processes, it would be possible to spawn either a foreground or a
background process.  However, these options are also not viable because
they would lead to the situation described before where there would be a
monitoring process per instance, which is not scalable.  Moreover, the
foreground process has an additional disadvantage: it would require
modifications the node daemon in order not to expect a terminating job,
which is the current node daemon design.

There is another design issue to have in mind.  We could reconsider the
place where to write the data that tell Ganeti whether an instance was
shutdown by an administrator or the user.  Instead of using the KVM
shutdown files presented above, in which the presence of the file
indicates a user shutdown and its absence an administrator shutdown, we
could store a value in the KVM runtime state file, which is where the
relevant KVM state information is.  The advantage of this approach is
that it would keep the KVM related information in one place, thus making
it easier to manage.  However, it would lead to a more complex
implementation and, in the context of the general transition in Ganeti
from Python to Haskell, a simpler implementation is preferred.

Finally, it should be noted that the KVM runtime state file benefits
from automatic migration.  That is, when an instance is migrated so is
the KVM state file.  However, the instance shutdown detection for KVM
does not require this feature and, in fact, migrating the instance
shutdown state would be incorrect.

Further considerations
======================

There are potential race conditions between Ganeti and the KVM daemon,
however, in practice they seem unlikely.  For example, the KVM daemon
needs to add and remove watches to the parent directories of the KVM
control directory until this directory is finally created.  It is
possible that Ganeti creates this directory and a KVM instance before
the KVM daemon has a chance to add a watch to the KVM control directory,
thus causing this daemon to miss the ``inotify`` creation event for the
QMP socket.

There are other problems which arise from the limitations of
``inotify``.  For example, if the KVM daemon is started after the first
Ganeti instance has been created, then the ``inotify`` will not produce
any event for the creation of the QMP socket.  This can happen, for
example, if the KVM daemon needs to be restarted or upgraded.  As a
result, it might be necessary to have an additional mechanism that runs
at KVM daemon startup or at regular intervals to ensure that the current
KVM internal state is consistent with the actual contents of the KVM
control directory.

Another race condition occurs when Ganeti shuts down a KVM instance
using force.  Ganeti uses ``TERM`` signals to stop KVM instances when
force is specified or ACPI is not enabled.  However, as mentioned
before, ``TERM`` signals are interpreted by the KVM daemon as a user
shutdown.  As a result, the KVM daemon creates a shutdown file which
then must be removed by Ganeti.  The race condition occurs because the
KVM daemon might create the shutdown file after the hypervisor code that
tries to remove this file has already run.  In practice, the race
condition seems unlikely because Ganeti stops the KVM instance in a
retry loop, which allows Ganeti to stop the instance and cleanup its
runtime information.

It is possible to determine if a process, in this particular case the
KVM process, was terminated by a ``TERM`` signal, using the `proc
connector and socket filters
<https://web.archive.org/web/20121025062848/http://netsplit.com/2011/02/09/the-proc-connector-and-socket-filters/>`_.
The proc connector is a socket connected between a userspace process and
the kernel through the netlink protocol and can be used to receive
notifications of process events, and the socket filters is a mechanism
for subscribing only to events that are relevant.  There are several
`process events <http://lwn.net/Articles/157150/>`_ which can be
subscribed to, however, in this case, we are interested only in the exit
event, which carries information about the exit signal.

.. vim: set textwidth=72 :
.. Local Variables:
.. mode: rst
.. fill-column: 72
.. End: