doc/design-lu-generated-jobs.rst

==================================
Submitting jobs from logical units
==================================

.. contents:: :depth: 4

This is a design document about the innards of Ganeti's job processing.
Readers are advised to study previous design documents on the topic:

- :ref:`Original job queue <jqueue-original-design>`
- :ref:`Job priorities <jqueue-job-priority-design>`


Current state and shortcomings
==============================

Some Ganeti operations want to execute as many operations in parallel as
possible. Examples are evacuating or failing over a node (``gnt-node
evacuate``/``gnt-node failover``). Without changing large parts of the
code, e.g. the RPC layer, to be asynchronous, or using threads inside a
logical unit, only a single operation can be executed at a time per job.

Currently clients work around this limitation by retrieving the list of
desired targets and then re-submitting a number of jobs. This requires
logic to be kept in the client, in some cases leading to duplication
(e.g. CLI and RAPI).


Proposed changes
================

The job queue lock is guaranteed to be released while executing an
opcode/logical unit. This means an opcode can talk to the job queue and
submit more jobs. It then receives the job IDs, like any job submitter
using the LUXI interface would. These job IDs are returned to the
client, who then will then proceed to wait for the jobs to finish.

Technically, the job queue already passes a number of callbacks to the
opcode processor. These are used for giving user feedback, notifying the
job queue of an opcode having gotten its locks, and checking whether the
opcode has been cancelled. A new callback function is added to submit
jobs. Its signature and result will be equivalent to the job queue's
existing ``SubmitManyJobs`` function.

Logical units can submit jobs by returning an instance of a special
container class with a list of jobs, each of which is a list of opcodes
(e.g.  ``[[op1, op2], [op3]]``). The opcode processor will recognize
instances of the special class when used a return value and will submit
the contained jobs. The submission status and job IDs returned by the
submission callback are used as the opcode's result. It should be
encapsulated in a dictionary allowing for future extensions.

.. highlight:: javascript

Example::

  {
    "jobs": [
      (True, "8149"),
      (True, "21019"),
      (False, "Submission failed"),
      (True, "31594"),
      ],
  }

Job submissions can fail for variety of reasons, e.g. a full or drained
job queue. Lists of jobs can not be submitted atomically, meaning some
might fail while others succeed. The client is responsible for handling
such cases.


Other discussed solutions
=========================

Instead of requiring the client to wait for the returned jobs, another
idea was to do so from within the submitting opcode in the master
daemon. While technically possible, doing so would have two major
drawbacks:

- Opcodes waiting for other jobs to finish block one job queue worker
  thread
- All locks must be released before starting the waiting process,
  failure to do so can lead to deadlocks

Instead of returning the job IDs as part of the normal opcode result,
introducing a new opcode field, e.g. ``op_jobids``, was discussed and
dismissed. A new field would touch many areas and possibly break some
assumptions. There were also questions about the semantics.

.. vim: set textwidth=72 :
.. Local Variables:
.. mode: rst
.. fill-column: 72
.. End: