===================
Multi-version tests
===================

.. contents:: :depth: 4

This is a design document describing how tests which use multiple
versions of Ganeti can be introduced into the current build
infrastructure.

Desired improvements
====================

The testing of Ganeti is currently done by using two different
approaches - unit tests and QA. While the former are useful for ensuring
that the individual parts of the system work as expected, most errors
are discovered only when all the components of Ganeti interact during
QA.

However useful otherwise, until now the QA has failed to provide support
for testing upgrades and version compatibility as it was limited to
using only one version of Ganeti. While these can be tested for every
release manually, a systematic approach is preferred and none can exist
with this restriction in place. To lift it, the buildbot scripts and QA
utilities must be extended to allow a way of specifying and using
diverse multi-version checks.

Required use cases
==================

There are two classes of multi-version tests that are interesting in
Ganeti, and this chapter provides an example from each to highlight what
should be accounted for in the design.

Compatibility tests
-------------------

One interface Ganeti exposes to clients interested in interacting with
it is the RAPI. Its stability has always been a design principle
followed during implementation, but whether it held true in practice was
not asserted through tests.

An automatic test of RAPI compatibility would have to take a diverse set
of RAPI requests and perform them on two clusters of different versions,
one of which would be the reference version. If the clusters had been
identically configured, all of the commands successfully executed on the
reference version should succeed on the newer version as well.

To achieve this, two versions of Ganeti can be run separately on a
cleanly setup cluster. With no guarantee that the versions can coexist,
the deployment of these has to be separate. A proxy placed between the
client and Ganeti records all the requests and responses. Using this
data, a testing utility can decide if the newer version is compatible or
not, and provide additional information to assist with debugging.

Upgrade / downgrade tests
-------------------------

An upgrade / downgrade test serves to examine whether the state of the
cluster is unchanged after its configuration has been upgraded or
downgraded to another version of Ganeti.

The test works with two consecutive versions of Ganeti, both installed
on the same machine. It examines whether the configuration data and
instances survive the downgrade and upgrade procedures. This is done by
creating a cluster with the newer version, downgrading it to the older
one, and upgrading it to the newer one again. After every step, the
integrity of the cluster is checked by running various operations and
ensuring everything still works.

Design and implementation
=========================

Although the previous examples have not been selected to show use cases
as diverse as possible, they still show a number of dissimilarities:

- Parallel installation vs sequential deployments
- Comparing with reference version vs comparing consecutive versions
- Examining result dumps vs trying a sequence of operations

With the first two real use cases demonstrating such diversity, it does
not make sense to design multi-version test classes. Instead, the
programmability of buildbot's configuration files can be leveraged to
implement each test as a separate builder with a custom sequence of
steps. The individual steps such as checking out a given or previous
version, or installing and removing Ganeti, will be provided as utility
functions for any test writer to use.

Current state
-------------

An upgrade / downgrade test is a part of the QA suite as of commit
aa104b5e. The test and the corresponding buildbot changes are a very
good first step, both by showing that multi-version tests can be done,
and by providing utilities needed for builds of multiple branches.
Previously, the same folder was used as the base directory of any build,
and now a directory structure more accommodating to multiple builds is
in place.

The builder running the test has one flaw - regardless of the branch
submitted, it compares versions 2.10 and 2.11 (current master). This
behaviour is different from any of the other builders, which may
restrict the branches a test can be performed on, but do not
differentiate between them otherwise. While additional builders for
different versions pairs may be added, this is not a good long-term
solution.

The test can be improved by making it compare the current and the
previous version. As the buildbot has no notion of what a previous
version is, additional utilities to handle this logic will have to be
introduced.

Planned changes
---------------

The upgrade / downgrade test should be generalized to work for any
version which can be downgraded from and upgraded to automatically,
meaning versions from 2.11 onwards. This will be made challenging by
the fact that the previous version has to be checked out by reading the
version of the currently checked out code, identifying the previous
version, and then making yet another checkout.

The major and minor version can be read from a Ganeti repository in
multiple ways. The two are present as constants defined in source files,
but due to refactorings shifting constants from the Python to the
Haskell side, their position varies across versions. A more reliable way
of fetching them is by examining the news file, as it obeys strict
formatting restrictions.

With the version found, a script that acts as a previous version
lookup table can be invoked. This script can be constructed dynamically
upon buildbot startup, and specified as a build step. The checkout
following it proceeds as expected.

The RAPI compatibility test should be added as a separate builder
afterwards. As the test requires additional comparison and proxy logic
to be used, it will be enabled only on 2.11 onwards, comparing the
versions to 2.6 - the reference version for the RAPI. Details on the
design of this test will be added in a separate document.

Potential issues
================

While there are many advantages to having a single builder representing
a multi-version test, working on every branch, there is at least one
disadvantage: the need to define a base or reference version, which is
the only version that can be used to trigger the test, and the only one
on which code changes can be tried.

If an error is detected while running a test, and the issue lies with
a version other than the one used to invoke the test, the fix would
have to make it into the repository before the test could be tried
again.

For simple tests, the issue might be mitigated by running them locally.
However, the multi-version tests are more likely to be complicated than
not, and it could be difficult to reproduce a test by hand.

The situation can be made simpler by requiring that any multi-version
test can use only versions lower than the reference version. As errors
are more likely to be found in new rather than old code, this would at
least reduce the number of troublesome cases.