Ganeti-htools release notes
===========================


**Note**: After version 0.3.1, the htools sources have been integrated
into the ganeti core repository, and released together with the ganeti
releases. Thus this NEWS file is obsolete.

Version 0.3.1 (Fri, 11 Mar 2011)
--------------------------------

Minor bugfix release:

- Fixed source archive generation: the hscolour.css file was an invalid
  symlink, and the man pages were not correctly timestamped (leading to
  unneeded build-time rebuilds)
- Improved the Luxi backend to show which attribute fails parsing
- Small improvements to the man pages, and also ship the HTML version of
  man pages in the source archive


Version 0.3.0 (Fri, 04 Feb 2011)
--------------------------------

A significant release that breaks compatibility with Ganeti versions
below 2.4 due to the node group changes. Only the RAPI backend can talk
to older clusters, but it is recommended to use this version only with
Ganeti 2.4.

All commands are now multi-group aware (but to various degrees), so
allocation, balancing and capacity calculation respects the group layout
and will not create “broken” instances by using nodes from different
groups.

For a regular, single-group cluster, no changes should be directly
visible to the users. A multi-group cluster however will change some
things slightly:

- hbal will require a target group to operate on (no cluster-wide
  balancing yet)
- evacuation of (DRBD) instances from a node will be restricted to nodes
  in the same group, as inter-group moves are not implemented yet
- capacity, while showing correct data, will not give per-group details
  yet

There are other changes in this release:

- fixed a long-standing bug in hscan related to node memory data
- changed the text backend format, which unfortunately invalidates old
  files
- error handling improvements, so that invalid input data reports better
  where the error is
- the simulation backend changes its syntax, now it takes the allocation
  policy too, and can generate multiple groups
- (internal) man page generation moved to pandoc from hand-written,
  which is helpful as it can also generate HTML versions
- the balancing algorithm has been changed to work in parallel, if the
  code is linked against the multi-threaded runtime; this gives a very
  good speedup (~80% on 4 cores, ~60-70% of 12 cores)

Version 0.2.8 (Thu, 23 Dec 2010)
--------------------------------

A bug fix release:

- fixed balancing function for big clusters, which will improve corner
  cases where hbal didn't see any solution even though the cluster was
  obviously not well balanced
- fixed exit code of hbal in case of (Luxi) job errors
- changed the signal handling in hbal in order to make hbal control
  easier: instead of synchronising on the count of signals, make SIGINT
  cause graceful termination, and SIGTERM an immediate one
- increased the tag exclusion weight so that it has greater importance
  during the balancing
- slight improvement to the speed of balancing via algorithm tweaks


Version 0.2.7 (Thu, 07 Oct 2010)
--------------------------------

Bug fixes:

- fixed the error message for hail multi-evacuation mode
- improve evacuation mode for offline secondary nodes (ignore available
  memory)

New features:

- add a new option ``-S`` to hbal and hspace that saves the cluster
  state at the end of the processing in the text format used by the
  ``-t`` option, for later re-processing
- a two new options to hbal, -g and --min-gain-limit, that should help
  in limiting the number of balances steps with a low gain in the final
  stages
- hbal, when executing jobs, will now wait for the current jobs to
  finish at the first stop (e.g. ^C); if the user wants immediate exit,
  another signal should be sent
- added “normalized” physical CPU units in hspace output (NPU), which
  represents units of physical CPUs free/used, based on the max-cpu
  ratio


Version 0.2.6 (Mon, 26 Jul 2010)
--------------------------------

Exactly three months since the last release. Many internal changes, plus
a couple of important changes in the balancing algorithm.

First, the balancing may now introduce N+1 errors, if this solves other,
more critical problems. For the moment, this means that moving instances
away from offline nodes is allowed even if it creates N+1 errors, and
that means evacuation can be done in more cases.

Second, the scoring for N+1 has changed. In previous versions, it simply
counted the number of failing N+1 nodes, which means moving an instance
away from a N+1 failed node (but without the node 'clearing' the N+1
status) was not reflected in the cluster score. As such, the balancing
algorithm managed to clear N+1 errors only sometimes, since usually it
takes more than one move for this, and the first prerequisite move was
not 'rewarded' appropriately and thus it was not selected. Now, it is
possible to fix many more error cases than before: on a simulated 40
node cluster full with instances (symmetrically allocated on all nodes),
around five nodes can be evacuated before N+1 errors can be solved,
whereas 0.2.5 could evacuate at best one node.

There were some other internal changes to the scoring algorithm, such
that now the metrics have associated weights, and they are not all of
the same importance anymore. As of now, the only change is that offline
instances have a higher weight, which should favour proper node
evacuations.

Among the other changes:

- fixed the hspace KM_POOL_* metrics, which were returned as the final
  state and not as the delta between the initial and final states
- fixed hspace handling of N+1 failing clusters: before, it used to
  generate a 'fake' response, and the structure of this response was not
  always in sync with the real responses, leading to missing items;
  currently it proceeds correctly through the code (skipping the
  computation), and uses the same display mechanisms as the normal case
- fixed hscan exit code for RAPI failures: previously it finished with
  success even if all the clusters failed, which was creating issues
  with the live-test script; now it exits with exit code 2 for RAPI
  failures (unfortunately this is still not optimal as LUXI failures
  will use exit code 1, the same as the command line)
- changed the limit values for CPU/disk, which previously were used
  optionally, whereas now they are always used; the default cpu ratio
  limit is now 64 VCPUs per PCPU
- changed the internal handling of the short name vs. original
  (Ganeti-provided) name; now internally we always use the full name,
  and only in display routines we show the shortened (called 'alias')
  name; as a result, the -O and --excluded-instances options now accept
  both the full name and the shortened name
- changed internal handling of JSON conversions and errors, such that
  now we show a better context for failure messages, which should help
  with diagnosing the malformed message
- changed the names for a few node fields, and added some more nodes;
  this is most likely to help with debugging, and not with regular
  operation though
- changed the node fields option to allow the '+' prefix to mean 'extend
  the default fields list' rather than start from fresh (similar to
  Ganeti's implementation)
- a few internal changes related to the LUXI protocol implementation,
  which should make it more safe against potential bugs, one
  optiomization that should help with large messages, and some patches
  in preparation for potential expansion of the LUXI backend functionality

And finally, many improvements on unittests and the live-test
script. Test coverage is much enhanced, and the test infrastructure has
better error reporting; this should lead down-the-road to better code
and fewer bugs…


Version 0.2.5 (Mon, 26 Apr 2010)
--------------------------------

Some internal cleanup plus a few user-visible changes:

- new option for marking instances as 'do-not-move' during rebalancing
- allow ``hscan`` to scan the local cluster via Luxi
- add more metrics to ``hspace`` which show the delta between original
  state and final state better (only valid for tiered allocation)


Version 0.2.4 (Mon, 22 Feb 2010)
--------------------------------

Two improvements for node evacuation:

- hbal takes a new parameter ``--evac-mode`` that restricts the
  instances to be moved to the ones on offline/drained nodes, which
  should reduce the work done
- hail supports the new ``multi-evacuate`` mode of the IAllocator
  protocol, that will be released in a minor release on the Ganeti 2.1
  branch


Version 0.2.3 (Thu,  4 Feb 2010)
--------------------------------

A small release:

- Fixes selection of secondary node: previously, if the cluster had
  many N+1 failures, a N+1 failed node could be selected as secondary
  even if it did not have enough memory to allow the instance to be
  migrated/failed over to it; this is bad for automated tools, since
  we can get the cluster in an unhealthy state
- Switch the text backend to a single input file, that is generated
  now by hscan and shouldn't be generated manually via
  gnt-node/instance list anymore; this allows richer information to be
  kept in the file, and simplifies a little the internals of the text
  backend


Version 0.2.2 (Tue, 29 Dec 2009)
--------------------------------

Small release, 0.2.1 was broken and thus this was released earlier:

- Release 0.2.1 broke the LUXI backend due to a typo, fixed
- Added a live-test script that should catch errors like the above one
  in the future (needs a working, non-empty cluster)
- Changed RAPI and LUXI backends to treat drained nodes as offline,
  similar to the IAllocator backend change in 0.2.0 (which was wrongly
  marked as affecting all backends)
- Changed the metrics for offline instances and N1 score from percent to
  count, in order to increase the priority of evacuations
- Added a new metric (offline primary instances) which should fix the
  evacuation of a offline node in a 2-node cluster


Version 0.2.1 (Wed,  2 Dec 2009)
--------------------------------

- Added instance exclusion defined via instance tags
- Fixed the output of hspace to be again parseable from the shell


Version 0.2.0 (Tue, 10 Nov 2009)
--------------------------------

A significant release, with a few new major features:

- Added direct execution of the hbal solution when using the Luxi
  backend; the steps for each instance moves are submitted as a single
  jobs, and the different jobs are submitted as groups in order to
  parallelise the execution of moves
- Added support for balancing based on dynamic utilisation data for
  instances, fed in via a text file; by default, all instances are
  considered equal and this change also improves the equalisation of
  secondary instances per node
- Added support for tiered capacity calculation in hspace, where we
  start from a maximum instance spec and decrease the spec when we run
  out of resources; this should give a better measure of available
  capacity on 'fragmented' clusters; this is done separately from the
  current fixed-mode computation

Also there have been many minor improvements:

- Added option for showing instances (“--print-instances”), similar to
  the print nodes option
- Added support for customising the node list via an argument to the
  print nodes option in the form of a comma-separated list of field
  names; currently the field names are not documented, expecting further
  changes in a next release
- Enhanced the error reporting in the Luxi and Rapi backends
- Changed the handling of drained nodes, now being treated the same as
  offline nodes, for Ganeti 2.0.4+ compatibility
- A number of internal changes, simplifying code and merging some
  disparate functions
- Simplify the build system in relation to creation of archives


Version 0.1.8 (Tue, 29 Sep 2009)
--------------------------------

- Brown-paper-bag release fixing haddock issues


Version 0.1.7 (Mon, 28 Sep 2009)
--------------------------------

- Fixed a bug in the Luxi backend for big responses
- Fixed test suite exit code in presence of test failures
- Changed the migrate operation to run instead failover for instances
  which were marked as not running in the input data (this could have
  been changed since then, but it's better than today's always migrate)
- Added support for 'cheap' moves only (only migrate/failover) in
  balancing
- Added support for building without curl (thus no RAPI backend)


Version 0.1.6 (Wed, 19 Aug 2009)
--------------------------------

- Added support for Luxi (the native Ganeti protocol)
- Added support for simulated clusters (for hspace only)
- Added timeouts for the RAPI backend
- Fixed a few inconsistencies in the command line handling
- Fixed handling of errors while loading data
- The 'network' is a new dependency due to the Luxi addition


Version 0.1.5 (Thu, 09 Jul 2009)
--------------------------------

- Removed obsolete hn1 program; this allowed removal of a lot of
  supporting code
- Lots of changes in hspace: the output now is a shell fragment in order
  for script to source it or parse it easier; added failure reasons;
  optimised to use less memory for large clusters
- Optimized the scoring algorithm (used by all tools) so that now
  computations should be faster


Version 0.1.4 (Tue, 16 Jun 2009)
--------------------------------

- Added CPU count/ratio of virtual-to-physical CPUs to the cluster
  scoring methods; this means that now the balancer, the iallocator
  plugin and so on will try to keep the VCPU-to-PCPU ratio equal across
  the cluster
- Fixed some hscan bugs
- Fixed the way iallocator reads the total disk size (was broken and it
  was always falling back to summing the disk sizes)
- Internals: fixed most compile-time warnings


Version 0.1.3 (Fri, 05 Jun 2009)
--------------------------------

- Fix a bug in the ReplacePrimary instance moves, affecting most of the
  tools


Version 0.1.2 (Tue, 02 Jun 2009)
--------------------------------

- Add a new program, “hspace”, which computes the free space on a
  cluster (based on a given instance spec)
- Improvements in API docs and partially in the user docs
- Started adding unittests


Version 0.1.1 (Tue, 26 May 2009)
--------------------------------

- Add a new program, “hail”, which is an iallocator plugin and can
  allocate/relocate instances
- Experimental support for non-mirrored instances (hail supports them,
  hbal should no longer abort when it finds such instances and simply
  ignore them)
- The RAPI port and/or scheme can be overriden now, and even “file://”
  schemes can be used if the message body has been saved under the
  appropriate name
- Lots of code reorganization, esp. rewritten loading pipeline
- Better data checking and better error messages in case validation
  fails; tools now consider nodes with error in input data (‘?’ returned
  by ganeti) as offline
- Small enhancement to the makefile for simpler packaging


Version 0.1.0 (Tue, 19 May 2009)
--------------------------------

- Drop compatibility with Ganeti 1.2
- Add a new minimum score option (with a very low default), should help
  with very good clusters (but is still not optimal)
- Add a --quiet option to hbal
- Add support for reading offline nodes directly from the cluster


Version 0.0.8 (Tue, 21 Apr 2009)
--------------------------------

- hbal: prevent mismatches in wrong node names being passed to -O, by
  aborting in this case
- add the ability to write the commands (-C) to a script via (-C<file>),
  so that it can be later executed directly; this has also changed the
  commands to include the ncessary -f flags to skip confirmations
- add checks for extra argument in hbal and hn1, so that unintended
  errors are catched
- raise the accepted “missing” memory limit to 512MB, to cover usual Xen
  reservations


Version 0.0.7 (Mon, 23 Mar 2009)
--------------------------------

- added support for offline nodes, which are not used as targets for
  instance relocation and if they hold instances the hbal algorithm will
  attempt to relocate these away
- added support for offline instances, which now will no longer skew the
  free memory estimation of nodes; the algorithm will no longer create
  conditions for N+1 failures when such instances are later started
- implemented a complete model of node resources, in order to prevent an
  unintended re-occurrence of cases like the offline instance were we
  miscalculate some node resource; this gives warning now in case the
  node reported free disk or free memory deviates by more than a set
  amount from the expected value
- a new tool *hscan* that can generate the input text-file for the other
  tools by collection via RAPI
- some small changes to the build system to make it more friendly; also
  included the generated documentation in the source archive


Version 0.0.6 (Mon, 16 Mar 2009)
--------------------------------

- re-factored the hbal algorithm to make it stable in the sense that it
  gives the same solution when restarted from the middle; barring
  rounding of disk/memory and incomplete reporting from Ganeti (for
  1.2), it should be now feasible to rely on its output without
  generating moves ad infinitum
- the hbal algorithm now uses two more variables: the node N+1 failures
  and the amount of reserved memory; the first of which tries to ‘fix’
  the N+1 status, the latter tries to distribute secondaries more
  equally
- the hbal algorithm now uses two more moves at each step:
  replace+failover and failover+replace (besides the original failover,
  replace, and failover+replace+failover)
- slightly changed the build system to embed GIT version/tags into the
  binaries so that we know for a binary from which tree it was done,
  either via ‘--version’ or via “strings hbal|grep version”
- changed the solution list and in general the hbal output to be more
  clear by default, and changed “gnt-instance failover” to “gnt-instance
  migrate”
- added man pages for the two binaries


Version 0.0.5 (Mon, 09 Mar 2009)
--------------------------------

- a few small improvements for hbal (possibly undone by later changes),
  hbal is now quite faster
- fix documentation building
- allow hbal to work on non N+1 compliant clusters, but without
  guarantees that the end cluster will be compliant; in any case, this
  should give a smaller number of nodes that are not compliant if the
  cluster state permits it
- strip common domain suffix from nodes and instances, so that output is
  shorter and hopefully clearer


Version 0.0.4 (Sun, 15 Feb 2009)
--------------------------------

- better balancing algorithm in hbal
- implemented an RAPI collector, now the cluster data can be gathered
  automatically via RAPI and doesn't need manual export of node and
  instance list


Version 0.0.3 (Wed, 28 Jan 2009)
--------------------------------

- initial release of the hbal, a cluster rebalancing tool
- input data format changed due to hbal requirements


Version 0.0.2 (Tue, 06 Jan 2009)
--------------------------------

- fix handling of some common cases (cluster N+1 compliant from the
  start, too big depth given, failure to compute solution)
- add option to print the needed command list for reaching the proposed
  solution


Version 0.0.1 (Tue, 06 Jan 2009)
--------------------------------

- initial release of hn1 tool

.. vim: set textwidth=72 :
.. Local Variables:
.. mode: rst
.. fill-column: 72
.. End: