==================================
Migration speed accounting in Hbal
==================================

.. contents:: :depth: 2

Hbal usually performs complex sequence of moves during cluster balancing in
order to achieve local optimal cluster state. Unfortunately, each move may take
significant amount of time. Thus, during the sequence of moves the situation on
cluster may change (e.g., because of adding new instance or because of instance
or node parameters change) and desired moves can become unprofitable.

Usually disk moves become a bottleneck and require sufficient amount of time.
:ref:`Instance move improvements <move-performance>` considers
disk moves speed in more details. Currently, ``hbal`` has a ``--no-disk-moves``
option preventing disk moves during cluster balancing in order to perform fast
(but of course non optimal) balancing. It may be useful, but ideally we need to
find a balance between optimal configuration and time to reach this
configuration.

Avoiding insignificant disk moves
=================================

Allowing only profitable enough disk moves may become a first step to reach
a compromise between moves speed and optimal scoring. This can be implemented
by introducing ``--avoid-disk-moves *FACTOR*`` option which will admit disk
moves only if the gain in the cluster metrics is *FACTOR* times
higher than the gain achievable by non disk moves.

Avoiding insignificant long-time solutions
==========================================

The next step is to estimate an amount of time required to perform a balancing
step and to introduce a new term: ``long-time`` solution.

``--long-solution-threshold`` option will specify a duration in seconds.
A solution exceeding the duration is a ``long-time`` solution by definition.

With time estimations we will be able to filter Hbal's sequences and
eliminating long-time solutions which don't lead to a sufficient cluster metric
improvement. This can be done by the ``--avoid-long-solutions *FACTOR*`` option
which will allow only long solutions, whose K/N metrics are more than *FACTOR*,
where K is the number of times cluster metric has increased and N is an
estimated time to perform this solution divided by the threshold.

The default values for the new options are:

``--long-solution-threshold`` = 1000 seconds
(all adequate solutions are not ``long-time``)

``--avoid-long-solutions`` = 0.0
(no filtering by time estimations, feature disabled)

Network bandwidth cluster tags
==============================

The bandwidth between nodes, node groups and within whole cluster could be
specified "by hand" with the cluster tags.

Every node contains its own set of bandwidth tags. Tags from higher level are
inherited by lower levels: nodegroups inherit cluster tags and nodes inherit
nodegroups tags. Tags from lower level (if exist) override higher level tags
with the same bandwidth prefixes: node tags override nodegroup tags as well as
nodegroup tags override cluster tags.

Below are some examples of using bandwidth tags:
(The examples are provided using in the Htools Text backend format)

1) single nodegroup with 5 nodes, all bandwidths symmetric.

group-01|...|nic:100MBit|

*no* node-level bandwidth tags (they are inherited by group tags)

htools:bandwidth:nic
htools:bandwidth:nic:100MBit::nic:100MBit::100


2) 3 nodegroups, within each nodegroup symmetric bandwidths,
between nodegroups different bandwidths.

group-01|...|nic:1000MBit (overrides cluster tags)|
group-02|...|nic:100MBit (overrides cluster tags)|
group-03|...|inherited by cluster tags (nic:10MBit)|

*no* node-level bandwidth tags (they are inherited by group tags)

htools:bandwidth:nic
nic:10MBit
htools:bandwidth:nic:10MBit::nic:10MBit::10
htools:bandwidth:nic:100MBit::nic:100MBit::100
htools:bandwidth:nic:1000MBit::nic:1000MBit::1000
htools:bandwidth:nic:10MBit::nic:100MBit::10
htools:bandwidth:nic:10MBit::nic:1000MBit::10
htools:bandwidth:nic:100MBit::nic:1000MBit::100


Network bandwidth estimation
============================

Balancing time can be estimated by dividing the amount of data to be moved by
the current network bandwidth between the affected nodes.

We propose to add a new data collector, that will gather information about
network speed by sending a test file between between nodes. Accounting the
time, we can estimate average network speed between nodegroups in a cluster.

DataCollector implementation details
====================================

The new bandwidth data collector introduces an ability to collect actual
information about network speed between nodegroups in a cluster. We assume,
that the network bandwidth between any nodes within one nodegroup is almost the
same unlike the network speed between different nodegroups. So the proposed
data collector will provide the most necessary data for time estimations.

For sending packets *scp* utility will be used. The default size of file
to send is 5Mbyte in order to obtain adequate values of the network speed.

During *dcUpdate* every data collector sends a file with known size to a node
from another nodegroup (chosen randomly) and measures time to perform it. MonD
receives the response of *dcReport* from collectors, it fills nodegroup
bandwidth map in nodes. The information collected will be used to estimate the
time of balancing steps. In the case of existing network speed information for
some node from bandwidth tags as well as from bandwidth data collector the last
one will be chosen.