http://docs.ganeti.org/ lists the latest stable version of Ganeti. The core Ganeti team supports and maintains the two most recent stable versions of Ganeti.
However, the Ganeti team also fixes high severity bugs in older versions. The decision whether or not to fix a bug in an older version of Ganeti is made on a case-by-case basis, and depends on the complexity of the fix, the impact of the fix on existing users, and the risk of the fix introducing regressions.
In practical terms, our policy means that we support at minimum security issues in older versions of Ganeti shipped by some major Linux distributions (which usually means the last security-supported version is the one present in Debian stable). This support is bug-driven only.
The following distributions package Ganeti:
| Distribution | Package name |
|---|---|
| Debian | Wheezy ganeti2 |
Jessie ganeti | |
| We recommend using the newer Ganeti versions in wheezy-backports. | |
| Ubuntu | Up to Saucy: ganeti2 |
As of Trusty: ganeti | |
| RPMs (RHEL/CentOS/Scientific Linux and Fedora) | See the current packaging effort at GitHub. |
| Gentoo | See the current ebuild file. |
Ganeti does not place any restrictions on guest operating systems. However, the limitations of the selected hypervisor still apply (e.g. xen-pvm can’t run Windows guests).
Ganeti relies on user-provided OS installation scripts to install guest operating systems. These scripts allow for very flexible installation scenarios and are documented at http://docs.ganeti.org/. It is also possible to boot an instance from an image and then manually install the OS.
From Ganeti 2.6 onwards, the supported Xen versions are:
Ganeti supports the following Xen toolstacks:
| Version | Xen toolstacks supported |
|---|---|
| Before Ganeti 2.6 | xm only |
| Ganeti 2.6 and later | Both xm and xl toolstacks are supported, but must be specified at configure time |
| Ganeti 2.10 and later | xl toolstack is supported, with the option to switch between the toolstacks at runtime using a hypervisor parameter |
Below is a list of Ganeti-related projects of which we’re aware. Contact us if you want your project to be added!
SAN is fully supported in version 2.9., using the ExtStorage Interface. This interface is a contribution from GRNET. For more information, see:
You may need to write your own ExtStorageProvider scripts to comply with the interface.
The name “Ganeti” does not actually have a meaning. As one of the first team members of Ganeti put it “It was the best random name that we could come up with.” It was chosen to be kind of unique in the sense of if you google for it, it will be the first hit, which it still is, this has nothing to do with that Google supports the project.
Google does not offer paid support for Ganeti, nor do we know/recommend/endorse any third party that may do so. That being said, Ganeti the software is fully open and easily understandable by a good Linux admin/developer, and you can address questions to the following mailing lists:
Google uses Ganeti to provision virtual machines inside its corporate network. We have Ganeti clusters in offices and data centers, and use these clusters to provide basic network services, virtual workstations, and general Linux servers. Google doesn’t use Ganeti for production infrastructure.
We’d like to scale Ganeti further (more nodes, instances, and parallel jobs) and to integrate interesting networking/virtualization/storage technologies as they become available. We don’t have a precise long term roadmap as of now.
You can easily convert a previously-exported Ganeti instance into an OVF package by using ovfconverter from the tools directory of the source code. The OVF package is supported by VMWare, VirtualBox, and some additional virtualization software.
You can also use an instance exported from a tool such as VMWare or VirtualBox and convert it to a Ganeti config file using the command gnt-backup import.
The document Ganeti Instance Import/Export using Open Virtualization Format has a more detailed description of the internal design of the converter, as well as a list of the available command line options.
You can deploy Ganeti by following the Ganeti installation tutorial. The main steps include:
gnt-* commands.Note: The Ganeti walk-through provides more information on initializing and configuring the cluster.
Ganeti is very suitable for small/medium deployments, and for starting small and then scaling up. It’s easy to start with even just 1 or 3 physical nodes, and then to add more nodes as needed. Managing a few clusters comprising up to 120-150 nodes should be easy for a good Linux admin or two capable of using configuration management (CFEngine, Puppet, Chef, etc.).
If you plan to scale your infrastructure to a large deployment (tens of clusters and thousands of nodes), management by someone with Python/Haskell knowledge, and the ability to use/deploy extra tools to coordinate such a big infrastructure, is advisable.
Large and notable Ganeti deployments include:
| Deployment | Description | Scale |
|---|---|---|
| debian.org | Linux distribution | 5 servers, 70 VMs |
| fsffrance.org | ||
| Part of Google’s corporate computing infrastructure | ||
| grnet.gr | Greek Research & Technology Network | 10 clusters, >6000 VMs |
| osuosl.org | Oregon State University Open Source Lab | 10 clusters, ~120 VMs |
| skroutz.gr | Price comparison engine | 20 nodes, >100 VMs, 3 locations |
There is no automated deployment process for Ganeti. However, since Ganeti version 2.11, you can upgrade an existing Ganeti cluster ( >2.10 ) using the single command: gnt-cluster upgrade.
The ideal Ganeti configuration requires two separate networks (or at least two separate VLANs):
See Security in Ganeti for security best practices and discussion of Ganeti components that require particular attention on matters security-related.
Security-wise, disks of instances are separate logical volumes and are wiped before they’re attached to an instance. Other than separating instance disks, Ganeti relies on the hypervisor to separate instances. Newer versions of Ganeti (>= 2.12) also contain some measures to mitigate compromise of a node, should an instance manage to escape the hypervisor. See Security in Ganeti for more details.
Performance-wise, Partitioned Ganeti allows resources, up to the level of individual spindles, to be individually assigned to instances.
Yes. Ganeti itself makes no assumption about running on real hardware. Of course, in order to use nested virtualization, you must use virtualization solutions that are able to run inside each other. KVM at all levels is one popular nested virtualization solution. In fact, some developers use such a KVM setup for developing Ganeti. Additionally, the Ganeti buildbot (buildbot.ganeti.org) runs completely on Ganeti VMs and includes a proper (KVM-based) QA.
Ganeti has no concept of geo location; the most closely-related concept to geo location is a node group. Nodes in any particular node group are assumed to be equivalent for usage, which means that they’re equivalent to existing in the same location. Therefore, if two nodes require geo redundancy, they need to be located in separate node groups. Instances are only moved between node groups upon explicit request, and deciding how to allocate instances among geo locations is the user’s responsibility.
To avoid two instances that provide the same service (such as DNS) from ending up on the same physical node, we recommend that you use exclusion tags. For more information, see the hbal man page.
The health status of a Ganeti cluster can be verified with the command gnt-cluster verify. Ganeti-specific information can be obtained via the ganeti-mond. Individual Ganeti instances are best monitored independently of Ganeti with the same setup you’d use to monitor a physical machine.
Answer: It’s a good idea to use configuration management to fully and automatically install nodes, as well as to maintain consistent node configuration across a cluster. In particular, we recommend configuring specific (rather than minimal) versions to avoid surprises with available updates.
Ganeti introduced native Ceph RBD support in Ganeti version 2.6. Native support for Gluster volumes was introduced in Ganeti 2.11. If the performance tradeoffs of true distributed storage are unacceptable to you, Ganeti also has excellent support for the limited-replica DRBD technology.
Users of Ganeti 2.7 and newer versions may use the external storage backend to provide support for other storage systems, distributed or otherwise. You can also support distributed storage models through the “shared file” mechanism, so long as you can provide transparent replication of the shared file storage directory across all nodes (for example, with an NFS mount point).
In all cases, storage deployment and configuration is a task left to the administrator. Ganeti is simply configured to use an existing RBD pool, Gluster volume, or file system directory for instance data storage.
For more details on Ganeti and storage, see the Ganeti administrator’s guide and the Ganetti installation tutorial.
Ganeti supports backup of instances. Support for backup compression was added in the Ganeti 2.11 release. Instance snapshotting is not yet available.
As of January 2014, support for Ceph is mature. Gluster support is not tested well as it is maintained as a best effort.
Support for instances with disks of different types is a planned feature. We worked a bit on it, but it is not finished yet.
Ganeti supports instances with multiple volume groups. Simply create instances overriding the “vg” attribute for the disk parameter. Examples on how to do this can be found in Hochschule Darmstadt Wiki.
In comparison to other cloud solutions, Ganeti is much more lightweight. Ganeti is quite simple to set up and has very few dependencies on the required infrastructure. For example, Ganeti doesn’t even require a database. Also, Ganeti’s scope is more limited than other cloud solutions: it is made to manage small- to medium-sized clusters, rather than huge clouds and doesn’t provide a management UI out of the box. For a more in-depth comparison of VMWare, OpenStack, and Ganeti, see the IaaS presentation.
libvirt, on the other hand, is only a library which abstracts over different virtualization technologies. It does not provide any cluster management features. While Ganeti can be used as abstraction for the supported hypervisors, this functionality is not the main purpose of Ganeti.
Ganeti supports redundant storage solutions, the most important of which is DRBD. Ganeti also distributes instances to ensure that the secondary nodes have enough memory to immediately start an instance, should the primary node break.
Yes, Ganeti supports automatic failover. Automatic failover is opt-in, as some Ganeti system administrators prefer to either manually respond to breakages or to delegate such responses to some other system to take actions via RAPI.
harep is Ganeti’s auto-repair tool. harep can fix DRBD breakage, migrate and failover instances, and (as a last resort) reinstall instances. harep only acts on clusters, node groups, or instances for which a tag is set allowing the tool to take action.
As of Ganeti 2.17 there is work ongoing to create a maintenance daemon which subsumes harep and adds more features. See details here.
All of Ganeti’s functionality is exposed through a set of command line tools, which are comprehensively documented in their corresponding man pages.
A subset of Ganeti’s functionality is available through a web interface: the Remote API (RAPI). RAPI provides a REST-ful view of all the entities in a cluster, allowing them to be manipulated and queried. To address security concerns, RAPI provides authentication and encryption.
While Ganeti itself does not provide a web management console, outside developers who manage large Ganeti deployments have developed solutions for this purpose and have made them available to the public.
One simple solution is Oregon State University Open Source Lab’s Ganeti Web Manager, a Django-based web application that exposes the commands accessible via the RAPI in a user-friendly way.
Synnefo provides a more feature-rich UI; however, the UI relies on Synnefo abstractions and cannot be used to manage a plain Ganeti cluster.
Yes. Ganeti supports LXC and chroot-based environments, allowing instances to be created as though a real hypervisor were used. However, those are not tested well and are maintained on a best-effort basis.
Nodes: This number can range from 1 to 150. If you’re implementing a large number of nodes, they should ideally span multiple node groups. However, Ganeti does not strictly enforce this limit of 150. It is possible to create bigger clusters, you will just have to face performance issues then.
Instances: This number is difficult to recommend, as it depends upon the specification of the node, the size of the instance, etc. There is no hard limit Ganeti enforces.
Size and number of disks: These numbers are difficult to recommend, as they depend upon the resources available in the nodes. There is no hard limit Ganeti enforces.
There is a design document describing the integration of Ganeti and Open vSwitch (see doc/html/design-openvswitch.html). This design document is still being implemented; currently, Ganeti can configure an Open vSwitch switch on nodes and also associate VLANs to instances’ NICs.
There have also been external experiments integrating Ganeti and Open vSwitch, most notably, the AFOYI experiment and integration of KVM and Open vSwitch.
USB devices can be passed directly to KVM instances when they are created (see man/gnt-instance.html for more information). For Xen, Ganeti currently simply enables USB support for domU, but does not add any USB devices. It is possible to add USB devices manually from dom0 (see Xen USB Passthrough for details). Patches for USB device support for Xen are welcome.
OS image support is currently under development and was planned for release as part of Ganeti 2.11. See doc/html/design-os.html for a description of how OS images can be used as part of the OS installation for instances. However, this is not finished.
Ganeti first used Haskell as an experiment when we needed a compiled language that could perform fast computations on cluster metadata. Eventually, we found that Haskell had advantages in its type safety that made it easy for us to develop better code with fewer common errors without having to write extensive “type-specific” unit tests. At this point, we decided to expand our use of Haskell.
Only “infrastructural” code (job queue, locking) that was already very complicated in Python has been ported to Haskell. We’ll continue to use Python for Logical Units, backend RPCs, and hypervisor, storage, and networking implementations.
There are two main reasons why Haskell constitutes a low barrier:
For good examples of setup, see the Ganeti documentation (in particular, Haskell requirements) and codebase (in particular, devel/build_chroot).
If some part of the Haskell codebase is too inflexible for your use, the Ganeti team is happy to help you understand how to customize the codebase, or how to make the codebase more modular and scriptable, if possible.