doc/design-scsi-kvm.rst - ganeti - Git at Google

 ==========
 KVM + SCSI
 ==========

 .. contents:: :depth: 4

 This is a design document detailing the refactoring of device
 handling in the KVM Hypervisor. More specifically, it will use
 the latest QEMU device model and modify the hotplug implementation
 so that both PCI and SCSI devices can be managed.


 Current state and shortcomings
 ==============================

 Ganeti currently supports SCSI virtual devices in the KVM hypervisor by
 setting the `disk_type` hvparam to `scsi`. Ganeti will eventually
 instruct QEMU to use the deprecated device model (i.e. -drive if=scsi),
 which will expose the backing store as an emulated SCSI device. This
 means that currently SCSI pass-through is not supported.

 On the other hand, the current hotplug implementation
 :doc:`design-hotplug` uses the latest QEMU
 device model (via the -device option) and is tailored to paravirtual
 devices, which leads to buggy behavior: if we hotplug a disk to an
 instance that is configured with disk_type=scsi hvparam, the
 disk which will get hot-plugged eventually will be a VirtIO device
 (i.e., virtio-blk-pci) on the PCI bus.

 The current implementation of creating the QEMU command line is
 error-prone, since an instance might not be able to boot due to PCI slot
 congestion.


 Proposed changes
 ================

 We change the way that the KVM hypervisor handles block devices by
 introducing latest QEMU device model for SCSI devices as well, so that
 scsi-cd, scsi-hd, scsi-block, and scsi-generic device drivers are
 supported too. Additionally we refactor the hotplug implementation in
 order to support hotplugging of SCSI devices too. Finally, we change the
 way we keep track of device info inside runtime files, and the way we
 place each device upon instance startup.

 Design decisions
 ================

 How to identify each device?

 Currently KVM does not support arbitrary IDs for devices; supported are
 only names starting with a letter, with max 32 chars length, and only
 including the '.', '_', '-' special chars. Currently we generate an ID
 with the following format: <device type>-<part of uuid>-pci-<slot>.
 This assumes that the device will be plugged in a certain slot on the
 PCI bus. Since we want to support devices on a SCSI bus too and adding
 the PCI slot to the ID is redundant, we dump the last two parts of the
 existing ID. Additionally we get rid of the 'hot' prefix of device type,
 and we add the next two parts of the UUID so the chance of collitions
 is reduced significantly. So, as an example, the device ID of a disk
 with UUID '9e7c85f6-b6e5-4243-b27d-680b78c6d203' would be now
 'disk-9e7c85f6-b6e5-4243'.


 Which buses does the guest eventually see?

 By default QEMU starts with a single PCI bus named "pci.0". In case a
 SCSI controller is added on this bus, a SCSI bus is created with
 the corresponding name: "scsi.0".
 Any SCSI disks will be attached on this SCSI bus. Currently Ganeti does
 not explicitly use a SCSI controller via a command line option, but lets
 QEMU add one automatically if needed. Here, in case we have a SCSI disk,
 a SCSI controller is explicitly added via the -device option. For the
 SCSI controller, we do not specify the PCI slot to use, but let QEMU find
 the first available (see below).


 What type of SCSI controller to use?

 QEMU uses the `lsi` controller by default. To make this configurable we
 add a new hvparam, `scsi_controller_type`. The available types will be
 `lsi`, `megasas`, and `virtio-scsi-pci`.


 Where to place the devices upon instance startup?

 The default QEMU machine type, `pc`, adds a `i440FX-pcihost`
 controller on the root bus that creates a PCI bus with `pci.0` alias.
 By default the first three slots of this bus are occupied: slot 0
 for Host bridge, slot 1 for ISA bridge, and slot 2 for VGA controller.
 Thereafter, the slots depend on the QEMU options passed in the command
 line.

 The main reason that we want to be fully aware of the configuration of a
 running instance (machine type, PCI and SCSI bus state, devices, etc.)
 is that in case of migration a QEMU process with the exact same
 configuration should be created on the target node. The configuration is
 kept in the runtime file created just before starting the instance.
 Since hotplug has been introduced, the only thing that can change after
 starting an instance is the configuration related to NICs and Disks.

 Before implementing hotplug, Ganeti did not specify PCI slots
 explicitly, but let QEMU decide how to place the devices on the
 corresponding bus. This does not work if we want to have hotplug-able
 devices and migrate-able VMs. Currently, upon runtime file creation, we
 try to reserve PCI slots based on the hvparams, the disks, and the NICs
 of the instance. This has three major shortcomings: first, we have to be
 aware which options modify the PCI bus which is practically impossible
 due to the huge amount of QEMU options, second, QEMU may change the
 default PCI configuration from version to version, and third, we cannot
 know if the extra options passed by the user via the `kvm_extra` hvparam
 modify the PCI bus.

 All the above makes the current implementation error prone: an instance
 might not be able to boot if we explicitly add a NIC/Disk on a specific
 PCI slot that QEMU has already used for another device while parsing
 its command line options. Besides that, now, we want to use the SCSI bus
 as well so the above mechanism is insufficient. Here, we decide to put
 only disks and NICs on specific slots on the corresponding bus, and let
 QEMU put everything else automatically. To this end, we decide to let
 the first 12 PCI slots be managed by QEMU, and we start adding PCI
 devices (VirtIO block and network devices) from the 13th slot onwards.
 As far as the SCSI bus is concerned, we decide to put each SCSI
 disk on a different scsi-id (which corresponds to a different target
 number in SCSI terminology). The SCSI bus will not have any default
 reservations.


 How to support the theoretical maximum of devices, 16 disks and 8 NICs?

 By default, one could add up to 20 devices on the PCI bus; that is the
 32 slots of the PCI bus, minus the starting 12 slots that Ganeti
 allows QEMU to manage on its own. In order to by able to add
 more PCI devices, we add the new `kvm_pci_reservations` hvparam to
 denote how many PCI slots QEMU will handle implicitly. The rest will be
 available for disk and NICs inserted explicitly by Ganeti. By default
 the default PCI reservations will be 12 as explained above.


 How to keep track of the bus state of a running instance?

 To be able to hotplug a device, we need to know which slot is
 available on the desired bus. Until now, we were using the ``query-pci``
 QMP command that returns the state of the PCI buses (i.e., which devices
 occupy which slots). Unfortunately, there is no equivalent for the SCSI
 buses. We could use the ``info qtree`` HMP command that practically
 dumps in plain text the whole device tree. This makes it really hard to
 parse. So we decide to generate the bus state of a running instance
 through our local runtime files.


 What info should be kept in runtime files?

 Runtime files are used for instance migration (to run a QEMU process on
 the target node with the same configuration) and for hotplug actions (to
 update the configuration of a running instance so that it can be
 migrated). Until now we were using devices only on the PCI bus, so only
 each device's PCI slot should be kept in the runtime file. This is
 obviously not enough. We decide to replace the `pci` slot of Disk and
 NIC configuration objects, with an `hvinfo` dict. It will contain all
 necessary info for constructing the appropriate -device QEMU option.
 Specifically the `driver`, `id`, and `bus` parameters will be present to
 all kind of devices. PCI devices will have the `addr` parameter, SCSI
 devices will have `channel`, `scsi-id`, and `lun`. NICs and Disks will
 have the extra `netdev` and `drive` parameters correspondingly.


 How to deal with existing instances?

 Only existing instances with paravirtual devices (configured via the
 disk_type and nic_type hvparam) use the latest QEMU device model. Only
 these have the `pci` slot filled. We will use the existing
 _UpgradeSerializedRuntime() method to migrate the old runtime format
 with `pci` slot in Disk and NIC configuration objects to the new one
 with `hvinfo` instead. The new hvinfo will contain the old driver
 (either virtio-blk-pci or virtio-net-pci), the old id
 (hotdisk-123456-pci-4), the default PCI bus (pci.0), and the old PCI
 slot (addr=4). This way those devices will still be hotplug-able, and
 the instance will still be migrate-able. When those instances are
 rebooted, the hvinfo will be re-generated.


 How to support downgrades?

 There are two possible ways, both not very pretty. The first one is to
 use _UpgradeSerializedRuntime() to remove the hvinfo slot. This would
 require the patching of all Ganeti versions down to 2.10 which is practically
 imposible. Another way is to ssh to all nodes and remove this slot upon
 a cluster downgrade. This ugly hack would go away on 2.17 since we support
 downgrades only to the previous minor version.


 Configuration changes
 ---------------------

 The ``NIC`` and ``Disk`` objects get one extra slot: ``hvinfo``. It is
 hypervisor-specific and will never reach config.data. In case of the KVM
 Hypervisor it will contain all necessary info for constructing the -device
 QEMU option. Existing entries in runtime files that had a `pci` slot
 will be upgraded to have the corresponding `hvinfo` (see above).

 The new `scsi_controller_type` hvparam is added to denote what type of
 SCSI controller should be added to PCI bus if we have a SCSI disk.
 Allowed values will be `lsi`, `virtio-scsi-pci`, and `megasas`.
 We decide to use `lsi` by default since this is the one that QEMU
 adds automatically if not specified explicitly by an option.


 Hypervisor changes
 ------------------

 The current implementation verifies if a hotplug action has succeeded
 by scanning the PCI bus and searching for a specific device ID. This
 will change, and we will use the ``query-block`` along with the
 ``query-pci`` QMP command to find block devices that are attached to the
 SCSI bus as well.

 Up until now, if `disk_type` hvparam was set to `scsi`, QEMU would use the
 deprecated device model and end up using SCSI emulation, e.g.:

   ::

     -drive file=/var/run/ganeti/instance-disks/test:0,if=scsi,format=raw

 Now the equivalent, which will also enable hotplugging, will be to set
 disk_type to `scsi-hd`. The QEMU command line will include:

   ::

     -drive file=/var/run/ganeti/instance-disks/test:0,if=none,format=raw,id=disk-9e7c85f6-b6e5-4243
     -device scsi-hd,id=disk-9e7c85f6-b6e5-4243,drive=disk-9e7c85f6-b6e5-4243,bus=scsi.0,channel=0,scsi-id=0,lun=0


 User interface
 --------------

 The `disk_type` hvparam will additionally support the `scsi-hd`,
 `scsi-block`, and `scsi-generic` values. The first one is equivalent to
 the existing `scsi` value and will make QEMU emulate a SCSI device,
 while the last two will add support for SCSI pass-through and will
 require a real SCSI device on the host.

 .. vim: set textwidth=72 :
 .. Local Variables:
 .. mode: rst
 .. fill-column: 72
 .. End:
	==========
	KVM + SCSI
	==========

	.. contents:: :depth: 4

	This is a design document detailing the refactoring of device
	handling in the KVM Hypervisor. More specifically, it will use
	the latest QEMU device model and modify the hotplug implementation
	so that both PCI and SCSI devices can be managed.


	Current state and shortcomings
	==============================

	Ganeti currently supports SCSI virtual devices in the KVM hypervisor by
	setting the `disk_type` hvparam to `scsi`. Ganeti will eventually
	instruct QEMU to use the deprecated device model (i.e. -drive if=scsi),
	which will expose the backing store as an emulated SCSI device. This
	means that currently SCSI pass-through is not supported.

	On the other hand, the current hotplug implementation
	:doc:`design-hotplug` uses the latest QEMU
	device model (via the -device option) and is tailored to paravirtual
	devices, which leads to buggy behavior: if we hotplug a disk to an
	instance that is configured with disk_type=scsi hvparam, the
	disk which will get hot-plugged eventually will be a VirtIO device
	(i.e., virtio-blk-pci) on the PCI bus.

	The current implementation of creating the QEMU command line is
	error-prone, since an instance might not be able to boot due to PCI slot
	congestion.


	Proposed changes
	================

	We change the way that the KVM hypervisor handles block devices by
	introducing latest QEMU device model for SCSI devices as well, so that
	scsi-cd, scsi-hd, scsi-block, and scsi-generic device drivers are
	supported too. Additionally we refactor the hotplug implementation in
	order to support hotplugging of SCSI devices too. Finally, we change the
	way we keep track of device info inside runtime files, and the way we
	place each device upon instance startup.

	Design decisions
	================

	How to identify each device?

	Currently KVM does not support arbitrary IDs for devices; supported are
	only names starting with a letter, with max 32 chars length, and only
	including the '.', '_', '-' special chars. Currently we generate an ID
	with the following format: <device type>-<part of uuid>-pci-<slot>.
	This assumes that the device will be plugged in a certain slot on the
	PCI bus. Since we want to support devices on a SCSI bus too and adding
	the PCI slot to the ID is redundant, we dump the last two parts of the
	existing ID. Additionally we get rid of the 'hot' prefix of device type,
	and we add the next two parts of the UUID so the chance of collitions
	is reduced significantly. So, as an example, the device ID of a disk
	with UUID '9e7c85f6-b6e5-4243-b27d-680b78c6d203' would be now
	'disk-9e7c85f6-b6e5-4243'.


	Which buses does the guest eventually see?

	By default QEMU starts with a single PCI bus named "pci.0". In case a
	SCSI controller is added on this bus, a SCSI bus is created with
	the corresponding name: "scsi.0".
	Any SCSI disks will be attached on this SCSI bus. Currently Ganeti does
	not explicitly use a SCSI controller via a command line option, but lets
	QEMU add one automatically if needed. Here, in case we have a SCSI disk,
	a SCSI controller is explicitly added via the -device option. For the
	SCSI controller, we do not specify the PCI slot to use, but let QEMU find
	the first available (see below).


	What type of SCSI controller to use?

	QEMU uses the `lsi` controller by default. To make this configurable we
	add a new hvparam, `scsi_controller_type`. The available types will be
	`lsi`, `megasas`, and `virtio-scsi-pci`.


	Where to place the devices upon instance startup?

	The default QEMU machine type, `pc`, adds a `i440FX-pcihost`
	controller on the root bus that creates a PCI bus with `pci.0` alias.
	By default the first three slots of this bus are occupied: slot 0
	for Host bridge, slot 1 for ISA bridge, and slot 2 for VGA controller.
	Thereafter, the slots depend on the QEMU options passed in the command
	line.

	The main reason that we want to be fully aware of the configuration of a
	running instance (machine type, PCI and SCSI bus state, devices, etc.)
	is that in case of migration a QEMU process with the exact same
	configuration should be created on the target node. The configuration is
	kept in the runtime file created just before starting the instance.
	Since hotplug has been introduced, the only thing that can change after
	starting an instance is the configuration related to NICs and Disks.

	Before implementing hotplug, Ganeti did not specify PCI slots
	explicitly, but let QEMU decide how to place the devices on the
	corresponding bus. This does not work if we want to have hotplug-able
	devices and migrate-able VMs. Currently, upon runtime file creation, we
	try to reserve PCI slots based on the hvparams, the disks, and the NICs
	of the instance. This has three major shortcomings: first, we have to be
	aware which options modify the PCI bus which is practically impossible
	due to the huge amount of QEMU options, second, QEMU may change the
	default PCI configuration from version to version, and third, we cannot
	know if the extra options passed by the user via the `kvm_extra` hvparam
	modify the PCI bus.

	All the above makes the current implementation error prone: an instance
	might not be able to boot if we explicitly add a NIC/Disk on a specific
	PCI slot that QEMU has already used for another device while parsing
	its command line options. Besides that, now, we want to use the SCSI bus
	as well so the above mechanism is insufficient. Here, we decide to put
	only disks and NICs on specific slots on the corresponding bus, and let
	QEMU put everything else automatically. To this end, we decide to let
	the first 12 PCI slots be managed by QEMU, and we start adding PCI
	devices (VirtIO block and network devices) from the 13th slot onwards.
	As far as the SCSI bus is concerned, we decide to put each SCSI
	disk on a different scsi-id (which corresponds to a different target
	number in SCSI terminology). The SCSI bus will not have any default
	reservations.


	How to support the theoretical maximum of devices, 16 disks and 8 NICs?

	By default, one could add up to 20 devices on the PCI bus; that is the
	32 slots of the PCI bus, minus the starting 12 slots that Ganeti
	allows QEMU to manage on its own. In order to by able to add
	more PCI devices, we add the new `kvm_pci_reservations` hvparam to
	denote how many PCI slots QEMU will handle implicitly. The rest will be
	available for disk and NICs inserted explicitly by Ganeti. By default
	the default PCI reservations will be 12 as explained above.


	How to keep track of the bus state of a running instance?

	To be able to hotplug a device, we need to know which slot is
	available on the desired bus. Until now, we were using the ``query-pci``
	QMP command that returns the state of the PCI buses (i.e., which devices
	occupy which slots). Unfortunately, there is no equivalent for the SCSI
	buses. We could use the ``info qtree`` HMP command that practically
	dumps in plain text the whole device tree. This makes it really hard to
	parse. So we decide to generate the bus state of a running instance
	through our local runtime files.


	What info should be kept in runtime files?

	Runtime files are used for instance migration (to run a QEMU process on
	the target node with the same configuration) and for hotplug actions (to
	update the configuration of a running instance so that it can be
	migrated). Until now we were using devices only on the PCI bus, so only
	each device's PCI slot should be kept in the runtime file. This is
	obviously not enough. We decide to replace the `pci` slot of Disk and
	NIC configuration objects, with an `hvinfo` dict. It will contain all
	necessary info for constructing the appropriate -device QEMU option.
	Specifically the `driver`, `id`, and `bus` parameters will be present to
	all kind of devices. PCI devices will have the `addr` parameter, SCSI
	devices will have `channel`, `scsi-id`, and `lun`. NICs and Disks will
	have the extra `netdev` and `drive` parameters correspondingly.


	How to deal with existing instances?

	Only existing instances with paravirtual devices (configured via the
	disk_type and nic_type hvparam) use the latest QEMU device model. Only
	these have the `pci` slot filled. We will use the existing
	_UpgradeSerializedRuntime() method to migrate the old runtime format
	with `pci` slot in Disk and NIC configuration objects to the new one
	with `hvinfo` instead. The new hvinfo will contain the old driver
	(either virtio-blk-pci or virtio-net-pci), the old id
	(hotdisk-123456-pci-4), the default PCI bus (pci.0), and the old PCI
	slot (addr=4). This way those devices will still be hotplug-able, and
	the instance will still be migrate-able. When those instances are
	rebooted, the hvinfo will be re-generated.


	How to support downgrades?

	There are two possible ways, both not very pretty. The first one is to
	use _UpgradeSerializedRuntime() to remove the hvinfo slot. This would
	require the patching of all Ganeti versions down to 2.10 which is practically
	imposible. Another way is to ssh to all nodes and remove this slot upon
	a cluster downgrade. This ugly hack would go away on 2.17 since we support
	downgrades only to the previous minor version.


	Configuration changes
	---------------------

	The ``NIC`` and ``Disk`` objects get one extra slot: ``hvinfo``. It is
	hypervisor-specific and will never reach config.data. In case of the KVM
	Hypervisor it will contain all necessary info for constructing the -device
	QEMU option. Existing entries in runtime files that had a `pci` slot
	will be upgraded to have the corresponding `hvinfo` (see above).

	The new `scsi_controller_type` hvparam is added to denote what type of
	SCSI controller should be added to PCI bus if we have a SCSI disk.
	Allowed values will be `lsi`, `virtio-scsi-pci`, and `megasas`.
	We decide to use `lsi` by default since this is the one that QEMU
	adds automatically if not specified explicitly by an option.


	Hypervisor changes
	------------------

	The current implementation verifies if a hotplug action has succeeded
	by scanning the PCI bus and searching for a specific device ID. This
	will change, and we will use the ``query-block`` along with the
	``query-pci`` QMP command to find block devices that are attached to the
	SCSI bus as well.

	Up until now, if `disk_type` hvparam was set to `scsi`, QEMU would use the
	deprecated device model and end up using SCSI emulation, e.g.:

	::

	-drive file=/var/run/ganeti/instance-disks/test:0,if=scsi,format=raw

	Now the equivalent, which will also enable hotplugging, will be to set
	disk_type to `scsi-hd`. The QEMU command line will include:

	::

	-drive file=/var/run/ganeti/instance-disks/test:0,if=none,format=raw,id=disk-9e7c85f6-b6e5-4243
	-device scsi-hd,id=disk-9e7c85f6-b6e5-4243,drive=disk-9e7c85f6-b6e5-4243,bus=scsi.0,channel=0,scsi-id=0,lun=0


	User interface
	--------------

	The `disk_type` hvparam will additionally support the `scsi-hd`,
	`scsi-block`, and `scsi-generic` values. The first one is equivalent to
	the existing `scsi` value and will make QEMU emulate a SCSI device,
	while the last two will add support for SCSI pass-through and will
	require a real SCSI device on the host.

	.. vim: set textwidth=72 :
	.. Local Variables:
	.. mode: rst
	.. fill-column: 72
	.. End: