KVM and libvirt best practices

Submitted by dag on Thu, 2009/12/17 - 11:23

Setting up my first environment using KVM on RHEL, I am disappointed about the lack of best practices and standardization regarding KVM on GFS clustering. First thing that should be obvious when sharing a complex and flexible piece of software online for production use, don't allow too many things without proper best practices. Otherwise people will do things differently and possibly for the wrong reasons.

For example, while with ESX there are proper locations for putting disks and VM config files on a shared storage, with libvirt both are separated and to me its unclear what part of the default paths should reside on shared storage and what-not. I guess, like many others, I will have to learn it the hard way. What a shame.

It is disappointing that proper conventions are lacking for:

  • network bridges (eg. don't use vnetX for bridge names like many docs do)
  • disk images (eg. using separate system and data disks)
  • hardware/driver (eg. using virtio-net and virtio-blk better or not ?)
  • kernel options (eg. should we be using noop elevator, different clock source, divider ?)
  • architecture choice (eg. 32bit will probably be sufficient for most services, 64bit only when needed)
  • multipath naming (eg. userfriendly names ?)
  • LVM naming (eg. use unique names to prevent conflicts)
  • GFS and cluster configuration (eg. what conventions are used inside RHEV ?)

For early adopters of libvirt, being able to migrate to RHEV in the future is an important prerequisite. For Red Hat having an easy migration path from a (custom) libvirt setup to RHEV without little hassle is interesting as well.

Have you been setting up libvirt environments, share your best practices here ! If you found interesting links to information, let us know too. Let's fill this void.

what I've learned so far

- use libvirt! You can run kvm directly from the command line, wrap this into a script and reinvent the wheel. But you will not be able to use any management tools. If you don't need a GUI like VirtManager, there is virsh. The only caveat so far, there is no support for that fancy VESA-to-ncurses console wrapper (which allows text-mode installs within a ssh session (so no serial ttys are required)
- use virtio; if available, it's fast (and stable enough for me, since I've upgraded to a recent Kernel!)
- use labels to mount file systems, so that you can easily revert from virtio to scsi emulation (this is what rhev does)
- I name my bridge kvm0 e.g.; it should be obvious to anyone else that this is a special host.
- I store my VMs at /srv/kvm/ and configs in /etc/libvirt/; though, separating config from data looks somewhat odd on a single host environment.
- I'm using some routing tricks, so that I do not loose any ip addresses on my kvm0 bridge (which is necessary with some hosting providers, but this is another story that needs an extra post)
- I prefer to run 64bit clients because the 4 GB limit hits VMs the same way it hits physical machines
- naming storage: I think that's not important, but I derive the naming from the mount points/primary use in general. but: shall I put a (GPT) partition table inside of an iscsi volume?

KVM and RHEL clusters are not supported at the moment

Just thought you should know. According to RHEL, 5.4 KVM Clustering is a tech preview. We were trying to get it setup and ended up giving up on getting it to work due to odd issue after odd issue. Looks like Xen is it for now.

Yes, I know. We're looking at

Yes, I know. We're looking at clustering the hosts, not the guests :-) But clustering guests looks quite interesting as well.

Virtual Machine configuration

Personally I think the location of the configuration files for libvirt (and KVM) are troubling. The virtual machine configuration files are locally installed on the KVM host (/etc/libvirt), unlike the way VMware does it. VMware combines the virtual disk images (.vmdk) and configuration files (.vmx .vmxf) inside the same folder. (note: If you even name the folder with the extension .vmwarevm it is a contained vm for use on Fusion). This way it is easy to start a virtual machine on another node, without the need to move configuration files or migrate the node (you only need to add the .vmx to the server inventory). What would be a best practice for KVM+Libvirt?

unlike the way VMware does it ... now

VMware used to have local config files in the 1.x and 2.x versions of ESX. Version 3.x and up have centrally stored config files, which is much cleaner and easier for anything related to VM mobility (live migration, automatic restart on different host in case of host failure, ...)

Multipath: Don't use "user friendly names"

For multipathing, I recommend turning so-called user friendly names off. I'm talking about the user_friendly_names setting in multipath.conf. As far as I can tell, user_friendly_names' default value is actually "no", but Red Hat's package overrides this, for unknown reasons.

I find "user friendly names" very unfriendly: In an already complicated coctail of device mapper, LVM, etc, it's not useful to have LUNs aliased. Setting user_friendly_names to "no" means somewhat long device names, but much less confusion, and consistent naming across servers which might share LUNs.

Very nice post, by the way. I agree with all the recommendations put forward so far. Especially the one about using unique names for LVM elements, e.g. a volume group should be called HOSTNAME_rootvg, instead of just rootvg; this makes things much less confusing when you have to work with guest file systems from the host.

I agree, LVM names are

I agree, LVM names are sufficient for most manipulations. Everything underneath is opaque to the user, except maybe when tracing something and then you'll be happy to using something unique, rather than something configurable (and possibly confusing) :-)

libvirt changes iptables

One thing that annoyed me very much about libvirt is that it will change your iptables configuration as soon as you start a routed/natted network.

I patched my libvirt so that I can disable that. For me the default behavior is a no go because I need to control my firewall settings in one place.

KVM tip: data corruption

If you have a cluster of KVM servers, be careful - there is no locking across the cluster - AT ALL(!)

If you try to start the same virtualised host on two servers, nothing will stop you. You'll get silent data corruption and eventually you will notice one or both VMs panic.

Re: KVM tip: data corruption

> If you have a cluster of KVM servers, be careful - there is no locking across the cluster

Hi. What tool do you advice so avoid this problem ?

I've found pacemaker can be a

I've found pacemaker can be a good solution.

libvirt offers

libvirt offers locking/fencing