KISScluster: goals, environment, building blocks
Goals
Have two machines that can run KVM guests and are highly available. That means your VMs are statically assigned to run on one of the physical machines, and in case that machine goes down, the VMs automatically boot up on the other node without manual intervention.
I'd like to implement that according to the KISS principle. The other solutions that I've looked at are way too complex for my taste, see my original rant that started all this. That means to solve the common problems, and maybe let the uncommon ones unsolved (or rather, require manual intervention if they happen). It also directly leads to the next paragraph.
I'd also like to be able to use as much stuff that I already know as possible, and try not to learn yet another piece of software if it is avoidable. This is lazyness of the good kind, as any BOFH only has so much brain capacity, and using more of the well-known stuff keeps that know-how fresh, while only using special solutions seldomly means that when the shit hits the fan, you go searching for the documentation.
Oh: no pointy-clicky web interface. All command-line and config file driven.
Environment
A cluster consists of two nodes.
They have two network interfaces each:
Production, this goes to the "real network", or LAN.
Storage, which is a direct cable between the ports on the nodes, and carries the DRBD traffic plus a bit of cluster communication.
Actually, this is for the purposes of clustering; nothing prevents more interfaces (e.g. for hooking up VMs to various networks), but those additional interfaces are not used in the clustering decisions.
On the production network, an external IP address must be reachable via ICMP pings. The idea is that this will be the switch where the production interfaces are connected.
Building blocks/Tools
DRBD
DRBD "can be understood as network based raid-1", as its creators say. This is the storage layer that makes sure that VM images are kept consistent on both cluster nodes in real time.
KVM
KVM is the kernel-based Linux virtualization solution that makes it possible to run virtual machines.
(Nothing would prevent Xen from running on KISScluster, except the missing glue)
systemd
systemd, with its dependency handling and easy config file format, is the component which converts the cluster daemon's decisions on state into running services, KVM instances, DRBD devices and whatnot.
Ansible
Ansible is an automation solution, and it's the component which drives both the installation of a KISScluster and the management of the VMs running on it.
Debian
Debian is my Linux distro of choice, it's stable and has all the requirements of KISScluster packaged. This is not to say you couldn't run it on other distros, it's just more work.