Hugepages and KVM

I've seen the benefits of hugepages before when setting up Oracle and MySQL, but while doing some research I found an article on how to use them with KVM. However, patching /sbin/start_udev and creating an init script to make sure it stays patched just seems like a horrendous idea just to get hugetlbfs mounted on /dev/hugepages. My non-intrusive method is as follows:

1) teach MAKEDEV how to "create the directory" /dev/hugepages on boot. Actually it is creating an additional /dev/null device at /dev/hugepages/null, but it should be harmless to have multiple "null" (major 1, minor 3) devices and also harmless to mount on top of it.

echo 'c $ALLWRITE 1 3 1 1 hugepages/null' > /etc/makedev.d/01hugepages

2) tell udev to create it on boot if needed:

echo 'hugepages/null' > /etc/udev/makedev.d/52-hugepages.nodes

3) tell udev what the right permissions are for it:

echo 'KERNEL=="hugepages*", OWNER="root", GROUP="root", MODE="0775"' > /etc/udev/rules.d/52-hugepages.rules

4) Under CentOS/RHEL run "huge_page_setup_helper.py" to get your hugepages setup

5) Set the hugetlbfs to be mounted on boot:

echo 'hugetlbfs /dev/hugepages hugetlbfs defaults 0 0' >> /etc/fstab

That's it! After a reboot, you can check that hugepages are setup with "sysctl vm.nr_hugepages" and "grep -i huge /proc/meminfo" and check that hugetlbfs is mounted with "mount | grep huge".

Check /proc/meminfo once your KVM guests start to make sure the number of free pages decreases. If not confirm your guest's XML file has "<memoryBacking><hugepages/></memoryBacking>" below the "<currentmemory>" section and that they have "-mem-prealloc -mem-path /dev/hugepages/libvirt/qemu" in the qemu-kvm command line (it should be auto-set by libvirt).