Mageia2 on EC2: Boarding procedures

This is my second post on getting Mageia2 running on Amazon Web Services' Elastic Compute Cloud. See my first post in the series for an overview.

The first step to creating a Mageia2 install on EC2 is to have a local Mageia2 system as your seed setup. Why? Because you must use urpmi, the Mageia package installer. It is equivalent to apt-get in Ubuntu or yum in CentOS/RHEL/Amazon Linux. Yes, you can use rpm to install individual packages (and we will) but urpmi is what talks to the media sets (e.g. repositories) and makes sure you have all your dependencies installed. Besides the man page and the urpmi page on the Mageia wiki, I found a good quick reference guide that helped.

I'll not cover the setup of your seed Mageia2 system here - the installation is was breeze. I used VirtualBox under Windows 7 to install from the Dual-arch ISO CD, but you could likely use any old PC you have laying around and install however you like - USB key, Live CD, whatever.

Also, I'm not doing to document getting the EC2 command line tools working on your seed system. Installing java was simple ("urpmi java" I believe) and getting the API tools and AMI tools installed and configured is well documented by Amazon and plenty of others.

Once you have a functioning Mageia2 system and working EC2 AMI and API tools, then we're ready to begin.

The initial steps we'll be following are a mix between the Mageia chroot install and the official documentation on how to create an EC2 instance-store backed AMI. Another major factor was choosing a kernel. Now any good distribution ships with it's own kernel, and Mageia is no different. And of course you can use your own kernel in EC2. The most efficient way to do this is to use the PV-GRUB AKI provided by AWS to load the kernel that is present on your instance's disk, which is what we'll do.

For the most part, all of this went well after some trial-and-error. However I did run across a few issues:

1. Make sure you create a big enough loopback device. I started with 2GB and while it was enough for the base install, it wasn't enough once I started adding other packages later. My docs below use 8GB. The maximum is 10GB.
2. Make sure you choose the right PV-GRUB AKI (more on kernels in a moment)
3. Use a gzip compressed kernel, not a xz compressed kernel
4. Choose the right mirror for urpmi.addmedia (distro.ibiblio.org was extra slow for me - mirrors.kernel.org was much faster)
5. Make sure you install the critical packages. Without dhcp-client, you won't get your IP address and without an IP address, you're sunk. Same goes for sshd and sudo.
6. At this stage, I didn't pay attention to the ssh key pairs built into the EC2 provisioning system. I baked a new public key of the "mageia" user into the install.

When choosing a PV-GRUB AKI, the AWS documentation explains:

You must choose an AKI with "hd0" in the name if you want a raw or unpartitioned disk image (most images). Choose an AKI with "hd00" in the name if you want an image that has a partition table.

Since I am doing a direct mke2fs of the loopback image, it doesn't have a partition table. However, I was using the wrong PV-GRUB AKI, the one for loopbacks with partition tables, resulting in nothing working with error messages that were confusing. I'm sure you could fdisk your loopback image and create partitions if you want, but I didn't see it as necessary as I often use the other ephemeral disks for swap, etc. So, I had to use a "hd0" PV-GRUB to get it to work. 

After I got PV-GRUB set the system still wouldn't load. The error message from the system console was:

ERROR Invalid kernel: xc_dom_probe_bzimage_kernel: unknown compression format

In chatting with the very helpful "tmb" from the #mageia IRC channel on freenode, I was able to overcome this issue. The problem is that all Mageia kernels are xz compressed by default which is supported just fine by regular grub. However, the PV-GRUB AKI I was using didn't support xz compressed kernels. tmb provided me with a gzip compressed kernel to bootstrap by first EC2 instance running Mageia2. Thanks again tmb - without your help, I wouldn't have gotten this working!

After the kernel loaded, it was just straightforward trial and error troubleshooting till I was able to login. Here's the sanitized steps I used to get my first Mageia2 instance-store backed EC2 AMI uploaded:

# preparation
export PATH=$PATH:/sbin:/usr/sbin
# create a working directory
mkdir $HOME/ec2
# create ssh public key
ssh-keygen -t rsa -f $HOME/ec2/mageia -C "mageia@ec2" -P ""
# setup the image
dd if=/dev/zero of=mageia2-instance-store-v1.img bs=1M count=8192
# format it
mke2fs -F -j $HOME/ec2/mageia2-instance-store-v1.img

# everything forward needs to be done as root 
sudo bash -o vi

# mount the image for chroot
export MAGEIA_PUB_KEY=$HOME/ec2/mageia.pub
export CHRDIR=$HOME/ec2/loop
# mount the chroot location
mount -o loop $HOME/ec2/mageia2-instance-store-v1.img $CHRDIR

# create the minimum devices
mkdir $CHRDIR/dev
/sbin/makedev $CHRDIR/dev console
/sbin/makedev $CHRDIR/dev null
/sbin/makedev $CHRDIR/dev zero

# setup the minimum filesystems
mkdir $CHRDIR/etc
cat > $CHRDIR/etc/fstab << EOF
/dev/xvda1 /         ext3    defaults        1 1
none       /dev/pts  devpts  gid=5,mode=620  0 0
none       /dev/shm  tmpfs   defaults        0 0
none       /proc     proc    defaults        0 0
none       /sys      sysfs   defaults        0 0
EOF

# add required /proc filesystem
mkdir $CHRDIR/proc
mount -t proc none $CHRDIR/proc

# choose the best/fastest mirror
GET http://mirrors.mageia.org/api/mageia.2.x86_64.list | grep country=US
# setup the urpmi media locations in the chroot
urpmi.addmedia --distrib --urpmi-root $CHRDIR http://mirrors.kernel.org/mageia/distrib/2/x86_64
# install the minimum packages
urpmi --auto --urpmi-root $CHRDIR kernel-server basesystem urpmi locales-en sshd sudo dhcp-client

# MASSIVE HACK TIME
#
# kernel from tmb:
# http://tmb.mine.nu/Mageia/2/ec2/
#
mkdir tmb
pushd tmb
curl -O http://tmb.mine.nu.nyud.net/Mageia/2/ec2/kernel-server-3.4.18-1.mga2-1-1.mga2.x86_64.rpm
curl -O http://tmb.mine.nu.nyud.net/Mageia/2/ec2/kmod-7-7.mga2.x86_64.rpm
popd
# install custom kernel
rpm --root=$CHRDIR -Uhv tmb/*.rpm

# insure the new ramdisk is created properly
chroot $CHRDIR
cd /boot
mkinitrd initrd-3.4.18-server-1.mga2.img.a 3.4.18-server-1.mga2
exit

# set the kernel to load on boot
cat > $CHRDIR/boot/grub/menu.lst << EOF
default=0
timeout=0
title linux
  root (hd0)
  kernel /boot/vmlinuz-server ro root=/dev/xvda1 console=hvc0 BOOT_IMAGE=linux-nonfb
  initrd /boot/initrd-server.img
EOF

# configure the chroot network for ec2
cat > $CHRDIR/etc/sysconfig/network-scripts/ifcfg-eth0 << EOF
DEVICE=eth0
BOOTPROTO=dhcp
ONBOOT=yes
TYPE=Ethernet
USERCTL=yes
PEERDNS=yes
IPV6INIT=no
EOF
cat > $CHRDIR/etc/sysconfig/network << EOF
NETWORKING=yes
CRDA_DOMAIN=US
EOF

# configure ssh
test -f $CHRDIR/etc/ssh/sshd_config.orig || cp -p $CHRDIR/etc/ssh/sshd_config $CHRDIR/etc/ssh/sshd_config.orig
cat $CHRDIR/etc/ssh/sshd_config.orig |
    sed -e 's/^#UseDNS yes/UseDNS no/g' |
    sed -e 's/^PermitRootLogin no/PermitRootLogin without-password/g' > $CHRDIR/etc/ssh/sshd_config
# setup mageia account
chroot $CHRDIR /usr/sbin/useradd --create-home --home /home/mageia --shell /bin/bash mageia
mkdir --mode=0700 $CHRDIR/home/mageia/.ssh
(umask 0077; touch $CHRDIR/home/mageia/.ssh/authorized_keys)
cat $MAGEIA_PUB_KEY >> $CHRDIR/home/mageia/.ssh/authorized_keys
echo "set -o vi" >> $CHRDIR/home/mageia/.bashrc
chown -Rh 500:500 $CHRDIR/home/mageia/.ssh
(umask 0227; echo "mageia ALL=(ALL) NOPASSWD:ALL" > $CHRDIR/etc/sudoers.d/mageia)

# dismount the chroot
umount $CHRDIR/proc
umount -d $CHRDIR

# setup for EC2
export EC2_ID=[aws account id #]
export EC2_PRIVATE_KEY=[location of private key]
export EC2_CERT=[location of signing cert]
export EC2_ACCESS=[iam access key]
export EC2_SECRET=[iam secret key]

BUCKETNAME="$EC2_ID-mageia2-instance-store-v1"
# create S3 bucket
# you can use the AWS Console instead of s3cmd, if you like
s3cmd mb s3://$BUCKETNAME
# where to put the bundle parts
pushd $HOME/ec2
mkdir mageia2-instance-store-v1
# create the AMI bundle
# AKI for pv-grub-hd0_1.03-x86_64.gz partitionless PV-GRUB in US-East-1
AKIID="aki-88aa75e1"
ec2-bundle-image -i mageia2-instance-store-v1.img -d mageia2-instance-store-v1 -r x86_64 --kernel $AKIID -k $EC2_PRIVATE_KEY -c $EC2_CERT -u $EC2_ID
# put it on S3
ec2-upload-bundle -b $BUCKETNAME -m mageia2-instance-store-v1/mageia2-instance-store-v1.img.manifest.xml -a $EC2_ACCESS -s $EC2_SECRET
# register it
ec2-register $BUCKETNAME/mageia2-instance-store-v1.img.manifest.xml -n mageia2-instance-store-v1
popd