Building Icinga on Amazon Linux

As part of a datacenter to cloud migration for $WORK I recently needed to move our Icinga install to AWS. However, there is no current Icinga package for Amazon Linux. There exists a set of rpms when using rpmforge, but Amazon Linux is already setup for EPEL instead. The way forward? Build the rpms.

It actually wasn't very hard at all. I spun up a builder instance, built the rpms, copied them to an S3 bucket and then installed them on a fresh instance. Here's the steps:

1. prep the builder instance as root, getting all the dependencies installed.

yum -y --enablerepo=epel install fedora-packager rpmdevtools
yum -y --enablerepo=epel install gcc gd-devel zlib-devel libpng-devel libjpeg-devel libdbi-devel perl-ExtUtils-Embed
yum -y --enablerepo=epel install httpd-devel php php-devel php-gd php-ldap php-pdo php-xml php-pear
useradd makerpm

2. Never build an rpm as root. Use the "makerpm" user:

sudo su - makerpm
rpmdev-setuptree
cd rpmbuild/
cat > .rpmmacros << EOF
%_topdir      %(echo $HOME)/rpmbuild
%_smp_mflags  -j3
%__arch_install_post   /usr/lib/rpm/check-rpaths   /usr/lib/rpm/check-buildroot
EOF

3. Build icinga-core rpms:

sudo su - makerpm
cd rpmbuild/SOURCES/
wget https://github.com/Icinga/icinga-core/releases/download/v1.11.0/icinga-1.11.0.tar.gz
tar zxvf icinga-1.11.0.tar.gz icinga-1.11.0/icinga.spec
mv icinga-1.11.0/icinga.spec ../SPECS/
rmdir icinga-1.11.0
cd ..
rpmbuild -ba --define="_vendor redhat" SPECS/icinga.spec

4. Build icinga-web rpms:

sudo su - makerpm
cd rpmbuild/SOURCES/
wget https://github.com/Icinga/icinga-web/releases/download/v1.11.0/icinga-web-1.11.0.tar.gz
tar zxvf icinga-web-1.11.0.tar.gz icinga-web-1.11.0/icinga-web.spec
mv icinga-web-1.11.0/icinga-web.spec ../SPECS/
rmdir icinga-web-1.11.0
cd ..
rpmbuild -ba --define="_vendor redhat" SPECS/icinga-web.spec

This process leans heavily on the Build Icinga RPMS page which is based on the How to create an RPM package page.

Building a perl package for Amazon Linux, CentOS or RHEL

A previous post detailed on how to build a .deb file for a perl module on Ubuntu. However, I needed the updated module as a .rpm on an Amazon Linux system, so I created the procedure for that OS as well. It was a lot easier than I thought it would be.

# install general dependencies of Net::Amazon::EC2
sudo yum --enablerepo=epel install \
    perl-Net-Amazon-EC2 perl-File-Slurp perl-DBI perl-DBD-MySQL \
    perl-Net-SSLeay perl-IO-Socket-SSL perl-Time-HiRes perl-Params-Validate \
    perl-Date-Manip perl-DateTime perl-DateTime-Format-ISO8601 \
    ca-certificates

# install stuff required for the build
sudo yum --enablerepo=epel install cpanspec rpm-build.x86_64
sudo yum --enablerepo=epel install perl-Test-Exception perl-CPAN

# generate .spec file
PACKAGER="John Smith <jsmith@example.com>"
cpanspec --packager "$PACKAGER" -v Net::Amazon::EC2

# get the source
mkdir -p rpmbuild/SOURCES
wget -Orpmbuild/SOURCES/Net-Amazon-EC2-0.23.tar.gz \
    http://search.cpan.org/CPAN/authors/id/M/MA/MALLEN/Net-Amazon-EC2-0.23.tar.gz

# don't set these or tests will fail
unset AWS_ACCESS_KEY_ID
unset SECRET_ACCESS_KEY

# do the build
rpmbuild -bb perl-Net-Amazon-EC2.spec

Building a perl package for Ubuntu

I recently had a need to update an Ubuntu perl module for a client at $WORK. Specifically I needed to update the Net::Amazon::EC2 module to version 0.23 to support tagging of AWS resources. Looking at CPAN, the library went un-maintained for close to 3 years before a new author picked it up. Things are looking up though as it seems this fall's release of Ubuntu will likely have the new package.

Still, this project couldn't wait till then. Thankfully, I found a good article on how to roll your own Debian packages for perl modules. There were only 2 hangups. First, the "make test" step fails because it can't do the authentication for AWS. I've disabled the "make test" step in my procedure by exporting the variable DEB_BUILD_OPTIONS="nocheck". After that the module package builds fine, but there's no dependencies set so when installing it, it can't be used without a few rounds of dependency hell. To solve this, I just borrowed the dependency list from the current 0.14 version of the package. The borrowed dependencies were "perl, libmoose-perl, libparams-validate-perl, liburi-perl, libwww-perl, libxml-simple-perl".

Note, this is not a well-tuned setup. There's lots of peculiarities that the dh-make-perl and debuild process wants that can be glossed over. For example - if you don't have a pgp key for your $DEBEMAIL address in you private keychain, the .deb package will still build, but it won't be signed. This may or may not be a problem for you, YMMV.

Here's the steps I took for building the .deb:

#
# install some packages
#
sudo apt-get update
sudo apt-get install dh-make-perl libmodule-build-perl debhelper devscripts
sudo apt-file update

#
# do the build
#

# note: disable "make test" so you don't need valid AWS keys
# (I couldn't make it work because even though I set the end variables,
# I think dh-make-build cleans the environment before running make)
export DEB_BUILD_OPTIONS="nocheck"

export DEBFULLNAME="John Smith"
git config --global user.name $DEBFULLNAME
export DEBEMAIL="jsmith@example.com"
git config --global user.email $DEBEMAIL

wget http://search.cpan.org/CPAN/authors/id/M/MA/MALLEN/Net-Amazon-EC2-0.23.tar.gz
mkdir builder
cd builder
# debian is particular how original tar files are named
ln -s ../Net-Amazon-EC2-0.23.tar.gz libnet-amazon-ec2-perl_0.23.orig.tar.gz
tar -pzxvf libnet-amazon-ec2-perl_0.23.orig.tar.gz
cd Net-Amazon-EC2-0.23/
dh-make-perl --depends "perl, libmoose-perl, libparams-validate-perl, liburi-perl, libwww-perl, libxml-simple-perl" .
debuild
cd ..
# viola, a .deb package
dpkg --info libnet-amazon-ec2-perl_0.23-1_all.deb

Forcing a service to stop during Chef execution

My latest project at work has been to get a Chef installation going for a client who is migrating a site from a traditional datacenter to Amazon Web Services. And part of the unique snowflake of a deploy process for this application is that it must be restarted on every deploy because the config files are included in the deploy - apache's httpd.conf, tomcat's server.xml, etc. etc. Most importantly is that if you don't stop the service before the first deploy, it will most likely not stop at all because it makes significant changes to the configuration (pid file locations, etc.). NOTE: I would NOT have designed the deploy process this way, but I'm stuck with it for  now. So, in the meantime, this is how I deal with it:

1) Right after the service definition in the default recipe (e.g. apache2::default), create a semaphore file that forces the service to stop if this file doesn't exist (it shouldn't):

file "/tmp/gather-apache2-semaphore" do
    mode 00644
    owner "root"
    group "root"
    action :create
    content "must stop apache"
    notifies :stop, resources(:service => "apache2"), :immediately
end

2) Make sure your application deploy recipe notifies a restart of the service (e.g. apache2::example_app_config):

ruby_block "copy_httpd.conf" do
    block do
         FileUtils.copy_file("#{deploy_directory}/example_app/build_out/dist/conf/httpd.conf","/etc/apache2/httpd.conf")
    end
    notifies :restart, resources(:service => "apache2")
end

and/or if you are calling this from another recipe (e.g. example_app::deploy):

include_recipe "apache2"

bash "copy_example_app_htdocs.tar" do
    user "root"
    cwd "#{deploy_directory}/example_app/build_out/dist"
    code <<-EOH
        tar --extract --file=example_app_htdocs.tar --directory /etc/apache2/htdocs
    EOH
    only_if { ::File.exists?("/etc/apache2/htdocs") }
    notifies :restart, resources(:service => "apache2")
end

3) Finally, after all the key recipes have been run to deploy your unique snowflake, the service will have been restarted and you can clean-up your semaphore that will force the service to be restarted next time you deploy.

file "/tmp/gather-apache2-semaphore" do
    action :delete
end

Now the service will be guaranteed to be stopped while your application deploying.

Mageia2 on EC2: Cruising Altitude

This is my fourth post on getting Mageia2 running on Amazon Web Services' Elastic Compute Cloud. See my first post in the series for an overview.

In the last post, I addressed the problem of having only a test kernel by tweaking the Mageia kernel SRPM and creating a gzipped kernel that can be used with the version of PV-GRUB supplied by Amazon. Now I'll walk through the steps of building an EBS backend instance instead of an instance-store backed instance.

You need a working Mageia setup on an instance-store backed instance before you can create the EBS backed one. Just launch the AMI created in a previous step and then attach a 32GB EBS volume to it. Using the EC2 API tools, you attach the volume like this:

SIZE=32
TARGETAZ=us-east-1a
INSTID=i-09abcdef

CMD=($(ec2-create-volume --size $SIZE --availability-zone $TARGETAZ --type standard))
VOLID=${CMD[1]}
ec2-attach-volume $VOLID --instance $INSTID --device /dev/sdg


You will also need some other components:

1. "kernel-server" RPM created in a last post.
2. A copy of ec2-get-ssh.sh for the mageia user

The second component is so you don't have to embed passwords in your AMI, but instead uses ssh public keys that are imported to (or generated by) AWS.

Another difference is that we add the kernel to the skip.list for upgrades, as we don't want to get a non-gzipped kernel. So, here's the steps for setting it up:

mkdir $HOME/ec2

# everything forward needs to be done as root 
sudo bash -o vi
cd $HOME/ec2
export PATH=$PATH:/sbin:/usr/sbin

# setup the filesystem
/sbin/mkfs -t ext4 /dev/xvdg

# mount the image for chroot
export CHRDIR=$HOME/ec2/loop
mount /dev/xvdg $CHRDIR

# create the minimum devices
mkdir $CHRDIR/dev
/sbin/makedev $CHRDIR/dev console
/sbin/makedev $CHRDIR/dev null
/sbin/makedev $CHRDIR/dev zero

# setup the minimum filesystems
mkdir $CHRDIR/etc
cat > $CHRDIR/etc/fstab << EOF
/dev/xvda1 /         ext3    defaults        1 1
none       /dev/pts  devpts  gid=5,mode=620  0 0
none       /dev/shm  tmpfs   defaults        0 0
none       /proc     proc    defaults        0 0
none       /sys      sysfs   defaults        0 0
EOF

# add required /proc filesystem
mkdir $CHRDIR/proc
mount -t proc none $CHRDIR/proc

# choose the best/fastest mirror
GET http://mirrors.mageia.org/api/mageia.2.x86_64.list | grep country=US
# setup the urpmi media locations in the chroot
urpmi.addmedia --distrib --urpmi-root $CHRDIR http://mirrors.kernel.org/mageia/distrib/2/x86_64
# install the minimum packages
urpmi --auto --urpmi-root $CHRDIR basesystem urpmi locales-en sshd sudo dhcp-client

# MASSIVE HACK TIME
rpm --root=$CHRDIR -Uhv custom-kernel/kernel-server-3.3.8-2.mga2-1-1.mga2.x86_64.rpm

# cleanup desktop kernel
chroot $CHRDIR
urpme kernel-desktop-3.3.8-2.mga2-1-1.mga2
rm -f initrd-desktop.img  vmlinuz-desktop 
# confirm there's a good initrd
cd /boot
stat initrd-3.3.8-server-2.mga2.img
mkinitrd initrd-3.3.8-server-2.mga2.img 3.3.8-server-2.mga2
exit

# set the kernel to load on boot
cat > $CHRDIR/boot/grub/menu.lst << EOF
default=0
timeout=0
title linux
  root (hd0)
  kernel /boot/vmlinuz-server ro root=/dev/xvda1 console=hvc0 BOOT_IMAGE=linux-nonfb
  initrd /boot/initrd-server.img
EOF

# do not upgrade the kernel, until upstream fixes the xz/gz issue
test -f $CHRDIR/etc/urpmi/skip.list || cp -p $CHRDIR/etc/urpmi/skip.list $CHRDIR/etc/urpmi/skip.list.orig
cat > $CHRDIR/etc/urpmi/skip.list << EOF
# Here you can specify the packages that won't be upgraded automatically
# for example, to exclude all apache packages :
# /^apache/
/^kernel/
EOF

# configure the chroot network for ec2
cat > $CHRDIR/etc/sysconfig/network-scripts/ifcfg-eth0 << EOF
DEVICE=eth0
BOOTPROTO=dhcp
ONBOOT=yes
TYPE=Ethernet
USERCTL=yes
PEERDNS=yes
IPV6INIT=no
EOF
cat > $CHRDIR/etc/sysconfig/network << EOF
NETWORKING=yes
CRDA_DOMAIN=US
EOF

# configure ssh
test -f $CHRDIR/etc/ssh/sshd_config.orig || cp -p $CHRDIR/etc/ssh/sshd_config $CHRDIR/etc/ssh/sshd_config.orig
cat $CHRDIR/etc/ssh/sshd_config.orig |
    sed -e 's/^#UseDNS yes/UseDNS no/g' |
    sed -e 's/^PermitRootLogin no/PermitRootLogin without-password/g' > $CHRDIR/etc/ssh/sshd_config
# create login account
chroot $CHRDIR /usr/sbin/useradd --create-home --home /home/mageia --shell /bin/bash mageia
(umask 0227; echo "mageia ALL=(ALL) NOPASSWD:ALL" > $CHRDIR/etc/sudoers.d/mageia)

# setup ssh public key
cp ec2-get-ssh $CHRDIR/etc/rc.d/init.d/ec2-get-ssh
chmod 0750 $CHRDIR/etc/rc.d/init.d/ec2-get-ssh
chown root:root $CHRDIR/etc/rc.d/init.d/ec2-get-ssh
chroot $CHRDIR /sbin/chkconfig ec2-get-ssh on

# dismount the chroot
umount $CHRDIR/proc
umount -d $CHRDIR

Now that the EBS volume is all set, it needs to be snapshotted and registered as an AMI. Here's what you do:

ec2-detach-volume $VOLID --instance $INSTIT--device /dev/sdg

# create a snapshot
CMD=($(ec2-create-snapshot --description "Mageia 2" $EBSVOL))
SNAPID=${CMD[1]}

# create AMI
AKIID="aki-88aa75e1"
ec2-register --name "Mageia 2" --description "Mageia 2" \
    --architecture x86_64 --root-device-name /dev/sda1 \
    --block-device-mapping /dev/sda1=$SNAPID --kernel $AKIID

Now you're ready to launch your EBS back Mageia2 Linux instance! Enjoy!

Mageia2 on EC2: Stormy Weather

This is my third post on getting Mageia2 running on Amazon Web Services' Elastic Compute Cloud. See my first post in the series for an overview.

In my last post I described how to create and upload an AMI that allows you to run Mageia2 on EC2. There were two issues with that method:

1. The EC2 instances are using a one-off unverified kernel, obtained for testing purposes only.
2. The instances launched can only be instance-store backed, ephemeral disk.

Both of these problems are solvable. We'll address the kernel first. The solution? Compile your own!

As with any good open-source project, you can easily obtain the source code. The same holds for Mageia. For recompiling the kernel, I plucked the kernel's SRPM file off mirrors.kernel.org. Reviewing the source, there were considerable tweaks made by the Mageia development team - so much so that they were bundled together inside the SRPM. Once I was able to dig into that tarball, I found where to enable CONFIG_KERNEL_GZIP and disable CONFIG_KERNEL_XZ in the configuration.

Now it was a matter of getting a system to build the kernel on. Initially, I tried to do it on my local seed Mageia VM, but the 10GB disk was too small to hold all the compiled kernel sources. So, I launched an instance using the EC2 console of the freshly-uploaded Mageia2 AMI. This is where I ran into the limitation of the first revision of the AMI I created - 2GB was insufficient to install all the compiler dependencies needed for creating a kernel package - but 8GB was ok.

Finally, I launched an m2.xlarge instance via the EC2 console with an 8GB root disk on instance-store to do the compilation. I wanted an instance with at least 2 cores to speed up the compile and sufficient additional space on ephemeral disk (/dev/xvdb) that could hold the compiled kernel sources. It still took a considerable amount of time - approximately 2 hours to compile the RPM. When I have to do this again in the future, I might consider a high I/O instance to reduce the time spent waiting for the compile. Either way, the cost is negligible if you remember to shut it down after you're done - the m2.xlarge was around $1.35 for 3 hours and a hi1.4xlarge would be $3.10 for one hour. For those who follow this blog, you should recognize the build script.

EDIT: on 1/18/13, I recompiled the kernel using an hi1.4xlarge and the final timing from "time ./do-build.sh" was:

real    51m5.356s
user    173m1.470s
sys     32m44.040s

Of course, it took a little longer than that to install all the packages needed for building, but it can be done in less than 2 hrs.


Here are the steps for compiling the Mageia2 kernel:

#
# prep stuff done as root
#
sudo bash -o vi
# mount the ephemeral storage
mkfs -t ext4 /dev/xvdb
mkdir /media/extra
mount /dev/xvdb /media/extra
# create some swap
dd if=/dev/zero of=/media/extra/swapfile00 bs=1024 count=4194304
mkswap /media/extra/swapfile00
swapon /media/extra/swapfile00
# setup space for kernel building
mkdir /media/extra/kernel
chown $USER:$USER /media/extra/kernel
exit

#
# build stuff done as normal user
#
# prep for kernel building
cd $HOME
ln -s /media/extra/kernel
cd kernel/
# bring down the source
curl -O http://mirrors.kernel.org/mageia/distrib/2/SRPMS/core/updates/kernel-3.3.8-2.mga2.src.rpm
mkdir SOURCES
cd SOURCES
# extract the source
rpm2cpio ../kernel-3.3.8-2.mga2.src.rpm | cpio -i
# make a working copy of the .spec file
cp -p kernel.spec ..
# extract the mageia customizations
tar Jxf linux-3.3.8-mga2.tar.xz
cd 3.3.8-mga2/configs/
# modify the kernel config for gzip compression
cp -p x86_64.config x86_64.config.orig
vi x86_64.config
# diff of what it looks like when it's done
$ diff -u x86_64.config.orig x86_64.config
--- x86_64.config.orig  2012-07-12 08:53:47.000000000 +0000
+++ x86_64.config       2012-11-15 04:48:37.000000000 +0000
@@ -67,10 +67,10 @@
 CONFIG_HAVE_KERNEL_LZMA=y
 CONFIG_HAVE_KERNEL_XZ=y
 CONFIG_HAVE_KERNEL_LZO=y
-# CONFIG_KERNEL_GZIP is not set
+CONFIG_KERNEL_GZIP=y
 # CONFIG_KERNEL_BZIP2 is not set
 # CONFIG_KERNEL_LZMA is not set
-CONFIG_KERNEL_XZ=y
+# CONFIG_KERNEL_XZ is not set
 # CONFIG_KERNEL_LZO is not set
 CONFIG_DEFAULT_HOSTNAME="(none)"
 CONFIG_SWAP=y

# rebuild the mageia customizations
cd ../..
mv linux-3.3.8-mga2.tar.xz linux-3.3.8-mga2.tar.xz.orig
tar Jcf linux-3.3.8-mga2.tar.xz 3.3.8-mga2

# install builder dependencies
sudo urpmi easyrpmbuilder
sudo urpmi elfutils-devel zlib-devel binutils-devel newt-devel python-devel pciutils-devel asciidoc xmlto docbook-style-xsl

# setup the build script
cat do-build.sh
#!/bin/sh -x
rm -rf BUILD BUILDROOT RPMS SRPMS tmp || true
mkdir -p BUILD BUILDROOT RPMS SRPMS tmp

OPTS=""
OPTS="$OPTS --with=server"
OPTS="$OPTS --without=desktop"
OPTS="$OPTS --without=desktop586"
OPTS="$OPTS --without=netbook"
rpmbuild $OPTS -bb --define="_topdir $PWD" --define="_tmppath $PWD/tmp" kernel.spec 2>&1 | tee kernel-build.txt

# do the build
time ./do-build.sh

# save the rpm
scp -p RPMS/x86_64/kernel-server-3.3.8-2.mga2-1-1.mga2.x86_64.rpm $REMOTE_SERVER:

Mageia2 on EC2: Boarding procedures

This is my second post on getting Mageia2 running on Amazon Web Services' Elastic Compute Cloud. See my first post in the series for an overview.

The first step to creating a Mageia2 install on EC2 is to have a local Mageia2 system as your seed setup. Why? Because you must use urpmi, the Mageia package installer. It is equivalent to apt-get in Ubuntu or yum in CentOS/RHEL/Amazon Linux. Yes, you can use rpm to install individual packages (and we will) but urpmi is what talks to the media sets (e.g. repositories) and makes sure you have all your dependencies installed. Besides the man page and the urpmi page on the Mageia wiki, I found a good quick reference guide that helped.

I'll not cover the setup of your seed Mageia2 system here - the installation is was breeze. I used VirtualBox under Windows 7 to install from the Dual-arch ISO CD, but you could likely use any old PC you have laying around and install however you like - USB key, Live CD, whatever.

Also, I'm not doing to document getting the EC2 command line tools working on your seed system. Installing java was simple ("urpmi java" I believe) and getting the API tools and AMI tools installed and configured is well documented by Amazon and plenty of others.

Once you have a functioning Mageia2 system and working EC2 AMI and API tools, then we're ready to begin.

The initial steps we'll be following are a mix between the Mageia chroot install and the official documentation on how to create an EC2 instance-store backed AMI. Another major factor was choosing a kernel. Now any good distribution ships with it's own kernel, and Mageia is no different. And of course you can use your own kernel in EC2. The most efficient way to do this is to use the PV-GRUB AKI provided by AWS to load the kernel that is present on your instance's disk, which is what we'll do.

For the most part, all of this went well after some trial-and-error. However I did run across a few issues:

1. Make sure you create a big enough loopback device. I started with 2GB and while it was enough for the base install, it wasn't enough once I started adding other packages later. My docs below use 8GB. The maximum is 10GB.
2. Make sure you choose the right PV-GRUB AKI (more on kernels in a moment)
3. Use a gzip compressed kernel, not a xz compressed kernel
4. Choose the right mirror for urpmi.addmedia (distro.ibiblio.org was extra slow for me - mirrors.kernel.org was much faster)
5. Make sure you install the critical packages. Without dhcp-client, you won't get your IP address and without an IP address, you're sunk. Same goes for sshd and sudo.
6. At this stage, I didn't pay attention to the ssh key pairs built into the EC2 provisioning system. I baked a new public key of the "mageia" user into the install.

When choosing a PV-GRUB AKI, the AWS documentation explains:

You must choose an AKI with "hd0" in the name if you want a raw or unpartitioned disk image (most images). Choose an AKI with "hd00" in the name if you want an image that has a partition table.

Since I am doing a direct mke2fs of the loopback image, it doesn't have a partition table. However, I was using the wrong PV-GRUB AKI, the one for loopbacks with partition tables, resulting in nothing working with error messages that were confusing. I'm sure you could fdisk your loopback image and create partitions if you want, but I didn't see it as necessary as I often use the other ephemeral disks for swap, etc. So, I had to use a "hd0" PV-GRUB to get it to work. 

After I got PV-GRUB set the system still wouldn't load. The error message from the system console was:

ERROR Invalid kernel: xc_dom_probe_bzimage_kernel: unknown compression format

In chatting with the very helpful "tmb" from the #mageia IRC channel on freenode, I was able to overcome this issue. The problem is that all Mageia kernels are xz compressed by default which is supported just fine by regular grub. However, the PV-GRUB AKI I was using didn't support xz compressed kernels. tmb provided me with a gzip compressed kernel to bootstrap by first EC2 instance running Mageia2. Thanks again tmb - without your help, I wouldn't have gotten this working!

After the kernel loaded, it was just straightforward trial and error troubleshooting till I was able to login. Here's the sanitized steps I used to get my first Mageia2 instance-store backed EC2 AMI uploaded:

# preparation
export PATH=$PATH:/sbin:/usr/sbin
# create a working directory
mkdir $HOME/ec2
# create ssh public key
ssh-keygen -t rsa -f $HOME/ec2/mageia -C "mageia@ec2" -P ""
# setup the image
dd if=/dev/zero of=mageia2-instance-store-v1.img bs=1M count=8192
# format it
mke2fs -F -j $HOME/ec2/mageia2-instance-store-v1.img

# everything forward needs to be done as root 
sudo bash -o vi

# mount the image for chroot
export MAGEIA_PUB_KEY=$HOME/ec2/mageia.pub
export CHRDIR=$HOME/ec2/loop
# mount the chroot location
mount -o loop $HOME/ec2/mageia2-instance-store-v1.img $CHRDIR

# create the minimum devices
mkdir $CHRDIR/dev
/sbin/makedev $CHRDIR/dev console
/sbin/makedev $CHRDIR/dev null
/sbin/makedev $CHRDIR/dev zero

# setup the minimum filesystems
mkdir $CHRDIR/etc
cat > $CHRDIR/etc/fstab << EOF
/dev/xvda1 /         ext3    defaults        1 1
none       /dev/pts  devpts  gid=5,mode=620  0 0
none       /dev/shm  tmpfs   defaults        0 0
none       /proc     proc    defaults        0 0
none       /sys      sysfs   defaults        0 0
EOF

# add required /proc filesystem
mkdir $CHRDIR/proc
mount -t proc none $CHRDIR/proc

# choose the best/fastest mirror
GET http://mirrors.mageia.org/api/mageia.2.x86_64.list | grep country=US
# setup the urpmi media locations in the chroot
urpmi.addmedia --distrib --urpmi-root $CHRDIR http://mirrors.kernel.org/mageia/distrib/2/x86_64
# install the minimum packages
urpmi --auto --urpmi-root $CHRDIR kernel-server basesystem urpmi locales-en sshd sudo dhcp-client

# MASSIVE HACK TIME
#
# kernel from tmb:
# http://tmb.mine.nu/Mageia/2/ec2/
#
mkdir tmb
pushd tmb
curl -O http://tmb.mine.nu.nyud.net/Mageia/2/ec2/kernel-server-3.4.18-1.mga2-1-1.mga2.x86_64.rpm
curl -O http://tmb.mine.nu.nyud.net/Mageia/2/ec2/kmod-7-7.mga2.x86_64.rpm
popd
# install custom kernel
rpm --root=$CHRDIR -Uhv tmb/*.rpm

# insure the new ramdisk is created properly
chroot $CHRDIR
cd /boot
mkinitrd initrd-3.4.18-server-1.mga2.img.a 3.4.18-server-1.mga2
exit

# set the kernel to load on boot
cat > $CHRDIR/boot/grub/menu.lst << EOF
default=0
timeout=0
title linux
  root (hd0)
  kernel /boot/vmlinuz-server ro root=/dev/xvda1 console=hvc0 BOOT_IMAGE=linux-nonfb
  initrd /boot/initrd-server.img
EOF

# configure the chroot network for ec2
cat > $CHRDIR/etc/sysconfig/network-scripts/ifcfg-eth0 << EOF
DEVICE=eth0
BOOTPROTO=dhcp
ONBOOT=yes
TYPE=Ethernet
USERCTL=yes
PEERDNS=yes
IPV6INIT=no
EOF
cat > $CHRDIR/etc/sysconfig/network << EOF
NETWORKING=yes
CRDA_DOMAIN=US
EOF

# configure ssh
test -f $CHRDIR/etc/ssh/sshd_config.orig || cp -p $CHRDIR/etc/ssh/sshd_config $CHRDIR/etc/ssh/sshd_config.orig
cat $CHRDIR/etc/ssh/sshd_config.orig |
    sed -e 's/^#UseDNS yes/UseDNS no/g' |
    sed -e 's/^PermitRootLogin no/PermitRootLogin without-password/g' > $CHRDIR/etc/ssh/sshd_config
# setup mageia account
chroot $CHRDIR /usr/sbin/useradd --create-home --home /home/mageia --shell /bin/bash mageia
mkdir --mode=0700 $CHRDIR/home/mageia/.ssh
(umask 0077; touch $CHRDIR/home/mageia/.ssh/authorized_keys)
cat $MAGEIA_PUB_KEY >> $CHRDIR/home/mageia/.ssh/authorized_keys
echo "set -o vi" >> $CHRDIR/home/mageia/.bashrc
chown -Rh 500:500 $CHRDIR/home/mageia/.ssh
(umask 0227; echo "mageia ALL=(ALL) NOPASSWD:ALL" > $CHRDIR/etc/sudoers.d/mageia)

# dismount the chroot
umount $CHRDIR/proc
umount -d $CHRDIR

# setup for EC2
export EC2_ID=[aws account id #]
export EC2_PRIVATE_KEY=[location of private key]
export EC2_CERT=[location of signing cert]
export EC2_ACCESS=[iam access key]
export EC2_SECRET=[iam secret key]

BUCKETNAME="$EC2_ID-mageia2-instance-store-v1"
# create S3 bucket
# you can use the AWS Console instead of s3cmd, if you like
s3cmd mb s3://$BUCKETNAME
# where to put the bundle parts
pushd $HOME/ec2
mkdir mageia2-instance-store-v1
# create the AMI bundle
# AKI for pv-grub-hd0_1.03-x86_64.gz partitionless PV-GRUB in US-East-1
AKIID="aki-88aa75e1"
ec2-bundle-image -i mageia2-instance-store-v1.img -d mageia2-instance-store-v1 -r x86_64 --kernel $AKIID -k $EC2_PRIVATE_KEY -c $EC2_CERT -u $EC2_ID
# put it on S3
ec2-upload-bundle -b $BUCKETNAME -m mageia2-instance-store-v1/mageia2-instance-store-v1.img.manifest.xml -a $EC2_ACCESS -s $EC2_SECRET
# register it
ec2-register $BUCKETNAME/mageia2-instance-store-v1.img.manifest.xml -n mageia2-instance-store-v1
popd

Mageia2 on EC2: Flying in a different direction

At $WORK, we make the claim that as a client's systems oversight service, we are "distribution agnostic" - meaning we'll help you out regardless of what Linux distribution you're running. Most of the time, we work with Ubuntu, RHEL, CentOS or Amazon Linux. However, a client recently decided on running Mageia2 GNU/Linux on Amazon Web Services' Elastic Compute Cloud, so I had to pick up the challenge.

Now as far as distributions go, it seems to me that Mageia is "ok" - a fork of Mandriva that has a nice community developed around it. Unfortunately, what hasn't developed in any interest in running it on EC2. There seems to be a few VPS providers that support it, but not any of the popular ones that I know of. Still, AWS EC2 has the capability of running your own Linux distribution and even your own kernel so I pushed up my sleeves and dug into getting it going. As there's a lot to this configuration, I will break this down over a few posts. Here is an outline:

1. Boarding procedures - performing a chroot install on a local Mageia system 
2. Stormy weather - recompiling the Linux kernel to get Mageia running on EC2
3. Cruising altitude - performing another chroot install for an EBS backed instance

I'll be posting sanitized code samples and links to the key documents that explain the steps.

EDIT 2/7/2013: I've put the code samples up on github.

Log housekeeping with python

This week I was able to finally finish some python code I've been writing for $WORK - a script to rotate webserver logs directly to S3. This task was similar to something I'd done a long, LONG time ago (14 years since the first rev!) in a programming language far, far away. I had a much bigger chip on my shoulder then -  site analytics really isn't done with log parsing anymore, so I skipped the whole test for open filehandles and email notifications. Anywhere, here it is - enjoy!

https://github.com/dialt0ne/rotate-to-s3

Kudos to Justin for critiquing my python.

EBS snapshots and LVM2

I've been meaning to try this for a while to see how it goes - use LVM2 to take an "instantaneous" snapshot of an EBS volume and then let AWS take it's time. I found LVM wasn't as quick as I'd like. Also, I have performance tested this either, so I don't know how bad the latency will be. Either way, I think it's an easy way to get a consistent backup:

# prep
export MYAZ="us-east-1a"
export MYINST="i-XXXXXXXX"
# create 1st EBS volume and attach
ec2-create-volume --size 2 --availability-zone $MYAZ
export VOL0="vol-XXXXXXXX"
ec2-attach-volume $VOL0 --instance $MYINST --device /dev/sdf

# create LVM partition 1
fdisk /dev/sdf
# add to LVM
pvcreate /dev/sdf1
# create a volume group
vgcreate vol0 /dev/sdf1
# create a logical volume
lvcreate -l80%FREE -n test vol0
# format it
mke2fs -j -m0 /dev/vol0/test
# mount it
mkdir -p /mnt/vol0/test
mount /dev/vol0/test /mnt/vol0/test

# lock data consistently

# create LVM snapshot
lvcreate -L300M -s -n test2 /dev/vol0/test

# unlock data consistently

# create EBS snapshot
ec2-create-snapshot $VOL0
export VOL0_SNAP="snap-XXXXXXXX"

# remove LVM snapshot
lvremove vol0/test2

# create 2nd EBS volume from snapshot and attach
ec2-create-volume --snapshot $VOL0_SNAP --availability-zone $MYAZ
export VOL1="vol-YYYYYYYY"
ec2-attach-volume $VOL1 --instance $MYINST --device /dev/sdf

# import snapshot as new volume group
vgimportclone -n vol1 /dev/sdg1
# activate new volume group
vgchange -a y vol1
# mount it
mkdir -p /mnt/vol1/test2
mount /dev/vol1/test2 /mnt/vol1/test2

EDIT 2013.02.10: "lock data consistently" - I highly recommend "fsfreeze" which is built into most Linux distributions nowadays.

python!

After a break, I've decided to pick up a new project - learn python. I have a specific work goal in mind - create a small application to create, manage, remove, map, etc. CloudFront distributions that use a custom origin server, i.e. not S3, across multiple AWS accounts. It seems that boto is way to go. It has an uphill battle - the AWS provided SDKs are really quite easy to use and are well documented.

EDIT: Gah! Talking about leaping before looking! I pull up the boto API reference page on cloudfront and I see the following:

This module is not well tested. Paging of distributions is not yet supported. CNAME support is completely untested. Use with caution. Feedback and bug reports are greatly appreciated.

Sounds not quite ready for a) learning python b) using at work. <sigh> Learning python is still the next project, but I won't be using it for this concept.

Autoscaling revisited

Shortly after writing my previous post about AWS autoscaling, Amazon updated the autoscaling methodology. Instead of triggers they now use autoscaling policies and alarms from CloudWatch to initiate the policy actions. So here's how I create and remove policies and alarms from an autoscaling group. Note: you can't have both triggers and policies on a group, you have to remove the triggers first before adding the policies.

#
# create policies that will scale the group up and down
# note: cooldown is how many seconds to wait before
#       applying the policy again
#
export COOLDOWN=300
export SCALEUP=`as-put-scaling-policy $ASGROUP-scaleUp \
        --auto-scaling-group $ASGROUP \
        --cooldown $COOLDOWN \
        --adjustment=1 \
        --type ChangeInCapacity`
if [ $? -eq 0 ]; then echo OK - $SCALEUP; else echo ERROR; fi

export SCALEDOWN=`as-put-scaling-policy $ASGROUP-scaleDown \
        --auto-scaling-group $ASGROUP \
        --cooldown $COOLDOWN \
        --adjustment=-1 \
        --type ChangeInCapacity`
if [ $? -eq 0 ]; then echo OK - $SCALEDOWN; else echo ERROR; fi

#
# create alarms to implement policies
#

#
# example: Latency on the ELB
#
mon-put-metric-alarm \
        --alarm-name $ASGROUP-HighLatency \
        --namespace "AWS/ELB" \
        --metric-name Latency \
        --statistic Average \
        --period 60 \
        --comparison-operator GreaterThanThreshold \
        --threshold 5.0 \
        --unit Seconds \
        --evaluation-periods 5 \
        --dimensions "LoadBalancerName=$LBNAME" \
        --alarm-actions $SCALEUP
if [ $? -eq 0 ]; then echo OK; else echo ERROR; fi

mon-put-metric-alarm \
        --alarm-name $ASGROUP-LowLatency \
        --namespace "AWS/ELB" \
        --metric-name Latency \
        --statistic Average \
        --period 60 \
        --comparison-operator LessThanThreshold \
        --threshold 0.5 \
        --unit Seconds \
        --evaluation-periods 5 \
        --dimensions "LoadBalancerName=$LBNAME" \
        --alarm-actions $SCALEDOWN
if [ $? -eq 0 ]; then echo OK; else echo ERROR; fi

And to clean up:

mon-delete-alarms --alarm-name $ASGROUP-HighLatency --force
mon-delete-alarms --alarm-name $ASGROUP-LowLatency --force
as-delete-policy $ASGROUP-scaleUp --auto-scaling-group $ASGROUP --force
as-delete-policy $ASGROUP-scaleDown --auto-scaling-group $ASGROUP --force

On Cloud n+1

I spent the last few days setting up an autoscaling pool of servers on the Amazon Elastic Compute Cloud. They really have done an excellent job of putting together a great toolset and documentation. I've made some notes on how to do a basic setup, including using the EC2 Elastic Load Balancer. Another cool tool I was able to use for this project was Ubuntu's pre-built EC2 images and the cloud-init package, making auto-deployment of the servers very easy to do.

# Notes on setting up Amazon AWS Auto Scaling
# ===========================================
# ATonns Tue Oct 26 17:37:12 EDT 2010
#

export AVAILZONE="us-east-1a"
#
# create a launch config
#
export LCNAME="test-lc"
as-create-launch-config $LCNAME \
 --image-id ami-f5e0049c \
 --instance-type m1.small
#
# other key args:
#
# /* security group */
# --group {groupname}
# /* meta-data file */
# --user-data-file {filename}
#

#
# create a load balancer
#
export LBNAME="test-lb"
elb-create-lb $LBNAME --headers \
 --availability-zones $AVAILZONE \
 --listener "protocol=http,lb-port=80,instance-port=80" 
#
# add some thresholds that will kick instances out
#
export LBTESTURI="/DONOTREMOVE.php"
elb-configure-healthcheck $LBNAME --headers \
 --target "HTTP:80$LBTESTURI" \
 --interval 5 \
 --timeout 2 \
 --unhealthy-threshold 2 \
 --healthy-threshold 5

#
# create auto-scale group
#
export ASGROUP="test-asg"
as-create-auto-scaling-group $ASGROUP \
 --availability-zones $AVAILZONE \
 --launch-configuration $LCNAME \
 --min-size 1 \
 --max-size 5 \
 --load-balancers $LBNAME

#
# create a trigger
#
export ASTRIGGER="test-trig"
as-create-or-update-trigger $ASTRIGGER \
 --auto-scaling-group $ASGROUP \
 --period 60 \
 --unit Seconds \
 --dimensions "LoadBalancerName=$LBNAME" \
 --namespace "AWS/ELB" \
 --measure Latency \
 --statistic Average \
 --lower-threshold 0.25 \
 --upper-threshold 0.75 \
 --breach-duration 300 \
 --lower-breach-increment=-1 \
 --upper-breach-increment 1 

#
# more metrics
#
http://goo.gl/A4pAd

------------

#
# remove everything
#
as-delete-trigger $ASTRIGGER --auto-scaling-group $ASGROUP --force
as-update-auto-scaling-group $ASGROUP --min-size 0 --max-size 0
count="-1"
while [ $count -ne 0 ]
do
 count=0
 for i in `as-describe-auto-scaling-groups $ASGROUP --show-long`
 do
  type=`echo $i | cut -d, -f1 -`
  if [ $type = INSTANCE ]
  then
   count=`expr $count + 1`
  fi
 done
 echo $count instances left
done
procs="-1"
while [ $procs -ne 0 ]
do
 procs=0
 for i in `as-describe-scaling-activities $ASGROUP --show-long | cut -d, -f4 -`
 do
  if [ "$i" != "Successful" ]
  then
   procs=`expr $procs + 1`
  fi
 done
 echo $procs processes still running
done
as-delete-auto-scaling-group $ASGROUP --force
as-delete-launch-config $LCNAME --force
elb-delete-lb $LBNAME --force