Building a perl package for Amazon Linux, CentOS or RHEL

A previous post detailed on how to build a .deb file for a perl module on Ubuntu. However, I needed the updated module as a .rpm on an Amazon Linux system, so I created the procedure for that OS as well. It was a lot easier than I thought it would be.

# install general dependencies of Net::Amazon::EC2
sudo yum --enablerepo=epel install \
    perl-Net-Amazon-EC2 perl-File-Slurp perl-DBI perl-DBD-MySQL \
    perl-Net-SSLeay perl-IO-Socket-SSL perl-Time-HiRes perl-Params-Validate \
    perl-Date-Manip perl-DateTime perl-DateTime-Format-ISO8601 \
    ca-certificates

# install stuff required for the build
sudo yum --enablerepo=epel install cpanspec rpm-build.x86_64
sudo yum --enablerepo=epel install perl-Test-Exception perl-CPAN

# generate .spec file
PACKAGER="John Smith <jsmith@example.com>"
cpanspec --packager "$PACKAGER" -v Net::Amazon::EC2

# get the source
mkdir -p rpmbuild/SOURCES
wget -Orpmbuild/SOURCES/Net-Amazon-EC2-0.23.tar.gz \
    http://search.cpan.org/CPAN/authors/id/M/MA/MALLEN/Net-Amazon-EC2-0.23.tar.gz

# don't set these or tests will fail
unset AWS_ACCESS_KEY_ID
unset SECRET_ACCESS_KEY

# do the build
rpmbuild -bb perl-Net-Amazon-EC2.spec

Piping STDOUT to one command but STDERR to a different command

Found this awesome stackoverflow answer and had to write it up as a note to myself:

./foobar.pl > >( logger -t stdout ) 2> >( logger -t stderr )

Specifically, I hope to use this to replicate all EBS snapshots taken on an instance, e.g.:

ec2-consistent-snapshot > >( ec2-replicate-snapshots ) 2> >( logger -t $PROGNAME )

Mageia2 on EC2: Cruising Altitude

This is my fourth post on getting Mageia2 running on Amazon Web Services' Elastic Compute Cloud. See my first post in the series for an overview.

In the last post, I addressed the problem of having only a test kernel by tweaking the Mageia kernel SRPM and creating a gzipped kernel that can be used with the version of PV-GRUB supplied by Amazon. Now I'll walk through the steps of building an EBS backend instance instead of an instance-store backed instance.

You need a working Mageia setup on an instance-store backed instance before you can create the EBS backed one. Just launch the AMI created in a previous step and then attach a 32GB EBS volume to it. Using the EC2 API tools, you attach the volume like this:

SIZE=32
TARGETAZ=us-east-1a
INSTID=i-09abcdef

CMD=($(ec2-create-volume --size $SIZE --availability-zone $TARGETAZ --type standard))
VOLID=${CMD[1]}
ec2-attach-volume $VOLID --instance $INSTID --device /dev/sdg


You will also need some other components:

1. "kernel-server" RPM created in a last post.
2. A copy of ec2-get-ssh.sh for the mageia user

The second component is so you don't have to embed passwords in your AMI, but instead uses ssh public keys that are imported to (or generated by) AWS.

Another difference is that we add the kernel to the skip.list for upgrades, as we don't want to get a non-gzipped kernel. So, here's the steps for setting it up:

mkdir $HOME/ec2

# everything forward needs to be done as root 
sudo bash -o vi
cd $HOME/ec2
export PATH=$PATH:/sbin:/usr/sbin

# setup the filesystem
/sbin/mkfs -t ext4 /dev/xvdg

# mount the image for chroot
export CHRDIR=$HOME/ec2/loop
mount /dev/xvdg $CHRDIR

# create the minimum devices
mkdir $CHRDIR/dev
/sbin/makedev $CHRDIR/dev console
/sbin/makedev $CHRDIR/dev null
/sbin/makedev $CHRDIR/dev zero

# setup the minimum filesystems
mkdir $CHRDIR/etc
cat > $CHRDIR/etc/fstab << EOF
/dev/xvda1 /         ext3    defaults        1 1
none       /dev/pts  devpts  gid=5,mode=620  0 0
none       /dev/shm  tmpfs   defaults        0 0
none       /proc     proc    defaults        0 0
none       /sys      sysfs   defaults        0 0
EOF

# add required /proc filesystem
mkdir $CHRDIR/proc
mount -t proc none $CHRDIR/proc

# choose the best/fastest mirror
GET http://mirrors.mageia.org/api/mageia.2.x86_64.list | grep country=US
# setup the urpmi media locations in the chroot
urpmi.addmedia --distrib --urpmi-root $CHRDIR http://mirrors.kernel.org/mageia/distrib/2/x86_64
# install the minimum packages
urpmi --auto --urpmi-root $CHRDIR basesystem urpmi locales-en sshd sudo dhcp-client

# MASSIVE HACK TIME
rpm --root=$CHRDIR -Uhv custom-kernel/kernel-server-3.3.8-2.mga2-1-1.mga2.x86_64.rpm

# cleanup desktop kernel
chroot $CHRDIR
urpme kernel-desktop-3.3.8-2.mga2-1-1.mga2
rm -f initrd-desktop.img  vmlinuz-desktop 
# confirm there's a good initrd
cd /boot
stat initrd-3.3.8-server-2.mga2.img
mkinitrd initrd-3.3.8-server-2.mga2.img 3.3.8-server-2.mga2
exit

# set the kernel to load on boot
cat > $CHRDIR/boot/grub/menu.lst << EOF
default=0
timeout=0
title linux
  root (hd0)
  kernel /boot/vmlinuz-server ro root=/dev/xvda1 console=hvc0 BOOT_IMAGE=linux-nonfb
  initrd /boot/initrd-server.img
EOF

# do not upgrade the kernel, until upstream fixes the xz/gz issue
test -f $CHRDIR/etc/urpmi/skip.list || cp -p $CHRDIR/etc/urpmi/skip.list $CHRDIR/etc/urpmi/skip.list.orig
cat > $CHRDIR/etc/urpmi/skip.list << EOF
# Here you can specify the packages that won't be upgraded automatically
# for example, to exclude all apache packages :
# /^apache/
/^kernel/
EOF

# configure the chroot network for ec2
cat > $CHRDIR/etc/sysconfig/network-scripts/ifcfg-eth0 << EOF
DEVICE=eth0
BOOTPROTO=dhcp
ONBOOT=yes
TYPE=Ethernet
USERCTL=yes
PEERDNS=yes
IPV6INIT=no
EOF
cat > $CHRDIR/etc/sysconfig/network << EOF
NETWORKING=yes
CRDA_DOMAIN=US
EOF

# configure ssh
test -f $CHRDIR/etc/ssh/sshd_config.orig || cp -p $CHRDIR/etc/ssh/sshd_config $CHRDIR/etc/ssh/sshd_config.orig
cat $CHRDIR/etc/ssh/sshd_config.orig |
    sed -e 's/^#UseDNS yes/UseDNS no/g' |
    sed -e 's/^PermitRootLogin no/PermitRootLogin without-password/g' > $CHRDIR/etc/ssh/sshd_config
# create login account
chroot $CHRDIR /usr/sbin/useradd --create-home --home /home/mageia --shell /bin/bash mageia
(umask 0227; echo "mageia ALL=(ALL) NOPASSWD:ALL" > $CHRDIR/etc/sudoers.d/mageia)

# setup ssh public key
cp ec2-get-ssh $CHRDIR/etc/rc.d/init.d/ec2-get-ssh
chmod 0750 $CHRDIR/etc/rc.d/init.d/ec2-get-ssh
chown root:root $CHRDIR/etc/rc.d/init.d/ec2-get-ssh
chroot $CHRDIR /sbin/chkconfig ec2-get-ssh on

# dismount the chroot
umount $CHRDIR/proc
umount -d $CHRDIR

Now that the EBS volume is all set, it needs to be snapshotted and registered as an AMI. Here's what you do:

ec2-detach-volume $VOLID --instance $INSTIT--device /dev/sdg

# create a snapshot
CMD=($(ec2-create-snapshot --description "Mageia 2" $EBSVOL))
SNAPID=${CMD[1]}

# create AMI
AKIID="aki-88aa75e1"
ec2-register --name "Mageia 2" --description "Mageia 2" \
    --architecture x86_64 --root-device-name /dev/sda1 \
    --block-device-mapping /dev/sda1=$SNAPID --kernel $AKIID

Now you're ready to launch your EBS back Mageia2 Linux instance! Enjoy!

Vi IMproved

Ubuntu tweak #2 - diediedie nano die!

$ sudo update-alternatives --config editor
There are 4 choices for the alternative editor (providing /usr/bin/editor).

  Selection    Path                Priority   Status
------------------------------------------------------------
* 0            /bin/nano            40        auto mode
  1            /bin/ed             -100       manual mode
  2            /bin/nano            40        manual mode
  3            /usr/bin/vim.basic   30        manual mode
  4            /usr/bin/vim.tiny    10        manual mode

Press enter to keep the current choice[*], or type selection number: 3
update-alternatives: using /usr/bin/vim.basic to provide /usr/bin/editor (editor) in manual mode

Ahh, much better!

Disable dnsmasq in NetworkManager

I have recently converted my work desktop to Ubuntu 12.10. Most things were better, but I was seeing horrible DNS lag from dnsmasq. To disable it, I've done the following:

sudo vi /etc/NetworkManager/NetworkManager.conf
# comment out dns=dnsmasq
sudo restart network-manager

This will regenerate your resolv.conf and you'll see your DNS servers directly and not localhost.

Mageia2 on EC2: Stormy Weather

This is my third post on getting Mageia2 running on Amazon Web Services' Elastic Compute Cloud. See my first post in the series for an overview.

In my last post I described how to create and upload an AMI that allows you to run Mageia2 on EC2. There were two issues with that method:

1. The EC2 instances are using a one-off unverified kernel, obtained for testing purposes only.
2. The instances launched can only be instance-store backed, ephemeral disk.

Both of these problems are solvable. We'll address the kernel first. The solution? Compile your own!

As with any good open-source project, you can easily obtain the source code. The same holds for Mageia. For recompiling the kernel, I plucked the kernel's SRPM file off mirrors.kernel.org. Reviewing the source, there were considerable tweaks made by the Mageia development team - so much so that they were bundled together inside the SRPM. Once I was able to dig into that tarball, I found where to enable CONFIG_KERNEL_GZIP and disable CONFIG_KERNEL_XZ in the configuration.

Now it was a matter of getting a system to build the kernel on. Initially, I tried to do it on my local seed Mageia VM, but the 10GB disk was too small to hold all the compiled kernel sources. So, I launched an instance using the EC2 console of the freshly-uploaded Mageia2 AMI. This is where I ran into the limitation of the first revision of the AMI I created - 2GB was insufficient to install all the compiler dependencies needed for creating a kernel package - but 8GB was ok.

Finally, I launched an m2.xlarge instance via the EC2 console with an 8GB root disk on instance-store to do the compilation. I wanted an instance with at least 2 cores to speed up the compile and sufficient additional space on ephemeral disk (/dev/xvdb) that could hold the compiled kernel sources. It still took a considerable amount of time - approximately 2 hours to compile the RPM. When I have to do this again in the future, I might consider a high I/O instance to reduce the time spent waiting for the compile. Either way, the cost is negligible if you remember to shut it down after you're done - the m2.xlarge was around $1.35 for 3 hours and a hi1.4xlarge would be $3.10 for one hour. For those who follow this blog, you should recognize the build script.

EDIT: on 1/18/13, I recompiled the kernel using an hi1.4xlarge and the final timing from "time ./do-build.sh" was:

real    51m5.356s
user    173m1.470s
sys     32m44.040s

Of course, it took a little longer than that to install all the packages needed for building, but it can be done in less than 2 hrs.


Here are the steps for compiling the Mageia2 kernel:

#
# prep stuff done as root
#
sudo bash -o vi
# mount the ephemeral storage
mkfs -t ext4 /dev/xvdb
mkdir /media/extra
mount /dev/xvdb /media/extra
# create some swap
dd if=/dev/zero of=/media/extra/swapfile00 bs=1024 count=4194304
mkswap /media/extra/swapfile00
swapon /media/extra/swapfile00
# setup space for kernel building
mkdir /media/extra/kernel
chown $USER:$USER /media/extra/kernel
exit

#
# build stuff done as normal user
#
# prep for kernel building
cd $HOME
ln -s /media/extra/kernel
cd kernel/
# bring down the source
curl -O http://mirrors.kernel.org/mageia/distrib/2/SRPMS/core/updates/kernel-3.3.8-2.mga2.src.rpm
mkdir SOURCES
cd SOURCES
# extract the source
rpm2cpio ../kernel-3.3.8-2.mga2.src.rpm | cpio -i
# make a working copy of the .spec file
cp -p kernel.spec ..
# extract the mageia customizations
tar Jxf linux-3.3.8-mga2.tar.xz
cd 3.3.8-mga2/configs/
# modify the kernel config for gzip compression
cp -p x86_64.config x86_64.config.orig
vi x86_64.config
# diff of what it looks like when it's done
$ diff -u x86_64.config.orig x86_64.config
--- x86_64.config.orig  2012-07-12 08:53:47.000000000 +0000
+++ x86_64.config       2012-11-15 04:48:37.000000000 +0000
@@ -67,10 +67,10 @@
 CONFIG_HAVE_KERNEL_LZMA=y
 CONFIG_HAVE_KERNEL_XZ=y
 CONFIG_HAVE_KERNEL_LZO=y
-# CONFIG_KERNEL_GZIP is not set
+CONFIG_KERNEL_GZIP=y
 # CONFIG_KERNEL_BZIP2 is not set
 # CONFIG_KERNEL_LZMA is not set
-CONFIG_KERNEL_XZ=y
+# CONFIG_KERNEL_XZ is not set
 # CONFIG_KERNEL_LZO is not set
 CONFIG_DEFAULT_HOSTNAME="(none)"
 CONFIG_SWAP=y

# rebuild the mageia customizations
cd ../..
mv linux-3.3.8-mga2.tar.xz linux-3.3.8-mga2.tar.xz.orig
tar Jcf linux-3.3.8-mga2.tar.xz 3.3.8-mga2

# install builder dependencies
sudo urpmi easyrpmbuilder
sudo urpmi elfutils-devel zlib-devel binutils-devel newt-devel python-devel pciutils-devel asciidoc xmlto docbook-style-xsl

# setup the build script
cat do-build.sh
#!/bin/sh -x
rm -rf BUILD BUILDROOT RPMS SRPMS tmp || true
mkdir -p BUILD BUILDROOT RPMS SRPMS tmp

OPTS=""
OPTS="$OPTS --with=server"
OPTS="$OPTS --without=desktop"
OPTS="$OPTS --without=desktop586"
OPTS="$OPTS --without=netbook"
rpmbuild $OPTS -bb --define="_topdir $PWD" --define="_tmppath $PWD/tmp" kernel.spec 2>&1 | tee kernel-build.txt

# do the build
time ./do-build.sh

# save the rpm
scp -p RPMS/x86_64/kernel-server-3.3.8-2.mga2-1-1.mga2.x86_64.rpm $REMOTE_SERVER:

Mageia2 on EC2: Boarding procedures

This is my second post on getting Mageia2 running on Amazon Web Services' Elastic Compute Cloud. See my first post in the series for an overview.

The first step to creating a Mageia2 install on EC2 is to have a local Mageia2 system as your seed setup. Why? Because you must use urpmi, the Mageia package installer. It is equivalent to apt-get in Ubuntu or yum in CentOS/RHEL/Amazon Linux. Yes, you can use rpm to install individual packages (and we will) but urpmi is what talks to the media sets (e.g. repositories) and makes sure you have all your dependencies installed. Besides the man page and the urpmi page on the Mageia wiki, I found a good quick reference guide that helped.

I'll not cover the setup of your seed Mageia2 system here - the installation is was breeze. I used VirtualBox under Windows 7 to install from the Dual-arch ISO CD, but you could likely use any old PC you have laying around and install however you like - USB key, Live CD, whatever.

Also, I'm not doing to document getting the EC2 command line tools working on your seed system. Installing java was simple ("urpmi java" I believe) and getting the API tools and AMI tools installed and configured is well documented by Amazon and plenty of others.

Once you have a functioning Mageia2 system and working EC2 AMI and API tools, then we're ready to begin.

The initial steps we'll be following are a mix between the Mageia chroot install and the official documentation on how to create an EC2 instance-store backed AMI. Another major factor was choosing a kernel. Now any good distribution ships with it's own kernel, and Mageia is no different. And of course you can use your own kernel in EC2. The most efficient way to do this is to use the PV-GRUB AKI provided by AWS to load the kernel that is present on your instance's disk, which is what we'll do.

For the most part, all of this went well after some trial-and-error. However I did run across a few issues:

1. Make sure you create a big enough loopback device. I started with 2GB and while it was enough for the base install, it wasn't enough once I started adding other packages later. My docs below use 8GB. The maximum is 10GB.
2. Make sure you choose the right PV-GRUB AKI (more on kernels in a moment)
3. Use a gzip compressed kernel, not a xz compressed kernel
4. Choose the right mirror for urpmi.addmedia (distro.ibiblio.org was extra slow for me - mirrors.kernel.org was much faster)
5. Make sure you install the critical packages. Without dhcp-client, you won't get your IP address and without an IP address, you're sunk. Same goes for sshd and sudo.
6. At this stage, I didn't pay attention to the ssh key pairs built into the EC2 provisioning system. I baked a new public key of the "mageia" user into the install.

When choosing a PV-GRUB AKI, the AWS documentation explains:

You must choose an AKI with "hd0" in the name if you want a raw or unpartitioned disk image (most images). Choose an AKI with "hd00" in the name if you want an image that has a partition table.

Since I am doing a direct mke2fs of the loopback image, it doesn't have a partition table. However, I was using the wrong PV-GRUB AKI, the one for loopbacks with partition tables, resulting in nothing working with error messages that were confusing. I'm sure you could fdisk your loopback image and create partitions if you want, but I didn't see it as necessary as I often use the other ephemeral disks for swap, etc. So, I had to use a "hd0" PV-GRUB to get it to work. 

After I got PV-GRUB set the system still wouldn't load. The error message from the system console was:

ERROR Invalid kernel: xc_dom_probe_bzimage_kernel: unknown compression format

In chatting with the very helpful "tmb" from the #mageia IRC channel on freenode, I was able to overcome this issue. The problem is that all Mageia kernels are xz compressed by default which is supported just fine by regular grub. However, the PV-GRUB AKI I was using didn't support xz compressed kernels. tmb provided me with a gzip compressed kernel to bootstrap by first EC2 instance running Mageia2. Thanks again tmb - without your help, I wouldn't have gotten this working!

After the kernel loaded, it was just straightforward trial and error troubleshooting till I was able to login. Here's the sanitized steps I used to get my first Mageia2 instance-store backed EC2 AMI uploaded:

# preparation
export PATH=$PATH:/sbin:/usr/sbin
# create a working directory
mkdir $HOME/ec2
# create ssh public key
ssh-keygen -t rsa -f $HOME/ec2/mageia -C "mageia@ec2" -P ""
# setup the image
dd if=/dev/zero of=mageia2-instance-store-v1.img bs=1M count=8192
# format it
mke2fs -F -j $HOME/ec2/mageia2-instance-store-v1.img

# everything forward needs to be done as root 
sudo bash -o vi

# mount the image for chroot
export MAGEIA_PUB_KEY=$HOME/ec2/mageia.pub
export CHRDIR=$HOME/ec2/loop
# mount the chroot location
mount -o loop $HOME/ec2/mageia2-instance-store-v1.img $CHRDIR

# create the minimum devices
mkdir $CHRDIR/dev
/sbin/makedev $CHRDIR/dev console
/sbin/makedev $CHRDIR/dev null
/sbin/makedev $CHRDIR/dev zero

# setup the minimum filesystems
mkdir $CHRDIR/etc
cat > $CHRDIR/etc/fstab << EOF
/dev/xvda1 /         ext3    defaults        1 1
none       /dev/pts  devpts  gid=5,mode=620  0 0
none       /dev/shm  tmpfs   defaults        0 0
none       /proc     proc    defaults        0 0
none       /sys      sysfs   defaults        0 0
EOF

# add required /proc filesystem
mkdir $CHRDIR/proc
mount -t proc none $CHRDIR/proc

# choose the best/fastest mirror
GET http://mirrors.mageia.org/api/mageia.2.x86_64.list | grep country=US
# setup the urpmi media locations in the chroot
urpmi.addmedia --distrib --urpmi-root $CHRDIR http://mirrors.kernel.org/mageia/distrib/2/x86_64
# install the minimum packages
urpmi --auto --urpmi-root $CHRDIR kernel-server basesystem urpmi locales-en sshd sudo dhcp-client

# MASSIVE HACK TIME
#
# kernel from tmb:
# http://tmb.mine.nu/Mageia/2/ec2/
#
mkdir tmb
pushd tmb
curl -O http://tmb.mine.nu.nyud.net/Mageia/2/ec2/kernel-server-3.4.18-1.mga2-1-1.mga2.x86_64.rpm
curl -O http://tmb.mine.nu.nyud.net/Mageia/2/ec2/kmod-7-7.mga2.x86_64.rpm
popd
# install custom kernel
rpm --root=$CHRDIR -Uhv tmb/*.rpm

# insure the new ramdisk is created properly
chroot $CHRDIR
cd /boot
mkinitrd initrd-3.4.18-server-1.mga2.img.a 3.4.18-server-1.mga2
exit

# set the kernel to load on boot
cat > $CHRDIR/boot/grub/menu.lst << EOF
default=0
timeout=0
title linux
  root (hd0)
  kernel /boot/vmlinuz-server ro root=/dev/xvda1 console=hvc0 BOOT_IMAGE=linux-nonfb
  initrd /boot/initrd-server.img
EOF

# configure the chroot network for ec2
cat > $CHRDIR/etc/sysconfig/network-scripts/ifcfg-eth0 << EOF
DEVICE=eth0
BOOTPROTO=dhcp
ONBOOT=yes
TYPE=Ethernet
USERCTL=yes
PEERDNS=yes
IPV6INIT=no
EOF
cat > $CHRDIR/etc/sysconfig/network << EOF
NETWORKING=yes
CRDA_DOMAIN=US
EOF

# configure ssh
test -f $CHRDIR/etc/ssh/sshd_config.orig || cp -p $CHRDIR/etc/ssh/sshd_config $CHRDIR/etc/ssh/sshd_config.orig
cat $CHRDIR/etc/ssh/sshd_config.orig |
    sed -e 's/^#UseDNS yes/UseDNS no/g' |
    sed -e 's/^PermitRootLogin no/PermitRootLogin without-password/g' > $CHRDIR/etc/ssh/sshd_config
# setup mageia account
chroot $CHRDIR /usr/sbin/useradd --create-home --home /home/mageia --shell /bin/bash mageia
mkdir --mode=0700 $CHRDIR/home/mageia/.ssh
(umask 0077; touch $CHRDIR/home/mageia/.ssh/authorized_keys)
cat $MAGEIA_PUB_KEY >> $CHRDIR/home/mageia/.ssh/authorized_keys
echo "set -o vi" >> $CHRDIR/home/mageia/.bashrc
chown -Rh 500:500 $CHRDIR/home/mageia/.ssh
(umask 0227; echo "mageia ALL=(ALL) NOPASSWD:ALL" > $CHRDIR/etc/sudoers.d/mageia)

# dismount the chroot
umount $CHRDIR/proc
umount -d $CHRDIR

# setup for EC2
export EC2_ID=[aws account id #]
export EC2_PRIVATE_KEY=[location of private key]
export EC2_CERT=[location of signing cert]
export EC2_ACCESS=[iam access key]
export EC2_SECRET=[iam secret key]

BUCKETNAME="$EC2_ID-mageia2-instance-store-v1"
# create S3 bucket
# you can use the AWS Console instead of s3cmd, if you like
s3cmd mb s3://$BUCKETNAME
# where to put the bundle parts
pushd $HOME/ec2
mkdir mageia2-instance-store-v1
# create the AMI bundle
# AKI for pv-grub-hd0_1.03-x86_64.gz partitionless PV-GRUB in US-East-1
AKIID="aki-88aa75e1"
ec2-bundle-image -i mageia2-instance-store-v1.img -d mageia2-instance-store-v1 -r x86_64 --kernel $AKIID -k $EC2_PRIVATE_KEY -c $EC2_CERT -u $EC2_ID
# put it on S3
ec2-upload-bundle -b $BUCKETNAME -m mageia2-instance-store-v1/mageia2-instance-store-v1.img.manifest.xml -a $EC2_ACCESS -s $EC2_SECRET
# register it
ec2-register $BUCKETNAME/mageia2-instance-store-v1.img.manifest.xml -n mageia2-instance-store-v1
popd

Mageia2 on EC2: Flying in a different direction

At $WORK, we make the claim that as a client's systems oversight service, we are "distribution agnostic" - meaning we'll help you out regardless of what Linux distribution you're running. Most of the time, we work with Ubuntu, RHEL, CentOS or Amazon Linux. However, a client recently decided on running Mageia2 GNU/Linux on Amazon Web Services' Elastic Compute Cloud, so I had to pick up the challenge.

Now as far as distributions go, it seems to me that Mageia is "ok" - a fork of Mandriva that has a nice community developed around it. Unfortunately, what hasn't developed in any interest in running it on EC2. There seems to be a few VPS providers that support it, but not any of the popular ones that I know of. Still, AWS EC2 has the capability of running your own Linux distribution and even your own kernel so I pushed up my sleeves and dug into getting it going. As there's a lot to this configuration, I will break this down over a few posts. Here is an outline:

1. Boarding procedures - performing a chroot install on a local Mageia system 
2. Stormy weather - recompiling the Linux kernel to get Mageia running on EC2
3. Cruising altitude - performing another chroot install for an EBS backed instance

I'll be posting sanitized code samples and links to the key documents that explain the steps.

EDIT 2/7/2013: I've put the code samples up on github.

EBS snapshots and LVM2

I've been meaning to try this for a while to see how it goes - use LVM2 to take an "instantaneous" snapshot of an EBS volume and then let AWS take it's time. I found LVM wasn't as quick as I'd like. Also, I have performance tested this either, so I don't know how bad the latency will be. Either way, I think it's an easy way to get a consistent backup:

# prep
export MYAZ="us-east-1a"
export MYINST="i-XXXXXXXX"
# create 1st EBS volume and attach
ec2-create-volume --size 2 --availability-zone $MYAZ
export VOL0="vol-XXXXXXXX"
ec2-attach-volume $VOL0 --instance $MYINST --device /dev/sdf

# create LVM partition 1
fdisk /dev/sdf
# add to LVM
pvcreate /dev/sdf1
# create a volume group
vgcreate vol0 /dev/sdf1
# create a logical volume
lvcreate -l80%FREE -n test vol0
# format it
mke2fs -j -m0 /dev/vol0/test
# mount it
mkdir -p /mnt/vol0/test
mount /dev/vol0/test /mnt/vol0/test

# lock data consistently

# create LVM snapshot
lvcreate -L300M -s -n test2 /dev/vol0/test

# unlock data consistently

# create EBS snapshot
ec2-create-snapshot $VOL0
export VOL0_SNAP="snap-XXXXXXXX"

# remove LVM snapshot
lvremove vol0/test2

# create 2nd EBS volume from snapshot and attach
ec2-create-volume --snapshot $VOL0_SNAP --availability-zone $MYAZ
export VOL1="vol-YYYYYYYY"
ec2-attach-volume $VOL1 --instance $MYINST --device /dev/sdf

# import snapshot as new volume group
vgimportclone -n vol1 /dev/sdg1
# activate new volume group
vgchange -a y vol1
# mount it
mkdir -p /mnt/vol1/test2
mount /dev/vol1/test2 /mnt/vol1/test2

EDIT 2013.02.10: "lock data consistently" - I highly recommend "fsfreeze" which is built into most Linux distributions nowadays.

rsync + FAT32 filesystem

Found a useful nugget in the rsync FAQ: if your destination filesystem when using rsync is a FAT32 filesystem you need to add the --modify-window=1 option due to problems with the modified times on FAT32. A working example would be:

rsync \
 --progress \
 --delete \
 --verbose \
 --archive \
 --modify-window=1 \
 /path/to/source/dir/ \
 /path/to/fat32/dir/

As always, remember to be careful about those trailing slashes!

Autoscaling revisited

Shortly after writing my previous post about AWS autoscaling, Amazon updated the autoscaling methodology. Instead of triggers they now use autoscaling policies and alarms from CloudWatch to initiate the policy actions. So here's how I create and remove policies and alarms from an autoscaling group. Note: you can't have both triggers and policies on a group, you have to remove the triggers first before adding the policies.

#
# create policies that will scale the group up and down
# note: cooldown is how many seconds to wait before
#       applying the policy again
#
export COOLDOWN=300
export SCALEUP=`as-put-scaling-policy $ASGROUP-scaleUp \
        --auto-scaling-group $ASGROUP \
        --cooldown $COOLDOWN \
        --adjustment=1 \
        --type ChangeInCapacity`
if [ $? -eq 0 ]; then echo OK - $SCALEUP; else echo ERROR; fi

export SCALEDOWN=`as-put-scaling-policy $ASGROUP-scaleDown \
        --auto-scaling-group $ASGROUP \
        --cooldown $COOLDOWN \
        --adjustment=-1 \
        --type ChangeInCapacity`
if [ $? -eq 0 ]; then echo OK - $SCALEDOWN; else echo ERROR; fi

#
# create alarms to implement policies
#

#
# example: Latency on the ELB
#
mon-put-metric-alarm \
        --alarm-name $ASGROUP-HighLatency \
        --namespace "AWS/ELB" \
        --metric-name Latency \
        --statistic Average \
        --period 60 \
        --comparison-operator GreaterThanThreshold \
        --threshold 5.0 \
        --unit Seconds \
        --evaluation-periods 5 \
        --dimensions "LoadBalancerName=$LBNAME" \
        --alarm-actions $SCALEUP
if [ $? -eq 0 ]; then echo OK; else echo ERROR; fi

mon-put-metric-alarm \
        --alarm-name $ASGROUP-LowLatency \
        --namespace "AWS/ELB" \
        --metric-name Latency \
        --statistic Average \
        --period 60 \
        --comparison-operator LessThanThreshold \
        --threshold 0.5 \
        --unit Seconds \
        --evaluation-periods 5 \
        --dimensions "LoadBalancerName=$LBNAME" \
        --alarm-actions $SCALEDOWN
if [ $? -eq 0 ]; then echo OK; else echo ERROR; fi

And to clean up:

mon-delete-alarms --alarm-name $ASGROUP-HighLatency --force
mon-delete-alarms --alarm-name $ASGROUP-LowLatency --force
as-delete-policy $ASGROUP-scaleUp --auto-scaling-group $ASGROUP --force
as-delete-policy $ASGROUP-scaleDown --auto-scaling-group $ASGROUP --force

Dumping memcached

I needed to see if memcached was getting the values I thought it was getting. Everyone knows about "stats" to see if it is getting activity, but I looked around and found that it is possible to get some of that data out without knowing how your app stores the data. So I wrote a script to do it:

#!/bin/sh
HOST="localhost"
if [ "$1" != "" ]; then HOST=$1; fi
COUNT=100
if [ "$2" != "" ]; then COUNT=$2; fi
for slab in `echo "stats items" | nc $HOST 11211 | grep :number | cut -d: -f2 -`
do
  for item in `echo "stats cachedump $slab $COUNT" | nc $HOST 11211 | grep "^ITEM" | cut -d" " -f2 -`
  do
    echo "get $item" | nc $HOST 11211
  done
done

On Cloud n+1

I spent the last few days setting up an autoscaling pool of servers on the Amazon Elastic Compute Cloud. They really have done an excellent job of putting together a great toolset and documentation. I've made some notes on how to do a basic setup, including using the EC2 Elastic Load Balancer. Another cool tool I was able to use for this project was Ubuntu's pre-built EC2 images and the cloud-init package, making auto-deployment of the servers very easy to do.

# Notes on setting up Amazon AWS Auto Scaling
# ===========================================
# ATonns Tue Oct 26 17:37:12 EDT 2010
#

export AVAILZONE="us-east-1a"
#
# create a launch config
#
export LCNAME="test-lc"
as-create-launch-config $LCNAME \
 --image-id ami-f5e0049c \
 --instance-type m1.small
#
# other key args:
#
# /* security group */
# --group {groupname}
# /* meta-data file */
# --user-data-file {filename}
#

#
# create a load balancer
#
export LBNAME="test-lb"
elb-create-lb $LBNAME --headers \
 --availability-zones $AVAILZONE \
 --listener "protocol=http,lb-port=80,instance-port=80" 
#
# add some thresholds that will kick instances out
#
export LBTESTURI="/DONOTREMOVE.php"
elb-configure-healthcheck $LBNAME --headers \
 --target "HTTP:80$LBTESTURI" \
 --interval 5 \
 --timeout 2 \
 --unhealthy-threshold 2 \
 --healthy-threshold 5

#
# create auto-scale group
#
export ASGROUP="test-asg"
as-create-auto-scaling-group $ASGROUP \
 --availability-zones $AVAILZONE \
 --launch-configuration $LCNAME \
 --min-size 1 \
 --max-size 5 \
 --load-balancers $LBNAME

#
# create a trigger
#
export ASTRIGGER="test-trig"
as-create-or-update-trigger $ASTRIGGER \
 --auto-scaling-group $ASGROUP \
 --period 60 \
 --unit Seconds \
 --dimensions "LoadBalancerName=$LBNAME" \
 --namespace "AWS/ELB" \
 --measure Latency \
 --statistic Average \
 --lower-threshold 0.25 \
 --upper-threshold 0.75 \
 --breach-duration 300 \
 --lower-breach-increment=-1 \
 --upper-breach-increment 1 

#
# more metrics
#
http://goo.gl/A4pAd

------------

#
# remove everything
#
as-delete-trigger $ASTRIGGER --auto-scaling-group $ASGROUP --force
as-update-auto-scaling-group $ASGROUP --min-size 0 --max-size 0
count="-1"
while [ $count -ne 0 ]
do
 count=0
 for i in `as-describe-auto-scaling-groups $ASGROUP --show-long`
 do
  type=`echo $i | cut -d, -f1 -`
  if [ $type = INSTANCE ]
  then
   count=`expr $count + 1`
  fi
 done
 echo $count instances left
done
procs="-1"
while [ $procs -ne 0 ]
do
 procs=0
 for i in `as-describe-scaling-activities $ASGROUP --show-long | cut -d, -f4 -`
 do
  if [ "$i" != "Successful" ]
  then
   procs=`expr $procs + 1`
  fi
 done
 echo $procs processes still running
done
as-delete-auto-scaling-group $ASGROUP --force
as-delete-launch-config $LCNAME --force
elb-delete-lb $LBNAME --force

Hugepages and KVM

I've seen the benefits of hugepages before when setting up Oracle and MySQL, but while doing some research I found an article on how to use them with KVM. However, patching /sbin/start_udev and creating an init script to make sure it stays patched just seems like a horrendous idea just to get hugetlbfs mounted on /dev/hugepages. My non-intrusive method is as follows:

1) teach MAKEDEV how to "create the directory" /dev/hugepages on boot. Actually it is creating an additional /dev/null device at /dev/hugepages/null, but it should be harmless to have multiple "null" (major 1, minor 3) devices and also harmless to mount on top of it.

echo 'c $ALLWRITE 1 3 1 1 hugepages/null' > /etc/makedev.d/01hugepages

2) tell udev to create it on boot if needed:

echo 'hugepages/null' > /etc/udev/makedev.d/52-hugepages.nodes

3) tell udev what the right permissions are for it:

echo 'KERNEL=="hugepages*", OWNER="root", GROUP="root", MODE="0775"' > /etc/udev/rules.d/52-hugepages.rules

4) Under CentOS/RHEL run "huge_page_setup_helper.py" to get your hugepages setup

5) Set the hugetlbfs to be mounted on boot:

echo 'hugetlbfs /dev/hugepages hugetlbfs defaults 0 0' >> /etc/fstab

That's it! After a reboot, you can check that hugepages are setup with "sysctl vm.nr_hugepages" and "grep -i huge /proc/meminfo" and check that hugetlbfs is mounted with "mount | grep huge".

Check /proc/meminfo once your KVM guests start to make sure the number of free pages decreases. If not confirm your guest's XML file has "<memoryBacking><hugepages/></memoryBacking>" below the "<currentmemory>" section and that they have "-mem-prealloc -mem-path /dev/hugepages/libvirt/qemu" in the qemu-kvm command line (it should be auto-set by libvirt).

Getting timing out of curl


curl -w " \
time_total %{time_total} \
time_connect %{time_connect} \
time_namelookup %{time_namelookup} \
time_pretransfer %{time_pretransfer} \
time_starttransfer %{time_starttransfer} \
time_redirect %{time_redirect}\n" http://www.example.com


Notes on burning a DVD using Linux

I can't believe I haven't done this before. I guess my desktop has always had a burner and a gui program to do this. Anyway after searching and finding some links, I'm making some notes for next time.

TITLE="012345678901234"
SOURCE_DIR="/path/to/files"
mkisofs -v -A $TITLE -V $TITLE -J -r -o dvd.iso $SOURCE_DIR
eject -t dvd
cdrecord -scanbus dev=ATA # find your DVD burner in the list
cdrecord -v dev=ATA:1,1,0 driveropts=burnfree -dao dvd.iso
eject dvd

I'm sure there's better ways to do this, but it worked pretty well for the first attempt.

rpm queryformat

Another reminder to myself - rpm query to show installed RPMs formatted with name of the file as they were installed (as per the default CentOS/RHEL naming scheme on the install media):

rpm -qa --qf '%{name}-%{version}-%{release}.%{arch}.rpm\n'

step 2: anti-spam, anti-virus

ow. my head.

Not only is the spam problem on the internet horrible, but so is the how-to-implement-spam-prevention problem. There's sooo many walkthroughs, guides, howtos and different packages for different UNIX flavors that to attempt to accomplish the task. Here's the list of tools I'm starting off with:

amavis-new
spamassassin
clamav
razor-agents
pyzor

I started with the adminspotting walkthrough but that's debian based and my CentOS box needs additional configuration. I read over the SA wiki, but still didn't fit right. I think the closest is the HowToForge howto, but my virtual setup is different (file based vs. mysql based). I also added OpenProtect's sa-update channel and I built my own pyzor rpm using the fedora spec file. Below are some key config steps. I might have missed one or two, but I think I got "the big ones". Of course, there are more components that I could add (dcc, DomainKeys, spf, etc. etc.) but my VM is already wheezing on memory and thats with only 2 amavisd children and zero mail traffic.

Man, what a pain in the ass.

---8<---
# install rpmforge pkgs
yum install spamassassin
yum install clamav-db clamav clamd
yum install amavisd-new yum install razor-agents
rpm -ihv /www/src/rpms/pyzor-0.4.0-11.noarch.rpm

#
# for /etc/postfix/main.cf:
#

#
# amavis
#
content_filter=smtp-amavis:[127.0.0.1]:10024
receive_override_options = no_address_mappings

#
# for /etc/postfix/master.cf:
#

#
# amavis setup
#
smtp-amavis unix - - n - 2 smtp
-o smtp_data_done_timeout=1200
-o smtp_send_xforward_command=yes

127.0.0.1:10025 inet n - n - - smtpd
-o content_filter=
-o local_recipient_maps=
-o relay_recipient_maps=
-o smtpd_restriction_classes=
-o smtpd_client_restrictions=
-o smtpd_helo_restrictions=
-o smtpd_sender_restrictions=
-o smtpd_recipient_restrictions=permit_mynetworks,reject
-o mynetworks=127.0.0.0/8
-o strict_rfc821_envelopes=yes
-o smtpd_error_sleep_time=0
-o smtpd_soft_error_limit=1001
-o smtpd_hard_error_limit=1000


[root@vps1 ~]# cd /etc
[root@vps1 etc]# rcsdiff -u clamd.conf
===================================================================
RCS file: RCS/clamd.conf,v
retrieving revision 1.1
diff -r1.1 clamd.conf
72c72,73
< LocalSocket /tmp/clamd.socket
---
> #LocalSocket /tmp/clamd.socket
> LocalSocket /var/run/clamav/clamd

[root@vps1 etc]# rcsdiff -u amavisd.conf
===================================================================
RCS file: RCS/amavisd.conf,v
retrieving revision 1.1
diff -u -r1.1 amavisd.conf
--- amavisd.conf 2008/07/13 17:56:22 1.1
+++ amavisd.conf 2008/07/14 02:35:48
@@ -18,7 +18,7 @@
$daemon_user = "amavis"; # (no default; customary: vscan or amavis), -u
$daemon_group = "amavis"; # (no default; customary: vscan or amavis), -g

-$mydomain = 'example.com'; # a convenient default for other settings
+$mydomain = 'localhost'; # a convenient default for other settings

# $MYHOME = '/var/amavis'; # a convenient default for other settings, -H
$TEMPBASE = "$MYHOME/tmp"; # working directory, needs to exist, -T
@@ -46,7 +46,8 @@
$enable_global_cache = 1; # enable use of libdb-based cache if $enable_db=1
$nanny_details_level = 2; # nanny verbosity: 1: traditional, 2: detailed

-@local_domains_maps = ( [".$mydomain"] ); # list of all local domains
+#@local_domains_maps = ( [".$mydomain"] ); # list of all local domains
+read_hash(\%local_domains, '/virtual/etc/vdomains');

@mynetworks = qw( 127.0.0.0/8 [::1] [FE80::]/10 [FEC0::]/10
10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 );
@@ -90,8 +91,8 @@
auth_required_release => 0, # do not require secret_id for amavisd-release
};

-$sa_tag_level_deflt = 2.0; # add spam info headers if at, or above that level
-$sa_tag2_level_deflt = 6.2; # add 'spam detected' headers at that level
+$sa_tag_level_deflt = 0.0; # add spam info headers if at, or above that level
+$sa_tag2_level_deflt = 4.0; # add 'spam detected' headers at that level
$sa_kill_level_deflt = 6.9; # triggers spam evasive actions (e.g. blocks mail)
$sa_dsn_cutoff_level = 10; # spam level beyond which a DSN is not sent
# $sa_quarantine_cutoff_level = 25; # spam level beyond which quarantine is off
@@ -132,7 +133,8 @@
$MIN_EXPANSION_QUOTA = 100*1024; # bytes (default undef, not enforced)
$MAX_EXPANSION_QUOTA = 300*1024*1024; # bytes (default undef, not enforced)

-$sa_spam_subject_tag = '***SPAM*** ';
+#$sa_spam_subject_tag = '***SPAM*** ';
+$sa_spam_subject_tag = '[SPAM] ';
$defang_virus = 1; # MIME-wrap passed infected mail
$defang_banned = 1; # MIME-wrap passed mail containing banned name
# for defanging bad headers only turn on certain minor contents categories:
@@ -143,11 +145,16 @@

# OTHER MORE COMMON SETTINGS (defaults may suffice):

-# $myhostname = 'host.example.com'; # must be a fully-qualified domain name!
+$myhostname = 'vps1.tonns.com'; # must be a fully-qualified domain name!

# $notify_method = 'smtp:[127.0.0.1]:10025';
# $forward_method = 'smtp:[127.0.0.1]:10025'; # set to undef with milter!

+$final_virus_destiny = D_REJECT;
+$final_banned_destiny = D_REJECT;
+$final_spam_destiny = D_PASS;
+$final_bad_header_destiny = D_PASS;
+
# $final_virus_destiny = D_DISCARD;
# $final_banned_destiny = D_BOUNCE;
# $final_spam_destiny = D_BOUNCE;


#
# NOTE: I also uncommented the clamav checks and commented out all the other
# AV checks, but that diff is too large to bother with here
#

# after following the OpenProtect update docs:
[root@vps1 ~]# cd /usr/share/spamassassin/
[root@vps1 spamassassin]# diff sa-update.cron.orig sa-update.cron
5c5
< /usr/bin/sa-update && /etc/init.d/spamassassin condrestart > /dev/null
---
> /usr/bin/sa-update --gpgkey D1C035168C1EBC08464946DA258CDB3ABDE9DC10 --channel saupdates.openprotect.com --channel updates.spamassassin.org && /etc/init.d/amavisd condrestart > /dev/null

#
# setup razor & pyzor
#
su -s/bin/bash amavis
razor-admin -create
razor-admin -register
pyzor discover

[root@vps1 ~]# cd /etc/mail/spamassassin/
[root@vps1 spamassassin]# diff local.cf.orig local.cf
9a10,22
>
> #pyzor
> use_pyzor 1
> pyzor_path /usr/bin/pyzor
>
> #razor
> use_razor2 1
> razor_config /var/amavis/.razor/razor-agent.conf
>
> #bayes
> use_bayes 1
> use_bayes_rules 1
> bayes_auto_learn 1

step 1: greylisting

Short story: I fiddled with gps for a while since it seems like it would perform better than postgrey. I've thrown in the towel for now. gps has the nice feature of whitelisting on sender, but it just seems like it has too much "other" baggage.

postgrey install:

yum install postgrey
chkconfig postgrey on

# add to /etc/sysconfig/postgrey
# OPTIONS="--unix=$SOCKET --delay=120 --auto-whitelist-clients=8 --greylist-text='Service temporarily unavailable. Please rety in %s seconds.' "

# add to /etc/postfix/main.cf:
# smtpd_recipient_restrictions =
# permit_mynetworks
# reject_unauth_destination
# check_policy_service unix:postgrey/socket

service postgrey start
service postfix restart

Long story: OMGWTFBBQ@$%^@#$!!!! You'd think using a nice database abstraction layer like libdbi would make gps a snap. But nooooo RedHat has to be a total pain in my ass. The include libdbi-dbd RPMs for MySQL and PostgreSQL but not for SQLite. And the one thing I don't want to run on my slicehost is an memory-hogging database server, so SQLite is really what I want. So after contemplating it, I just rolled my own spec file and that did it... mostly. gps and it's accompanying perl script gps-maintain.pl have different opinions on what 'dbtype' should be and what the accompanying db_dbtype_dbdir should be, but a post on the forums allowed me to hack it up so it was working. In the end, I spent a lot of time on it and if postgrey sucks the life out of my VM, I might reconsider gps. But for now, I'm tired of installing complex software.

dive! dive! dive!

Looks like my server located at Dorsai is experiencing extended downtime, reason unknown. I've cutover the key websites, but not all of them and not mail yet. I need to really get the spam filtering, etc. going first. Time to hustle that setup, on the quick. More to come shortly...