Log housekeeping with python

This week I was able to finally finish some python code I've been writing for $WORK - a script to rotate webserver logs directly to S3. This task was similar to something I'd done a long, LONG time ago (14 years since the first rev!) in a programming language far, far away. I had a much bigger chip on my shoulder then -  site analytics really isn't done with log parsing anymore, so I skipped the whole test for open filehandles and email notifications. Anywhere, here it is - enjoy!

https://github.com/dialt0ne/rotate-to-s3

Kudos to Justin for critiquing my python.

Self-signing a certificate... quickly

I've been using SSL/TLS certs for a long, long time - I've even had to re-issue my personal CA cert after it expired after 5 years. However, every time I've issued a self signed cert for an internal site, openssl prompted me interactively for the Country, State, Locality, etc. etc. blah, blah, blah. The lack of automation was exceptionally annoying. I knew the defaults could be customized so that only the Common Name would have to be entered, but that wasn't enough. The openssl req manual page has a non-working example of a config file that shouldn't prompt (Sample configuration containing all field values) but it doesn't work. After spending considerable time trying to craft a custom, template openssl.cnf file today, I finally found a blog post that mentions the -subj argument that completes the certificate request without any prompting. The only prompting now done is for the rsa command if you're encrypting your keyfile. And of course, this can be automated with the -passin arg, if needed. Here is a full example:

# FQDN of SSL/TLS site
CN="fhqwhgads.example.com"

# preflight
C="US"
ST="New York"
L="New York"
O="Example.com Inc."
OU="Systems Team"
emailAddress="devnull@example.com"

# create a private key
openssl genrsa -out ${CN}.key 2048
# create a certificate request
openssl req \
 -new \
 -subj "/C=$C/ST=$ST/L=$L/O=$O/OU=$OU/CN=$CN/emailAddress=$emailAddress" \
 -key ${CN}.key \
 -out ${CN}.csr
# create cert
openssl x509 -req -days 3650 -in ${CN}.csr -signkey ${CN}.key -out ${CN}.crt

#
# optional - encrypt key
#
# move key
mv ${CN}.key ${CN}.key.plain
# encrypt key
# (add '-passin pass:password' or '-passin file:pathname' for no prompting)
# see openssl(1) manpage
openssl rsa -des3 -in ${CN}.key.plain -out ${CN}.key.crypt
# rename key
mv ${CN}.key.crypt ${CN}.key
# clean up
rm ${CN}.key.plain

Disabling TRACE and TRACK methods

After reading a blog post about how to disable TRACE and TRACK for compliance, I've taken an extra step - limit HTTP requests to only "the big three":

        RewriteEngine On
        RewriteCond %{REQUEST_METHOD} !^(GET|HEAD|POST)
        RewriteRule .* - [F]

It's possible you might want to add "OPTIONS" to that list or "DELETE|PUT" to be RESTful, but as with most implementations, YMMV.

Splitting traffic with an F5 BigIP LTM iRule

Another item filed under "notetoself" - how to split traffic by URI with an iRule applied to a virtual server on an F5 BigIP LTM.

when CLIENT_ACCEPTED {
   set default_pool [LB::server pool]
}
when HTTP_REQUEST {
  if { [HTTP::uri] starts_with "/path/to/split/off" } {
    pool pool_to_split_to
  } else {
    pool $default_pool
  }
}

Normally, I am against this type of hack. I believe that content should have a unique location - if there's two URLs that can get you to the same bit of content, people will use them interchangeably and it will cause nothing but headaches. However, in this case the $default_pool content is a Tomcat stack running a custom framework backed by Oracle and the pool_to_split_to is a LAMP stack running drupal backed by MySQL e.g. they couldn't be any different. This is the best way to unify the URL to access both without creating unnecessary extra hops across the network (say, using apache's mod_proxy_http).

Autoscaling revisited

Shortly after writing my previous post about AWS autoscaling, Amazon updated the autoscaling methodology. Instead of triggers they now use autoscaling policies and alarms from CloudWatch to initiate the policy actions. So here's how I create and remove policies and alarms from an autoscaling group. Note: you can't have both triggers and policies on a group, you have to remove the triggers first before adding the policies.

#
# create policies that will scale the group up and down
# note: cooldown is how many seconds to wait before
#       applying the policy again
#
export COOLDOWN=300
export SCALEUP=`as-put-scaling-policy $ASGROUP-scaleUp \
        --auto-scaling-group $ASGROUP \
        --cooldown $COOLDOWN \
        --adjustment=1 \
        --type ChangeInCapacity`
if [ $? -eq 0 ]; then echo OK - $SCALEUP; else echo ERROR; fi

export SCALEDOWN=`as-put-scaling-policy $ASGROUP-scaleDown \
        --auto-scaling-group $ASGROUP \
        --cooldown $COOLDOWN \
        --adjustment=-1 \
        --type ChangeInCapacity`
if [ $? -eq 0 ]; then echo OK - $SCALEDOWN; else echo ERROR; fi

#
# create alarms to implement policies
#

#
# example: Latency on the ELB
#
mon-put-metric-alarm \
        --alarm-name $ASGROUP-HighLatency \
        --namespace "AWS/ELB" \
        --metric-name Latency \
        --statistic Average \
        --period 60 \
        --comparison-operator GreaterThanThreshold \
        --threshold 5.0 \
        --unit Seconds \
        --evaluation-periods 5 \
        --dimensions "LoadBalancerName=$LBNAME" \
        --alarm-actions $SCALEUP
if [ $? -eq 0 ]; then echo OK; else echo ERROR; fi

mon-put-metric-alarm \
        --alarm-name $ASGROUP-LowLatency \
        --namespace "AWS/ELB" \
        --metric-name Latency \
        --statistic Average \
        --period 60 \
        --comparison-operator LessThanThreshold \
        --threshold 0.5 \
        --unit Seconds \
        --evaluation-periods 5 \
        --dimensions "LoadBalancerName=$LBNAME" \
        --alarm-actions $SCALEDOWN
if [ $? -eq 0 ]; then echo OK; else echo ERROR; fi

And to clean up:

mon-delete-alarms --alarm-name $ASGROUP-HighLatency --force
mon-delete-alarms --alarm-name $ASGROUP-LowLatency --force
as-delete-policy $ASGROUP-scaleUp --auto-scaling-group $ASGROUP --force
as-delete-policy $ASGROUP-scaleDown --auto-scaling-group $ASGROUP --force

On Cloud n+1

I spent the last few days setting up an autoscaling pool of servers on the Amazon Elastic Compute Cloud. They really have done an excellent job of putting together a great toolset and documentation. I've made some notes on how to do a basic setup, including using the EC2 Elastic Load Balancer. Another cool tool I was able to use for this project was Ubuntu's pre-built EC2 images and the cloud-init package, making auto-deployment of the servers very easy to do.

# Notes on setting up Amazon AWS Auto Scaling
# ===========================================
# ATonns Tue Oct 26 17:37:12 EDT 2010
#

export AVAILZONE="us-east-1a"
#
# create a launch config
#
export LCNAME="test-lc"
as-create-launch-config $LCNAME \
 --image-id ami-f5e0049c \
 --instance-type m1.small
#
# other key args:
#
# /* security group */
# --group {groupname}
# /* meta-data file */
# --user-data-file {filename}
#

#
# create a load balancer
#
export LBNAME="test-lb"
elb-create-lb $LBNAME --headers \
 --availability-zones $AVAILZONE \
 --listener "protocol=http,lb-port=80,instance-port=80" 
#
# add some thresholds that will kick instances out
#
export LBTESTURI="/DONOTREMOVE.php"
elb-configure-healthcheck $LBNAME --headers \
 --target "HTTP:80$LBTESTURI" \
 --interval 5 \
 --timeout 2 \
 --unhealthy-threshold 2 \
 --healthy-threshold 5

#
# create auto-scale group
#
export ASGROUP="test-asg"
as-create-auto-scaling-group $ASGROUP \
 --availability-zones $AVAILZONE \
 --launch-configuration $LCNAME \
 --min-size 1 \
 --max-size 5 \
 --load-balancers $LBNAME

#
# create a trigger
#
export ASTRIGGER="test-trig"
as-create-or-update-trigger $ASTRIGGER \
 --auto-scaling-group $ASGROUP \
 --period 60 \
 --unit Seconds \
 --dimensions "LoadBalancerName=$LBNAME" \
 --namespace "AWS/ELB" \
 --measure Latency \
 --statistic Average \
 --lower-threshold 0.25 \
 --upper-threshold 0.75 \
 --breach-duration 300 \
 --lower-breach-increment=-1 \
 --upper-breach-increment 1 

#
# more metrics
#
http://goo.gl/A4pAd

------------

#
# remove everything
#
as-delete-trigger $ASTRIGGER --auto-scaling-group $ASGROUP --force
as-update-auto-scaling-group $ASGROUP --min-size 0 --max-size 0
count="-1"
while [ $count -ne 0 ]
do
 count=0
 for i in `as-describe-auto-scaling-groups $ASGROUP --show-long`
 do
  type=`echo $i | cut -d, -f1 -`
  if [ $type = INSTANCE ]
  then
   count=`expr $count + 1`
  fi
 done
 echo $count instances left
done
procs="-1"
while [ $procs -ne 0 ]
do
 procs=0
 for i in `as-describe-scaling-activities $ASGROUP --show-long | cut -d, -f4 -`
 do
  if [ "$i" != "Successful" ]
  then
   procs=`expr $procs + 1`
  fi
 done
 echo $procs processes still running
done
as-delete-auto-scaling-group $ASGROUP --force
as-delete-launch-config $LCNAME --force
elb-delete-lb $LBNAME --force

Getting timing out of curl


curl -w " \
time_total %{time_total} \
time_connect %{time_connect} \
time_namelookup %{time_namelookup} \
time_pretransfer %{time_pretransfer} \
time_starttransfer %{time_starttransfer} \
time_redirect %{time_redirect}\n" http://www.example.com