Making WooMe

At Woome we often suddenly create a new set of key information, such as a regularly changing nginx config. We use mercurial extensively, so the typical next step at this point initialise a repo for the data set and check everything in. As a team we’ve trained ourselves to check in changes to most admin data sets automatically, with a ticket number in the comment, so this is very nearly self documenting change tracking with easy rollback.

Now, that gives us good change accountability and recoverability however it does tend to leave us with mercurial repos littered around the infrastucture slowly becoming more and more important. What we now do is auto-locate these repos and publish them to a central box for hgweb access presenting us with a rapid bare-os recovering in the case of hardware failure. 

This script runs from cron on our admin server every few hours:

#!/bin/bash                                                                     
# sysopshosts is a flat file hostlist                                          
hostlist=$(cat ~woome/sysops/releases/sysopshosts  |tr -d ':')
base=/data/repos

cd $base
for host in $hostlist
do
    # locate any mercurial repos under /etc                                     
    repolist=$(ssh $host find /etc -iname .hg -print \
        |grep -v /etc/.hg|sed -e 's/\/.hg//' )
    # Work through them                                                         
    for repo in  $repolist
    do
        # create a directory of host32-etc-nginx for host32/etc/nginx           
        repodir=$(echo $repo |tr \/ \-)
        targetdir="${host}${repodir}"
        # if the target directory already exists then check                     
        # if the repo looks related                                             
        if [ -d $targetdir ]
        then
            HGUNRELATED=`hg -R $targetdir  in ssh://${host}//${repo}/ 2>&1 \    
                grep -cE "(unrelated|no suitable)"`
            if [ "${HGUNRELATED}" -gt "0" ]
            then
                echo "found unrelated repo - $targetdir moving aside"
                mv ${targetdir} ${targetdir}.unrelated.`date +%Y-%m-%d`
            fi
        fi
        # now attempt to either create or update the repo                       
        if [ -d $targetdir ]
        then
            echo Updating existing repo $targetdir
            pushd  $targetdir
            hg pull
            popd
        else
            echo Found new repo $targetdir
            hg clone ssh://${host}/${repo} ${targetdir}
        fi
    done
done

We use hgweb to publish the repo directory .

An example use case (simplified slightly). We had a hardware failure of our main nagios monitoring system during a data centre outage/aircon outage. We were immediately blind to the state of our infrastucture. We obtained sufficient monitoring to finish our recovery with the following steps on a spare server:

sudo apt-get install nagios2  lighttpd
rm -fr /etc/nagios2 /etc/lighttpd
hg clone ssh://host30/data/repos/host65-etc-nagis2 /etc/nagios2
hg clone ssh://host30/data/repos/host65-etc-lighttpd /etc/lighttpd
... minor config changes to lighttpd and dns for new host.....
sudo /etc/init.d/lighttpd restart
sudo /etc/init.d/nagios2 restart