Making WooMe

For years I’ve hated web analysis software - Its all sluggish as hell and its all a bit monolithic. Well - about 6 months ago one of our team had a ridiculous idea. He suggested “Why dont we stick the whole lot in a database, then we can just do stuff whenever we want.”

Clearly we sacked him because he was barking mad.

A couple of weeks later, we started mangling our log files and throwing them into a postgres database. Its been a revolutionary idea and here are a couple of things that used to be terribly difficult and are now trivial. We’re a real time web site - with a lot of database interaction and a very heavy server load per page load. Slow pages and page failures are a day to day concern and managing performance issues or the perception of them amongst the management team is something that takes a fair chunk of our day:

“Its allllll really slow”

select count(*) from access_y2010d193 where backendtime >5 and date >(now()-‘5 minutes’::interval);

“It feels like theres a lot of page load failures since the last release”

Munin graph including:

select count(*) from ”’ + today_partition + ”’ where backendstatus in  (500,501,502,503,504)  and date >(now() - ‘15 minutes’::interval) ;

“My page loads are all really slow - everything is taking over 10 seconds”

select count(*) from access_y2010d193 where ip=’complainers ip address” and backendtime >10 and date >(now()-‘1 hour’::interval);