upstream agile - software

Using haproxy with multiple backends a.k.a. content switching

We have been using haproxy as a load balancer for autoki (rails) for a while. The setup was relatively simple. We had two servers full of mongrels and one haproxy was simply throwing requests at them using round robin.

Now our requirements have gotten bigger. We have extracted the game into a separate application that is being deployed on its own server independently. The problem that we want to solve with this is that while we want to deploy changes to our servers whenever we want to, the game application has to keep running all the time without interruptions by a deployment and the problems that might be caused by it. Oh, and at the same time we didn’t want to change any urls. The game has been sitting in the external_games controller, so the url was autoki.com/external_games and since we have given this url to people we wanted it to stay the same.

So now we have a new server called external_games.autoki.com with 10 mongrels running on it and we want all requests to /external_games go there and everything else still go to the old servers. The solution to this is to let the load balancer decide which servers to forward the requests to. Since version 1.3 haproxy has implemented exactly this - and calls it content switching. To get thsi to work you have to tweak the haproxy.cfg file a bit. This is the old listen section with one pool of mongrels:

First we have split up our single proxy configuration into a frontend and a backend. We will actually be using two backends - one for gaming and one for the rest, and the frontend will decide which backend to use.

The rule to decide which backend to use is implemented using an acl. The following line creates a rule with the name game that evaluates to tue, if the url contains the string external_games. The haproxy documentation has a whole list of keywords you can use to match against all the contents of a http packet.

Now all we have to do is tell the frontend when to use which backend. We do this by defining a rule for the gaming backend and set a default backend for all requests not matching our acl.

That’s it. Now when we go to http://autoki.com/haproxy?stats we’ll see the two backends up and running. (only after adding stats enable to both backends actually)

post to del.icio.us

Railsconf Europe 2007 Roundup 3 (rubyworks)

rubyworks is a full rails stack put together also by thoughtworks. this time it’s all open source. it features haproxy (supposed to be much faster and light weight than apache_mod_balancer) as load balancer, mongrel as application server, monit and runit for monitoring and controlling mongrel, mysql/postgrs/oracle ruby bindings for database connectivity and also bindings for ferret, libxml, hpricot and rmagick.

everything is nicely packaged as rpms and debian packages so it can easily be installed within 5 minutes (it really works). even better, the packages at the same time pretend to be ruby gems, so you can safely install other debian packages *and* gems that depend on one of the libraries provided by rubyworks.

now we had already set up a cluster of 6 servers for autoki with everything set up more or less perfectly so why would we need rubyworks? answer: to steal the config files for runit/monit and haproxy. our mongrel setup has always been a bit shaky, especially when it came to restarting the mongrels after a deployment. after using the rubyworks setup with runit now everything is stable. (btw runit can run and supervise any process in *nix and is ready to be the successor of the old init which is used by most linux distros to start up all the processes. one advantage is that it starts all processes at once instead of piece by piece, plus runit handles putting a process in the background and keeping it alive there, something mongrel is especially bad at).

post to del.icio.us

Deine Tests wissen die Antwort.

Nachdem wir vor kurzem auf Rails 1.2.2 umgestiegen sind, verbrachte ich letztens einige Stunden damit die Ursache für folgenden Fehler zu finden, der beim starten von mongrel auftrat: 'remove_const': cannot remove Object::ModelTranslation (NameError) Die Logfiles zeigten, dass es Probleme beim Laden von Abhängigkeiten für Globalize gab. Allerdings war die Ursache völlig unklar. Ein Aufruf unserer Unit Tests mit rake lieferte dann die banale Lösung: der Datenbankserver lief nicht.

Das lehrt mal wieder, immer die Tests laufen lassen, wenn es Probleme gibt!

post to del.icio.us