gitorious-org-setup

We run Varnish for caching on gitorious.org. Varnish is basically a zero-config setup, and will do wonderful things to the responsiveness of your app provided you take care of two things:

• Any request with a Set-Cookie response header will not be cached by Varnish
• As long as the Cache-Control response header is set to public, Varnish will cache the request for as long as specified by the max-age parameter.

Varnish is set up to handle port 80 (HTTP) on our servers, and is set up with a single backend: the private Nginx server mentioned below. This means that Varnish will cache as much as it can of any requests on port 80 of gitorious.org.

We run Nginx on port 443, since Varnish doesn't run SSL. The server running on gitorious.org:443 will serve any static files directly from Rails.root on the server, and proxy any other request to the public Varnish server on port 80. By sending these requests through Varnish, we get to use the same cache for HTTP and HTTPS.

Nginx is also set up to listen on a private port, where it receives requests from (only) Varnish. Like the HTTPS Nginx server, this will deliver any static assets directly, and pass all other requests over a UNIX socket to Unicorn.

Nginx is also set up to deliver sending of other files, intercepting the X-Accel-Redirect response headers emitted by Gitorious; equivalent of Apache mod_xsendfile's X-Sendfile headers. To enable this, we have frontend_server:nginx in the gitorious.yml file on gitorious.org, and the configuration in Nginx looks like this:

# Will deliver /srv/gitorious/tarballs-cache/filename.tar.gz
location /tarballs/ {
  internal;
  alias /srv/gitorious/tarballs-cache/;
}

location /git-http/ {
  internal;
  alias /srv/gitorious/repositories/;
}

If a user requests https://gitorious.org/gitorious/mainline/archive-tarball/master Gitorious will (once the tarball has been generated) respond with an X-Accel-Redirect header like /tarballs/gitorious-mainline-$sha1.tar.gz ($sha1 is which SHA1 the master branch points to at request time), which is picked up by Nginx by the first rule above. Nginx will resolve this to the file /srv/gitorious/tarballs-cache/$sha1.tar.gz and deliver this file directly.

The /tarballs/ locations are marked as private in Nginx, which means a user isn't allowed to request them directly. Using Apache with mod_x_sendfile the X-Sendfile header would contain the full path to the repository, while Nginx lets us maintain a symbolic mapping resolved by Nginx itself.

The same mechanism is used for Git over HTTP.

Unicorn is a Ruby based HTTP server leaning heavily on fundamental UNIX concepts. Unicorn works by starting a master process which loads the full Rails environment. Once this is done, it will run fork(2) to create 16 child processes (this is how many workers we have running on gitorious.org). These child processes will inherit the socket set up by the master process, which means the kernel will take care of load balancing the requests between the active worker processes.

Unicorn is designed for chaotic situations, like the one we have on gitorious.org. An IO intensive application like Gitorious will run into problematic situations caused by things like IO load all the time, and our previous setup (Apache and Passenger) would end up with some really CPU and memory hungry processes running for a long time. Our Unicorn setup has a strict timeout of 30 seconds for any request, which means that any request that takes more than 30 seconds to complete will cause the worker process to be killed. And once the worker is killed, the master will immediately fork again, with the new child process ready to serve requests right away.

Like the good UNIX citizen Unicorn is, the easiest way to communicate with it is using signals. We use the following signals on gitorious.org:

• We send a USR2 to the master process after deploying a new version of Gitorious. This causes the master process to spawn a new master process; using the newly deployed code. Once the new master is started, it looks for a PID file for the "old" master process in pid_dir/unicorn.pid.oldbin. If this file exists, it sends a QUIT signal to that, which causes it do shut down itself and all its worker processes. This gives us a zero downtime deployment, which is a big deal for us.
• We send a USR1 to the master process after rotating the logs (done by logrotate). This causes the master and worker processes to reopen the log files.

The Unicorn configuration file we use on gitorious.org is practically identical to the one in Gitorious mainline, except we use a full path in RAILS_ROOT since expanding a relative path would resolve to Capistrano's app/releases directory.

As you may be aware, Git supports two kinds of Git over HTTP. The "dumb" protocol, which basically creates an HTTP request for every commit, and "smart" HTTP which uses HTTP as a carrier for git-upload-pack. The git-scm.com site has all the details.

Gitorious ships with a simplistic version of the former, a Rails metal which uses Nginx' X-Accel-Redirect or Apache's X-Sendfile to deliver Git objects directly from the file system. This method is inefficient and somewhat error-prone, but after setting up the new servers last week the stability of this solution was not near good enough.

Friday afternoon we set up another backend for Git over HTTP, using a Jgit based servlet for doing Git over HTTP. I ran it using Maven for a few hours, after setting up Varnish to proxy traffic to git.gitorious.org to this Java app. This speeded up HTTP cloning a lot, so we decided to run it through the weekend.

After running it for around 48 hours the Jetty server was using around 5 GB of (resident) RAM, having served around 80.000 git clone operations through the weekend.

The app we're running for this is a single Java webapp, and the source code is at https://gitorious.org/gitorious/jgit-http-server. We're working on a push+pull implementation in Jruby, also using JGit, but we really like the way Java and Jgit works for us.

On Ubuntu/Debian servers:

sudo apt-get install redis-server
update-rc.d redis-server defaults
sudo service start redis-server

On RHEL/CentOS-like systems:

sudo yum install redis
sudo chkconfig redis on
sudo /etc/init.d/redis start

This is a simple setting in gitorious.yml:

messaging_adapter: resque

This depends on which server you're running. If you're using Passenger:

touch tmp/restart.txt

If you're using Unicorn

kill -USR2 /path/to/unicorn.pid

The bin/rake script shipping with Gitorious will run a rake task from anywhere, setting up the correct RAILS_ENV, HOME environment variables and ensuring the task is run as the user specified as gitorious_user in gitorious.yml, and Resque workers are run with Rake:

QUEUE=* /path/to/gitorious/bin/rake resque:work

To run dedicated workers for single queues, change the QUEUE environment variable, eg.

QUEUE="/queue/GitoriousPush" /path/to/gitorious/bin/rake resque:work

Since the bin/rake task can be called directly, we simply added an Upstart script with an exec stanza (no shell required) to control the Resque workers:

description "Run a Resque worker on all queues"
author "Marius Mårnes Mathiesen <marius@gitorious.com>"

start on started gitorious/unicorn
stop on runlevel [06]

env QUEUE=*
env PIDFILE=/path/to/gitorious/pids/resque-worker1.pid
exec /path/to/gitorious/bin/rake resque:work