Multiplay Labs

tech hits and tips from Multiplay

Archive for April 17th, 2013

Caching Steam Downloads @ LAN’s

with 32 comments

This article has been superseded by LANcache – Dynamically Caching Game Installs at LAN’s using Nginx

Gone are the days when users install most of their games from physical media such as CD’s / DVD’s instead they rely on high bandwidth internet connections and services such as EA’s Origin and Valve’s Steam.

This causes issues for events which bring like minded gamers together at LAN events such as Multiplay’s Insomnia Gaming Festival, due to the massive amount of bandwidth required to support these services with users patching games and downloading new ones to play with their friends.

With the release of Steam’s new Steampipe creating a local cache of steam download’s, so that game installs at these types of events is significantly quicker and requires dramatically less internet bandwidth, has become much easier.

Steampipe uses standard web requests from Valve Software’s new content servers so standard proxy technology can be used to cache requests.

This article describes how to use FreeBSD + nginx to create a high performance caching proxy for all steam content served by Steampipe

Hardware Specifications
The following is our recommended specification for a machine. Obviously the higher the spec the better, in particular more RAM.

  • Quad Core CPU
  • 32GB RAM (the more the better)
  • 6 x 1TB HD’s
  • 2 x 120GB SSD’s
  • 1Gbps NIC

Machine Install
Our setup starts by performing a FreeBSD ZFS install using mfsBSD we use a RAIDZ on the HD’s e.g.

zfsinstall -d da0 -d da1 -d da2 -d da3 -d da4 -d da5 \
    -r raidz -t /cdrom/8.3-RELEASE-amd64.tar.xz -s 4G

Once the OS is installed setup the L2ARC on the SSD’s

zpool add cache da6 da7

Next install the required packages:

pkg_add -r nginx

Enable FreeBSD http accept filter which improves performance of processing http connections as they are handled in kernel mode:

echo 'accf_http_load="YES"' >> /boot/loader.conf
kldload accf_http

Enable nginx to start on boot by adding nginx_enable="YES" to /etc/rc.conf:

echo 'nginx_enable="YES"' >> /etc/rc.conf

Configuring Nginx
First create some directories:

mkdir -p /data/www/steamproxy
mkdir -p /data/www/logs

The trick to mirroring the steam content so that duplicate requests are served locally is two fold.

  • First spoof *.cs.steampowered.com, content[1-8].steampowered.com and send all traffic to the proxy box
  • Configure nginx to on demand mirror the requests under /depot/ using none spoofing resolvers.

Here’s our example /usr/local/etc/nginx.conf. It uses Google’s open DNS servers for the resolver but any non-spoofed resolvers will do.

user www www;
worker_processes  4;
 
error_log  /data/www/logs/nginx-error.log;
 
events {
	worker_connections 8192;
	multi_accept on;
	use kqueue;
}
 
http {
	include mime.types;
 
	access_log  /data/www/logs/nginx-access.log;
	keepalive_timeout 65;
 
	# steam mirror
	server {
		listen 80 accept_filter=httpready default;
		server_name _;
		access_log /data/www/logs/steam-access.log;
		error_log /data/www/logs/steam-error.log;
		root /data/www/steamproxy;
		index index.html;
		resolver 8.8.8.8;
 
		location /depot/ {
			try_files $uri @mirror;
			access_log /data/www/logs/depot-local.log;
		}
 
		location / {
			proxy_next_upstream error timeout http_404;
			proxy_pass http://$host$request_uri;
			proxy_redirect off;
			proxy_set_header Host $host;
			proxy_set_header X-Real-IP $remote_addr;
			proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
			add_header X-Mirror-Upstream-Status $upstream_status;
			add_header X-Mirror-Upstream-Response-Time $upstream_response_time;
			add_header X-Mirror-Status $upstream_cache_status;
		}
 
		location @mirror {
			proxy_store on;
			proxy_store_access user:rw group:rw all:r;
			proxy_next_upstream error timeout http_404;
			proxy_pass http://$host$request_uri;
			proxy_redirect off;
			proxy_set_header Host $host;
			proxy_set_header X-Real-IP $remote_addr;
			proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
			add_header X-Mirror-Upstream-Status $upstream_status;
			add_header X-Mirror-Upstream-Response-Time $upstream_response_time;
			add_header X-Mirror-Status $upstream_cache_status;
			access_log /data/www/logs/depot-remote.log;
		}
	}
}

We use on demand mirroring instead of proxy “caching” as the files in steam /depot/ never change so storing them permanently eliminates the overhead of cache cleaners and other calculations.

Don’t forget this can also be used with smaller hardware to reduce steam bandwidth requirements in an office environment too 🙂

Update – 2013/08/19 – Added content[1-8].steampowered.com as additional content server hosts which need proxying
Update – 2013/09/10 – Changed from $uri to $request_uri (which includes query string args) required for authentication now.
Update – 2014/04/30 – Superseded by LANcache

Written by Dilbert

April 17th, 2013 at 1:01 pm

Posted in FreeBSD,Gaming,Nginx