Multiplay Labs

tech hits and tips from Multiplay

Caching Steam Downloads @ LAN’s

with 30 comments

This article has been superseded by LANcache – Dynamically Caching Game Installs at LAN’s using Nginx

Gone are the days when users install most of their games from physical media such as CD’s / DVD’s instead they rely on high bandwidth internet connections and services such as EA’s Origin and Valve’s Steam.

This causes issues for events which bring like minded gamers together at LAN events such as Multiplay’s Insomnia Gaming Festival, due to the massive amount of bandwidth required to support these services with users patching games and downloading new ones to play with their friends.

With the release of Steam’s new Steampipe creating a local cache of steam download’s, so that game installs at these types of events is significantly quicker and requires dramatically less internet bandwidth, has become much easier.

Steampipe uses standard web requests from Valve Software’s new content servers so standard proxy technology can be used to cache requests.

This article describes how to use FreeBSD + nginx to create a high performance caching proxy for all steam content served by Steampipe

Hardware Specifications
The following is our recommended specification for a machine. Obviously the higher the spec the better, in particular more RAM.

  • Quad Core CPU
  • 32GB RAM (the more the better)
  • 6 x 1TB HD’s
  • 2 x 120GB SSD’s
  • 1Gbps NIC

Machine Install
Our setup starts by performing a FreeBSD ZFS install using mfsBSD we use a RAIDZ on the HD’s e.g.

zfsinstall -d da0 -d da1 -d da2 -d da3 -d da4 -d da5 \
    -r raidz -t /cdrom/8.3-RELEASE-amd64.tar.xz -s 4G

Once the OS is installed setup the L2ARC on the SSD’s

zpool add cache da6 da7

Next install the required packages:

pkg_add -r nginx

Enable FreeBSD http accept filter which improves performance of processing http connections as they are handled in kernel mode:

echo 'accf_http_load="YES"' >> /boot/loader.conf
kldload accf_http

Enable nginx to start on boot by adding nginx_enable="YES" to /etc/rc.conf:

echo 'nginx_enable="YES"' >> /etc/rc.conf

Configuring Nginx
First create some directories:

mkdir -p /data/www/steamproxy
mkdir -p /data/www/logs

The trick to mirroring the steam content so that duplicate requests are served locally is two fold.

  • First spoof *.cs.steampowered.com, content[1-8].steampowered.com and send all traffic to the proxy box
  • Configure nginx to on demand mirror the requests under /depot/ using none spoofing resolvers.

Here’s our example /usr/local/etc/nginx.conf. It uses Google’s open DNS servers for the resolver but any non-spoofed resolvers will do.

user www www;
worker_processes  4;
 
error_log  /data/www/logs/nginx-error.log;
 
events {
	worker_connections 8192;
	multi_accept on;
	use kqueue;
}
 
http {
	include mime.types;
 
	access_log  /data/www/logs/nginx-access.log;
	keepalive_timeout 65;
 
	# steam mirror
	server {
		listen 80 accept_filter=httpready default;
		server_name _;
		access_log /data/www/logs/steam-access.log;
		error_log /data/www/logs/steam-error.log;
		root /data/www/steamproxy;
		index index.html;
		resolver 8.8.8.8;
 
		location /depot/ {
			try_files $uri @mirror;
			access_log /data/www/logs/depot-local.log;
		}
 
		location / {
			proxy_next_upstream error timeout http_404;
			proxy_pass http://$host$request_uri;
			proxy_redirect off;
			proxy_set_header Host $host;
			proxy_set_header X-Real-IP $remote_addr;
			proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
			add_header X-Mirror-Upstream-Status $upstream_status;
			add_header X-Mirror-Upstream-Response-Time $upstream_response_time;
			add_header X-Mirror-Status $upstream_cache_status;
		}
 
		location @mirror {
			proxy_store on;
			proxy_store_access user:rw group:rw all:r;
			proxy_next_upstream error timeout http_404;
			proxy_pass http://$host$request_uri;
			proxy_redirect off;
			proxy_set_header Host $host;
			proxy_set_header X-Real-IP $remote_addr;
			proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
			add_header X-Mirror-Upstream-Status $upstream_status;
			add_header X-Mirror-Upstream-Response-Time $upstream_response_time;
			add_header X-Mirror-Status $upstream_cache_status;
			access_log /data/www/logs/depot-remote.log;
		}
	}
}

We use on demand mirroring instead of proxy “caching” as the files in steam /depot/ never change so storing them permanently eliminates the overhead of cache cleaners and other calculations.

Don’t forget this can also be used with smaller hardware to reduce steam bandwidth requirements in an office environment too :)

Update – 2013/08/19 – Added content[1-8].steampowered.com as additional content server hosts which need proxying
Update – 2013/09/10 – Changed from $uri to $request_uri (which includes query string args) required for authentication now.
Update – 2014/04/30 – Superseded by LANcache

Written by Steven Hartland

April 17th, 2013 at 1:01 pm

Posted in FreeBSD,Gaming,Nginx

30 Responses to 'Caching Steam Downloads @ LAN’s'

Subscribe to comments with RSS or TrackBack to 'Caching Steam Downloads @ LAN’s'.

  1. [...] work is based on the nginx configuration file written for the purpose by Steven Hartland at Multiplay. However the attached configuration file iterates on the idea in the following [...]

  2. Have you tested this at a LAN? How much offloading were you able todo? Did your system requirements end up being over, under or right on?

    mhohman

    4 Aug 13 at 18:59

  3. Yes we use it at the UK’s largest LAN Insomina Gaming Festival as well as our smaller even StratLAN

    We managed to offload over 1Gbps, could still have done more to improve throughput with improved HW most importantly more RAM for caching the hot set. We’ll likely add L2ARC next event too to improve disk throughput more.

    Steven Hartland

    4 Aug 13 at 19:30

  4. Thanks for the quick reply! Approximately how much of your inbound traffic was from steams cdn vs other traffic? Did this make a large impact on perceived speed?

    mhohman

    4 Aug 13 at 19:42

  5. Have you guys deployed any other types of caching (Besides basic http cache) to keep more traffic local?

    mhohman

    4 Aug 13 at 19:45

  6. This kept all the steam traffic local apart from the initial request so there was no need for additional caching for steam downloads :)

    Steven Hartland

    4 Aug 13 at 19:48

  7. With regards inbound traffic, steam would pretty much saturate the 1Gbps inbound line if we let it, with the proxy this was significantly reduced only using the inbound line for the first request for a piece of content.

    Obviously there are other types of content which then have space to breath, so to speak, such as standard web traffic and streaming services like youtube; some of which themselves could benefit from proxying but in an LAN environment getting the games is a key factor.

    Steven Hartland

    5 Aug 13 at 00:07

  8. I was wondering what you are using as the firewall infront of this. We are using PFSense and are load balancing / traffic shaping with dual WAN. I want to see about incorporating this into our setup. Any light you can shed would be awesome.

    PFSense has SQUID as a package proxy. If I have to use an external then I would have to make a firewall rule and my fear is that it will bog the router with all the other stuff going on.

    sideout

    8 Aug 13 at 12:58

  9. Pfsense

    Steven Hartland

    8 Aug 13 at 13:00

  10. Would it be possible to get a peek at your config on PFSense as to how your sending the request to Nginx or an example of your firewall rules?

    I would be eternally grateful.

    sideout

    8 Aug 13 at 13:09

  11. Or did you just use the remote cache method with squid?

    sideout

    8 Aug 13 at 13:12

  12. No PFSense rules required as detailed above its all DNS based.

    Steven Hartland

    8 Aug 13 at 15:06

  13. Okay. Thanks for the info. So basically your spoofing all DNS for steam to hit the proxy box then and it serves it up from there.

    I will try this out . Thanks again.

    sideout

    8 Aug 13 at 16:04

  14. Just fired this up @ the BIG GAME 34 and it took ~30Mb/s off our average bandwidth usage (so far) – excellent work, thanks for the recipe.

    putty_thing

    10 Aug 13 at 14:08

  15. So i have the server up and applied the config file and I have DNS set to point to the server for *.steampowered.com / *.cs.steampowered.com but steam only starts in offline mode.

    Any hints on what to look at? I can see entries in the steam-access.log file.

    sideout

    12 Aug 13 at 13:14

  16. Well i am getting closer. I am getting a steam timeout when trying to install. If someone has some documentation on the config on the Freebsd side that would be awesome or willing to help me out , it would be greatly appreicated. I have nginx installed on freebsd like above but i can’t get it to cache.

    sideout

    12 Aug 13 at 17:25

  17. […] With their huge (thousands of guests) i-series LAN events, Multiplay had the same problem as us but on a much larger scale. They were one of the first to take advantage of the new distribution platform, using a combination of DNS overrides and a custom HTTP proxy using Nginx. They very kindly published their approach. […]

    Thank you, Valve

    15 Sep 13 at 12:52

  18. I noticed that the dns is missing some hosts like content2a.steampowered.com & valve[1-300].cs.steampowered.com so a lot of files are not being cached.

    mlwalk3r

    11 Oct 13 at 15:12

  19. To add what I posted before this is my method,
    address=/.steampowered.com/172.16.0.3
    address=/store.steampowered.com/208.64.202.69
    this will cache all the sub-domains they use and removes the store avoiding a man in the middle attack.

    mlwalk3r

    11 Oct 13 at 15:30

  20. not to spam I notice that they use cdn for there side so if you do it my way with the dns its,

    address=/store.steampowered.com/208.64.202.69
    address=/cdn3.steampowered.com/4.26.249.126
    address=/cdn3.store.steampowered.com/209.84.3.254

    thanks!

    mlwalk3r

    12 Oct 13 at 01:37

  21. I had to “chown -r www /data/www/steamproxy”
    Otherwise I was getting errors in the nginx logs that it couldn’t create files in steamproxy and it wasn’t saving any cached files.

    Curium

    21 Oct 13 at 12:18

  22. Yep that’s part of a standard install, usually chown -r www:www

    Steven Hartland

    21 Oct 13 at 12:22

  23. […] setup is based off of the great work of Astrolox which was based on a guide from Multiplay with modifications to also cache LoL content. The following config will create a on demand mirror […]

  24. […] having copies of the same steam game. It’s merely a simplified version of the well thought Caching Steam Downloads @ LAN’s article. Obviously, to do this, you need to have your own home server. For instance, it should work out of […]

  25. “We use on demand mirroring instead of proxy “caching” as the files in steam /depot/ never change so storing them permanently eliminates the overhead of cache cleaners and other calculations.”

    While this does seem to be what should happen based on nginx docs, I’ve found that this configuration still cares about the last modified timestamp. Unfortunately this prevents it from working properly with a Greyhole-based volume as file storage, since Samba doesn’t handle that properly. The files are saved, but the local copy is never actually used. The next download of the same file goes right back out to the server, where if I move the storage location to a directory on a normal disk everything works as expected.

    When saving on the Greyhole pool nginx throws an error for every file complaining about utimes() being an unsupported operation, but nowhere am I able to tell it to actually stop caring about that.

    wolrah

    17 Jan 14 at 14:20

  26. […] With their huge (thousands of guests) i-series LAN events, Multiplay had the same problem as us but on a much larger scale. They were one of the first to take advantage of the new distribution platform, using a combination of DNS overrides and a custom HTTP proxy using Nginx. They very kindly published their approach. […]

  27. During insomnia51 we tested out a large rework of the our steam cache, which we now call lancache as its not just steam and includes: Valves Steam, Riot games, Blizzard games, Hirez games, Sony games and Origin. Watch out for an update with all the details soon :)

    Steven Hartland

    29 Apr 14 at 00:00

  28. […] year we posted our Caching Steam Downloads @ LAN’s article which has been adopted by many of the LAN event organisers in the community as the baseline […]

  29. […] […]

Leave a Reply

You must be logged in to post a comment.