Multiplay Labs

tech hits and tips from Multiplay

LANcache – Dynamically Caching Game Installs at LAN’s using Nginx

with 314 comments

Last year we posted our Caching Steam Downloads @ LAN’s article which has been adopted by many of the LAN event organisers in the community as the baseline for improving download speeds and helping avoid internet saturation when you have 10’s – 1000’s of gamers at events all updating and installing new games from Steam.

This rework builds on the original concepts from our steam caching, brings in additional improvements from the community, such as the excellent work by the guys @ as as well as other enhancements.

Due to the features used in this configuration it requires nginx 1.6.0+ which is the latest stable release at the time of writing.

Nginx Configuration
In order to make the configuration more maintainable we’ve split the config up in to a number of smaller includes.

In the machines directory you have lancache-single.conf which is the main nginx.conf file that sets up the events and http handler as well as the key features via includes: custom log format, cache definition and active vhosts.

include lancache/log_format;
include lancache/caches;
include vhosts/*.conf;

The custom log format adds three additional details to the standard combined log format “$upstream_cache_status” “$host” “$http_range”. These are useful for determine the efficiency of each segment the cache.

In order to support the expanding number of downloads supported by LANcache we’ve switched the config from static mirror to using nginx’s built in caching.

In our install we’re caching data to 6 x 240GB SSD’s configured in ZFS RAIDZ so we have just over 1TB of storage per node.
To ensure we don’t run out of space we’ve limited the main installs cache size to 950GB with custom loader details to ensure we can init the cache quicker on restart.
The other cache zone is used for none install data so is limited to just 10GB.

We also set proxy_temp_path to a location on the same volume as the cache directories so that temporary files can be moved directly to the cache directory avoid a file copy which would put extra stress on the IO subsystem.

proxy_cache_path /data/www/cache/installs levels=2:2 keys_zone=installs:500m inactive=120d max_size=972800m loader_files=1000 loader_sleep=50ms loader_threshold=300ms;
proxy_cache_path /data/www/cache/other levels=2:2 keys_zone=other:100m inactive=72h max_size=10240m;
proxy_temp_path /data/www/cache/tmp;

Here we define individual server entries for each service we’ll be caching, we do this so that each service can configure how its cache works independently.
In order to allow for direct use of the configs in multiple setups without having to edit the config files themselves we made use of named entries for all listen addresses.

The example below shows the origin server entry which listens on lancache-origin and requires the spoof entries

We use server_name as part of the cache_key to avoid cache collisions and so add _ as the wildcard catch all to ensure all requests to this servers IP are processed.

For performance we configure the access log with buffering.

# origin
server {
        listen lancache-origin accept_filter=httpready default;
        server_name origin _;
        # DNS entries:
        # lancache-origin
        access_log /data/www/logs/lancache-origin-access.log main buffer=128k flush=1m;
        error_log /data/www/logs/lancache-origin-error.log;
        include lancache/node-origin;

The include is where all the custom work is done, in this case lancache/node-origin. There are currently 5 different flavours of node: blizzard, default, origin, pass and steam.

Origin’s CDN is pretty bad in that currently prevents the caching of data, due to this we’re force to ignore the Cache-Control and Expires headers. The files themselves are very large 10GB+ and the client uses range requests to chunk the downloads to improve performance and provide realistic download restart points.

By default nginx proxy translates a GET request with a Range request to a full download by stripping the Range and If-Range request headers from the upstream request. It does this so subsequent range requests can be satisfied from the single download request. Unfortunately Origin’s CDN prevents this so we have to override this default behaviour by passing through the Range and If-Range headers. This means the upstream will reply with a 206 (partial content) instead of a 200 (OK) response and hence we must add the range to the cache key so that additional requests are correctly.

The final customisation for Origin is to use $uri in the proxy_cache_key, we do this as the Origin client uses a query string parameter sauth=<key>

Blizzard have large downloads too, so to ensure that requests are served quickly we cache 206 responses in the same way as for Origin.

All Steam downloads are located under /depot/ so we have a custom location for that which ignores the Expires header as Steam sets a default Expires header.

We also store the content of requests /serverlists/ as these requests served by give us information about hosts used by Steam to process download request. The information in these files could help identify future DNS entries which need spoofing.

Finally the catch all / entry caches all other items according to the their headers.

This is the default which is used for riot, hirez and sony it uses standard caching rules which caches based on the proxy_cache_key "$server_name$request_uri"

Required DNS entries
All of the required DNS entries are for each service are documented their block server in vhosts/lancache-single.conf which as of writing is:
lancache-steam * * *







lancache-microsoft * *

You’ll notice that each entry starts with lancache-XXXX this is entry used in the listen directive so no editing of the config is required for IP allocation to each service. As we’re creating multiple server entries and each is capturing hostnames using the _ wildcard each service must have its own IP e.g. lancache-steam =, lancache-riot =, lancache-blizzard =, lancache-hirez =, lancache-origin = and lancache-sony =

Hardware Specifications
At Insomnia 51 we used a hybrid of this configuration which made use of two machines working in a cluster with the following spec:

  • Dual Hex Core CPU’s
  • 128GB RAM
  • 6 x 240GB SSD’s ZFS RAIDZ
  • 6 x 1Gbps Nics
  • OS – FreeBSD 10.0

These machines where configured in a failover pair using CARP @ 4Gbps lagg using LACP. Each node served ~1/2 of the active cache set to double the available cache space to ~1.8TB, with internode communication done on using a dedicated 2Gbps lagg

LANcache Stats from Insomnia 51
For those that are interested at its initial outing at Insomnia 51 LANcache:

  • Processed 6.6 million downloads from the internet totalling 2.2TB
  • Served 34.1 million downloads totalling 14.5TB to the LAN
  • Peaked at 4Gbps (the max capacity) to the LAN

Config Downloads

* 2015-09-17 – Updated info about required DNS entries
* 2015-10-09 – Linked initial public version of configs on github

Written by Dilbert

April 30th, 2014 at 3:33 pm

Posted in FreeBSD,Gaming,Nginx

314 Responses to 'LANcache – Dynamically Caching Game Installs at LAN’s using Nginx'

Subscribe to comments with RSS or TrackBack to 'LANcache – Dynamically Caching Game Installs at LAN’s using Nginx'.

  1. ok, so im having a weird issue, steam doesnt seem to be hitting my nginx for some reason, riot is caching fine though. if i ping any of the steam servers from any machine on the network it pings my nginx machine like it should. everything in my dnsmasq settings looks right, no idea what could be going on any ideas?


    6 Jun 16 at 19:05

  2. Check with wireshark on the client machines where they are downloading from and the dns names they used to get there, you could be seeing a new set of hosts we’ve not encountered before.


    7 Jun 16 at 20:13

  3. yeah sorry i fixed it, seems steam added * i missed that somewhere lol. thanks though :).


    8 Jun 16 at 23:01

  4. Hi I ran in to a few issues when running this. I am using Ubuntu 14.04 instead of FreeBSD. I have the cache working, but I wanted to circle back to some commands I commented out that were causing nginx to crash. Any help would be appreciated thanks!

    “accept_filter=httpready” are not supported on this platform

    line 19 – “unknown directive “proxy_cache_revalidate”
    line 35 – “unknown directive “proxy_cache_purge”

    line 11 – “unknown directive “range_cache_size”

    line 14 – “unknown directive “range_cache_size”

    line 18 – “unknown directive “range_cache_size”

    line 14 – “unknown directive “range_cache_size”


    4 Jul 16 at 19:56

  5. httpready requires the accf_http_load=”YES” in /boot/loader.conf you can load it on the fly with kldload accf_http
    range_cache_size requires the currently unreleased module, see the download options for the older config which doesn’t require this.


    4 Jul 16 at 21:39

  6. Do I need to be using FreeBSD for KLDLOAD? I set this up in Ubuntu.


    5 Jul 16 at 01:20

  7. Yes you do otherwise just remove it as Linux doesn’t have kernel level http filters.


    5 Jul 16 at 08:18

  8. I guess my next question would be. What is the benefit of it? Faster downloads? More sessions? I don’t use Linux much so my knowledge is minimal. I have it working, but if I can make the performance better with KLDLOAD I’ll go through the trouble of setting up FreeBSD.


    5 Jul 16 at 14:07

  9. Its a performance feature, it avoids the switch from kernel to user space before the session is ready.

    We use FreeBSD because it has good all round OS which doesn’t suffer from the issue Linux does where there may different sources of the truth depending on the flavour you use. That and features such as full ZFS are a massive administration win.

    With regards performance there’s a reason why netflix uses FreeBSD for it’s content distribution system 😉


    5 Jul 16 at 15:52

  10. It looks like my Steam clients are hitting an IP ( rather than hostname now. Has anyone else experienced this?


    18 Jul 16 at 01:57

  11. Nope


    18 Jul 16 at 09:20

  12. […] At home I have a custom-built NAS that I use for all of my storage needs. I use it all the time to store files like game client downloads and backups of my games on Steam. It just makes it easier to restore after an install or when a friend needs to copy a game over. Recently however I was referred to LANcache. […]

  13. Hi, trying the slice configuration, the proxy_max_temp_file_size is accepting a max size of: 2047m
    any ideas?


    31 Jul 16 at 16:04

  14. If its accepting the option whats the issue?


    31 Jul 16 at 23:26

Leave a Reply

You must be logged in to post a comment.