Multiplay Labs

tech hits and tips from Multiplay

GeoIP C interface isn’t all thread safe

without comments

It’s documented that GeoIP C interface after v1.3.6 is thread safe, with the exception of GEOIP_CHECK_CACHE, however this is not really the case.

A large number of the C API functions such as: GeoIP_new, GeoIP_open_type, GeoIP_db_avail call the internal _GeoIP_setup_dbfilename function which statically initialises the GeoIPDBFileName global pointer without any locking.

This means that if several threads call any of these methods to gain a geoip handle then its possible to trash the memory structure while its being read and hence result in a crash.

So if your using the GeoIP C interface in a threaded application ensure that you use locking around the use of these methods.

Written by Dilbert

July 4th, 2013 at 2:11 pm

Posted in Code

Configuring Supermicro IPMI interface NIC using ipmitool

without comments

Newer Supermicro IPMI interfaces come configured by default in “failover” mode which means that the IPMI will bind to either the dedicated IPMI NIC port or share with one the the machine NIC ports.

This can cause IPMI to come up on wrong NIC and hence be inaccessible if the dedicated NIC doesn’t detect a link.

You can use ipmitool to change this behavour

First query the current setting:
ipmitool raw 0x30 0x70 0x0c 0

The result will be one of the following
0x00 = Dedicated
0x01 = Onboard / Shared
0x02 = Failover

Next to configure it you can use one of the following.

For older models:
ipmitool raw 0x30 0x70 0x0c 1 1 0

For X9 motherboards:
ipmitool raw 0x30 0x70 0x0c 1 0

References for this can be found here:
http://www.supermicro.com/support/faqs/faq.cfm?faq=9829
http://www.supermicro.com/support/faqs/faq.cfm?faq=14417

Written by Dilbert

June 27th, 2013 at 5:03 pm

Posted in FreeBSD,Networking

Force failing a disk in RAID set attached to an Areca controller

without comments

Sometimes a disk may be failing and timing out yet still working enough so that the RAID controller doesn’t drop it from the array.

This results in very poor performance from the entire array as the controller waits for timeout’s on the problem disk.

If your using an Areca controller there’s a command that can be used to manually fail such a disk allow it to rebuild onto an available Hotspare without having to have someone onsite to remove the problem disk.

WARNING – Using these commands can damage your array so be very sure before using them!

This can be achieved via the Areca web interface from the Raidset Functions -> Rescue Raidset option. In the text entry field enter the following command:-
FailDisk <disk channel>

The <disk channel> MUST be entered correctly otherwise you could fail the wrong disk for example entering
FailDisk Ch2 will fail disk attached to Channel 4 and not Channel 2.

The correct input is includes the 0 i.e. Ch02 you see in the Hierarchy display so in full:
FailDisk Ch02

The areca-cli also provides a similar option e.g. disk fail drv=2 but this only works on new controllers and not older ones e.g. ARC-1220 (at least not with FW 1.49)

Note a rather confusing version of this information can be found on Areca’s FAQ

Written by Dilbert

June 18th, 2013 at 9:01 pm

Posted in OS's

Fixing bad FreeBSD boot.config on ZFS

without comments

Today we had a machine rebooted with a broken /boot.config file, preventing it from booting.

It took us some time to find the solution to for a full ZFS machine so worth a mention.

From boot prompt simply enter:

root:/boot/zfsloader

Written by Dilbert

May 14th, 2013 at 1:46 pm

Posted in FreeBSD,ZFS

Caching Steam Downloads @ LAN’s

with 32 comments

This article has been superseded by LANcache – Dynamically Caching Game Installs at LAN’s using Nginx

Gone are the days when users install most of their games from physical media such as CD’s / DVD’s instead they rely on high bandwidth internet connections and services such as EA’s Origin and Valve’s Steam.

This causes issues for events which bring like minded gamers together at LAN events such as Multiplay’s Insomnia Gaming Festival, due to the massive amount of bandwidth required to support these services with users patching games and downloading new ones to play with their friends.

With the release of Steam’s new Steampipe creating a local cache of steam download’s, so that game installs at these types of events is significantly quicker and requires dramatically less internet bandwidth, has become much easier.

Steampipe uses standard web requests from Valve Software’s new content servers so standard proxy technology can be used to cache requests.

This article describes how to use FreeBSD + nginx to create a high performance caching proxy for all steam content served by Steampipe

Hardware Specifications
The following is our recommended specification for a machine. Obviously the higher the spec the better, in particular more RAM.

  • Quad Core CPU
  • 32GB RAM (the more the better)
  • 6 x 1TB HD’s
  • 2 x 120GB SSD’s
  • 1Gbps NIC

Machine Install
Our setup starts by performing a FreeBSD ZFS install using mfsBSD we use a RAIDZ on the HD’s e.g.

zfsinstall -d da0 -d da1 -d da2 -d da3 -d da4 -d da5 \
    -r raidz -t /cdrom/8.3-RELEASE-amd64.tar.xz -s 4G

Once the OS is installed setup the L2ARC on the SSD’s

zpool add cache da6 da7

Next install the required packages:

pkg_add -r nginx

Enable FreeBSD http accept filter which improves performance of processing http connections as they are handled in kernel mode:

echo 'accf_http_load="YES"' >> /boot/loader.conf
kldload accf_http

Enable nginx to start on boot by adding nginx_enable="YES" to /etc/rc.conf:

echo 'nginx_enable="YES"' >> /etc/rc.conf

Configuring Nginx
First create some directories:

mkdir -p /data/www/steamproxy
mkdir -p /data/www/logs

The trick to mirroring the steam content so that duplicate requests are served locally is two fold.

  • First spoof *.cs.steampowered.com, content[1-8].steampowered.com and send all traffic to the proxy box
  • Configure nginx to on demand mirror the requests under /depot/ using none spoofing resolvers.

Here’s our example /usr/local/etc/nginx.conf. It uses Google’s open DNS servers for the resolver but any non-spoofed resolvers will do.

user www www;
worker_processes  4;
 
error_log  /data/www/logs/nginx-error.log;
 
events {
	worker_connections 8192;
	multi_accept on;
	use kqueue;
}
 
http {
	include mime.types;
 
	access_log  /data/www/logs/nginx-access.log;
	keepalive_timeout 65;
 
	# steam mirror
	server {
		listen 80 accept_filter=httpready default;
		server_name _;
		access_log /data/www/logs/steam-access.log;
		error_log /data/www/logs/steam-error.log;
		root /data/www/steamproxy;
		index index.html;
		resolver 8.8.8.8;
 
		location /depot/ {
			try_files $uri @mirror;
			access_log /data/www/logs/depot-local.log;
		}
 
		location / {
			proxy_next_upstream error timeout http_404;
			proxy_pass http://$host$request_uri;
			proxy_redirect off;
			proxy_set_header Host $host;
			proxy_set_header X-Real-IP $remote_addr;
			proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
			add_header X-Mirror-Upstream-Status $upstream_status;
			add_header X-Mirror-Upstream-Response-Time $upstream_response_time;
			add_header X-Mirror-Status $upstream_cache_status;
		}
 
		location @mirror {
			proxy_store on;
			proxy_store_access user:rw group:rw all:r;
			proxy_next_upstream error timeout http_404;
			proxy_pass http://$host$request_uri;
			proxy_redirect off;
			proxy_set_header Host $host;
			proxy_set_header X-Real-IP $remote_addr;
			proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
			add_header X-Mirror-Upstream-Status $upstream_status;
			add_header X-Mirror-Upstream-Response-Time $upstream_response_time;
			add_header X-Mirror-Status $upstream_cache_status;
			access_log /data/www/logs/depot-remote.log;
		}
	}
}

We use on demand mirroring instead of proxy “caching” as the files in steam /depot/ never change so storing them permanently eliminates the overhead of cache cleaners and other calculations.

Don’t forget this can also be used with smaller hardware to reduce steam bandwidth requirements in an office environment too 🙂

Update – 2013/08/19 – Added content[1-8].steampowered.com as additional content server hosts which need proxying
Update – 2013/09/10 – Changed from $uri to $request_uri (which includes query string args) required for authentication now.
Update – 2014/04/30 – Superseded by LANcache

Written by Dilbert

April 17th, 2013 at 1:01 pm

Posted in FreeBSD,Gaming,Nginx

Making mysqldump v5.5 work with v5.6 servers

without comments

Due to a bugs in mysqldump that ship with v5.5 & v5.6 neither are compatible with each others DB making backups across version impossible.

The first issue is that v5.5 mysqldump against a v5.6 server failes with: “Couldn't execute 'SET OPTION SQL_QUOTE_SHOW_CREATE=1'” the second is that v5.6 mysqldump against a v5.5 server fails with “Couldn't execute 'SELECT @@GTID_MODE': Unknown system variable 'GTID_MODE' (1193)

There is however a dirty hack which works around this. Starting with mysqldump from v5.5 edit the binary and search for the test “SET OPTION SQL_QUOTE_SHOW_CREATE” and replace each character in “OPTION” with a space, so you end up with: “SET        SQL_QUOTE_SHOW_CREATE

Written by Dilbert

April 9th, 2013 at 1:44 pm

Posted in MySQL

FreeBSD LACP balancing

without comments

When using FreeBSD’s lagg interface one of the protocol options is LACP which is useful when bonding with switches to provide redundancy as well as additional capacity.

When we tried this with 8.3-RELEASE the results weren’t what was expected with only one NIC in the LACP seeing real amounts traffic in either direction.

After digging around in the LAGG code it became apparent that the hashing algorithm being used was the most likely cause of the issue as in our case we had FLOWTABLE removed from our kernel.

Looking in the FreeBSD source tree we found r232629 which adds the ability to configure the hashing method to include layer 2 (mac), layer 3 (IP) and layer 4 (port) details. Applying this to 8.3 and then configuring for l3 hashing with lagghash l3 significantly improved the balancing across the NIC’s for our application, so if your using LACP and aren’t seeing good balancing across your physical interfaces this is well worth a shot.

Written by Dilbert

March 21st, 2013 at 3:59 pm

Posted in FreeBSD

Listing package names only in FreeBSD

without comments

It’s sometimes useful to be able to get a list package names only or packages which match a specific pattern. The little used -E flag to pkg_info achieves this quickly and easily.

pkg_info -E '*'

Written by Dilbert

March 8th, 2013 at 11:06 am

Posted in FreeBSD

Renaming root zfs pool on FreeBSD

without comments

While not often required its sometimes nice to be able to rename the root zfs pool.

Armed with a mfsbsd cdrom this is a relatively painless process under FreeBSD (8.3-RELEASE in out case).

In this example we’re renaming the default root zpool from tank to zroot
1. Boot from mfsbsd cdrom
2. Import the root pool into an alternative location specifying a cachefile, this is in the important bit, as using -R sets cachefile=none which won’t generate the changes required.

zpool import -o altroot=/mnt -o cachefile=/boot/zfs/zpool.cache tank zroot

3. Copy the new cachefile to zpool updating the existing one.

cp /boot/zfs/zpool.cache /mnt/boot/zfs/zpool.cache

4. Update /mnt/boot/loader.conf with the new details e.g. vfs.root.mountfrom="zfs:zroot/root"
5. Reboot

If you have more thank one pool then always best to import them all even if not renaming them to ensure the new zpool.cache contains all your pools

Written by Dilbert

January 14th, 2013 at 5:01 pm

Posted in FreeBSD,ZFS

Making gstat work in FreeBSD jail’s

without comments

gstat requires /dev/devstat so for it to work under a jail you need enable devfs an ensure that /dev/devstat is available.

To enable devfs for all jails set the following in /etc/rc.conf

jail_devfs_enable="YES"

To add devstat to the default jail ruleset you can add the following to /etc/devfs.rules

[devfsrules_jail=4]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add path devstat unhide

Written by Dilbert

December 12th, 2012 at 2:46 pm

Posted in FreeBSD