Minecraft Monitoring with Zabbix Part 2

This is just a quick update on the previous post about monitoring zabbix, and assumes you already have zabbix set up and monitoring the basic properties of your server. (if you don’t, see here.)

We’re going to add a simple php script on our minecraft server that queries the number of players online, then return this value to the zabbix server by adding it ot the agent as a UserParameter.

Adding the PHP:

No point in reinventing the wheel, so we’re going to grab  xPaw’s MinecraftQuery.class.php from here and save it on the minecraft server as “MinecraftQuery.class.php”

Then we need to add our own file which uses this one, queries our server ip and port, and returns the number:

<?php
require_once(‘MineCraftQuery.class.php’);

$Query = new MinecraftQuery( );

try
{
//$arg[1] is the argument passed in when calling the php, in this case the port
$Query->Connect( ‘127.0.0.1’, $argv[1] );

$result = $Query->GetInfo( );

echo $result[“Players”];
}
catch( MinecraftQueryException $e )
{
//return 0 if something goes wrong
echo 0;
}

unset($Query);
unset($result);
?>

Now we add a custom user parameter to the bottom of /etc/zabbix/zabbix_agentd.conf :

UserParameter=minecraft.players[*],php /home/mc/ServerQuery.php $1

The first part is the key to identify the item in zabbix server: the * in brackets tells the agent to use the values that the server is sending in place of any numbers prepended with $. This means with a single UserParameter on the agent, we can query multiple minecraft servers from the zabbix server, simply by passing a different port in with the trigger. The second part is the command to run our script, change it to match your script’s path.

Finally, on the server we need to reload the zabbix agent so it know about the new key:

/etc/init.d/zabbix-agent restart

Adding the Host Item

In the zabbix web interface, go to configuration -> Hosts, select your minecraft server and go to Item -> Create Item.

For the key, enter the key you used as a UserParamter, but replace the * with the server query port.

Thats all you need to do to add the item, but to make it a bit neater you can set the Units to “players” and create a new application group such as “minecraft players”

Here’s mine:

Capture

Repeat the above for any extra servers you have, and add all of the items to a graph to display them together (see part 1 for more on graphs) You should end up with something like this:

Capture

To link externally to your graphs you need to enable guest readonly access, and use the following address:

http://mydomain.com/zabbix/chart2.php?graphid=<graphID>

(you can find the graph ID in the URL when you are configuring the graph)

MC DDOS protection part 3: Monitoring

So now our tunnels are set up, our round-robin SRV records are distributing our players across our VPS’s, next it would be nice if we could check the traffic passing across them, and also keep an eye on the system stats for each machine.

In this section we’re going to look at the basic structure of Zabbix, the terminology, and adding custom graphs.

Basics

Zabbix is made up of 3 parts:

Server Core
  • Receives and processes stats from each server
  • Stores the data
  • Can send alerts based on certain criteria (“triggers”)
Server Web Interface
  • Displays graphs and information about the servers
  • Can be on a separate machine from the core
Zabbix Agent
  • Runs on each of the servers you want to monitor
  • Reports the servers status to Zabbix
  • Can be customised with extra scripts

Installing zabbix core and web interface

To install Zabbix you will need a webserver and a mysql database. Your webserver can be on another machine to the one you’re installing zabbix server on, you just need to tell it where during the configuration.

Zabbix has some good documentation so I won’t re-write it, the version I used is here but feel free to use a later release, the only thing that’s important is that you don’t use the default from the apt repository: it will most likely be < v2.0 and not support interface auto discovery properly.

You will also need to enable access to port 10051 incoming to your zabbix server, either through your firewall or its iptables.

Installing zabbix agent on your servers

As above: add the repository, update,  and “apt-get install zabbix-agent”.

Few extra notes:

  • Make sure you’re on a version later than 2.0 with “zabbix-agent –version”
  • Change “Server=127.0.0.1” in /etc/zabbix/zabbix_agentd.conf to your zabbix server IP
  • Restart the agent with “/etc/init.d/zabbix_agentd restart”
  • Allow port 10050 through iptables with ” iptables -A INPUT -p tcp –dport 10050 -j ACCEPT”

Host Configuration

Assuming you’ve now installed Zabbix and aren’t screaming at me for not including more detail, you’re probably now staring at the web GUI wondering where to start. Here are the basic terms you need to know:

Items

  • Measure a specific value on the server, e.g. CPU load

Applications

  • Group Items into categories, e.g. “OS Memory” = all memory items.

Discovery Rules

  • Auto-generate new items by scanning the system for extra hard drives, interfaces, etc.

Triggers

  • Perform an event when an item matches certain criteria, e.g. send an email when CPU load exceeds 100%

Graphs

  • Self explanatory, graph an item or set of items on a host.

Screens

  • Can be used to create a page with multiple graphs.

Templates

  • Group all of the above together into a package that can be easily added to a new host with minimal extra configuration.

Add new host

New hosts can be added via configuration > Hosts > Create host

Give the host a descriptive name, add it to “linux servers” group, and enter it’s IP address. You then need to add a template to tell it what to monitor. “Template OS Linux” covers everything we need for our VPS, including an interface discovery script for our GRE tunnels.

Save your new host, and go back to Configuration > Hosts. After a few minutes you should see Status: Monitored and a green Z icon indicating that everything is fine.

If you have a grey icon, you may have not added any templates (hence nothing to monitor.

If there is a red icon then there is most likely an issue with communication to the host, hover over the red icon to see the details, check that your firewall rules are allowing the correct ports, and that the IP’s are correct.

Graphing:

If you navigate to Monitoring > Graphs and select your new host.  There should be a graph available with a name like “Network traffic on gre1”, which if you have any players connected will look something like this:

Capture

Using this page when can now switch between each host we’ve added and view the bandwidth usage on each interface. Great start, but it would be nice if we could combine the information into one place, which we’ll do with a custom graph.

Creating a custom graph

Since our MC server host has an interface for each tunnel, we can add each item for each tunnel onto a new custom graph:

Go to Configuration > Hosts and select “Graphs” on the entry for your MC server.

Select “Create Graph” in the top right corner.

You can leave everything default except the name and the Items.

Select “add items” and then select each incoming and outgoing item for your GRE interfaces. You can change any colours you want, and preview your changes in the preview tab.

Once you’re happy with the result, save the result and your new graph should appear under Monitoring > Graphs.

Capture

Screens

Depending on how many VPS’s you have and which interfaces you have selected, your graph may look a bit messy, even with some custom colours assigned to interfaces. Another option is to use a screen to display the data, with each interfaces as a separate graph, but all on the same page.

To do this go to Configuration > Screens and “Create Screen.”

Give it a name and decide an initial size, you can change it later.

Save it and it should appear in the list of screens: select your screen and you will be presented with a grid and an option to change the data for each one.

Select the data you want and you’re done, you can now view your new screen under Monitoring > Screens:

Capture

And we’re done for now. Thanks for reading, next up: traffic shaping on our VPS’s


			
				
			
			

Minecraft DDOS protection part 2: Cloudflare

So we’ve set up our GRE tunnels and redirected all of the minecraft traffic hitting our VPS’s down their respective tunnels. The only problem is that there’s nothing to distribute players between all our VPS’s, which is where DNS records come into play. And while we’re at it we’ll protect our website too.

Before we continue, I don’t work for cloudflare, I’m not getting paid by them, and I’m not too keen on some of their slightly over the top marketing. With that said, here’s why I’m using them:

Why cloudflare?

Fast

Cloudflare use an anycast DNS network so the nearest DNS server to you is the one that responds, you can read more about it here

Free

Their basic plan covers everything we need to do, and is completely free.

Easy to use

I’ve bought domains from a few companies, and cloudflares web interface has to be the nicest I’ve seen, their API is also well documented and easy to work with.

Extras

Cloudflare performs some caching, lowering the bandwidth and resource usage of your site, provides analytics for you with zero setup, and if your site goes down, it will host a static cache of your web pages where possible.

Website Setup

There are a few options for website protection setup depending on your web server location:

Hosting website on the same server as minecraft:

Cloudflare isn’t required here, you can simply duplicate your VPS’s iptables rules and replace 25565 with 80, forwarding the web requests through the tunnel to the protected server. With that said I’d still recommend going with cloudflare for the reasons above.

Hosting with another provider:

Since you’re using a different connection and IP, attacks on the minecraft server won’t affect the site and don’t expose the MC server IP. However your website could still be attacked, and if your host doesn’t provide and DDOS protection, you may as well use cloudflare’s.

Hosting on another server on the same connection

This is the situation I’m in, my webserver sits behind a firewall, on the same LAN and internet connection as my MC server. I can’t use my tunnels to forward traffic as their endpoints sit on the wrong server, and leaving the webserver on the global IP will leave that IP exposed to attackers.

Moving to cloudflare

To use cloudflare for your domain you need to need to change your nameservers with your registrar, cloudflare has covered how to do this with the most common registrars here.

Once that’s done it will take a while to move over, in the mean time cloudflare will provide you with a list of subdomains it’s picked up automatically, and it allows you do decide whether or not to use cloudflares network to protect them. It also adds an extra “direct” subdomain which is set to bypass their network. I would recommend removing this since anyone that knows you’re using cloudflare, or simply guesses that you might be, can use it to try and get your real IP. Everything else on their site is well documented so I won’t go over it again.

Adding records

Adding A Records

To start with, we need to add an A record for each VPS, so that instead of IP’s, we have a nice readable subdomain we can point our SRV records to, for example “minecraft.itschr.is”. This is as easy as clicking “Add” and then entering the new subdomain name and IP.

Adding SRV records

Since the domain pointing to our website is now sitting behind a cloudflare IP and not our server IP, and cloudflare doesn’t support forwarding MC traffic, we need to add extra information to tell the players’ Minecraft client what to do when trying to connect to our domain.

SRV records enable us to point our domain to an alternative A record, and even tell the players’ client an alternative port to use to connect.

Wikipedia’s description covers SRV records better than I can, so I’ll just focus on what’s important to us.

  • name

This is location that the client must connect to to receive this SRV record: if set to “@” it will be the domain (itschr.is,) if set to anything else it will be a subdomain (minecraft.itschr.is.)

  • target

Where the SRV record redirects the client to, in this case our VPS’s A record

Result

Now if we add multiple SRV records with the same name, cloudflare will rotate through each target, serving up a new one every time the TTL expires.

Lets say we have 2 VPS’s in the US, and 2 in Europe. We can create an SRV record for each VPS, and player connections will be distributed across all of them. On top of that, we can also pair the US servers together under under a “us.itschr.is” subdomain, and the EU servers together under “eu.itschr.is.” This then allows the players to use a server nearer to them for general use (faster ping but lower attack-redundancy) and still use the main domain if they have issues connecting.

If an attacker takes out a single VPS, it will only affect players connected via that VPS, or players being served that VPS address through SRV records (which will change after the 5 minute TTL)

That’s all for now, I’ll come back to this later and talk about how we can lower the TTL further, and automatically remove the SRV record of a VPS under attack.

Minecraft DDOS protection part 1: GRE Tunnels

Chiisana from /r/admincraft has a really nice write-up about basic DDOS protection here, so rather than rehashing it I’ll just write a brief summary and then continue to extend the ideas a little, covering GRE Tunnels, Zabbix monitoring, and DNS automation with Cloudflare.

Basic Idea

When dealing with a Denial of Service by sheer bandwidth, by the time the traffic hits our server it’s already too late: no amount of shaping or blocking will change the fact that our link is saturated. To protect it, we sit a few cheap VPS’s between the players and the server and limit the bandwidth before it hits our server. The players are distributed between VPS’s using round robin DNS, and if attackers saturate one VPS link, the others will still remain up.

Adding GRE Tunnels

Rather than using tcptunnel, I’m using GRE tunnels, so that the minecraft server gets the IP of the player connecting, and not the VPS IP.

The BuyVM guide is a good start for setting up a GRE tunnel, but it mixes up the VPS and server commands a little, and doesn’t really explain what the commands are doing.

VPS commands:

echo ‘net.ipv4.ip_forward=1’ >> /etc/sysctl.conf

iptunnel add gre1 mode gre local VPS_GLOBAL_IP remote MC_GLOBAL_IP ttl 255

ip addr add 192.168.10.2/30 dev gre1
ip link set gre1 up

iptables -t nat -A POSTROUTING -s 192.168.10.0/30 -j SNAT –to-source VPS_GLOBAL_IP
iptables -t nat -A PREROUTING -p tcp -d VPS_GLOBAL_IP –dport 25565 -j DNAT –to-destination 192.168.10.1:25565
iptables -A FORWARD -p tcp -d 192.168.1.1 –dport 25565 -m state –state NEW,ESTABLISHED,RELATED -j ACCEPT

Breakdown

  • echo ‘net.ipv4.ip_forward=1’ >> /etc/sysctl.conf

This enables IP forwarding, and only needs to be run once.

  • iptunnel add gre1 mode gre local VPS_GLOBAL_IP remote MC_GLOBAL_IP ttl 255

This creates a new tunnel interface with the name “gre1”, with type set to “gre”,  local and remote IP’s defining the tunnel endpoints.

  • ip addr add 192.168.10.2/30 dev gre1

We then give the interface a private IP address used to communicate with the interface on the other end of the tunnel.

  • ip link set gre1 up

Bring the interface up, it should now be listed when running “ifconfig”

  • iptables -t nat -A POSTROUTING -s 192.168.10.0/30 -j SNAT –to-source VPS_GLOBAL_IP

Re-write the source IP of all packets coming from our tunnel out to the players to the global IP of the VPS

  • iptables -t nat -A PREROUTING -p tcp -d VPS_GLOBAL_IP –dport 25565 -j DNAT –to-destination 192.168.10.1:25565

Change the destination port and IP of the packets incoming from the VPS’s global IP to the other end of the tunnel

  • iptables -A FORWARD -p tcp -d 192.168.10.1 –dport 25565 -m state –state NEW,ESTABLISHED,RELATED -j ACCEPT

Finally we re-route the traffic down our GRE tunnel to the MC server.

MC Server commands:

iptunnel add gre1 mode gre local MC_GLOBAL_IP remote VPS_GLOBAL_IP ttl 255
ip addr add 192.168.10.1/30 dev gre1
ip link set gre1 up

echo ‘101 ATGRE’ >> /etc/iproute2/rt_tables
ip rule add from 192.168.10.0/30 table ATGRE
ip route add default via 192.168.10.2 table ATGRE

Breakdown

The first block is just the same as for the VPS except the IP’s are swapped round; remember not to use a subnet that conflicts with your LAN subnet.

  • echo ‘101 GRETUN’ >> /etc/iproute2/rt_tables

Linux has a pretty powerful set of routing features, in this case allowing us to define multiple routing tables so that we can define some source routing policies. This command adds a new routing table into rt_tables, that we can use to add some rules specific to our tunnel.

  • ip rule add from 192.168.10.0/30 table GRETUN

Tell our system that all traffic from 192.168.10.0/30 will be dealt with by our “GRETUN” routing table.

  • ip route add default via 192.168.10.2 table ATGRE

Send any traffic from our 192.168.10.0/30 subnet (that we have no route for) to the other end of our tunnel, the VPS.

Once your happy with your config, you can add it to /etc/rc.local so that it runs each time the system boots. This means you can change it easily simply by restarting.

Notes:

  • If your MC server is sitting behind a firewall and has no global IP, you can use 0.0.0.0 as the MC IP in the MC server config, point your VPS’s tunnel to your firewall’s global IP, and set up the firewall to pass gre traffic through to your MC server.
  • For any other services sitting on the same machine as the MC server, you can simply duplicate the iptables rules on the VPS, replacing the ports and protocol with that of your other service.
  • Since my website sits on a different machine behind the firewall, I’m going to use Cloudflare to mask it’s IP, which I’ll cover later on.