Hands up if you use www.speedtest.net ?

Speedtest.net® by OOKLA is the de-facto standard for measuring Internet connectivity and the performance and availability of your Internet connection. But did you know it has a command-line interface? This means that it can easily be integrated with Zabbix.

Small and medium sized enterprises, schools, not-for-profit organisations largely depend on the same Internet provision that we use at home. The only difference is likely to be that with a “business grade” Internet connection you’ll probably have a defined service level in terms of performance and availability.

Nobius shows you how

This article shows how Zabbix and OOKLA can work together in harmony to give you a straightforward view of your Internet performance and availability.

We assume that you have basic familiarity with Zabbix and Linux.

We should also mention that this is a very BASIC configuration. There are some steps that are crude to say the least but the objective here is to show conceptually how it can be done. This guide is published as a “kickstart” guide only. We strongly encourage you to seek further knowledge and understanding rather than blindly following my words.

Let’s start by listing the “ingredients” we’ll need….

1. Zabbix
   We used a straightforward 4.4.0 virtual appliance for simplicity. Definitely not recommended for serious monitoring, but it works as an example.
2. Internet Access from Your Zabbix Server
   We’re going to set this up using the Zabbix Server to run Speedtest. You can also configure things so that Speedtest runs from a Zabbix proxy – useful if you want to monitor Internet connectivity at remote locations.
3. The Speedtest® CLI
   This can be downloaded from here. Download the version for your OS. The Zabbix appliance VM we’re using is based on Ubuntu Xenial.

That’s the ingredients.

I’m going to assume you’ve got Zabbix running and you’ve installed the Speedtest® CLI on your Zabbix server.

First, check Speedtest works OK on your Zabbix server by running:
appliance@zabbix:~$ speedtest -h

N.B. The first time you run Speedtest CLI you’ll be asked to accept some licensing and GDPR notices. These messages only appear the first time you run Speedtest.

After accepting the licensing information, you should see something like this:

Notice the ‘-f’ parameter. This tells Speedtest how to show the output. This is an important feature because Zabbix can handle JSON directly.

• If you run appliance@zabbix:~$ speedtest -f json you get something like this;
  which is pretty much perfect for what we’ll need, even though it looks complicated.
  
• Now the fun bit starts. We’re going to configure Speedtest CLI as a UserParameter in Zabbix. This will allow us to use the output to generate various Zabbix items. Remember a Zabbix item can only contain one value but the output when we run Speedtest CLI will give us multiple values. To handle this, we need to create an item containing the entire JSON output from the Speedtest CLI and then create dependent items to obtain individual metrics. Schematically, it looks like this;

With each dependent item being a child of the master item.

Only the master item actually executes the Speedtest CLI.

The steps to make it work

Step 1: Configure the UserParameter;

a) Edit (vi / vim) the /etc/zabbix/zabbix_agent.conf file and find the “UserParameter” section:

b) Add the following line:
UserParameter=nobius.speedtest,sudo -u zabbix usr/bin/speedtest --accept-license --accept-gdpr -f json

c) In the same file, find the TimeOut section and change the Timeout parameter value to 30:

Save the file and restart the Zabbix agent.

Explainer for Step 1.

For step 1.b you might think that including the ‘sudo -u’ part is a little wrong. However, it seems that the writers of the Speedtest CLI® – OOKLA have some code in Speedtest CLI ® that causes it to crash when called by the Zabbix agent. This has been raised with OOKLA and we’re awaiting a response. This does mean though that we MUST perform Step 2.(below). Carefully.
In Step 1.c, we are increasing the Timeout value for processing. Running Speedtest CLI ® can take a few seconds – certainly more than the default of 3 seconds. In our test environment it takes on average 20 – 23 seconds.

Step 2: Make sure the zabbix agent can run the Speedtest CLI program.

a) Add the user zabbix to the sudoers files.
There are lots of ways to do this (it’s Linux after all) one way is to edit the /etc/sudoers file (use visudo NOT vi, vim or nano).
Add the following line at the end:
username     ALL=(ALL) NOPASSWD:ALL

b) Important Step. When Speedtest CLI is run for the first time, it creates a directory structure in the home directory of the user running it. This needs to be replicated under the home directory of the zabbix user:

appliance@zabbix:~$ cd ~appliance	# Assuming you’re using the Zabbix appliance and logged on as appliance.
appliance@zabbix:~$ ls -a . .. .bash_history .bash_logout  .bashrc .config .profile .ssh  .sudo_as_admin_successful	# If you’ve already run speedtest, you’ll see a directory called .config. This directory tree needs to be replicated in the home directory of the zabbix user.
appliance@zabbix:~$ cp -R ~appliance/.config ~zabbix	# This creates the .config/ookla/speedtest-cli.json file.
root@zabbix:~$ chown -R zabbix:zabbix ~zabbix/.config	Sudo into root and changes the ownership of the directories and files to zabbix:zabbix.

Explainer for Step 2.
For Step 2.a we appreciate that this may not be the best way to do it. As we said, this is a simple route to working. The zabbix user itself does not have a login shell defined, which does mitigate this step a little.
For Step 2.b we manually recreated the structure of directory and files that are created by Speedtest CLI itself when executed for the first time. Zabbix doesn’t have a login shell defined and we didn’t want to set one up for obvious reasons.

Step 3. Test that the UserParameter is working

appliance@zabbix:~$ zabbix_get -s 127.0.0.1 -k nobius.speedtest should result in something like this:

{"type":"result","timestamp":"2020-02-03T22:11:08Z","ping":{"jitter":0.16400000000000001,"latency":18.266999999999999},"download":"bandwidth":8467913,"bytes":55840457,"elapsed":6615},"upload":"bandwidth":2332421,"bytes":32429585,"elapsed":15015},"packetLoss":0,"isp":"BT","interface":"internalIp":"192.168.146.129","name":"ens33","macAddr":"00:0C:29:6A:32:26","isVpn":false,"externalIp":"86.170.184.184"},"server":{"id":3504,"name":"TNP Ltd.","location":"Manchester","country":"Great Britain","host":"speedtest.tnp.net.uk","port":8080,"ip":"5.61.120.11"},"result":{"id":"2c00e1b4-fb2b-4424-a25fe2ca085a97b4","url":"https://www.speedtest.net/result/c/2c00e1b4-fb2b-4424-a25f-e2ca085a97b4"}}

You’ll be pleased to know that the rest of the process is done entirely within the Zabbix UI, but it’s probably worth doing a little recap…

So far we have:

• Installed the Speedtest CLI binary.
• Configured Zabbix to run the Speedtest CLI binary as a UserParameter.
• Configured the SpeedTest CLI so it runs in the context of the Zabbix agent.

We now need to configure the master Zabbix item and all the dependent items.

Step 4 In the Zabbix UI, create a new item (in our case for the Zabbix host) :

The important field here is the “Key”. This is set to the name of the UserParameter. Make a note of the item’s Name – you’ll need it for the dependent items.

Step 5 Now create the dependent items with some preprocessing:

The important field here is the Item Type. Make sure it is set to “Dependent Item”.

Also, make sure that the Master Item field is set to “Nobius Internet” – i.e. the Item you just created in step 4. Use the “Select” button to make it easier to find.

Step 6 Then select the Preprocessing tab :

The important field here is the JSONPath parameter. We can’t go into the nuances of JSON here but in simple terms, if you look carefully at the output of the Speedtest CLI, you’ll see a section that looks like this :
"ping":{"jitter":0.16400000000000001,"latency":18.266999999999999}

You can thing of “ping” as the section you want Zabbix to look at and the entry “latency” as the field within the section. Specifying “ping.latency” gives the value 18.2669999999999. In JSONPath terms, this is represented as $.ping.latency.
If you wanted the value for jitter, you’d use $.ping.jitter

To create the remaining items, repeat sections 5 and 6 creating new item names and specifying new item keys.

Top Tip : We use http://www.jsonquerytool.com/ to figure out JSONPaths. Just paste your JSON into the input and enter your query to see whether you have the right JSONPath. We find this is quicker than using Zabbix’s built-in tests. Having said that we ALWAYS double check with the Zabbix JSONPath “test” dialogue.

For completeness, here’s the set of JSONPath parameters we used with the Speedtest CLI – it’s not all the values you can get, but it is the interesting ones….

$.ping.latency	The ping response time contained in the JSON from Speedtest CLI
$.ping.jitter	The packet jitter rate measured by Speedtest CLI
$.download.bytes	How many bytes were downloaded during the test
$.upload.bytes	Bytes uploaded during the test
$.packetloss	Number of data packets lost as measured by Speedtest CLI

In summary, this is a crude and basic proof of concept on how to use the Speedtest CLI ® released by OOKLA with Zabbix. Nobius doesn’t warrant this guide or provide any guarantees as to suitability for a particular purpose.

There are a number of refinements that Nobius have made to this including graphs and maps that we use with our customers.

Feel free to use as you see fit and any comments are always appreciated. If you have questions, please contact us

The Nobius Team