Measuring IP Connectivity with RIPE Atlas

One of the most important but often overlooked criteria when choosing a hosting provider is the performance of its network. Poor network performance for any kind of online service leads to low customer satisfaction. What is the best way to measure network performance that most accurately simulates a customers experience with a network? We found this question to be an interesting idea to explore during a Leaseweb hackathon!

RIPE Atlas

RIPE Atlas is the RIPE NCC’s main Internet data collection system. Its a global network of devices, called probes and anchors, that actively measure Internet connectivity. Anyone can access this data via Internet traffic maps, streaming data visualizations, and an API. RIPE Atlas users can also perform customized measurements to gain valuable data about their own networks.

Due to the size and reach of the Atlas project, it’s one of the most important internet measurement initiatives internationally. What’s great is that the project is run by RIPE, but driven by the global community of RIPE members (and non-members!) who contribute some of their resources to help the project. This improves visibility into the inner-workings of the global Internet. It’s used by Internet professionals all over the world to improve quality of their network, debug issues, and learn. Leaseweb contributes to the RIPE Atlas by hosting 7 ‘anchors’ in various data centers all over the globe.

Since Leaseweb already contributes to RIPE Atlas with these anchors, it’s an obvious choice as a source of random probes to be used against those anchors and compared with other infrastructure providers’ anchors. (By the way, if you would like to do your own measurements and contribute to the RIPE Atlas project at the same time, you can request a probe for your home or office right here!). You can read more on the structure of the Atlas network (and how probes, anchors and the RIPE backend work together) in various posts on the RIPE labs pages.

The main elements needed to use RIPE Atlas are measurements, sources and targets. These elements all have their own function to facilitate experiments.

Getting The Data

Triggering one-off measurements and fetching the results of those measurements can be done using the RIPE Atlas API. RIPE-NCC also developed a wrapper around the RIPE Atlas API to allow anyone to communicate with the RIPE Atlas API using Python. It is maintained by RIPE Atlas developers and is therefore the best choice for consuming their API. Using it requires an API key. Wrapper is open-source and available on GitHub.

Installation is very simple using pip:

$ pip install ripe.atlas.cousteau

Included classes in the Python script:

from ripe.atlas.cousteau import (
Ping,
Traceroute,
AtlasSource,
AtlasCreateRequest,
AtlasLatestRequest
)

Measurement Types

RIPE Atlas offers several measurement types – Ping, Traceroute, DNS, HTTP, SSL, NTP, and WiFi. Creating a measurement allows you to specify a type of test (or ‘experiment’) to perform. Typically, these have something to do with latency for a service, but there are also options to check things like resolving a domain name and checking DNS propagation. These measurements can be one-off or recurring. For our Hackathon project we used one-off measurements.

Sources

Probes defined as sources are used to trigger the measurement. They can be defined explicitly or taken from a pool (area, country, prefix or AS number). Below, the defined source takes 50 random probes from Europe to be used as a source of measurement.

source = AtlasSource(
    type="area",
    value= "North-Central",
    requested=50
)

Targets

Targeted IPs that the measurements are run against. In this example, ping is run against one of Leaseweb’s anchors.

pingLSW = Ping(
    af=4,
    target="5.79.112.97", 
    description="Ping nl-haa-as60781.anchors.atlas.ripe.net"
)

Create Request

Defining a request combines all of the elements together: measurements and sources. It also requires a start time, API key and – in this case – a one-off flag.

atlas_request = AtlasCreateRequest(
    start_time=datetime.utcnow(),
    key=ATLAS_API_KEY,
    measurements=[pingLSW, pingOVH, pingAzure, pingAWS, pingUni],
    sources=,
    is_oneoff=True
    )

To summarize what this request will do:

  • We specify a number of ping tests to various endpoints
  • We specify where we want to have those request come from and how many we want
  • …and we bundle those tests into a single one-off request.

Calling RIPE Atlas API now is simple

response = atlas_request.create()

After calling this function, Atlas will launch tests towards 50 random probes in the area we designated, and will store the results.

Returned values, stored in response, are measurement counts. In this case, there are 5 values as there are 5 measurements defined. The actual results have to be retrieved separately, so the next step is fetching the measurement data. Here, class AtlasLatestRequest is used:

results = AtlasLatestRequest(msm_id=measurement_id).create()

The results variable now has stored all of the measurement details needed to calculate and compare latencies. It’s clear to see how powerful Atlas is. In a few simple lines we’ve generated latency information from 50 end points to multiple targets!

Visualizing Data

Visualizing data that was fetched from RIPE Atlas API was done with pygal Python library that supports various chart types and styles. For this Hackathon project pygal.Bar() was used to draw out the comparison results. (Pygal usage is out of the scope of this blog post). Two charts below show the data from measurements taken from Europe and from Russia.

Conclusion and Going Forward

This Hackathon project showed the basic features of RIPE Atlas and what it can accomplish.  RIPE Atlas also maintains a parsing library name Sagan, available in GitHub, that handles format changes in measurement results and always returns a native Python object.

RIPE Atlas has a huge amount of functionality and can be easily used in your own experiments and measurements. Remember to always use the Atlas infrastructure responsibly.

Other tools for using permanent measurementsstreaming data, and building dashboards are RIPE Prometheus exporter, working as a metric exporter of RIPE Atlas measurement results that exports collected metrics to Prometheus. Grafana is common tool that works well with Prometheus to create dashboards with useful metrics, gathered from RIPE Atlas measurements.  

Comments? Questions? Other ways to use RIPE Atlas and receive measurements? I’d love to hear your feedback! 

Share

Google PageRank still matters for SEO

visitors_sept

On Monday, September 21st we moved this blog to another domain. This has made the traffic on the blog go up from 2000 to 2800 unique visitors per day. You can see the effect in the graph above (source: count-per-day plugin).

Google PageRank

lsw_domains

At LeaseWeb we run several websites with different Google PageRank scores as you can see in the picture above (source: www.seomastering.com). In order to let the sites benefit from the highest valued domain we could move the sites from subdomains to paths. We did that for this site (LeaseWeb labs).

www.leaseweblabs.com => www.leaseweb.com/labs

This higher Google PageRank led to an expected increase of Google search traffic. It went from 1250 to 2000 (+60%) clicks per day. It is incredible what better Google PageRank can do for the traffic of a site.

Moving domains? 301 everything!

Moving content from one domain to another (or from http to https) is not without risk. If you do not proper 301 redirect your content, you may lose traffic/rank due to penalties for having 302 redirects or no redirects at all.

I suggest that you create a second property on the Google’s Search Console so that you can see the search traffic move from one site to the other. You can also tell Google in the search console that the site has moved.

google_clicks

On the above graph (from Google’s Search Console) I have drawn a red line to show the 6 day transition from the old to the new domain.

Share

Docky makes Xubuntu look pretty

Xubuntu is my favorite Linux distribution. It is fast, pretty and minimal. Right now I am running version 15.04 and 14.04 (LTS). Previous versions used to have a sort of a dock on the bottom, but current versions do not have that. In the current version the top panel holds the “Application Menu”, “Window Buttons” and the “Notification Area”. When you move that panel down it will be familiar to Windows XP users. In order to make the transition for Mac OSX users easier you may some more adjustments. Apple desktop users need features like: a dock, expose and a quick launcher.

1) Docky is a good looking icon dock

xfce_docky

Docky is a dock that looks great and indicates open programs, so it can be a replacement for your “Window buttons” in the taskbar. Removing these from the panel is easy (right-click). I recommend installing docky with the following command:

sudo apt-get install docky

There are two tricks you may want to execute after installation, read about them below.

Docky: Remove the anchor icon

Docky has by default an anchor icon in the dock. It allows you to configure the dock. This can also be done by right-clicking a border or a separator, so you don’t really need the anchor icon. You can simple remove the anchor icon using the following command (source):

gconftool-2 --type Boolean --set /apps/docky-2/Docky/Items/DockyItem/ShowDockyItem False

You may replace “False” at the end of the line with “True” to revert the change.

Docky: Wrong Thunar and Terminal icons

The “Thunar” and “Terminal” icons may look bad/weird and the applications may have double entries in the dock. This problem is less serious than it looks. It is caused by Docky not being able find the corresponding application shortcuts (that contain the icon path). This can easily be fixed with the following commands (source):

sudo cp /usr/share/applications/xfce4-terminal.desktop ~/.local/share/applications
sudo cp /usr/share/applications/Thunar.desktop ~/.local/share/applications

NB: You may encounter other applications with this problem, but this easily fixed in a similar fashion.

2) Skippy-XD adds expose functionality

The “expose” functionality is in which a click-able miniature version of all windows are shown in a non-overlapping layout to enable quick and pretty switching of applications. For Ubuntu there is “Skippy-XD” an application that does exactly that. It has a daemon mode and a run once mode. The daemon mode is very usable. Here is how to install skippy-xd (source):

sudo add-apt-repository ppa:landronimirc/skippy-xd-daily
sudo apt-get update
sudo apt-get install skippy-xd

To make start during login navigate in the main menu to “Settings” > “Session and Startup” > “Application Autostart” and add the following command:

skippy-xd --start-daemon

To make F3 act as the expose hot-key navigate in the main menu to “Settings” > “Keyboard” > “Application Shortcuts” and add the following command:

skippy-xd --activate-window-picker

Now logout and login to see whether the application indeed works as intended.

3) Launchy is a great quick launcher

You know how convenient you can search on OSX with a Ctrl-Space? This can easily and beautifully be configured using the “Launchy” application. This can easily be installed using the following command:

sudo apt-get install launchy launchy-plugins

After installation you may have to edit the settings to make launchy properly index you “Documents” and “Downloads” folders. Also make sure you check out the plugin configurations.

Conclusion

After installing and configuring these applications your Xubuntu is feeling a bit more like OSX. For people that are switching operating systems this may be a good thing (they could also try Elementary OS). I personally dislike having a dock and prefer the gnome 2 layout with 2 panels: one on top with the main menu and one on the bottom with the window buttons. This is exactly why power users like Linux: you can customize it to fit your needs.

Share

When (not why) “Agile” and especially Scrum are terrible

This article is a response to “Why ‘Agile’ and especially Scrum are terrible” as posted on Michael O. Church’s blog. He is not so much attacking the agile manifesto, as he he is attacking the reality of “agile”, the way it portraits itself and the excesses.

If you read the comments on Reddit then you find that there are two camps: scrum advocates and scrum haters. The haters complain how scrum leads to bad behavior and the advocates argue that this behavior is caused by doing scrum wrong. In this post I will focus on this behavior that, both parties agree, should be avoided.

1) Being judged personally

You should not be judged on hour by hour productivity as a software developer, because constant daily competition is not healthy. What particular tasks, technical debt or research tasks each developer worked on in a sprint is irrelevant. Peer pressure is the most effective way to keep people productive and looking at individual performance kills peer pressure. If there is micro-management going on, then that has nothing to do with being “Agile”.

Although there may be a high level of transparency, that does not mean that management should be evaluating people’s performance at the end of each sprint. The story points are designed to prevent management from doing this. The retrospective allows for criticizing the performance and effectiveness of people in the team by the team.

Especially web-based “Agile” registration tools must be used with care (or avoided). They are not part of “scrum” as white-boards and stickies are preferred. Also they may add lots of bureaucracy and may be misused to draw conclusions. Conclusion that may destroy the valuable trust relationship between employer and employee as they are more likely to be wrong than right.

If you see “agile” and “scrum” as tools to achieve “accountability” or “predictability” you are missing the point. It is merely a method to talk about work done instead of work to be done.

2) Developers being considered the lowest rank

The “product owners” and “scrum masters” do NOT outrank “team members”. Although they prioritize the user stories, they should not not choose which ones get executed. Also they should not assign tickets. If they feel superior or they are treated (or paid) as such, then something is wrong and not in the last place because good team members are all-round technical experts and these are extremely hard to find.

Team members are stakeholders as well. They can not only choose which tickets to work on (from the top of the stack), but also add stories (often technical debt) and argue why they are important.

One of the goals of agile is to eliminate blame and fear in the workplace. In a good team people do not fear to say “I was stuck with my code all day and got nothing done” during a stand-up. People should understand their pain and instead of blaming them offer some help.

3) Short term focus

Under Agile, technical debt can pile up not being addressed, because non-technical business people decide on priorities. They will not see a problem until it’s far too late or, at least, too expensive to fix it.

This pileup may be the result of  a straightforward application of “scrum/agile” principles. The focus may easily shift to only things that bring business value or “visible features” as I like to call them. These principles might work when imposed/implemented by an engineering team onto itself, but they break apart instantly the moment the management tries to impose them from outside.

There’s no reason why major refactoring can’t be tackled as a story. And having metrics to measure this (e.g. steadily increasing time to deliver features) should make it easier, not harder, to make the business case.

Conclusion

Short term work under high scrutiny is not fun nor agile, avoid it.

Share

Heka monolog decoder

This post is about how to use heka to give your symfony 2 application logs the care they deserve.

Application logs are very important for the quality of the product or service you are offering.

They help you find out what went wrong so you can explain and fix a bug that was reported recently. Or maybe to gather statistics to see how often a certain feature is used. For example how many bare metal reinstallation requests were issued last month and how many of those failed. Valuable information that you could use to decide what feature you are going to work on next.

At LeaseWeb we use quite a lot of php and Seldaek’s monolog is our logging library of choice. Recently we open sourced a heka decoder on github here. For you who do not know heka yet, check out their documentation.

Heka is an open source stream processing software system developed by Mozilla. Heka is a “Swiss Army Knife” type tool for data processing, useful for a wide variety of different tasks, such as …

Heka runs as a system daemon just like logstash or fluentd. Heka is written in go and comes with an easy to use plugin system based on lua. It has almost no dependencies and is lightweight. James Turnbull has written a nice article on how to get started with heka.

send symfony 2 application logs to elastic search

How better to explain with an example.

Lets say you have a symfony 2 application and you want the application logs to be sent to an Elastic Search platform.

On a debian based os you can download one of the heka debian packages from github.

    $ wget https://github.com/mozilla-services/heka/releases/download/v0.9.2/heka_0.9.2_i386.deb
    $ sudo dpkg -i heka_0.9.2_i386.deb

To configure heka you are required to edit the configuration file located at /etc/hekad.toml.

    $ vim /etc/hekad.toml

Take your time to read the excellent documentation on heka. There are many ways of using heka but we will use it as a forwarder:

  1. Define an input, where messages come from.
  2. Tell heka how it should decode the monolog log line into a heka message
  3. Tell heka how to encode the message so Elastic Search will understand it
  4. Define an output, where should the messages be sent to.

First we define the input:

    [Symfony2MonologFileInput]
    type = "LogstreamerInput"
    log_directory = "/var/www/app/logs"
    file_match = 'prod\.log'
    decoder = "Symfony2MonologDecoder"

Adjust `log_directory` and `file_match` according to your setup. As you can see we alread told heka to use the `Symfony2MonologDecoder` to we will define that one next:

    [Symfony2MonologDecoder]
    type = "SandboxDecoder"
    filename = "/etc/symfony2_decoder.lua"

Change the `filename` with the path where you placed the lua script on your system.

Now we have defined the input we can tell heka where to output messages to:

    [ESJsonEncoder]
    index = "%{Hostname}"
    es_index_from_timestamp = true
    type_name = "%{Type}"

    [ElasticSearchOutput]
    message_matcher = "TRUE"
    server = "http://192.168.100.1:9200"
    flush_interval = 5000
    flush_count = 10
    encoder = "ESJsonEncoder"

In the above example we assume that your Elastic Search server is running at 192.168.100.1.

And thats it.

A simple log line in app/logs/prod.log:

    [2015-06-03 22:08:02] app.INFO: Dit is een test {"bareMetalId":123,"os":"centos"} {"token":"556f5ea25f6af"}

Is now sent to Elastic Search. You should now be able to query your Elastic Search for log messages, assuming the hostname of your server running symfony 2 is myapi:

    $ curl http://192.168.100.1:9200/myapi/_search | python -mjson.tool
    {
        "_shards": {
            "failed": 0,
            "successful": 5,
            "total": 5
        },
        "hits": {
            "hits": [
                {
                    "_id": "ZIV7ryZrQRmXDiB6thY_yQ",
                    "_index": "myapi",
                    "_score": 1.0,
                    "_source": {
                        "EnvVersion": "",
                        "Hostname": "myapi",
                        "Logger": "Symfony2MonologFileInput",
                        "Payload": "Dit is een test",
                        "Pid": 0,
                        "Severity": 7,
                        "Timestamp": "2015-06-03T20:08:02.000Z",
                        "Type": "logfile",
                        "Uuid": "344e7cae-6ab7-4fb2-a770-d2cbad6653c3",
                        "channel": "app",
                        "levelname": "INFO",
                        "bareMetalId": 123,
                        "os": "centos",
                        "token": "556f5ea25f6af"
                    },
                    "_type": "logfile"
                },
        // ...
        }
    }

What is important to notice is that the keys token, bareMetalId and os in the monolog log line end up in Elastic Search as an indexable fields. From your php code you can add this extra information to your monolog messages by supplying an associative array as a second argument to the default monolog log functions:

    <?php

    $logger = $this->logger;
    $logger->info('The server was reinstalled', array('bareMetalId' => 123, 'os' => 'centos'));

Happy logging!

Share