Creating a Real-time HoneyPot Attack Map

Every device connected to the internet is open for cyber attacks. It takes less than one minute before a system is attacked once it is connected to the internet. Recently, I worked on a hackathon project to visualize honeypot attacks on a map in real-time.

A honeypot is a computer system that mimics a target for hackers. It tries to fool hackers into thinking it is a real computer system, distracting them from other targets.

Initial Setup

There are many honeypot systems around but for this project I used T-Pot created by Deutsche Telekom Security. It consists of many honeypot daemons and tools out-of-the-box and is easy to setup.

To create a live map, we need to have the following:

  • Running T-Pot instance
  • Small server with a webserver (nginx) and Node.js/NPM

Follow the instructions to install T-Pot. Confirm your T-Pot instance is running and you see attacks appearing in the dashboards.

There are many instructions on how to install a server with Nginx and Node.js (one example can be found here).

Node.js Application to Receive Logs

On the webserver, we create a small Node.js application that will do two simple tasks:

  • Receive data from the T-Pot installation (logstash)
  • Run a small websocket server to broadcast the received data to connected clients

Install required packages
In our Node.js application we use two packages: `express` and `ws`. First install both packages:

npm install ws express

Now we create a small application called `server.js`:

vi server.js

Insert the following code into the file:

#!/usr/bin/env nodejs
const http = require('http');
const WebSocket = require('ws');
const express = require('express');
const app = express();

const PORT = 8080;
const WS_PORT = 8081;

app.use(express.json());

// Create a WebSocket Server so clients can connect to it
const wss = new WebSocket.Server({ port: WS_PORT })
wss.on('connection', function connection(ws) {
});

// Now we create a simple HTTP server which receives a message
// and forwards the message to the connected WebSocket clients
app.get('/', (req, res) => {
  res.send('Hello');
});

app.post('/', (req, res) => {
  wss.clients.forEach(function each(client) {
    if (client.readyState === WebSocket.OPEN) {
      client.send(JSON.stringify(req.body));
    }
  });
  res.sendStatus(200);
});

app.listen(PORT, () => {
   console.log('The server is running at port 8080!');
});

This small Node.js application simply listens on port 8080 to receive a POST message. WebSockets can connect to port 8081. Once a message is received on port 8080, it is sent to all connected WebSocket clients.

Test Application

To test your application, make it executable:

chmod +x server.js

Now run the application:

./server.js

The output will be:

The server is running at port 8080!

You can use a process manager like PM2 to daemonize your application:

sudo npm install -g pm2
pm2 start server.js

PM2 will restart applications automatically if the application crashed or is killed. In order to have your application to run after a system (re)boot, you will need to execute another command:

pm2 startup systemd

This command output might include a command which needs to be run with superuser privileges:

sudo env PATH=$PATH:/usr/bin /usr/lib/node_modules/pm2/bin/pm2 startup systemd -u your_user — hp /home/your_user

Webpage Showing the Map

Now we will create a small webpage, and add some Javascript code. This code will open a websocket to receive updates and plot them on a map. To create a world map, I used Mapbox GL JS. You will need to create a (free) account in order to create an API key that will be used to create a map.

If you have a server running default Nginx, create a new `index.html` in the web-root folder:

cd /var/www/html
vi index.html

Insert the following HTML code into the file:

<!DOCTYPE HTML>
<html>
   <head> 
   <link href="https://cdn.jsdelivr.net/npm/bootstrap@5.0.0-beta1/dist/css/bootstrap.min.css" rel="stylesheet" integrity="sha384-giJF6kkoqNQ00vy+HMDP7azOuL0xtbfIcaT9wjKHr8RbDVddVHyTfAAsrekwKmP1" crossorigin="anonymous">
   <script src='https://api.mapbox.com/mapbox-gl-js/v2.0.1/mapbox-gl.js'></script>
    <link href='https://api.mapbox.com/mapbox-gl-js/v2.0.1/mapbox-gl.css' rel='stylesheet' />  
    <style>
        @import url(https://fonts.googleapis.com/css?family=Inconsolata:400,500);

        body { background-color: black }
        #map { height: calc(100vh - 275px);; width: ; z-index: 1; }
        .table { color: #fff; font-family: Inconsolata,sans-serif; font-size: 15px; border-color: #525252;}    
        .thead { font-weight: 700; color: #525252; }            
        
        @-webkit-keyframes flashrow {
           from { background-color: #525252; }
           to { background-color: var(--bs-table-bg); }
         }
         @-moz-keyframes flashrow {
           from { background-color: #525252; }
           to { background-color: var(--bs-table-bg); }
         }
         @-o-keyframes flashrow {
           from { background-color: #525252; }
           to { background-color: var(--bs-table-bg); }
         }
         @keyframes flashrow {
           from { background-color: #525252; }
           to { background-color: var(--bs-table-bg); }
         }
         .flashrow {
           -webkit-animation: flashrow 1.5s; /* Safari 4+ */
           -moz-animation:    flashrow 1.5s; /* Fx 5+ */
           -o-animation:      flashrow 1.5s; /* Opera 12+ */
           animation:         flashrow 1.5s; /* IE 10+ */
         }
    </style>   
   </head>   
   <body>
   <div class="container">   
    <div class="row">
      <div class="col">
         <div id="map"></div>
      </div>
   </div>
   <div class="row">
      <div class="col" id="ticker">
         <table id="tickettable" class="table table-black ticker">
            <thead class="text-uppercase thead">
               <tr>
                  <th class="col-lg-1 thead">Time</th>
                  <th class="col-lg-2 thead">Country</th>
                  <th class="col-lg-3 thead">AS Organisation</th>
                  <th class="col-lg-6 thead">TYPE</th>
               </tr>
            </thead>
            <tbody>            
            </tbody>
         </table>         
      </div>
   </div>
</div>
<script src='map.js'></script>
</html>

This page simple loads Mapbox GL Javascript libraries and some styles. It also loads another Javascript file (at the bottom) which will open the Websocket and update the map.
Let’s create this Javascript file:

vi map.js

and insert the following code into the file:

// Set the IP to your webserver IP
const WEBSOCKET_SERVER = 'ws://<YOUR_WEBSERVERIP:8081>';
// Set your mapboxGL AccessToken
const MAPBOX_TOKEN = 'YOUR_ACCESS_TOKEN';

// Remove points from map after x-seconds
var displayTime = 300;

// Set some defaults for the map

var framesPerSecond = 15; 
var initialOpacity = 1;
var opacity = initialOpacity;
var initialRadius = 3;
var radius = initialRadius;
var maxRadius = 15;
let points = new Map();
var timers = [];

//Set your accessToken here
mapboxgl.accessToken = MAPBOX_TOKEN;

//Create new mapboxGl Map. Set your used style
var map = new mapboxgl.Map({
    container: 'map',
    style: 'mapbox://styles/leaseweb/ckkiepmg40ds717ry6l0htwag',
    center: [0, 0],
    zoom: 1.75
});

// Create a popup, but don't add it to the map yet.
var popup = new mapboxgl.Popup({
   closeButton: false,
   closeOnClick: false
});

// Once the map is loaded, we open the Websockets
map.on('load', function () {   
   openWebSockets(map); 
});

function openWebSockets(map) { 
   if ("WebSocket" in window) {
      // Let us open a web socket
      var ws = new WebSocket( WEBSOCKET_SERVER); 

      ws.onopen = function() {
         // Web Socket is connected, send data using send()         
         console.log("WS Open...");
      };
   
      ws.onmessage = function (event) { 
         var received_msg = JSON.parse(event.data);             
         addPoint(received_msg);            
      };

      ws.onerror = function(error) {
         console.log('Websocket error: ');
         console.log(error);
      }

      ws.onclose = function() { 
         // websocket is closed.
         console.log("Connection is closed..."); 
      };

   } else {
      // The browser doesn't support WebSocket
      alert("WebSocket NOT supported by your Browser!");
   }    
}

function animateMarker(timestamp, pointId) {
   if(!(pointId === undefined)) {
      if (points.has(pointId)) {         
        timers[pointId] = setTimeout(function() {
            requestAnimationFrame(function(timestamp) {
              animateMarker(timestamp, pointId);
            });
            

            radius = points.get(pointId)[0];
            opacity = points.get(pointId)[1];
            
            radius += (maxRadius - radius) / framesPerSecond;            
            opacity -= ( .9 / framesPerSecond );
            if (opacity < 0) {
               opacity = 0;
            }

            map.setPaintProperty('point-'+pointId, 'circle-radius', radius);
            map.setPaintProperty('point-'+pointId, 'circle-opacity', opacity);
         
            if (opacity <= 0) {
                radius = initialRadius;
                opacity = initialOpacity;
            } 
            points.set(pointId,[radius, opacity ]);        
        }, 1000 / framesPerSecond);
     } else {
      //The point is removed, we don't do anything at this moment
     }
  }
}

function addPoint(msg) {
   geo = JSON.parse(msg.geoip);     
   var ip = geo.ip;
   //Create a geohash based on the lat/lon of the IP. We used factor 7 to prevent overlapping point animations
   var geohash = encodeGeoHash(geo.latitude, geo.longitude, 7);
   //Get the AS Organisation name (or unknown)
   var ASORG = (geo.as_org === undefined ? 'Unknown' : geo.as_org);

   //Remove the flashrow style from last added row
   var flashrows = document.getElementById("tickettable").getElementsByClassName('flashrow');
   while (flashrows[0]) {
      flashrows[0].classList.remove('flashrow');
   }
   
   //Get table to add the newly added point information
   var tbody = document.getElementById("tickettable").getElementsByTagName('tbody')[0];
   
   tbody.insertRow().innerHTML = '<tr><td class="flashrow">' + new Date().toLocaleTimeString() + '</td>' +
         '<td class="flashrow">' + geo.country_name + '</td>' +
         '<td class="flashrow">' + ASORG + '</td>' +
         '<td class="flashrow">' + msg.protocol.toUpperCase() + ' Attack on port ' + msg.dest_port +'</td>' +    
         '</tr>';
   
   //If we have more than 5 items in the list, remove the first one
   if (tbody.rows.length > 5) {
      tbody.deleteRow(0);
   }

   //Add the point to the map if it is not already on the map
   if (!(geohash === undefined)) {               
      if (!(points.has(geohash))) {
         
         //Add the point to hash to keep of all active points and prevent duplicate points.
         points.set(geohash, [initialRadius, initialOpacity ]);               

         //Set a timer to remove the poinrt after 5minutes
         setTimeout(function() { removePoint(geohash) }, displayTime * 1000);            

         map.addSource('points-'+geohash, {
           "type": "geojson",
           "data": {
               "type": "Feature",
               "geometry": {
                  "type": "Point",
                  "coordinates": [ geo.longitude, geo.latitude]
               },
               "properties": {
                  "description": "<strong>" + ASORG + " (AS " + geo.asn +")</strong><p>IP: " + ip + "<BR>City: " + (geo.city_name === undefined ? 'Unknown' : geo.city_name) + 
                     "<BR>Region: " + (geo.region_name === undefined ? 'Unknown' : geo.region_name) + "<BR>Country: " + (geo.country_name === undefined ? 'Unknown' : geo.country_name) + "</P>"
               }
           }
         });
         
         map.addLayer({
           "id": "point-"+geohash,
           "source": "points-"+geohash,
           "type": "circle",
           "paint": {
               "circle-radius": initialRadius,
               "circle-radius-transition": {duration: 0},
               "circle-opacity-transition": {duration: 0},
               "circle-color": "#dd7cbf"
           }
         });  

         map.on('mouseenter', 'point-'+geohash, function (e) {
            // Change the cursor style as a UI indicator.
            map.getCanvas().style.cursor = 'pointer';
             
            var coordinates = e.features[0].geometry.coordinates.slice();
            var description = e.features[0].properties.description;
             
            // Ensure that if the map is zoomed out such that multiple
            // copies of the feature are visible, the popup appears
            // over the copy being pointed to.
            while (Math.abs(e.lngLat.lng - coordinates[0]) > 180) {
               coordinates[0] += e.lngLat.lng > coordinates[0] ? 360 : -360;
            }
             
            // Populate the popup and set its coordinates
            // based on the feature found.
            popup.setLngLat(coordinates).setHTML(description).addTo(map);
         });
             
         map.on('mouseleave', 'point-'+geohash, function () {
            map.getCanvas().style.cursor = '';
            popup.remove();
         });

         //Animate the added point.
         animateMarker(0, geohash);               
      }
   }
}

function removePoint(ip) {
   clearTimeout(timers[ip]);
   points.delete(ip);
   map.removeLayer('point-'+ip);
   map.removeSource('points-'+ip);         
}

function encodeGeoHash(latitude, longitude, precision) {
  var BITS = [16, 8, 4, 2, 1];

  var BASE32 = "0123456789bcdefghjkmnpqrstuvwxyz";
  var isEven = 1;
  var lat = [-90.0, 90.0];
  var lon = [-180.0, 180.0];
  var bit = 0;
  var ch = 0;
  precision = precision || 12;

  var geohash = "";
  while (geohash.length < precision) {
    var mid;
    if (isEven) {
      mid = (lon[0] + lon[1]) / 2;
      if (longitude > mid) {
        ch |= BITS[bit];
        lon[0] = mid;
      } else {
        lon[1] = mid;
      }
    } else {
      mid = (lat[0] + lat[1]) / 2;
      if (latitude > mid) {
        ch |= BITS[bit];
        lat[0] = mid;
      } else {
        lat[1] = mid;
      }
    }

    isEven = !isEven;
    if (bit < 4) {
      bit++;
    } else {
      geohash += BASE32[ch];
      bit = 0;
      ch = 0;
    }
  }
  return geohash;
}; 

You will need to make two small modification at the top of the file:

  • Set the IP of your webserver
  • Set your Mapbox access token

Once you have done this, open the page in your browser and a map should appear. NOTE: nothing will happen at this moment 🙂

Configure Logstash

Now we are all set for the last part: configuring Logstash to also forward (some) logs to our Node-application.
On your T-Pot server, we need to get the Logstash configuration as described on the T-Pot Wiki:

docker exec -it logstash ash
cd /etc/logstash/conf.d/
cp logstash.conf /data/elk/logstash.conf
exit

Open the Logstash configuration and add the following lines to the output section, after the Elasticsearch output:

if [type] == "ConPot" and [dest_port] and [event_type] == "NEW_CONNECTION" and [src_ip] != "${MY_INTIP}" {
   http {
     url => "http://${HTTP_LOGIP}"
     http_method => "post"
     mapping => {
         "type" => "%{type}"
         "protocol" => "Elastic"
         "source" => "%{src_ip}"
         "dest_port" => "%{dest_port}"
         "geoip" => "%{geoip}"
     }
   }
}
if [type] == "Ciscoasa" and [src_ip] != "${MY_INTIP}" {
   http {
     url => "http://${HTTP_LOGIP}"
     http_method => "post"
     mapping => {
          "type" => "%{type}"
         "protocol" => "Ciscoasa"
         "source" => "%{src_ip}"
         "geoip" => "%{geoip}"
     }
   }
}
if [type] == "Mailoney" and [dest_port] and [src_ip] != "${MY_INTIP}" {
   http {
     url => "http://${HTTP_LOGIP}"
     http_method => "post"
     mapping => {
          "type" => "%{type}"
         "protocol" => "Mail"
         "source" => "%{src_ip}"
         "dest_port" => "%{dest_port}"
         "geoip" => "%{geoip}"
     }
   }
}
if [type] == "ElasticPot" and [dest_port] and [src_ip] != "${MY_INTIP}" {
   http {
     url => "http://${HTTP_LOGIP}"
     http_method => "post"
     mapping => {
          "type" => "%{type}"
         "protocol" => "Elastic"
         "source" => "%{src_ip}"
         "dest_port" => "%{dest_port}"
         "geoip" => "%{geoip}"
     }
   }
}
if [type] == "Adbhoney" and [dest_port] and [src_ip] != "${MY_INTIP}" {
   http {
     url => "http://${HTTP_LOGIP}"
     http_method => "post"
     mapping => {
          "type" => "%{type}"
         "protocol" => "ADB"
         "source" => "%{src_ip}"
         "dest_port" => "%{dest_port}"
         "geoip" => "%{geoip}"
     }
   }
}
if [type] == "Dionaea" and [dest_port] and [src_ip] != "${MY_INTIP}" {
   http {
     url => "http://${HTTP_LOGIP}"
     http_method => "post"
     mapping => {
          "type" => "%{type}"
         "protocol" => "%{[connection][transport]}"
         "service" => "%{[connection][protocol]}"
         "source" => "%{src_ip}"
         "dest_port" => "%{dest_port}"
         "geoip" => "%{geoip}"
     }
   }
}
if [type] == "Fatt" and [protocol] != "ssh" and [src_ip] != "${MY_INTIP}" {
   http {
     url => "http://${HTTP_LOGIP}"
     http_method => "post"
     mapping => {
          "type" => "%{type}"
         "protocol" => "%{protocol}"
         "source" => "%{src_ip}"
         "dest_port" => "%{dest_port}"
         "geoip" => "%{geoip}"
     }
   }
}
if [type] == "Cowrie" and [dest_port] and [protocol] and [src_ip] != "${MY_INTIP}" {
   http {
     url => "http://${HTTP_LOGIP}"
     http_method => "post"
     mapping => {
          "type" => "%{type}"
         "protocol" => "%{protocol}"
         "source" => "%{src_ip}"
         "dest_port" => "%{dest_port}"
         "geoip" => "%{geoip}"
     }
   }
}
if [type] == "HoneyTrap" and [dest_port] and [src_ip] != "${MY_INTIP}" {
   http {
     url => "http://${HTTP_LOGIP}"
     http_method => "post"
     mapping => {
          "type" => "%{type}"
         "protocol" => "%{[attack_connection][protocol]}"
         "source" => "%{src_ip}"
         "dest_port" => "%{dest_port}"
         "geoip" => "%{geoip}"
     }
   }
}

We need to add a new variable to the docker environment:

vi /opt/tpot/etc/compose/elk_environment

Then add the following line to the file, to exclude the T-Pot server in the Logstash messages:

HTTP_LOGIP=<YOUR_TPOTSERVER_IP>:8080

Now add a new docker volume for the Logstash service:

vi /opt/tpot/etc/tpot.yml

Go to the Logstash service section and add the following line:

- /data/elk/logstash.conf:/etc/logstash/conf.d/logstash.conf

Now we are all set and it’s time to restart your T-port service:

systemctl start tpot

That’s It!

Now take a look at your map. If there are attacks on your server, they should appear on the map and in the listing below.

You can trigger an event by, for example, opening a regular SSH session to your T-Pot server:

ssh <TPOT_SERVER>

Simply close the connection once it is established, and your location should appear on the map.

Daily figures

Almost immediately when you start running a honeypot you will see attacks. Within one day, I saw over 200.000 attacks, mostly on common ports like HTTP(S), SSH and SMTP. You can use this data to make your environments more safe, or just use them for some fun projects.

Some notes
As this was a quick project in limited time, there are definitely some optimisations or better coding that can take place 🙂 The Javascript will give a few errors after some time, probably due to points being removed from the map while a call to update the same point happens at the same time.
In addition, some points on the map will suddenly run on steroids, animating at higher frames than they did initially. The Node.js application was made quick and dirty but is suitable for this demo.

Technical Careers at Leaseweb

We are searching for the next generation of engineers and developers to help us build the infrastructure to automate our global hosting services! If you are interested in finding out more, check out our Careers at Leaseweb.

Share

Using Correlation IDs in API Calls

Over the years, the IT industry has moved from a single domain, monolithic architecture to a microservice architecture. In a microservice architecture, complex processes are split into smaller and simpler sub-processes. While this kind of architecture has many benefits, there are also some downsides – for example, if you send one request to a Leaseweb API, it ends up in multiple requests in other backend systems [FIGURE 1]. How do you keep track of requests and responses processed by multiple systems? This is where Correlation IDs come into play.

[FIGURE 1: Example request/response flow]

Using a Correlation ID

A Correlation ID is a unique, randomly generated identifier value that is added to every request and response. In a microservice architecture, the initial Correlation ID is passed to your sub-processes. If a sub-system also makes sub-requests, it will also pass the Correlation ID to those systems.

How you pass the Correlation ID to other systems depends on your architecture. At Leaseweb we are using REST APIs a lot, with HTTP headers to pass on the Correlation ID. As a rule, we assign a Correlation ID as soon as possible, and always use a Correlation ID if it is passed on. Our public API only accepts Correlation IDs from internally trusted clients. For any other client (such as an employee or customer API clients) a new Correlation ID is generated for the request.

Real Value of Correlation IDs

The real value of Correlation IDs is realized when you also log the Correlation IDs. Debugging or tracing requests becomes much easier, as you can search all of your logs for the same Correlation ID. Combined with central logging solutions (such as the ELK stack), searching logs becomes even easier and can be done by non-technical colleagues. Providing tools to your colleagues to troubleshoot issues allows them to have more responsibility and gives you more time to work on more technical projects.

We mainly use Correlation IDs at Leaseweb for debugging purposes. When an error occurs, we provide the Correlation ID to the client/customer. If users provide the Correlation ID when submitting a support ticket, we can visualize the entire process needed to fulfil the client’s initial intent. This has significantly improved the time it takes us to fix bugs.

[FIGURE 2: Example of one Correlation ID with multiple requests]

Debugging issues is a time-consuming process if Correlation IDs are not used. When your environment scales, you will need to find solutions to group transactions happening in your systems. By using a Correlation ID, you can easily group requests and events in your systems, allowing you to spend more time fixing the problem and less time trying to find it.

Practical examples on how to implement Correlation IDs

The following examples use Symfony, a popular web application framework. These concepts can also be applied to any other framework, such as Laravel, Django, Flask or Ruby on Rails.

If you are unfamiliar with the concept of Service Containers and Dependency Injection, we recommend reading the excellent Symfony documentation about it here: https://symfony.com/doc/current/service_container.html

Using Monolog to append Correlation IDs to your application logs

When processing a HTTP request your application often logs some information – such as when an error occurred, or an important change made in your system that you want to keep track of. When using the Monolog logging library in PHP (https://seldaek.github.io/monolog/), you can use the concept of “Processors” (read more about that here on symfony.com).

One way to do this is by creating a Monolog Processor class:

<?php

namespace App\Monolog\Processor;

use Symfony\Component\HttpFoundation\RequestStack;

class CorrelationIdProcessor
{
    protected $requestStack;

    public function __construct(RequestStack $requestStack)
    { 

       $this->requestStack = $requestStack;

    }
 
    public function processRecord(array $record)
    {
        $request = $this->requestStack->getCurrentRequest();

        if (!$request) {
            return;
        }

        $correlationId = $request->headers->get(‘X-My-Correlation-ID');

        if (empty($correlationId)) {
             return;
        }

        // If we have a correlation id include it in every monolog line
        $record['extra']['correlation_id'] = $correlationId;
 
        return $record;
    }
}

Then register this class on the service container as a monolog processor in services.yml:

# app/config/services.yml

services:
  App\Monolog\Processor\CorrelationIdProcessor:
    arguments: ["@request_stack"]
    tags:
      - name: monolog.processor
        method: processRecord

Now, every time you log something in your application with Monolog:

$this->logger->info('shopping_cart_emptied', [‘cart_id’ => 123]);

You will see the Correlation ID of the HTTP Request in your log files:

$ grep ‘shopping_cart_emptied’ var/logs/prod.log

[2020-07-03 12:14:45] app.INFO: shopping_cart_emptied {“cart_id”: 123} {"correlation_id":"d135d5f1-3dd0-45fa-8f26-55d8d6a44876"}

You can utilize the same pattern to log the name of the user that is currently logged in, the remote IP address of the API client, or anything else that makes troubleshooting faster for you.

Using Guzzle to append Correlation IDs when making sub-requests

If your API makes API calls to other microservices (and you use Guzzle to do this) you can make use of Handlers and Middleware.

Some teams at Leaseweb depend on many downstream microservices, and can therefore have multiple guzzle clients as services on the service container. While each Guzzle client is configured with its own base URL and/or authentication, it is possible for all of the Guzzle clients to share the same HandlerStack.

First, create the middleware:

<?php

namespace App\Guzzle\Middleware;

use Symfony\Component\HttpFoundation\RequestStack;
use Psr\Http\Message\RequestInterface;

class CorrelationIdMiddleware
{
    protected $requestStack;
 
    public function __construct(RequestStack $requestStack)
    {
        $this->requestStack = $requestStack;
    }

    public function __invoke(callable $handler)
    {
        return function (RequestInterface $request, array $options = []) use ($handler) {
            $request = $this->requestStack->getCurrentRequest();

            if (!$request) {
                return $handler($request, $options);
            }

            $correlationId = $request->headers->get(‘X-My-Correlation-ID');

            if (empty($correlationId)) {
                 return $handler($request, $options);
            } 
 
            $request = $request->withHeader(‘X-My-Correlation-ID’, $correlationId);
 
            return $handler($request, $options);
        };
    }
}

Define this middleware as service on the service container and create a HandlerStack:

# app/config/services.yml

services:
  correlation_id_middleware:
    class: App\Guzzle\Middleware:
    arguments: ["@request_stack"]

  correlation_id_handler_stack:
    class: GuzzleHttp\HandlerStack
    factory: ['GuzzleHttp\HandlerStack', 'create']
    calls:
      - [push, ["@correlation_id_middleware", "correlation_id_forwarder"]]

With these two services defined, you can now configure all your Guzzle clients using the HandlerStack so that the Correlation ID of the current HTTP request is forwarded to downstream HTTP requests:

# app/config/services.yml

services:
  my_downstream_api:
    class:
    arguments:
      - base_uri: https://my-downstream-api.example.com
        handler: "@correlation_id_handler_stack”

Now every API call that you make to https://my-downstream-api.example.com will include the HTTP request header ‘X-My-Correlation-ID’ and have the same value as the Correlation ID of the current HTTP request. You can also apply the same Monolog and Guzzle tricks described here to the downstream API.

Expose Correlation IDs in error responses

The missing link between these processes is to now expose your Correlation IDs to your users so they can also log them or use them in support cases they report to your organization.

Symfony makes this easy using Event Listeners. You can define Event Listeners in Symfony to pre-process HTTP requests as well as to post-process HTTP Responses just before they are returned by Symfony to the API caller. In this example, we will create a HTTP Response listener and add the Correlation ID of the current HTTP request as a HTTP Header in the HTTP Response.

First, we create a service on the Service Container:

<?php
 
namespace App\Listener;
 
use Symfony\Component\HttpFoundation\RequestStack;
use Symfony\Component\HttpKernel\Event\FilterResponseEvent;

class CorrelationIdResponseListener
{
    protected $requestStack;
 
    public function __construct(RequestStack $requestStack)
    {
        $this->requestStack = $requestStack;
    }

    public function onKernelResponse(FilterResponseEvent $event)
    {
        $request = $this->requestStack->getCurrentRequest();

        if (!$request) {
            return;
        }

        $correlationId = $request->headers->get(‘X-My-Correlation-ID');

        if (empty($correlationId)) {
             return;
        }

        $event->getResponse()->headers->set(‘X-My-Correlation-ID’, $correlationId);
    }
}

Now configure it as a Symfony Event Listener:

# app/config/services.yml

services:
  correlation_id_response_listener:
    class: App\Listener\CorrelationIdResponseListener
    arguments: ["@request_stack"]
    tags:
      - { name: kernel.event_listener, event: kernel.response, method: onKernelResponse }

Every response that is generated by your Symfony application will now include a X-My-Correlation-ID HTTP response header with the same Correlation ID as the HTTP request.

The Value of Correlation IDs

Using Correlation IDs throughout your whole stack gives you more insight into all (sub)requests during a transaction. Using the right tools allows others to debug issues, giving your developers more time to work on new awesome features.

Implementing Correlation IDs isn’t hard to do, and can be achieved quickly depending on your software stack. At Leaseweb, the use of Correlation IDs has saved us hours of time while debugging issues on numerous occasions.

Technical Careers at Leaseweb

We are searching for the next generation of engineers and developers to help us build infrastructure to automate our global hosting services! If you are interested in finding out more, check out our Careers at Leaseweb.

Share