GoAccess is a top for Nginx or Apache

goaccess-dashboard

GoAccess is an open source real-time web log analyser and interactive viewer that runs in a terminal on Linux systems. The name suggest that is written in the Go language, but it actually is not (it is written in C).

Monitoring

Effectively it is like the “top” command, but instead of showing processes it is giving you insight in the traffic on your webserver. The tool provides all kinds of top lists useful for spotting irregularities. Great features include the listing of IP addresses by percentage of hits and showing the status codes of the served responses. Typically you would use this tool if your webserver is reporting high load or high bandwidth usage and you want to find out what is going on. The outcomes of the analysis with this tool can then be used to adjust your firewall or caching rules.

Installing

On my Ubuntu 15.04 I can install version 0.8.3 with:

sudo apt-get install goaccess

If you are on any Debian based Linux and you want to run the latest (0.9.2) version you can simply run:

wget http://tar.goaccess.io/goaccess-0.9.2.tar.gz
tar -xvzf goaccess-0.9.2.tar.gz
cd goaccess-0.9.2/
sudo apt-get install build-essential
sudo apt-get install libglib2.0-dev
sudo apt-get install libgeoip-dev
sudo apt-get install libncursesw5-dev
./configure --enable-geoip --enable-utf8
make
sudo make install

This will install the software in “/usr/local/bin/” and the manual in “/usr/local/man/”.

Running

Running the software is as easy as:

man goaccess
goaccess -f /var/log/apache2/access.log

The software will prompt you the log format. For me the “NCSA Combined Log Format” log format worked best. For nginx I just ran:

goaccess -f /var/log/nginx/access.log

It is really awesome, try it!

Fix Ubuntu SSLv3 POODLE issue in Nginx and Apache

Are you running an HTTPS website on Ubuntu (or any other Linux) with Nginx or Apache? You may be at risk! A man-in-the-middle attack may be effective. This is explained yesterday by Google in their publication about the POODLE attack. POODLE is an acronym for “Padding Oracle On Downgraded Legacy Encryption”. The attack uses a fall-back to the 18 year old “SSLv3” protocol. The security researchers propose that the easiest “fix” is to disable this legacy SSL variant. Fortunately only a small part of your visitors will be impacted:

All modern browsers and API clients will support TLSv1 and later. Disabling SSLv3 will inconvenience WindowsXP users who browse using Internet Explorer 6 – nginx blog

The attack is registered as CVE-2014-3566. Now let’s quickly look at the commands we need to execute:

Disable SSLv3 on Apache

1) We need to edit the file that holds the Apache SSL configuration:

sudo nano /etc/apache2/mods-enabled/ssl.conf

2) Find the following line:

SSLProtocol all -SSLv2

3) Add the option “-SSLv3” so that the line will look like this:

SSLProtocol all -SSLv2 -SSLv3

4) Now restart Apache to make the change effective:

sudo service apache2 restart

Disable SSLv3 on Nginx

1) We need to search in all virtualhost configuration files for the use of the “ssl_protocols” directive:

maurits@nuc:~$ grep -R ssl_protocols /etc/nginx/sites-*
/etc/nginx/sites-available/default:    ssl_protocols SSLv3 TLSv1 TLSv1.1 TLSv1.2;

2) We need to edit each file that holds the “ssl_protocols” directive:

sudo nano /etc/nginx/sites-available/default

3) Find the following line:

ssl_protocols SSLv3 TLSv1 TLSv1.1 TLSv1.2;

4) Remove the option “SSLv3” so that the line will look like this:

ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

5) Now restart Nginx to make the change effective:

sudo service nginx reload

Bonus: Disable SSLv3 on HAProxy

1) Edit the “/etc/haproxy.cfg" file and find your “bind" line. Append “no-sslv3". For example:

bind :443 ssl crt <crt> ciphers <ciphers> no-sslv3

2) Now restart HAProxy to make the change effective:

sudo service haproxy reload

IMAP/POP3/SMTP

You should disable SSLv3 on all applications you run, so also on your IMAP/POP3/SMTP daemons. Think about Courier-imap, Dovecot, Sendmail and Postfix. For more information read this post on AskUbuntu.

Testing your server

A server that does not support SSLv3 will give the following output when trying to force a SSLv3 connection:

maurits@nuc:~$ openssl s_client -connect www.nginx.com:443 -ssl3 < /dev/null 2>&1 | grep New
New, (NONE), Cipher is (NONE)

A server that is still supporting SSLv3 (and may thus be vulnerable) will give the following output:

maurits@nuc:~$ openssl s_client -connect www.google.com:443 -ssl3 < /dev/null 2>&1 | grep New
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-RC4-SHA

NB: You can also test other services with this command, but then you need to change 443 to the appropriate port number.

 

Buffered Nginx log reading using PHP and logrotate

The problem: We want to continuously read the Nginx access logs on our high-traffic web nodes for real-time analytic purposes.

Logrotate every 10 seconds

Not possible! The minimum that logrotate allows is hourly. Apart from that I do not think this is very practical. I would still prefer to keep the logfiles hourly, for easy disk management. I do not like an hour of delay on my statistics. I would prefer something like one minute until it reaches the MySQL database.

Using a named pipe with mkfifo

The idea is that you can make a pipe with “mkfifo” and configure Nginx to log to that file. Sounds good until you realize that the buffer size of the pipe is quite limited (a few kb) and it will just silently drop log lines on buffer overflow. Also Igor Sysoev advised that you should not do it and since he knows what he is talking about, I suggest you stay away from it. 🙂

Watch the log file with inotify

This idea is to install a watcher on the log file that triggers an event when the file is written to. This is a good idea, but the amount of events on a busy server may be just a little too high for this to be really efficient. We would rather not be notified every time the filesystem sees a change on the Nginx access log file. EDIT: This is actually not a bad idea as long as you enable buffered Nginx access logging, see next post.

Open the file and read until EOF every x seconds

Open the file and keep it open. Sleep for x seconds and read the file until the end. Send that data to your statistics cluster and return to the sleep. This works great until the file gets rotated. Once the file gets rotated, you first need to finish reading the old file until you start reading the new one. Fortunately, logrotate moves the file by default and a open moved file can still be read using the original file descriptor. When we find that that the file has been moved y seconds ago and not been written to for z seconds, then we can decide to close the rotated file and open the new current file. This way we can ensure that we do not lose any log lines. Certainly x, y, and z need some values found using trial and error. I think five, three, and one seconds should be a good starting point.

Configure logrotate to rotate hourly for 48 hours

This can be done using the following logrotate script:

/var/log/nginx/*.log {
        hourly
        missingok
        rotate 48
        compress
        delaycompress
        notifempty
        create 0640 www-data adm
        sharedscripts
        prerotate
                if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
                        run-parts /etc/logrotate.d/httpd-prerotate; \
                fi \
        endscript
        postrotate
                [ -s /run/nginx.pid ] && kill -USR1 `cat /run/nginx.pid`
        endscript
}

Make sure you also move the logrotate cron job from daily to hourly using:

sudo mv /etc/cron.daily/logrotate /etc/cron.hourly/logrotate

Read the log files

The following code is an example in PHP that will demonstrate how this log reader will work. Note that Python and Golang may be more suitable languages for implementing these kinds of long running programs. This script just prints the data it reads. Streaming the log lines to your big data analytics cluster is left as an exercise to the reader. 😉

<?php
$logfile = "/var/log/nginx/access.log";
$var_x = 5;
$var_y = 3;
$var_z = 1;
$f = fopen($logfile, "r");
fseek($f,0,SEEK_END);
while (true) {
  $data = stream_get_contents($f);
  echo $data; // or send the data somewhere
  clearstatcache();
  $s1 = fstat($f);
  $s2 = stat($logfile);
  $renamed  = $s1["dev"] != $s2["dev"] || $s1["ino"] != $s2["ino"];
  if ($renamed && time()-$s1["ctime"]>$var_y && time()-$s1["mtime"]>$var_z) {
    echo "renamed\n";
    while (!file_exists($logfile)) sleep(1);
    fclose($f);
    $f = fopen($logfile,"r");
  } else sleep($var_x);
}
fclose($f);

While running the script above you can make sure the access log is written to another terminal using:

siege -c 100 http://localhost/

Now you can force the rotation to be executed using:

sudo logrotate -f /etc/logrotate.d/nginx

And see how the script gracefully handles it.

Symfony on HHVM 3 and Nginx 1.4 vs PHP 5.5 and Apache 2.4

symfony_hhvm_nginx

Installing Symfony

From symfony.com/download we get the latest (2.4.4) version of Symfony. I have unpacked it and put it in the directory “/home/maurits/public_html”. In “app/AppKernel.php” I moved the “AcmeDemoBundle” to the production section and in “routing.yml” I added the “_acme_demo” route that was originally in “routing_dev.yml”.

Test environment

I tested on my i5-2300 CPU with 16GB of RAM and an SSD. To run a benchmark, I installed on my Ubuntu 14.04 both Apache 2.4 with PHP 5.5 and Nginx 1.4 with HHVM 3. I used Apache Bench (ab) to test the “/demo” path on both web servers. In Apache, I disabled XDebug and enabled Zend OPcache.

Install and configure HHVM 3 with Nginx 1.4 for Symfony

For HHVM, we find pre-built (64-bit) packages listed on Github. This is how you install them on Ubuntu 14.04:

wget -O - http://dl.hhvm.com/conf/hhvm.gpg.key | sudo apt-key add -
echo deb http://dl.hhvm.com/ubuntu trusty main | sudo tee /etc/apt/sources.list.d/hhvm.list
sudo apt-get update
sudo apt-get install hhvm

First we install Nginx from the Ubuntu 14.04 repository using:

sudo apt-get install nginx

Now we configure Nginx using the “normal” FastCGI configuration:

server {
    listen             8080;
    server_name        sf2testproject.dev;

    root /home/maurits/public_html/web;

    location / {
        # try to serve file directly, fallback to rewrite
        try_files $uri @rewriteapp;
    }

    location @rewriteapp {
        # rewrite all to app.php
        rewrite ^(.*)$ /app.php/$1 last;
    }

    location ~ ^/(app|app_dev|config)\.php(/|$) {
        fastcgi_pass 127.0.0.1:9000;
        fastcgi_split_path_info ^(.+\.php)(/.*)$;
        include fastcgi_params;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        fastcgi_param HTTPS off;
    }
}

Single request

When doing a single request and analyzing the response times using the Firebug “Net” panel there is no noticeable difference. This is probably because the threads do not have to compete for CPU. So let’s skip this and do some load testing.

Apache Bench results (Symfony 2.4 / Apache 2.4 / PHP 5.5)

maurits@nuc:~$ ab -c 10 -n 2000 http://sf2testproject.dev/demo/
This is ApacheBench, Version 2.3 <$Revision: 1528965 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking sf2testproject.dev (be patient)
Completed 200 requests
Completed 400 requests
Completed 600 requests
Completed 800 requests
Completed 1000 requests
Completed 1200 requests
Completed 1400 requests
Completed 1600 requests
Completed 1800 requests
Completed 2000 requests
Finished 2000 requests

Server Software:        Apache/2.4.7
Server Hostname:        sf2testproject.dev
Server Port:            80

Document Path:          /demo/
Document Length:        4658 bytes

Concurrency Level:      10
Time taken for tests:   9.784 seconds
Complete requests:      2000
Failed requests:        0
Total transferred:      9982000 bytes
HTML transferred:       9316000 bytes
Requests per second:    204.42 [#/sec] (mean)
Time per request:       48.918 [ms] (mean)
Time per request:       4.892 [ms] (mean, across all concurrent requests)
Transfer rate:          996.36 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       3
Processing:    18   49  10.9     50      90
Waiting:       15   41  10.1     41      78
Total:         18   49  10.9     50      91

Percentage of the requests served within a certain time (ms)
  50%     50
  66%     54
  75%     56
  80%     58
  90%     62
  95%     65
  98%     69
  99%     73
 100%     91 (longest request)

Apache Bench results (Symfony 2.4 / Ningx 1.4 / HHVM 3)

maurits@nuc:~$ ab -c 10 -n 2000 http://sf2testproject.dev:8080/demo/
This is ApacheBench, Version 2.3 <$Revision: 1528965 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking sf2testproject.dev (be patient)
Completed 200 requests
Completed 400 requests
Completed 600 requests
Completed 800 requests
Completed 1000 requests
Completed 1200 requests
Completed 1400 requests
Completed 1600 requests
Completed 1800 requests
Completed 2000 requests
Finished 2000 requests

Server Software:        nginx/1.4.6
Server Hostname:        sf2testproject.dev
Server Port:            8080

Document Path:          /demo/
Document Length:        4658 bytes

Concurrency Level:      10
Time taken for tests:   4.678 seconds
Complete requests:      2000
Failed requests:        0
Total transferred:      9900000 bytes
HTML transferred:       9316000 bytes
Requests per second:    427.50 [#/sec] (mean)
Time per request:       23.392 [ms] (mean)
Time per request:       2.339 [ms] (mean, across all concurrent requests)
Transfer rate:          2066.52 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       1
Processing:     8   23  11.5     21      84
Waiting:        8   22  11.3     20      84
Total:          8   23  11.5     21      84

Percentage of the requests served within a certain time (ms)
  50%     21
  66%     26
  75%     30
  80%     32
  90%     39
  95%     46
  98%     54
  99%     58
 100%     84 (longest request)

Conclusion

On both setups, I did not do any optimization. I just installed and ran the benchmark. It seems that Symfony 2.4 on HHVM is about twice as fast as on PHP 5.5. In a real life setup this means you need half the machines. This seem like a good cost reduction at first, but it may not be as good as it looks. I believe that in reality most Symfony applications are doing database and/or API calls. These will not be faster when using HHVM, since HHVM only speeds up PHP execution. This is why I think that the difference will be smaller (than a factor 2) for a real life Symfony application. What do you think?

“HTTP secure link” using OpenResty Nginx module

Whenever you generate a link on your website to let your visitor download some content, you run into the risk that people will start sharing that link with others. If you want to discourage that you can require the user to login before downloading. If this is not an option for you then you can generate links that are only valid for a few seconds. This is a feature that Nginx supports using the Nginx module HttpSecureLinkModule.

Standard HttpSecureLinkModule

You can specify a Unix timestamp on which the link should no longer be valid; you add this timestamp to the query string like this:

http://localhost/package.zip?e=1402056061&s=be0c677337fec22820b916e48c54755e

In this URL (that uses ‘GET’ parameters) the ‘e’ parameter is the expiry timestamp and the ‘s’ parameter is the signature. This signature is calculated by taking a secret and appending the expiry timestamp to it and calculating a hash over that.

$signature = md5( $secret . $path . $expiry );

OpenResty Nginx modules

But what if you want to use a variant on that? For instance, if you want to restrict on certain networks or use a different hashing algorithm for the signature? My idea was to use the excellent building blocks from the set-misc-nginx-module that is made by Yichun Zhang (a.k.a. “agentzh”). This Nginx module is not included in Nginx by default, but is a part of the OpenResty Nginx distribution or as Yichun Zhang explaines:

By taking advantage of various well-designed Nginx modules, OpenResty effectively turns the nginx server into a powerful web app server. OpenResty is not an Nginx fork. It is just a software bundle. – openresty.org

We did 3 pull requests to the “set-misc-nginx-module” that allow for the needed functionality. You can compile this Nginx module into your Nginx by pointing the “–add-module” configure option to your checkout of this module. We have added the “set_encode_base64url”, “set_decode_base64url”, “set_expired” and “set_ip_matches” directives for building a custom secure link Nginx module.

Installation instructions

The configuration for the Nginx module would be like this:

# GET /?e=1893452400&n=MC4wLjAuMC8w&s=QVvNrcn-j4smvOFhuPTUU7EgoI8
# returns 403 when signature on the arguments is not correct OR
# when expire time is passed or network does not match.
# It is an alternative to the HttpSecureLinkModule in Nginx.
# This example has expiry "2030-01-01" and network "0.0.0.0/0".
location / {
  set_hmac_sha1 $signature 'secret-key' "e=$arg_e&n=$arg_n";
  set_encode_base64url $signature;
  if ($signature != $arg_s) {
    return 403;
  }
  set_expired $expired $arg_e;
  if ($expired) {
    return 403;
  }
  set_decode_base64url $network $arg_n;
  set_ip_matches $ip_matches $network $remote_addr;
  if ($ip_matches = 0) {
    return 403;
  }
  root /var/www;
}

Where the PHP code to generate the link would be:

<?php
function base64url_encode($s)
{ return rtrim(strtr(base64_encode($s),"+/", "-_"),"=");
}
$secret = "secret-key";
$expiry = strtotime("2030-01-01 00:00:00");
$network = base64url_encode("0.0.0.0/0");
$query = "e=$expiry&n=$network";
$signature = base64url_encode(hash_hmac('sha1',$query,$secret,true));
$url = "http://localhost/?$query&s=$signature";
echo "$url\n";

Output would be:

http://localhost/package.zip?e=1893452400&n=MC4wLjAuMC8w&s=QVvNrcn-j4smvOFhuPTUU7EgoI8