Redis sorted set stores score as a floating point number

Today I was playing a little with our statistics. I was writing some “Go” (golang) that was requesting a top 20 customers from Redis using a “Sorted Set” and the “ZREVRANGEBYSCORE” command. Then I found out that the score of the sorted set was actually stored as a double precision floating point number. Normally, I would not be bothered about this and use the float storage for integer values.

But this time I wanted to make a top 20 of the real-time monthly data traffic of our entire CDN platform. A little background: Hits on the Nginx edges are measures in bytes and the logs are streamed to our statistics cluster. Therefor the real-time statistics counters for the traffic are in bytes. Normally we use 64 bit integers (in worst case they are signed and you lose 1 bit).

2^64 = 9,223,372,036,854,775,807
      EB, PB, TB, GB, MB, kB,  b

If you Google for: “9,223,372,036,854,775,807 bytes per month in gigabit per second” you will find that this is about 26 Tbps on average. We do not have such big customers yet, so that will do for now. So an “int64” will do, but how about the double precision float? Since it has a floating point, theory says it can not reliably count when numbers become too large. But how large is too large? I quickly implemented a small script in golang to find out:

package main

import (
	"fmt"
)

func main() {
	bits := 1
	float := float64(1)
	for float+1 != float {
		float *= 2
		bits++
	}
	fmt.Printf("%.0f = %d bits\n", float, bits)
}

Every step the script doubles the number and tries to add 1 until it fails. This is the output showing when the counting goes wrong:

9007199254740992 = 54 bits

So from 54 bits the counting is no longer precise (to the byte). What does that mean for our CDN statistics? Let’s do the same calculation we did before:

2^54 = 9,007,199,254,740,992
      PB, TB, GB, MB, kB,  b

If you Google for: “9,007,199,254,740,992 bytes per month in gigabits per second” you will find that this is about 25 Gbps on a monthly average. We definitely have customers that do much more than that.

I quickly calculated that the deviation would be less than 0.000000000000001%. But then I realized I was wrong: At 26 Tbps average the deviation might as well be as big as 1 kB (10 bits). Imagine that the customer is mainly serving images and JavaScript from the CDN and has an average file size of 10 kB. In this case the statistics will be off by 10% during the last days of the month!

Okay, this may be the worst case scenario, but still I would not sleep well ignoring it. I feel that when it comes to CDN statistics, accuracy is very important. You are dealing with large numbers and lots of calculation and as you see this may have some unexpected side effects. That is why these kind of seemingly small things keep me busy.

Symfony2 Memcache session locking

In one of the previous posts we wrote about session reliability. Today we will talk about “locking session data”. This is another session reliability topic and we will look at the problems that may occur in Symfony2 and how to solve them.

Session locking

Session locking is when the web server thread acquires an exclusive lock on the session data to avoid concurrent access. Browsers use HTTP 1.1 keep-alive and would normally just use one open TCP connection and reuse that to get all dynamic content. When loading images (and other static content) the browser may decide to use multiple TCP connections (concurrent) to get the data as fast as possible. This also happens when using AJAX. This may (and will most likely) lead to different workers (threads) on the web server answering these concurrent requests concurrently.

Each of the requests may read the session data update and write it back. The last write wins, so some writes may get lost. This can be countered by applying session locking. The session lock will prevent race conditions from occurring and prevent any corrupted data appearing in the session. This can easily be understood by looking at the following two images.

session-access-without-lockingsession-access-with-locking

The left image shows concurrent requests without session locking and the right shows concurrent requests with session locking. This is very well described in this post by Andy Bakun. Note that the above images are also from that post. Reading the Andy Bakun post allows you to truly understand the session locking problem (and the performance problems that AJAX may cause).

Symfony2 sessions

In Symfony2 one would normally use the NativeFileSessionHandler, which will just use the default PHP session handler. This works flawless in most cases. PHP uses “flock” to acquire an exclusive lock on the local filesystem. But when you scale out and run a server farm with multiple web servers you cannot use the local filesystem. You might be using a shared (NFS) filesystem and run into problems with “flock” (see the Linux NFS FAQs). If you use the database you may run into performance problems, escpecially when applying locking.

This leaves Memcache or Redis as options for session storage. These are fast key/value stores that can be used for session storage. Unfortunately the Symfony2 session storage implementations for Memcache (in Symfony) and Redis (in phpredis) do not implement session locking. This potentially leads to problems, especially when relying on AJAX calls as explained above. Note that other frameworks (like CakePHP) also do not implement session locking when using Memcache as session storage. Edit: This post has inspired the guys from SncRedisBundle and this Symfony2 bundle now supports session locking, which is totally awesome!

Custom save handlers

One can write “Custom Save Handlers” as described by the Symfony2 documentation:

Custom handlers are those which completely replace PHP’s built in session save handlers by providing six callback functions which PHP calls internally at various points in the session workflow. Symfony2 HttpFoundation provides some by default and these can easily serve as examples if you wish to write your own. — Symfony2 documentation

But you should be careful, since the examples do not implement session locking.

LswMemcacheBundle to the rescue

At LeaseWeb we love (to use) Memcache. Therefore, we have built session locking into our LswMemcacheBundle. It actually implements acquiring a “spin lock” with the timeout set to PHP’s “max_execution_time” (defaults to 30 seconds). The spin lock tries to acquire the lock every 150 ms (configurable). It will also hold the lock for a maximum time of the PHP “max_execution_time”. By using Memcache’s built-in key expire mechanism, we can ensure the lock is not held indefinitely.

This (spin-lock) implementation is a port of the session locking code from the memcached PECL module (written in C). Our bundle enables locking by default. If you want, you can disable the locking by setting the “locking” configuration parameter to “false” as described in the documentation.

This session locking code was also ported to SncRedisBundle and submitted as PR #109. LswMemcacheBundle is open-source and can be found on our GitHub account:

https://github.com/LeaseWeb/LswMemcacheBundle

Replacing SOA API calls by EDA request/replies using Redis

This post aims at simplifying and explaining Enterprise Application Integration (EAI) concepts using more pictures and examples than words.

soa_vs_eda

The main difference between Event Driven Architecture (EDA) and Service Oriented Architecture (SOA) is that EDA is asynchronous and SOA synchronous.

soa

In the picture above you see how SOA typically connects anything to anything directly.

eda

The EDA pattern, on the other hand, uses a single communication channel (message bus or queueing server).

Ideally we stop thinking about API calls (request/response) and we move to “events”. “Events” are things that happen in your application that might be of interest to other applications. When a customer needs to be created by your website you can make an API call and wait for an “OK”, or you could rewrite your application to “fire and forget” and handle errors separately. But what if an application has a question that needs to be answered? For example: What if you have a “customer reference” and want to find the corresponding “email address”?

The solution is to use the Request/Reply Enterprise Integration Pattern (EIP). Most Enterprise Messaging Systems (EMS) offer an Enterprise Service Bus (ESB) on which the message format (set of optional and required headers) is specified, but the data format is free. It is up to the applications to run their own integration engine, which has adapters for different data types for the various applications it wants to talk with.

A SugarCRM example

The following message could be in the “SugarCRM/Requests” queue:

Timestamp:  1382055093876 ms
MessageId:  345781632423
ReplyQueue: SugarCRM/Replies
Data:       {"method":"GET","url":"/Accounts/f1eeca5f-c0eb-891e-db92-52323b958a87"}

The SugarCRM application has an integration engine that monitors the “SugarCRM/Requests” queue. It will receive the above message, transforms it into a API request and executes it. The result that it receives will be transformed into the following message and put in the “SugarCRM/Replies” queue:

Timestamp:  1382055096943 ms
MessageId:  345781632567
ReplyTo:    345781632423
Data:       {"id":"f1eeca5f-c0eb-891e-db92-52323b958a87","name":"RR. Talker Co","date_entered":"2013-09-12T18:09:00-04:00","date_modified":"2013-09-12T18:09:00-04:00","modified_user_id":"1","modified_by_name":"Administrator","created_by":"1","created_by_name":"Administrator","description":"","deleted":false,"assigned_user_id":"seed_jim_id","assigned_user_name":"Jim Brennan","team_count":"","team_name":[{"id":"East","name":"East","name_2":"","primary":true}],"linkedin":"","facebook":"","twitter":"","googleplus":"","account_type":"Customer","industry":"Electronics","annual_revenue":"","phone_fax":"","billing_address_street":"67321 West Siam St.","billing_address_street_2":"","billing_address_street_3":"","billing_address_street_4":"","billing_address_city":"Santa Fe","billing_address_state":"NY","billing_address_postalcode":"44150","billing_address_country":"USA","rating":"","phone_office":"(949) 400-8060","phone_alternate":"","website":"www.imim.name","ownership":"","employees":"","ticker_symbol":"","shipping_address_street":"67321 West Siam St.","shipping_address_street_2":"","shipping_address_street_3":"","shipping_address_street_4":"","shipping_address_city":"Santa Fe","shipping_address_state":"NY","shipping_address_postalcode":"44150","shipping_address_country":"USA","email":[{"email_address":"phone95@example.it","invalid_email":false,"opt_out":false,"primary_address":true,"reply_to_address":false},{"email_address":"qa.qa@example.tv","invalid_email":false,"opt_out":false,"primary_address":false,"reply_to_address":false}],"email1":"phone95@example.it","parent_id":"","sic_code":"","parent_name":"","email_opt_out":false,"invalid_email":false,"campaign_id":"","campaign_name":"","my_favorite":false,"_acl":{"fields":{}},"following":true,"_module":"Accounts"}

Tutorial: EDA with Redis

One of the more KISS – but less conventional – ways of implementing the above scheme would be to use Redis to store queues. The “SugarCRM/Requests” queue can be implemented as a “List” of “MessageId” values. The messages can be stored with “Hash” data types in the global namespace using the Redis “HMSET” command, where the headers can be individual keys. The unique “MessageId” values can be generated using the Redis “INCR” command. The “SugarCRM/Replies” queue can be implemented as a “Hash” data type as well, where  the “ReplyTo” can be used as the key and the “MessageId” as the value.

Phpredis as Redis client in Ubuntu 12.04 LTS

Redis can be run with either Predis or phpredis. This post will explain how you can install the phpredis client on Ubuntu 12.04 LTS. This post assumes you run the standard LAMP stack. We have talked about running Redis in Symfony2 before, but then we assumed you would be running Predis.

Redis is an open source, BSD licensed, advanced key-value store. It is often referred to as a data structure server since keys can contain strings, hashes, lists, sets and sorted sets. — source: redis.io

The main advantage of running phpredis over running Predis is that it is faster, because phpredis is written in C, while Predis is written in PHP. So, if you want to run the fastest client for your server than follow these simple instructions:

Instructions for installing phpredis (bleeding edge)

1) Make sure you have the dependencies installed:

sudo apt-get install redis-server php5-dev build-essential xsltproc

2) Now clone the phpredis repo, configure and make:

git clone git@github.com:nicolasff/phpredis.git
cd phpredis/
phpize
./configure
make

3a) Install the compiled files directly:

sudo cp modules/redis.so `php-config --extension-dir`
sudo cp rpm/redis.ini /etc/php5/conf.d/
sudo service apache2 restart

3b) or create, view and install the Debian package:

./mkdeb-apache2.sh
dpkg -c phpredis-x86_64.deb
sudo dpkg -i phpredis-x86_64.deb

Instructions for installing phpredis (stable)

1) Make sure you have the dependencies installed:

sudo apt-get install redis-server dh-make-php php5-dev build-essential

2) Now get Redis from PECL and make a package:

mkdir redis
cd redis
export DEBFULLNAME="Maurits van der Schee"
export DEBEMAIL="m.vanderschee@leaseweb.com"
dh-make-pecl redis

3) or create, view and install the Debian package:

cd php-redis-2.2.3/
debuild
dpkg -c ../php5-redis_2.2.3-1_amd64.deb
sudo dpkg -i ../php5-redis_2.2.3-1_amd64.deb

Test to see whether it works or not

To test the loaded module create a PHP file that you run from the command-line or via Apache2:

        <?php
        $redis = new Redis() or die("Cannot load Redis module.");
        $redis->connect('localhost');
        $redis->set('random', rand(5000,6000));
        echo $redis->get('random');

That’s it! Easy. And it performs way better than Predis.

Reliable queues in Redis

You can run Redis, for instance, if you need a fast and scalable queuing system in a distributed platform. You can implement such a queue using the LPUSH and BRPOP commands. With LPUSH you add elements to the start of the list, while the BRPOP removes elements from the list, blocking when no elements are available. But if you want that queue to be reliable, you can consider using the BRPOPLPUSH command, that can add the returned element to an ‘in progress’ queue before returning it to the client. This way you can remove the element from the ‘in progress’ queue when you successfully handled them, or put them back on the original queue if you failed.

Redis bundle for Symfony2

redis_screenshot

The  SncRedisBundle integrates Predis and phpredis into your Symfony2 application.

“Redis is an open source, BSD licensed, advanced key-value store. It is often referred to as a data structure server since keys can contain stringshasheslistssets and sorted sets.” — redis.io

At LeaseWeb we developed the LswMemcacheBundle. This is not because we do not like Redis. Maybe Redis serves a different purpose, since it has support for data structures, but is single threaded on the other hand. Having data structures could lead to better (more efficient) cache usage, while being single threaded does make a difference on the maximum amount of concurrent get and set operations (see benchmark). So without choosing for either Redis or Memcache we want to tell you about SncRedisBundle for Symfony.

The biggest differences between Redis and Memcache we found:

  • Redis has support for data structures
  • Redis has support for persistent storage
  • Redis has more features (in general)
  • php-redis and memcached are both C based clients, but php-redis is not yet in the Ubuntu repository and predis might be slower.
  • Memcache is multi-threaded and gets higher IOPS on a multi core server

If you run a high traffic PHP site, consider running Redis and/or Memcache and share your thoughts with us.