Symfony2 Memcache session locking

In one of the previous posts we wrote about session reliability. Today we will talk about “locking session data”. This is another session reliability topic and we will look at the problems that may occur in Symfony2 and how to solve them.

Session locking

Session locking is when the web server thread acquires an exclusive lock on the session data to avoid concurrent access. Browsers use HTTP 1.1 keep-alive and would normally just use one open TCP connection and reuse that to get all dynamic content. When loading images (and other static content) the browser may decide to use multiple TCP connections (concurrent) to get the data as fast as possible. This also happens when using AJAX. This may (and will most likely) lead to different workers (threads) on the web server answering these concurrent requests concurrently.

Each of the requests may read the session data update and write it back. The last write wins, so some writes may get lost. This can be countered by applying session locking. The session lock will prevent race conditions from occurring and prevent any corrupted data appearing in the session. This can easily be understood by looking at the following two images.

session-access-without-lockingsession-access-with-locking

The left image shows concurrent requests without session locking and the right shows concurrent requests with session locking. This is very well described in this post by Andy Bakun. Note that the above images are also from that post. Reading the Andy Bakun post allows you to truly understand the session locking problem (and the performance problems that AJAX may cause).

Symfony2 sessions

In Symfony2 one would normally use the NativeFileSessionHandler, which will just use the default PHP session handler. This works flawless in most cases. PHP uses “flock” to acquire an exclusive lock on the local filesystem. But when you scale out and run a server farm with multiple web servers you cannot use the local filesystem. You might be using a shared (NFS) filesystem and run into problems with “flock” (see the Linux NFS FAQs). If you use the database you may run into performance problems, escpecially when applying locking.

This leaves Memcache or Redis as options for session storage. These are fast key/value stores that can be used for session storage. Unfortunately the Symfony2 session storage implementations for Memcache (in Symfony) and Redis (in phpredis) do not implement session locking. This potentially leads to problems, especially when relying on AJAX calls as explained above. Note that other frameworks (like CakePHP) also do not implement session locking when using Memcache as session storage. Edit: This post has inspired the guys from SncRedisBundle and this Symfony2 bundle now supports session locking, which is totally awesome!

Custom save handlers

One can write “Custom Save Handlers” as described by the Symfony2 documentation:

Custom handlers are those which completely replace PHP’s built in session save handlers by providing six callback functions which PHP calls internally at various points in the session workflow. Symfony2 HttpFoundation provides some by default and these can easily serve as examples if you wish to write your own. — Symfony2 documentation

But you should be careful, since the examples do not implement session locking.

LswMemcacheBundle to the rescue

At LeaseWeb we love (to use) Memcache. Therefore, we have built session locking into our LswMemcacheBundle. It actually implements acquiring a “spin lock” with the timeout set to PHP’s “max_execution_time” (defaults to 30 seconds). The spin lock tries to acquire the lock every 150 ms (configurable). It will also hold the lock for a maximum time of the PHP “max_execution_time”. By using Memcache’s built-in key expire mechanism, we can ensure the lock is not held indefinitely.

This (spin-lock) implementation is a port of the session locking code from the memcached PECL module (written in C). Our bundle enables locking by default. If you want, you can disable the locking by setting the “locking” configuration parameter to “false” as described in the documentation.

This session locking code was also ported to SncRedisBundle and submitted as PR #109. LswMemcacheBundle is open-source and can be found on our GitHub account:

https://github.com/LeaseWeb/LswMemcacheBundle

Share

Testing your project with PHPUnit and Selenium

PHPUnit is a very handy tool you can use to test and detect possible mistakes in your PHP project.

Functional testing is also possible with PHPUnit using default headless browser (crawler). It works perfect as long as you don’t need to test functionality which uses JavaScript. For this purpose, you can use PHPUnit and Selenium.

Selenium is a powerful test tool that allows you to perform tests for web applications in any programming language using any mainstream browser.

Installation requirements

Before you start using Selenium to test your PHP project, you need to install the following:

  1. Selenium Server
  2. PHPUnit_Selenium package

1. Installing Selenium Server

Perform the following steps to install the Selenium Server:

    1. Download a distribution archive of Selenium Server.
    2. Unzip the distribution archive and copy selenium-server-standalone-2.35.0.jar (check the version suffix) to a folder from where you will run the server (such as /usr/local/bin).
    3. Start the Selenium Server by running the following command:
java -jar /usr/local/bin/selenium-server-standalone-2.35.0.jar

2. Installing PHPUnit_Selenium package

      The PHPUnit_Selenium package is necessary for natively accessing the Selenium Server from PHPUnit.
      To install it, run the following command:
pear install phpunit/PHPUnit_Selenium

Using Selenium in PHPUnit tests

There are two Selenium test cases:

    • PHPUnit_Extensions_Selenium2TestCase
    • PHPUnit_Extensions_SeleniumTestCase

PHPUnit_Extensions_Selenium2TestCase test case allows you to use the WebDriver API (partially implemented).

<?php
class WebTest extends PHPUnit_Extensions_Selenium2TestCase
{
    protected function setUp()
    {
        $this->setBrowser('firefox');
        $this->setBrowserUrl('http://www.example.com/');
    }

    public function testTitle()
    {
        $this->url('http://www.example.com/');
        $this->assertEquals('Example WWW Page', $this->title());
    }

}
?>

PHPUnit_Extensions_SeleniumTestCase test case implements the client/server protocol to talk to Selenium Server as well as specialized assertion methods for web testing.

<?php
require_once 'PHPUnit/Extensions/SeleniumTestCase.php';

class WebTest extends PHPUnit_Extensions_SeleniumTestCase
{
    protected function setUp()
    {
        $this->setBrowser('*firefox');
        $this->setBrowserUrl('http://www.example.com/');
    }

    public function testTitle()
    {
        $this->open('http://www.example.com/');
        $this->assertTitle('Example WWW Page');
    }
}
?>

Using different browsers with WebDrivers

For running tests with different browsers you should have WebDriver of that browser. As an example, we’ll try to use the Google Chrome browser and its WebDriver.

  1. Download WebDriver.
  2. To make Selenium Server aware of this WebDriver, perform one of the following tasks:
    • Store the Chrome WebDriver binary in the system path
    • Start the Selenium Server with -Dwebdriver.chrome.driver=path/to/your/chromedriver

You can now set Chrome as a browser for your functional test:

...
$this->setBrowser('chrome');
...

Running headless Selenium Server with Xvfb

Sometimes you don’t want the browser to be launched at your desktop during testing, or you may not be using Xserver. To make it work, one of the easiest solutions is to use Xvfb.

Xvfb is an X11 server that performs various graphical operations in memory, without displaying any screen output.

So, let’s try it. At first ensure that you have Xvfb installed on your server. If not, you can install it from your OS repository. For example, to install it on Ubuntu, run the following command:

sudo apt-get install xvfb

Once installed, to run your Selenium Server in Xvfb, run the following command:

DISPLAY=:1 xvfb-run java -jar selenium-server-standalone-2.35.0.jar

Once the server starts, run any of the test case examples. The output should display as follows:

PHPUnit 3.8.0 by Sebastian Bergmann.

F

Time: 4 seconds, Memory: 6.25Mb

There was 1 failure:

1) WebTest::testTitle
Failed asserting that two strings are equal.
--- Expected
+++ Actual
@@ @@
-'Example WWW Page'
+'Example Domain'

/your/project/Tests/Functional/WebTest.php:14

FAILURES!
Tests: 1, Assertions: 1, Failures: 1.

That’s all for now. Enjoy testing!

Source references:

  1. PHPUnit and Selenium (official documentation)
  2. Selenium Server downloads list
Share

7 bundles for highly effective software development

photo

At the Dutch Symfony Meetup on 4th of September (that was generously hosted by my employer LeaseWeb) I have been giving a talk about LeaseWeb’s open-sourced Symfony2 bundles. The slides are available!

1. Software consistency in large teams

Bundle: LswDefaultRoutingBundle – v1.0.1
Github URL: https://github.com/LeaseWeb/LswDefaultRoutingBundle
Description: Symfony2 bundle that provides default routing, relative routing and default templating.

Things like “conventions” and the compliance of those could be measurements for software quality. But having conventions means that you can actually choose not to follow them, which would lead to lower software quality, so why would one allow it? One of these cases is the Symfony routing system. The routes need to be individually specified in the routing configuration using annotations (special comment blocks) in the Controller code of the application or in an YML (or XML) file that resides either on application level or on bundle level.

Well, as you see this written down you might ask yourself: Why do you allow the programmer to have all these choices. Fellow programmers will answer: these are features of the routing system. It is awesome, because it supports all ways I might want to specify my routes. People who look at software quality and cost of maintenance may disagree: having to look at all these places to find a defined route in a software package might make adding a feature or doing a bug-fix more expensive. Also it is prone to typos to have to specify every route individually.

In Symfony1, one could choose to create a default route to avoid having to specify every route individually. In Symfony2 this option is no longer available and the DefaultRoutingBundle has exactly that mission: “bring back default routing functionality to Symfony2”. The aim is to do that in such a way it respects all features and tools the Symfony2 routing system has.

Apart from routing one can also argue that software development gets cheaper when bundles are reused in different software teams. Not only because development times are lower, but also because maintenance can be combined and it provides a way sharing lessons learned between teams.

2. Debugging API based applications

Bundle: LswApiCallerBundle – v1.0.1
Github URL: https://github.com/LeaseWeb/LswApiCallerBundle
Description: Symfony2 bundle that adds cURL API call functionality with full Web Debug Toolbar integration.

In modern web development, the use of web based API’s is omni-present. Connecting to these API’s is not hard at all, but debugging your Symfony application when these API’s fail can be a nightmare. This bundle provides insights  into: call parameters, call responses, time taken per call, total call time, number of calls and errors during calls. This information can be accessed from the Web Debug Toolbar of Symfony in the same way you can access this information for debugging SQL queries made on the database.

The API caller bundle understands different data formats (like XML, JSON and HTML), but can also display responses as plain text. To help developers create healthy (fast) applications the bundle shows the total amount of milliseconds spent to do API calls. It also shows the amount of API calls that are made from a single page. It will these show numbers in green, yellow and red to warn developers to not make too many API calls.

3. Caching for high traffic web applications

Bundle: LswMemcacheBundle – v1.1.0
Github URL: https://github.com/LeaseWeb/LswMemcacheBundle
Description: Symfony2 bundle for Memcache Doctrine caching and session storage in the Web Debug Toolbar.

When building a real high traffic web page you need Memcache. Memcache is a caching server that is very popular for high traffic websites. Especially when you have a web-farm it will become a necessity to use Memcache. You can use your database server(s) as cache server(s) by running Memcache on them.

This bundle provides debugging capabilities, multi-server support and a feature that is rarely used, but very valuable in extreme caching cases: Anti-Dog-Pile support. It also supports MemCached v2, so it is very future proof.

4. High performance and industry standard translations

Bundle: LswGettextTranslationBundle – v1.0.1
Github URL: https://github.com/LeaseWeb/LswGettextTranslationBundle
Description: Symfony2 bundle that adds native (faster) gettext translation support and is easy to use.

Translating agencies are used to work with gettext (worlds most popular translation engine for open source software). Adding translation support to any application is not only complex, it is also a lot of work. A large part of the complexity is in the work-flow. How do you handle updates to the software? How to extract untranslated texts and how to send them to the translators for all supported languages. What formats do they accept? How do you separate the software over different agencies and important: can you reuse part of the translations, like with bundles?

Then there is a performance problem. How expensive is it to render a multi-lingual page? Does it not slow down the application too much? All these questions are answered with this bundle. Since most people will run Linux web-servers with native gettext support we’ve chosen that to get maximum performance. And to optimize translation reuse the choice was made to use one translation file per bundle (per language). The bundle provides commands to extract language files from source-code and the tool “poedit” can be used to enter the missing translations.

5. Better testing support for staging environments

Bundle: LswVersionInformationBundle – v1.0.1
Github URL: https://github.com/LeaseWeb/LswVersionInformationBundle
Description: Symfony2 bundle that adds output of SVN and Git “status” commands to the Web Debug Toolbar.

One of the things you want to know when testing an application is what you version of the software you are testing. In our projects we have chosen Subversion (and recently Git) as our SCM to keep track of versions. Within these systems we use branches and tags to identify versions. But only the programmers who create those tags and the system administrators that deploy these versions know what version is running. This bundle has the mission to “make visible to the testers of the application what version of the software they are testing”. One of the more advanced things the bundle does is that is shows a list of dirty files (if any).

6. Consistent updates for high quality staging

Bundle: LswAutomaticUpdateBundle – v1.0.1
Github URL: https://github.com/LeaseWeb/LswAutomaticUpdateBundle
Description: Symfony2 bundle that enables automatic updates of the application from the Web Debug Toolbar.

This will show in the web debug toolbar which of your (composer) packages are not running stable versions. It also shows when the “composer update” was last run, so that you can see whether or not software maintenance was executed. It also provides (an experimental) feature to update your environment from the web debug toolbar.

7. Symfony1 to symfony2 migration

Bundle: LswRemoteTemplateBundle – v1.0.1
Github URL: https://github.com/LeaseWeb/LswRemoteTemplateBundle
Description: Symfony2 bundle that allows you to load your (view) template from a remote site.

When you want to migrate from Symfony1 to Symfony2 you should probably try to avoid a “big bang” migration. It would be too risky and too much work. Probably managers will not let you work on the migration that could take months, will not bring much new functionality and causes high risk for regression. This bundle allows you to load the template from your symfony1 application while actually sharing your session between both Symfony1 and Symfony2. This way you can gradually migrate from Symfony1 to Symfony2 one Controller or even one view at a time.

Share

Bootstrap 3.0.0 RC1 customization

Bootstrap is an HTML, CSS, JavaScript framework that you can use as a basis for creating websites or web applications. It is a well documented tool that simplifies the creation of beautiful form based applications. Especially when you are using Symfony2 and you use MopaBootstrapBundle. When we discussed version 2, we explained how it influences your front-end development cost. Now version 3 is about to be released. MopaBootstrapBundle has already two PR (pull requests) open for supporting Bootstrap 3. If you want to modify the Bootstrap default colors to match your company’s new website layout, this blog provides the instructions. However, make sure you first check out the Bootstrap 2 to 3 migration guide.

1) Download Bootstrap

git://github.com/twbs/bootstrap.git
cd bootstrap

2) Install npm and node (Ubuntu)

curl https://npmjs.org/install.sh | sudo sh
sudo ln -s /usr/local/bin/npm /usr/bin/npm
sudo npm cache clean -f
sudo npm install -g n
sudo n stable

Or using the package manager (thank you Mischa):

sudo apt-get install python-software-properties python g++ make;
sudo add-apt-repository ppa:chris-lea/node.js;
sudo apt-get update;
sudo apt-get install nodejs;

3) Install grunt-cli globally and grunt locally

sudo npm install -g grunt-cli
npm install grunt
npm install grunt-contrib-connect grunt-contrib-clean grunt-contrib-concat grunt-contrib-jshint
npm install grunt-contrib-uglify grunt-contrib-qunit grunt-contrib-watch grunt-recess

4) We need Jekyll as well..

sudo apt-get install rubygems
sudo gem install jekyll

5) Now do the actual modification

Now edit the files in the “less” directory (start with “variables.less”) with you favorite editor. Note that less is some sort of programming language that allows you to script consistent CSS generation. One of the examples is that Bootstrap has a “brand-primary” color that can be used in different shades using the “lighten” and “darken” functions.

6) Test your creation

grunt dist
jekyll serve

Questions? Comments? Let us know!

Share

@Secure annotation for Symfony 2.3

We created the LswSecureControllerBundle to provide @Secure annotation in Symfony 2.3 to secure actions in controllers by specifying required roles. This functionality was provided by the JMSSecurityExtraBundle, included in Symfony 2.0, 2.1 and 2.2. In  Symfony 2.3 it was removed because of a license incompatibility between Symfony Core and the JMSSecurityExtraBundle. On the official blog Fabien Potencier (author of Symfony) writes:

As the Symfony Standard Edition is under the MIT license, we did not want to have anything else in its deps. That helps people developing GPL projects on top of Symfony. That said, if you are developing your own project, using dependencies under an Apache license should not be a problem at all. So, feel free to add them back. — Fabien Potencier in a response

So, the JMS bundles are no longer included by default and honestly, apart from the license, it is a huge pile of code. Especially as the only feature we use from it, is the @Secure annotation to require roles for executing specific actions in the controllers. Please note that it is not only a beefy bundle, but it also has an impressive set of dependencies:

And don’t get me wrong; I believe the JMSSecurityExtraBundle does a lot of other things that are very valuable and fully justifies its size and its dependencies. I also highly respect Johannes Schmitt for his efforts in the Symfony community and his coding skills. However, if you only need this one feature and want to stay lean (122 lines of code) or if you need a more permissive license, then our bundle is the one for you.

Installation and usage

secure_test
Figure 1: Symfony 2.3 denying access based on a @Secure annotation

Installation of this bundle is just like any other; details can be found in the README. We will show the usage by providing an example based on the standard Symfony 2.3 AcmeDemoBundle. We use the @Secure annotation in the AcmeDemoBundle to secure the “hello world” page requiring the role “ROLE_TEST” to execute.

In “src/Acme/DemoBundle/Controller/SecuredController.php” you should add the following line on top, but under the namespace definition:

use Lsw\SecureControllerBundle\Annotation\Secure;

To require the “ROLE_TEST” for “helloAction” in the “SecuredController” you should add @Secure annotation to the DocBlock of the “helloAction” like this (line 2):

    /**
     * @Secure(roles="ROLE_TEST")
     * @Route("/hello", defaults={"name"="World"}),
     * @Route("/hello/{name}", name="_demo_secured_hello")
     * @Template()
     */
    public function helloAction($name)
    {
        return array('name' => $name);
    }

Note that you can only use the @Secure annotation on actions that are protected by a firewall. You can configure the firewall in the “app/config/security.yml” file.

Credits

Creating the bundle would not have been possible without Matthias Noback’s excellent posts:

Thank you Matthias!

Support

We are committed to support this bundle and, as with all our bundles, you can contact us via the LswSecureControllerBundle Github page if you find a bug or want a feature to be added.

Share