Symfony2 Memcache session locking

In one of the previous posts we wrote about session reliability. Today we will talk about “locking session data”. This is another session reliability topic and we will look at the problems that may occur in Symfony2 and how to solve them.

Session locking

Session locking is when the web server thread acquires an exclusive lock on the session data to avoid concurrent access. Browsers use HTTP 1.1 keep-alive and would normally just use one open TCP connection and reuse that to get all dynamic content. When loading images (and other static content) the browser may decide to use multiple TCP connections (concurrent) to get the data as fast as possible. This also happens when using AJAX. This may (and will most likely) lead to different workers (threads) on the web server answering these concurrent requests concurrently.

Each of the requests may read the session data update and write it back. The last write wins, so some writes may get lost. This can be countered by applying session locking. The session lock will prevent race conditions from occurring and prevent any corrupted data appearing in the session. This can easily be understood by looking at the following two images.

session-access-without-lockingsession-access-with-locking

The left image shows concurrent requests without session locking and the right shows concurrent requests with session locking. This is very well described in this post by Andy Bakun. Note that the above images are also from that post. Reading the Andy Bakun post allows you to truly understand the session locking problem (and the performance problems that AJAX may cause).

Symfony2 sessions

In Symfony2 one would normally use the NativeFileSessionHandler, which will just use the default PHP session handler. This works flawless in most cases. PHP uses “flock” to acquire an exclusive lock on the local filesystem. But when you scale out and run a server farm with multiple web servers you cannot use the local filesystem. You might be using a shared (NFS) filesystem and run into problems with “flock” (see the Linux NFS FAQs). If you use the database you may run into performance problems, escpecially when applying locking.

This leaves Memcache or Redis as options for session storage. These are fast key/value stores that can be used for session storage. Unfortunately the Symfony2 session storage implementations for Memcache (in Symfony) and Redis (in phpredis) do not implement session locking. This potentially leads to problems, especially when relying on AJAX calls as explained above. Note that other frameworks (like CakePHP) also do not implement session locking when using Memcache as session storage. Edit: This post has inspired the guys from SncRedisBundle and this Symfony2 bundle now supports session locking, which is totally awesome!

Custom save handlers

One can write “Custom Save Handlers” as described by the Symfony2 documentation:

Custom handlers are those which completely replace PHP’s built in session save handlers by providing six callback functions which PHP calls internally at various points in the session workflow. Symfony2 HttpFoundation provides some by default and these can easily serve as examples if you wish to write your own. — Symfony2 documentation

But you should be careful, since the examples do not implement session locking.

LswMemcacheBundle to the rescue

At LeaseWeb we love (to use) Memcache. Therefore, we have built session locking into our LswMemcacheBundle. It actually implements acquiring a “spin lock” with the timeout set to PHP’s “max_execution_time” (defaults to 30 seconds). The spin lock tries to acquire the lock every 150 ms (configurable). It will also hold the lock for a maximum time of the PHP “max_execution_time”. By using Memcache’s built-in key expire mechanism, we can ensure the lock is not held indefinitely.

This (spin-lock) implementation is a port of the session locking code from the memcached PECL module (written in C). Our bundle enables locking by default. If you want, you can disable the locking by setting the “locking” configuration parameter to “false” as described in the documentation.

This session locking code was also ported to SncRedisBundle and submitted as PR #109. LswMemcacheBundle is open-source and can be found on our GitHub account:

https://github.com/LeaseWeb/LswMemcacheBundle

Share

Fast Symfony2 served on a stable Xubuntu

Ingredients:

Introduction:

This recipe is for people who want to install symfony2 quickly with all dependencies, without having to read the excellent documentation that can be found here:

  1. The quick tour
  2. The book
  3. The cookbook

This recipe can be used by linux users or by windows users that run linux in a virtual machine.

This will get you:

  • A working Symfony2 installation
  • Pretty URL’s (using mod_rewrite)
  • Optimized execution speed (using php5-apc)
  • Configured timezone settings and internationalization support (using php5-intl)
  • Run as a specific user and not as www-data (using apache2-mpm-itk)
  • Created a database and installed a database management tool (phpmyadmin).

The instructions are meant for a development platform. If you use these instructions on a production platform the least you should do is choose a more secure password. NB: You also might want to run Ubuntu Server instead of Xubuntu in such a case.

This will NOT get you:

  • A debugging environment (using XDebug)
  • An PHP IDE with debugging integration (Eclipse PDT)

For serious web development you might want the above: a full-featured IDE and a step-by-step debugger. This will be discussed in another post.

Instructions:

The lines below with a “-” are actions and the lines with a “$” are commands:


- Download and install Virtualbox
- Download xubuntu iso
- Configure a new Virtual Machine with:
  OS set to: Linux/Ubuntu 64
  4 cores (assuming you have 8 cores)
  2048 MB RAM (assuming you have at least 4096)
  20 GB VDI dynamic disk
- Select the Xubuntu iso file in the first run wizard
- Install Xubuntu by following the steps
- Install additional drivers (guest additions) and reboot
- Open up a web browser
- Go to http://symfony.com/download and click download
- Save "Symfony_Standard_Vendors_2.x.xx.tgz" and unpack it
- Move the "Symfony" folder to your home directory and rename it to "public_html"
$ sudo apt-get install lamp-server^ php-apc php5-intl php5-sqlite phpmyadmin
 choose a mysql root password
 choose to configure phpmyadmin for apache2
$ sudo apt-get install apache2-mpm-itk
$ sudo a2enmod rewrite
$ sudo nano /etc/php5/apache2/php.ini
 NB: use Ctrl-W Ctrl-R to search and replace
 search: ;date.timezone =
 replace: date.timezone = Europe/Amsterdam
 NB: enter your local timezone if you are not in Europe/Amsterdam
 (see http://en.wikipedia.org/wiki/List_of_tz_database_time_zones)
 search: short_open_tag = On
 replace: short_open_tag = Off
$ pwd
 NB: note down the output, as this is the path of your home folder (in my case: /home/maurits)
$ id -un
 NB: note down the output, as this is your user name (in my case: maurits)
$ id -gn
 NB: note down the output, as this is your group name (in my case: maurits)
$ sudo nano /etc/apache2/sites-enabled/000-default
 NB: use Ctrl-W Ctrl-R to search and replace
 search: /var/www
 replace: {YOUR_HOME_FOLDER}/public_html/web
 NB: you have to enter the home folder path you found from the pwd command instead of {YOUR_HOME_FOLDER}
$ sudo nano /etc/apache2/sites-enabled/000-default
 NB: add the following line under the DocumentRoot
 AssignUserId {YOUR_USER_NAME} {YOUR_GROUP_NAME}
 NB: you have to enter your user name and group name instead of {YOUR_USER_NAME} {YOUR_GROUP_NAME}
$ sudo service apache2 restart
- Open web browser and go to http://127.0.0.1/phpmyadmin
- Login with root user and chosen root password
- Click "Privileges"
- Click "Add a new User"
- Enter/choose:
 username: symfony
 host: localhost
 password: symfony
 retype: symfony
- Select "Create database with same name and grant all privileges"
- Click "Create User"
- Open a web browser and go to http://127.0.0.1/config.php
- verify that symfony does not complain about anything
- click "Configure your Symfony Application online"
- Enter/choose:
 Driver: MySQL (PDO)
 Host: localhost
 Name: symfony
 User: symfony
 Password: symfony
 Password again: symfony
- Let symfony generate a "Global Secret"
- Choose "Next Step"
- Symfony should say: Your distribution is configured!
- Click "Go to the Welcome page"
- Click "Run the Demo"
- Play around with the demo to see that everything works

Bonus instructions:

Some nice tips on how to customize your Xubuntu:

That is it for today, have fun!

Share

Migration to Symfony2 continued

On December 21, 2011 Stefan Koopmanschap wrote an excellent article on this blog titled “Painless (well, less painful) migration to Symfony2.” In his article he explained the advantages of doing a gradual migration. The technical solution that he proposed to make this possible was to “…wrap your old application into your Symfony2 application.” He even provided us the tool (The IngewikkeldWrapperBundle code) to do so.

We were very much inspired by his passionate elucidation and we were fully convinced of the urge to start migrating to Symfony2 as soon as possible. However, he also provided us with a “A word of caution” about 2 things: performance and authentication/authorization. This might get some people worried, but not us: it challenged us to find a solution for those two open issues.

1. Performance

As Stefan Koopmanschap explains, in his solution you “…use two frameworks for all your legacy pages” and “…two frameworks add more overhead than one.” Since our Symfony1 application (the LeaseWeb Self Service Center) is not the fastest you’ve ever seen, some customers are even complaining about it’s speed, this got us a little worried. Losing performance was not really an option, so we had to find a solution.

Symfony 1 & 2 both use a Front Controller architecture (one file handling all requests) we were just looking for seperating traffic between the two front controllers. Stefan proposed to do so using Symfony 2 routing and make it use a fall-back route to handle your legacy URLs. We hereby propose to do it using a .htaccess rewrite. This has virtually no overhead, because every Symfony request gets rewritten by mod_rewrite anyway.

2. Authentication/authorization

He also wrote: “Another issue is sessions.” Further clarifying the problem by stating: “If your application works with authentication and authorization, you’ll now have to work with two different systems that have a different approach to authentication and authorization”. Since our application requires both authentication and authorization we had to come up with a solution here. We decided to move the authentication (login page) to Symfony2 and make Symfony1 “trust” this authentication done by “Symfony2”.

To realize this solution we had to enable Symfony1 to “see” and “understand” the Symfony2 session. First we made sure that both applications use the same name by setting the Symfony2 “framework_session_name” setting in “app/config/config.yml” to “symfony”. Then we reverse engineered the Symfony2 session storage and found that it serializes some PHP object into it. To be able to unserialize those objects we had to register an autoload function in Symfony1 using “spl_autoload_register”

Finally, instructions

To solve the performance problem we installed Symfony2 in the “sf2” directory inside the Symfony1 project (next to “apps”) and we started by changing the lines in our “web/.htaccess” file from:

# redirect to front controller
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^(.*)$ index.php [QSA,L]

And added these lines above it:

# redirect to new symfony2 front controller
RewriteCond %{REQUEST_FILENAME} !-f
# but only if URL matches one from this list:
RewriteCond %{REQUEST_URI} ^/users/login
# end of list
RewriteRule ^(.*)$ sf2/web/$1 [QSA,L]

To support the Symfony2 authentication and authorization in Symfony1 we created a “Symfony2AuthenticationFilter” class. This filter can be loaded by putting it under “lib/filter” folder in your Symfony1 project and add the following lines in “apps/ssc/config/filters.yml”:

symfony2AuthenticationFilter:
    class: Symfony2AuthenticationFilter

For configuration of the filter we added a few new application settings to “/apps/ssc/config/app.yml”:

all:
    symfony2:
        paths: ['sf2/vendor/symfony/src', 'sf2/src', 'sf2/vendor/bundles', 'sf2/vendor/doctrine-common/lib']
        attribute: '_security_main'

This path setting shows that Symfony2 is located in the “sf2” sub-directory of the Symfony1 project. The attribute reflects the name of the Symfony2 firewall. The code of the Symfony2AuthenticationFilter is this:

function symfony2_autoload ($pClassName)
{
  $sf2Paths = sfConfig::get('app_symfony2_paths');

  foreach ($sf2Paths as $path)
  {
    $path = sfConfig::get('sf_root_dir') . DIRECTORY_SEPARATOR . $path;
    $file = $path . DIRECTORY_SEPARATOR . str_replace('\\', DIRECTORY_SEPARATOR ,$pClassName ) . ".php";

    if (file_exists($file))
    {
      include($file);
      break;
    }
  }
}

spl_autoload_register("symfony2_autoload");

use Symfony\Component\Security\Core\Authentication\Token\UsernamePasswordToken;
use Symfony\Component\Security\Core\User\User;
use Symfony\Component\Security\Core\Role\Role;

class Symfony2AuthenticationFilter extends sfFilter
{
  public function execute($filterChain)
  { // get session data
    $sessionData = null;
    $symfony2Attribute = sfConfig::get('app_symfony2_attribute');
    if (isset($_SESSION['_symfony2']['attributes'][$symfony2Attribute]))
    { $sessionData = unserialize($_SESSION['_symfony2']['attributes'][$symfony2Attribute]);
    }
    // get sf1 username
    if (!$this->getContext()->getUser()->isAuthenticated()) $sf1UserName = false;
    else $sf1UserName = $this->getContext()->getUser()->getUserName();
    // get sf2 username
    if (!$sessionData) $sf2UserName = false;
    else $sf2UserName = $sessionData->getUser()->getUserName();
    // if usernames do not match
    if ($sf1UserName!=$sf2UserName)
    { if ($sf2UserName) // if symfony2 is signed in
      { // signin to symfony1
        $this->getContext()->getUser()->setUserName($sf2UserName);
        $this->getContext()->getUser()->setAuthenticated(true);
        $this->getContext()->getUser()->clearCredentials();
      }
      else // if symfony2 is not signed in
      { // signout from symfony1
        $this->getContext()->getUser()->setUserName(false);
        $this->getContext()->getUser()->setAuthenticated(false);
        $this->getContext()->getUser()->clearCredentials();
        // redirect to current page
        $path = $this->getContext()->getRequest()->getPathInfo();
        $this->getContext()->getController()->redirect($path);
      }
    }
    // and execute next filter
    $filterChain->execute();
  }
}

Update (Jan 22, 2013): You can now find the bundle we created for this on the LeaseWeb Labs on GitHub account and Packagist:

Share

Painless (well, less painful) migration to Symfony2

Over the past months I’ve talked to several companies that run a symfony 1 codebase, sometimes as old as symfony 1.0, which is big, very big. Often these applications or websites are at the heart of their business. Yet, Symfony2 is stable now, and symfony 1 support is going to stop at some point. Regularly, they ask me for advice on migrating their legacy codebase to a brand new Symfony2 project. Often enough, I have to inform them that migration from a symfony 1 to a Symfony2 codebase is a lot of effort and a huge investment in time (and therefore money). Many companies then wonder whether they should do it at all. The thing is, there is no need to do it all at once.

Gradual migration

It is much easier to do a gradual migration. Start with one part of your application, and bit by bit migrate your logic and application. The traditional way of doing such migrations is to create a new project and have parallel development on the old and the new version of the application. The problem with this, though, is that when you make a change to your old application, you have to make the same change in the new codebase, essentially doubling the amount of work for each feature you need to implement.

Wouldn’t it be much cooler if you only need to change this once? This is of course an option. You could wrap your old application into your Symfony2 application, and have different parts of your application be handled by different versions of your codebase. Once you’ve ported a new section of your application to Symfony2, you change the routing in your application to point those URLs to your Symfony2 bundle instead of your old symfony 1 project, and that part is done. You can move on to the next part.

A word of caution

While this approach is ideal because you can migrate in several phases, there are some issues with this approach. First of all, this will have a performance impact on your application. Instead of a single framework, you now use two frameworks for all your legacy pages. Result: An extra overhead because, well, two frameworks add more overhead than one. And though Symfony2 is in this case only a proxy, it does add an extra layer. Of course, you can use the cool caching features of Symfony2 to add a caching layer in front of your symfony 1 application (which might actually speed up your application), but this definitely does not apply to all projects. So before simply applying some caching to your application, think about whether it will work. And keep testing 🙂

Another issue is sessions. If your application works with authentication and authorization, you’ll now have to work with two different systems that have a different approach to authentication and authorization. So far, I have not yet figured out how to manage this, but in the projects I’ve used this approach for so far, I didn’t really need to do this. But this is something you really need to consider.

A tool

I created a simple bundle that could assist in using this approach. The IngewikkeldWrapperBundle adds a simple fallback route in your Symfony2 application, so that any URL not handled by your Symfony2 application is automatically forwarded to your legacy codebase. The legacy codebase handles the request and returns the response, and that’s it. And once you’ve had the time to work on porting a new part of the codebase to Symfony2, you simply add a route to your new bundle that matches the old URL pattern for that part of your application, and Symfony2 starts serving that part instead of symfony 1.

Share

Developing a mobile web application with IUI

At LeaseWeb we have a hosting control panel called “Self Service Center” or SSC. We are starting to make our first steps into making the Self Service Center available on a mobile platform.

We have a couple of options on how to implement this:
• Build native applications for each mobile platform ( mainly Android & iOS )
• Build a mobile accessible website

We had to make a list of advantages and disadvantages of the 2 options:

We prefer to keep a single code stack, for maintenance reasons, and so we can make every feature immediately available in the mobile version. This is why we decided to make a proof-of-concept of a mobile accessible website.

The first problem we encountered was to find out how to implement this mobile website the best way possible in the Symfony framework. At first, we checked if this was possible by doing only CSS changes, so without touching the templates. It did not took us long to come to the conclusion that this was not enough and we had to revise all of the templates.

By looking up the API of the Symfony framework on a mobile device we saw that they had a mobile version of it on their website. We were very curious what technology they used and how they implemented it, so we started digging. We found out that they were using the IUI framework on their mobile API page and although it didn’t work flawlessly we were impressed with the way it worked.

So what is this IUI and how does it work?
With IUI you have to think of your webpage as 1 big HTML file which you keep adding content to. So you keep updating your current HTML with new adaptations through AJAX request. When you click a button / link the new page is loaded off-screen. When the screen is loaded it slides smoothly into your screen. The old screens are not unloaded so you can smoothly slide back to the previous screen using the back button in the upper left corner.

Figure 1 – The red box shows the area that is visible on the screen, the other pages can slide in.

Using this framework you can make any webpage feel like a native iPhone application. A single IUI web page can contain multiple related iPhone screens. This makes IUI map on the Symfony MVC framework quite nicely, because all the data of a single MVC view will not fit on one iPhone screen, but it might fit on multiple.

Also for backend developers it is really nice that you don’t have to design all the screen layouts. You can just follow the layout that is provided with the framework and your screens immediately look professional. Although we created an impressive proof-of-concept there were some problems we did encounter:

A broken back button
When you post a form with validation errors, the same page – containing the form – is loaded with the errors displayed next to the input fields. This new page has the same id as the old form causing the back functionality to be broken.

The IUI back functionality works using a “stack”. This stack contains the ids of the previously loaded pages and forms. When the page loads it adds the id to the stack. When a validation error has occurred the stack contains the wrong values, because the validated form and the original form have the same id.

The possible solutions we found are:
1) To fix this we can give the validated form a different id than the original form.
2) Show the (validation) errors on a new page or popup.

An iPhone like way to handle field validation error is to show popups containing the validation errors. This can be achieved in IUI using “Dialog” type forms. First we tried using JavaScript “alert” calls but we had trouble, because these pop-ups are blocking and it is hard to use them without modifying IUI internals.

Displaying flash messages
More or less the same problem occurs when displaying “flash” messages. “Flash” messages are messages often colored yellow, red or green (for warnings, errors and confirmations) and are set on top of the screen displaying the results of the last page load. In IUI the flash messages are not working properly because pages are not reloaded on “back button” navigation. In an iPhone application these messages typically show in a popup.

Conclusion
We think IUI is a good and easy way to convert your current website to a mobile website in a MVC framework. Problems we encountered are related to error and status messages that a typical web application shows inside (and on top of) forms. Unfortunately IUI does not seem prepared for showing these. We’ve found a workaround by showing these messages in “Dialog” type forms.

LeaseWeb Self Service Center – https://secure.leaseweb.com
IUI framework – http://www.iui-js.org/
Symfony framework – http://www.symfony-project.org

Share