Writing a compiler in C# for pre-compiled templates

I have written high performance web application frameworks for JavaScript and PHP. Today I decided to start writing a high performance web application framework in C# (on Linux using Mono). Speed and simplicity are again the goals for this MVC framework. I have read that HttpListener is very fast (comparable to NodeJS), so that seemed like a logical starting point. The framework is far from done, but one of the most interesting things I did so far was that I looked into pre-compiled templates for optimal ease of use (template errors are shown during compile time). Normally these templates can be written in T4 or Razor, but the compile time errors are not so understandable (to me).

I hand-coded a few templates in C# and wrote some input in a template language that resembles “aspx” syntax (but is not). After that I wrote a compiler that would turn the template into that exact C# code to be compiled. The template:

<%@ Page Inherits="Layouts.Default" Using="System.Text,System.Text.RegularExpressions" %>
<% Context.Data ["title"] = Context.Data ["Name"]; %>
Email: <input type="text" value="<%= Context.Data["Email"] %>"></input>
Test: <input type="text" value="<%= "some string" %>"></input>

The generated C# code:

/**
 * WARNING: Generated file, do not edit, changes will be lost!
 **/

using System;
using System.Text;
using System.Text.RegularExpressions;

namespace Maussoft.Mvc.CsTest.Views.Test
{
	public class Index: Layouts.Default
	{
		public override void Content()
		{
			WriteLine();
			Context.Data ["title"] = Context.Data ["Name"]; WriteLine();
			WriteLine(@"Email: <input type=""text"" value=""{0}""></input>",  Context.Data["Email"]);
			Write(@"Test: <input type=""text"" value=""{0}""></input>",  "some string");
		}
	}
}

I started with some regular expressions, but soon found out that I had to write a real parser. The compiler deliberately outputs multiple commands per line to ensure every line of the template corresponds to a single line of code. This is one of the design goals and this behavior should improve the ease of debugging of the code, since the errors on the compiled template will be reported on the generated C# code.

I was looking into turning this into a “Custom Tool”, but after seeing the implementation differences between MonoDevelop and Visual Studio I decided on making an executable that your run before building (as a pre-build step). Next step for the framework is probably to add something like Dapper to have simple database support. Also a run-time debugger toolbar in the browser (like symfony2 has) would be beneficial.

All code is available on Github.

Share

Chef server API integration with PHP

In this post I will show you a quick example of how you can integrate with the chef server api from php.

If you don’t know chef I recommend to have a look at https://www.chef.io. Chef is a configuration management tool, similar to ansible or puppet.

Chef turns infrastructure into code. With Chef, you can automate how you build, deploy, and manage your infrastructure.

At LeaseWeb our infrastructure that supports our business consists of many machines. For us it was a logical step to use a configuration management tool to manage all those servers and we chose chef. We also use chef to automate most of our (web) application deployments.

While our “chef managed” infrastructure was getting bigger, deploying fixes and features got easier and more frequent we needed something so our organisation is able to know what is being deployed and when.

Php is the main language we use here and we use Guzzle for quick and easy integration with rest api’s and web services.

Guzzle is a PHP HTTP client that makes it easy to send HTTP requests and trivial to integrate with web services.

Read more about guzzle here http://guzzle.readthedocs.org/.

We have created a plugin for Guzzle3 that implements the chef server authentication algorithm as described in their documentation https://docs.chef.io/auth.html

The plugin can be found on our github page https://github.com/LeaseWeb/chefauth-guzzle-plugin.

The plugin takes care of adding all the necessary http headers and signing the request to make a fully authenticated call to the chef server.

To start consuming the chef server rest api either checkout the source code with git or add the plugin as a dependency to your project using `composer`:

    php composer.phar require "leaseweb/chef-guzzle-plugin":"1.0.0"

Once you have created a user in chef the two things you need to get started is the client name of this user (in this example we assume my-dashboard) and the private key of this client (in this example we assume it is stored in my-dashboard.pem):

    <?php

    use Guzzle\Http\Client;
    use LeaseWeb\ChefGuzzle\Plugin\ChefAuth\ChefAuthPlugin;

    // Supply your client name and location of the private key.
    $chefAuthPlugin = new ChefAuthPlugin("my-dashboard", "my-dashboard.pem");

    // Create a new guzzle client
    $client = new Client('https://manage.opscode.com');
    $client->addSubscriber($chefAuthPlugin);

    // Now you can make calls to the chef server
    $response = $client->get('/organizations/my-organization/nodes')->send();

    $nodes = $response->json();

There is a ton of things you can do with the chef api, read more about it here https://docs.chef.io/api_chef_server.html

Hopefully this plugin will make it easier to integrate your chef’ed infrastructure in your company processes.

We are playing around with:

  • automatically generating release notes for our applications,
  • automatically update our issue tracking systems after a chef deployment
  • and many more.
Share

How to mock MySQLi when unit testing with PHPUnit

PHPUnit is the most used unit testing framework for PHP. Today I wanted to unit test some PHP code that relies on MySQLi. The dilemma is that you either need an actual database and load a fixture or you need to mock the database. As Claudio Lasalla clearly puts:

Unit tests are not “unit” tests if they test things other than the System Under Test (SUT).

And further explains:

Unit tests check on the behavior of units. Think of a class as being a unit. Classes, more often than not, have external dependencies. Tests for such classes should not use their real dependencies because if the dependencies have defects, the tests fail, even though the code inside the class may be perfectly fine.

This theory made total sense to me. That’s why I decided to mock the MySQLi dependency. In this post I will show you just how far I came before I realized this was not going to work out (for me).

The code

The test class, that extends “PHPUnit Framework TestCase”, has an extra method “expectQueries()”. The class looks like this:

<?php

class MySQL_CRUD_API_Test extends PHPUnit_Framework_TestCase
{
	private function expectQueries($queries)
	{
		$mysqli = $this->getMockBuilder('mysqli')
			->setMethods(array('query','real_escape_string'))
			->getMock();
		$mysqli->expects($this->any())
			->method('real_escape_string')
			->will($this->returnCallback(function($str) { return addslashes($str); }));
		$mysqli->expects($this->any())
			->method('query')
			->will($this->returnCallback(function($query) use ($queries) {
				$this->assertTrue(isset($queries[$query]));
				$results = $queries[$query];
				$mysqli_result = $this->getMockBuilder('mysqli_result')
					->setMethods(array('fetch_row','close'))
					->disableOriginalConstructor()
					->getMock();
				$mysqli_result->expects($this->any())
					->method('fetch_row')
					->will($this->returnCallback(function() use ($results) {
						static $r = 0;
						return isset($results[$r])?$results[$r++]:false;
					}));
				return $mysqli_result;
			}));

		return $mysqli;
	}

	public function testSomeSubjectThatUsesMysqli()
	{
		$mysqli = $this->expectQueries(array(
			"SELECT * FROM `table`" =>array(array('1','value1'),array('2','value2'),array('3','value3')),
			"SELECT * FROM `table` LIMIT 2" =>array(array('1','value1'),array('2','value2')),
			// other queries that may be called
		));
		// do something that uses $mysqli
	}
}

The subject-under-test is actually doing something like this:

$result = $mysqli->query("SELECT * FROM `table`");
while ($row = $result->fetch_row()) {
	// do something with the data in $row
}
$result->close();

And in the test it will return the corresponding rows for the queries that you execute. Nice huh?

Not ready

This is a proof-of-concept of a mock of the MySQLi component for PHPUnit. The ‘real_escape_string’ function has a sloppy implementation. It does not (yet) support the much used ‘prepare’, ‘execute’ or ‘fetch_fields’ methods. To give an idea of the completeness, for MySQLi it now support 2/62 functions and properties, for MySQLi Statement 0/28 and for MySQLi Result 2/15. Apart from this incompleteness there is the problem that you may need to support meta information, such as field names and types, to have a fully working mock. If you feel like continuing my work, then feel free to take my code.

Conclusion

Although this was a nice exercise and it may even be the right thing to do in theory, it did not seem to make much sense (to me) in practice. So I gave up on this approach and my current implementation runs all tests against a real database. It loads a database from a SQL file (fixture) in the static ‘setUpBeforeClass()’ function. This may not be so ‘correct’ or ‘clean’ (from a unit testing point of view), but it is much faster to write and easier to maintain.

My question for you: Am I wrong or is the theory wrong? Please tell me using the comments.

Share

MindaPHP now has Memcache support

The PHP framework I am building (MindaPHP) already contains support for MySQL and cURL. Today I have added Memcache support. Memcache can be used for two main purposes in PHP: session storage and application caching. In the framework we only support debugging Memcache for application caching. This is how the debugger looks when the Cache class is used:

mindaphp_cache

The cache is used to store the results from the Bing query. You can try this on: http://maurits.server.nlware.com/hello/bing (click the debugger link in the bottom bar after searching).

Memcache for application caching

Now you can speed up your application using the following commands:

$var = Cache::get($key)
$success = Cache::set($key,$var,$expire=0)
$success = Cache::delete($key)
$success = Cache::add($key,$var,$expire=0)
$success = Cache::replace($key,$var,$expire=0)
$var = Cache::increment($key,$value=1)
$var = Cache::decrement($key,$value=1)

The commands “get” and “set” do retrieval and storage of values in the cache based on the “key” parameter. The commands “add” and “replace” are comparable to “set”, but either fail when the key does (in case of add) or does not (in case of replace) exist. The “increment” and “decrement” commands can be used for counters, but beware that “increment” fails when the key does not exist. This is why you may want to call “add” before you increment.

Memcache for session storage

If you want to use Memcache for session storage in PHP (with any framework), you configured this in “php.ini” with the following statements:

session.save_handler = memcache
session.save_path = "tcp://localhost:11211"

Note that you need the Memcache daemon and the php Memcache extension installed. The following command installs the required software on a Debian based Linux (like Ubuntu):

sudo apt-get install php5-memcache memcached

Have fun accelerating you application!

Share

MindaPHP now has RESTful API support

When building applications today you need to follow cool new architectures like “Microservice design” and “API-first”. APIs play an increasingly important role in applications today. Together with database abstraction layers they bring the database technology further from the business logic than ever.

In line with the “Ease of learning” vision for MindaPHP, I decided to add a minimal RESTful API client in the form of a cURL wrapper. It has full integration with the debugger as you can see below:

mindaphp_api

The wrapper class hardly influences the performance of cURL when the debugger is disabled. When the debugger is enabled the performance and the memory usage may be affected, but this gives you a great deal of control as you can see in the image above. In the example below you see how you can use the cURL wrapper to call the Bing search engine and extract the first 10 links for a search query.

<?php
$query = isset($_POST['q'])?$_POST['q']:'';
$results = array();

if ($query) {
    if (Curl::call('GET','http://www.bing.com/search',array('q'=>$query),$result)==200) {
        
        $dom = new DOMDocument();
        @$dom->loadHTML($result);
        
        $xpath = new DOMXpath($dom);
        $elements = $xpath->query('//ol["b_results"]/li[@class="b_algo"]//h2/a');

        foreach ($elements as $element) {
            $text = $element->nodeValue;
            $link = $element->getAttribute("href");
            $results[] = compact('text','link');
        }
    }
}

For more information check out Github or the demo on one of the links below:

 

Code: https://github.com/mevdschee/MindaPHP
Demo: http://maurits.server.nlware.com/hello/bing

Share