Fluid web forms using AJAX

Sometimes you want a web form that changes based on the input. The web form can give the user feedback while filling it in: We often see web forms put green check-marks behind (server side) validated fields. Some fields may turn red after you filled them in wrong. This validation can either be done client side (in JavaScript) or server side (using AJAX).

I’ve created a small application (available on Github) that looks like this:

fluid_form

When you fill in a field and it loses focus, the entire web form is submitted and updated. This includes field validation on all fields and marking them red when validation fails. Note that only after you submitted the form you want validate that the required fields are filled in. So you need to differentiate between validation before and after submit. The form has a hidden “submitted” field to facilitate that.

This demo is using the following components:

  1. Bootstrap 3 (requires jQuery)
  2. jQuery-populate (plugin by Dave Stewart)

All validation and calculation is done server side.

Why? Well, some validation can simply not be done client side. Promo codes in your web shop’s shopping cart is a good example (for security reasons). Another reason may be that sending all your validation data in a JSON object may in some cases overload your client. For instance, when you have hundreds of products with lots of product specific options. This may lead to slow rendering, due to excessive JavaScript usage. This may be especially problematic on less powerful clients, like mobile phones.

There is a downside: In order to allow for real-time server side validation you need to be able to sustain way more page loads. If the form is submitted after every changed input field, you need the web form handling to be light-weight. I managed to get the initial page load and the submit and validate to be 1 (one!) millisecond in vanilla PHP on my local machine (see screenshot). I guess that if your page loads are this cheap, you may consider using a little more server side technology. 🙂

Play with it, feel it!

git clone https://github.com/mevdschee/bootstrap-fluid-form.git
cd bootstrap-fluid-form
php -S localhost:8888

Point your browser to http://localhost:8888 and test the application. Pay special attention to the speed and the usability. Does your application resemble this? If not, then you should ask yourself… why not?

PHP-CRUD-API now has transforms!

Last week I created a new GitHub project called “PHP-CRUD-API” and it allows you to quickly setup a simple REST API with CRUD functionality by just adding a single “api.php” file to your project and configuring it’s database connection. Today I show how the relational support of this project works.

Supported table relations

There are three types of table relations supported:

  • BelongsTo
  • HasMany
  • HasAndBelongsToMany

Blog example

When you use the “list” command of the API it allows you to specify multiple tables. If these tables have relations (foreign keys) the output will be filtered in such a way that only relevant records are returned. This is what the API outputs when you list posts and comments filtered on a specific post:

{
    "posts": {
        "columns": [
            "id",
            "user_id",
            "category_id",
            "content"
        ],
        "records": [
            [
                "1",
                "1",
                "1",
                "blog started"
            ]
        ]
    },
    "comments": {
        "relations": {
            "post_id": "posts.id"
        },
        "columns": [
            "id",
            "post_id",
            "message"
        ],
        "records": [
            [
                "1",
                "1",
                "great"
            ],
            [
                "2",
                "1",
                "fantastic"
            ]
        ]
    }
}

Not so useful right? You would probably like to see something like this:

{
    "posts": [
        {
            "id": "1",
            "comments": [
                {
                    "id": "1",
                    "post_id": "1",
                    "message": "great"
                },
                {
                    "id": "2",
                    "post_id": "1",
                    "message": "fantastic"
                }
            ],
            "user_id": "1",
            "category_id": "1",
            "content": "blog started"
        }
    ]
}

That is exactly what the function “php_crud_api_transform()” does. You run this function on the client after receiving the API response. This is beneficial as it uses the CPU and RAM of the API consumer instead of that of the API server. This transformation function is implemented in PHP and JavaScript, so that you can make spiders and users with browsers equally happy!

<?php
function php_crud_api_transform(&$tables) {
	$getobjs = function(&$tables,$table_name,$where_index=false,$match_value=false) use (&$getobjs) {
		$objects = array();
		foreach($tables[$table_name]['records'] as $record) {
			if ($where_index===false || $record[$where_index]==$match_value) {
				$object = array();
				foreach ($tables[$table_name]['columns'] as $index=>$column) {
					$object[$column] = $record[$index];
					foreach ($tables as $relation=>$reltable) {
						foreach ($reltable['relations'] as $key=>$target) {
							if ($target == "$table_name.$column") {
								$columnidx = array_flip($reltable['columns']);
								$object[$relation] = $getobjs($tables,$relation,$columnidx[$key],$record[$index]);
							}
						}
					}
				}
				$objects[] = $object;
			}
		}
		return $objects;
	};
	$tree = array();
	foreach ($tables as $name=>$table) {
		if (!isset($table['relations'])) {
			$tree[$name] = $getobjs($tables,$name);
		}
	}
	return $tree;
}

And the JavaScript version:

function php_crud_api_transform(tables) {
	var array_flip = function (trans) {
		var key, tmp_ar = {};
		for (key in trans) {
			tmp_ar[trans[key]] = key;
		}
		return tmp_ar;
	};
	var get_objects = function (tables,table_name,where_index,match_value) {
		var objects = [];
		for (var record in tables[table_name]['records']) {
			record = tables[table_name]['records'][record];
			if (!where_index || record[where_index]==match_value) {
				var object = {};
				for (var index in tables[table_name]['columns']) {
					var column = tables[table_name]['columns'][index];
					object[column] = record[index];
					for (var relation in tables) {
						var reltable = tables[relation];
						for (var key in reltable['relations']) {
							var target = reltable['relations'][key];
							if (target == table_name+'.'+column) {
								column_indices = array_flip(reltable['columns']);
								object[relation] = get_objects(tables,relation,column_indices[key],record[index]);
							}
						}
					}
				}
				objects.push(object);
			}
		}
		return objects;
	};
	tree = {};
	for (var name in tables) {
		var table = tables[name];
		if (!table['relations']) {
			tree[name] = get_objects(tables,name);
		}
	}
	return tree;
}

Check out all the source code on my GitHub account: https://github.com/mevdschee/php-crud-api

PHP asset proxy increases website availability

remote_assets

Don’t you hate it when your site does not work, because you linked jQuery from “code.jquery.com” and that site is suffering connection problems? This may also happen with stylesheets or with font files. To counter this problem (but not lose the convenience of remote loaded assets) I created  an “asset proxy” in PHP. It will cache the assets in a cache folder on your web server, so that you do not have to worry about downtime of other services. You can configure how often the cache should be refreshed. When the external source is not available during a refresh the stale cache files will be used and there is no downtime at all!

proxy_assets

Install asset-proxy.php in your webroot. Then replace all references in your HTML from:

 href="http://fonts.googleapis.com/css?family=Droid+Sans:400,700"

to:

 href="/asset-proxy.php/fonts.googleapis.com/css?family=Droid+Sans:400,700"

Make sure you edit the list of allowed hostnames in the header of the PHP file and that you set an appropriate refresh time (in seconds). If the assets are not available upon refresh the stale files are served.

// hostnames for which "GET" requests can be proxied over "HTTP" (no ssl)
$hostnames = array(
	'fonts.gstatic.com',
	'maxcdn.bootstrapcdn.com',
	'netdna.bootstrapcdn.com',
	'fonts.googleapis.com',
	'ajax.googleapis.com',
);

// maximum age of a file before being refreshed
$refresh_age = 24*3600;

// directory where the cache resides (should exist and not be served)
$cache_dir = '/tmp/cache';

// strip the leading "/proxy.php/" from the URL
$url = substr($_SERVER['REQUEST_URI'], strlen($_SERVER['SCRIPT_NAME'].'/'));

// if there is no URL specified show bad request error
if(!$url || !strpos($url,'/')){
	header('Bad Request', true, 400);
	exit;
}

// get the hostname which should be the first segment (until the first slash)
$hostname = substr($url, 0, strpos($url, '/'));

// if the hostname is not in the list of allowed hostnames show forbidden error
if (!in_array($hostname, $hostnames)) {
	header('Forbidden', true, 403);
	exit;
}

// calculate the cached filename and check whether it already exists
$filename = $cache_dir.'/'.md5($url);
$file_exists = file_exists($filename);

// get the file age if the file exists
if ($file_exists) {
	$file_age = time()-filemtime($filename);
}

// if cache exists and is fresh, let's read the file, else retrieve it with cURL
if ($file_exists && $file_age<$refresh_age) {
	$result = file_get_contents($filename);
} else {
	// set some headers on the cURL call to pretend we are a user
	$sent_headers = array();
	foreach (array('User-Agent','Accept','Accept-Language','Referer') as $header) {
		$key = 'HTTP_'.strtoupper(str_replace('-','_',$header));
		if (isset($_SERVER[$key])) {
			$sent_headers[] = $header.': '.$_SERVER[$key];
		}
	}

	// make sure we do net get chunked, deflated or gzipped content
	$sent_headers[] = 'Accept-Encoding: ';
	$sent_headers[] = 'Cache-Control: max-age=0';
	$sent_headers[] = 'Connection: keep-alive';

	// initialize cURL with the URL, our headers and set headers retrieval on
	$curl = curl_init('http://'.$url);
	curl_setopt_array($curl, array(
			CURLOPT_HEADER => true,
			CURLOPT_RETURNTRANSFER => true,
			CURLOPT_BINARYTRANSFER => true,
			CURLOPT_HTTPHEADER => $sent_headers
	));

	// execute cURL call and get status code
	$result = curl_exec($curl);
	$status = curl_getinfo($curl, CURLINFO_HTTP_CODE);
	curl_close($curl);

	if ($status == 200) {
		// file was successfully retrieved
		if (file_put_contents($filename, $result)===false) {
			// show error on unsuccessful write
			header('Internal Server Error', true, 500);
			exit;
		}
	} else if ($file_exists) {
		// serve stale
		$result = file_get_contents($filename);
		// reset refresh timer
		touch($filename);
	}

}

// split the message in raw headers and body
if (strpos($result,"\r\n\r\n")!==false) {
	list($raw_headers,$body) = explode("\r\n\r\n", $result, 2);
} else {
	list($raw_headers,$body) = array($result,'');
}

// convert raw headers into an array
$raw_headers = explode("\n", $raw_headers);

// parse raw headers into received headers
$received_headers = array();
foreach ($raw_headers as $h) {
	$h = explode(':', $h, 2);
	if (isset($h[1])) {
		$received_headers[$h[0]] = trim($h[1]);
	}
}

// set certain headers for the output
$headers = array('Content-Type','Content-Encoding','Cache-Control','ETag','Last-Modified','Vary');
foreach ($headers as $header) {
	if (isset($received_headers[$header])) {
		header($header.': '.$received_headers[$header]);
	}
}

// replace the absolute URL's in the output
foreach ($hostnames as $hostname) {
	$body = preg_replace('/(https?:)?\/\/'.str_replace('.','\.',$hostname).'\//',
		$_SERVER['SCRIPT_NAME'].'/'.$hostname.'/', $body);
}

// set the new content length properly
header('Content-Length: '.strlen($body));

// echo the contents of the body
echo $body;

Best thing since sliced bread.. 😉 And only 128 lines of PHP code! Source code is on Github:

https://github.com/mevdschee/asset-proxy.php

 

Please stop using pop-up windows in web applications

In the Nineties, we were writing desktop applications with pop-ups. These desktop applications consisted of multiple windows that popped up. I was programming Delphi back in these days, where windows were called forms. The naming probably came from their main purpose: data entry into the bundled Paradox database. This is comparable with the forms that we see today on the web and they were equally abused for other purposes.

The purpose of forms in a database driven application is to facilitate CRUD (Create, Read, Update, Delete) operations. That is why you need the List, Add, Edit and Delete forms. Maybe the Delete form is not needed and this can be just a conformation dialog. To simplify the application flow, it used to be possible to make pop-up windows “modal”. This meant that you could not ignore them and had to click them away before you could continue. This is typically something you would want when you want the user to confirm an action or acknowledge a critical error.

JavaScript (like Delphi back in the day) has simplified making modal pop-ups by offering us the functions “alert()” and “confirm()”. But let’s take an example of a typical company database application. Such an application may have an overview showing a listing of customers. Maybe you can search in this list. If you click on a specific customer you may be able to see a list of their orders. In the Delphi days, we would have a window with customers and when you clicked on the “view orders” button, it would open up a new window with this information.

lightbox_popup
Figure 1: Example of a jQuery lightbox styled pop-up in WordPress

On the web, we first tried to copy this model by opening new browser windows in web applications. Then came the era of pop-up ads and ad blockers, and people started moving away from the multiple browser window strategy. This move was stimulated even more when browsers started having tabs. Then we saw that developers started making jQuery lightbox styled pop-ups on top of other pages. These are still used a lot, but I feel they lead to horrible user experiences wherever they are used.

In the end, most of the developers saw the light (fortunately). They probably realized that you do not need a stack of windows since the web browser already allows you to go back (and forward by opening new pages) in this stack using the back button and the links you can click. Also, the browser creators acknowledged that in order to make users use the “alert()” and “confirm()” functions, they had to make sure these pop-ups rendered in much prettier way. Until then, they resembled the JavaScript error pop-ups from the Nineties.

So today, when I stumble across an HTML anchor tag with an TARGET property, I cringe. It hurts even more when I see people use jQuery lightbox styled pop-ups. Not only because they are almost always a pain to close, but also because they do not work properly on different screen sizes (like phones). However, the worst thing about this form of pop-up, is the wrong expectation that people have about the underlying page. What should happen when the pop-up is closed? Should it be reloaded, so it is updated? Or can it have old data? I don’t know, can you tell me?

To add something constructive to this rant, I will also propose some new rules for pop-up lovers that have a hard time forgetting the Nineties:

  1. Everything is a page, and your application can most probably be represented by a tree (with some jumps back).
  2. Use clickable breadcrumbs to show the current path in the tree structure of your application.
  3. The back button should work everywhere and warn when needed (about reposts or expired pages).
  4. Make sure all your pages have a single, structured, short, but descriptive URL.
  5. For confirmation, rely on the JavaScript “confirm()” function.
  6. Use top of page colored flash messages to show success or failure.

Are you in the business of making web applications that mainly do CRUD operations on a database? Have you still not sworn off pop-ups? Do you think I am wrong? Please use the comments to discuss.

Loading remote data into bootstrap tabs

Many people are using Bootstrap for their website layout nowadays, and we all know that Bootstrap comes with quite a few handy JavaScript tools out-of-the-box. One of those tools is Bootstrap tabs, which lets you quickly create tabs to make your site more accessible. One feature, however, is missing from tabs, which would enhance the user experience even more: loading remote data with Ajax. Let us say you have a page that loads slowly because it is generating some statistics for a user. If you render it in tabs using the default construction, you might end up with a slow landing page.

The user might not always want to view his statistics, and it is extremely inconvenient for a user to open a completely new page just to see a few statistics.
This is where the Bootstrap remote tabs plugin comes in, this easy-to-use and lightweight (only 1.7kb) plugin will allow you to set an external source quickly in your existing tab configuration:

<!-- Nav tabs -->
 <ul>
 <li class="active"><a href="#home">Dashboard</a></li>
 <li><a href="#stats">Statistics</a></li>
 </ul>

<!-- Tab panes -->
 <div>
 <div id="home">
 <p>User dasboard<p>
 </div>
 <div id="stats"></div>
 </div>

The plugin reads the data-tab URL setting inside the ‘anchor’ tag of the tab link. No further configuration is required,
although the plugin allows for several other configuration options described in the README file.

So, how does this plugin accomplish what it does? First, it iterates over all the tab components it finds:

$('[data-toggle=tab]').each(function(k, tab) {
	// rest of code here
}

So, now we have the tab component itself. The next step is to a possible remote data url from the tab.

url = tabObj.attr('data-tab-url');
tabDiv = $(tabObj.attr('href'));
// also retrieve other settings

So now we need to listen to an event to trigger the retrieval of the remote data. In the case of the tabs it’s the show event we want to listen for:

var tabEvent = ‘bs-show’; //listen to the show event of bootstrap v3 tabs
tabObj.on(tabEvent, function(e) {
	// call execute request
}

Now we can do a Ajax call to retrieve data from the URL previously passed to the tab, and inject that content into the container that the tab is connected to.

$.ajax({
            url: url,
            success: function(data) {
                if (data) {
                   // don’t load the data more than once until it’s finished
                   if(!trigger.hasClass("loaded")) {
                      trigger.addClass("loaded");
                   }
                    tabContainer.html(data);
                }
            },
            fail: function(data) {
                // the call failed
            }
        });

The code above does a simple jQuery Ajax request, retrieves the remote content and injects it into the tab container. The actual script does a little bit more than that, but those are only applicable to the expanded feature set. This basic example gets the job done for the basic functionality of this plugin.

You can view a live demo here: http://leaseweb.github.io/bootstrap-tabs-demo/ or clone the repo and try it out for yourself: https://github.com/LeaseWeb/bootstrap-remote-tabs