Replacing SOA API calls by EDA request/replies using Redis

This post aims at simplifying and explaining Enterprise Application Integration (EAI) concepts using more pictures and examples than words.


The main difference between Event Driven Architecture (EDA) and Service Oriented Architecture (SOA) is that EDA is asynchronous and SOA synchronous.


In the picture above you see how SOA typically connects anything to anything directly.


The EDA pattern, on the other hand, uses a single communication channel (message bus or queueing server).

Ideally we stop thinking about API calls (request/response) and we move to “events”. “Events” are things that happen in your application that might be of interest to other applications. When a customer needs to be created by your website you can make an API call and wait for an “OK”, or you could rewrite your application to “fire and forget” and handle errors separately. But what if an application has a question that needs to be answered? For example: What if you have a “customer reference” and want to find the corresponding “email address”?

The solution is to use the Request/Reply Enterprise Integration Pattern (EIP). Most Enterprise Messaging Systems (EMS) offer an Enterprise Service Bus (ESB) on which the message format (set of optional and required headers) is specified, but the data format is free. It is up to the applications to run their own integration engine, which has adapters for different data types for the various applications it wants to talk with.

A SugarCRM example

The following message could be in the “SugarCRM/Requests” queue:

Timestamp:  1382055093876 ms
MessageId:  345781632423
ReplyQueue: SugarCRM/Replies
Data:       {"method":"GET","url":"/Accounts/f1eeca5f-c0eb-891e-db92-52323b958a87"}

The SugarCRM application has an integration engine that monitors the “SugarCRM/Requests” queue. It will receive the above message, transforms it into a API request and executes it. The result that it receives will be transformed into the following message and put in the “SugarCRM/Replies” queue:

Timestamp:  1382055096943 ms
MessageId:  345781632567
ReplyTo:    345781632423
Data:       {"id":"f1eeca5f-c0eb-891e-db92-52323b958a87","name":"RR. Talker Co","date_entered":"2013-09-12T18:09:00-04:00","date_modified":"2013-09-12T18:09:00-04:00","modified_user_id":"1","modified_by_name":"Administrator","created_by":"1","created_by_name":"Administrator","description":"","deleted":false,"assigned_user_id":"seed_jim_id","assigned_user_name":"Jim Brennan","team_count":"","team_name":[{"id":"East","name":"East","name_2":"","primary":true}],"linkedin":"","facebook":"","twitter":"","googleplus":"","account_type":"Customer","industry":"Electronics","annual_revenue":"","phone_fax":"","billing_address_street":"67321 West Siam St.","billing_address_street_2":"","billing_address_street_3":"","billing_address_street_4":"","billing_address_city":"Santa Fe","billing_address_state":"NY","billing_address_postalcode":"44150","billing_address_country":"USA","rating":"","phone_office":"(949) 400-8060","phone_alternate":"","website":"","ownership":"","employees":"","ticker_symbol":"","shipping_address_street":"67321 West Siam St.","shipping_address_street_2":"","shipping_address_street_3":"","shipping_address_street_4":"","shipping_address_city":"Santa Fe","shipping_address_state":"NY","shipping_address_postalcode":"44150","shipping_address_country":"USA","email":[{"email_address":"","invalid_email":false,"opt_out":false,"primary_address":true,"reply_to_address":false},{"email_address":"","invalid_email":false,"opt_out":false,"primary_address":false,"reply_to_address":false}],"email1":"","parent_id":"","sic_code":"","parent_name":"","email_opt_out":false,"invalid_email":false,"campaign_id":"","campaign_name":"","my_favorite":false,"_acl":{"fields":{}},"following":true,"_module":"Accounts"}

Tutorial: EDA with Redis

One of the more KISS – but less conventional – ways of implementing the above scheme would be to use Redis to store queues. The “SugarCRM/Requests” queue can be implemented as a “List” of “MessageId” values. The messages can be stored with “Hash” data types in the global namespace using the Redis “HMSET” command, where the headers can be individual keys. The unique “MessageId” values can be generated using the Redis “INCR” command. The “SugarCRM/Replies” queue can be implemented as a “Hash” data type as well, where  the “ReplyTo” can be used as the key and the “MessageId” as the value.


API first architecture or the fat vs thin server debate

API first architecture is an architecture that treats the API user as the primary user of the application. This means that API is not an alternative view in the MVC paradigm, but it has the highest priority. The main differentiators are that in “API first” the architecture enforces a complete, responsive, and well-documented API. This is especially important when targeting: mobile (Apps connect to the API), resellers (their presentation layer uses the API) and highly integrated, but decoupled, multi-product environments.

MVC reuse

The MVC architecture has been popular for a long time already. In 2004, it’s popularity skyrocketed when Ruby on Rails was released. MVC allows for high reuse in the case you have a front-end / back-end application (in the CMS sense), where customers use the front-end and employees use the back-end. This does require that you choose the same software stack for both the front-end and back-end, and make those applications as similar as possible. When the MVC strategy is executed properly, many parts of the application can be reused. Some of the parts that can be reused are: DBAL/ORM, Business Logic, Presentation and AAA. Specifically AAA (Authentication, Authorization and Accounting) can be reused by allowing employees to impersonate customers, use the same login screen and share logging facilities.

Mobile views for MVC

In 2007, Apple introduced the iPhone and from that time on the importance of web application (and websites) on small screens quickly grew. MVC applications were, and still are, very suited to facilitate small screens. All that is needed is a separate or adjusted set of views that is usable on a smartphone or tablet. The strategy of creating a single set of views that is suitable for mobile and also for desktop is called “mobile first“. This is the most cost-effective and radical approach, which requires strong leadership and decision taking, because all the software needs dramatic change, as all the views need to be adjusted. The alternative is to maintain two sets of views: one for mobile and one for desktop. The alternative views are often hosted on a “m.” subdomain. This is a simple and transparent approach.

Adding API to MVC

The dreadful “HTML5 vs Native” app development debate is going on right now and I quote Danny Brown:

Any company creating mobile apps today faces an important decision, Native or HTML5? Each one has its advantages, but choosing the wrong one could be costly. – Danny Brown

Choosing Native requires you to build a complete, responsive, and well-documented API, while choosing HTML5 requires you to redesign the views. There are arguments to defend either path and it depends on the situation what choice is best. There is one approach that will always fail: building an API as views on top of MVC. Let me explain why that fails and why so many people do it anyway.

Typically the (server-side) MVC approach leads to pages with 200 ms of load time. In this approach the server does three things: database abstraction, business logic, and presentation. This is why it is also referred to as “fat server”. An API is not responsible for presentation, executes smaller business logic per request and is therefor named “thin server“. A good API is highly optimized for speed and has typical load times under 20 ms. The means that when a mobile page is constructed there can be multiple (up to 10) calls made to render the full page within 300 ms.

Still when one took the MVC approach and is now lacking an API, the easiest thing to do is to add a few views that output JSON and call it a “RESTful API”. All you need to do is write some documentation and the boss will be happy. The fact that this API is totally unusable in real life, because it does not scale and is horribly slow, will only be noticed when the API is actually used and there is no way back.


Twitter & API first

In 2010, Twitter announced their “API first” strategy. They call this a JavaScript architecture, since they created a web application in JavaScript in a similar architecture as the mobile “apps”. This allows them to have full reuse of the API they build. Where initially the API was something “extra”, next to their web application, it then became the foundation of all other development. Their API is focused on delivering optimal integration for JavaScript programmers by using a RESTful JSON API. But they are also serving their application using traditional pages:

In order to support crawlers and users without JavaScript, we needed a rendering system that runs on both server and client. – Twitter

This approach of delivering traditional pages, while still using the “API first” strategy is what I call “Hybrid”. In the diagram below I’ve tried to enumerate the different approaches.



Optimal reuse brings down costs, but optimal reuse can only be achieved when there is a strong architectural strategy to follow. Refactoring code to increase architectural compliance does not bring value to the business directly. It will bring down the cost of change eventually, but the level of trust needed for these decisions is not easily gained.

Further reading

This is not the first post about “API first”. Check out the following links if you want to learn more about it:


I created a deck of 11 slides about “api first” for a presentation I gave on the 26th of November at LeaseWeb.