Meta-programming: automate software creation

The secret to success in the business software field is to automate the software creation process. If you do so, you gain a big advantage as you have to program or customize less software than your competition. I have identified three types of meta-programming. They all have their advantages and disadvantages.

1) Higher subject abstraction

You can create software that sells “cars” that has, for instance, a table with “occasions”, containing columns for “brand” and “color”.

You can also make software that sells “products”, that have “properties”. Both could be tables, and there could be a foreign key between them. The properties table could contain the columns for “product” (a reference), “name”, “value” and “required”.

I see this pattern when I look at software that is not specialized for a niche. It is clear that generic web shop software like WooCommerce does not know what the web shop owner is going to sell and thus has no other choice than to generalize. It often also allows you to add “custom” fields, often with cryptic and generated names.

Is this bad? Yes and no. Yes, as this creates complex data structures that do not perform and are hard to understand. No, as you may have to do less software development. So in fact you pay your more effective software development by lower software performance and worse structured data.

2) Higher code abstraction

Programming language constructs like reflections, “generics” in C#, “method_missing” in Ruby, “magic methods” in PHP and “proxy classes” in Java enable programmers to write less code that is more generic or reusable.

Is this bad? Yes and no. Yes, when your code starts to get variables named “instance” and “class” it becomes hard to grasp what it is actually doing. No, as it may actually lead to less and more powerful code. In fact you are paying your more effective software development by less readable and harder to modify software.

The real problem is that this approach becomes really ugly when the software is aging. The code will start to contain lots of exceptions implemented as if/else statements in the wrong abstraction level.

3) Generating the code

When you generate code you can still add exceptions afterwards. Also you can still use specific, readable code and simple data structures for the domain you are automating. This actually gives you the best of both worlds: a ‘natural’ abstraction level for data and code, but still lower software development costs. Yes, it may give code duplication at first, but as soon as the code ages, you will see that the exceptions you will have to add actually justify this.

Another advantage of this method is that it can be applied in any programming language as no special language constructs are required.

Further reading

While you are at it, why not check out Rails 3 Generators: Scaffolding and MVC and Scaffolding for Rails Newbs? I’m sure you’ll love it!

Why speed is more important than scalability

Software developers creating web applications like to talk about scalability as if it is totally unrelated to computing efficiency. They also like to argue that abstractions are important. They will tell you that their DBAL, Router, View Egine, Dependency Injection and even ORM do not have that much overhead (only a little). They will argue that the learning curve pays off and that the performance loss is not that bad. In this post I’ll argue that page loading speed is more important than scalability (or pretty abstractions).

Orders of magnitude of speed on the web

Just to get an idea of speed, I tried to search for a web action for every order of magnitude:

  • 0.1 ms – A simple database lookup from memory
  • 1 ms – Serving small static content from RAM
  • 10 ms – An very simple API that only does a DB lookup
  • 100 ms – A complex page load that calls multiple APIs
  • 1000 ms – Nothing should take this long… 🙂

But I’m sure that your website has pages that take a full second or more (even this site has). How can that be?

CPUs and web farms

Most severs have 1 to 4 CPUs. Each CPU has 2 to 32 cores. If you do a single web request, then you are using (at most) a single core of a single CPU. Maybe that is why people say that page loading speed is irrelevant. If you have more visitors, then they will use other cores or even other CPUs. This is true, but what if you have more concurrent requests than visitors? You can simply add machines and configure a web farm, as most people do.

At some point you may have 16 servers running your popular web application with page loads that average at 300 ms. If you could bring down the page load time to 20 ms, you could run this on a single box! Imagine how much simpler that is! The “one big box” strategy is also called the “big iron” strategy. Often programmers are not very careful with resources. This is because a software developers tend to aim for beautiful abstractions and not for fast software.

Programming languages matter

Only when hardware enthusiasts and software developers work together you may get efficient software. Nice frameworks may have to be removed. Toys like Dependency Injection that mainly bring theoretical value may have to be sacrificed. Also languages need to be chosen for execution speed. Languages that typically score good are: C, C# (Mono), Go and even Java (OpenJDK). Languages that typically score very bad: PHP, Python, Ruby and Perl (source: benchmarksgame). This is understandable, as these are all interpreted languages. But it should be considered when building a web application.

Stop using web application frameworks

But even if you use an interpreted language (which may be 10x slower), then you can still have good performance. Unless of course you build applications consisting of tens of thousands of lines of code that all need to be loaded. Normally people would not do that as it would take too much time to write. (warning: sarcasm) In order to be able to fail – against all odds – people have created frameworks. These are tens of thousands of lines of code that you can install on your server. With it your small and lean application will still be slow (/sarcasm). I think it is fair to say that frameworks will make your web application typically 10-100x slower.

Now let that be exactly the approach that is seen in the industry as “best practice”. Unbelievable right?

5 reasons your application is slow

If you are on shared hardware (VM or shared webhosting), then you need to fix that first. I’m sure switching to dedicated hardware will give you a better (and more consistent) performance pattern. Still, each of the following problems may give you an order of a magnitude of speed decrease:

  1. Not enough RAM
  2. No SSD disks (or wrong controllers)
  3. Using an interpreted programming language
  4. A bloated web framework
  5. Not using Memcache where possible

How many of these apply to you? Let me know in the comments.

Finally

This post will probably be considered offending by programmers that like VMs, frameworks and interpreted languages. Please, don’t use that negative energy in the comments. Use it to “Go” and try Gorilla, I dare you! It is not a framework and it is fast, very fast! It will not only going to be interesting and a lot of fun, it may also change your mind about this article.

NB: Go is almost as fast as the highly optimized C code of Nginx (about 10-100x faster than PHP).

Hey government! Agile software tenders please..

This is a follow-up on my Dutch article on Agile government software development tenders. I believe that the current tender strategy on software builds is causing a waste of (tax) money by our (Dutch) government.

Apparently governments think that bigger tenders lead to lower costs. They probably think that higher volume allows for lower margins. This is an economic rule and this means that it should be true for all industries, right? Not for software development. Big tenders lead to BDUF (Big Design Up Front), which means: less control, steering, validation and thus higher risk of failure.

If you calculate the risk of failure as part of the costs, then the simple economic rule “higher volume allows for lower margins” is no longer true (for software development).

So for the above misconception software development projects are made as big as possible. The government will try to describe what should be build in this huge contract. In the software industries these descriptions are called “specs” (or specifications). There can be hundreds of specs. The contractor has no other option than to “build to spec”, which means that even if the specification is wrong. he has to build it wrong, because the spec is part of the contract and this is agreed.

Specs fail, Agile does not

Even if the spec was written correctly in 2011 and the specs were finished by 2012 and then the project finished in 2014, it may be that the specs are no longer valid. This is what we call in software development a “moving target”. So now there are 3 options:

  1. Ignore that we are building something that is not needed anymore.
  2. Adjust the specs, but this means we have to renegotiate price and delivery time.
  3. Move the work to “extra work” category in which it will be charged an hourly fee.

Every deviation or unclarity in the original spec can be used to earn extra money or make the deadline fluid. Writing a spec is very hard. So hard that it is, in my opinion, impossible to write an unambiguous and good spec in a reasonable amount of time.

So as a government you have to choose between two evils. Either you hold the contractor to build something that is not useful or you change the spec. In the first case the project fails and in the second case you have to pay extra and/or accept that the deadlines shifts.

To make matters worse the government rules clearly state that the cheapest offer should be chosen. I believe that contractors understand the rules of the game and that is why they are underbidding. This leads to an even more stubborn contractor as he needs to “earn back” the money he “lost” to win the tender.

Not lean at all! Reduce waste now

This is a very ugly game that is played with our tax money. And in my opinion the government should apply what we have learned years ago in the product software business: work Agile. In Agile we only commit on effort and not on spec. We let a group of experts work for 2 weeks on a project, look at the result, steer the direction and continue.

Governments should make a few domain experts available for the project. These people should be seen as the “customer on-site”. The domain experts and the contractor winning the bid should use a SCRUM way of working to guarantee short feedback cycles.

Then one last thing: Software building should no longer be seen as projects, but should be considered products. If people work on software for a limited time, they are less committed to its quality and reduced complexity. Also the software that the government offers should be constantly changing as does our world, the government regulations and all other software around us.

Further reading:

  1. http://www.computerworld.co.nz/article/492220/agile_approach_may_clash_govt_tender_policy/
  2. http://www.computerweekly.com/news/2240187680/Government-tenders-for-agile-digital-developers
  3. http://diginomica.com/2014/04/22/uk-government-spending-millions-agile-development-coaching-training/

 

Agile ontwikkelen bij ICT aanbestedingen overheid?

Hieronder een (zo goed als) lopend verhaal over de ICT problematiek bij aanbestedingen door de Nederlandse overheid. Het bestaat uit een viertal quotes over hetzelfde onderwerp, maar uit verschillende bronnen. Ze geven een overzicht van de pijnpunten en pleiten tevens voor het gebruik van de “Agile” methode bij ICT aanbestedingen.

Bart van Oosterhout in Intermediair (2007):

Een op de vijf grote ict-projecten bij de overheid komt nooit van de grond, en dat kostte tot nu toe miljarden. De Tweede Kamer wil nu een parlementair onderzoek. Maar in de ict-sector weet men allang waarom het zo vaak misgaat. Over megalomanie, fantasy- deadlines en de wederzijdse wurggreep van de aanbesteding.

Sander Hulsman quote Marc Maas (QNH Consulting) in Computable op 05-09-2014:

Net als bij infrastructuur besteden overheden te grote percelen aan. Dit heeft verschillende effecten. Zo zorgt een grote dealwaarde voor grote belangen bij bedrijven om de deal binnen te halen. Natuurlijk zorgt dat voor de gewenste prijsdruk, maar zorgt ook voor neveneffecten. Zo wordt de deal te scherp aangeboden waarbij bedrijven genoodzaakt zijn om te sturen op meerwerk. Dit kunnen ze omdat de klant de aanbesteding te generiek uitvraagt, waardoor veel interpretatieruimte ontstaat. Deze wordt gebruikt door de leverancier. Dit zorgt voor misverstanden en frustratie.

Paul Weghorst schrijft in 2012 op scrum.nl:

Binnen het uitbesteden is een traditie gegroeid van uitgebreide specificaties opstellen, een aanbieding aan de markt vragen in een proces met minimale interactie, de goedkoopste laten winnen en vervolgens een contract opstellen met hoge boeteclausules. Of dit allemaal echt wordt voorgeschreven door de wetgever is nog maar de vraag, maar het is wel de standaard van vandaag. We weten ook dat dit zelden verloopt zoals het bedoeld is: gemiddeld 35% van de specificaties verandert, en het levert software op waarvan 65% niet of nauwelijks wordt gebruikt. Er wordt stevig geëscaleerd, de relatie komt flink onder druk gestaan, maar ondanks 9 jaar ervaring in zakendoen onder aanbestedingen, weet ik dat boetes zelden worden geëffectueerd (de opdrachtgever was zelf nooit van 100% onbesproken gedrag).

Michiel Steltman schrijft in zijn column op rijnstreekbusiness.nl in mei 2014 over Agile:

Die aanpak bleek wel goed te werken. Software bedrijven als Microsoft en Google weten dat ook ontwikkelen al sinds jaren ‘Agile’. Maar zo gaat dat niet bij onze overheid. Die blijft volharden in de illusie van voorspelbaarheid. En roept met de aanbestedingswet in de hand steeds opnieuw dezelfde ellende over zich af. De hoogste tijd dus om het over een andere boeg te gaan gooien. Laten we bij het volgende grote systeem nu eens alleen afspreken om met elkaar een paar duizend uur te gaan klussen aan een probleem. Met als doelstelling ‘we zien wel waar we uitkomen’.

Ik ben voor! Hoe dat moet? Neem een voorbeeld aan deze Agile aanbesteding in het Verenigd Koninkrijk van 40 miljoen Britse Pond.

NB: De ruim 40 developers van LeaseWeb gebruiken de “Agile” methode met veel succes voor de interne software development projecten.

Building software is like building a LEGO house

One of the problems of the software development industry, is non-software people making decisions about software builds. Many of these people believe that building software is like building houses, which it is not. People see the software business, approach it like building a house, specifying upfront and asking when it is finished. Assigning project managers as general contractors and hiring cheap labor to execute the build. I can hear some managers who are reading this think: Wow! Can we do that? That is the answer! No, it is not, we tried it for more than a decade and it failed spectacularly. They even came up with a name for it: “out-sourcing”.

But even though we move from Big Design Up Front (BDUF) to Agile the analogy of building houses is hard to get rid of. Yes, we know we do Agile with sprints, no deadlines just sprint goals and a backlog with user stories and a road-map with epics. But we are still building a house. Well.. no.. we are actually not building a house. The whole house analogy is wrong. We are building a LEGO house. Yes, something like this:

lego-house

Was it a coincidence that Paul Hammant used a LEGO picture in his post about software architecture? I think not. Here are some of the points that show why building LEGO house is a good analogy for building product software:

  • Innovation: Best builds come from experimenting constantly and thinking out-of-the-box.
  • Flexible: LEGO house foundations can be replaced carefully. Anything can be replaced any time.
  • Looks: Building a pretty LEGO house is very hard and requires special skills.
  • Agility: There is no such thing as a “done” LEGO house. It is alive and constantly changing.
  • Requirements: You don’t plan building a LEGO house. You know what you want when it evolves.
  • Specs: LEGO is enjoyed best by building whatever comes to mind, not by following specs.
  • Compatibility: Everything connects to everything, with hardly any effort.
  • Freedom: LEGO can be build without having to comply with all kinds of regulation.
  • Creative joy: Creating software is as much fun as creating with LEGO.
  • Value: People pay for the promise of easy integration and high flexibility.
  • Validation: A real-life test by the target audience tells you things you could not imagine.
  • Mindset: Grown-ups do not understand LEGO, like business does not understand software development.
  • Out-sourcing: If somebody else builds your LEGO house, you will get something you don’t like.
  • Standards: Build with non-standard blocks and you’ll get stuck quickly.

Can you think of other things in which the analogy is right? Use the comments! Don’t comment saying that I do not take software development serious. I believe that software development is a serious business, but it is also seriously fun. To put it stronger: If it does not remind you of playing with LEGO when you were still a kid, you are doing it wrong.