When (not why) “Agile” and especially Scrum are terrible

This article is a response to “Why ‘Agile’ and especially Scrum are terrible” as posted on Michael O. Church’s blog. He is not so much attacking the agile manifesto, as he he is attacking the reality of “agile”, the way it portraits itself and the excesses.

If you read the comments on Reddit then you find that there are two camps: scrum advocates and scrum haters. The haters complain how scrum leads to bad behavior and the advocates argue that this behavior is caused by doing scrum wrong. In this post I will focus on this behavior that, both parties agree, should be avoided.

1) Being judged personally

You should not be judged on hour by hour productivity as a software developer, because constant daily competition is not healthy. What particular tasks, technical debt or research tasks each developer worked on in a sprint is irrelevant. Peer pressure is the most effective way to keep people productive and looking at individual performance kills peer pressure. If there is micro-management going on, then that has nothing to do with being “Agile”.

Although there may be a high level of transparency, that does not mean that management should be evaluating people’s performance at the end of each sprint. The story points are designed to prevent management from doing this. The retrospective allows for criticizing the performance and effectiveness of people in the team by the team.

Especially web-based “Agile” registration tools must be used with care (or avoided). They are not part of “scrum” as white-boards and stickies are preferred. Also they may add lots of bureaucracy and may be misused to draw conclusions. Conclusion that may destroy the valuable trust relationship between employer and employee as they are more likely to be wrong than right.

If you see “agile” and “scrum” as tools to achieve “accountability” or “predictability” you are missing the point. It is merely a method to talk about work done instead of work to be done.

2) Developers being considered the lowest rank

The “product owners” and “scrum masters” do NOT outrank “team members”. Although they prioritize the user stories, they should not not choose which ones get executed. Also they should not assign tickets. If they feel superior or they are treated (or paid) as such, then something is wrong and not in the last place because good team members are all-round technical experts and these are extremely hard to find.

Team members are stakeholders as well. They can not only choose which tickets to work on (from the top of the stack), but also add stories (often technical debt) and argue why they are important.

One of the goals of agile is to eliminate blame and fear in the workplace. In a good team people do not fear to say “I was stuck with my code all day and got nothing done” during a stand-up. People should understand their pain and instead of blaming them offer some help.

3) Short term focus

Under Agile, technical debt can pile up not being addressed, because non-technical business people decide on priorities. They will not see a problem until it’s far too late or, at least, too expensive to fix it.

This pileup may be the result of  a straightforward application of “scrum/agile” principles. The focus may easily shift to only things that bring business value or “visible features” as I like to call them. These principles might work when imposed/implemented by an engineering team onto itself, but they break apart instantly the moment the management tries to impose them from outside.

There’s no reason why major refactoring can’t be tackled as a story. And having metrics to measure this (e.g. steadily increasing time to deliver features) should make it easier, not harder, to make the business case.

Conclusion

Short term work under high scrutiny is not fun nor agile, avoid it.

About Frank, our fantastic handyman

johnny-automatic-electric-drillAt home, we have a handyman. His name is Frank. He is very experienced and all-round and was able to take on every construction or repair job we gave him in the past years. Frank is easy going, never stressed out and often takes a break to smoke a cigarette and enjoy the weather. Although we pay him by the hour, we do not complain about his smoking as he asks a reasonable fee and works quite efficiently. Also, my wife really likes the guy, as she feels he can be trusted and always understands her really well.

For instance, last year he redid our kitchen, which turned out really nice. And although we wanted to go for a four ring gas hob he convinced us to take a five-ring hob with a central wok burner. He argued this would hardly take any extra space and would be very convenient in case we needed to use a bigger pan. Every time we have visitors over, my wife refers to Frank and how happy she is about the choice we made.

This year we were getting the bathroom done, but since we only have one shower in our apartment I asked Frank: “When will the bathroom be done?”. He answered “that depends”, “why are you asking?”. I told him about my concern of using the shower and he smiled. “I can make sure the shower can be used every time I leave as long as you don’t mind the mess.” he replied. Well… that solved one of my worries, but I was not completely satisfied, so I kept asking.

“But, when will the bathroom be done? I mean.. how much money is it going to cost?”. “Those are two different questions” Frank answered, “The first one depends on how often and how long I will be here and the second thing depends on what the bathroom should look like and what other more important jobs you have around the house”. “Hmmm… I understand”, I mumbled, but had the unpleasant feeling that he was dodging the question.

My wife has a very busy and responsible job, but she is also the one that decides on the interior related things in our house. Not that I do not care, but we both know that she has “feeling” for these things and I (an IT guy) do not. I told my wife that I could let Frank in and pour him his coffee, but that I did not see how we would organize this bathroom rebuild. She had an easy solution: She asked Frank to promise to work 3 days a week for at least 2 months. He worked on Monday, Tuesday and Thursday, since he had another job going on Wednesday and Friday. Every Thursday evening he would leave late, so he could meet my wife to show what he did and discuss with her what he was going to work on next week.

I would not have taken that route and would have probably asked an interior designer to draw the bathroom in 3D. Then I would have asked several building contractors to quote me with a price and a delivery date. I’m sure my wife would not have been as pleased as she is now with our new bathroom. The bathroom turned out exactly as we wanted, she even feels she has built it herself. Another good thing is that we only paid Frank for the work he actually did and the materials he needed. Of course not everything went flawlessly: Frank had to reroute a sewer pipe because we wanted the toilet in a different position. That was a lot of hassle, but my wife really wanted it like that, so it was our own choice in the end.

I’m glad I’m in IT doing agile projects, because construction work is not my cup-of-tea. 😉

Hey government! Agile software tenders please..

This is a follow-up on my Dutch article on Agile government software development tenders. I believe that the current tender strategy on software builds is causing a waste of (tax) money by our (Dutch) government.

Apparently governments think that bigger tenders lead to lower costs. They probably think that higher volume allows for lower margins. This is an economic rule and this means that it should be true for all industries, right? Not for software development. Big tenders lead to BDUF (Big Design Up Front), which means: less control, steering, validation and thus higher risk of failure.

If you calculate the risk of failure as part of the costs, then the simple economic rule “higher volume allows for lower margins” is no longer true (for software development).

So for the above misconception software development projects are made as big as possible. The government will try to describe what should be build in this huge contract. In the software industries these descriptions are called “specs” (or specifications). There can be hundreds of specs. The contractor has no other option than to “build to spec”, which means that even if the specification is wrong. he has to build it wrong, because the spec is part of the contract and this is agreed.

Specs fail, Agile does not

Even if the spec was written correctly in 2011 and the specs were finished by 2012 and then the project finished in 2014, it may be that the specs are no longer valid. This is what we call in software development a “moving target”. So now there are 3 options:

  1. Ignore that we are building something that is not needed anymore.
  2. Adjust the specs, but this means we have to renegotiate price and delivery time.
  3. Move the work to “extra work” category in which it will be charged an hourly fee.

Every deviation or unclarity in the original spec can be used to earn extra money or make the deadline fluid. Writing a spec is very hard. So hard that it is, in my opinion, impossible to write an unambiguous and good spec in a reasonable amount of time.

So as a government you have to choose between two evils. Either you hold the contractor to build something that is not useful or you change the spec. In the first case the project fails and in the second case you have to pay extra and/or accept that the deadlines shifts.

To make matters worse the government rules clearly state that the cheapest offer should be chosen. I believe that contractors understand the rules of the game and that is why they are underbidding. This leads to an even more stubborn contractor as he needs to “earn back” the money he “lost” to win the tender.

Not lean at all! Reduce waste now

This is a very ugly game that is played with our tax money. And in my opinion the government should apply what we have learned years ago in the product software business: work Agile. In Agile we only commit on effort and not on spec. We let a group of experts work for 2 weeks on a project, look at the result, steer the direction and continue.

Governments should make a few domain experts available for the project. These people should be seen as the “customer on-site”. The domain experts and the contractor winning the bid should use a SCRUM way of working to guarantee short feedback cycles.

Then one last thing: Software building should no longer be seen as projects, but should be considered products. If people work on software for a limited time, they are less committed to its quality and reduced complexity. Also the software that the government offers should be constantly changing as does our world, the government regulations and all other software around us.

Further reading:

  1. http://www.computerworld.co.nz/article/492220/agile_approach_may_clash_govt_tender_policy/
  2. http://www.computerweekly.com/news/2240187680/Government-tenders-for-agile-digital-developers
  3. http://diginomica.com/2014/04/22/uk-government-spending-millions-agile-development-coaching-training/

 

Agile ontwikkelen bij ICT aanbestedingen overheid?

Hieronder een (zo goed als) lopend verhaal over de ICT problematiek bij aanbestedingen door de Nederlandse overheid. Het bestaat uit een viertal quotes over hetzelfde onderwerp, maar uit verschillende bronnen. Ze geven een overzicht van de pijnpunten en pleiten tevens voor het gebruik van de “Agile” methode bij ICT aanbestedingen.

Bart van Oosterhout in Intermediair (2007):

Een op de vijf grote ict-projecten bij de overheid komt nooit van de grond, en dat kostte tot nu toe miljarden. De Tweede Kamer wil nu een parlementair onderzoek. Maar in de ict-sector weet men allang waarom het zo vaak misgaat. Over megalomanie, fantasy- deadlines en de wederzijdse wurggreep van de aanbesteding.

Sander Hulsman quote Marc Maas (QNH Consulting) in Computable op 05-09-2014:

Net als bij infrastructuur besteden overheden te grote percelen aan. Dit heeft verschillende effecten. Zo zorgt een grote dealwaarde voor grote belangen bij bedrijven om de deal binnen te halen. Natuurlijk zorgt dat voor de gewenste prijsdruk, maar zorgt ook voor neveneffecten. Zo wordt de deal te scherp aangeboden waarbij bedrijven genoodzaakt zijn om te sturen op meerwerk. Dit kunnen ze omdat de klant de aanbesteding te generiek uitvraagt, waardoor veel interpretatieruimte ontstaat. Deze wordt gebruikt door de leverancier. Dit zorgt voor misverstanden en frustratie.

Paul Weghorst schrijft in 2012 op scrum.nl:

Binnen het uitbesteden is een traditie gegroeid van uitgebreide specificaties opstellen, een aanbieding aan de markt vragen in een proces met minimale interactie, de goedkoopste laten winnen en vervolgens een contract opstellen met hoge boeteclausules. Of dit allemaal echt wordt voorgeschreven door de wetgever is nog maar de vraag, maar het is wel de standaard van vandaag. We weten ook dat dit zelden verloopt zoals het bedoeld is: gemiddeld 35% van de specificaties verandert, en het levert software op waarvan 65% niet of nauwelijks wordt gebruikt. Er wordt stevig geëscaleerd, de relatie komt flink onder druk gestaan, maar ondanks 9 jaar ervaring in zakendoen onder aanbestedingen, weet ik dat boetes zelden worden geëffectueerd (de opdrachtgever was zelf nooit van 100% onbesproken gedrag).

Michiel Steltman schrijft in zijn column op rijnstreekbusiness.nl in mei 2014 over Agile:

Die aanpak bleek wel goed te werken. Software bedrijven als Microsoft en Google weten dat ook ontwikkelen al sinds jaren ‘Agile’. Maar zo gaat dat niet bij onze overheid. Die blijft volharden in de illusie van voorspelbaarheid. En roept met de aanbestedingswet in de hand steeds opnieuw dezelfde ellende over zich af. De hoogste tijd dus om het over een andere boeg te gaan gooien. Laten we bij het volgende grote systeem nu eens alleen afspreken om met elkaar een paar duizend uur te gaan klussen aan een probleem. Met als doelstelling ‘we zien wel waar we uitkomen’.

Ik ben voor! Hoe dat moet? Neem een voorbeeld aan deze Agile aanbesteding in het Verenigd Koninkrijk van 40 miljoen Britse Pond.

NB: De ruim 40 developers van LeaseWeb gebruiken de “Agile” methode met veel succes voor de interne software development projecten.

Waterfall versus Agile – comparing software development methodologies

One of the first decisions you need to make as a project manager before starting a project implementation is: Which development methodology should I use? There are several methodologies you can use to manage a project. This blog contains reasons and information about two methodologies which can help you make this decision.

What is Waterfall?

waterfall

The Waterfall model is a sequential design process, often used in software development processes, in which progress is seen as flowing steadily downwards (like a waterfall) through the phases of Conception, Initiation, Analysis, Design, Construction, Testing, Production/Implementation, and Maintenance. This means that when each of the eight stages are completed the developers move on to the next step. Once a step has been completed developers can’t go back to a previous step without reviewing the whole project and starting from the beginning. There is few room for changes or errors, so a project outcome and an extensive plan must be set in the beginning and then followed carefully.

What is Agile?

agile

Instead of a sequential design process the Agile methodology follows an incremental team based approach. Developers start with a minimalistic project design, and then begin to work on small modules (sprints with a defined duration and a running list of deliverables planned one sprint in advance). The deliverables are prioritized by business value as determined by the customer. At the end of each sprint project priorities are evaluated and tests are run. These sprints allow bugs to be discovered in an early stage and customer feedback can easily be included into the design before the next sprint is run.

Comparison between Waterfall and Agile

In this picture you can see the differences between both methodologies at a glance.

Figure3

Image source: Waterfall versus Agile

Advantages and disadvantages of both methodologies

Advantages Disadvantages
Waterfall
  • The customer knows what to expect in size, costs and timeline
  • Minimal impact in case of employee turnover due to strong documentation
  • Developers and customers agree on what will be delivered up front
  • Progress is more easily measured, as the full scope is known in advance
  • Team members can work on different projects simultaneously
  • Customer presence isn’t strictly required after the requirements phase
  • Makes it easier when multiple software components must be designed for integration with external systems due to early known design
  • Once a step has been completed, developers can’t easily go back to a previous stage and make changes
  • The initial requirements can’t be changed without heavy impact
  • Testing the whole product at the end is risky due to finding bugs in early written code
  • A customer’s evolving needs are not taken into account. When changes are demanded this will impact the deadlines and budget
  • Possibility of a dissatisfied customer with their delivered software product due to not seeing the product until it is almost finished
Agile
  • Value is added every sprint
  • Easier to add features that are up-to-date with the latest industry developments
  • Project priorities are set before every sprint and evaluated after each sprint
  • Customer feedback is allowed and will contribute to the final end product
  • Testing each sprint (also by the customer) ensures no surprises at the end
  • High level of customer involvement (strong sense of ownership)
  • Short time to market: quickly produce a basic version of working software
  • Transparency is high
  • Deviations on the final product can arise due to no existence of a definitive plan
  • Customers need to have the time for the high degree of involvement otherwise the product will not match their requirements
  • Additional sprints could be added due to added requirements, because not all these requirements are clear during the start of the sprints, this may increase costs for the customer
  • The iterative nature may lead to a reduction in overall system quality, as there is less emphasis on understanding the finished system as a whole early in the project

Conclusion

After putting the information together, what can we conclude from all this? Both methodologies have their own strengths and weaknesses. The choice of methodology really depends on the goals you want to achieve. This all comes down to the context of a project. In fact, will it not be better to combine aspects from both methodologies to make the best possible software development process for the project?

When should you use the Waterfall methodology?

  1. When customers will not have the ability to change the scope of the project once it has begun
  2. When a clear image of what the final product should be must be worked out in advance
  3. When definition and quality are the keys to success instead of fast delivery

When should you use the Agile methodology?

  1. When customers need to be able to change the scope of the project
  2. When there isn’t a clear picture of what the final product should look like
  3. When quick production is more important than the quality of the product
  4. When the product is intended for an industry with fast changing standards, like software development
  5. When you have skilled developers available who are adaptable and able to think independently

Ultimately, although the way in which you do your work is important, most important is delivering a solid and supportable product that satisfies the customer’s needs!