Jan De Baere is an Agile Enterprise coach and well known international speaker. One of the subjects he often speaks about is estimation. He also specialises in the organisational & leadership part of business agility.
FT: Why did you give a recent talk on Sense and Nonsense of Estimates?
JDB: Well, I’m always very surprised when I start with new clients, not so much about the low quality of estimations, but basically about the awareness of the low quality of estimations. When I ask a client, “How good are your predictions and your estimations?” they say, “Well, not too bad,” and they estimate at about 80% of the stuff they promise that actually gets done. When you actually check it, it turns out to be between 20 and 30% of the things that they think they can do that they actually do, and they always say it’s just an exceptional period, but it turns out when you keep measuring, that it’s the norm.
FT: Where does your own knowledge come from on estimation techniques?
JDB: Well, I started to do some investigation together with Tom …… and then I followed the course on Harvard on PredictionX. So that has some background. Then we also used stuff from #NoEstimates movement from Vasco Duarte and the actual Agile Metrics from Daniel Vacanti was also a source of inspiration there.
FT: Do you think there is an evolution in the way activities are predicted?
JDB: Absolutely. So, what I learned from the PredictionX course is that there are basically three periods of time when it comes down to predictions and estimations, and the first period is about Omens, Oracles & Prophecies and basically what they did was they get some information from somewhere, it could be birds, it could be tarot cards, it could be basically anything. And then somebody integrates that information and then they use some predictive system and then a prediction comes out. That’s basically how it works. So you have some information, you have somebody with experience, he has a system and a prediction comes out. That’s basically how they did it in the ancient phase, so where you had things like astrology, haruspicy, Oracle of Delphi, I think we all know the tarot cards, tasseography and that kind of stuff. So these are the three periods.
FT: You mentioned haruspicy, what is that? How do you even pronounce it?
JDB: Haruspicy. It’s a very interesting one because it’s a technique that has been used for more than 1000 years. It’s described in a very high level of detail and basically what it is, they hear the question and then they pray to the Gods, the Gods would put signs in the liver or the lungs of sheep particularly, and then they would slaughter the sheep and then look at the signs that the Gods put in there. It is described in such a high detail that it’s almost scientifically how they did it and how they practiced it, and they did it for more than 1000 years. They knew there was a certain level of uncertainty in it, so when it was really important, they did twice the same thing and the outcome should have been twice the same, then it would be noted to be certain that the prediction was okay. Basically, that is what I still see in most estimates being done today. Project managers have techniques that they used for a couple of decades and then we ask some people some information, we do some hocus-pocus in Excel or MS Project, and then we make a prediction, end of story. So there’s the same level of making predictions as it was in the days of omens.
FT: You talked about the three points of predictions. So, what are the other two?
JDB: Yeah. Now we don’t use haruspicy or tarot cards or oracles anymore. We use a very simple technique and that’s the second period. We call it the feedback loop. When you make a prediction, you simply measure and evaluate the accuracy of your prediction and then you make a change to your predictive system. So you have a feedback loop. In other words, if you take estimations seriously, just measure the accuracy of your estimations as much as possible and have feedback loops as short as possible.
The third period is about computer models to do simulations, so we will not be talking about this one.
FT: What are other things that we should know about measuring and estimation?
JDB: Once you start measuring, there are a couple of things that you should know. Measuring works extremely well when you do it in order to improve yourself. Think about your smartwatch or activity tracker. It works brilliantly if you set an objective for yourself, and then you measure if you’re obtaining that objective. The moment you use measurements in order to control people, it loses all meaning because it will be gamed, so if you use measurements for control, it loses all its information value. That’s something that you should know.
FT: Can you give me an example on that?
JDB: If you want teams to work harder and you measure the speed of work being delivered, the speed will rise regardless of reality. In other words you will be gamed.
FT: So, can we predict everything?
JDB: The answer is no. Basically, what I refer to is the Cynefin framework. I will not go into full details but there are a couple of domains of work. One domain is the “obvious” domain. Here you have a direct link between action and result. That type of work is so straightforward that you don’t have to estimate but you monitor that work. For example running an automated script. Another domain of work is the complicated domain. This is the domain of the expert in a stable environment. A third domain is the complex domain. Here you typically have a dynamic context and things beyond your control. There is also a link between action and result but it only becomes clear in retrospect. In the complex domain you can also estimate but a better name is gambling. For example, some teams can predict how much work, or “output” they can deliver in a specific period. This makes that the work is in the complicated domain. What is not predictable is the “outcome” of their work, how much more sales you will have? The increase of website use by adding a new feature. How much more will our website be used? This is in the complex domain as we do not control the users.
FT: They’re a bit harder, for sure.
JDB: Yeah, and it leaves people confused, all these things, and they say everything is predictable, but we should know which one is complex, which one is complicated and which one is obvious. Complicated things have a stable predictability of more than 80%, obvious work more than 95%.
FT: Yeah, some things are harder to predict, yeah. What is the final message you wanted people to leave your talk with?
JDB: Well, basically there are 3 points I want to stress.
1) If predictions and estimations are important, start measuring. If you don’t do that, you might as well slaughter sheep.
2) If you measure, you should measure in order to improve yourself, not to control another.
3) The third thing is known, learn what is predictable and what is not predictable and by the way you can do that by measuring.