By now everyone would have at least heard of the longest match in Tennis history that was played at this year’s Wimbledon between John Isner and Nicolas Mahut. As unearthly as their will, effort, and stamina sounds, the match was also a very good example of an earthly peculiarity – ‘errors in software applications’.
Wimbledon Scoreboard (Image courtesy: Wikimedia Commons)
During that legendary final set, the stadium scoreboard went kaput after displaying ’47-47′. So the stunned spectators as well as the weary chair umpire had to rely only on manual records of the game score for the never-ending set.
Why? Well, nobody at IBM had thought that two supposedly sane individuals would ever reach that figure, let alone cross it! Not only that, even the online scoreboard reset itself after displaying ’50-50′! Were the programmers wrong? Now, that’s a tricky question.
For any amount of budget, there has to be a limit until where software applications, like automatic scoring systems, can be tested. You can either trust a wise man who magically comes up with certain values, or base your limits on certain empirical data, the fact is that these limits cannot be infinity. Software testers are paid their wages to put the application through the paces. Yes, there are fixed rules and conditions (test cases) to satisfy, but hardcore testers pride themselves in thinking of innovative conditions where the application might break. All this is added to the already exhaustive list of test scenarios, but how exhaustive can ‘exhaustive’ be! There will always be certain cases and non-conventional types of inputs where even the best pieces of softwares will crack.
The impact of this might not sound as catastrophic in a Tennis match scenario, but consider this happening to a near-perfect auto-pilot application in the aircraft you might be boarding during your business trip next week, or to a highly secure online-banking and money-transfer application which your bank uses to handle multi-million dollar transactions. The knowledge of some recent unsolved aircraft accidents and some computer hacking incidences is enough to realize the severity of the issue. A bit ruffled now, are you?
The point is not to get hysterical about the state of affairs, but to remain alert. Always look at the odds of these things happening, e.g. the ratio of air-accidents to the amount of aircraft taking off and landing on airports all over the world on any given day, or the ratio of banking-frauds to the total amount of money being handled online every hour. There are hundreds of thousands of people working in the software field which are dedicated towards producing secure and highly-reliable pieces of software, be it for the microwave oven in your kitchen or the International Space Station (ISS) flying many miles above you.
Many would remember the recent controversies and product recalls by the auto major Toyota regarding the software (embedded firmware) in its hi-tech cars. As more features are added to existing systems and software gets more and more complex, the chances of it behaving erratically even after months of thorough testing increases! Also, not every issue that occurs is easily reproducible back at the factory. Not everybody in this world is a software developer and so not everybody actually understands this fact. The thing we all have to remember before starting the blame-game when some untoward incident happens is that nothing can be 100% secure; that’s a myth. People work day and night to iron out any known (and imaginatively unknown) issues and test cases, which, let me tell you, can run into thousands. There is always scope for improvement which, like it or not, takes time. Neither can companies stop selling cars until that time, nor can airlines stop flying the aircraft in their arsenal until every possible glitch is resolved.
The 2035 edition of the Audi RSQ Sport Coupe (Image courtesy: PopularMechanics.com)
We humans aren’t perfect, then how can we expect the machines we make to be so; unless of course they are made by a Positronic brain, as mentioned in Issac Asimov‘s sci-fi classic ‘I, Robot‘!
The best analogy I can think of is that of a letter by Pete Docter, the director of the highly successful Pixar movies ‘Monsters, Inc.‘ (2001) and ‘Up‘ (2009), where he had quoted Pixar’s John Lasseter. He had said – “Pixar films don’t get finished, they just get released.” I think the same logic applies to the world of software applications!
Credo in between the lines 
good one… keep it up 🙂
Thanks Manoj 🙂
I’ve used software that was as near to perfect as it could possibly get based on my usage requirements. My hat goes off to the nameless individuals that went through the painstaking process of getting their software to that state. There is always room for improvement, but that is more about enhancement vice testing.
In my opinion, the belief that software cannot be 100% tested is the primary reason why people don’t bother trying to get to that state. It’s about the people and process, not about the time and money. As you point out, it’s the hardcore testers that get the job done.
The Pixar analogy is pretty good. I think in their case, not finished means not visually perfect. As with any art form, there will always be critics. Software either does what it was intended to do or it doesn’t. – it’s very black and white. If a user has a requirement outside of the intended functionality and specifications, it’s really a problem with the requirements, not the testing.
In the Wimbledon case, was the root cause of the problem the specifications or testing? That would be good to know.
Cheers
Thanks, and I’m glad that you liked the analogies! You raised a good point that sometimes good documentation of the requirements / specs can help software programmers, and in turn the software itself to get closer to that 100% mark. It does come down to imagination on the part of the programmers and testers – I mean, who would have thought that those guys would battle it out on the Tennis court to that extent!
What’s funny is that even then there are gray areas. In the Wimbledon example, the IBM guys said that the bug fix was only temporary and that the scoreboard would have failed again if they had gone on for another 25 games! So even after knowing exactly what they were up against, the software was just too darn complicated to be modified to pass the new testing criteria! 🙂
Your reasoning is perfect when the software is relatively manageable, and the test / operating conditions are well defined. How many times have we had highly critical errors in the operating systems we use on our computers! Once or twice a year maybe if you really play around with it. Similarly the firmware built into our TV sets and microwave ovens stays reliable for years. Those programs have clearly defined goals. Problems arise when the complexity increases and the operating conditions become unpredictable, like in airplanes flying through rough weather, or the software controlling the Large Hadron Collider for example. The ‘Toyota car recalls’ this year were due to similar reasons – there were just too many inputs and conditions to go through for the testing team! We can devise methods to get close to the magical number of 100%, but really it is an Alice-in-Wonderland scenario – you take two steps ahead and still find yourself in the same relative place!
To make the software 100% bug free is really depends on the scale and scope of the software. As you given the above Wimbledon example it might happened that they might have assumed the some cases. Also as I said the scale at which you are designing the software if it is very large scale you can not be always sure that it will be 100% bug free e.g Kalpana Chawla died in the Space Shuttle Columbia disaster (though its not the example of software failure) it was the fault in the design and problem seriousness was not considered . Same might have happened in the Wimbledon.
Yes, true. As software gets bigger and more complex, there is more chance of it behaving in an odd fashion for a certain input, or more chance of it being vulnerable to outside attack. We, builders and testers of such software only have the limits of our imagination to find out how well it performs in certain situations; but as was clearly evident in the Wimbledon scoreboard example, nobody had thought that two players (however good they might be) would play 50+ games in the final set! They built the software on this premise, and it stood the test of time for so many years, until that eventful day! 🙂