2. First step of the software development journey: get prepared
My previous article was trying to stimulate interest in software development, especially to scientists. The end goal is to use the enormous computing power available like you use your hands or your brain. To use the large amount of cloud servers as an extension of your thoughts. For those convinced, now it’s time to explore why most of us fail to use this great potential.
The Complexity of Software
One of the best known developers at Microsoft, Scott Hanselman, warns us to Stop saying learning to code is easy:
When we tell folks — kids or otherwise — that programming is easy, what will they think when it gets difficult? And it will get difficult. That’s where people find themselves saying “well, I guess I’m not wired for coding. It’s just not for me.” — Scott Hanselman
I think every magic formula, every great principle and all senior guidelines you will follow in software will lead you in some deep troubles. Damn if you do, damn if you don’t.
Let’s be specific, most developers nowadays will tell you they practice TDD, in short for Test Driven Development. In their bible: Test Driven Development By Example from Kent Beck, the founder of this technique and one of the greatest developers alive these days, at the end (Appendix II), Beck illustrates the power of its principles through a recursive routine to calculate the Fibonacci sequence. But the algorithm he crafted with TDD is known to be the absolute worst way of calculating it: this program can literally paralyze any powerful computer if we ask too great a number to the function proposed by Beck…
We don’t know why he chose this example and neither if Beck was aware of this flaw. Nevertheless, it’s a great lesson: even with great principles, great care and multiple reviews from many great engineers, we can be absolutely wrong.
The manager of the first commercial operating system and computer on the market: the IBM System 360 put it this way:
plan to throw one away
Despite all the best practices, Brooks warns us that we shouldn’t be surprised to have to throw away our first work. There is No Silver Bullet...
Don’t get me wrong, I don’t want to frighten people, I want you to be prepared, to understand the context and the history of your future work, of the lines of codes you are about to write.
At the root of information technology: the Sumerians
So, why IT becomes hard? First, IT, as we all know, is an information technology communicated by a system of writing: programming languages (Python, C#, Ruby, etc.), domain languages (express in programming libraries or truly new languages built on top of programming languages) and data languages (Datagrams, JSON, etc.) to speak respectively of processes, domains and evidence. This system is there to organize information for our needs. And not the reverse: we create languages, and on top of them, domain languages to organize what we can’t express with the current ones available. Solving a new problem in software is always improving our system of writing, sort of…
The Sumerian invents writing in the same way: they needed ways to represent all debts, taxes and wealth everybody has or owe to everybody else. The anthropologist Jared Diamond explains this in its famous Guns, Germs, and Steel:
The independent invention that we can trace in greatest detail is history’s oldest writing system, Sumerian cuneiform. For thousands of years before it jelled, people in some farming villages of the Fertile Crescent had been using clay tokens of various simple shapes for accounting purposes, such as recording numbers of sheep and amounts of grain. In the last centuries before 3000 B.C., developments in accounting technology, format, and signs rapidly led to the first system of writing.
— Jared Diamond
Diamond emphasizes that creating a system of writing is incredibly hard when no one is existing:
INVENTING A WRITING system from scratch must have been incomparably more difficult than borrowing and adapting one. (…) For example, they had to figure out how to decompose a continuous utterance into speech units, regardless of whether those units were taken as words, syllables, or phonemes. They had to learn to recognize the same sound or speech unit through all our normal variations in speech volume, pitch, speed, emphasis, phrase grouping, and individual idiosyncrasies of pronunciation. (…) Probably all other peoples who have developed writing since then have borrowed, adapted, or at least been inspired by existing systems.
All this for accounting… The same is true for IT: we essentially built first the Assembly Language, then the C language on top of it, then most of all the other languages you know are built on top of the C.
The Sumerian discovered then many problems doing their accounting: more and more folks started to use their system to hold their debts and taxes. They needed a system to find the right information when desired: a sort of librarianship. The Sumerian could realize then, they can’t just write the name of one to hold him accountable, they maybe also need his address to distinguish him from someone else or they need to know his relatives to transfer his debt in case he disappears. This address should be written in another system, maybe a land tenure, and so on. In software, we call this a scalability problem. These kinds of issues arise when more and more data is poured into the system of information.
Pouring more information on the system implies that more people have to communicate and synchronized themselves. If each new person has to communicate with all the people already working on it, the communication problem growth quadratically (n², where n is the number of people). Brooks famous formula expressed it fabulously: what one programmer can do in one month, two programmers can do in two months.
You can now maybe feel why IT becomes hard. It’s not necessarily hard at the beginning, scalability makes it hard. And that’s why many IT projects derails.
Embracing Failure
Only those who dare to fail greatly can ever achieve greatly.
— Robert F. Kennedy
One of my biggest and scariest failure happened when I was employed at Clear Destination, a wonderful operation research software firm. We had a powerful optimizer that plan routes for home deliveries of big items (furniture, appliances, etc.). But each delivery client firm had to promote an “expert” to operate the optimizer which sometimes leads to uneven results. What if we could replace all these people by an artificial dispatcher that take cares of providing an optimal solution without human intervention? This was my first truly big task in my career. I tried hard to succeed: I learned concurrent programming, microservices architecture, data access patterns, automated tests in order to achieve this, but I ran out of time and I deliver only a fragile prototype. Nevertheless, I know how to design and deliver such a system now; plan to throw one away…
Fred Brooks failed at levels not bearable for someone like me: he made a 100 million dollars mistake during his time at IBM. And 100 million in 1970 was a lot of money! But, I think we could say, he is a legend now.
Lastly, software development isn’t school: in school you learn to fear failure, now you must embrace it, look for it. This is how we learn to learn, the fundamental next step in the software journey; if you are interested in my next article.
Conclusion
I hope this article gives you the courage to not giving up. If you liked it, don’t forget to applaud and follow me.