Pseudo scientific Software Engineering

Edsger Dijkstra didn't like software engineers:

The required techniques of effective reasoning are pretty formal, but as long as programming is done by people that don't master them, the software crisis will remain with us and will be considered an incurable disease. And you know what incurable diseases do: they invite the quacks and charlatans in, who in this case take the form of Software Engineering gurus.

That smarts. A lot. Not least because whenever I'm asked what I do for a living I reply that I'm a “Software Engineer”.

Dijkstra is right of course. Software Engineering is pseudo science¹ [1Dijkstra didn't use the term pseudo science, but that is what it appeared as on Reddit.] rather than science proper in all the ways that Karl Popper meant when he first described the difference. And yes, software engineers are quacks compared to computer scientists. But that's OK because we're not doing the same job.

I think there really are two closely related disciplines. Computer Science is formal methods and proofs as Dijkstra described. Its use is where the specifications are known. Software Engineering isn't helped by its misnomer, because it isn't engineering at all. It's about communication and discovery.

Software Engineering techniques attempt to put working code in front of somebody as quickly as possible in order to try to answer the question “is this what you meant?” It works from a basis of unknown and un-knowable specifications. As a software engineer you cannot prove that your software meets its specification because the first specification that you get which you could hope to try to prove is when you have working software in front of people who are happy using it — you'll know when you're there because it'll be at the end of a successful project.

Part of the confusion comes because different parts of the same project have each of these two different perspectives. There are algorithmically challenging parts of any real world application that certainly need to be done using the approach Dijkstra advocates, but there are also parts where the maxim about the definition of porn² [2“I know it when I see it”.] is far more illuminating to show us the right approach when trying to understand user' needs. Of course it's often the same developers dealing with both aspects.

This is why software engineering techniques like Agile try to get something — anything — in front of the customer as early as possible³ [3We tried to get to understand the users enough to be able to work to the level of formal proofs. The approach is called “waterfall” and it was shown not to be very good in practice. No matter how much the software conformed to its specification, it still didn't do what was actually needed.]. The truth is that despite being the only species on the planet that holds meetings we're not actually very articulate about anything we can't point to, and often not even then.

The task of the software engineer is to give the user something to point to so that we can give them the best chance to tell us what they really want and need. Popper's notion of “prediction” to turn it into science doesn't work because there isn't anything formal enough to predict until the project is over.

The whole practice revolves around the software engineer's empathy to the needs of the user (something we're generally famously lacking) producing the right result. Software engineers know that they're not going to get paid to learn what they have to produce separately form producing it, so they have developed techniques to allow them to deliver the best quality software possible at the same time that they finally understand the requirements.

Software engineering is pseudo science and rightly so. It will continue to be so right up until the point when users and businesses can give us software specifications with a set of axioms we can use to derive our proofs from. Right now we're lucky if we get a list of conjectures to work from.