Feb 12

Since the last company I was with was sold two years ago I have been doing lots of things with different technologies and slowly putting together yet another Micro ISV (this will be my fourth) and as a truly independent one man show, sometime you go off looking for inspiration and what a better place to go to then the Ted Talksas there are many great and accomplished people offering their advice, experience and ultimately their passion for nothing other then for you to sit down and listen for like 20 minutes.  If you have never heard of, or listened to any of these talks then you are truly missing out on a great treasure of knowledge, experience and passion which is free for the taking.

Now the other day someone recommended that I listen to Clifford Stoll’s Ted Talk entitled ‘Clifford Stoll on ... everything’ and his name was familiar to me as I had read his books and while not totally agreeing with ‘Silcon Snake Oil’ I did found his different point of view enlightening.  Now before you watch it be warned that Cliff is somewhat like what you would expect in a cheap Hollywood movie about a whacked out old Astronomer who maybe spent a little too much time staring into space, he is hyper and what most people would consider ADHD but he makes an interesting statement about 1:18 into the presentation (among other interesting statements throughout his presentation, I really like the one about how to tell what the future will be like, and I like one side objects as well).

‘The first time you do something, its science, the second time, its engineering, the third time, its just being a technician’

Now I started life as a Mechanical Engineer, had a couple of oil patch jobs (well tester, field operator, production engineer, etc), and worked in a steel mill that made tubular goods (mostly casing for oil and gas wells) and then decided to go back to university and take computer science as I discovered I really like designing and creating software for other people to use.  Now over the last twenty years I think I have gotten pretty good at designing and creating software as every piece of software I have been involved with is still in use today and I’ve only failed once to deliver the product and that involved a corporate merger and it made perfect business sense to shutdown our project.  Now most of the software I’ve created was commercial in nature and has sold or generated almost a billion dollars and won a number of different awards etc, so I think I have some proof that I have at least some ability as a software scientist.

Now what Cliff said struck me as being very interesting as I believe that software design/development is still a science as often its the first time its been done, or at least done that way so at least its still mostly science.  You don’t write new software to replace an old system unless something new is needed.  Now sometimes its just a matter of porting the application as is to a new platform, and that isn’t science and really only requires a technician as ‘its been done many times before’.  Now what is different between the first time you do something and the second time, ie what is the difference between a scientist and and engineer?  I believe a scientist looks for the new or unexpected and an engineer tries to explain why and how it works.  When I worked in the steel mill I could look at the compositional analysis of the steel and I could tell you which pieces of equipment were going to have problems or work particularly well that run, as I knew what effect the different components in steel had at various levels.  For example if the sulphur content was high, I knew the continuous ERW welder wasn’t going to work very well and the pipe would likely fail because of a brittle weld (ie it would fail the flat test and also fail ultrasound with hook cracks etc in the weld), however the threaders and bevelers would do well as the higher sulphur content meant the steel cuttings would chip easily and reduce the build up of heat on the cutting tools and hence they would last longer.  So as an Engineer I knew the relationships between cause and effect and what the balance points were and why (the sulphur interfered with the slip planes in the lattice structure and impurity segregation of the steel increasing the brittleness of the steel, blah, blah, blah, blah).

Now when I see people claiming to be Software Engineers and suggesting different methodologies or technologies etc, I often wonder how well they know the relationships and causes and effects of what they are expounding.  I see various groups claiming the benefits of different frameworks, methodologies etc but yet they have no real data or evidence to back up their claims, now this bothers me as these don’t sound like the engineers I knew before as they sound far more like scientists who might have noticed something interesting, but they don’t really understand what they saw or why.  Now I often hear the excuse that there is no way to measure different causes and effects or compare different methodologies or frameworks etc, but I don’t believe that.  There have been various studies attempted and for the most part their findings are inconclusive, so either we need to try harder to find the answers or perhaps the answer is simply that some of this ‘stuff’ just doesn’t make that much difference and after all these years, I’m starting to believe that methodologies etc just don’t matter that much.  My experience over the years is the most important factor in creating good software is a good team of passionate people.  For example take a team of experienced, talented and passionate developers, and give them crappy tools etc and I believe they will still produce a better product then a crappy team using all the so called ‘cool’ tools and methodologies etc and I think most people would agree with this.

Now I’m concerned about the changes that are being made to the development process without understanding the reasons/benefits or cause and effect relationships.  For example I see various frameworks being extolled as being huge development aids, but these same frameworks five or ten years from now could become maintenance nightmares as we have already seen a number of ‘promoted’ frameworks die.  Often I’ve seen levels of abstraction get to the point that frankly its impossible to understand what the original problem was, think that could become a maintenance issue when someone other then the original programmer tries to modify that code, I certainly think so.  The software industry used to measure code defects as number of defects per x lines of code, now how much code is being saved or added by using some of these frameworks, often I think use of some of these frameworks substantially increases the number of lines of code in the product, which then typically increases the overall number of code defects in the product.  When companies tried to reduce the number of code defects per x lines of code via training, tools etc they typically found they quickly got into diminishing returns and ROI was limited.  Often I found a better strategy to reducing the overall number of defects was to reduce the number of lines of code as this tended to have all sorts of benefits, besides dropping the product defect counts, including improved future maintenance as it was easier to understand the code and what it was doing.  Now when considering the state of the development world I sometimes find myself thinking we are moving in the wrong direction and adding complexity to our code when perhaps moving to a minimalist approach to coding might be a better idea in the long run.

Now every good scientist knows processes, rules, guidelines etc for investigating their science and often these are made up as they go and learn, or experience feedback from their existing processes etc, and they adapt to changes and various influences so they can ultimately refine and sharpen their focus as they near their ‘discovery’.  Now this ‘way’ of the scientist sounds like so called modern development methodologies like Agile etc, so why do we call it ‘Engineering’ as that is a blatant mislabelling, as there really isn’t any engineering rigor.  I wonder if Cliff would agree?