29 July 2012

Over Engineering Is Killing Creative Software

Is Agile good? Is it creating over emphasis on code quality and testing? Are they necessary evil or a balance needs to be struck somewhere? In a lot of teams TDD (Test Driven Development) and the transgression to BDD (Behavior Driven Development) has become a dominate way of development methodologies. But, are they really the only way? Should they really be what determines whether code meets the requirements? Shouldn't it be all about making sure you know your requirements and domain model first before you test? I feel pragmatism is really the best policy towards development. Agile approaches at times impinge on the flexibility that is so well required for creativity and innovation within applications development. Agile it seems has become a major factor in getting things out quickly where documentation is left behind and testing has become more of a factor. At times, the whole process just reeks in bugs. One can never be fully assured that no issues will arise once the code is released into production no matter how much unit, functional, integration, load, acceptance, continuous testing is done during the development, test, and stage phases. In continuous delivery perhaps one would not even require so many phases further reducing testing stage cycles but still inducing the risk of bug creep. As a person from a Computer Science background, I feel Software Engineering should not be what drives project development but what inspires better coding standards. In end, Software Engineering is a sub-field that empowers Computer Science providing the necessary skills towards developing in the large. However, in the ever changing landscape of the web and the increasing data processing requirements it is really the principles of Computer Science that pave the way towards extending and envisioning what could not be possible before. 

It feels digressing everything solely on grounds of methods-driven development expounded from Software Engineering makes the process a very uninteresting and tedious at times. People who study Computer Science by the very nature are interested in solving problems, finding algorithms that could do things better, or making solutions happen naturally that most people could not conceive of before. In process, Computer Science in all its endeavors allows one to build on the research of giants to further the potential of computers and systems as a whole. Software Engineering delivers processes, at times, more processes than is necessary and arching over Computer Science like a god father. There is a difference in the way a Software Engineer thinks about a problem and how a Software Developer thinks about a problem. There is also a difference in the way they go about finding a solution. The underlining difference arises from their background whether it be in Computer Science or Software Engineering. Through past experiences I have found that  at times, teams with a lot of people with Software Engineering backgrounds tend to focus more on processes whether they be testing, management, code release and what have you. Whereas, people in development teams coming from Computer Science background seem to approach the development from a holistic approach defined either by the optimization of an algorithm to pragmatic use of software tools. It is here where creativity is nourished and where innovation reeks interesting outcomes. Artificial Intelligence is a sub-field of Computer Science that over the years has become a major contributing factor to the Intelligent Web per se and to the derivation of information from the huge amount of data for which the term 'Big Data' has become synonymous.

I feel the way to go with development of projects is a pragmatic one. Keeping a balance on level of testing, project management, and the maintenance of code quality. Especially, against time pressures, many project teams can not support so much over engineering approaches to exist unless forced upon by architects at times. It seems only natural that people move further a field from Agile to more lean and second method phases beyond just the use of TDD or BDD. Flexibility is important and clearly from the development trend it can be seen that the old Waterfall cycle was a major contributing factor to movement away and towards developing better ways of approaching projects. However, one cannot always apply TDD to everything, and Agile at times has problems making sure teams maintain sufficient documentation. Also, developments too inclined to use Agile methods and tools end up leaving behind the customer in the process - pretty much losing the gist of what Agile was all about, test first and often, and deliver so the customer is always in the loop of work progress and acceptance. Open source frameworks and tools have allowed a lot flexibility for code reuse and rapid development of user cases in project delivery cycles. The trend is likely to emerge in more open source projects and support of such projects by industry. Even to factor in that so many open source frameworks and tools have only just been developed for development which makes it even more possible for teams to develop on time and check for correctness. And, above all else it has provided a way to reuse other peoples' code without reinventing the wheel of creativity - building better and productive code that matters. 

Agile in all its glory is a nice approach with a set of methods whether they be Scrum, TDD, or BDD are good to have in the mind set. However, use of them should be approached from what is the best way to go about a problem. Just like the use of design patterns, not all can be applied to all problems, all the time. Creativity and innovation is what makes even such methods and tools possible. So, killing the mind set with only accepting one way of doing something is like coming in the way of progress. Open source is all about freedom to create and collaborate. A balance in process-driven attrition and freedom to innovate, create and empower development is the best policy.