11 June 2021

Thorny Weeds of Data Science

In industry, the area of data science is a bit like navigating through a large field of thorny weeds. There are just too many people pitching themselves as experts, that don't understand what they are doing. Many of them with Phd backgrounds who have the complete inability to translate theory into practice. The field is a breeding ground of insecure people in teams with pure academic backgrounds. For everything, they require help, additional resources, adding to the frustration of their peers but also to the management who have to support their work with unnecessary inefficiencies and extensive funding. The patterns of recruitment or lack thereof seem to be typical across many organizations. Often these false hopes of hiring Phd individuals to lead research work in business seems to stem from clueless investors who neither have the interest nor the sense to understand the practical aspects of the field. Interview rounds for candidates becomes a foregone conclusion of misplaced, inept, and aberrant hiring. As a result, the entire organization becomes guilty of compounding issues in hiring incapable, pretentious, and arrogant individuals, lacking basic common sense, who may apparently have steller academic credentials. In many cases, it is questionable on the merit of a Phd qualification, where much of it may have been gained via crowdsourcing platforms or even via the extensive help of the academic advisor for the thesis write-up. Most things that a Phd caliber individual can do, a non-Phd individual can do it better and translate it into a product. If a Phd individual cannot convert theory into practice then what is the point of such a hire? Considering the fact, only one percent of the population has a Phd, is it any wonder why organizations are so ill-informed about calling it a skills shortage when there isn't any to begin with and should really focus on correcting their idealized job requirements. Invariably, organizations learn the hard way when projects fail to deliver and there is no tangible return on investment from a Phd hire. Is this an evidence of a failure of the education system, of the entire technology industry, or perhaps both. Flawed online training courses, mentoring, and certification courses further amplify this ineffective practice. Bad code and models, just breeds bad products to the end-user which ultimately effects investment returns from lack of business performance where targets need to be offset through additional end-user support plans. It still stands to reason, that for any company, people are the biggest asset. Hiring the right people and looking beyond the fold of their credentials is paramount. In fact, hiring astute generalists is far more important than hiring specialists in the long-term. A specialist in a certain area, at a given point in time, is likely to be an outdated specialist in the short-term to long-term cycle of work. Organizations that function as businesses need to strategize their game plan and forecast for the future, which may just be a few quarterly cycles ahead of time. The quickest way to failed startup is to increase costs by hiring Phds and increasing head count of staff to support their work. One needs to wonder why hire so many people to support someone with a Phd unless they are practically incompetent. Academics invariably cannot translate into practice which impacts delivery cycles of work. Phds as a result become the weakest link in many cases, inhibiting and hampering the cycle of productivity and innovation both for the short-term as well as long-term growth of an organization.