Big Data Talent Crunch

Video On-Demand
December 18, 2015
September 2016 Newsletter
September 25, 2016

Big Data Talent Crunch

Big Data Vacancy Image


Fast paced growth and reputation of Big Data becoming a main stream technology in under 5 years caught few people by surprise.  However, tepid growth of big data talent in comparison to avalanche of big data was an unpleasant surprise and a hard reality.

McKinsey Global Institute predicted that “By 2018, United States alone could face a shortage of 140,000 to 190,000 people with deep analytical skills as well as 1.5 million managers and analysts with the know-how to use the analysis of big data to make effective decisions….. a 50% to 60% gap between supply and requisite demand of deep analytic talent. ”  The report highlights that these positions will be unfilled due to a lack of applicants with expertise and experience.


How the organizations are bracing for big data avalanche and ominous warning of resource shortage?  For many organizations large and small, bigger challenge than the  exponential growth of big data is the lack of resources to harness the power of big data solutions.  Scary situation facing many organizations, private and public sector, is inadequate talent pool in big data domain.  Rampant shortage of big data resources is a serious hindrance to organizational ability to mature analytics practice and is stifling organizational ability to fully harvest benefits.

This problem is even more acute outside of silicon valley or  wall street, in particular, in the Midwest region.  While top talent flock to high flying west coast or Wall street powered lucrative east coast, Midwest is in tough spot to attract this talent. Multidisciplinary knowledge and skill set required for big data specialists is making the situation even worse.

The shortage is already being felt across a broad spectrum of industries, including retail, banking & finance,  insurance, pharmaceuticals, and transportation.  The supply and demand imbalance in big data talent is hurting organizations to the point that talent discovery emerged as a bigger problem for organization than data discovery. Shortage is in crisis mode and organizations are hitting panic button.


How does an organization recognize that it is impacted by big data talent crunch?  Few obvious signs include IT leadership struggles to kick start big data initiative or fail to deploy to production in reasonable timeframe, dismal project completion rate, business analyst dissatisfaction about unstructured data analysis, lagging behind competition in big insights, etc.  Similarly, on HR front,  positions for big data experts and data scientists remain vacant for extended period that HR already gave up on these Reqs.

If half of the above mentioned symptoms are applicable to an organization, then it is safe for the organization ask itself whether to stay ahead of the curve by employing appropriate measures or maintain status quo and eventually press panic button.


How to Solve Big Data Resource challenge? Pain of big data talent shortage is felt by far too many organizations, so it stands to reason that unique ways of finding that talent would come to the forefront.  Situation could be different from industry to industry and organization to organization. However, some common themes that help address are described below.

  • Systematic and strategic approach that include a combination of near-term and longer-term strategies. Traditional ways of assigning resources to projects may no longer effective and should be replaced by newer ways such as, partnering with academic institutions, attracting millennial generation, crowd sourcing, outsourcing, insourcing (organic growth), etc.
  • Lack of awareness or crystal clear view of required skill set for big data is the fundamental challenge facing organizational leadership attempting to address resource shortage. Appropriate knowledge of data analytics and leadership capacity constraints are undermining many companies’ efforts to achieve that. Educate leadership and HR dept about big data.  Big data literacy program or training courses for HR are some of the proven techniques organizations can leverage. In the future, HR leadership will need to ensure that their teams have the competency and culture to support effective analytics.
  • Greater collaboration and open dialogue among different stakeholders, in particular – HR, business, finance, IT is required to define and quantify the value proposition of big data, and the value leakage from inadequate big data talent and the measures to be implemented to alleviate the problem.
  • Organizations should have an innovative talent acquisition and retention Paradigm shift in recruitment thought processes and totally revamped recruitment processes are some of the drastic measures applied by successful organizations to acquire, train and retain resources. Creative approaches for recruitment, encouraging organic growth via training, mentorship, pairing seasoned big data experts with less experienced talent ensure a steady supply of talent to meet growing organizational demands.  Generous longer term incentives to retain talent, and elevation of data heroes improve organic growth and maturity of big data talent.  Efforts should be in developing multi-talented and multi-skilled resources that are more suitable for big data.
  • While focusing on growing resource pool, HR leaders should be cognizant of poaching or plundering and be prepared for turnover and ballooning churn rate of big data resources and hence, factor high attrition rate for risk mitigation.
  • Invest and partner with local educational Shape academic curriculum, and customize internship and campus recruitment policies to better accommodate big data talent. STEM investment certainly pays off, but the pace of demand is way beyond the supply, and has already eclipsed the earlier predictions of growth of big data and advance analytics.
  • Accelerating Legacy retirement and releasing talent to focus on big data will be a double win for organizations.
  • Outsource to meet the demand. These resources can be entrusted with data preparation, visualization, data discovery and preliminary analysis, thereby offloading some of the burden.  Some organizations are turning to analytics service providers such as cEpoch for help.
  • Crowdsourcing could very well represent the wave of the future, at least when it comes to hiring data scientists. While crowdsourcing has its own challenges, it has also proven to be an effective strategy for augmenting big data bench strength.



Generating volumes of extraneous data is futile if there are no resources to derive actionable insights and solutions. Widening gap between technology capability and resource competency is disturbing and needs urgent attention.   Unfortunately, there is no easy button or magic wand to simply resolve this problem. Comprehensive strategy, concerted execution and meticulous efforts are required to overcome this barrier.    Organizations should apply rapid action plan to stay ahead of the curve, otherwise they miss opportunities to capitalize on benefits of big data solutions.  After all, Big Data need not be a pipe dream for mainstream organizations with limited IT and marketing budgets.



Venu Yalavarthi

I’m a frequent writer about enterprise architecture, data management, and big data. If you would like to read my regular posts, please click ‘Follow‘ and feel free to also connect via cEpoch.

Leave a Reply

Your email address will not be published. Required fields are marked *