Why it is hard to build a Big Data team

Increasingly, I speak to managers who are adopting big data tools and developing PoCs to prove how they can make use of them. Just last week I spoke to a data architect who mentioned that if he didn’t get exposure to big data tech sooner rather than later, his current RDBMS skills may become redundant within the next few years. While that is likely an exaggeration, it is certainly an interesting point. Companies that would have never previously had the capability to interpret ‘Big Data’ are now exploring a variety of NoSQL platforms. In particular, the massive performance benefits gained from Spark and real-time/streaming tools have opened up a whole new world beyond just MapReduce. I don’t claim to be a data engineer, but as a recruiter for this sector, what I do is spend all day, every day interacting with big data developers, architects and managers (as well as keeping a close eye on the latest Apache incubator projects). Due to this, I have seen some recurring themes that have become trends when companies look to create and build their big data teams that are coming to the fore.

Candidate demand

• The demand for Big Data professionals is very much a present day issue as the data companies have grand plans for is waiting for the right data developer to use the best tech to extract valuable insights from it.
• The best candidates receive massive interest, often gain multiple offers from a range of companies. Your business is now no longer just competing with large corporations such as Facebook, Twitter or Yahoo. Startups and SMEs are also vying for the best candidates.
• Candidates are seeing pay rises twice that of the normal rate, as illustrated in our salary guide.

Candidate shortage

• The number of candidates with hands-on, production level Big Data experience is incredibly limited. We go to great lengths to find the candidates who can add real value to companies.
• The growth and exciting future for the big data industry has led to increased interest in big data jobs, particularly for those from RDBMS or software. engineering backgrounds. This leaves the industry in a difficult predicament: high demand + low supply = massive competition. There are countless examples of companies that have failed to recruit a Big Data team after a year of looking.

Competition to get ahead and stand out

• Planning - Companies need to have a data road map detailing their future plans. Candidates want to clearly know what they are getting into and what to expect from a job.
• Innovation - Why get stuck on batch processing? The most exciting positions that candidates love are in data innovations teams, playing with real-time/streaming tech and new languages.
• Personal development, growth and training – with the data science market experiencing similar growth, many big data engineers are looking for a job that not only offers the chance to work with machine learning and similar fields; but training, mentoring towards clear career progression as standard.
• Speed – the length of the interview process is often seen as a reflection of the amount of red tape developers have to go through to get a job. The longer and more convoluted the process, the more put off some people may be.
• Complacency – don’t rest on your laurels, it’s unlikely that you’ll get 10s of CVs through when you are looking to fill a data role, so when you find a candidate you like, move swiftly to show your interest to them as quality candidates don’t come around often.

By implementing these small but effective improvements to your recruiting process and how you develop data talent will see you create a team that is a success in this ever more digital analytics landscape. Companies who don’t create and nurture strong, dynamic teams will fall by the wayside.

It’s Harnham’s job to help you achieve this goal. Get in touch with us to tell you how. T: (020) 8408 6070 E: info@harnham.com