Why it is hard to build a Big Data team

Ross Whatling our consultant managing the role
Posting date: 6/6/2017 3:19 PM

Increasingly, I speak to managers who are adopting big data tools and developing PoCs to prove how they can make use of them. Just last week I spoke to a data architect who mentioned that if he didn’t get exposure to big data tech sooner rather than later, his current RDBMS skills may become redundant within the next few years. While that is likely an exaggeration, it is certainly an interesting point. Companies that would have never previously had the capability to interpret ‘Big Data’ are now exploring a variety of NoSQL platforms. In particular, the massive performance benefits gained from Spark and real-time/streaming tools have opened up a whole new world beyond just MapReduce. I don’t claim to be a data engineer, but as a recruiter for this sector, what I do is spend all day, every day interacting with big data developers, architects and managers (as well as keeping a close eye on the latest Apache incubator projects). Due to this, I have seen some recurring themes that have become trends when companies look to create and build their big data teams that are coming to the fore.

Candidate demand

  • The demand for Big Data professionals is very much a present day issue as the data companies have grand plans for is waiting for the right data developer to use the best tech to extract valuable insights from it.
  • The best candidates receive massive interest, often gain multiple offers from a range of companies. Your business is now no longer just competing with large corporations such as Facebook, Twitter or Yahoo. Startups and SMEs are also vying for the best candidates.
  • Candidates are seeing pay rises twice that of the normal rate, as illustrated in our salary guide.

Candidate shortage

  • The number of candidates with hands-on, production level Big Data experience is incredibly limited. We go to great lengths to find the candidates who can add real value to companies.
  • The growth and exciting future for the big data industry has led to increased interest in big data jobs, particularly for those from RDBMS or software. engineering backgrounds. This leaves the industry in a difficult predicament: high demand + low supply = massive competition. There are countless examples of companies that have failed to recruit a Big Data team after a year of looking.

Competition to get ahead and stand out

  • Planning - Companies need to have a data road map detailing their future plans. Candidates want to clearly know what they are getting into and what to expect from a job.
  • Innovation - Why get stuck on batch processing? The most exciting positions that candidates love are in data innovations teams, playing with real-time/streaming tech and new languages.
  • Personal development, growth and training – with the data science market experiencing similar growth, many big data engineers are looking for a job that not only offers the chance to work with machine learning and similar fields; but training, mentoring towards clear career progression as standard.
  • Speed – the length of the interview process is often seen as a reflection of the amount of red tape developers have to go through to get a job. The longer and more convoluted the process, the more put off some people may be.
  • Complacency – don’t rest on your laurels, it’s unlikely that you’ll get 10s of CVs through when you are looking to fill a data role, so when you find a candidate you like, move swiftly to show your interest to them as quality candidates don’t come around often.

By implementing these small but effective improvements to your recruiting process and how you develop data talent will see you create a team that is a success in this ever more digital analytics landscape. Companies who don’t create and nurture strong, dynamic teams will fall by the wayside.

It’s Harnham’s job to help you achieve this goal. Get in touch with us to tell you how. T: (020) 8408 6070 E: info@harnham.com

Related blog & news

With over 10 years experience working solely in the Data & Analytics sector our consultants are able to offer detailed insights into the industry.

Visit our Blogs & News portal or check out the related posts below.

 Hiring a BI Manager – Trends and Challenges

Hiring a BI Manager – Trends and Challenges

With all the talk of big data and data science being able to predict what colour shirt I will buy in four years’ time (probably white or blue for those who don’t know me!), effective business intelligence is sometimes passed by or considered old news. The reality is that companies are realising that they can get much more from their business intelligence and are changing their strategies to deliver interactive, insight-driven and visualised reports. Not every data-driven decision needs machine learning algorithms behind it, and quality business intelligence enables all managers to be effective decision-makers. These strategies are creating some obvious trends in the market, resulting in a change in expectations when hiring a BI Manager. Key BI TrendsData Visualisation – Companies of all sizes are implementing Qlikview and Tableau (amongst many other tools) to create attractive, interactive visualisations, to harness intelligence, in a way that will capture attention in a presentation. Insight Driven - A BI professional can’t simply develop automated reports anymore. Analysts are often required to offer suggestions for business change and present insight to decision makers. Hands-on Management – BI managers and even heads of business intelligence are expected to keep coding well into their management years, with the logic that problems can be spotted quicker when they are in the trenches, coupled with strategic and line management work. Data Ambassadors – BI professionals are becoming door-to-door data sellers, coaching teams in a business on the benefits of using data to optimise their teams and decisions to save or bring in more money. Heads are in the Cloud – Companies are using cloud-based data warehouses such as Redshift to save on storage costs, whilst creating a centralised data warehouse for BI. Alternative Data Sources – Companies are looking to use the web and social media data, alongside numerous other sources to generate deep insights for managers. The BI Manager EffectI am completely sold that all of these features represent the future of business intelligence. The few companies that are doing all of the above well enough, are doing advanced work in the area and these companies will be leveraging big commercial gains from their business intelligence teams. The problem is that only a few businesses are doing all of the above, so only a handful of professionals have the relevant experience, and as a result expect top dollar to bring all of those skills. Therefore, it is prudent to be flexible with your hiring requirements. Look for a bright, passionate candidate, who can readily grasp the shift in business intelligence trends, and is keen to plug skills gaps. An enthusiastic business intelligence professional will get up to speed with whatever they were missing. Don’t be too quick to dismiss those who are not ready-made BI managers on paper. Message to CandidatesFor all aspirational or existing business intelligence managers and leaders, I would advise you try to stay hands on as long as possible. I know some of you dream of never seeing a line of SQL code again, however, the trend in hiring for hands-on business intelligence management positions means that keeping your tech skills sharp will really keep your options open moving forward. It would be great to hear your experiences, so please feel free to comment below on the trends you see in your business. Have you needed to remain hands on as you progress within your career? Or are you looking for a multi-skilled BI manager, and it is proving hard?

Which data visualisation tools should you be using?

Which data visualisation tools should you be using?

There is no denying the ever increasing applications of Data visualisation tools for the modern business. Everyone is aware of them in a business environment, whether they are conscious of it or not. Personally I have been making use of data from a sales perspective long before I was conscious of the titles Business Intelligence and Analytics. Using a simple geo-mapping tool to visualise account locations and statuses, I then shared this with senior management and the sales team. A very simple but useful application, and a far cry from sticking coloured pins in a map! Visualising your options A new breed of data visualisation tools have become prevalent in recent years. Given the number of vendors, varied features and implementation costs, it can be difficult to know which ones to: A) Implement as a Business B) Become familiar with as a BI Analyst/Developer. The purpose of Business Intelligence visualisation tools is something everyone can appreciate. As for centuries people have used pictorial or graphical representations of data for it to be universally understood. The usage and investment in this part of a corporation’s BI platform seems to be ever increasing. According to the trustradius 2015 BI survey: Medium (101-1000) and large (1001+) size companies are most likely to invest in this area of their business in 2015- even more so than big data (Hadoop, Cloudera) and predictive analytics solutions (R, SPSS etc) How to stay in demandWhen talking to clients, particularly regarding new positions being created and growth of BI teams, visualisation experience is increasingly sought after. Sometimes this is to maximise existing tools and platforms, and often to help assimilate their usage into the new Business Intelligence solutions and maximise their use for end users across the business. There are many differences between vendors, but the ones I encounter being used most often are Tableau, Qlikview, Spotfire and the MS BI platform. They all have pros and cons depending on existing systems, data streams, usage plans and of course (and often most pertinent) budgetary issues. From a client perspective, there will be numerous contributory factors to consider when selecting which one to implement, however I think it is important to highlight you will never be able to attract all candidates with any one choice. There will always be analysts and developers with strong subjective preference for certain tools. The positives to take from this is that many BI professionals I speak to are eager to learn a new tool. Furthermore, the general consensus between talented analysts and developers seems to be that established skills are highly transferrable between top vendor’s product offerings. This view seems to be mirrored by hiring managers, not just in the larger corporations. Of course for certain companies, teams and projects, the skills focus will lean towards and include prior experience with a particular vendor’s product as a necessity. However, often, as long as the back end data warehouse and data stream experience is compatible, there is generally a level of flexibility when hiring. Who is using who?This sentiment also rings true with the type of job specifications we are seeing. There is both increased demand for visualisation experience and a degree of flexibility on the particular tools analysts and developers have used in the past. With regard to job type, as a general rule, spotfire seems to be the most favoured for visual analytics, Tableau for visual OLAP, Qlikview for data visualisation drilldown and the MS BI stack for back end Data visualisation which makes sense given its direct compatibility with SQL server databases. Are you on the right track?So the message for hiring managers seems to be to research the type of profiles you’re looking to bring into your BI team and focus your efforts on candidates with that particular skillset. And finally aim to get a handle on which tools are most compatible in terms of transferable skills and usability. With regard to BI professionals it is impossible to have experience with all of the BI visualisation tools out there, but the key is to prioritise the ones you know are most compatible with the industry you are hoping to work in long term. For example if you are keen to progress your career towards a BI manager role within a big 4 consultancy, and you’re finding the job specs focus on particular tools you have not used. Keep up to date through courses, webinars, conferences and media events. From all the BI and analytics professionals I have worked with (including Developers, Analysts, SAS programmers, Programmers, DBA’s and consultants) over 49% have used one of the top four vendors for visualisation and reporting. This is only set to increase in 2016 as the market consolidates, and conversely the proliferation of broader skills and experience won’t go to waste with the advent of new tools coming to market. What is your experience as a BI professional or someone trying to build a team? How do you view the suitability of particular tools with regard to the stages of the BI solution from Data Warehousing?

RELATED Jobs

Salary

£70000 - £90000 per annum

Location

London

Description

Being one of the largest organizations globally; rich in data you will join a world-class function and work closely with key stakeholders.

Salary

£55000 - £65000 per annum + Additional Benefits

Location

London

Description

I am working with a leading Aviation company looking to hire a Data Engineer capable streaming vast quantities in real time.

Salary

£55000 - £70000 per annum + Additional Benefits

Location

London

Description

I am currently working with a Global Airline company looking to hire a Senior Data Engineer to build a real time Data platform.

Salary

600000kr - 800000kr per annum

Location

Stockholm

Description

This is an exclusive opportunity to work as a Senior Data Engineer in a large media company in Stockholm.

recently viewed jobs