Defragmenting Data Analytics

Guest Blog our consultant managing the role
Author: Guest Blog
Posting date: 1/12/2021 2:25 PM
This week's guest blog is written by Moray Barclay

                  Around 20 years ago I was showing some draft business plans with cashflow projections to my new boss. His name was Marc Destrée and I concluded by saying I’d like to get the finance department involved. “No”, Marc replied. He paused for several seconds, looked up from his desk, and explained "Do the internal rate of return. Then we discuss. Then we give it to finance."

He was right of course, for three reasons which together represent best practice. Firstly, it cemented the separate accountabilities between the different job functions responsible for the business case and financial governance. Secondly, there were no technical barriers to separating the “cashflow creation process” and the “P&L creation process” as everyone in the organisation used the same product: Excel. Thirdly, it assigned the right skills to activities.

Today, organisations have no equivalent best practice upon which to build their data analytics capability. The lack of best practice is caused by fragmentation: fragmentation of job functions, fragmentation of products, and fragmentation of skills. This is not necessarily a bad thing: fragmentation drives innovation, and those organisations who get it right will gain huge competitive advantage.

But the application of best practice mitigates against unnecessary fragmentation and hence unnecessary inefficiencies.

So how could best practice be applied to an organisation’s data analytics capability? In other words, how we do defragment data job functions, data products and data skills?

Defragmenting data job functions

A good starting point to understanding best practice for data job functions is the informative and well-written publication “The scientist, the engineer and the warehouse”, authored by the highly respected Donald Farmer of TreeHive Strategy. He includes references to four job functions: (i) the data scientist, (ii) the data engineer, (iii) the business intelligence analyst and (iv) the departmental end user. 

(i) The data scientist: The accountability of the data scientist is to build data science models using their skills in maths and coding to solve business problems. In addition to using open source technologies, such as python and R, data scientists can and do use data science platforms such as Knime which enable them to spend more time on maths and less time on coding - more on data science platforms later.

(ii) The data engineer: The accountability of the data engineer is to build robust and scalable data pipelines which automate the movement and transformation of data across the organisation’s infrastructure, using their skills in database engineering, database integration, and a technical process called extract/transform/load (ETL) and its variants – more on ETL production platforms later.

(iii) The business intelligence (BI) analyst: Donald Farmer’s publication does not address the accountabilities of the BI analyst in any detail because that is not its focus. Unlike the clearly defined roles of data scientists and data engineers, there are no best practice descriptions for the role of BI analyst. Typical accountabilities often include designing data visualisations from existing datasets, building these visualisations into reports or online dashboards and automating their production, and configuring end users to ensure they only have access to data that they are approved to see.

Beyond these core accountabilities, BI analysts sometimes create entirely new datasets by building complex analytic models to add value to existing datasets, using either a suitable open source technology (such as python, but used in a different way to data scientists) or a data analytic platform such as Alteryx which enables the creation of code-free analytic models.

One final point - a BI analyst might also build data science models, albeit typically more basic ones than those built by data scientists. BI analysts will inevitably become more like data scientists in the future driven by their natural curiosity and ambitions, vendors creating combined data science platforms and data analytic platforms, and organisations wanting to benefit from the integration of similar functions.

(iv) The departmental end-user: A departmental end-user is generally the most data-centric person within a department: it might be a sales operations professional within a sales department for example. I am told that when Excel was first introduced into organisations in the 1980’s, there would be a “go-to Excel expert”; self-evidently over time everyone learned how to use it. I was there when CRM systems like and Netsuite appeared 20 years later, and the same thing happened: initially there would be one or two pioneers, but eventually everyone learned to use it. The same democratisation is happening and will continue to happen with business intelligence. In the same way that CRM and Excel are used by everyone who needs to, soon anyone will be able to build their own data visualisations and reports to help identify and solve their own problems. In some organisations such as BP this is already well-established. And why stop there? If a departmental end-user can model different internal rates of return and create visualisations, then why should they not apply their own data science techniques to their own datasets?

But this can only happen if the role of the BI analyst has an accountability for democratisation, in addition to those mentioned earlier.In summary, the following is a list of best practice accountabilities for the BI analyst:

(1) Build and automate the initial set of business intelligence reports and visualisations

(2) Create the data governance framework to enable self-service by departmental end-users

(3) Act as the initial go-to business intelligence expert

(4) Evangelise a data-driven culture and mentor those who want to become proficient in self-service

(5) Deploy resources which over time make redundant the role of a go-to business intelligence expert

(6) Over time, increase time devoted to creating innovative datasets by building complex analytic models which add value to existing datasets - using open source technologies and/or a data analytic platform

(7) Work with the data science function in such a way that over time the data science function and the BI function can be merged

The above best practice eventually results in the role of the BI analyst, or the BI analyst team, becoming redundant, much in the way that the role of a dedicated Excel specialist died out in the mid-1980’s. As mentioned earlier, as BI analysts will move into data science, this should not result in people losing their jobs. 

Defragmenting data products

Unlike open source technologies there is a highly fragmented data product landscape. Products include data science platforms, data analytic platforms, platforms which are more visualisation-centric, and platforms which are more focused on data governance. There are also ETL production platforms which are in the domain of the data engineer but which include functionality to build some types of analytic models.

Fragmented markets eventually consolidate. Even the broadest three cloud vendors, Amazon, Google and Microsoft, do not cover the entire landscape. For visualisation there is Quicksight, Data Studio, and Power BI respectively as well as competitive products, most obviously Tableau; for ETL production platforms there is Athena, Cloud Dataflow and Azure Data Factory, as well as competitive products such as Talend. But smaller vendors have the lead in data science platforms and data analytic platforms. The hiring by Microsoft of the python inventor Guido van Rossum two months ago points to their ambitions in data science platforms and data analytic platforms. Market consolidation in 2021 seems inevitable, but the details of actual acquisitions are not obvious. After all, it was which bought Tableau in 2019: not Amazon, Google or Microsoft.

Best practice for organisations is to consider possible vendor consolidation as part of their procurement process, because product fragmentation means there is a corresponding fragmentation of skills.

Defragmenting data skills

Fragmentation of data skills means that the market for jobs, particularly contract jobs, is less elastic than it could be. The fragmentation of skills is partly caused by the fragmentation of products and their associated education resources and certification.

Vendor’s product pricing typically falls into three categories: (i) more expensive commercial products (c. £500 - £5000 per user per month) which include free online education resources and certification; (ii) inexpensive commercial products (c. £5 to £50 per user per month) which usually require a corporate email address but have free online education resources and reasonably-priced certification exam fees (c £100- £200); and (iii) products which are normally expensive but have an inexpensive licensed version that cannot be used for commercial purposes, again including free online education resources and certification. The latter approach is best practice for solving the fragmentation of skills because the barriers to learning (i.e. high product cost or the need for a corporate email address) are removed.

Best practice includes the Microstrategy Analyst Pass, which is available to anyone and costs $350 per year including a non-commercial product licence, online education resources and access to certification exams.

University students (as well as self-educated hackers) learn open source technologies and one would expect that those skills are sufficient for them to enter the workplace in any data analytics environment. Yet several vendors who provide the more expensive commercial products (c. £500 - £5000 per user per month) and do not have discounted licences for non-commercial purposes make one exception: universities. At face value, this seems benign or even generous. But it contributes to the inelasticity of the job market at graduate level because an unintended consequence is that some graduate data analytics jobs require the graduate to be competent in a product before they have started work.

Best practice is for organisations to employ graduates based on their skills in maths, statistics and open source technologies, not product.

In seeking corporate acquisitions, vendors might find that their customers value “education bundling” as much as “product bundling”. Customers who are happy to pick, for example, the best visualisation product and the best data storage product from different vendors might be more attracted to their people using a single education portal with the same certification process across all products. And if an organisation can allocate 100% of its education budget to a single vendor then it will surely do so.

Best practice is for vendors to consider the value of consolidating and standardising education resources, and not just products, when looking at corporate acquisitions.

Defragmentating data analytics

The consequence of implementing a best practice data analytics capability based on the principles of defragmentation has profound consequences for an organisation. It enables a much richer set of conversations to the one which took place 20 years ago.

A young business development manager is showing some draft business plans to their new boss. They conclude by saying they’d like to get a data scientist involved. “No”, the boss replies. He pauses for several seconds, looks up from his desk and explains "Segment our customer base in different ways using different clustering techniques. Then run the cashflow scenarios. Then we discuss. Then we give it to data science."

You can view Moray's original article here.

Moray Barclay is an Experienced Data Analyst working in hands-on coding, Big Data analytics, cloud computing and consulting.

Related blog & news

With over 10 years experience working solely in the Data & Analytics sector our consultants are able to offer detailed insights into the industry.

Visit our Blogs & News portal or check out the related posts below.

Three Ways Data Impacts The Customer Experience

In 2019, over 50 per cent of companies had adopted Big Data, with a further 38 per cent citing that they would be investing in it in the future. As it stands, we can assume that now, at least three-quarters of businesses will have invested in Big Data capabilities. By 2022, the annual revenue from the global big data and business analytics market is expected to reach $274.3 billion.  The lucrative nature of this industry stems from a recognition by many companies that it’s no longer good enough to guess what customers might want or need from your product or service, but to have hard evidence to back up your choices. Not only does this make for much happier, more satisfied customers, but it undoubtedly improves the bottom line.  Here are three examples which showcase how Data can positively impact the customer experience: 1. Create a more intuitive website journey From heatmapping the areas of interest (or disinterest) on your website through eye movement or mouse tracking to traffic analysis through tools such as Google Analytics, Data can give you both real-time and overall information about the success of your website.  You can analyse areas of the website where consumers ‘linger’ or click through, such as content pieces, links or assets, which proves to give added value or entice them to learn more about your business. You can also see areas where little to no activity happens, allowing you to create a new, perhaps more engaging, strategy.  The use of data for website ensures your get the design and content right in less time. The cost of redesigning a website can be a hefty cost for any business. The fewer times a website needs chopping and changing, the more cost-effective it will be, not forgetting to mention a much smoother and more efficient process for customers. 2. Building loyalty through personalisation In a report featured in Forbes by The Harris Poll, 76 per cent of Americans are more likely to complete a purchase if the customer journey has been personalised to them, their needs and wants. The story is similar in the UK, 80 per cent of companies report seeing an uplift after employing personalisation tactics.  However, personalisation must go one step further than just addressing a person by name in an email nowadays. It means targeting consumers with specific and relevant ads that actually take their interest instead of bombarding them with a scattergun approach, as well as looking at areas such as location-specific targeting and device optimised outreach. This can be made possible by combining marketing data, such as brand interactions, combined with sales data, previous purchases, and customer service data, the feedback given. These aspects allow you to create an in-depth and meaningful customer journey map, help you understand what turns specific consumers on, or off, and ensures your marketing messages and outreach are pertinent.  3. Be prepared for problems before they occur Data can give incredible insight into what’s working currently for a business but, arguably, its strengths lie in giving accurate understanding into the potential risks or problems that are likely to occur in the future.  According to Clarion Tech, there are seven areas in which Data can play a crucial role in minimising risk, errors or issues for a vast range of businesses. From making sense of unused business data to making companies proactive instead of reactive, minimising misleading forecasts to diminishing customer service challenges, data can be the solution to a wealth of problems.  Not only do these kinds of errors leave a bitter taste in the mouths of customers who may struggle to revisit your business after a bad experience, but they can negatively affect your bottom line too. Nipping them in the bud before they happen is an incredible card to have to hand, and one that could be the saviour of your business.  To learn more about how working with a Data & Analytics specialist could help bolster the success of your business, contact our team or, if you're looking for your next opportunity, check out our latest roles. 

Data Analytics vs. Data Science: Which Should You Pursue?

Businesses are recognizing the increasing importance of data experts to help the company grow. As a result, the hiring demand for Data Scientists and Data Management Analysts has grown by 46% since 2019. This projection will only continue to rise in the next few years. So if you’re planning to become a data analyst or a data scientist, then here’s what you need to know. Data Analytics and Data Science: What's the Difference? Data Analysts and Data Scientists are both proficient in statistics and experienced in using database management systems. However, the key differences between these two professions revolve around their purpose for using the data. The Role of a Data Analyst These professionals organize and examine structured data to create solutions that will drive a business’ growth. They are tasked with studying sets of data using various tools, such as Excel and SQL, to uncover insights and trends that will serve as an answer to certain queries. For example, they can provide data-driven answers that can explain your marketing campaigns’ conversion rates or improve the logistics of your products. Then, they present these findings to concerned individuals and departments so they can formulate strategies that would boost revenue, efficiency, and other improvements. The Role of a Data Scientist Data Scientists are required to use their mathematical and programming skills to build statistical models that can provide solutions for a company’s potential problems. These professionals handle huge sets of both structured and unstructured data and prepare these for processing and analysis. They have to be very proficient in programming to utilize Predictive Analytics, statistics, and Machine Learning in unearthing meaningful insights from all the collected data. Their multidisciplinary approach towards data helps them draw conclusions that are valuable for specific business needs and goals. Career Paths for Aspiring Data Analysts Businesses, governments, and other institutions are on the search for individuals who are qualified in interpreting and communicating data. Data analysts are often offered huge salaries and great work benefits because the demand is so high and yet, the pool of talent is very limited. You can become qualified for a wide array of careers in data analytics through a comprehensive master’s degree program that will teach you how to interpret data and present actionable insights. These careers span from digital marketers to quantitative analysts. Graduates can work in governments and insurance companies as financial analysts who are in charge of assessing financial statements and economic trends to boost profit. On the other hand, you can also work as a marketing analyst whose responsibilities involve monitoring sales venues and evaluating consumer data. Their salaries range from $62,000 (Insight Analysts) to as much as $225,000 (highly paid Customer Analysts). Career Paths for Aspiring Data Scientists Data Scientists are experts in statistical analysis and in programming languages, such as Python and R. Thus, the average starting salary for professionals in this field is around $100,000 per year. Data Scientists would need to earn a bachelor’s degree and a master’s degree in computer science so that they would be adept at using complex software programs that are necessary for the position. If you’re more interested in software development, then you can work as a data engineer. These professionals create infrastructures that can gather and store data that analysts and other scientists may need to use. Data modellers, on the other hand, use techniques and databases to design and document data architecture. You can become a great asset to top companies in the US by pursuing a degree and a career in data analytics or data science. In this digital age, you can only expect that the demand for these positions would rise as data becomes increasingly important in driving business growth.  Written by Jena Burner for


recently viewed jobs