The future of DW's in the age of Big Data

Kirsty Garshong our consultant managing the role
Posting date: 7/27/2013 2:47 PM

Many companies are saddled with data warehouses that weren’t designed to handle big data, but they can evolve their data warehouses into “analytics warehouses” capable of processing structured and unstructured data.

Enterprise data warehouses have reached a crossroads. Companies have spent millions of dollars designing, implementing, and updating them, but few organizations have realized the return they expected from their investments, according to Richard Solari, a director with Deloitte Consulting LLP’s Information Management service line.

The disappointing ROI largely stems from an inherent inadequacy in data warehouses: They were designed to handle the kind of structured data stored in ERP systems, not the unstructured data from social media, mobile devices, Web traffic, and other sources now streaming into enterprises. By Solari’s estimate, 90 percent of the data warehouses he’s observed process just 20 percent of an enterprise’s data. Consequently, many enterprises have only been able to use their data warehouses for historical analysis and past performance reporting.

The bottom line, says Solari: “Companies are using expensive infrastructure to generate back office reports.”

Organizations’ prospects for obtaining an acceptable return from their data warehousing investments may continue to diminish as long as this infrastructure fails to keep pace with big data.

The good news: Vendors are building new generations of data warehouses with advanced statistical capabilities for performing analytics and forecasting, according to Robert Stackowiak, Oracle Corp.’s vice president of information architecture and big data. They’re also improving integration with emerging platforms like Hadoop that process large volumes of unstructured data, he says.

Because newer generations of data warehouses are designed to federate structured and unstructured data, they may provide enterprises with a 360-degree view of their operations and, with that broader perspective, the ability to make better decisions about the future, according to Solari.

Companies running legacy data warehouses don’t have to junk their infrastructure and start anew. Solari says they can add capabilities to their existing data warehouse infrastructure that can allow it to grow into an “analytics warehouse.”

“Data warehouses are going to look very different in five years, and organizations should begin preparing for that transition,” says Solari.

Introducing the Analytics Warehouse

Fundamentally, the analytics warehouse functions as a central repository for an enterprise’s structured and unstructured data. In a traditional data warehousing architecture, structured data from ERP systems, CRM systems, file shares, and line of business applications is batch processed into the enterprise data warehouse using ETL (extract, transform, load) database processes. Software for running ad hoc queries and business intelligence systems take data from the warehouse environment, which may include operational data stores and data marts, to generate reports for users.

The architecture for the analytics warehouse builds on the traditional data warehouse architecture in three primary ways:

1. A distributed file system (like Hadoop) sits between source data systems and the data warehouse. It collects, aggregates, and processes huge volumes of unstructured data, and stages it for loading into the data warehouse.

2. Structured and unstructured data from back end systems can be brought into the data warehouse in real- and near-real time.

3. Engines that use statistical and predictive modeling techniques to perform data discovery, visualization, inductive and deductive reasoning, and real-time decision-making reside between the data warehouse and end users. These engines identify patterns in big data. They can also complement and feed traditional ad hoc querying tools and business intelligence applications.


“In the past, companies couldn’t integrate these disparate technologies with the data warehouse because each technology required different file formats and data schemas,” says Stackowiak. “Today, you can integrate these technologies, and the result is that companies can access more of their data—not just the 20 percent from enterprise systems—and convert it into valuable, profitable information.”

Companies interested in building out their traditional data warehouse infrastructures may consider starting with reporting, if they don’t already have reporting capabilities in place, suggests Solari. Then, they can begin integrating analytics technologies to their reporting framework.

“When companies start bringing this data together and federating it inside a data warehouse, the total cost of ownership for the data warehouse may begin to go down while the ROI goes up,” says Solari. “The ability to integrate big data technologies, analytics technologies, back office systems, and traditional data warehouses has the potential to fundamentally change the economics of data warehousing for the better.”


Click here for the article on the web.


Related blog & news

With over 10 years experience working solely in the Data & Analytics sector our consultants are able to offer detailed insights into the industry.

Visit our Blogs & News portal or check out the related posts below.

Marketing Analytics - Then, Now & In the Future: A Q&A with Sarah Nooravi

We recently spoke to Sarah Nooravi, an Analytics professional with a specialism in Marketing who was named one of LinkedIn’s Top Voices in Analytics.  Sarah found herself working in Analytics after being attracted to the culture, creativity and the opportunity to be challenged. Having spent the first four years of her career working within the Marketing space, she has seen a real transition in the way that Analytics and Data Science has informed Marketing decisioning.  “I started my career in a Marketing agency within the entertainment industry, at the time it was doing things that most of the entertainment industry hadn’t considered doing yet”.  At the start of her career she’d meet entertainment giants with advertising budgets of millions of dollars who were, at the time, making mostly gut decisions with how to approach campaigns. “It was common that I’d hear, ‘I think our audience is females over the age of 35 with a particular interest and we should just target them’” she expands.  However, agencies quickly recognised the need for something more Data-driven. Entertainment businesses were going too narrow and were misunderstanding their audiences. The next step was to embed into these businesses the insights from a greater variety of sources, including social media, and to introduce more testing. That translated into a better media buying strategy that could be continuously optimised. It was a big step forward in the utilisation of Data within this realm and its clear focus on ROI.  Suddenly, the market was changing, “There was a massive spike of agencies popping up and claiming to leverage Data Science and Machine Learning to provide better optimisations for entertainment companies, mobile gaming – you name it. There was a huge momentum shift from using these gut decisions to leveraging agencies that could prove that”.  What she saw next seemed only natural, with more agencies offering Data-driven optimisation, companies looked to develop this capability internally. Sarah elaborates; “Now I am seeing these companies starting to take ownership of their own media buying and bringing the Marketing and Data Science in-house”. This shift in-house has been propelled by the major players, companies like Facebook, Google and Nooravi’s own company, Snapchat, working directly with companies to help them optimise their campaigns. This shift has changed the landscape of Marketing Analytics, specifically within the advertising space. Sarah explains, “You no longer need an agency to optimise your, for example, Facebook campaigns, because Facebook will do it for you. They are minimising the number of people behind the campaigns. You give up a little of your company’s Data for a well optimised campaign and you don’t have to hire a media buyer. There is definitely a movement now to becoming more Data-driven. Companies are really leveraging A/B tests and also testing out different creatives”.  It is this change in strategy that is seemingly taking the Marketing Analytics challenge to the next level. With opportunities to pinpoint specific audiences, companies are using their Data to understand how to approach their content, take the opportunity to experiment, and to find out what it takes to resonate with their audience. Sarah has seen the potential of this first hand: “We are starting to see a lot of AR and VR. There are meaningful ways to engage with technology to connect with the world. Moving forward, content will have to become more engaging. People’s attention spans are becoming shorter and with each decision someone makes it is changing the direction of content in the future. There has been a massive shift from static images to video advertisement and, more recently, from video into interactive video like playable adverts. People want to engage with adverts in order to understand a company’s message”.  It is within this space that she sees a gap for the future of ROI positive advertising:  “The biggest issue that I find with the creative and the content is that the value add is missing. The resonance with the brand or company, their values and mission is what is missing. Analytics alone cannot fix that. You need to understand what the company stands for, people want to connect with brands because of what they stand for – whatever it is. Especially in a time like we are dealing with right now, a pandemic, advertising spending has gone down. However, maybe there is a way to properly message to people that would resonate. Not that you want them to buy your stuff but maybe right now is the perfect time to do outreach and to help people understand your brand”. The ability to understand and predict customer behaviour is evolving, but with that, so is the customer. Whereas at the moment, you can build out experiments, you can create models that will be able to, as Sarah explains, “in real-time decide whether a user’s behaviour is indicative of one that is going to churn” and then try and create offers to increase retention.   This is the challenge of the current analytics professional – our behaviours in a global pandemic have shifted consumers into a new world. Now working for Snap Inc, she sees the potential of this from a new perspective. Naturally, like most social media channels and communication technologies, they have seen an increase in usage over the last month.  “People are wanting to communicate more as we are forced to social distance. However, we are seeing different regions engaging a lot more heavily. For example, it's Ramadan right now, people want to share those moments with one another and at the moment the way that they are having to do that is changing”.  So, it will be a question for all those required to predict behaviours to determine how many of these new lines of communication, these new habits, will have evolved. Once people are out of quarantine, are they going to continue to utilise the apps, games, social channels in the same way that they are currently? It certainly is going to be something that many within the marketing analytics space will be trying to forecast.  If you’re looking to take your next step in Marketing Analytics, or are looking to build out your team, Harnham may be able to help.  Take a look at our latest opportunities or get in touch with one of our expert consultants to find out more. 

How To Write The Perfect Tech Job Description

It’s a challenge finding the right Data & Analytics candidate. Add in the number of companies fighting for that perfect profile and for many it can seem like an uphill battle. But there’s a simple way to cut through the noise; better job descriptions.  As a specialist recruitment agency within the Data & Analytics space, we have seen a real variety of job descriptions over the years, from the bright and innovative to the long and technical. And it may surprise you to learn that candidates still ask regularly to see official job descriptions and it is part of their decision-making process.  Unfortunately, they are also often a part of the recruitment process that can be rushed or created from out-of-date previous descriptions. There are some real benefits, however, to putting the time and effort required into creating something fresh.   If you’ve recruited a role like a Data Scientist before, you know that the problem isn’t usually getting enough candidates through the door, it’s about getting the right ones. A well-crafted job description leads to better quality applicants. It also helps those candidates become more engaged and excited about your business.  So, with that in mind, here are our five top tips for businesses looking to help their role stand out from the crowd.  CHECK YOUR JOB TITLE  You might think that calling your BI Analyst a ‘Data Ninja’ is going to get you the top talent, but it would probably mostly cause confusion. It is important that you align the job title to a clear and market relevant job title. Often internal job titles can be the biggest blocker in aligning your vacancy to the market.  Consider changing the job title for external purposes to make it more closely aligned to the market. Here are some common examples:  An AVP Analyst within a Marketing Analytics team is more closely aligned to a Senior Marketing Analyst. A Data Scientist job title aligned to a role with no machine Learning or algorithmic development may be better titled a Statistical Analyst.  CREATE A COMPELLING JOB RUN-THROUGH  Our consultants agreed unanimously that one of the weakest areas of job descriptions tends to be the more detailed description of what the role actually is. Too often job descriptions just list lots of different responsibilities, but these are often very generic or basic.  Before starting to write the job brief, ask members of your team that do the role already – what gets them excited?  You will likely find that it has to do more with the types of projects i.e. the application of technical elements, that appeals most to candidates. If you can, bring the role to life in a meaningful way. For example, relating it to projects that your team has done is a really enticing method of exciting a candidate about the potential of the role. Create A Tailored Experience Section Uninspiring job descriptions often have long lists of key skills required, often with irrelevant skills included. Keep your requirements to around 5 or 6 key bullet points, asking yourself what the most important requirements are and clearly laying those out.  On top of that often companies get too focused on requesting years of experience. We strongly discourage companies from specifying years of experience in a job advert as, within the UK, most European countries and a number of US states this is classified as age discrimination. Instead of including years of experience, carve out what it is that you want your ideal candidate to have done before instead, this will often correlate to their experience level. For example: 5+ years' experience in a Marketing Analytics could easily be transformed to Proven commercial experience in a Marketing Analytics environment with exposure to pre and post campaign analysis, customer analysis,  customer segmentation and predictive modelling.  DON’T FORGET TO SELL YOURSELVES Another key area where many companies fall down is effectively selling their opportunity and company to the prospective candidates. Whether an active or passive job-seeker, candidates are likely deciding whether this is the right fit for them based on what they are reading. Many job descriptions completely forgo any type of sales pitch above an initial description of what the company does, perhaps because they expect the candidate to know them and want them.  These are the areas we’d suggest bringing to life to effectively sell your opportunity: Writing in your brands personality. Consider the right tone of voice to match your company culture and style of working. Introduce yourself. Whether you’re a brand name or not, use this chance to actually tell people about what you really do and what you really stand for. Share what it’s like to work for the company. Include the culture, work environment, targets, challenges and of course reference to perks and benefits on offer too. Consider the candidate. What appeals to a talented Data Scientist will differ from what appeals to an HR professional. Make sure you tailor your overall pitch to the type of candidate you are seeking.  WORK ON THE LOOK AND FEEL  A little effort on the aesthetic look of your job description an go a long way.  On top of a nice overall look, keep the length to a maximum of 1.5 pages. Utilise bullet points and bold formatting to keep the description some-what ‘skimmable’.  If you’re looking to hire a Data & Analytics professional, Harnham can help. Get in touch with one of our expert consultants to find out more. 

RELATED Jobs

Salary

£80000 - £100000 per annum

Location

London

Description

Fully remote role for a Senior Software Engineer in healthcare.

Salary

US$275000 - US$300000 per annum + Additional Benefits

Location

Boston, Massachusetts

Description

A biopharma company in Boston are looking for a VP Enterprise Architect to join their business.

Salary

US$150000 - US$190000 per annum + Additional Benefits

Location

Miami, Florida

Description

Interested in joining a leading horse racing brand as a Data Architect/Senior Data Engineer in Florida? Read below to learn more!

recently viewed jobs