The future of DW's in the age of Big Data

Kirsty Garshong our consultant managing the role
Posting date: 7/27/2013 2:47 PM

Many companies are saddled with data warehouses that weren’t designed to handle big data, but they can evolve their data warehouses into “analytics warehouses” capable of processing structured and unstructured data.

Enterprise data warehouses have reached a crossroads. Companies have spent millions of dollars designing, implementing, and updating them, but few organizations have realized the return they expected from their investments, according to Richard Solari, a director with Deloitte Consulting LLP’s Information Management service line.

The disappointing ROI largely stems from an inherent inadequacy in data warehouses: They were designed to handle the kind of structured data stored in ERP systems, not the unstructured data from social media, mobile devices, Web traffic, and other sources now streaming into enterprises. By Solari’s estimate, 90 percent of the data warehouses he’s observed process just 20 percent of an enterprise’s data. Consequently, many enterprises have only been able to use their data warehouses for historical analysis and past performance reporting.

The bottom line, says Solari: “Companies are using expensive infrastructure to generate back office reports.”

Organizations’ prospects for obtaining an acceptable return from their data warehousing investments may continue to diminish as long as this infrastructure fails to keep pace with big data.

The good news: Vendors are building new generations of data warehouses with advanced statistical capabilities for performing analytics and forecasting, according to Robert Stackowiak, Oracle Corp.’s vice president of information architecture and big data. They’re also improving integration with emerging platforms like Hadoop that process large volumes of unstructured data, he says.

Because newer generations of data warehouses are designed to federate structured and unstructured data, they may provide enterprises with a 360-degree view of their operations and, with that broader perspective, the ability to make better decisions about the future, according to Solari.

Companies running legacy data warehouses don’t have to junk their infrastructure and start anew. Solari says they can add capabilities to their existing data warehouse infrastructure that can allow it to grow into an “analytics warehouse.”

“Data warehouses are going to look very different in five years, and organizations should begin preparing for that transition,” says Solari.

Introducing the Analytics Warehouse

Fundamentally, the analytics warehouse functions as a central repository for an enterprise’s structured and unstructured data. In a traditional data warehousing architecture, structured data from ERP systems, CRM systems, file shares, and line of business applications is batch processed into the enterprise data warehouse using ETL (extract, transform, load) database processes. Software for running ad hoc queries and business intelligence systems take data from the warehouse environment, which may include operational data stores and data marts, to generate reports for users.

The architecture for the analytics warehouse builds on the traditional data warehouse architecture in three primary ways:

1. A distributed file system (like Hadoop) sits between source data systems and the data warehouse. It collects, aggregates, and processes huge volumes of unstructured data, and stages it for loading into the data warehouse.

2. Structured and unstructured data from back end systems can be brought into the data warehouse in real- and near-real time.

3. Engines that use statistical and predictive modeling techniques to perform data discovery, visualization, inductive and deductive reasoning, and real-time decision-making reside between the data warehouse and end users. These engines identify patterns in big data. They can also complement and feed traditional ad hoc querying tools and business intelligence applications.


“In the past, companies couldn’t integrate these disparate technologies with the data warehouse because each technology required different file formats and data schemas,” says Stackowiak. “Today, you can integrate these technologies, and the result is that companies can access more of their data—not just the 20 percent from enterprise systems—and convert it into valuable, profitable information.”

Companies interested in building out their traditional data warehouse infrastructures may consider starting with reporting, if they don’t already have reporting capabilities in place, suggests Solari. Then, they can begin integrating analytics technologies to their reporting framework.

“When companies start bringing this data together and federating it inside a data warehouse, the total cost of ownership for the data warehouse may begin to go down while the ROI goes up,” says Solari. “The ability to integrate big data technologies, analytics technologies, back office systems, and traditional data warehouses has the potential to fundamentally change the economics of data warehousing for the better.”


Click here for the article on the web.


Related blog & news

With over 10 years experience working solely in the Data & Analytics sector our consultants are able to offer detailed insights into the industry.

Visit our Blogs & News portal or check out the related posts below.

2020: The Year of the Data Engineer

Data Engineers are the architects of Data. They lay the foundation businesses use to collect, gather, store, and make Data usable. Each iteration of the Data as it moves along the pipeline is cleaned and analysed to be used by Data professionals for their reports and Machine Learning models. A ROLE IN HIGH DEMAND Even as businesses reopen, reassess, and for some, remain remote, the demand for Data Engineers is high. Computer applications, Data modelling, prediction modelling, Machine Learning, and more need Data professionals to lay the groundwork to help businesses benefit in today’s Data-driven culture. The word gets thrown around a bit, but when the majority of business has moved online, Data-driven is the name of the game. Having a Data plan, a Data team, and all aligned with your business strategy is imperative to the way business is done today. This type of innovation can offer insight for better business decisions, enhance customer engagement, and improve customer retention without missing a beat.  Without Data Engineers, Data Scientists can’t do their jobs. Understanding the amount of Data, the speed at which is delivered, and its variety need Engineers to create reliable and efficient systems. Like many Data professional jobs, even still in 2020, Data Engineers are in high demand. Yet a skills shortage remains. This has created an emerging field of professionals from other backgrounds who are looking to take on the role of Data Engineer and fill the gap. Whether by necessity or design, these individuals build and manage pipelines, automate projects, and see their projects through to the end result. CAREER OPPORTUNITIES OUTSIDE THE NORM As this growing trend emerges, it has created career opportunities for those with experience outside the normal channels of Data Engineering study. While it might involve individuals from backgrounds such as software Engineering, Databases, or something similarly IT-related, some businesses are upskilling their employees with talent. Rapid growth, reskilling, upskilling, and ever-constant changes still leave businesses with a shortage of Data Engineers to meet the demand. It’s critical to fill the gap for success. According to LinkedIn’s 2020 Emerging Jobs Report, Data Engineering is listed in the top 10 of jobs experiencing growth. THREE STEPS TOWARDS BECOMING A DATA ENGINEER This is a vital role in today’s organisations. So, if you’re in the tech industry and want to take a deeper dive into Data as a Data Engineer, what steps can you take? This is a time like no other. There’s time to assess your goals, take online classes, and get hands on with projects. Though having a base of computer science, mathematics, or business-related degree is always a good start. Be well-versed in such popular programming languages such as SQL, Python, R, Hadoop, Spark, and Amazon Web Services (AWS).Prepare for an entry-level role once you have your bachelor’s degree.Consider additional education to stay ahead of the curve. This can include not only professional certifications, but higher education degrees as well. The more experience, hands-on as well as academic, you have the more in demand you’ll be as a Data Engineer. Data scientists might be the rockstars of Data, but Data Engineers set the stage. As business processes have shifted online, looking for your next job has become more daunting than ever before. If you’re looking for your next opportunity in Data, take a look at our current jobs or get in touch with one of our expert consultants to find out more. 

How To Hire With Video Interviews

Virtual interviewing may have erupted over the last few months but the trends are showing that this is something that is likely to last well beyond the remote reality that many people are facing. Virtual interviewing is not as easy as it seems, in fact we’ve found our clients asking us over and over again for advice on how to run an effective video interview process. With that in mind, we’ve compiled a list of some of our top tips for clients and hiring managers for a successful video interview:  1. DON’T FORGET THE PRE-INTERVIEW PREP Confirm: Just like you would confirm a face to face interview with an email with the right address, instruction of how to get there and what to expect – the same applies for virtual interviews. Ensure to email candidates in advance with a link, information about who they are meeting and, most importantly, what you expect from a dress code. One of candidates biggest areas of concern is usually about what to wear for a virtual interview, so setting this out clearly in an email is a great way to start the process off on the right foot.Do not forget to provide instructions for using the video conferencing platform. Whether it is zoom, skype, google hangouts or another – keep in mind the candidate may not be familiar with your platform of choice.  Test:  Make sure to log onto to the interview early to ensure your camera, microphone and set up works. Be sure to ensure that your image is clear and that the volume is adequate. It is likely that the candidate will do the same and will ensure that the first few minutes of the interview aren’t focused on the technical side of things and ‘can you hear/see me?’.  2. PROVIDE A CLEAR STRUCTURE Opening: A usual face to face interview provides opportunity for warming a candidate up, however this time there is no shaking of hands and asking about commute.Just because you are video interviewing does not mean therefore that icebreakers shouldn’t exist, consider still incorporating an icebreaker to put the candidate at ease.  Ease concerns: One of the biggest concerns that candidates have when video interviewing is that there is a lot more out of their control in comparison to sitting in a meeting room opposite your interviewer. To ease any worries that the candidate might have, and to create a great candidate experience, let them know that background noise is okay and not to panic if the connection drops out. It’s likely that the candidate will have done everything they can to stop both of these from occurring, but ultimately, they could happen and it’s important the candidate knows that this will not negatively affect their outcome.  Set the agenda: Once you are through the icebreaker and have eased concerns, make sure to set an agenda for the interview. Let the candidate know what to expect. For example, introduction, CV run through, competency questions, Q&A and end. End the interview the right way, finish up by telling the candidates about the next steps and the timescales that you expect for that.  3. PREPARE THE QUESTIONS IN ADVANCE Due to the nature of video interviews, you will find the experience quite different to what you were used to. Usually you would have the CV and question sheet in front of you on the table, or on a laptop and the candidate separate to that. This time, you will potentially have all of that information on one screen. Preparing for how to optimise your screen and information therefore is important so that you can focus more on the candidate.  Read up on the candidate: Complete your CV read through and background prior to the interview to ensure that you do not need to rely wholly on the CV to make sense of the candidate’s answers. Don’t try and wing it: Prepare your questions in advance, have the questions in front of you and use them to help you to keep the interview on track and ensure all your questions get answered.  4. BE AWARE THAT EYE CONTACT IS DIFFERENT One of the biggest issues that clients and candidates alike feedback to us is that the concept of eye contact when video interviewing has as slightly different meaning. Having real eye contact in a virtual interview is challenging, because it means that you are going to be looking at the camera and not at the candidate, which takes some adjusting to.  Top Tips: Train yourself to look at the camera when you are talking, as this will give the candidate more of that personal feeling.Avoid the temptation to gape at your image on the screen, or the candidate when  you are speaking. If possible, turn off your picture so that the only image that shows on the screen is that of the candidate – this avoids the very familiar desire to look at oneself.  5. AVOID DIGITAL DISTRACTIONS There’s only so much you can do to stop your child running into the room, or your partner forgetting you’re on an interview and heading to the fridge but you can control the digital interruptions. It is important that you give the candidate your full attention. If your entire process is virtual, these are the sole ways that the candidate has to judge whether this is the right opportunity for them – so remember that this is a key part of their experience. Turn off notifications: Interviewing on a computer means that you are more likely to be distracted by your emails, IM messages, we’d advise turning off your notifications for both emails and IMs and closing all unnecessary tabs. Turn your phone onto airplane mode or DND. Harnham are currently supporting our clients within the Data & Analytics space on running completely remote interview processes for candidates. If you're looking to hire we can help you optimise your process in order to get the best talent then get in touch with one of our expert consultants. 

RELATED Jobs

Salary

US$120000 - US$140000 per annum + Additional Benefits

Location

McLean, Virginia

Description

An aviation industry partner is looking to bring on a Tableau Insights Analyst to join their Virginia office.

Salary

£50000 - £70000 per annum

Location

London

Description

An exciting role working for a technology-driven business who are in a growth stage. You will report directly into the Data Director and have lots of ownership

Salary

£50000 - £70000 per annum

Location

London

Description

This fintech brand received £200 million funding last year!

recently viewed jobs