The future of DW's in the age of Big Data

Kirsty Garshong our consultant managing the role
Posting date: 7/27/2013 2:47 PM

Many companies are saddled with data warehouses that weren’t designed to handle big data, but they can evolve their data warehouses into “analytics warehouses” capable of processing structured and unstructured data.

Enterprise data warehouses have reached a crossroads. Companies have spent millions of dollars designing, implementing, and updating them, but few organizations have realized the return they expected from their investments, according to Richard Solari, a director with Deloitte Consulting LLP’s Information Management service line.

The disappointing ROI largely stems from an inherent inadequacy in data warehouses: They were designed to handle the kind of structured data stored in ERP systems, not the unstructured data from social media, mobile devices, Web traffic, and other sources now streaming into enterprises. By Solari’s estimate, 90 percent of the data warehouses he’s observed process just 20 percent of an enterprise’s data. Consequently, many enterprises have only been able to use their data warehouses for historical analysis and past performance reporting.

The bottom line, says Solari: “Companies are using expensive infrastructure to generate back office reports.”

Organizations’ prospects for obtaining an acceptable return from their data warehousing investments may continue to diminish as long as this infrastructure fails to keep pace with big data.

The good news: Vendors are building new generations of data warehouses with advanced statistical capabilities for performing analytics and forecasting, according to Robert Stackowiak, Oracle Corp.’s vice president of information architecture and big data. They’re also improving integration with emerging platforms like Hadoop that process large volumes of unstructured data, he says.

Because newer generations of data warehouses are designed to federate structured and unstructured data, they may provide enterprises with a 360-degree view of their operations and, with that broader perspective, the ability to make better decisions about the future, according to Solari.

Companies running legacy data warehouses don’t have to junk their infrastructure and start anew. Solari says they can add capabilities to their existing data warehouse infrastructure that can allow it to grow into an “analytics warehouse.”

“Data warehouses are going to look very different in five years, and organizations should begin preparing for that transition,” says Solari.

Introducing the Analytics Warehouse

Fundamentally, the analytics warehouse functions as a central repository for an enterprise’s structured and unstructured data. In a traditional data warehousing architecture, structured data from ERP systems, CRM systems, file shares, and line of business applications is batch processed into the enterprise data warehouse using ETL (extract, transform, load) database processes. Software for running ad hoc queries and business intelligence systems take data from the warehouse environment, which may include operational data stores and data marts, to generate reports for users.

The architecture for the analytics warehouse builds on the traditional data warehouse architecture in three primary ways:

1. A distributed file system (like Hadoop) sits between source data systems and the data warehouse. It collects, aggregates, and processes huge volumes of unstructured data, and stages it for loading into the data warehouse.

2. Structured and unstructured data from back end systems can be brought into the data warehouse in real- and near-real time.

3. Engines that use statistical and predictive modeling techniques to perform data discovery, visualization, inductive and deductive reasoning, and real-time decision-making reside between the data warehouse and end users. These engines identify patterns in big data. They can also complement and feed traditional ad hoc querying tools and business intelligence applications.


“In the past, companies couldn’t integrate these disparate technologies with the data warehouse because each technology required different file formats and data schemas,” says Stackowiak. “Today, you can integrate these technologies, and the result is that companies can access more of their data—not just the 20 percent from enterprise systems—and convert it into valuable, profitable information.”

Companies interested in building out their traditional data warehouse infrastructures may consider starting with reporting, if they don’t already have reporting capabilities in place, suggests Solari. Then, they can begin integrating analytics technologies to their reporting framework.

“When companies start bringing this data together and federating it inside a data warehouse, the total cost of ownership for the data warehouse may begin to go down while the ROI goes up,” says Solari. “The ability to integrate big data technologies, analytics technologies, back office systems, and traditional data warehouses has the potential to fundamentally change the economics of data warehousing for the better.”


Click here for the article on the web.


Related blog & news

With over 10 years experience working solely in the Data & Analytics sector our consultants are able to offer detailed insights into the industry.

Visit our Blogs & News portal or check out the related posts below.

Weekly News Digest: 12th - 16th April 2021

This is Harnham’s weekly news digest, the place to come for a quick breakdown of the week’s top news stories from the world of Data & Analytics.      Express Pharma: The five biggest data challenges for life sciences Life Sciences has grown exponentially over the past 12 months. As the COVID-19 pandemic devastated the world, Life Science companies were in a race against time to create a life-changing vaccine and help us all back on the road to recovery.  In 2019, the Life Science market was valued at around $7.5bn. After this year’s influx of activity, the market is estimated to grow by over double in the next decade, reaching $18bn by 2030.  However, despite the positive growth the industry has had, this doesn’t mean Life Sciences will be free of challenges. In fact, with such a spike in the amount of data held by so many Life Science companies as they tried to work on a vaccine, data storage is now one of the main concerns for anyone working within the field.  In this article by Express Pharma, Vimal Venkatram, Country Manager for Snowflake India, highlights the five key data hurdles Life Sciences will continue to have to overcome in the following decade. These include data performance, data exchange and collaboration, data quality, data management and scaling, and regulatory compliance.  Read the full story here.  Harnham: How can organisations tap into the huge pool of neurodiverse data talent? For many companies, the past year has led to an increased focus on diversity and inclusion within businesses – a fantastic step forward. However, when we think of diversity, we usually assume people are talking about gender, ethnicity, sexuality and perhaps even physical disability. One area that is regularly missed from discussion is that of neurodiversity.  An umbrella term coined by sociologist, Judy Singer, neurodiversity can cover a wide range of neurological conditions such as dyslexia, autism, ADHD, ADD and dyspraxia. Our head of internal recruitment, Charlie Waterman, explores why neurodiverse talent shouldn’t be overlooked, and how Data & Analytics specifically can do more to tap into and harness this incredible pool of talent.` Exploring how employers can create a smooth recruitment process, successful onboarding programmes and retention schemes, this article highlights how all of this can be tailored to be accessible for anyone with an invisible disability. To read more on this topic, click here. Computer Weekly: What has a year of homeworking meant for the DPO? Employers in a significant number of industries across the world have had to uproot from the office to working from home because of the COVID-19 pandemic. For many of these employers, it appears that remote working, or a hybrid model of working, will become the norm post-pandemic.  But what has this sudden shift meant for the likes of Data Protection Officers (DPOs)? Most of these professionals have had to get to grips with managing and handling sensitive data from the comfort of their own living room. According to data from IBM, 70 per cent of DPOs believe that the shift to remote working will increase the likelihood of data breaches. So how can DPOs enjoy the benefits and perks of working from home, without the stress of poorly managed or breached data? In this article by Computer Weekly, steps are outlined on how DPOs can work closely with IT teams to minimise any data risk that could happen. This includes: Not allowing DPOs access to everything if it’s not necessaryDiscouraging local storage of dataRegularly reviewing security standards To read the full article, visit the website here.  Solutions Review: The three best Data Engineering books on our reading lists There’s no better feeling than getting stuck into a really good book. Not only can it be a great way to escape the stresses of everyday life, but by continuously absorbing new information, your knowledge on a specific subject can grow immensely.  Any branch of Data & Analytics, but especially Data Engineering, requires employees to always be thinking one step ahead, staying on top of new trends and keeping up to date with specific coding languages. While everyone learns in very different ways, reading is a brilliant education tool. Whether you’re a visual learner, an auditory learner or a reading learner, books and audiobooks could be the key to expanding your knowledge.  Solutions Review provides Data Engineers with three of the best books on the market at the moment to help you keep on top of your professional development. Data Driven Science and Engineering by Brunton and KutzData Engineering with Python by Crickard An introduction to agile Data Engineering by using data vault 2.0 by Graziano To read more about each of these books, click here.  We've loved seeing all the news from Data & Analytics in the past week, it’s a market full of exciting and dynamic opportunities. To learn more about our work in this space, get in touch with us at  info@harnham.com.    

How Can Organisations Tap Into The Huge Pool Of Neurodiverse Data Talent?

Ensuring that our workplaces are thriving with a diverse range of talent is, rightly, a topic that many organisations are focussing on. Yet, for the most part, this dialogue is centred around gender, ethnicity, sexuality and perhaps even physical disability. It is fairly uncommon therefore to see close attention given to exploring the challenges surrounding neurodiversity in organisations around the globe. Generally speaking, the term neurodiversity encompasses autism, attention deficit disorders, dyslexia, dyscalculia, dyspraxia and other neurological conditions. To hear a range of diverse viewpoints and perspectives is to contribute to an inclusive society and organisation. Leaving neurodiversity aside is no longer acceptable. Our research in the US highlights how 26 per cent of US adults have some form of disability, yet disabled individuals only account for 3.5 per cent of those working in Data & Analytics. As the global skills shortage worsens, it stands to reason that businesses will want to access this previously untapped talent pool. We know that in the UK, 56 per cent of organisations continue to experience skills shortages and in the US, two-thirds of employers hiring for full-time, permanent employees say they can’t find qualified talent to fill open jobs. An often-overlooked area of diversity is the impact a disability can have on an individual’s professional career. It’s no secret that all organisations would like to construct the best team – but are you doing enough to consider underrepresented talent? Creating a smooth recruitment and interview process One of the first barriers that neurodiverse candidates may encounter when seeking to enter an organisation is the recruitment and interview process. For these individuals, undergoing testing in this way puts pressure on communication skills, a tool that often allows us to better understand, connect and empathise with one another. When it comes to the recruitment process, the traditional in-person interview process — which assesses communication skills and personality fit — can be difficult to negotiate for neurodiverse candidates. In fact, this can be said to have been heightened by the pandemic too. The switch to virtual interviewing has added a new challenge to how neurodiverse candidates are able to participate in the process as miscommunication and interruptions come into the picture. For employers, tapping into the pool of data professionals with these invisible disabilities requires them to take the stress out of the interview and assessment process. It is critical to consider someone’s potential ability to do the job and the core skills that they have linking directly to the role on offer. Onboard a successful neurodiverse candidate efficiently Regardless of the size of an organisation, from global corporation to growing SME, they all share the same need to onboard new hires successfully and with limited disruption. It is this process that begins the relationship between an employee and an employer and although there will have been interactions through the recruitment process, it is the initial welcome into the organisation that will set the tone for the relationship moving forward. For neurodiverse employees this can be a daunting prospect; meeting new people while also familiarising themselves with a new environment and routine requires ongoing support and help from the employer. There are a number of ways that organisations can make this easier, from in-person or virtual meetings with smaller groups of the team to scheduled one-to-one chats with colleagues, the first few steps can be made more comfortable by promoting an inclusive culture. However, as there are such wide-ranging differences between neurodiverse conditions and individual requirements, employers need to implement policies that are tailored and highly individualised. Creating such policies and programmes can be complex and time-consuming, but it is critical to include your team in this. Ultimately it will boost your bottom line and the array of perspectives and views that are shared within the organisation. Retaining neurodiverse employees Neurodiverse candidates are capable, intelligent and have creative-thinking minds. To ensure their tenure within an organisation is lengthy and successful, we need to support these professionals and equip them with the tools and support they need to thrive. A standardised approach will not satisfy every need, and so it is important that every person in your organisation is accommodated as far as possible. The importance of this could not be clearer, as the BIMA Tech Inclusion & Diversity Report details how neurodivergent employees are more likely to be impacted by poor mental health (84 per cent against 49 per cent for neurotypical workers). This suggests that beyond attracting neurodiverse talent into the organisation, employers need to focus on the quality of the experience within the team. For example, take the time to book in regular meetings between the employee and their line manager. This will ensure that projects run smoothly, and any concerns or questions can be raised in a controlled environment. Listen to your team and their lived experiences to make informed and accurate plans to facilitate their growth within the team. After all, each employee brings a set of unique skills to a company. As more organisations realise the benefits of hiring neurodivergent candidates into their teams, employers have to act quickly to make routes into the business as accessible as possible. Ultimately, hiring neurodiverse people makes complete business sense. We know that diverse teams perform better, so now is the time to step up and tap into the huge pool of neurodiverse data talent. If you’re in the world of Data & Analytics and looking to take a step up or find the next member of your team, we can help. Take a look at our latest opportunities or get in touch with one of our expert consultants to find out more.

RELATED Jobs

Salary

£50000 - £60000 per annum + Benefits

Location

Hertfordshire

Description

This large Travel/Hotel brand has had extensive investment and they are looking for a Data Scientist to deliver actionable insights using statistical analysis.

Salary

£55000 - £60000 per annum + bonus and benefits

Location

Hertfordshire

Description

A market leader in the hospitality & leisure space are looking for a Data Scientist for their Hertfordshire office.

Salary

£35000 - £70000 per annum + bonus and benefits

Location

London

Description

A high-growth Food/Tech company are looking for multiple Supply Chain Analysts.

recently viewed jobs