The Future of DW's in the Age of Big Data



Many companies are saddled with data warehouses that weren’t designed to handle big data, but they can evolve their data warehouses into “analytics warehouses” capable of processing structured and unstructured data.

Enterprie data warehouses have reached a crossroads. Companies have spent millions of dollars designing, implementing, and updating them, but few organizations have realized the return they expected from their investments, according to Richard Solari, a director with Deloitte Consulting LLP’s Information Management service line.

The disappointing ROI largely stems from an inherent inadequacy in data warehouses: They were designed to handle the kind of structured data stored in ERP systems, not the unstructured data from social media, mobile devices, Web traffic, and other sources now streaming into enterprises. By Solari’s estimate, 90 percent of the data warehouses he’s observed process just 20 percent of an enterprise’s data. Consequently, many enterprises have only been able to use their data warehouses for historical analysis and past performance reporting.

The bottom line, says Solari: “Companies are using expensive infrastructure to generate back office reports.”

Organizations’ prospects for obtaining an acceptable return from their data warehousing investments may continue to diminish as long as this infrastructure fails to keep pace with big data.

The good news: Vendors are building new generations of data warehouses with advanced statistical capabilities for performing analytics and forecasting, according to Robert Stackowiak, Oracle Corp.’s vice president of information architecture and big data. They’re also improving integration with emerging platforms like Hadoop that process large volumes of unstructured data, he says.

Because newer generations of data warehouses are designed to federate structured and unstructured data, they may provide enterprises with a 360-degree view of their operations and, with that broader perspective, the ability to make better decisions about the future, according to Solari.

Companies running legacy data warehouses don’t have to junk their infrastructure and start anew. Solari says they can add capabilities to their existing data warehouse infrastructure that can allow it to grow into an “analytics warehouse.”

“Data warehouses are going to look very different in five years, and organizations should begin preparing for that transition,” says Solari.

Introducing the Analytics Warehouse

Fundamentally, the analytics warehouse functions as a central repository for an enterprise’s structured and unstructured data. In a traditional data warehousing architecture, structured data from ERP systems, CRM systems, file shares, and line of business applications is batch processed into the enterprise data warehouse using ETL (extract, transform, load) database processes. Software for running ad hoc queries and business intelligence systems take data from the warehouse environment, which may include operational data stores and data marts, to generate reports for users.

The architecture for the analytics warehouse builds on the traditional data warehouse architecture in three primary ways:

1. A distributed file system (like Hadoop) sits between source data systems and the data warehouse. It collects, aggregates, and processes huge volumes of unstructured data, and stages it for loading into the data warehouse.

2. Structured and unstructured data from back end systems can be brought into the data warehouse in real- and near-real time.

3. Engines that use statistical and predictive modeling techniques to perform data discovery, visualization, inductive and deductive reasoning, and real-time decision-making reside between the data warehouse and end users. These engines identify patterns in big data. They can also complement and feed traditional ad hoc querying tools and business intelligence applications.

“In the past, companies couldn’t integrate these disparate technologies with the data warehouse because each technology required different file formats and data schemas,” says Stackowiak. “Today, you can integrate these technologies, and the result is that companies can access more of their data—not just the 20 percent from enterprise systems—and convert it into valuable, profitable information.”

Companies interested in building out their traditional data warehouse infrastructures may consider starting with reporting, if they don’t already have reporting capabilities in place, suggests Solari. Then, they can begin integrating analytics technologies to their reporting framework.

“When companies start bringing this data together and federating it inside a data warehouse, the total cost of ownership for the data warehouse may begin to go down while the ROI goes up,” says Solari. “The ability to integrate big data technologies, analytics technologies, back office systems, and traditional data warehouses has the potential to fundamentally change the economics of data warehousing for the better.”


Click here for the article on the web.



<< Click here to see more recent news articles >>

 

 

Harnham blog & news

With over 10 years experience working solely in the Data & Analytics sector our consultants are able to offer detailed insights into the industry.

Visit our News & Blogs portal or check out our recent posts below.

Leading Remote Teams in the Land of Beer, Barbeque, and Tech

Charlotte, North Carolina known for its beer and barbeque is not the first location to jump to mind when you think digital hub. But here’s the rub, it is. In the Fall of 2019, the city and Microsoft signed a three-year digital alliance. And as the country moved from office locations to work-from-home and remote operations, Charlotte became a prime destination for tech.  Four Skills for Leading in the New Normal Whether you’ve learned to balance work time with virtual schooling or have been working from home for years, there are some skillsets which set leaders apart. The first one may surprise you. Let others lead – According to our most recent salary guide, one of the main reasons people leave their jobs is due to poor management. With the rise of remote working, hierarchy has flattened to a degree as everyone must discipline themselves. Micromanagement becomes almost moot as everyone leans into this learning curve. Lead by example and let others take over the leadership driver’s seat from time-to-time. Balance both Soft and Technical Skills – While technical skills are the backbone of subject matter experience in a Digital Analytics role, it’s the soft skills which can help set you apart. Sure, you’ll want to know the ins and outs of web analytics and optimization, but you’ll also need to have the skills to explain findings and offer recommendations to address client needs. Know When to Pivot – Life throws us curveballs. Consider 2020, for example. Whether you must pivot for survival or simply need to take things in a new direction, knowing when to pivot and how to explain it to your employees is a leadership skill inherent for this new normal. Be Approachable – With open door policies moved online, leaders will want to determine the best way to recreate opportunities for employees that need to talk outside the daily or weekly staff meeting. Having insight into how your team works best, can help you guide them toward success. Programs to know and experience to have often include the technical knowledge you’ll need to ensure your client makes the most informed decisions.  ‘Smart Cities’ of the Future At the beginning of 2020, as the coronavirus came to call, someone joked that in 2020 we’d hoped to have flying cars and smart cities of the future, but instead were being taught how to wash our hands. Perhaps they weren’t far off on the smart cities quip as both Houston, Texas and Charlotte, North Carolina are on the cutting edge of creating these smart cities. Things like Massive Open Online Courses (MOOCs), LinkedIn Learning, even the Entrepreneur Store offers classes and bundles in everything from computer language learning such as C++, R, and Python to Digital Marketing and Graphic Design. But, learning these things and more on a Microsoft campus can catapult students into more jobs and helps guide cities in smart power grids, smart busses, autonomous cars, and the list goes on. Experience in a design agency who works within both the B2C and the B2B verticals helps to expand opportunities exponentially. Remote working opportunities have opened up worlds of collaboration, teamwork, and focus on the next steps into the future. Whether you’re interested in a remote working leadership role in the beachy Carolinas or looking for your first or next role in Big Data, Web Analytics, Marketing & Insight, Life Science Analytics, and more, check out our current vacancies or contact one of our recruitment consultants to learn more.   For our West Coast Team, contact us at (415) 614 - 4999 or send an email to sanfraninfo@harnham.com.   For our Mid-West and East Coast teams contact us at (212) 796-6070 or send an email to newyorkinfo@harnham.com.  

Career Priorities Are Shifting For The US Data & Analytics Market Place

Positions in Data & Analytics are highly sought after in the US. With a widening talent pool due to the rise of flexible working, shifting skillsets and requirements for more professionals working in data and tech, the sector continues to experience unprecedented growth; regardless of the global pandemic pulling at the very core of the business community. For candidates, the Data & Analytics job market will be more active than ever. In our most recent US Data & Analytics Salary Guide, this was clearly identified. In fact, post-COVID, more respondents were either actively seeking, or willing to move for, a new role than they were in 2019. As businesses look to streamline existing processes and establish new ones, they are more reliant on Data professionals than ever before. Yet, candidates in this market have re-evaluated their needs and the incentives they value, shifting their priorities in the wake of COVID-19. With the impact of COVID-19 felt in almost every part of our day-to-day lives, it wasn’t long until conversation turned to what ‘the new normal’ would look like once the worst of the pandemic had passed. However, for those in the Data & Analytics space, the new normal wasn’t as new as first thought - it had been gradually establishing itself over the past few years. When seeking a new role, respondents prioritized career progression over salary increases, as well as an emphasis on job security and working for a stable and growing business. The needs of the multi-generational workforce in a post-COVID world has never been more dynamic than it is today. It was evidenced that financial ambitions have decreased slightly as a result of the pandemic. Interestingly, ‘a more competitive salary’ was the top reason for seeking a new role. Post-COVID, not only do respondents seek career progression over a salary increase, they also put a significantly greater emphasis on job security and the need to join a stable and growing business.  Specifically, for Data Engineers in the current market space, 80 per cent of professionals would leave their role if the right opportunity came up. This falls in parallel to digital analytics experts, of which 82 per cent would leave their role if the right opportunity came up. This is perhaps unsurprising as the full economic impact of the pandemic has yet to be established. Attracting the best hire for your business is going to be one of the core challenges of our time. Period. The focus for American organizations needs to be on hiring for the needs of tomorrow, rather than for the demands of today. A forward-thinking business will have the critical vision needed in order to keep pace with the market in its current state. Data professionals are in the driving seat; that’s a certainty. Their holistic skillset, understanding and application of core technologies such as SQL, Python, AWS, Kubernetes and Spark position them well when seeking to secure their next role. It is up to organizations to maintain their momentum as we acknowledge that for Data & Analytics, continuing on the trajectory of growth in the ‘new normal’ is something very few outside of this space saw coming. If you're looking for a new opportunity in the world of Data & Analytics or to expand your Data team, we may be able to help. Take a look at our latest opportunities or get in touch with one of our expert consultants to find out more: For our West Coast Team, contact us at (415) 614 - 4999 or send an email to sanfraninfo@harnham.com.    For our Mid-West and East Coast teams contact us at (212) 796-6070 or send an email to newyorkinfo@harnham.com.  

Navigating the New Normal Now and In the Future

At the beginning of every year and particularly the start of a new decade, we often find ourselves reassessing our priorities. Though this year was no different, it does have the twist of a pandemic. Businesses and individuals were forced to pivot toward a new normal.  Together, we’ve moved quickly and one thing this year’s Salary Guide has shown us is that priorities have changed. In previous years, focus was on higher salaries and it was expected to stay in a position only two or three years before moving on to the next job. But this year, stability is the name of the game. And so, priorities shift. Candidates now want to stay in a role that could lead to career progression.   What’s Changed?  In the Data & Analytics industry, flexible working options have steadily increased, though it was already a way of life for many. Add into the mix, the increased need to work from home for employee safety or in controlled environments for those whose role did not allow for remote working such as those in the life sciences.  Financial considerations were no longer top dog in hiring and retaining top talent. Longevity, career progression, and good management are highly desired. When everyone is online and flexible working options are the new normal, it’s important to have strong leadership.  Four Future Changes to Come   With the world online, working, and learning from home, bandwidth has become a non-renewable resource. Too much traffic. Too many people online could cause issues particularly during prime working and learning hours. Though with everything online, what does it mean to have prime hours now?  Setting Priorities – Just because you’re working from home doesn’t mean you should lock yourself in your office for hours on end. Set aside time for food, family, and fun. Elevate the Home Office - Larger, more stable devices may see a resurgence. The home office is truly just that with the standard desktop and monitor to more easily see information. For those in many industries, two screens really help set the tone. Flexible Shopping Options are Here to Stay – When you’re working and learning from home, time is of the essence. The benefits of delivery, pick-up, and even some more expanded food centers could change the way we eat and gather. Retail is being redesigned for the new normal.Logistics of Social Distancing meet Machine Learning – As we focus on social distancing, mobile applications may shift toward a more logistics focused future using crowdsourcing and Big Data. From contact tracing to food buying, it will be important to have technologies that can keep up with people at all hours of the day and night to deliver goods and services.  In our recently released 2020 Salary Guide we discuss each specialism, what’s working and what isn’t. And how businesses can hire and retain top talent to keep their projects on track and their businesses running smoothly. If you’re interested in Data and Technology, Risk or Digital Analytics, Life Sciences Analytics, Marketing and Insight, Data Science, or Computer Vision, take a look at our current vacancies. If you’d like to learn more, get in touch with one of our expert consultants. For our West Coast Team, contact us at (415) 614 - 4999 or send an email to sanfraninfo@harnham.com.    For our Mid-West and East Coast teams contact us at (212) 796-6070 or send an email to newyorkinfo@harnham.com.  

How Digital Analytics & Marketing Help Shape E-Commerce

Since the days of store window displays, the shopping experience has been just that; an experience. So, what happens when you want to recreate the visceral and emotional experience of shopping from within the confines of a computer screen, smartphone, or tablet? Enter Digital and Marketing Analytics. According to a recent report by Adobe, e-commerce sales have seen a 77% jump year-over-year which at any other time in history should have taken 4 to 6 years. The COVID-19 pandemic has helped to escalate and drive these numbers leaving some businesses scrambling and others raking it in. So, what’s the difference? Data professionals. The Role of Data in E-Commerce Whether you buy online and pick up in store or have a product delivered to your door, the role of shopping has irrevocably changed. Ensuring consumers get the personalized experience they’re used to from the days of brick-and-mortar stores, retail stores have turned to Digital and Marketing Analytics to give their customers the shopping experience they’re used to within a different format. Professionals within the UI/UX Design vertical are particularly sought after. Buying habits are changing and competition is fierce. So, how does Data affect e-commerce? Here are 3 examples: Know Your Customer. No longer a lamented visage from yesteryear, knowing your customer is inherent to the survival of an online retailer.  Data professionals bring to life the customer through historical data, demographics, and creates products and services which elicit an emotional response to stop, look around, and buy. And if the cart gets abandoned, follow up email campaigns to jog the memory that you were here and were thinking about buying this or that item. Get Personal. Personalize shopping has evolved into the personal shopping experience. What made someone buy a product during their last visit? Dynamic presentation and emotion-driven verbiage can certainly contribute.  When an online retailer knows what the buyer wants or needs and presents it in a way that resonates. This is the personalization once the domain of sales clerks.  From the Data you enter at checkout to the social media platforms and search engines with information like what you’re looking for, where you’re located, your purchase history and more can sometimes leave the personalized experience out in the cold. Bridging the old personalization with the new is the key ingredient for successful online retailers. Sell Where Your Consumers are Buying. If your customers are on Facebook looking for a product or service, sell to them where they’re looking. Though reviews are still important, even more prevalent is the range of social influencers to help buyers make decisions. Enter social commerce. Layout, design, Data gathering, collecting, and analyzing all have a slightly different flavor within this construct.  Data Professionals in Demand Within the Digital Analytics and Marketing specialism are a variety of Data professionals in demand as retailers are forced online. Businesses need E-Commerce Analysts to help present the new normal for the online retailer. As buy online and pick up in store convenience increases as well as purchases made from devices such as your smartphone, there’s been a rise in demand for Web & Mobile Product Management as well. Digital Transformation is no longer at the doorstep of business, it has crossed the threshold. And rather than focus on one tool or another, the impetus is on the importance of having more than one tool at a candidate’s disposal. What candidates want has changed and evolved as well. Salary and bonuses have dwindled while the demand for experience and professional development have become rallying cries for finding and retaining top talent. For more information In our recently released 2020 Salary Guide we discuss each specialism. What’s working. What isn’t. And how businesses can hire and retain top talent to keep their projects on track and their businesses running smoothly. You can download your copy here.  If you’re interested in Data and Technology, Risk or Digital Analytics, Life Sciences Analytics, Advanced Analytics & Insight, Data Science, or Computer Vision, we invite you to check out our latest jobs. If you’d like to learn more, contact one of our expert consultants: For our West Coast Team, contact us at (415) 614 - 4999 or send an email to sanfraninfo@harnham.com.   For our Mid-West and East Coast teams contact us at (212) 796-6070 or send an email to newyorkinfo@harnham.com.  

Recently Viewed jobs