Data is the new oil - crude oil



When Nasdaq stopped trading this week, it again showed how global firms are at the mercy of a power that created them

"Data is the new oil," declared Clive Humby, a Sheffield mathematician who with his wife, Edwina Dunn, made £90m helping Tesco with its Clubcard system. Though he said it in 2006, the realization that there is a lot of money to be made – and lost – through the careful or careless marshalling of "big data" has only begun to dawn on many business people.

The crash that knocked out the Nasdaq trading system was only one example; in the past week, Amazon, Google and Apple have all suffered breaks in service that have affected their customers, lost sales or caused inconvenience. When Amazon's main shopping site went offline for nearly an hour, estimates suggested millions of dollars of sales were lost. When Google went offline for just four minutes this month, the missed chance to show adverts to searchers could have cost it $500,000.

Michael Palmer, of the Association of National Advertisers, expanded on Humby's quote: "Data is just like crude. It's valuable, but if unrefined it cannot really be used. It has to be changed into gas, plastic, chemicals, etc to create a valuable entity that drives profitable activity; so must data be broken down, analyzed for it to have value."

For Amazon and Google especially, being able to process and store huge amounts of data is essential to their success. But when it goes wrong – as it inevitably does – the effects can be dramatic. And the biggest problem can be data which is "dirty", containing erroneous or garbled entries which can corrupt files and throw systems into a tailspin. That can cause the sort of "software glitch" that brought down the Nasdaq – or lead to servers locking up and a domino effect of overloading.

"Whenever I meet people I ask them about the quality of their data," says Duncan Ross, director of data sciences at Teradata, which provides data warehousing systems for clients including Walmart, Tesco and Apple. "When they tell me that the quality is really good, I assume that they haven't actually looked at it."

That's because the systems businesses use increasingly rely on external data, whether from governments or private companies, which cannot be assumed to be reliable. Ross says: "It's always dirty."

And that puts businesses at the mercy of the occasional high-pressure data spill. Inject the wrong piece of data and trouble follows. In April, when automatic systems read a tweet from the Associated Press Twitter feed which said the White House had been bombed and Barack Obama injured, they sold stock faster than the blink of an eye, sending the US Dow index down 143 points within seconds. But the data was dirty: AP's Twitter feed had been hacked.

The statistics are stunning: about 90% of all the data in the world has been generated in the past two years (a statistic that is holding roughly true even as time passes). There are about 2.7 zettabytes of data in the digital analytics universe, where 1ZB of data is a billion terabytes (a typical computer hard drive these days can hold about 0.5TB, or 500 gigabytes). IBM predicts that will hit 8ZB by 2015. Facebook alone stores and analyzes more than 50 petabytes (50,000 TB) of data.

Data is also moving faster than ever before: by last year, between 50% and 70% of all trades on US stock exchanges was being done by machines which could execute a transaction in less than a microsecond (millionth of a second). Internet connectivity is run through fibre optic connections where financial companies will seek to shave five milliseconds from a connection so those nanosecond-scale transactions can be done even more quickly.

We're also storing and processing more and more of it. But that doesn't mean we're just hoarding data, says Ross: "The pace of change of markets generally is so rapid that it doesn't make sense to retain information for more than a few years.

"If you think about something like handsets or phone calls, go back three or four years and the latest thing was the iPhone 3GS and BlackBerrys were really popular. It's useless for analysis. The only area where you store data for any length of time is regulatory work."

Yet the amount of short-term data being processed is rocketing. Twitter recently rewrote its entire back-end database system because it would not otherwise be able to cope with the 500m tweets, each as long as a text message, arriving each day. (By comparison, the four UK mobile networks together handle about 250m text messages a day, a figure is falling as people shift to services such as Twitter.)

Raffi Krikorian, Twitter's vice-president for "platform engineering" – that is, in charge of keeping the ship running, and the whale away – admits that the 2010 World Cup was a dramatic lesson, when goals, penalties and free kicks being watched by a global audience made the system creak and quail.

A wholesale rewrite of its back-end systems over the past three years means it can now "withstand" events such as the showing in Japan of a new film called Castle in the Sky, which set a record by generating 143,199 tweets a second on 2 August at 3.21pm BST. "The number of machines involved in serving the site has been decreased anywhere from five to 12 times," he notes proudly. Even better, Twitter has been available for about 99.9999% of the past six months, even with that Japanese peak.

Yet even while Twitter moved quickly, the concern is that other parts of the information structure will not be resilient enough to deal with inevitable collapses – and that could have unpredictable effects.

"We've had mains power for more than a century, but can have an outage caused by somebody not resetting a switch," says Ross. "The only security companies can have is if they build plenty of redundancy into the systems that affect our lives."


Click here for the article on the web.



<< Click here to see more recent news articles >>

 

 

Harnham blog & news

With over 10 years experience working solely in the Data & Analytics sector our consultants are able to offer detailed insights into the industry.

Visit our News & Blogs portal or check out our recent posts below.

Leading Remote Teams in the Land of Beer, Barbeque, and Tech

Charlotte, North Carolina known for its beer and barbeque is not the first location to jump to mind when you think digital hub. But here’s the rub, it is. In the Fall of 2019, the city and Microsoft signed a three-year digital alliance. And as the country moved from office locations to work-from-home and remote operations, Charlotte became a prime destination for tech.  Four Skills for Leading in the New Normal Whether you’ve learned to balance work time with virtual schooling or have been working from home for years, there are some skillsets which set leaders apart. The first one may surprise you. Let others lead – According to our most recent salary guide, one of the main reasons people leave their jobs is due to poor management. With the rise of remote working, hierarchy has flattened to a degree as everyone must discipline themselves. Micromanagement becomes almost moot as everyone leans into this learning curve. Lead by example and let others take over the leadership driver’s seat from time-to-time. Balance both Soft and Technical Skills – While technical skills are the backbone of subject matter experience in a Digital Analytics role, it’s the soft skills which can help set you apart. Sure, you’ll want to know the ins and outs of web analytics and optimization, but you’ll also need to have the skills to explain findings and offer recommendations to address client needs. Know When to Pivot – Life throws us curveballs. Consider 2020, for example. Whether you must pivot for survival or simply need to take things in a new direction, knowing when to pivot and how to explain it to your employees is a leadership skill inherent for this new normal. Be Approachable – With open door policies moved online, leaders will want to determine the best way to recreate opportunities for employees that need to talk outside the daily or weekly staff meeting. Having insight into how your team works best, can help you guide them toward success. Programs to know and experience to have often include the technical knowledge you’ll need to ensure your client makes the most informed decisions.  ‘Smart Cities’ of the Future At the beginning of 2020, as the coronavirus came to call, someone joked that in 2020 we’d hoped to have flying cars and smart cities of the future, but instead were being taught how to wash our hands. Perhaps they weren’t far off on the smart cities quip as both Houston, Texas and Charlotte, North Carolina are on the cutting edge of creating these smart cities. Things like Massive Open Online Courses (MOOCs), LinkedIn Learning, even the Entrepreneur Store offers classes and bundles in everything from computer language learning such as C++, R, and Python to Digital Marketing and Graphic Design. But, learning these things and more on a Microsoft campus can catapult students into more jobs and helps guide cities in smart power grids, smart busses, autonomous cars, and the list goes on. Experience in a design agency who works within both the B2C and the B2B verticals helps to expand opportunities exponentially. Remote working opportunities have opened up worlds of collaboration, teamwork, and focus on the next steps into the future. Whether you’re interested in a remote working leadership role in the beachy Carolinas or looking for your first or next role in Big Data, Web Analytics, Marketing & Insight, Life Science Analytics, and more, check out our current vacancies or contact one of our recruitment consultants to learn more.   For our West Coast Team, contact us at (415) 614 - 4999 or send an email to sanfraninfo@harnham.com.   For our Mid-West and East Coast teams contact us at (212) 796-6070 or send an email to newyorkinfo@harnham.com.  

Career Priorities Are Shifting For The US Data & Analytics Market Place

Positions in Data & Analytics are highly sought after in the US. With a widening talent pool due to the rise of flexible working, shifting skillsets and requirements for more professionals working in data and tech, the sector continues to experience unprecedented growth; regardless of the global pandemic pulling at the very core of the business community. For candidates, the Data & Analytics job market will be more active than ever. In our most recent US Data & Analytics Salary Guide, this was clearly identified. In fact, post-COVID, more respondents were either actively seeking, or willing to move for, a new role than they were in 2019. As businesses look to streamline existing processes and establish new ones, they are more reliant on Data professionals than ever before. Yet, candidates in this market have re-evaluated their needs and the incentives they value, shifting their priorities in the wake of COVID-19. With the impact of COVID-19 felt in almost every part of our day-to-day lives, it wasn’t long until conversation turned to what ‘the new normal’ would look like once the worst of the pandemic had passed. However, for those in the Data & Analytics space, the new normal wasn’t as new as first thought - it had been gradually establishing itself over the past few years. When seeking a new role, respondents prioritized career progression over salary increases, as well as an emphasis on job security and working for a stable and growing business. The needs of the multi-generational workforce in a post-COVID world has never been more dynamic than it is today. It was evidenced that financial ambitions have decreased slightly as a result of the pandemic. Interestingly, ‘a more competitive salary’ was the top reason for seeking a new role. Post-COVID, not only do respondents seek career progression over a salary increase, they also put a significantly greater emphasis on job security and the need to join a stable and growing business.  Specifically, for Data Engineers in the current market space, 80 per cent of professionals would leave their role if the right opportunity came up. This falls in parallel to digital analytics experts, of which 82 per cent would leave their role if the right opportunity came up. This is perhaps unsurprising as the full economic impact of the pandemic has yet to be established. Attracting the best hire for your business is going to be one of the core challenges of our time. Period. The focus for American organizations needs to be on hiring for the needs of tomorrow, rather than for the demands of today. A forward-thinking business will have the critical vision needed in order to keep pace with the market in its current state. Data professionals are in the driving seat; that’s a certainty. Their holistic skillset, understanding and application of core technologies such as SQL, Python, AWS, Kubernetes and Spark position them well when seeking to secure their next role. It is up to organizations to maintain their momentum as we acknowledge that for Data & Analytics, continuing on the trajectory of growth in the ‘new normal’ is something very few outside of this space saw coming. If you're looking for a new opportunity in the world of Data & Analytics or to expand your Data team, we may be able to help. Take a look at our latest opportunities or get in touch with one of our expert consultants to find out more: For our West Coast Team, contact us at (415) 614 - 4999 or send an email to sanfraninfo@harnham.com.    For our Mid-West and East Coast teams contact us at (212) 796-6070 or send an email to newyorkinfo@harnham.com.  

Navigating the New Normal Now and In the Future

At the beginning of every year and particularly the start of a new decade, we often find ourselves reassessing our priorities. Though this year was no different, it does have the twist of a pandemic. Businesses and individuals were forced to pivot toward a new normal.  Together, we’ve moved quickly and one thing this year’s Salary Guide has shown us is that priorities have changed. In previous years, focus was on higher salaries and it was expected to stay in a position only two or three years before moving on to the next job. But this year, stability is the name of the game. And so, priorities shift. Candidates now want to stay in a role that could lead to career progression.   What’s Changed?  In the Data & Analytics industry, flexible working options have steadily increased, though it was already a way of life for many. Add into the mix, the increased need to work from home for employee safety or in controlled environments for those whose role did not allow for remote working such as those in the life sciences.  Financial considerations were no longer top dog in hiring and retaining top talent. Longevity, career progression, and good management are highly desired. When everyone is online and flexible working options are the new normal, it’s important to have strong leadership.  Four Future Changes to Come   With the world online, working, and learning from home, bandwidth has become a non-renewable resource. Too much traffic. Too many people online could cause issues particularly during prime working and learning hours. Though with everything online, what does it mean to have prime hours now?  Setting Priorities – Just because you’re working from home doesn’t mean you should lock yourself in your office for hours on end. Set aside time for food, family, and fun. Elevate the Home Office - Larger, more stable devices may see a resurgence. The home office is truly just that with the standard desktop and monitor to more easily see information. For those in many industries, two screens really help set the tone. Flexible Shopping Options are Here to Stay – When you’re working and learning from home, time is of the essence. The benefits of delivery, pick-up, and even some more expanded food centers could change the way we eat and gather. Retail is being redesigned for the new normal.Logistics of Social Distancing meet Machine Learning – As we focus on social distancing, mobile applications may shift toward a more logistics focused future using crowdsourcing and Big Data. From contact tracing to food buying, it will be important to have technologies that can keep up with people at all hours of the day and night to deliver goods and services.  In our recently released 2020 Salary Guide we discuss each specialism, what’s working and what isn’t. And how businesses can hire and retain top talent to keep their projects on track and their businesses running smoothly. If you’re interested in Data and Technology, Risk or Digital Analytics, Life Sciences Analytics, Marketing and Insight, Data Science, or Computer Vision, take a look at our current vacancies. If you’d like to learn more, get in touch with one of our expert consultants. For our West Coast Team, contact us at (415) 614 - 4999 or send an email to sanfraninfo@harnham.com.    For our Mid-West and East Coast teams contact us at (212) 796-6070 or send an email to newyorkinfo@harnham.com.  

How Digital Analytics & Marketing Help Shape E-Commerce

Since the days of store window displays, the shopping experience has been just that; an experience. So, what happens when you want to recreate the visceral and emotional experience of shopping from within the confines of a computer screen, smartphone, or tablet? Enter Digital and Marketing Analytics. According to a recent report by Adobe, e-commerce sales have seen a 77% jump year-over-year which at any other time in history should have taken 4 to 6 years. The COVID-19 pandemic has helped to escalate and drive these numbers leaving some businesses scrambling and others raking it in. So, what’s the difference? Data professionals. The Role of Data in E-Commerce Whether you buy online and pick up in store or have a product delivered to your door, the role of shopping has irrevocably changed. Ensuring consumers get the personalized experience they’re used to from the days of brick-and-mortar stores, retail stores have turned to Digital and Marketing Analytics to give their customers the shopping experience they’re used to within a different format. Professionals within the UI/UX Design vertical are particularly sought after. Buying habits are changing and competition is fierce. So, how does Data affect e-commerce? Here are 3 examples: Know Your Customer. No longer a lamented visage from yesteryear, knowing your customer is inherent to the survival of an online retailer.  Data professionals bring to life the customer through historical data, demographics, and creates products and services which elicit an emotional response to stop, look around, and buy. And if the cart gets abandoned, follow up email campaigns to jog the memory that you were here and were thinking about buying this or that item. Get Personal. Personalize shopping has evolved into the personal shopping experience. What made someone buy a product during their last visit? Dynamic presentation and emotion-driven verbiage can certainly contribute.  When an online retailer knows what the buyer wants or needs and presents it in a way that resonates. This is the personalization once the domain of sales clerks.  From the Data you enter at checkout to the social media platforms and search engines with information like what you’re looking for, where you’re located, your purchase history and more can sometimes leave the personalized experience out in the cold. Bridging the old personalization with the new is the key ingredient for successful online retailers. Sell Where Your Consumers are Buying. If your customers are on Facebook looking for a product or service, sell to them where they’re looking. Though reviews are still important, even more prevalent is the range of social influencers to help buyers make decisions. Enter social commerce. Layout, design, Data gathering, collecting, and analyzing all have a slightly different flavor within this construct.  Data Professionals in Demand Within the Digital Analytics and Marketing specialism are a variety of Data professionals in demand as retailers are forced online. Businesses need E-Commerce Analysts to help present the new normal for the online retailer. As buy online and pick up in store convenience increases as well as purchases made from devices such as your smartphone, there’s been a rise in demand for Web & Mobile Product Management as well. Digital Transformation is no longer at the doorstep of business, it has crossed the threshold. And rather than focus on one tool or another, the impetus is on the importance of having more than one tool at a candidate’s disposal. What candidates want has changed and evolved as well. Salary and bonuses have dwindled while the demand for experience and professional development have become rallying cries for finding and retaining top talent. For more information In our recently released 2020 Salary Guide we discuss each specialism. What’s working. What isn’t. And how businesses can hire and retain top talent to keep their projects on track and their businesses running smoothly. You can download your copy here.  If you’re interested in Data and Technology, Risk or Digital Analytics, Life Sciences Analytics, Advanced Analytics & Insight, Data Science, or Computer Vision, we invite you to check out our latest jobs. If you’d like to learn more, contact one of our expert consultants: For our West Coast Team, contact us at (415) 614 - 4999 or send an email to sanfraninfo@harnham.com.   For our Mid-West and East Coast teams contact us at (212) 796-6070 or send an email to newyorkinfo@harnham.com.  

Recently Viewed jobs