DATA IS THE NEW OIL - CRUDE OIL

Krishen Patel our consultant managing the role
Posting date: 9/2/2013 2:16 PM

When Nasdaq stopped trading this week, it again showed how global firms are at the mercy of a power that created them

"Data is the new oil," declared Clive Humby, a Sheffield mathematician who with his wife, Edwina Dunn, made £90m helping Tesco with its Clubcard system. Though he said it in 2006, the realization that there is a lot of money to be made – and lost – through the careful or careless marshalling of "big data" has only begun to dawn on many business people.

The crash that knocked out the Nasdaq trading system was only one example; in the past week, Amazon, Google and Apple have all suffered breaks in service that have affected their customers, lost sales or caused inconvenience. When Amazon's main shopping site went offline for nearly an hour, estimates suggested millions of dollars of sales were lost. When Google went offline for just four minutes this month, the missed chance to show adverts to searchers could have cost it $500,000.

Michael Palmer, of the Association of National Advertisers, expanded on Humby's quote: "Data is just like crude. It's valuable, but if unrefined it cannot really be used. It has to be changed into gas, plastic, chemicals, etc to create a valuable entity that drives profitable activity; so must data be broken down, analyzed for it to have value."

For Amazon and Google especially, being able to process and store huge amounts of data is essential to their success. But when it goes wrong – as it inevitably does – the effects can be dramatic. And the biggest problem can be data which is "dirty", containing erroneous or garbled entries which can corrupt files and throw systems into a tailspin. That can cause the sort of "software glitch" that brought down the Nasdaq – or lead to servers locking up and a domino effect of overloading.

"Whenever I meet people I ask them about the quality of their data," says Duncan Ross, director of data sciences at Teradata, which provides data warehousing systems for clients including Walmart, Tesco and Apple. "When they tell me that the quality is really good, I assume that they haven't actually looked at it."

That's because the systems businesses use increasingly rely on external data, whether from governments or private companies, which cannot be assumed to be reliable. Ross says: "It's always dirty."

And that puts businesses at the mercy of the occasional high-pressure data spill. Inject the wrong piece of data and trouble follows. In April, when automatic systems read a tweet from the Associated Press Twitter feed which said the White House had been bombed and Barack Obama injured, they sold stock faster than the blink of an eye, sending the US Dow index down 143 points within seconds. But the data was dirty: AP's Twitter feed had been hacked.

The statistics are stunning: about 90% of all the data in the world has been generated in the past two years (a statistic that is holding roughly true even as time passes). There are about 2.7 zettabytes of data in the digital analytics universe, where 1ZB of data is a billion terabytes (a typical computer hard drive these days can hold about 0.5TB, or 500 gigabytes). IBM predicts that will hit 8ZB by 2015. Facebook alone stores and analyzes more than 50 petabytes (50,000 TB) of data.

Data is also moving faster than ever before: by last year, between 50% and 70% of all trades on US stock exchanges was being done by machines which could execute a transaction in less than a microsecond (millionth of a second). Internet connectivity is run through fibre optic connections where financial companies will seek to shave five milliseconds from a connection so those nanosecond-scale transactions can be done even more quickly.

We're also storing and processing more and more of it. But that doesn't mean we're just hoarding data, says Ross: "The pace of change of markets generally is so rapid that it doesn't make sense to retain information for more than a few years.

"If you think about something like handsets or phone calls, go back three or four years and the latest thing was the iPhone 3GS and BlackBerrys were really popular. It's useless for analysis. The only area where you store data for any length of time is regulatory work."

Yet the amount of short-term data being processed is rocketing. Twitter recently rewrote its entire back-end database system because it would not otherwise be able to cope with the 500m tweets, each as long as a text message, arriving each day. (By comparison, the four UK mobile networks together handle about 250m text messages a day, a figure is falling as people shift to services such as Twitter.)

Raffi Krikorian, Twitter's vice-president for "platform engineering" – that is, in charge of keeping the ship running, and the whale away – admits that the 2010 World Cup was a dramatic lesson, when goals, penalties and free kicks being watched by a global audience made the system creak and quail.

A wholesale rewrite of its back-end systems over the past three years means it can now "withstand" events such as the showing in Japan of a new film called Castle in the Sky, which set a record by generating 143,199 tweets a second on 2 August at 3.21pm BST. "The number of machines involved in serving the site has been decreased anywhere from five to 12 times," he notes proudly. Even better, Twitter has been available for about 99.9999% of the past six months, even with that Japanese peak.

Yet even while Twitter moved quickly, the concern is that other parts of the information structure will not be resilient enough to deal with inevitable collapses – and that could have unpredictable effects.

"We've had mains power for more than a century, but can have an outage caused by somebody not resetting a switch," says Ross. "The only security companies can have is if they build plenty of redundancy into the systems that affect our lives."


Click here for the article on the web.

Related blog & news

With over 10 years experience working solely in the Data & Analytics sector our consultants are able to offer detailed insights into the industry.

Visit our Blogs & News portal or check out the related posts below.

Three Years Of GDPR: The Evolution Of Data Protection

Since its inception in 1991, the World Wide Web – or the internet – has grown immeasurably, with its capabilities exceeding the expectations of anyone who witnessed its implementation only 30 years ago. Now, it’s hard to think of a world without it; where would we be without unlimited knowledge at the touch of a button, the ability to maintain friendships with people halfway across the world or cat videos? Of course, the internet isn’t always a positive place. As the popularity of the online world grew, there also became an increased risk, particularly to our identities and our money.  In 1998, to combat the mismanagement of data both online and offline, Parliament passed the Data Protection Act. Compiled of eight different principles, from fair and lawful processing to disallowing data transfers from outside of the EU, this law aimed to help reduce the risk of data mismanagement and data breaches, while holding the power to fine and prosecute those who didn’t comply.  In January 2012, the European Commission wanted to take these laws one step further. As we began to enter a digital-first age, where the online world began to blend seamlessly with our daily lives, questions around whether the Data Protection Act of 1998 was robust enough to protect EU citizens.  On May 25th, 2018, the General Data Protection Regulation (GDPR) was introduced. Not only did this new law enforce tougher rules around data protection, including the protection of genetic data and biometrics, but it made business data collection far more transparent. For the first time, internet users were able to see exactly how and why their data was being used, and they were given the autonomy to opt-out of giving away sensitive data. Additionally, consumers now have the right to request ‘to be forgotten’, with all stored data being wiped from a business’ database with the click of a button.  As we edge closer to the three-year anniversary of the implementation of GDPR, we look at how the new laws have impacted both consumers and businesses, for better and for worse.  Consumer trust Both sides of the coin tell a very different story when it comes to consumer trust and GDPR. The general consensus amongst businesses across the EU is that GDPR has greatly improved consumer trust, with 73 per cent reporting that the regulations have notably improved data security. Unfortunately, this sentiment isn’t shared by consumers.  84 per cent feel that GDPR hasn’t been taken seriously by businesses, and the level of security they feel when giving data to certain sectors varies hugely. While financial services, such as banks, have gained nearly half of consumers’ trust, hospitality, for example, are lagging behind with not even a quarter of consumers happy with the level of security.  But, looking at data breaches that have occurred since the implementation of GDPR, this level of dissatisfaction and worry from consumers comes as no surprise. From 280 million Microsoft users’ data being left unprotected to over a million of Mashable’s staff and consumer data being leaked by hackers, GDPR hasn’t necessarily solved the problems it was set out to manage, and consumers are concerned.  Consumer control Despite the worry of continued breaches and hacks, consumers do feel however that GDPR has improved the control they have over their own data. From being able to opt-in instead of having to opt-out, to having greater choice over the information given away through cookies, consumers feel much happier to be able to walk away from the brands they don’t trust and/or have no interest in.  Education around Data privacy  GDPR, since its inception, has been something that has eluded many. Filled with jargon and lacking much in the way of accessible educational assets, consumers – while aware of their data concerns – are still unsure of how to protect themselves against hacks or breaches. For example, only 14 per cent of internet users encrypt private conversations and only a third change their passwords regularly.  While GDPR has undoubtedly been a positive step forward for businesses and consumers alike, it is clear there is room for great improvement. It is expected that as the world continues to evolve into a digital-first society, especially post-COVID as many of us move online for good in our working lives, and the need for much-improved data security becomes paramount, GDPR laws and business compliance will need to continue to evolve and improve and fast.  If you're looking for your next opportunity, or to build out your Data & Analytics team, we can help. Take a look at our latest opportunities or get in touch with one of our expert consultants to find out more. 

Weekly Digest: 26th - 30th April 2021

This is Harnham’s weekly news digest, the place to come for a quick breakdown of the week’s top news stories from the world of Data & Analytics. Computer Weekly: Microsoft outlines five-year plan for accessibility tech 1 in 5 employers have stated that they would be less likely to hire someone if they were disabled. A damning and worrying statistic highlighting the severe disability divide that still exists in the working world.  In a bid to help put a stop to this serious lack of inclusion, Microsoft have teamed up with the Department for Work and Pensions (DWP) to train up to 26,000 members of staff and work coaches to help them create a more accessible recruitment and working experiences for those with disabilities.  The three key areas of focus for Microsoft include: Educating workers to have a better understanding of accessibility.Showing how applied sciences can be used to create opportunities for all.Dedicating its cause to constructing inclusive office environments, whether on or offline. Brilliantly summed up by Brad Smith, Microsoft President: “Our work begins by guaranteeing that Microsoft’s personal merchandise are accessible by design, in order that as we advance our options and performance, we can assist everybody throughout the spectrum of incapacity be extra productive.” Read more on this fantastic story here.  Analytics India Mag: Why Data Engineering is the fastest growing tech job in 2021 As a result of COVID-19, businesses have had to work hard to not only navigate the ‘new normal’ but thrive in it. Remaining relevant and staying one step ahead of the competition has been, and will continue to be, crucial – and for this reason alone, Data Engineering is undoubtedly going to the fastest growing sector this year and perhaps beyond.  During this year’s SkillUp event, Sourav Saha, academic dean at Praxis Business School, and Prasad Srinivasa, assistant vice president at Genpact, spoke about the exciting career opportunities in data engineering. In a world driven by data, it is crucial that companies are using the data they have available to drive the success of their business. From being able to forecast future trends to gather consumer sentiment, data, and data engineers, will undoubtedly drive business success. Srinivasa highlights the three key roles he is expecting to emerge and boom over the next 6 – 12 months: Data OrchestrationData Architecture and Governance Data Strategy To read more on what to expect for the future of Data Engineers, click here.  Silicon Republic: How to ensure your Life Science career thrives after COVID Life Sciences became the saviour of the COVID-19 pandemic. Those within the sector worked tirelessly to create and deliver vaccines around the world, giving us all the hope we needed to get to the end of this crisis.  However, once we finally see the back of the COVID-19 pandemic, many Life Science specialists beg the question – what next? How do we continue to thrive in our careers and any future prospects post-pandemic? This insightful article from Silicon Republic highlights five key steps to ensure specialists can be prepared to take the next steps in their working journey once the dust has settled.  1. Take control and be proactive Before you do anything else, take proactive steps to look at the opportunities your current employer might be able to present you. Don’t be afraid to put yourself forward.  2. Look for innovation As the whole world adapts to the new normal, it’s more than likely your company is going to innovate to stay one step ahead of the competition. Explore where this innovation is likely to happen and put ideas forward to spearhead change.  3. Upskill across the board From those harder skills, such as technical knowledge, to softer skills, such as communication and empathy, will all need to be revisited and boosted as we come out of the pandemic. Make sure you take your learning into your own hands and show initiative.  4. Reflect on your career options If you’re ready to make the move away from where you are, make sure you’ve got a clear idea of what it is you want next before taking the leap. Tailor your CV, brush up on your knowledge and don’t be afraid to engage with a recruiter if you need guidance.  5. Learn how to present well in remote interviews  Computer to computer isn’t the same as face-to-face. Get up to speed with online etiquette and make the best first impression.  Read the full article here.  TechBullion: 3 sectors revolutionised by AI The pandemic has accelerated many businesses uptake and implementation of AI. But which sectors have we seen reap the most reward from this fantastic technology? TechBullion explores.  Insurance: Insurance firms have seen AI boost customer satisfaction like never before. Whether that’s through faster processing of claims, a reduction in fraud or improving loss prevention, AI has been making the sector smoother and more efficient.  Entertainment: The likes of Netflix and Spotify have taken AI and used it to transforms how they service consumers. With many users wanting a service that provides a seamless, efficient, and relevant experience, entertainment platforms have been able to create software that can learn personal preferences and offer timely suggestions based on users evolving needs and wants.   Education: The use of AI within education has become vast. From making every day learning easier through AI-based games that can be tailored to all learning types and needs to the personalisation of students’ curriculum depending on the results of past tests – the abilities are endless.  To read more on this, visit TechBullion here.  We've loved seeing all the news from Data & Analytics in the past week, it’s a market full of exciting and dynamic opportunities. To learn more about our work in this space, get in touch with us at info@harnham.com.       

RELATED Jobs

Salary

£40000 - £50000 per annum + social events, flexible working,

Location

London

Description

An exciting and growing new analytics consultancy looking to change the future of analytics.

Salary

£55000 - £70000 per annum

Location

London

Description

Leading research agency looking for a Research Manager to work on customer-focused research and insight projects.

Salary

£65000 - £75000 per annum

Location

City of London, London

Description

This organisation owns some of the leading entertainment and media channels with the main goal to keep their customers entertained!

recently viewed jobs