DATA IS THE NEW OIL - CRUDE OIL

Krishen Patel our consultant managing the role
Posting date: 9/2/2013 2:16 PM

When Nasdaq stopped trading this week, it again showed how global firms are at the mercy of a power that created them

"Data is the new oil," declared Clive Humby, a Sheffield mathematician who with his wife, Edwina Dunn, made £90m helping Tesco with its Clubcard system. Though he said it in 2006, the realization that there is a lot of money to be made – and lost – through the careful or careless marshalling of "big data" has only begun to dawn on many business people.

The crash that knocked out the Nasdaq trading system was only one example; in the past week, Amazon, Google and Apple have all suffered breaks in service that have affected their customers, lost sales or caused inconvenience. When Amazon's main shopping site went offline for nearly an hour, estimates suggested millions of dollars of sales were lost. When Google went offline for just four minutes this month, the missed chance to show adverts to searchers could have cost it $500,000.

Michael Palmer, of the Association of National Advertisers, expanded on Humby's quote: "Data is just like crude. It's valuable, but if unrefined it cannot really be used. It has to be changed into gas, plastic, chemicals, etc to create a valuable entity that drives profitable activity; so must data be broken down, analyzed for it to have value."

For Amazon and Google especially, being able to process and store huge amounts of data is essential to their success. But when it goes wrong – as it inevitably does – the effects can be dramatic. And the biggest problem can be data which is "dirty", containing erroneous or garbled entries which can corrupt files and throw systems into a tailspin. That can cause the sort of "software glitch" that brought down the Nasdaq – or lead to servers locking up and a domino effect of overloading.

"Whenever I meet people I ask them about the quality of their data," says Duncan Ross, director of data sciences at Teradata, which provides data warehousing systems for clients including Walmart, Tesco and Apple. "When they tell me that the quality is really good, I assume that they haven't actually looked at it."

That's because the systems businesses use increasingly rely on external data, whether from governments or private companies, which cannot be assumed to be reliable. Ross says: "It's always dirty."

And that puts businesses at the mercy of the occasional high-pressure data spill. Inject the wrong piece of data and trouble follows. In April, when automatic systems read a tweet from the Associated Press Twitter feed which said the White House had been bombed and Barack Obama injured, they sold stock faster than the blink of an eye, sending the US Dow index down 143 points within seconds. But the data was dirty: AP's Twitter feed had been hacked.

The statistics are stunning: about 90% of all the data in the world has been generated in the past two years (a statistic that is holding roughly true even as time passes). There are about 2.7 zettabytes of data in the digital analytics universe, where 1ZB of data is a billion terabytes (a typical computer hard drive these days can hold about 0.5TB, or 500 gigabytes). IBM predicts that will hit 8ZB by 2015. Facebook alone stores and analyzes more than 50 petabytes (50,000 TB) of data.

Data is also moving faster than ever before: by last year, between 50% and 70% of all trades on US stock exchanges was being done by machines which could execute a transaction in less than a microsecond (millionth of a second). Internet connectivity is run through fibre optic connections where financial companies will seek to shave five milliseconds from a connection so those nanosecond-scale transactions can be done even more quickly.

We're also storing and processing more and more of it. But that doesn't mean we're just hoarding data, says Ross: "The pace of change of markets generally is so rapid that it doesn't make sense to retain information for more than a few years.

"If you think about something like handsets or phone calls, go back three or four years and the latest thing was the iPhone 3GS and BlackBerrys were really popular. It's useless for analysis. The only area where you store data for any length of time is regulatory work."

Yet the amount of short-term data being processed is rocketing. Twitter recently rewrote its entire back-end database system because it would not otherwise be able to cope with the 500m tweets, each as long as a text message, arriving each day. (By comparison, the four UK mobile networks together handle about 250m text messages a day, a figure is falling as people shift to services such as Twitter.)

Raffi Krikorian, Twitter's vice-president for "platform engineering" – that is, in charge of keeping the ship running, and the whale away – admits that the 2010 World Cup was a dramatic lesson, when goals, penalties and free kicks being watched by a global audience made the system creak and quail.

A wholesale rewrite of its back-end systems over the past three years means it can now "withstand" events such as the showing in Japan of a new film called Castle in the Sky, which set a record by generating 143,199 tweets a second on 2 August at 3.21pm BST. "The number of machines involved in serving the site has been decreased anywhere from five to 12 times," he notes proudly. Even better, Twitter has been available for about 99.9999% of the past six months, even with that Japanese peak.

Yet even while Twitter moved quickly, the concern is that other parts of the information structure will not be resilient enough to deal with inevitable collapses – and that could have unpredictable effects.

"We've had mains power for more than a century, but can have an outage caused by somebody not resetting a switch," says Ross. "The only security companies can have is if they build plenty of redundancy into the systems that affect our lives."


Click here for the article on the web.

Related blog & news

With over 10 years experience working solely in the Data & Analytics sector our consultants are able to offer detailed insights into the industry.

Visit our Blogs & News portal or check out the related posts below.

How to Succeed in Self-Service BI

How to Succeed in Self-Service BI

Business Intelligence, along with Business Analytics and Big Data, is one of the terms often associated with decision-making processes in organisations.  However, there is little discussion around the importance of what skills decision makers in your organisation need to use the technology efficiently.  In recent years, the development of user-friendly tools for BI processes, Self-Service BI are increasing. Self-Service BI is an approach to BI where anyone in an organisation can collect and organise data for analysis without the assistance of data specialists. As a result of this, many businesses have invested in comprehensive storage and information processing tools. However, many are beginning to find that they are not able to realise the gains of these investments as they were expecting, may often due to underestimating the difficulties of introducing these systems into the current processes and transforming existing knowledge into actual actions and decisions.  In a worst-case scenario, if left unplanned, Self Service BI can sabotage your successful BI deployment by cutting mass user adoption, impairing query performance, failing to reduce report backlogs, and increasing confusion over the “single truth”. To prevent this from happening, here are our top three tips for ensuring the right implementation of SSBI in your company: UNDERSTAND YOUR USERS’ NEEDS There are three major user areas for analytics tools: strategic, tactical and operational. The strategic users make few, but important decisions. The tactical users make many decisions during a week and need updated information daily. Operational users are often closest to the customer, and this group needs data in its own applications in order to carry out a large number of requests and transactions.  Understanding the different needs of each group is necessary to know what information should be available at each given frequency to help scale the BI solution.  HARNESS THE POWER OF ADVANCED USERS To ensure a successful BI deployment, utilising advanced users is key. Self-service BI is not a one-size fits all approach. Casual users usually don’t have the time to learn the tool and will often reach out to ‘Power Users’ to create what they need. Hence, these users can become the go-to resource for creating ad-hoc views of data. Power Users are the ideal advocates for your business’ self-service BI implementation and should be able to help spur user adoption.  UPGRADE INTERNAL COMPETENCIES  Our final tip for a successful implementation is to communicate the new tool thoroughly to the users.  It is highly unlikely that employees who have not been involved in the actual development project will immediately understand what the tool should be used for, who needs it, and what it should replace. By upgrading internal competencies, you can avoid becoming dependent on external assistance. Establishing a cross-organizational BI competence centre of 5-10 members, who meet regularly to share their experiences will help drives and prioritise future use of the tool. The added benefit of a successful implementation is that it will generate new ideas from users for how the organisation can use data to make better decisions. If you have the skillset to implement Business Intelligence solutions, we may have a role for you.  Take a look at our latest opportunities or get in contact with our team. 

Real Time Pricing - Coming to a store near you

Real Time Pricing - Coming to a store near you

Real-time pricing: coming to a store near you.Personal shopping is on the brink of taking on a whole new meaning. The advancement of mobile technology and the information held on individuals' shopping histories means product prices could soon adapt as shoppers walk up and down their supermarket aisle.Gone are the days of retailers only being able to actively manage the price of a small number of products once a week. Algorithmic pricing and real-time competitive pricing data allows the changing of product prices on the fly.Amazon is at the forefront of such "real-time pricing" initiatives, which have traditionally been the preserve of online-only retailers.However, brick-and-mortar retailers in the US are showing their UK counterparts the limitless possibilities when it comes to dynamic pricing.Independent consumer electronics retailer Abt Electronics pipes competitive pricing data gathered by Dynamite Data into its point-of-sale systems to allow staff to negotiate prices at the point-of-sale, according to Dynamite Data chief executive Diana Schulz.Meanwhile, another one of Dynamite Data’s unnamed clients uses electronic shelf labels and re-prices every product in their stores each morning based on the prices of its rivals.The ability to change prices dynamically is not simply the preserve of all-powerful brands such as Walmart or Target either.Schulz explained that her company has "seen these types of technologies in both large and mid-sized retailers" despite the "investment in technology and competitive data that is typically needed".Commercial sensitivitiesBack in the UK things are not quite as close to a Minority Report-style personalized shopping experience.Even online-only specialists Shop Direct and Ocado claim they do not engage in real-time pricing, while those that do heavily use real-time data to adapt their prices such as the airline brands are reluctant to discuss the issues.EasyJet declined to comment when contacted because of commercial sensitivities around discussing pricing-related issues.Grocers Tesco, Asda and  Sainsbury’s have all claimed they do not engage in real-time pricing, with the latter two both citing the logistical difficulties in aligning such a strategy across their physical stores and online presence.A Sainsbury’s spokesman claims real-time pricing would result in "chaos", while an Asda spokeswoman saying such a strategy would be a "nightmare".Yet, despite such a negative perspective from UK brands, experts are confident real-time pricing will arrive on these shores sooner or later.Simon Spyer, a partner of VCCP data arm Conduit who began his career working on the Sainsbury's Nectar business, believes the UK will begin to see "more and more" of matching rivals’ prices dynamically, particularly in the grocery and electrical sectors.He explained that real-time pricing is likely to affect "anything where the product is largely commoditized" and in instances where the only way retailers can differentiate that product is by "being really keen on price".Electronic labelsAs it stands the major barrier for implementing "real-time pricing" in-store is changing the prices to match the online price, a hurdle that could be removed by the electronic shelf labels being pioneered in the US.Schemes like Tesco Price Promise and Asda Price Guarantee already use real-time data to 'price match'In the UK various retailers have dipped their toes into the water when it comes to electronic shelf-labeling including a Nisa Local store in Shrewsbury that launched a trial in August last year to carry out automatic pricing and timed promotional updates, alongside QR codes and meal deals.Tesco has also experimented with electronic labeling on various occasions with trials in 2006 and 2008, but the retail giant has yet to combine real-time pricing with its electronic labels.Spyer claims "the capability is definitely there both online and offline – it is whether there is a business rationale for investing in it".However, with major UK supermarkets lacking a pressing reason to implement real-time pricing, that investment may be slow in arriving, argues Kaye Coleman, the founder of price consultancy Ripe Strategic.Coleman explains: "The supermarkets already do price matching – it is not so sophisticated but price matching is already happening".Schemes including the Tesco Price Promise, the Asda Price Guarantee and the Sainsbury’s Brand Match currently use real-time data to "price match" by offering money off the next shop.A cynic could argue the supermarkets should knock money off at the till rather than relying on customers to redeem their vouchers at the next shop, but such an action could hit the companies' bottom line.Mobile sophisticationThe growing sophistication of mobile marketing is also likely to revolutionize the way brands approach their price matching."If you can come up with a value proposition where I check-in [on my mobile] when I walk through the store for the first time and that presents me with a personalized experience based on my purchase history then I could see the benefit for a customer and a retailer," said Spyer.The trick for retailers is persuading customers to adopt such behavior, but the offer of being delivered ever-changing personalized price offers and messages in-store is a compelling proposition.Personalization is already a priority for retailers. Sainsbury’s uses anonymized shopping data gathered from the Nectar card to personalize offers.The levels of personalization offered by Sainsbury’s are increasingly complex. If a female customer buys folic acid they will be sent promotions on other pregnancy-related supplements during the pregnancy period and offers on nappies further down the line.UK retailers are sure to keep a close eye on developments over the Atlantic, with Schulz claiming she knows of clients that are piloting technologies that enable in-store personalized discounts.The challenges on the high-street mean there will inevitably be more casualties, but real-time pricing does not have to be the sole preserve of online-only retailers.Innovative ways of manipulating real-time data could be the shot in the arm the high-street retail industry so desperately needs.This article was first published on marketingmagazine.co.ukClick here for the article on the web.

RELATED Jobs

Salary

£50000 - £75000 per annum + + Benefits

Location

City of London, London

Description

This is the leading company in the online food retail space operating in the UK They are seeking an Infosec Team lead

Salary

£50000 - £60000 per annum

Location

City of London, London

Description

*SENIOR INSIGHT ANALYST - CENTRAL LONDON - SQL & PYTHON/R - UP TO £60,000*

Salary

£42 - £65000 per annum + bonus and benefits

Location

London

Description

A well known marketplace app are seeking a Senior Analyst.

Salary

£36000 - £37000 per annum + Yes

Location

Milton Keynes, Buckinghamshire

Description

Junior Data Scientist, London United Kingdom.

recently viewed jobs