DATA IS THE NEW OIL - CRUDE OIL

Krishen Patel our consultant managing the role
Posting date: 9/2/2013 2:16 PM

When Nasdaq stopped trading this week, it again showed how global firms are at the mercy of a power that created them

"Data is the new oil," declared Clive Humby, a Sheffield mathematician who with his wife, Edwina Dunn, made £90m helping Tesco with its Clubcard system. Though he said it in 2006, the realization that there is a lot of money to be made – and lost – through the careful or careless marshalling of "big data" has only begun to dawn on many business people.

The crash that knocked out the Nasdaq trading system was only one example; in the past week, Amazon, Google and Apple have all suffered breaks in service that have affected their customers, lost sales or caused inconvenience. When Amazon's main shopping site went offline for nearly an hour, estimates suggested millions of dollars of sales were lost. When Google went offline for just four minutes this month, the missed chance to show adverts to searchers could have cost it $500,000.

Michael Palmer, of the Association of National Advertisers, expanded on Humby's quote: "Data is just like crude. It's valuable, but if unrefined it cannot really be used. It has to be changed into gas, plastic, chemicals, etc to create a valuable entity that drives profitable activity; so must data be broken down, analyzed for it to have value."

For Amazon and Google especially, being able to process and store huge amounts of data is essential to their success. But when it goes wrong – as it inevitably does – the effects can be dramatic. And the biggest problem can be data which is "dirty", containing erroneous or garbled entries which can corrupt files and throw systems into a tailspin. That can cause the sort of "software glitch" that brought down the Nasdaq – or lead to servers locking up and a domino effect of overloading.

"Whenever I meet people I ask them about the quality of their data," says Duncan Ross, director of data sciences at Teradata, which provides data warehousing systems for clients including Walmart, Tesco and Apple. "When they tell me that the quality is really good, I assume that they haven't actually looked at it."

That's because the systems businesses use increasingly rely on external data, whether from governments or private companies, which cannot be assumed to be reliable. Ross says: "It's always dirty."

And that puts businesses at the mercy of the occasional high-pressure data spill. Inject the wrong piece of data and trouble follows. In April, when automatic systems read a tweet from the Associated Press Twitter feed which said the White House had been bombed and Barack Obama injured, they sold stock faster than the blink of an eye, sending the US Dow index down 143 points within seconds. But the data was dirty: AP's Twitter feed had been hacked.

The statistics are stunning: about 90% of all the data in the world has been generated in the past two years (a statistic that is holding roughly true even as time passes). There are about 2.7 zettabytes of data in the digital analytics universe, where 1ZB of data is a billion terabytes (a typical computer hard drive these days can hold about 0.5TB, or 500 gigabytes). IBM predicts that will hit 8ZB by 2015. Facebook alone stores and analyzes more than 50 petabytes (50,000 TB) of data.

Data is also moving faster than ever before: by last year, between 50% and 70% of all trades on US stock exchanges was being done by machines which could execute a transaction in less than a microsecond (millionth of a second). Internet connectivity is run through fibre optic connections where financial companies will seek to shave five milliseconds from a connection so those nanosecond-scale transactions can be done even more quickly.

We're also storing and processing more and more of it. But that doesn't mean we're just hoarding data, says Ross: "The pace of change of markets generally is so rapid that it doesn't make sense to retain information for more than a few years.

"If you think about something like handsets or phone calls, go back three or four years and the latest thing was the iPhone 3GS and BlackBerrys were really popular. It's useless for analysis. The only area where you store data for any length of time is regulatory work."

Yet the amount of short-term data being processed is rocketing. Twitter recently rewrote its entire back-end database system because it would not otherwise be able to cope with the 500m tweets, each as long as a text message, arriving each day. (By comparison, the four UK mobile networks together handle about 250m text messages a day, a figure is falling as people shift to services such as Twitter.)

Raffi Krikorian, Twitter's vice-president for "platform engineering" – that is, in charge of keeping the ship running, and the whale away – admits that the 2010 World Cup was a dramatic lesson, when goals, penalties and free kicks being watched by a global audience made the system creak and quail.

A wholesale rewrite of its back-end systems over the past three years means it can now "withstand" events such as the showing in Japan of a new film called Castle in the Sky, which set a record by generating 143,199 tweets a second on 2 August at 3.21pm BST. "The number of machines involved in serving the site has been decreased anywhere from five to 12 times," he notes proudly. Even better, Twitter has been available for about 99.9999% of the past six months, even with that Japanese peak.

Yet even while Twitter moved quickly, the concern is that other parts of the information structure will not be resilient enough to deal with inevitable collapses – and that could have unpredictable effects.

"We've had mains power for more than a century, but can have an outage caused by somebody not resetting a switch," says Ross. "The only security companies can have is if they build plenty of redundancy into the systems that affect our lives."


Click here for the article on the web.

Related blog & news

With over 10 years experience working solely in the Data & Analytics sector our consultants are able to offer detailed insights into the industry.

Visit our Blogs & News portal or check out the related posts below.

Where Tech Meets Tradition

Where Tech Meets Tradition

If you’re lamenting the decline of handmade traditional products, cast your cares aside. There’s a new Sheriff in town and its name is, Tech. Just a generation ago, children would leave the farm or the family business, go to school, and then move on to make their place in the world doing their own thing. Away from family.  Today, the landscape has changed and those who have left are coming home. But this time, they’re bringing technology with them to help make things more efficient and more productive. Is Tech-Assisted Still Handmade? In a word, yes. Artists still make things “from scratch”, except now technologies allow them to not only see their vision in real-time, but their customers, too. Have you ever wondered what the image in your head might look like on paper or in metal? What about the design of prosthetic arms and healthcare devices by 3D printers? You’re still designing, creating.  But just like any new technology, there’s still a learning curve. Even for cutting-edge craftspeople who find that sometimes, the line between craftsmanship and high-tech creativity may be a bit of a blur. Not to mention the expense for either the equipment required or being able to offer art using traditional tools at technology-assisted prices. Somewhere between the two, there is a trade-off. It’s up to the individual to determine where and what that trade-off is. Life in the Creative Economy One of Banksy’s paintings shredded itself upon purchase at an auction recently. AI is making music and writing books. Augmented Reality, Virtual Reality, and Blockchain all have their place in the creative economy from immersive entertainment to efficient manufacturing processes. Each of these touches the way we live now. In a joint study between McKinsey and the World Economic Forum, 'Creative Disruption: The impact of emerging technologies on the creative economy', the organisations broke down the various technologies used in the creative economy and how they’re driving change. For example: AI is being used to distill user preferences when it comes to curating movies and music. The Associated Press has used AI to free up reporters’ time and the Washington Post has created a tool to help it generate up to 70 articles a month, many stories of which they wouldn’t have otherwise dedicated staff.Machine Learning has begun to create original content. Virtual Reality and Augmented Reality have come together as a new medium to help move people to get up, get active, and go play whether it’s a stroll through a virtual art gallery or watching your children play at the playground.  Where else might immersive media play out? Content today could help tell humanitarian stories or offer work-place diversity training. But back to the artisan handicrafts.  Artistry with technology Whilst publishing firms may be looking to use AI to redefine the creative economy, they are not alone. Other artists utilising these technologies include:  SculptorsDigital artistsPaintersJewellery makersBourbon distillers America’s oldest distiller has gotten on the technology bandwagon and while there is no rushing good Bourbon, but you can manage the process more efficiently. They’ve even taken things a step further and have created an app for aficionados to follow along in the process. Talk about crafted and curated for individual tastes and transparency. It may seem almost self-explanatory to note how other artisans are using technology. But what about distilleries? What are they doing? They’re creating efficiency by: Adding IoT sensors for Data Analytics collection Adding RFID tags to their barrels Creating experimental ageing warehouses (AR, anyone?) to refine their craft. Don’t worry, though. These changes won’t affect the spirit itself. After all, according to Mr. Wheatley, Master Distiller, “There’s no way to cheat mother nature or father time.” Ultimately, the idea is to not only understand the history behind the process, but to make it more efficient and repeatable. A way to preserve the processes of the past while using the advances of the present with an eye to the future. If you’re interested in using Data & Analytics to drive creativity, we may have a role for you. Take a look at our latest opportunities or get in touch with one of our expect consultants to find out more. 

How Will New Financial Risk Regulations Affect European Banks?

How Will New Financial Risk Regulations Affect European Banks?

The financial crisis of 2007-2008 changed banking. The world moved from taking mortgage loans in our dogs’ names to introducing strict regulations for banks prohibiting them from giving out loans to “anyone” without assessing Risk properly. In 2010 the Basel Committee on Banking Supervision (BCBS) introduced BASEL III, a regulatory framework that builds on BASEL I, and BASEL II. This framework changed how banks and financial institutions asses risk. It introduced an Advanced Internal Rate Based Approach (Commonly known as the AIRB approach).  Now, the committee has introduced new changes and, by 2022, all banks and institutions will have to implement the revised IRB Framework, as well as new revised regulations for the standardised approach, CVA Framework and new frameworks for Operational Risk and Market Risk. So, what does this mean for those working Risk? Change Is Coming Change is inevitable, no matter what you do. If you work in Risk Management and Compliance, change is something you can expect to happen, often. As mentioned above, by 2022 there will be lots of changes. The Basel Committee calls this initiative the “finalised reforms”, or BASEL IV which builds on the current regulatory framework BASEL III. Quickly summarised, the changes limit the reduction in capital that effect banks IRB models.  This change is predicted to impact banks in Sweden and Denmark the most, with estimations that capital ratio will fall by 2.5-3%, far higher than the 0.9% expected for the average European bank.  So what does all this mean for Swedish and Danish banks?  What’s Happening Now? One of the main things that Swedish and Danish banks need to revise for these new regulations, are their internal models. The new regulations introduced a new definition of Probability of Default, measured through a model commonly known as a PD model. Effectively this means that every bank must “re-develop” their internal PD Models in the IRB approach. Consequently, we are already seeing a clear response from the banks in their strategies moving forward. It has already become quite apparent that many banks are looking to make IRB model development their focus for 2019-2020 and 2021. This has resulted in a boom in the hiring space for developers with experience in IRB Modelling and Credit Risk Modelling in general, which in turn has led to high demand in the face of the low supply of these types of candidates. Understandably aware of this, modellers are now looking to negotiate higher salaries.  What You Can Do  For candidates that hold the right experience, there are good opportunities at hand. If so inclined, they can utilise this chance to finally see if the grass actually is greener on the other side, or not. However, there are a couple of things worth considering before making a move.   Firstly, are you actually keen on switching jobs? Your skills are probably equally in demand at your current employer and, if you are having doubts about moving from the get-go, you may well be able to negotiate a rise without pursuing a new opportunity. However, if you are serious about finding something new, this is a great time to do so. The majority of banks have found that these new regulations are creating an unsustainable workload,  and are now looking for talent externally to expand their teams. This means that the experienced modeller can pretty much have their pick of the litter.  Furthermore, if you are a junior modeller, there are now plenty of opportunities for you to enter a niche area known for being exciting and innovative. So, wherever you are in your career, these regulatory changes  are likely to have a large impact and open up new avenues for you to explore.   We all know that regulations in banking and finance are now essential, we all agree, even if they can be a little frustrating. However, what people often fail to think of are the opportunities new regulatory requirements create. In the case of BASEL IV, we’re already seeing an increase in demand for strong talent, and a demand for people who are passionate about Risk Management and model development.  For businesses, new regulations also provide the chance to not only improve their teams, but to  create new models that can be utilised to optimise and automate. A lot of financial institutions are already aware of this and are using these models to gain competitive advantage over their competitors, as well as to stay one hundred percent compliant.  If you’re looking to build out you Risk Management team or take on a new Risk opportunity for yourself, we may be able to help. Take a look at our latest opportunities or get in touch with one of our expert consultants to find out more. 

RELATED Jobs

Salary

£50000 - £65000 per annum

Location

London

Description

This role involves working with the most established media agency in the UK

Salary

£600 - £650 per day

Location

Greater London

Description

Hi all, I'm currently recruiting for a Solutions Architect who designing and implementing solutions on Microsoft Azure

Salary

£30000 - £45000 per annum + competitive bonus + benefits

Location

London

Description

A great opportunity to join an exciting and ambitious credit card company as the Lead Analyst on Application Fraud.

Salary

£80000 - £95000 per annum + competitive benefits package

Location

London

Description

A leading consultancy are seeking an experienced Forensic Analytics Director to oversee a team responsible for high volume transaction monitoring.

recently viewed jobs