Data is the new oil - crude oil



When Nasdaq stopped trading this week, it again showed how global firms are at the mercy of a power that created them

"Data is the new oil," declared Clive Humby, a Sheffield mathematician who with his wife, Edwina Dunn, made £90m helping Tesco with its Clubcard system. Though he said it in 2006, the realization that there is a lot of money to be made – and lost – through the careful or careless marshalling of "big data" has only begun to dawn on many business people.

The crash that knocked out the Nasdaq trading system was only one example; in the past week, Amazon, Google and Apple have all suffered breaks in service that have affected their customers, lost sales or caused inconvenience. When Amazon's main shopping site went offline for nearly an hour, estimates suggested millions of dollars of sales were lost. When Google went offline for just four minutes this month, the missed chance to show adverts to searchers could have cost it $500,000.

Michael Palmer, of the Association of National Advertisers, expanded on Humby's quote: "Data is just like crude. It's valuable, but if unrefined it cannot really be used. It has to be changed into gas, plastic, chemicals, etc to create a valuable entity that drives profitable activity; so must data be broken down, analyzed for it to have value."

For Amazon and Google especially, being able to process and store huge amounts of data is essential to their success. But when it goes wrong – as it inevitably does – the effects can be dramatic. And the biggest problem can be data which is "dirty", containing erroneous or garbled entries which can corrupt files and throw systems into a tailspin. That can cause the sort of "software glitch" that brought down the Nasdaq – or lead to servers locking up and a domino effect of overloading.

"Whenever I meet people I ask them about the quality of their data," says Duncan Ross, director of data sciences at Teradata, which provides data warehousing systems for clients including Walmart, Tesco and Apple. "When they tell me that the quality is really good, I assume that they haven't actually looked at it."

That's because the systems businesses use increasingly rely on external data, whether from governments or private companies, which cannot be assumed to be reliable. Ross says: "It's always dirty."

And that puts businesses at the mercy of the occasional high-pressure data spill. Inject the wrong piece of data and trouble follows. In April, when automatic systems read a tweet from the Associated Press Twitter feed which said the White House had been bombed and Barack Obama injured, they sold stock faster than the blink of an eye, sending the US Dow index down 143 points within seconds. But the data was dirty: AP's Twitter feed had been hacked.

The statistics are stunning: about 90% of all the data in the world has been generated in the past two years (a statistic that is holding roughly true even as time passes). There are about 2.7 zettabytes of data in the digital analytics universe, where 1ZB of data is a billion terabytes (a typical computer hard drive these days can hold about 0.5TB, or 500 gigabytes). IBM predicts that will hit 8ZB by 2015. Facebook alone stores and analyzes more than 50 petabytes (50,000 TB) of data.

Data is also moving faster than ever before: by last year, between 50% and 70% of all trades on US stock exchanges was being done by machines which could execute a transaction in less than a microsecond (millionth of a second). Internet connectivity is run through fibre optic connections where financial companies will seek to shave five milliseconds from a connection so those nanosecond-scale transactions can be done even more quickly.

We're also storing and processing more and more of it. But that doesn't mean we're just hoarding data, says Ross: "The pace of change of markets generally is so rapid that it doesn't make sense to retain information for more than a few years.

"If you think about something like handsets or phone calls, go back three or four years and the latest thing was the iPhone 3GS and BlackBerrys were really popular. It's useless for analysis. The only area where you store data for any length of time is regulatory work."

Yet the amount of short-term data being processed is rocketing. Twitter recently rewrote its entire back-end database system because it would not otherwise be able to cope with the 500m tweets, each as long as a text message, arriving each day. (By comparison, the four UK mobile networks together handle about 250m text messages a day, a figure is falling as people shift to services such as Twitter.)

Raffi Krikorian, Twitter's vice-president for "platform engineering" – that is, in charge of keeping the ship running, and the whale away – admits that the 2010 World Cup was a dramatic lesson, when goals, penalties and free kicks being watched by a global audience made the system creak and quail.

A wholesale rewrite of its back-end systems over the past three years means it can now "withstand" events such as the showing in Japan of a new film called Castle in the Sky, which set a record by generating 143,199 tweets a second on 2 August at 3.21pm BST. "The number of machines involved in serving the site has been decreased anywhere from five to 12 times," he notes proudly. Even better, Twitter has been available for about 99.9999% of the past six months, even with that Japanese peak.

Yet even while Twitter moved quickly, the concern is that other parts of the information structure will not be resilient enough to deal with inevitable collapses – and that could have unpredictable effects.

"We've had mains power for more than a century, but can have an outage caused by somebody not resetting a switch," says Ross. "The only security companies can have is if they build plenty of redundancy into the systems that affect our lives."


Click here for the article on the web.



<< Click here to see more recent news articles >>

 

 

Harnham blog & news

With over 10 years experience working solely in the Data & Analytics sector our consultants are able to offer detailed insights into the industry.

Visit our News & Blogs portal or check out our recent posts below.

Four Ways Advanced Analytics Drives Business Forward

2020 was an unprecedented year for shifting businesses online. Technology, never quite in the background, arrived center stage to help drive transformation in a variety of industries. Many businesses were forced to change their processes, how they interacted with their employees, customers, and with each other. One of these major shifts was in Advanced Analytics and Insight. Stemming from a Marketing perspective which had specific deliverables of demographics, location, and consumer histories, advancements found a place in working with unstructured Data. Working in tandem with these new analytical insights, artificial intelligence brought learning, problem-solving, planning, and other naturally human behaviors to life. This includes in the creative fields, not just in traditional industries like Finance or Retail.  In a study conducted by Forrester on behalf of Intel, though most businesses know Analytics can drive their business forward, less than half are taking advantage of these transformational technologies. Below are a few ways Advanced Analytics can drive business forward. 1. Decision Science and the CDO Roles Will Grow In a seemingly counterintuitive measure, while most businesses were cutting back in IT, Data and Analytics budgets were expanded. As the Chief Data Officer and Decision Science roles increase in importance, businesses who know the value of their Data can derive actionable insights and business decisions from these executive level communicators. 2. Access to a variety of Data Sources Will Help to Streamline Business Operations With most businesses operating strictly online or in a hybrid ecosystem, optimization of processes is key. In the ever-changing market systems, buyer behaviors and the consumer journey will increase dependency on Data and Analytics as businesses seek to meet consumer demand. Offering bespoke solutions and coordinating such Data sources as chatbots and call centers, businesses will have the opportunity to create a seamless system as they adopt and implement technologies such as Advanced Analytics and AI. In the right mindset, these practices can also drive partnerships within their ecosystems from Data Science to technology vendors with AI capabilities.  3. Sharpening Focus on Measurable Projects to Increase ROI Rather than rely on third parties, Data will become part of the business offering value in their operations. It will drive how they operate, deliver, and understand the needs of their consumer. Owning and managing their own Data will provide unique insights they may not have been aware of before. Sharpening their focus to get a good return on their analytics investment, businesses will broaden their ecosystem. Seeing the bigger picture, businesses will also want to access more specific insights that drive actionable answers to their questions. 4. Machine Learning, NLP, and Domain Expertise Can Help Scale Data Modelling As AI, Advanced Analytics, NLP, and Machine Learning platforms come into full swing and in combination, new Data Modelling opportunities can increase insight. Automated processes of Data classifications will drive scale increasing both the amount of Data and a granular level of detail to be extracted.  The specialization of these Data platforms will only grow in importance. In our always-on, always on demand world, the need for Advanced Analytics professionals and a variety of posts in the Data profession, businesses will expect strong domain knowledge. They’ll be looking for professionals and platforms which can help them understand specific use cases. Rather than just simple demographics and birds-eye views of their consumers, they’ll want to drill down to not only what they can provide now in terms of goods and services, but anticipate what consumers will want and need for the future. In the last year, we’ve absorbed a lot of information, and have struggled to distill it in actionable insights. But, if you’re interested in Marketing and Insight, and would like to shift into Advanced Analytics and Insight, we may have a role for you. Not your bag, but interested in Life Sciences, Decision Science, Machine Learning, or Robotics just to name a few, Harnham may have a role for you. Check out our current vacancies or contact one of our expert consultants to learn more.  For our West Coast Team, contact us at (415) 614 - 4999 or send an email to sanfraninfo@harnham.com.  For our Mid-West and East Coast teams contact us at (212) 796-6070 or send an email to newyorkinfo@harnham.com.  

Back To Basics In The Business Of Data Science

Though COVID-19 and the US Election still dominates the news, there’s a lot going in the world of Data Science, too. 2020 ramped up efforts in the healthcare industry to combat the pandemic. Cybersecurity is entering a renaissance of sorts as we tackle the misinformation age, but there’s some fun stuff, too. Here a few Data Science trends finding foundation and leverage in 2021. Businesses Kick Data Science into High Gear To stay viable, businesses are kicking their Data Strategy into high gear. From the top down, there will be an estimated increase of Chief Data Scientists to help businesses make critical business decisions from e-commerce to SaaS. Trajectory of Data Analysts Upskilling to Data Scientist  Once relegated to sampling bits of data and leaving others to break it down into workflow, Data Scientists could see a boost of responsibility. The demand for soft skills, upskilling, and cross-training could reduce the need to have Machine Learning and Data Engineers process empowering the Data Scientists to do more. Breaking the Mold With businesses and education moved online, businesses will be challenged to keep employees engaged. Training and education are now available to employees and would-be Data Scientists at home for on-the-job training as they face new technologies being developed, use new tools, and lessened demand on the college degree, but the experience in applying what’s been learned. Machine Learning Gets Smarter AI and Machine Leaning applications will focus on charting algorithms to understand cause-and-effect. But it won’t happen overnight. Teaching and testing machines is intense and time consuming. These technologies might present probability, but can’t determine definites. Yet.  Applying Machine Learning strategies to business problems through systems will focus businesses on finding solutions rather than focusing on building products that aren’t in their wheelhouse. Adding neuroscience and computational neuroscience into the mix for Machine Learning will see these fields grow. Ultimately, Machine Learning and AI are estimated to be the final piece in the puzzle when it comes to Data Science strategies for a variety of industries.  Back to Basics As everyone gets organized in their new ways of doing business, Data Scientists are getting back to basics. Their solving big problems with better tools, technologies, and open-source information now available. The push for open access scientific and medical journals along with the global team environment offers a variety of ways in which Data Scientists can come together to focus on problems more efficiently than anyone else. In other news, projects such as the new James Webb Telescope, the open access drive for scientific and medical journals, and the latest space race information, Data Scientists have been busy getting these projects off the ground as well. Though 2020 took us by surprise in so many ways, we took what we had and ran with it. So, as we enter 2021, we’re on a unique footing from Machine Learning and AI to Data Science with the added boost of nuero-and computational science to employ every tool at our disposal. Businesses have ramped up their efforts and are empowering the professionals in the Data Strategy teams to help them make critical business decisions with an eye toward the future. Data Scientists are getting back to basics while leveraging their skillsets from open access, online education sources, and on-the-job training to solve the big problems we face. And of course, while we have our eyes on the sky when it comes to space exploration this year and our feet on the ground as we work to vaccinate populations against COVID-19, and Chief Data Scientists split their focus to improve business bottom lines, we know demand will remain high for those in the Data industry. If you’re interested in Data & Analytics, Harnham may have a role for you. Check out our current vacancies or contact one of our expert consultants to learn more.  For our West Coast Team, contact us at (415) 614 - 4999 or send an email to sanfraninfo@harnham.com.  For our Mid-West and East Coast teams contact us at (212) 796-6070 or send an email to newyorkinfo@harnham.com.

Could Mobile Gaming Help Us Solve Real World Problems?

From Alexandra Ocasio-Cortez and Ilhan Omar’s Twitch play Among Us to Fortnite helping kids get into college, the mobile gaming industry has leapt into the new decade with gusto. It doesn’t hurt to have a few well-known names behind it, too. But it’s what AOC and Oman accomplished -normalcy - that begs the question. Could mobile gaming help us solve real world problems?  A Cultural Beginning             Cultural institutions, such as museums, have had a rough go of it this year. But innovation, creativity, and collaboration have come together to offer opportunities to get a bird’s eye view of the art world. Whether you play Occupy White Walls to create your own gallery bring real art into your gaming world through the Getty Museum and Nintendo’s Animal Crossing collaboration. Just a few ways these could be jumping off points to discussion for problem-solving include: Opportunities abound to host audiences from around the world without a head count capA chat function to discuss what you see, what you like, what you don’t, and what you’d like to seeCultural institutions become more open allowing anyone and everyone into its virtual wallsPerhaps even simulations and to imagine what-if scenarios for the rest of usGames could host exhibitions such as the Monterey Bay Aquarium of California who’s partnered with Animal Crossing and a fossil expert form the Field Museum in Chicago who hosts virtual tours through Twitch Games give us the opportunity to imagine what’s possible. And these games are bringing real life events and activities straight to your fingertips in mobile gaming. Ad-Tech and Analytics are In the Game Since social distancing has become the norm, gaming has exploded. Once all the numbers are in, mobile game downloads are expected to see a nearly 40% increase in 2020 from 2019. No business who sees the potential here is standing by, the least of which is advertising. If you remember cable TV or maybe still have one, the free channels were often supported by commercials. In some television shows, products were given strategic placement. Okay, so it’s probably still happening today, but now we’re used to it. Skip ahead a few generations. Hello, Ready Player One fans, and advertisers have a new platform. Or at least, they’re working on it. There are still a few kinks to work out. Some game developers are designing games to help allow advertisers to fit seamlessly into games.  The audience of one engagement of TV has moved to a community engagement of many in the gaming world. Over two and half billion people are gamers across demographics of age and location. Social media still has the highest ad buys from television, but gaming is quickly catching up. As advertisers and businesses get in the game, it’s estimated there will be a monumental shift in the collective. Games have always served as a device to teach. Chess and checkers teach strategy. Monopoly teaches business and banking. Life teaches us to follow different paths and see what happens (not so different from Second Life). So, what could games teach us now with its ever increasing role in bringing groups together for engagement, community, and discussion? What role will you play in the coming year? If you’re interested in Data & Analytics, Harnham may have a role for you. Check out our current vacancies or get in touch with one of our expert consultants to learn more.  For our West Coast Team, contact us at (415) 614 - 4999 or send an email to sanfraninfo@harnham.com.  For our Mid-West and East Coast teams contact us at (212) 796-6070 or send an email to newyorkinfo@harnham.com.  

Business Intelligence Is About Asking The Right Questions

You’ve dotted all the ‘Is’, crossed all the ‘Ts’. You’ve ensured your business priorities were aligned with your mission and objectives. But, how can you know if you’re on the right path, especially in light of today’s uncertainties. Your crystal ball may be in the cloud, but to find its clarity, you have to be asking the right questions. Below are three questions to consider moving forward. 1. How Collaborative Are We? As businesses shift online and teams expand globally, collaborative business intelligence streamlines decision-making. A combination of BI tools, software, and social technologies to inform, engage, analyze, and form insights of what customers want and need. This form of collaboration takes decision-making out of its siloes. Not unlike the Socratic method, collaborative business intelligence solves problems through shared information to find common ground. Using business intelligence software to provide opportunities for predictive modeling, visual analysis of data and business metrics, businesses analysts can interpret and inform, in a more efficient streamlined process. 2. How Secure is Our Data? Whether big business, small business, or medium business, no one is immune to cyberattacks. The ever- increasing rise of these attacks pinpoints just how important keeping data secure is for all businesses. Breaches cause not only monetary loss, but ultimately, consumer trust leading to more loss. The importance of Data security cannot be overstated.  Now that a majority of businesses are making flexible and remote work options available, it’s imperative businesses work to keep data secure. Consumers today are much more concerned today about how and why their Data is used, and many may decline to offer it, not wanting to put themselves at risk of a possible cyberattack.  3. What’s the Best Platform to Drive Actionable Insights from Our Analytics? Much like the trend of collaborative BI, businesses are focused on combining business processes and workflows into one platform, so everyone has access to the same Data. It’s within these platforms, that businesses cannot only determine what action to take and implement those actions all in one place. Platforms become the hub of the wheel and the spokes are analytics of a particular industry, business, or department in which insights can be implemented. Some platforms on the move include Sisense and Sharepoint. Google Analytics Intelligence (GAI) might be the most well-known especially if you’re just getting started asking the right questions for your business.  If you want insight into the state of your business, know any major consumer traffic changes, or want to know the probable conversion rate of web browsers to customers, GAI can help you get those answers. Because it uses machine learning to help, it’s important to know not necessarily what questions to ask, but how to ask them. How to Ask a Computer the Right Questions If you’ve been working in a collaborative BI team and asking each other questions based on the data you’ve collected, it may be a bit of a mindset shift for asking questions of a computer. So, how you phrase your question, what you want to know, and how you ask may require a bit of thought to find the answers you’re looking for. Below are a few guidelines to consider when posing the questions.Follow the TrendIf you want to know what’s trending in your business, you might ask: How many products were sold last week?How many customers did I have today?Where are my customers located?What time were the most customers shopping?  Which is Best? When you want to know what product is selling the most and through which means. Follow the performance. These questions might include: Which channel converted the most customers?Which product sold the most? Which product sold the least?Which hour was best for customer traffic? Compare and Contrast These are questions or commands that enable you to compare two sets of data side by side, such as how your business performed week to week, day to day, or year to year. While most questions begin with ‘which’ or ‘how’, the compare and contrast questions can get a bit more complex. In these questions, you begin with what you want to know such as conversion rate, revenue shares, traffic, or trend.  As this year comes to a close, what questions will you ask of yourself? Are you ready for a change? A new role? If you’re a business, what questions will you ask to move your company forward in the new year? If you’re interested in Big Data & Analytics, we may have a role for you. Check out our current opportunities or contact one of our expert consultants to learn more.  For our West Coast Team, contact us at (415) 614 - 4999 or send an email to sanfraninfo@harnham.com.  For our Mid-West and East Coast teams contact us at (212) 796-6070 or send an email to newyorkinfo@harnham.com.  

Recently Viewed jobs