Harnham
SEARCH
Shortlist

Big Data Demystified



People at a Big Data Seminar

The term “Big Data” as applied to IT was coined around 2011, and various persons have laid claim to having been the first person to coin it. It has become a buzz word that is sometimes misunderstood and often abused. Here we will try to demystify it so we can understand what it is and how we can realize its real value.

What is Big Data?

Many alternative definitions of Big Data have been published. One of the most insightful of these was proposed by Gartner and has become the accepted standard. It defines Big Data as “High volume, high velocity, and/or high variety information assets that require new forms of processing to enable enhanced decision making, insight discovery and process optimization."

These three V’s of Big Data, volume, velocity and variety, have more recently been augmented by the addition of a fourth V, veracity.

Volume

It is now almost a cliché to say that 90% of the existing data has been generated in the last two years. Around 2.4 trillion gigabytes of data are generated globally each day. Much of it arises through the internet and information generating digital analytics devices such as smartphones, digital analytics cameras and CCTVs and it is growing exponentially. It is estimated that by 2020 worldwide corporate data will exceed 35,000 exabytes where an exabyte is one quintillion bytes. 
Despite its size, big data is not  all about volume. In principal any volume of data could be processed using conventional database software if volume was the only issue.

Velocity

Data velocity, the rate at which information flows, is also increasing as a similar rate to volume. The increase in data velocity is in line with improving technologies are developing in accordance with Moore’s Law. 

Variety

The pivotal V that makes data Big Data is variety. While standard database software handles structured data, Big Data is often unstructured and cannot be processed with the same tool sets. In fact it can be a combination of various data categories including structured, semi-structured and unstructured. Typically it could consist of XML, database tables, audio and video files, text messages, tweets, and so forth.

Veracity

Veracity is an obvious addition. Unless the data is relevant, accurate and can be trusted, it is of little or no value. Ensuring the veracity of Big Data can be a challenge as it is difficult to control its quality. Any organization using Big Data must have the means of deciding whether it is beneficial and the extent to which it can be trusted.

Dealing with Big Data

One of the more popular ways of dealing with Big Data is Apache Hadoop. Named after a toy elephant it is an open source project designed to enable the large scale storage and processing of big data sets across server clusters.
It is hugely saleable and can readily be scaled from a single server to many thousands of servers either in premise or in the cloud. Originally developed by Yahoo and Google, its users include Yahoo, Facebook, Twitter, LinkedIn, and many more.

In addition to its scalability, it provides an inexpensive approach to massive parallel computing. It is flexible and can handle any kind of structured or unstructured data from unlimited sources which can be joined and aggregated and is fault tolerant.

Another popular approach is NoSQL. Also open source, it is a database framework that enables the storage and processing of large quantities of structured and unstructured data.

Big Data in Practice

Big Data and analytics have proven to be a success for many organizations. Its applications have included:

  • Understanding and targeting customers –  one of the main applications of Big Data. Examples of this in practice include Amazon Recommendations and personalized Tesco money-off coupons
  • Business processes – applications include optimizing supply route logistics, stock control based on social media trends, and HR processes including recruitment
  • Healthcare – for instance the side effects of drugs, correlations between lifestyle and health, the human genome, and the spread of infections
  • Big science – for instance the LHC at CERN generates one petabytes of data a second. Although most of it is discarded, CERN scientists store and process 30 petabytes a year using 65,000 servers.
  • Security and law enforcement – in the US the NSA uses Big Data in its war on terrorism as does GCHQ in the UK.

Is Big Data Over-hyped?

While the value of big data is clear, it isn’t a panacea. Certainly it has failed to live up to many of its early expectations and, according to some commentators, it has passed the peak of inflated expectations and has descended into the “trough of disillusionment”.

The backlash set in following the failure of Google Flu Trends which claimed to identify flu outbreaks using search queries. It got it spectacularly wrong, overestimating cases significantly from 2009 to 2013.

Big Data has several intrinsic weaknesses. These include:

  • While Big Data can detect subtle correlations, it can’t show causal relationships. This can lead to bad and dangerous conclusions. For instance the increasing number of autism diagnoses has been highly correlated with organic food sales.
  • Big data throws up correlations that appear to be statistically significant, but they happen just by chance simply because of the volume of data. The harder you look the more patterns you find even though they aren’t really there.
  • Big Data advocates have claimed that searching for models is no longer relevant as Big Data alone can deliver the answers. This is a dangerous and potentially catastrophic position that fortunately is losing sway.

Finally

Big Data is a large amount of data that may be structured, unstructured or both. It is characterized by its volume, velocity, and variety, and to be valuable it must have veracity too. However its real value is realized only when analytics are used to extract from it useful information.

It has changed how we do business, interact with each other and our customers, and protect our citizens from terrorism. Its benefits are clear, but so too are its potential dangers. Regardless, it’s here to stay so we should ensure that we learn how to handle it.

 

Harnham blog & news

With over 10 years experience working solely in the Data & Analytics sector our consultants are able to offer detailed insights into the industry.

Visit our News & Blogs portal or check out our recent posts below.

MACHINE LEARNING ENTERS BIOINFORMATICS AND ITS FUTURE IS BRIGHT

Machine Learning Enters Bioinformatics and its Future is Bright

Ever wondered how your email system knows which emails to show you and which to put in your junk or spam folder? Enter Machine Learning. It learns what you open and read and after a time can differentiate what you ignore, toss, or move to spam. Now imagine that same type of learning in the life sciences. As scientific advances move toward Data and Machine Learning to scale their knowledge, you can imagine the possibilities. After all, as you read this, trends in the life sciences, specifically with an eye toward bioinformatics showcase machine learning such as genome sequencing and the evolutionary of tree structures. Human and Machine Learning with a Common Goal There has been so much data provided over the past few decades, no mere mortal could possibly collect and analyze it all. It is beyond the ability of human researchers to effectively examine and process such massive amounts of information without a computer’s help.  So, machines must learn the algorithms and they do so in any number of ways. For the most part, it’s a comparison of what we know, or is already in a databank, with the information we have and don’t yet know. Unrecognized genes are identified by machines taught their function. The Future is Bright Machine Learning is giving other fields within the life sciences both roots and wings.  Imagine scientists being able to gain insight and learn from early detection predictions. This type of knowledge is already in play using neuroimaging techniques for CT and MRI capabilities. This is useful on a number of levels, not the least of which is in brain function; think Alzheimer’s Research, for example.  The hurdle? It isn’t the availability of such vast amounts of data, but the available computing resources. Add to that, humans will be the ones to check and counter-check validity which can in turn become more time-consuming and labor intensive than the computer’s original analysis. And it’s this hurdle which leads to a caveat emptor, or “buyer beware” of sorts. Caveat Emptor: Continue to Question Your Predictions In other words, how much can you trust the discoveries made using Machine Learning techniques in bioinformatics? The answer? Never assume. Always double check. Verify. But as you do so, know this. Work is already in progress for next-generation systems which can assess their own work.  Some discoveries cannot be reproduced. Why? Sometimes it’s more about asking the right question. Currently, a machine might look at two different clusters of data and see that they’re completely different. Rather than state the differences, we’re still working on a system that has the machine asking a different kind of question. You might think of it as a more human question that goes a bit deeper.  Imagine a machine that might say something noting the fact that some of the data is grouped together, but if different, it might say while it sees similarities, but am uncertain about these other groups of data. They’re not quite the same, but they’re close.  Machine Learning is intended to learn from itself, from its users, and from its predictions. Though a branch of statistics and computer science, it isn’t held to following explicit instructions. Like humans, it learns from data albeit at a much faster rate of speed. And its possibilities are only getting started. Want to see where Bioinformatics can take your career? We may have a role for you. If you’re interested in Big Data and Analytics, take a look at our our current vacancies or contact one of our recruitment consultants to learn more. For our West Coast Team, call (415) 614 - 4999 or send an email to sanfraninfo@harnham.com.  For our Mid-West and East Coast Teams, call (212) 796 - 6070 or send an email to newyorkinfo@harnham.com.

Riley Hsia

Riley Hsia

12. September 2019

How to lead a Data team

How To Lead A Data Team

Dream teams from sports to business are an ideal everyone aspires to live up to. But what is it every basketball or football dynasty has which makes them a dream team? What is it that brings individuals together to overcome odds, set examples, find solutions, and create the next best thing? Good management.  The need for good management is no different in the Data Science world. Yet according to our latest Salary Guide, poor management is one of the top five reasons Data professionals leave companies. So, let’s take a look at what poor management is, what causes it, and how businesses can better retain Data talent. What’s Your Data Science Strategy? Most businesses know they need a Data team. They may also assume that a Data Scientist who performed well can lead a Data team. But that isn’t necessarily the case. Managers have to know things like P&L statements, how to build a business case, make market assessments, and how to deal with people. And that’s just for a start.  The leader of a Data team has a number of other factors to consider as well such as Data Governance, MDM, compliance, legal issues around the use of algorithms, and the list goes on. At the same time, they also need to be managing their team with trust, authenticity, and candor. The list of responsibilities can be daunting and if someone is given too much too soon and without support, it can be a recipe for disaster. Other businesses might believe that a top performing Data Scientist would make a good manager. Yet these are two different fields. Or you might look at it this way. If you are willing to upskill a top performing Data professional and train them in managerial skills, giving them the education and support they need, that is one solution. Another solution is to create a Data Science strategy which brings in people with business backgrounds. Data Science is a diverse field and people come from a number of backgrounds not just Computer Science or Biostatistics, for example.  Now that you’ve seen what might cause a manager to fail, let’s take a look at a few tips to help you succeed. Seven Tips for Managing a Data Team Managing a team is about being able to hire, retain, and develop great talent. But if the manager has no management training, well, that’s how things tend to fall apart. Here a few tips to consider to help ensure you and your team work together to become the dream team of your organization: Build trust by caring about your team. Help define their role within the organization. Ensure projects are exciting and that they’re not being asked to do project with vague guidelines or unrealistic timeframes.Be open and candid. Remember, Data Scientists are trained in how to gather, collect, and analyze information. If anyone can see right through a façade, it will be these Data professionals. Have those “tough” conversations throughout every stage of the hiring, onboarding, and day-to-day, so that no one is caught unaware.Offer consistent feedback. And ask for it for yourself as well from your team.Ensure your team understands the business goals behind their projects. Let them in on the bigger picture. Think long-term recruitment for a permanent role, not short-term. If you have an urgent project, consider contracting it out. Prioritize diversity to include academic discipline and professional experience. Does the way this person view the world expand the knowledge of your team’s knowledge? Dream teams don’t always have to agree. Sometimes, the best solutions are found when there are other opinions. Finding the perfect, “Full Stack” Data Scientist or Data Engineer or Analyst is not impossible, and retaining them can be even easier. If you’ve done your job well, your team will trust you, have a balanced skillset, and understand how their work supports the organization and its goals. For more information on how to be a great manager, check out this article from HBR.  Ready for the next step?  Check out our current vacancies or contact one of our recruitment consultants to learn more. For our West Coast Team, call (415) 614 - 4999 or send an email to sanfraninfo@harnham.com.  For our Mid-West and East Coast Teams, call (212) 796 - 6070 or send an email to newyorkinfo@harnham.com.

Thomas Daughtrey

Thomas Daughtrey

11. July 2019

AI and Predictive Analytics Check-in to Hospitality

With summer in full swing, many of us are either planning our vacations, or have already enjoyed them and are thinking of where to go next. Regardless of location, we’re all looking for the same thing; a great experience to remember for years to come. No matter how exciting our trip, we all want our plans to run smoothly and, luckily, AI is here to help.  Today we have more options and more buying power than ever before. The ease with which we can search and select via our phones has kept businesses on their toes and driven them to look beyond traditional service.  By incorporating AI, the hospitality sector is implementing new ways to serve their customers more easily and efficiently. Fueling the AI Hospitality Experience The hospitality industry has a notoriously high turnover rate, relying heavily on seasonal workers, and those early in their careers. But, with AI, digital analysis, and predictive analytics entering the industry, new technologies are providing alternative customer service solutions: Predictive Analysis  Automation  Smart Domotics Advertising Predictive Analysis  As ‘the customer is always right’, the best way to create a smooth and memorable experience is to know what they want and then give it to them.  Given that difficulties can arise when there are too few staff for the number of guests, there is a need to be proactive when planning, in order to be reactive on the day. Utilizing Machine Learning, facilities can predict staffing and supply needs, planning for a more streamlined and, ultimately, better service.  Automation  Automating repetitive operations such as check-ins and check-outs, room assignments, and housekeeping deliveries gives staff more time to focus on the customer. As small and large hospitality businesses compete with the growing success of home sharing platforms, such as AirBnB, AI can give traditional facilities a fresh edge. In addition, rapid and efficient responses lead to greater customer satisfaction which, in turn, leads to a healthier bottom line.  Smart Domotics  More and more hotels are looking to the Internet of Things and Linked Technologies as they evolve into ‘smart’ destinations. With devices that can measure everything from room temperature to customer preferences, facilities can adapt in order to create an optimal environment. Furthermore, interaction with these ‘smart’ technologies can help hotels evolve over time, placing a greater emphasis on elements that prove to be the most popular with customers.  Advertising  From targeted Social Media campaigns to personalized gifts on arrival, Analytics can enhance the entire customer experience. When booking, users can engage with Chatbots 24/7, adding an element of humanity to the online booking experience.  When customers engage with resort apps and website, AI technologies cross-check their interactions and adapt their recommendations accordingly. With more people travelling than ever, the effort of keeping up with travelers the world over, night and day, is shifting to AI, thereby allowing the workforce more freedom to tend to customer needs.  AI in the Cloud The world of digital is transforming our lives, and the rise of Cloud technologies has taken digital analysis to the next level. With the advancements in AI, the hotel industry needs professionals who can create apps, collect and translate data, and, of course, build rigid infrastructures.  If you want to help hotel owners get a leg up on their competition and have a hand in creating a memorable travel experience for someone, we may have a role for you. To learn more, check out our current vacancies.  For the West Coast team call us at (415) 614-4999 or email us at sanfraninfo@harnham.com. For our Mid-West and East Coast Teams call us at (212) 796-6070, or email newyorkinfo@harnham.com.

Leah Topham

Leah Topham

24. July 2018

Why Texas is the place to be for technology jobs

Why Texas is the place to be for technology jobs The big data market is heating up the world over, and perhaps no more so than in Texas. The Dallas, Austin and Houston areas in particular are experiencing a massive boom in big data jobs, with many large tech companies making the move from Silicon Valley to enjoy all that Texas has to offer. But why the shift towards the southern state, and what does it mean for candidates looking big data jobs and broader technology roles? Tax-free Texas The Texan market is looking increasingly lucrative for both young start-ups and established tech companies alike.  One of the most significant factors in this rapid growth is the favourable tax conditions in the state. There’s no corporate or individual income tax, with Texas ranking 47 out of 50 states when it comes to taxes paid per $1,000 of personal income. As California tax rates hitting up to 10.84 for corporations and 12.3% for individuals, it’s understandable that entrepreneurs and big business alike are looking to the southern state for bigger breaks on tax day. On top of this, Texas offers favourable funding and regulatory conditions for young and growing businesses, providing a ‘pro-business’ environment for corporations to thrive. Texas State offers billions of dollars in incentives to businesses every year, providing all the more reasons for those in the technology industry to think hard about making the move. With a state government that celebrates business and provides easy to navigate laws and regulations, many businesses find the transition from Silicon Valley to Austin smooth and seamless. As organisations in San Francisco are priced out of the area, some of the nation’s top talent are moving to pastures greener – and for many, that means Texas. The living is easy On top of the tax breaks gained when moving to Texas, many movers and shakers experience a favourable quality of life. The cost of living is low – for example, the median home value in Austin is $321,600 compared to San Francisco’s $1,1943,300 – with relatively cheap utilities and the second-largest GDP in the nation. The market is robust, which has resulted in money being poured back into cities and communities to make them more attractive to businesses and young families. People can move to Austin and get more bang for their buck than they can in many other parts of the country, enjoying not only a booming technology market, but also superior housing and affordable living. Add in a comfortable climate and famously friendly locals and you’ve got a part of the country that is becoming increasingly appealing to even the most seasoned technology professionals. Technology is taking off Texas is huge when it comes to the technology industry. There was a 41.4% jump in technology industry employment between 2001 and 2013, resulting in large numbers of jobs being taken up across Austin and the wider state. And in 2016 alone, Texas added a huge 11,000 new technology jobs to its market, ranking it second of the 50 states in tech industry employment. The tech hub of Austin alone is home to employers such as Dell, Apple, Microsoft and Samsung, plus an increasingly significant number of start-ups peppering the landscape with innovation. There are a range of incubators and universities that feed into the city’s talent pool, with Austin ranking third in the list of US cities providing the most technology jobs in 2017. However, such growth doesn’t stop Austin and its other Texan counterparts from being a friendly and accessible place to work. There is less of the cut-throat nature that comes with tech in Silicon Valley, and more of a community, collaborative approach. Meanwhile, Dallas-Fort Worth is enjoying being the second-largest data center market in the country, offering an abundance of big data jobs to savvy business people.  Working in Texas Much of the Texas technology market is geared towards candidates currently, with more jobs than skilled employees to fill them. Companies are doing more to attract top talent to Texas, including offering generous benefits packages, relocation allowances and flexible work conditions, and the expectation is that this market will only continue to grow. If you’re looking for technology jobs in Austin or further afield in Texas, we might have just what you’re looking for. Take a look at our US data and technology jobs here.

Stephanie Brooks

Stephanie Brooks

05. July 2017

Recently Viewed jobs