Harnham

Big Data Demystified



People at a Big Data Seminar

The term “Big Data” as applied to IT was coined around 2011, and various persons have laid claim to having been the first person to coin it. It has become a buzz word that is sometimes misunderstood and often abused. Here we will try to demystify it so we can understand what it is and how we can realize its real value.

What is Big Data?

Many alternative definitions of Big Data have been published. One of the most insightful of these was proposed by Gartner and has become the accepted standard. It defines Big Data as “High volume, high velocity, and/or high variety information assets that require new forms of processing to enable enhanced decision making, insight discovery and process optimization."

These three V’s of Big Data, volume, velocity and variety, have more recently been augmented by the addition of a fourth V, veracity.

Volume

It is now almost a cliché to say that 90% of the existing data has been generated in the last two years. Around 2.4 trillion gigabytes of data are generated globally each day. Much of it arises through the internet and information generating digital analytics devices such as smartphones, digital analytics cameras and CCTVs and it is growing exponentially. It is estimated that by 2020 worldwide corporate data will exceed 35,000 exabytes where an exabyte is one quintillion bytes. 
Despite its size, big data is not  all about volume. In principal any volume of data could be processed using conventional database software if volume was the only issue.

Velocity

Data velocity, the rate at which information flows, is also increasing as a similar rate to volume. The increase in data velocity is in line with improving technologies are developing in accordance with Moore’s Law. 

Variety

The pivotal V that makes data Big Data is variety. While standard database software handles structured data, Big Data is often unstructured and cannot be processed with the same tool sets. In fact it can be a combination of various data categories including structured, semi-structured and unstructured. Typically it could consist of XML, database tables, audio and video files, text messages, tweets, and so forth.

Veracity

Veracity is an obvious addition. Unless the data is relevant, accurate and can be trusted, it is of little or no value. Ensuring the veracity of Big Data can be a challenge as it is difficult to control its quality. Any organization using Big Data must have the means of deciding whether it is beneficial and the extent to which it can be trusted.

Dealing with Big Data

One of the more popular ways of dealing with Big Data is Apache Hadoop. Named after a toy elephant it is an open source project designed to enable the large scale storage and processing of big data sets across server clusters.
It is hugely saleable and can readily be scaled from a single server to many thousands of servers either in premise or in the cloud. Originally developed by Yahoo and Google, its users include Yahoo, Facebook, Twitter, LinkedIn, and many more.

In addition to its scalability, it provides an inexpensive approach to massive parallel computing. It is flexible and can handle any kind of structured or unstructured data from unlimited sources which can be joined and aggregated and is fault tolerant.

Another popular approach is NoSQL. Also open source, it is a database framework that enables the storage and processing of large quantities of structured and unstructured data.

Big Data in Practice

Big Data and analytics have proven to be a success for many organizations. Its applications have included:

  • Understanding and targeting customers –  one of the main applications of Big Data. Examples of this in practice include Amazon Recommendations and personalized Tesco money-off coupons
  • Business processes – applications include optimizing supply route logistics, stock control based on social media trends, and HR processes including recruitment
  • Healthcare – for instance the side effects of drugs, correlations between lifestyle and health, the human genome, and the spread of infections
  • Big science – for instance the LHC at CERN generates one petabytes of data a second. Although most of it is discarded, CERN scientists store and process 30 petabytes a year using 65,000 servers.
  • Security and law enforcement – in the US the NSA uses Big Data in its war on terrorism as does GCHQ in the UK.

Is Big Data Over-hyped?

While the value of big data is clear, it isn’t a panacea. Certainly it has failed to live up to many of its early expectations and, according to some commentators, it has passed the peak of inflated expectations and has descended into the “trough of disillusionment”.

The backlash set in following the failure of Google Flu Trends which claimed to identify flu outbreaks using search queries. It got it spectacularly wrong, overestimating cases significantly from 2009 to 2013.

Big Data has several intrinsic weaknesses. These include:

  • While Big Data can detect subtle correlations, it can’t show causal relationships. This can lead to bad and dangerous conclusions. For instance the increasing number of autism diagnoses has been highly correlated with organic food sales.
  • Big data throws up correlations that appear to be statistically significant, but they happen just by chance simply because of the volume of data. The harder you look the more patterns you find even though they aren’t really there.
  • Big Data advocates have claimed that searching for models is no longer relevant as Big Data alone can deliver the answers. This is a dangerous and potentially catastrophic position that fortunately is losing sway.

Finally

Big Data is a large amount of data that may be structured, unstructured or both. It is characterized by its volume, velocity, and variety, and to be valuable it must have veracity too. However its real value is realized only when analytics are used to extract from it useful information.

It has changed how we do business, interact with each other and our customers, and protect our citizens from terrorism. Its benefits are clear, but so too are its potential dangers. Regardless, it’s here to stay so we should ensure that we learn how to handle it.

 

Harnham blog & news

With over 10 years experience working solely in the Data & Analytics sector our consultants are able to offer detailed insights into the industry.

Visit our News & Blogs portal or check out our recent posts below.

The Art And Science Of Tech In HR

The Art & Science Of Tech In HR

Human Resources. Talent Acquisition. People Officer. No matter what title a person has whose responsibility is finding the right person, for the right job, at the right time, the heart of the matter is they’re dealing with people. You know, humans. So, as the word “tech” becomes a suffix to every industry, think FinTech, MarTech, AdTech, and so on disrupting the old ways to make way for new, more efficient ways, what does that mean for the human resource industry? While there remains some concern about the industry in the age of the fourth industrial revolution, for the most part, many companies see technology in HR as good overall. But first, let’s get some misunderstandings out of the way. What Tech in HR Is Not Automation, Robotics, and AI are not taking over jobs, they’re helping to make a Hiring Manager’s job easier. By automating certain processes, it can give hiring professionals the time they lack to get back to the human side of things. You know, human-to-human communication. In fact, it’s not really disrupting jobs at all, rather it’s disrupting tasks. It’s either substituting or augmenting, just as humans do but wish they didn’t have to, so they could spend more time on the people side of things. Though we’ve mentioned a few things, Data Science in HR is not, there still remains some debate. Check out this article on whether or not Data Science will save the industry or forever change it leaving years of best practices in the dust? What Tech in HR Means for Growth Today’s tech world is a robust ecosystem of art and science. In Human Resources, the merging of a person looking for a job with a company looking for someone qualified to fill it takes a delicate touch of personality and technical prowess. HR has always been a science, of sorts. Now, it’s got the tools to help your company grow and keep up with your competitors. These tools include software applications which can offer improved reporting metrics, people analytics, Machine Learning, and NLP to help you improve such areas customer service. Would you rather have a person at the computer 24/7 or a chatbot to answer basic questions until a human can get online and take the conversation further? This is just one way in which technology is helping to improve the industry, not hurt it. Besides, when your customers are happy, so is your company’s bottom line. There is not a one-size-fits all approach and as technology learns to navigate the complex systems of people, businesses, and their behaviors, but there is much to learn. However, with the near limitless storage, computer processing ability and availability, and data models for predicting outcomes, the sky really is the limit. The mark of HR will be its ability on using data to steer the organization’s future with an upskilled and reskilled workforce. How to Not Get Left Behind There’s comes a time in every organization’s life in which it knows what it needs to grow, but isn’t yet ready to part with the old ways. The only problem, especially in today’s world, is that world doesn’t wait, and if you’re not ready and willing to pivot, you could get left behind. KPMG International’s Future of HR global study, which surveyed 1,200 global HR executives, reports conflicting attitudes and approaches to the challenge between action and inertia. According to the report, only 41% of respondents recognized the need for transformation but were in a “wait and see” position, while 72% of CEOs reported they’d rather be the disruptor than the disrupted. So, where is the disconnect? We know now, that even with a degree, pedigree, and a solid set of technical skills as a foundation, transformation is happening at a breakneck speed. So how can companies ensure their employees can keep up? One of the hottest topics across industries is reskilling and upskilling. Following in the footsteps of Google, IBM, and Microsoft, businesses today are more focused on ensuring individuals can demonstrate their skills, not just list them on a document. Think skills assessments and simulations, practice projects, virtual and mock experiences. It’s the “try before you buy” model. This is a near 360-degree shift from historical practices.  Every industry had its growing pains as it marched forward toward progress, but with the demand for highly skilled data professionals who can also resonate at personal level across the business landscape (read: soft skills), then the potential, if executed well, could be a boon for the industry.  Disruptions are meant to shake up the way we do business and answers the needs of our buyers. Easier, faster, more efficient, smoother, and the list goes on. Technology can free up time by taking on those repetitive and redundant tasks, giving you more time to be with your customer, whoever and wherever they are. Are you looking for a new opportunity in Data & Analytics? We may have a role for you. Check out our current vacancies or get in touch with one of our expert consultants to learn more. For our West Coast Team, call (415) 614 - 4999 or send an email to sanfraninfo@harnham.com.  For our Mid-West and East Coast Teams, call (212) 796 - 6070 or send an email to newyorkinfo@harnham.com.

Kyle Wong

Kyle Wong

15. August 2019

AR & VR Breathe Life Back Into Our Lives

AR & VR Breathe Life Back Into Our Lives

Remember the bicycles piled in the yard that showed where your friends were? What about the jingle of coins in your pocket as you headed to the arcade? Stranger Things may have recalled these distant memories, but what if you could still see bicycles piled against your neighbors’ door or listen to the jingle of coins as you played with your friends? This is Augmented Reality and its aim now is to get you off the couch, exercising, playing, and enjoying camaraderie. Ready, Player One? Five Trends to Watch As we navigate the next stage of our virtual and augmented realities, there are a few trends to watch. AR, VR, and AI Will “See” Objects - Augmented Reality (AR), combined with Artificial Intelligence (AI) and Computer Vision, will help computers “see” and label what’s being seen. Machine Learning will ramp up to offer increasingly correct identification of objects whether it’s the dinosaur exhibit at the natural history museum or connects the dots for stargazing in a planetarium.VR Gets More “Real” for the Mainstream – In other words, developers are able to offer more immersive experiences. New developments in hardware technology, such as eyeball tracking and field-of-view help power the idea users can interact and explore less like a video game and more like real-life.AR Can Help Keep Your Eyes on the Road – Remember KIT? The car who talked, could self-diagnose, and navigated with barely a hand on the wheel? It’s not so far-fetched now as it once seemed as vehicle manufacturers increasingly opt for voice assistants and some begin to offer graphics’ overlay of footage around the car. Others go a step farther, projecting data onto the car windshield, assisting with navigation, lane identification, and potential hazards along the way.AR-Based Entertainment Branches Out – No more piled high pizza boxes, sitting in the dark of your parent’s basement. AR today is about getting moving; physical activity and balancing tech life with real life. Toys and Gaming companies are on the bandwagon to get families moving. Location-based gaming has grown by leaps and bounds as has interactive projection-based technology. Some applications can transform your immediate environment into an immersive gaming experience while others can transform playgrounds which, when unlocked by parents, can offer choose-your-own adventure types of projects to complete. Enter whole being of mind, body, spirit. And we’ve barely scratched the surface.Worldbuilding, Personalized Character Building, and Augmented Board Games – Each of these are about bridging the gap between the physical and digital world and is most changed when it comes to board games.  As exciting as these trends are, it’s not all fun and games when it comes to the next advances of AR. This technology is also being used to help discover early onset of Alzheimer’s and most recently has found its way into the operating room. Or at least, it will be soon as companies look to bring the AR and VR technologies into the OR. Training and Teaching with AR AR & VR technologies are being used for teaching and training in just about every industry it seems. From Walmart to the Military, business is seeing the benefits of allowing individuals to practice their roles without the associated real-world risks. But when it comes to the healthcare industry, this is where AR begins to really shine. What if we could predict not only detect early onset dementia, but catch the onset of mental illness as well? One company, recently cleared by the FDA, is working toward just that, with the specific goal of predicting Alzheimer’s early. With an estimated cost of around $290 billion which could rise to over $1 Trillion by 2050, their claims of 94% accuracy in detection 6 to 10 years early is good news for families. Memories are how we share our stories and no family wants to miss theirs. Could technology as disparate as it once made us be bringing us closer together? With these latest, that just may be the case. So, whether you want to let out your inner child, practice a new skill without real-world ramifications, or simply know your memories will be with you for a lifetime, AR offers something for everyone. Every industry. Every business. It has infused our world and is breathing life back into our lives.  Want to be the wizard behind the curtain, the Data Analyst to know what’s next, or the Data Engineer who builds the next great technological advance in the AR spectrum of services and capabilities? We may have a role for you. Check out our current vacancies or contact one of our recruitment consultants to learn more. For our West Coast Team, call (415) 614 - 4999 or send an email to sanfraninfo@harnham.com.  For our Mid-West and East Coast Teams, call (212) 796 - 6070 or send an email to newyorkinfo@harnham.com.

Mairead Holt

Mairead Holt

08. August 2019

Diversity in Data

Coming Together: Diversity In The Workplace

Diversity. It’s a hot topic fused into every discussion from the board room to the upper echelons of government. And it’s more than the gender pay gap. It’s about bringing people together with different backgrounds, ideologies, processes, and creative persuasions. This is how you create your dream team – different ideas focused on a common goal.  We all know this at our core, but what does it mean to put it into action? In our 2019 Salary Guide, we covered a wide spectrum of how to hire and retain employees and within it discovered where the diversity gaps lay within the Data & Analytics industry. Marketing & Insight roles may have had the smallest gap, but Data Science teams had the largest. On the heels of our Salary Guide, keep an eye out for our upcoming Annual Diversity Report. But before we can do anything, we need to talk about culture. Specifically, changing our business culture to embrace diversity and inclusion. Did you know this change could help attract top talent and drive stronger teams to more innovative results? How to Put Diversity Initiatives Into Practice According to a 2018 study from McKinsey, diverse companies from their general workforce to their leadership are 33 percent more likely to have higher profitability than their competitors who are not as diverse. For gender diversity, the margin is 21 percent. Add to this, government regulations, candidates’ evaluation of diversity in an organization, and the company’s own plans to improve their efforts, and the task seems daunting enough. Diversity questions are no longer relegated to the just one group or another, now the question comes from every direction. Every location.  Below are just a few ways to begin your Diversity and Inclusion Initiatives Be Open to New Things and Establish a Sense of Belonging. Create a space where each person can bloom and shine. Establish a connection in which people are relaxed and can be themselves. Give them opportunities to create and engage in the workplace. Leadership is Key. Lead by example. Show empathy and avoid making diversity and inclusion the exclusive domain of HR. This is a time for everyone to get involved, to understand, and to remember why diversity is important. It’s more than ticking a box and it’s not one-size fits all. But each step forward is for the betterment of the business.Make Inclusion Part of Your Company Culture. Don’t think of diversity and inclusion as a one-day workshop. This is the time where you bring everyone together to learn what it means to be inclusive. Have you ever felt out of place or uncomfortable in a situation? Did anyone come to include you into the goings on or were you left to your own devices? Put yourself in other’s shoes and have your team do the same. Then identify behaviors and build new habits which support open and honest communication. It is okay for people to disagree, it’s what leads to real change. The key is to not let biases be mistaken for healthy discussion.You are Your Brand. Wear it Well. Company culture and brand are linked. They are infused in the products and services you offer to the world. Does your company culture of diversity reflect that of your customer base? What do you want your brand to say? How do you want to be known as a company? What new idea might you come up with divergent voices giving their thoughts and opinions? These are just a few of the ways you begin to establish a diverse and inclusive culture in your business. Now is the time to adapt your processes to scale for these behaviors. Ask yourself about the makeup of your meetings. Who attends? Who speaks? Is anyone being left out whose input you value? People First, Then Data As companies struggle to begin their diversity initiatives, there are still some caveats. The first is to remember you’re hiring people, not the data on a spreadsheet. Every business knows the importance of data and People Analytics is no different. But the problem begins with too small numbers as businesses try to pinpoint where they need to improve their diversity efforts.  Limited data about certain groups within the larger can be misleading. So, while it may seem counterintuitive, the answer is to broaden categories. Rather than focus on ethnicities, age, background consider the group overall. It’s the breaking down of demographics, where businesses begin to misstep on their path to diversity. So, what are some steps you can take to help improve your diversity initiatives? Well, here are a few to get you started. Avoid sample-based analyses. Focus on a range of outcomes. For example, are women represented well? What about women of color? Is there a wider variety of ages? Is the male contingent homogenous or are different demographics represented?Talk to employees and dig past first glances. Interviews with staff help to remind you these are people who cannot be defined by a statistic. It’s during this time you can learn more about them, their struggles, aspirations, and cultural insights. Descriptions take into account surface information such as race, gender, socio-economic status, and so on. Decisions based on data ripple through people’s careers and can affect their livelihoods.Ensure managers are engaged as allies. Just as leadership is key to ensuring the company culture embraces diversity, so too are the managers. It is they who are crucial as they make key hiring decisions, determine projects, and develop employees for advancement. Leadership offers an overview, but it’s the managers who shape the day-to-day of the employee experience.  One Final Thought… Most of us don’t start out intending to exclude anyone, but we naturally gravitate toward people like us or whom we imagine can best benefit our business. But when we open ourselves up to the possibilities of a more diverse workforce, the possibilities are endless. To begin, however, we must understand where the problems are, and from there fix them. At Harnham, we’re proud to be diverse in our company culture. In our inaugural Diversity Report, we showcased our near 50-50 split within our leadership and our efforts to be inclusive throughout our organization.  If you’re interested in Big Data and Analytics, we may have a role for you. Take a look at our current vacancies or contact one of our expert consultants to learn more. For our West Coast Team, call (415) 614 - 4999 or send an email to sanfraninfo@harnham.com.  For our Mid-West and East Coast Teams, call (212) 796 - 6070 or send an email to newyorkinfo@harnham.com.

Elizabeth Sobel

Elizabeth Sobel

30. July 2019

Data Privacy Rules

Privacy Rules. Or Does It?

Privacy and transparency are two sides of the same coin. With the amount of Data we give and companies consume, we want to know our personal information is safe. We want to know we are safe from illegal use of our information. But as conversations about FaceApp, Facebook, Cambridge Analytica, and Google privacy issues make the rounds again, we now know to be much more cautious. Companies would be wise to follow. States Take the Reins to Enact Privacy Rules Though the General Data Protection Regulation (GDPR) passed last year in Europe, businesses in the U.S. have not been so constrained. While there is no overarching federal law, states have taken steps to protect the privacy of their residents and have passed their own Data protection laws. Though all 50 states have enacted notification laws to inform consumers if personal information has been compromised, only California and Vermont have instituted laws requiring businesses to make real change in their Data operations. Other states, from Oregon to Virginia have expanded their definitions of identifying information and increased fines to $500,000 for breaches of privacy. These more stringent rules affect such information as that from Electronic Health Records to tax preparers. And when it comes to Data disposal, companies are required to shred or modify in some way any personal information before tossing it away. Student information is particularly protected in Iowa, in which online efforts against selling their information or otherwise siphoning from online profiles are expressly forbidden by state law.  These are just a few of the rules in place and vary slightly from state to state. So, how can you ensure you’re in compliance? Some Tips to Ensure You’re in Compliance If you need to create or amend your Data Management program, here are a few tips to consider: Conduct a gap assessment. What existing procedures are in place which may need to be revised?Ensure your legal teams work closely with your IT, business, and marketing teams to monitor changes and reassess your company’s mitigation controls. How effective are those controls within this legal landscape?Ensure the consumer Data you’re collecting is “critical” to the company. Create a process to receive, review, and fulfil customer requests. But also consider how you handle their information should a customer wish to opt out.Train employees on how to handle personal information. Create and maintain procedures on policy changes and best practices for your Data protection policies. A final note on the above tips, though each state has their own laws, it’s important to consider Golden Rule when it comes to privacy; how would you want your personal information handled? Data is a Commodity. Trust is Valued.  Broken promises have tarnished trust in companies. According to a recent survey by SAP, nearly 70% of customers said they don’t trust brands with their personal information. So as companies strive to offer the best customer experience, remember it’s more than flashing lights and deep discounts.  Customers want to know their personal Data is safe. So how you ensure this is the case and maintain your customer’s trust? Be transparent. Collect customer information with clear intentions and keep your customers informed of changes to policies.  Legal verbiage in policies are to protect companies. It’s time to rethink this strategy and enact policies to protect customers. Though their wariness is warranted, consider how not being transparent and protecting your own business has been detrimental to the customer experience. By being proactive in Data policy compliance laws, you let customers know you’re putting their needs first. That builds trust and loyalty to your business. Isn’t that what every business strives to attain? Even the tech companies realize its import and impact. Earlier this year, tech companies laid out what they’d like to see in federal Data privacy laws. The key takeaway? One set of rules for all is preferred over the slightly differing state laws. If you’re interested in Big Data and Analytics, we may have a role for you. Check out our current vacancies or get in touch with one of our expert consultants to learn more. For our West Coast Team, call (415) 614 - 4999 or send an email to sanfraninfo@harnham.com.  For our Mid-West and East Coast Teams, call (212) 796 - 6070 or send an email to newyorkinfo@harnham.com.

Bryan Minami

Bryan Minami

25. July 2019

Recently Viewed jobs