Harnham
SEARCH
Shortlist

Big Data Trends to Keep Your Eye on



big data processing

Big data trends to keep your eye on


Big data just keeps getting bigger, and 2016 has been no exception. Organizations are increasingly realizing that data can reveal major insights and lead to better-informed decisions and business advantages. As we edge closer to the end of the year, we thought it was time to take stock of the emerging trends that are only set to get bigger.

More scrutiny on security


The last few years have showcased how data leaks and hacks can have a disastrous effect on organizations and consumers. The scandalous Ashley Madison hack highlighted the need for ultra-secure systems when dealing with delicate consumer information, while the TalkTalk data hack showed the risk of hacks on high-profile businesses and the need for enhanced password security measures from consumers.

As a consequence of a string of high-profile data breaches, we’ve seen many organizations place increased scrutiny on how data is protected and used. As the new EU General Data Protection Regulations come into play for many businesses, we’ll likely see more spending on data security measures and preparation for further enhancements to tighten up sensitive data. With the EU Data Protection Regulation coming into effect across the EU, it’s likely the protection of and access to big data will become more critical than ever.

Big data in healthcare


The healthcare field has seen enormous benefits from the development of big data in recent years. 2016 has seen more medical information become digitized, loaded up to the cloud and analyzed in more technological ways than ever. The emergence of big data in the medical world could see more information and facts readily available to complement the opinions of doctors, bringing about a technologically-driven movement in how patient data is processed and reviewed. Big data also allows medical and test data to be shared more freely and quickly all over the world, allowing for more rapid and precise establishment of correlations in aggregated data.

Smart data for the masses


2016 was predicted by many to be the year that big data became smarter but not necessarily bigger. Instead of looking to gather as much information as they possibly can, analysts and scientists have instead been focusing on data that provides more insight than before, including features such as time, location and links between data sets.

The use of data has also extended beyond major organizations with dedicated big data programs through to smaller start-ups and even consumers who look to collect data to do things better, faster and smarter. If you use devices to track your activity levels, sleep patterns or even where and on what you spend your money, you’re part of the new data revolution.

What’s next?


If you’re interested in contributing to the future of big data, we might have the role for you. We specialize in Data and Analytics recruitment and always have a wide range of vacancies at both junior and senior level. Take a look at our current vacancies or contact us to find out more.

SHARE THIS PAGE

 

Harnham blog & news

With over 10 years experience working solely in the Data & Analytics sector our consultants are able to offer detailed insights into the industry.

Visit our News & Blogs portal or check out our recent posts below.

Big Data In Politics – Win, Lose, Or Draw

Big Data In Politics – Win, Lose, Or Draw

In the movie Definitely, Maybe starring Ryan Reynolds, there’s a scene in which he must sell tables for a political campaign dinner fundraiser. He makes call after call with no luck. Finally, in frustration, he speaks plainly and finds a connection between the politician and the prospective donor. In an instant, he understands. Make the connection and you can’t go wrong. This is the 90’s version of micro-targeting. Online advertising today has honed targeted Marketing to an art form and it’s infused every industry from Fisherman’s Wharf to Wall Street to Washington. Messages are crafted on detailed profiles of what makes us unique such as hopes, fears, dreams, emotional triggers, and more which is then taken out of the hands of humans. Enter such deep, personal details into automated technologies and you’ll get automated reactions. How did we get here? Ever since Cicero’s brother, Quintus, who approached politics with a do anything to win mindset, we’ve been working toward this point. But, when it comes to technological advances within politics, George Simmel put it best when he wrote around 1915, “the vast intensive and extensive growth of our technology…entangles us in a web of means, and means toward means, more and more intermediate stages, causing us to lose sight of our real ultimate ends.”  What does this mean? It means we have moved so quickly and with such intensity as we push inwards while reaching outward, we get tangled up in our own systems. Before we know it, it’s difficult to separate the means from their ends, and we lose sight of our purpose. In other words, it can be hard to keep our sense of direction with our constant distraction of tasks, systems, and processes. According to Simmel, this would soon morph into what he called a ‘fragmentary character.’ Like a mosaic, we put the pieces back together and assemble the bits to fit our concept of the world.   The Digitizing of Campaigns Traditional campaigning has traditionally looked much like the movie scene mentioned above with phone banks, whiteboards, and handmade signs. But, today, things are changing. Everyone has at least one smart device which can sync information in real time to a range of devices. Algorithms and predictive modeling help reduce the guesswork, though gut feeling and instinct still prevail. At least, for now. Our machines are learning how to learn about us and define what we believe and wish to see by historical Data, or rather our past behaviors. Where psychographic profiling meets micro-targeting. What was once only seen in the Marketing world has now entered politics. Just like marketers want to know what people are interested in, so to do politicians wish to know what voters think. To do this, both industries will study behavioral and attitudinal profiles to help understand a demographic better or discern a gap in the marketplace. In consumer research, companies rely on psychographic micro-targeting to reach smaller groups and individuals. The key question here is to ask is to what extent are politicians prepared to pass laws that restrict their own opportunities to know more about voters. Just as the next generation of voters are coming, so too are the next generation of tools being developed.  One Final Thought… Over the last 20 years or so, we have built an immense Data structure from mobile devices to social media to modelling processes and more. With this kind of connectivity combined with fragmentary media, the use of Data Analysis has a big role to play going forward. If we seek change in our political and social infrastructures, we will have to reimagine the structures currently in place. From algorithmic modelling to AI and Machine Learning, the possibilities for new ideologies has emerged blurring the lines between context and production in which Data underpins capitalism. As those in Data Analytics continue to pursue an uninterrupted (read: non-fragmentary) vision of the world, we find ourselves at a new stage in history of where both looking back and looking forward at the same time informs our future.   Where would you like to go? If you’re interested in Big Data & Analytics, we may have a role for you. Take a look at our latest opportunities or contact one of our expert consultants to find out more:  For our West Coast Team, call (415) 614 - 4999 or send an email to sanfraninfo@harnham.com.  For our Mid-West and East Coast Teams, call (212) 796 - 6070 or send an email to newyorkinfo@harnham.com.

Joshua Poore

Joshua Poore

10. October 2019

Going Green With Big Data

Going Green With Big Data

Greta Thunberg sailed the Atlantic to come the UN to talk about climate change. Her mother, a renowned opera singer, has given up air travel to support her daughter’s efforts. There is a zero-waste movement to lessen our trash and help alleviate the carbon footprints from our buying, traveling and more. These are steps humans have made. Yet technological advances may make it possible to flip the script for the environment and Big Data has a big role to play.   What are Some of the Advances Taking Place? Technological advances have brought us breakthroughs in modern science and in every industry. Now, we are at a time and place in where our technologies cam help tackle climate change. From modeling to predictions, we can begin to build not just a map of environmental concerns, but begin to build a road toward a solution. Below are just a few of the ways technology is being used to advance solutions for climate change. AI modeling makes it easier to identify problemsPredictive Analytics models can create different scenarios to see ‘what happens if?’Big Data is used to identify areas which need immediate attention This is just the tip of the iceberg when it comes to using technology to predict and identify climate concerns. While some parts of the world contribute more to the problem than others, Big Data has made it possible to draw conclusions where the hardest hit areas are and is key to addressing the problem. But whatever Data brings, the information is useless if it isn’t used to formulate and put forward better environmental practices and policies.  Ways to Upscale Urban Data Science  Manhattan, Berlin, and New Delhi, as varied as they are, have one thing in common. They’re often sites for case studies when it comes to analyzing our environment. However, our advances continue to improve and we’re able to learn from state-of-the-art Data infrastructures. These can include such things as social media data combined with earth observations to see how they might better integrate. A research publication in Berlin suggest three routes for expanding knowledge. They are: Mainstream Data collectionsAmplify Big Data and Machine Learning to scale solutions and maintain privacyUse computational methods to analyze qualitative Data With these advances in place, there is a chance urban climate solutions could effect change on a global scale. With the proper Data of urban areas in place, including that of related greenhouse gases, socio-economic issues, and climate threats, Data professionals can get a clearer picture of what needs to be done. Building on the advances that are in place with the integrated technologies of AI, Predictive Analytics, and Big Data helps make big strides in combatting climate change. According to reports, only about 100 cities make up 20% of the global carbon footprint. Yet 97% of climate concerns are focused in urban areas. There’s still a lot which remains to be done to combat the greatest issue of our age, but working hand in hand – machine and human – we just might find ourselves on reprieve and the chance to leave the world better than we found it for the next generation. The next Greta Thunbergs of the world. If you’re interested in Big Data & Analytics, we may have a role for you. Check out our current opportunities or get in touch with one of our expert consultants to learn more.  For our West Coast Team, call (415) 614 - 4999 or send an email to sanfraninfo@harnham.com.  For our Mid-West and East Coast Teams, call (212) 796 - 6070 or send an email to newyorkinfo@harnham.com.

Thomas Daughtrey

Thomas Daughtrey

03. October 2019

Data Engineer Or Software Engineer: What Does Your Business Need?

We are in a time in which what we do with Data matters. Over the last few years, we have seen a rapid rise in the number of Data Scientists and Machine Learning Engineers as businesses look to find deeper insights and improve their strategies. But, without proper access to the right Data that has been processed and massaged, Data Scientists and Machine Learning Engineers would be unable to do their job properly.   So who are the people who work in the background and are responsible to make sure all of this works? The quick answer is Data Engineers!... or is it? In reality, there are two similar, yet different profiles who can help help a company achieve their Data-driven goals.  Data Engineers  When people think of Data Engineers, they think of people who make Data more accessible to others within an organization. Their responsibility is to make sure the end user of the Data, whether it be an Analyst, Data Scientist, or an executive, can get accurate Data from which the business can make insightful decisions. They are experts when it comes to data modeling, often working with SQL.  Frequently, “modern” Data Engineers work with a number of tools including Spark, Kafka, and AWS (or any cloud provider), whilst some newer Databases/Data Warehouses include Mongo DB and Snowflake. Companies are choosing to leverage these technologies and update their stack because it allows Data teams to move at a much faster pace and be able to deliver results to their stakeholders.   An enterprise looking for a Data Engineer will need someone to focus more on their Data Warehouse and utilize their strong knowledge of querying information, whilst constantly working to ingest/process Data. Data Engineers also focus more on Data Flow and knowing how each Data sets works in collaboration with one another.    Software Engineers - Data Similar to a Data Engineers, Software Engineers - Data ( who I will refer to as Software Data Engineers in this article) also build out Data Pipelines. These individuals might go by different names like Platform or Infrastructure Engineer. They have to be good with SQL and Data Modeling, working with similar technologies such as Spark, AWS, and Hadoop. What separates Software Data Engineers from Data Engineers is the necessity to look at things from a macro-level. They are responsible for building out the cluster manager and scheduler, the distributed cluster system, and implementing code to make things function faster and more efficiently.  Software Data Engineers are also better programers. Frequently, they will work in Python, Java, Scala, and more recently, Golang. They also work with DevOps tools such as Docker, Kubernetes, or some sort of CI/CD tool like Jenkins. These skills are critical as Software Data Engineers are constantly testing and deploying new services to make systems more efficient.   This is important to understand, especially when incorporating Data Science and Machine Learning teams. If Data Scientists or Machine Learning Engineers do not have a strong Software Engineers in place to build their platforms, the models they build won’t be fully maximized. They also have to be able to scale out systems as their platform grows in order to handle more Data, while finding ways to make improvements. Software Data Engineers will also be looking to work with Data Scientists and Machine Learning Engineers in order to understand the prerequisites of what is needed to support a Machine Learning model.   Which is right for your business?  If you are looking for someone who can focus extensively on pulling Data from a Data source or API, before transforming or “massaging” the Data, and then moving it elsewhere, then you are looking for a Data Engineer. Quality Data Engineers will be really good at querying Data and Data Modeling and will also be good at working with Data Warehouses and using visualization tools like Tableau or Looker.   If you need someone who can wear multiple hats and build highly scalable and distributed systems, you are looking for a Software Data Engineer. It's more common to see this role in smaller companies and teams, since Hiring Managers often need someone who can do multiple tasks due to budget constraints and the need for a leaner team. They will also be better coders and have some experience working with DevOps tools. Although they might be able to do more than a Data Engineer, Software Data Engineers may not be as strong when it comes to the nitty gritty parts of Data Engineering, in particular querying Data and working within a Data Warehouse.  It is always a challenge knowing which type of job to recruit for. It is not uncommon to see job posts where companies advertise that they are looking for a Data Engineer, but in reality are looking for a Software Data Engineer or Machine Learning Platform Engineer. In order to bring the right candidates to your door, it is crucial to have an understanding of what responsibilities you are looking to be fulfilled. That's not to say a Data Engineer can't work with Docker or Kubernetes. Engineers are working in a time where they need to become proficient with multiple tools and be constantly honing their skills to keep up with the competition. However, it is this demand to keep up with the latest tech trends and choices that makes finding the right candidate difficult. Hiring Managers need to identify which skills are essential for the role from the start, and which can be easily picked up on the job. Hiring teams should focus on an individual's past experience and the projects they have worked on, rather than looking at their previous job titles.  If you're looking to hire a Data Engineer or a Software Data Engineer, or to find a new role in this area, we may be able to help.  Take a look at our latest opportunities or get in touch if you have any questions. 

Bryan Minami

Bryan Minami

26. September 2019

Sean Byrnes, CEO Of Outlier.Ai, On Creating A Business With Values

Sean Byrnes, CEO Of Outlier.Ai, On Creating A Business With Values

We sat down with Sean Byrnes, CEO of Outlier, an Analytics solution provider, to learn more about Outlier and its values. He also shared us with the power a candidate has when applying and interviewing for jobs in the tech industry today as well as how employers can retain their top talent. How long has Outlier been in business? We’ve been in business about four years. Prior to starting Outlier, I had another company called Flurry which we sold in 2014. When we sold it, I took a year off to reset my internal counter. I needed to reorient my work/life balance and then, when I felt I was in good shape to get back into things, we kicked off the launch of Outlier.  When you began Outlier, did you plan to map out your values of diversity, inclusion, transparency, and work/life balance? Or did it evolve as the company grew? Planning out our values is one of the first things my co-founder, Mike Kim, did when we were starting the company. It was intentional. We sat down and wrote down those values you see on the website.   I learned a lot in my previous company and there were a lot of things I did which followed the values we have which we didn’t follow intentionally. It just felt…right. Growing up in New York, a very diverse place, having a diverse team always felt more natural to me than a homogenous team. Add to that, having just spent a year with my new daughter and adjusting to being a parent and what that meant; it put things in perspective.  If there’s one thing I’ve learned it’s this. There’s no reward for running a company well. There’s no reward for following good practices and treating your employers with respect. In fact, the reason I started my first company was that I’d worked a lot of places that treated employers like resources or like widgets.  You put salary in one side and productivity comes out the other. I wanted to work somewhere that treated people like people. So, I started Flurry, and now have come into Outlier with a solid idea of our values and company culture. As important as tech skills are, there’s no data to support a feedback mechanism and predict success. Success comes from unexpected places.  So, by sitting down and writing out those values, we weren’t just signing up for a contract for how we would treat the people.  In our way, we were standing up as an example of how you can build a tech company that didn’t follow these bad habits.   Someone said, not too long ago, that the best employees stay in the same company for about 20-months before moving on to their next project. Often, it’s because the employee no longer feels creative.  In thinking through your company values statements, what would you say to a business who’s hoping to both attract and retain their top talent? Or do you think it’s better for these individuals to rotate off in order to keep things moving?  Tech companies typically have two problems: Companies are so desperate to keep going, they do whatever they can to survive. They promise themselves they’ll make compromises and fix the short cuts when the time is right.  So, they hire people who need not share their values or may not meet their criteria with a promise that when the company is successful, they will fix it and that time never comes. You’re never at a point where you’re so successful you can go back and remake those mistakes or fix them.For many, recruiting is a one-way system. You search for and hire a candidate, then another, and another without really putting much thought into it.  The reality is that you can’t build a high growth business that way. What you have to believe is that if you spend the time to find the great people that enjoy working on what you working on, where you treat them with respect, you give them not just responsibility but also the authority to do things that you create gravity. You create a world where the people you hire pull in the next group of people because people really want to work them, they want to be in that environment. A lot of our most recent hires here at Outlier are people that came to us, who wanted to work in this environment and when they saw how great our team are they want to add to that and it becomes a self-fulfilling cycle where the larger your core of great people the better your gravity is, the more great people it pulls in and so the gravity needs to expand and that’s how you build a high grow organization. You don’t build a high growth organization by having the best sourcing process, by having the largest recruiting team, the best employ onboarding. You grow the fastest if you can create a community and an environment people want to work in and when it becomes self-evident of that then you start to pull people in. And so those become the culmination.  How do the values that you’ve initiated affect your company? Have you found that those values foster a deeper loyalty and higher moral than in other places and other business that sound a little bit like yours?  We have a very low attrition rate. Yeah, people have left due to life changes, but otherwise we still have the same people we started with though their roles may have changed as we grew. There was nothing we set out to do, nothing on purpose to keep them, but just did things which felt right. In retrospect, it comes down to this. If you create an environment where people are learning, where they feel valued, and they enjoy the work, why would they leave? Where would you go? What could you prefer to that environment? I think by focusing on this early, it’s led to more employee retention.  My hope is that everybody’s who’s here will continue along the journey as we build the company together. Another example is our work/life balance. Both my co-founder and I are parents of young kids. My first few hires were parents of young kids. So, being family friendly has always been a core value of ours from the beginning.   Being family friendly is both a conceptual and a practical concept. Our policy is to be in the office three days per week and work from home two days per week. This gives our parents a chance to do things like take their kids to the doctor or attend a parent/teacher meeting. It’s become an enormously strong aspect for us because there’s a lot of people who have young kids and they don’t want to work in your stereotypical tech company. You know the one, the company who expects you to be in the office for twelve hours a day seven days a week.   Our employees like the challenge of building tech companies, but they need the flexibility we offer, too. So, just by having a simple stand of valuing people as people and parents as parents, means we have a competitive advantage in recruiting for roles. The senior people who can contribute at a very high level where they suddenly didn’t have any options and this is a chance for them to keep doing what they doing. They’re able to do what they love and not feel they have to compromise their family on its behalf.   It’s the kind of thing where very simple principles becomes an enormously strong competitive strong advantage in todays market. While it hasn’t been true for very long, it has definitely been the case so far for us. If you’re interested in Big Data and Analytics, we may have a role for you. Take a look at our latest opportunities or contact one of our expert consultants to learn more.  For our West Coast Team, call (415) 614 - 4999 or send an email to sanfraninfo@harnham.com.   For our Mid-West and East Coast Teams, call (212) 796 - 6070 or send an email to newyorkinfo@harnham.com.

Ben Jones

Ben Jones

19. September 2019

Recently Viewed jobs