Big data: 4 predictions for 2014







Big data was seen as one of the biggest buzzwords of 2013, when companies often used the term inappropriately and in the wrong context. This year, people will finally understand what it means

One could look back at 2013 and consider it the breakthrough year for big data, not in terms of innovation but rather in awareness. The increasing interest in big data meant it received more mainstream attention than ever before. Indeed, the likes of Google, IBM, Facebook and Twitter all acquired companies in the big data space. Documents leaked by Edward Snowden also revealed that intelligence agencies have been collecting big data in the form of metadata and, amongst other things, information from social media profiles for a decade.

And beyond all of that, big data became everyone's most hated buzzword in 2013 after it was inappropriately used everywhere, from boardrooms to conferences. This has led to countless analysts, journalists and readers calling for people to stop talking about big data. A good example could be seen in the Wall Street Journal last week, where a reader wrote in complaining:

A lot of companies talk about it but not many know what it is.

While that's a problem, it leads to my first prediction:

1. In 2014, people will finally start to understand the term big data. Because, as it stands, many do not.

The truth is that we've only really just started to talk about big data and companies aren't going to stop screaming about their latest big data endeavors. In fact, it's only January and the social bookmarking network Pinterest has already acquired image recognition platform VisualGraph. (Why? Pinterest want to understand what users are "pinning" and create better algorithms to help users better connect with their interests).

So let's get 2014 off on the right foot with a definition of big data, from researchers at St Andrews, that's fairly easy to understand:

The storage and analysis of large and/or complex data sets using a series of techniques including, but not limited to: NoSQL, MapReduce and machine learning.

The main elements revolve about volume, velocity and variety. And the word 'big'? If your personal laptop can handle the data on an Excel spreadsheet, it's not big.

Matt Asay, a journalist with ReadWriteWeb, also does a good job in explaining what makes a big data problem (as opposed to more traditional business intelligence).

If you know what questions to ask of your transactional cash register data, which fits nicely into a relational database, you probably don't have a big data problem. If you're storing this same data and also an array of weather, social and other data to try to find trends that might impact sales, you probably do.

2. Consumers will begin to (voluntarily) give up certain elements of privacy for personalization.

We've all heard of cookies – and we know that our actions around the internet affect the adverts that we see on websites and the suggested items we receive on Amazon. This is a concept that we've not only become accustomed to but also accept. After all, if we're going to have information put in front of us, we'd rather that we could relate to it.

But there have been problems in the past. Some websites have taken advantage of customers, for example increasing the prices for a flight that they've previously expressed interest in (consumers might worry that the price will go up even further and therefore decide to buy a ticket).

But as more companies instil big data techniques, customers will cooperate, on the premise that they will benefit. This is likely to follow Tesco's methodology, whereby customers are sent vouchers for goods that they are likely to buy anyway, creating a win-win situation for both parties. Customers, generally, are happy to receive a discount and retailers are pleased customers are coming back (especially if vouchers have an expiry date).

3. Big data-as-a-service will become a big deal

Despite claims from analysts that all businesses will look to hire data scientists, this just isn't going to happen. Firstly, there's a shortfall of data scientists, which goes some way in explaining why companies are retraining existing staff to work with big data) and secondly, not all companies are ready to (nor do they need to) invest in full-time data scientists to analyze and explain their data.

Instead, just as in other areas, I expect a wave of companies hustling to enter the big data-as-a-service space, an idea that began to creep into the latter parts of 2013. This could be anything from small and medium businesses signing up to anything from entire packages of storing, analyzing, explaining and visualizing data to more compact services, which focus on transferring data to cloud-based servers to allow for an accessible way of questioning the data in the future.

4. And finally... remember how Hadoop is an open-source software? Expect a lot more of that.

Hadoop, famously named after a toy elephant, is a well known piece of software to anyone curious about data science and it provides the backbone for many big data systems, allowing businesses to store and analyze masses of data. Most importantly, it's open source, which means that its implementation was inexpensive, allowing many organizations to understand, rather than ignore, the data they were collecting.

Quentin Gallivan, the chief executive of business analytics software firm Pentaho, explained last month that the rise of new open-source software will bring about more innovation and more ways of understand the data. He said:

New open source projects like Hadoop 2.0 and YARN, as the next generation Hadoop resource manager, will make the Hadoop infrastructure more interactive... projects like STORM, a streaming communications protocol, will enable more real-time, on-demand blending of information in the big data ecosystem.


Click here for the article on the web.



<< Click here to see more recent news articles >>

 

Harnham blog & news

With over 10 years experience working solely in the Data & Analytics sector our consultants are able to offer detailed insights into the industry.

Visit our News & Blogs portal or check out our recent posts below.

How To Lead A Remote Team

GitHub. How-To Geek. Toptal. Zapier. These are just a few of the businesses which have been 100% remote since their inception. On the front lines of the remote working lifestyle, they understand the benefits and the challenges businesses new to remote teams are facing. In recent years, some businesses tried offering remote working, then pulled workers back in house. Some offer part-time work-from-home opportunities, but still required a few days in the office. In today’s climate, businesses have been forced to develop remote working strategies and everyone is learning how to manage this new way of doing business. There are a variety of apps and software to help businesses work as close to the old ways as possible. So, as remote workers learn how to balance their personal and professional lives while at home, leaders also must learn how to manage remotely. Below are a few immediate benefits of remote teams for business. Three Immediate Benefits of Remote Working Less overheadsLess office politicsIncreased productivity However, there’s more to leading remote workers than the above. As teams tighten, it’s important to ensure everyone is on the same page. Most of what’s been done within the office walls can also be done virtually. Planning a staff meeting? Zoom. WebEx. GoToMeeting. Google Hangouts. Microsoft Teams. These are all video conferencing call software applications you can use to not only conduct your weekly staff meetings, but client meetings as well. Record the meetings so you can refer to comments or questions, you may have missed or to refresh your memory of planned next steps. Managing projects? Think Trello. Asana. Monday. Basecamp. Slack. Assign teams. Have a space just for brainstorming or just for fun. Emulate the office environment of those quick hallway meetings and watercooler chats. Need to access documents from anywhere? Google Drive. Dropbox. Box. These are just a few of the applications which allow anyone with a link to jump in and add their two cents, comment or correct. Does your team have all the necessary equipment? Computers, phones, access codes, passwords? How can you help them best do their jobs? These are just a few things to think about when planning for and leading remote teams.  Leading Remote Teams One of the issues remote workers once faced was “out of sight, out of mind” leadership. In an office, it can be easy for a manager to walk past someone and communicate on the fly or for remote workers to feel left out in company events, strategy sessions, or general camaraderie with others in the office. Companies who have always been or are moving to fully virtual teams have now levelled the playing field. However, this change marks just how important communication is for your team. Often, the best managers will check-in regularly. These managers are good listeners who offer feedback, check on progress, and determine if there’s too much workload or if an employee is ready to take on another project. Managers who communicate without micromanaging engage in trust and accountability of their employees. Having this kind of ownership of habits and behaviors can improve productivity and the trust goes both ways. Reach out to everyone on your team regularly. To ensure you’re including everyone when you assign tasks or projects, it’s a good idea to have a list of employees with their photos. Set Clear Boundaries and Guidelines It’s even more important when working from home to set clear hours for working and avoid burnout. Add to this the best methods of communication to use such as a project management app like Slack, a quick text or Skype call, or an email. Don’t forget to make allowances for how things like childcare may be handled as many are not only working from home, but may also be homeschooling, too. Ensuring everyone is clear on when they can turn to colleagues versus when they should reach out to their leaders helps put everyone on the same page.  Keep Everyone Connected Remote working is a lifestyle. But not everyone may be ready for it or have the personality for it. One benefit of the brick-and-mortar office is camaraderie and connection, even if it’s just a smile and wave in the hallway. Even the most resolute remote workers feel lonely and disconnected at times. Staying on task can feel paramount to success, but to meet the new challenges of a virtual team go a little deeper. Not only is it important to make time for personal interaction, but businesses can organize virtual check-ins and fun events. Think Trivia time, funny videos or GIFs, even virtual pizza parties.  It’s important in this time of social distancing and remote working lifestyle to stay connected and engaged with your workforce. It could even make your business stronger and more in sync than ever before. Not only can it be of benefit to your business, but can also offer and open up opportunities for more future employees. Without a location base, you open yourself up for the right person for the job, no matter where they are.In the wake of work-from-home policies, remote working, and the shifting landscape of working outside the office, technology careers are prime opportunities to both gain increased knowledge in your chosen field or begin your career path.  If you’re interested in Big Data & Analytics or other Data professional opportunities, check out our current vacancies or get in touch one of our expert consultants to learn more.   For our Mid-West and East Coast teams contact us at (212) 796-6070 or send an email to newyorkinfo@harnham.com.   For our West Coast Team, contact us at (415) 614 - 4999 or send an email to sanfraninfo@harnham.com.  

Drawing the Line Between Work and Family When Working Remotely

So, you’ve been told to work from home. Finally, more time with the family… Wait, more time with the family? How will you get your work done while also homeschooling your children? Maybe you’re married and your spouse is home, too. The routine of work has been disrupted, and though many businesses were already turning to remote work, this is something else.  So, how do you morph from leaving the house to go to the office to simply walking into your kitchen or home office to begin your day? In other words, how do you draw the line between work and family when you’re working from home? We know it can be difficult and unsettling in this troubled time, so we have a few tips to get you started. Getting Started in Your Remote Working Lifestyle DEFINE YOUR WORK SPACE What room can you designate in your house to be your “office.” It’s best to have someplace with a door, but this isn’t always possible. Is it the kitchen table? Ok, but this will mean you need to set strict ground rules about the hours you’re “on.” Make sure everyone understands when you’re “at work.” Whether it’s your kitchen table, a quiet room, or the end of your sofa with your laptop, these are your remote working tools. In some cases, it may even be a good idea to invest in noise-canceling headphones to help you stay focused. HAVE SET HOURS Define what hours you’re working and stick to them. Begin and end your day at the same time. Don’t forget to schedule breaks – coffee break, lunch, a stretch of the legs – around the same time each day as well. Work with your team to set hours for when you’ll be online working and respond to off-hour messages within your working hours. Without designated hours, it can feel like you’re constantly available and always “on” blurring the lines between work and family. Get some fresh air when you can. Step outside for a walk or a coffee, whatever brings you outside can help recharge and energize you for the work ahead. ENSURE YOU HAVE THE RIGHT TOOLS Remote working apps, videoconferencing tools, and cloud-based filing systems help ensure the job gets done. Make sure you keep your passwords in a safe place and be extra cautious when logging in from a new location. Is your computer up-to-date? Does it have all the security measures and capacity in place for the additional online tools and apps you may need to add? Making sure you address these things can help to solidify your workspace and ensure you’re able to meet with your team online and get the job done. FOLLOW THE 20-20-20 RULE The American Academy of Ophthalmology recommends the 20-20-20 rule: every 20 minutes look away from your screen and focus your eyes on something 20 feet away for 20 seconds. TAKE A BREAK FROM TALKING ABOUT WORK If you’re not used to working-from-home, loneliness can quickly set in. Remember those quick hallway chats or discussions over lunch or coffee? Take that impulse and use it when talking with your team. Have a virtual coffee break. Take a break from work and talk about hobbies, something funny that happened to you, or even just how you’re feeling away from everyone. You won’t be alone in these feelings with everyone in the same work-from-home boat. These tips can help you put your best foot forward for your remote working lifestyle. But don’t forget, you can use these same rules for family time, too. In the wake of work-from-home policies, remote working, and the shifting landscape of working outside the office, technology careers are prime opportunities to both gain increased knowledge in your chosen field or begin your career path.  If you’re interested in remote Big Data & Analytics opportunities, we may have a role for you. Take a look at our current vacancies or contact one of our expert consultants to find out more.   For our West Coast Team, contact us at (415) 614 - 4999 or send an email to sanfraninfo@harnham.com.   For our Mid-West and East Coast teams contact us at (212) 796-6070 or send an email to newyorkinfo@harnham.com.  

How Computer Vision Engineers Develop the Eyes of AI

Facial Recognition software. Autonomous vehicles. Drone delivery. Robotics in manufacturing. 3D Printing. No longer the stuff of science fiction, these advancements are at the heart of the next evolution in the digital age. Developments are not just being made in the tech hubs of Silicon Valley, Austin, or New York, but in the mid-West. Ann Arbor, Michigan home to the University of Michigan and not too far from where Henry Ford first introduced mass production with the help of automation has been advancing robotic technologies across a variety of fields.  Giving machines their own set of eyes does require someone to ensure they have the right information to do their jobs. Enter the Computer Vision Engineer. It’s estimated this field will see a rise of 19% demand through 2026. It’s also a relatively small field with only 5,400 new job openings. So, like many professions, demand is high yet a shortage remains of those Data professionals with the right skillsets. The Business of a Computer Vision Engineer While there are a variety of roles within the field of Computer Vision, the role of Computer Vision engineer focuses on two areas. Those areas are: Writing code in Python/C++ Integrate Data Visualization, image analysis, and imaging simulation controls In addition to these areas, these scientists focus on research, implementation, reaching across teams both human and machine to help solve real world problems. And as important as knowledge and application theory are, it’s the hands-on experience which raises the bar for most employers and client companies.  Using image recognition, machine learning, and segmentation can help machines learn to differentiate various images. Being able to “see” what the computer may see and correcting it to ensure it’s more like human vision takes a special skillset. This can include: Computer Vision librariesDatabase managementComponent or object-oriented softwareAnalytical, logical, and critical thinkingClear reasoning It’s these skillsets along with a background in mathematics and computer languages like C++ which pave the Computer Vision engineer career path.  The Future of Computer Vision  The days of the generalist are long behind us. Now, more than ever, technologies like machine vision require a dedicated focus. With every field from healthcare to law enforcement to manufacturing utilizing these technologies, the future of Computer Vision performs a broader range of functions.   In Ann Arbor, at the University of Michigan and in partnership with Ford Motor Company, advancements race through every field not the least of which is manufacturing. As they transition toward full automation using the Internet of Things and more autonomous processes, it’s even more important to ensure Computer Vision models understand what they’re “seeing.” Computer Vision engineers will help to advance technologies which make machines easier to train and more easily figure out images better than they do now. Used in conjunction with other technologies like neural networks and other subsets of AI, machines will be able to see and interpret in the same way humans see and interpret.  And as far as we’ve come, there remains more applications and benefits not yet explored. The possibilities are endless. Current and future advancements will pave the way for AI to be as human as we are bringing our once science fiction ideas to life.  One Final Thought… Though Computer Vision engineering can be drilled down to even more focused professions, the term itself is broad. But the specializations are basic with a demand for not only highly skilled professionals with the right educational background, but also hands-on experience. This detail is more important now than ever before, especially for Computer Vision teams seeking leadership roles who can take their applications to the next level and on a global scale.  Some of the basic specializaitons include, but are not limited to: Camera imaging geometryFeature detection and matchingImage classification and scene analysis In the wake of work-from-home policies, remote working, and the shifting landscape of working outside the office, technology careers are prime opportunities to both gain increased knowledge in your chosen field or begin your career path.  If you’re ready to take the next step in your career, we may be able to help. Take a look at our current vacancies or get in touch with one of our expert consultants to learn more.   For our West Coast Team, contact us at (415) 614 - 4999 or send an email to sanfraninfo@harnham.com.   For our Mid-West and East Coast teams contact us at (212) 796-6070 or send an email to newyorkinfo@harnham.com.  

Why You Should Always Be Learning In Data Science: Tips From Kevin Tran

Last month we sat down with Kevin Tran, a Senior Data Scientist at Stanford University, to chat about Data Science trends, improvements in the industry, and his top tips for success in the market.  As one of LinkedIn’s Top Voices of 2019 within Data & Analytics. his thoughts on the industry regularly garner hundreds of responses, with debates and discussions bubbling up in the comments from colleagues eager to offer their input.  This online reputation has allowed him to make a name for himself, building out his own little corner of the internet with his expertise. But for Tran, it’s never been about popularity. “It’s not about the numbers,” he says without hesitation. “I don’t care about posting things just to see the number of likes go up.” His goal is always connection, to speak with others and learn from them while teaching from his own background. He’s got plenty of stories from his own experiences. For him, sharing is a powerful way to lead others down a path he himself is still discovering.  When asked about the most important lesson he’s learned in the industry, he says it all boils down to staying open to new ideas.  “You have to continue to learn, and you have to learn how to learn. If you stop learning, you’ll become obsolete pretty soon, particularly in Data Science. These technologies are evolving every day. Syntax changes, model frameworks change, and you have to constantly keep yourself updated.”  He believes that one of the best ways to do that is through open discussion. His process is to share in order to help others. When he has a realisation, he wants to set it in front of others to pass along what he’s learned; he wants to see how others react to the same problem, if they agree or see a different angle. It’s vital to consider what you needed to know at that stage. Additionally, this exchange of ideas allows Tran to learn from how others tackle the same problems, as well as get a glimpse into other challenges he may have not yet encountered.  “When I mentor people, I’m still learning, myself,” Tran confesses. “There’s so much out there to learn, you can’t know it all. Data Science is so broad." At the end of the day, it all comes down to helping each other and bringing humanity back to the forefront. In fact, this was his biggest advice for both how to improve the industry and how to succeed in it. It’s a point he comes back to with some regularity in his writing. “It doesn’t matter how smart you are, stay humble and respect everyone,” one post reads. “Everyone can teach you something you don’t know.” Treating people well, understanding their needs, and consciously working to see them as people instead of numbers or titles—this, Tran argues, is how you succeed in the business. To learn and grow, you must work with people, especially people with different skills and mindsets. Navigating your career is not all technical, even in the world of Data. “The thing that cannot be automated is having a heart,” he tells me sagely. Beyond this, Tran stresses the need for a solid foundation. The one thing you can’t afford to do is take shortcuts. You have to learn the practicalities and how to apply them, but to be strong in theory as well.  Understanding what is happening underneath the code will keep you moving forward. He compares knowing the tools to learning math with a calculator. “If you take the calculator away, you still need to be able to do the work. You need the underlying skills too, so that when you’re in a situation without the calculator, you can still provide solutions.” By constantly striving to collaborate and improve, Tran believes the Data industry has the best chance of innovating successfully.  If you’re looking for a new challenge in an innovative and collaborative environment, we may have a role for you. Take a look at our latest opportunities or get in touch with one of our expert consultants to find out more. 

Recently Viewed jobs