The big data and data engineering market is growing at a rapid pace. Valued at 44 billion in 2021, the industry’s market size is expected to reach 120 billion USD by 2027 (roughly £88 billion). Are you interested in breaking into this fast-growing space, but not sure how to get started?
One of our consultants, Lottie Musgrove, sat down with Chris Hopkinson to learn about his journey into data engineering, and how he managed to break into this extremely competitive field.
Hopkinson has been working in data and analytics for nearly a decade. Currently working as a Lead Data Architect/Consultant, Hopkinson is starting a new role as Head of Data Engineering and BI at Evri soon. Here’s a summary of Musgrove and Hopkinson’s discussion.
Question #1: What motivated you to pursue a career in data engineering?
I used to be an accountant, but I didn’t really enjoy the role. So, I started thinking about what I actually enjoyed, and what I wanted to get out of work. And I realised that I was really interested in how we reported information as accountants, how that information flowed around the businesses, and then how we could improve those information systems.
I’d already started to teach myself SQL and was already starting to automate things in my accounting role to help optimise our processes. So, I realised that there was something there, that this was an area that I was really interested in, and also something that could have a real
business impact.
So, I went ahead and did a master’s in what was then called Business Intelligence and Data Mining, and that’s really what brought me into the world of data.
Question #2: What were some of the challenges that came up for you during your transition from accounting to data?
For me, it was challenging to get the right type of experience in the early days of my data career, and to find a company that was willing to take a chance on me, to allow me to get the hands-on experience I needed to “break in”. Eventually, I was fortunate enough to have someone take me under their wing, and allow me to join their boutique consultancy firm, which gave me an opportunity to work on data projects
with big companies and ultimately get my foot in the door.
Question #3: What have been some of the major milestones or turning points in your career?
My first proper role in analytics was a big deal, but I think the advent of massively online courses really opened the flood gates in terms of learning. I remember going online to learn about R, machine learning, and just finding the whole world of online resources and courses quite fascinating. The online resources I found gave me the opportunity to learn about things that I would have absolutely no chance of learning otherwise. Obviously now, there’s a plethora of material available online, but I feel fortunate that I found those online resources when I did, and that I was able to build my knowledge base around that.
Another real milestone for was getting access to tools like Hadoop and the world of big data. Before then, everything was either in a server or a database, whereas Hadoop and big data created an opportunity to start to do things at a much larger scale. This allowed us to look at and analyse things like network data. It was a learning curve but it really enlightening, it really opened up a world of possibilities. And finally, AWS and the cloud was a big milestone that stood out to me.
The cloud allowed me to start thinking in more of a system/architecture basis, which really changed my idea of how we can build systems, and what we can do to give our organisation the best chance of garnering valuable insights from its data.
Question #4: What do you do to keep continuing your own development?
I think the challenge now is finding the signal from the noise. There are so many people talking about data, writing blogs, and so much hype around AI, that finding the actual useful information can be difficult.
Because anybody can use a generative model to summarise something that someone else has written, I think it’s important to look at the credentials of the people that you’re following and make sure they’ve got the kind of experience that means they can talk into the area of experience that they profess to have.
I look for people that are working for companies that are building data products, or people that are leading AI consultancies – people that can share ideas and trends they’re sending first-hand as industry experts.
Question #5: Since you started your career, what have been some of the biggest changes in the industry that you’ve noticed?
It’s moving away from a small and narrow stack that most companies use, to a plethora of different tools that are available through the cloud.
Before, you could learn a tool like the SQL server stack, and pretty much be set for life. But now, as more companies adopt the cloud, the number of different services available has skyrocketed, which has changed what you have to do (and what you have to know) as a data
engineer.
Now, you need to start thinking in terms of systems more and more. There are also the changes that have come about through the adoption of software engineering principles, and the creation of DevOps and CI/CD. I think that data engineering as a field is still
on a path of discovery. We’re still trying to figure out how all these different specialisms
intersect and work together.
Question #6: How do you stay passionate about a field that’s constantly changing?
I really enjoy learning new things, so learning a new programming language is fascinating to me. What I see is that people who are willing to invest in continual learning really stand out in what’s becoming a really complicated landscape. For me, I really need to see my work have an impact on the business. And one of the nice things about engineering is that it’s a practical application of technical knowledge to business
problems.
So, making sure my work has some impact on the business keeps me focused and grounded on the things that matter. Working with other people that are passionate really helps too. You can get inspired by people that are excited about something, and by people who are good at what they do, and you can play off each other.
Question #7: How do you see the role of data engineer evolving?
There could be more separation into specialisms because there comes a point where there’s too much knowledge to expect from a single role.
We’re already seeing this already with the separation of analytics engineering and data engineering, so we may see more things that emerge like that.
Question #8: Are there any trends that you think data engineers should be focusing on right now?
Different paradigms of storage (S3 vs. database vs. document store) and processing (via Spark, a single machine, or a streaming platform). Getting a good grounding of all those things will help you make good decisions on what you should be using for a particular business problem that you’re trying to solve.
Also, while you don’t need to become a data scientist, gaining an understanding the needs of that domain will help you understand what you need to know about in order to add value to these different things that are happening in the business.
Question #9: What kind of advice do you have for somebody trying to break into the field?
Firstly – go for it. Secondly, get your hands on something. If you’re struggling to get access to a data engineering role specifically, maybe look at the adjacent roles like analyst roles. Anything that involves writing SQL, touching data – whether that’s building reports or spreadsheets – just
something that gets you in that space.
If your role doesn’t have it already try to bake it into your role and show your manager what you’re doing. That might not work for everyone but that’s certainly the path I took. But definitely, get hands-on experience and start to think about how you can do this stuff in your current role, which will help in the future when you start to apply for new roles.
Are you a data professional that’s looking to make a career change, or break into the industry? Get in touch today.