Senior Data Engineer
City of London, London / £75000 - £85000
£75000 - £85000
City of London, London
SENIOR DATA ENGINEER
CENTRAL LONDON, CURRENTLY FLEXIBLE / REMOTE / HOME WORKING
UP TO £80,000 + 10% BONUS+ 7% NON-CONTRIBUTORY PENSION
Harnham is partnered with an energy trading regulating company, working with companies supplying renewable energy. The Data Engineer will sit in the Data Science team, migrating their services to Snowflake and streaming real-time data through Kafka.
This company has created algorithms that gauge the demand for renewable energy, helping set the price for the commodity. They work with renewable suppliers, providing them with statistics and looking into variables like weather to help predict consumption.
The company has data scattered around their AWS platform but is looking to migrate to Snowflake and introduce Kafka.
- You will be migrating from Redshift/Postgres databases to Snowflake
- You will own the Kafka streams to ensure real-time data is being provided to the Data Analysts/Scientists
- You will be working alongside Snowflake consultants to set up the infrastructure using Terraform
YOUR SKILLS & EXPERIENCES:
- An understanding of Snowflake and how to implement the warehouse
- Worked with streaming techs (Kafka/Kinesis)
- Expert Python programming
- Nice to haves: Kubernetes/Docker for deployment
- 30min chat with the Hiring Manager
- 60min technical discussion with the Hiring Manager and a team member
- 30min values and culture chat with another senior stakeholder and someone from HR
HOW TO APPLY:
Please register your interest by sending your CV to Henry Rodrigues via the apply link on this page. (The Company has outlined a fully remote interview process and has a remote onboarding policy in place).
Data Engineer Or Software Engineer: What Does Your Business Need? | Harnham US Recruitment post
We are in a time in which what we do with Data matters. Over the last few years, we have seen a rapid rise in the number of Data Scientists and Machine Learning Engineers as businesses look to find deeper insights and improve their strategies. But, without proper access to the right Data that has been processed and massaged, Data Scientists and Machine Learning Engineers would be unable to do their job properly. So who are the people who work in the background and are responsible to make sure all of this works? The quick answer is Data Engineers!… or is it? In reality, there are two similar, yet different profiles who can help help a company achieve their Data-driven goals. Data Engineers When people think of Data Engineers, they think of people who make Data more accessible to others within an organization. Their responsibility is to make sure the end user of the Data, whether it be an Analyst, Data Scientist, or an executive, can get accurate Data from which the business can make insightful decisions. They are experts when it comes to data modeling, often working with SQL. Frequently, “modern” Data Engineers work with a number of tools including Spark, Kafka, and AWS (or any cloud provider), whilst some newer Databases/Data Warehouses include Mongo DB and Snowflake. Companies are choosing to leverage these technologies and update their stack because it allows Data teams to move at a much faster pace and be able to deliver results to their stakeholders. An enterprise looking for a Data Engineer will need someone to focus more on their Data Warehouse and utilize their strong knowledge of querying information, whilst constantly working to ingest/process Data. Data Engineers also focus more on Data Flow and knowing how each Data sets works in collaboration with one another. Software Engineers – DataSimilar to a Data Engineers, Software Engineers – Data ( who I will refer to as Software Data Engineers in this article) also build out Data Pipelines. These individuals might go by different names like Platform or Infrastructure Engineer. They have to be good with SQL and Data Modeling, working with similar technologies such as Spark, AWS, and Hadoop. What separates Software Data Engineers from Data Engineers is the necessity to look at things from a macro-level. They are responsible for building out the cluster manager and scheduler, the distributed cluster system, and implementing code to make things function faster and more efficiently. Software Data Engineers are also better programers. Frequently, they will work in Python, Java, Scala, and more recently, Golang. They also work with DevOps tools such as Docker, Kubernetes, or some sort of CI/CD tool like Jenkins. These skills are critical as Software Data Engineers are constantly testing and deploying new services to make systems more efficient. This is important to understand, especially when incorporating Data Science and Machine Learning teams. If Data Scientists or Machine Learning Engineers do not have a strong Software Engineers in place to build their platforms, the models they build won’t be fully maximized. They also have to be able to scale out systems as their platform grows in order to handle more Data, while finding ways to make improvements. Software Data Engineers will also be looking to work with Data Scientists and Machine Learning Engineers in order to understand the prerequisites of what is needed to support a Machine Learning model. Which is right for your business? If you are looking for someone who can focus extensively on pulling Data from a Data source or API, before transforming or “massaging” the Data, and then moving it elsewhere, then you are looking for a Data Engineer. Quality Data Engineers will be really good at querying Data and Data Modeling and will also be good at working with Data Warehouses and using visualization tools like Tableau or Looker. If you need someone who can wear multiple hats and build highly scalable and distributed systems, you are looking for a Software Data Engineer. It’s more common to see this role in smaller companies and teams, since Hiring Managers often need someone who can do multiple tasks due to budget constraints and the need for a leaner team. They will also be better coders and have some experience working with DevOps tools. Although they might be able to do more than a Data Engineer, Software Data Engineers may not be as strong when it comes to the nitty gritty parts of Data Engineering, in particular querying Data and working within a Data Warehouse. It is always a challenge knowing which type of job to recruit for. It is not uncommon to see job posts where companies advertise that they are looking for a Data Engineer, but in reality are looking for a Software Data Engineer or Machine Learning Platform Engineer. In order to bring the right candidates to your door, it is crucial to have an understanding of what responsibilities you are looking to be fulfilled.That’s not to say a Data Engineer can’t work with Docker or Kubernetes. Engineers are working in a time where they need to become proficient with multiple tools and be constantly honing their skills to keep up with the competition. However, it is this demand to keep up with the latest tech trends and choices that makes finding the right candidate difficult. Hiring Managers need to identify which skills are essential for the role from the start, and which can be easily picked up on the job. Hiring teams should focus on an individual’s past experience and the projects they have worked on, rather than looking at their previous job titles. If you’re looking to hire a Data Engineer or a Software Data Engineer, or to find a new role in this area, we may be able to help. Take a look at our latest opportunities or get in touch if you have any questions.
The Six Steps Of Data Governance | Harnham Recruitment post
The value that data analysis can provide to organisations is becoming increasingly clear. But with all the buzz around the endless ways that data can be used to revolutionise your business processes, it can be overwhelming to know where to start. Fundamentally, what you can do with your data and how useful it may be will hinge on its quality. This is the case no matter what data you may have, whether that be customer demographics or manufacturing inventories. High-quality data is also imperative for utilising exciting and innovative new technology such as Machine Learning and AI. It’s all very well investing in tech to harness your data assets to, for example, better inform decision making, but you won’t be able to glean any useful analysis if the data is full of gaps and inconsistencies. Many will be looking at this new tech and be tempted to run before they can walk. But building quality data sets and water-tight, long-lasting processes will form the foundation for any future developments and should not be overlooked. This is where Data Governance comes into its own.Data Governance (DG) is an effective step in improving your data and turning it into an invaluable asset. It has numerous definitions but according to Data Governance Institute (DGI), “Data Governance is the exercise of decision-making and authority for data-related matters.“Essentially DG is the process of managing data during its life cycle. It ensures the availability, useability, integrity and security of your data, based on internal data standards and policies that control data usage. Good data governance is critical to success and is becoming increasingly more so as organisations face new data privacy regulations and rely on data analytics to help optimise operations and drive business decision-making. As Ted Friedman from Gartner said: ‘Data is useful. High-quality, well-understood, auditable data is priceless.’Without DG, data inconsistencies in different systems across an organisation might not get resolved. This could complicate data integration efforts and create data integrity issues that affect the accuracy of business intelligence (BI) reporting and analytics applications.Data Governance programs can differ significantly, depending on their focus but they tend to follow a similar framework:Step 1: Define goals and understand the benefits The first step of developing a strategy should be to ensure that you have a comprehensive understanding of the process and what you would like the outcome to be.A strong Data Governance strategy relies on ‘buy in’ from everyone in the business. By stressing the importance of complying with the guidelines which you will later set, you will be helping to encourage broad participation and ensure that there is a concerted and collaborated effort to maintain high standards of data quality. Leaders must be able to comprehend the benefits themselves before communicating them to their team so it may be worth investing in training around the topic.Step 2: Analyse and assess the current dataThe next step is essentially sizing up the job at hand, to see where improvements might need to be made. Data should be assessed against multiple dimensions, such as the accuracy of key attributes, the completeness of all required attributes and timeliness of data. It may also be valuable to spend time analysing the root causes of inferior data quality.Sources of poor data quality can be broadly categorised into data entry, data processing, data integration, data conversion, and stale data (over time) but there may be other elements at play to be aware of.Step 3: Set out a roadmapYour data governance strategy will need a structure in which to function, which will also be key to measuring the progress and success of the program. Set clear, measurable, and specific goals – as the saying goes – you cannot control what you cannot measure. Plans should include timeframes, resources and any costs involved, as well as identifying the owners or custodians of data assets, the governance team, steering committee, and data stewards who will all be responsible for different elements. Including business leaders or owners in this step will ensure that programs remain business-centric.Step 4: Develop and plan the data governance programBuilding around the timeline outlined you can then drill down to the nitty-gritty. DG programs vary but usually include:Data mapping and classification – sorting data into systems and classifying them based on criteria.Business glossary – establishing a common set of definitions of business terms and concepts – helping to build a common vocabulary to ensure consistency.Data catalogue – collecting metadata and using it to create an indexed inventory of available data assets.Standardisation – developing polices, data standards and rules for data use to regulate proceduresStep 5: Implement the data governance programCommunicating the plan to your team may not be a one-step process and may require a long-term training schedule and regular check-ins. The important thing to realise is that DG is not a quick fix, it will take time to be implemented and fully embraced. It also may need tweaks as it goes along and as business objectives change. All DG strategies should start small and slowly build up over time – Rome wasn’t built in a day after all. Step 6: Close the loopArguably the most important part of the process is being able to track your progress and checking in at periodic intervals to ensure that the data is consistent with the business goals and meets the data rules specified. Communicating the status to all stakeholders regularly will also help to ensure that a data quality discipline is maintained throughout.Looking for your next big role in Data & Analytics or need to source exceptional talent? Take a look at our latest Data Governance jobs or get in touch with one of our expert consultants to find out more.
Keepers of the Data Kingdom: the Analytics Engineer | Harnham US Recruitment post
If it seems the Data world is drilling down further into niche specialities, you’re right. Considering the swathes of information sent and received on a day-by-day, minute-by-minute, and second-by-second basis, is it any wonder? The sheer volume, depending on your business and what you want to know, requires not just a Data team, but must now include someone with a particular skillset, including the tech-savvy analyst who can speak to the executive team.So, who holds it all together? These swathes of information. Who organizes the information in a cohesive order, so anyone with a map, can make their own analyses? Enter the Analytics Engineer.What Makes an Analytics Engineer an Analytics Engineer?Though it’s a rather new speciality within the Data Scientist scope—think Machine Learning Engineer, Software Engineer, Business Analyst, etc—at its core, the definition of an Analytics Engineer is this: “The analytics engineer sits at the intersection of the skill sets of data scientists, analysts, and data engineers. They bring a formal and rigorous software engineering practice to the efforts of analysts and data scientists, and they bring an analytical and business-outcomes mindset to the efforts of data engineering.” Michael Kaminsky, consultant, and former Director of Analytics at Harry’s.In other words, analytics engineers, using best software engineering practices transform data through testing and documentation so that data analysts begin with cleaner data to analyze. As technically savvy as the engineer must be, they must also be able to explain to stakeholders what they’re looking at so they can formulate their own insights. Five Roles and Responsibilities of the Analytics EngineerLike all new niche specialities, there are core responsibilities to consider as well as that of skillsets required to either study to become an Analytics Engineer or to discover if you’re one already. How? Consider the questions you ask, your studies within Data Science, Computer Science, Statistics, and Math, and your balance between technical skills and soft skills. Below are five things to consider when thinking about this role:Programming language experience. Experience with programming languages like R and Python along with strong SQL skills which are at the core of this role. DBT technology knowledge. As the driving force behind the rise of Analytics Engineer as a separate role, it’s imperative anyone interested in pursuing it should have a firm grasp of DBT — the Data Build Tool — that allows the implementation of analytics code using SQL. Data tracking expertise using Git. Data modelling. Clean, tested, and raw data which allow executives and analysts to view their Data, understand it within the database or its warehouse. Data transformation. Analytics Engineers determine what Data is most useful and transform it to ensure it fits related tasks. It’s part of building the foundational layer so businesses can answer their own questions. Key Changes Leading to the Shift in Data RolesWith every technological advancement their comes new players to the field. The difference is here is that the job description already existed. We were only missing a title. But from the traditional Data team to the modern Data team, there are a few key changes that point directly to the rise of this niche field. Cloud warehouses (like Snowflake, Redshift, BigQuery) and the arrival of the DBT the foundational layer which can be built on top of modern data warehouses are the first two that come to mind. Then, the Software-as-a-Service (SaaS) tools like Stitch and Hevo are capable of integrating Data from a variety of sources, and the introduction of tools like Mode and Looker allows anyone interested in drawing insight from Data to do so on their own.Who Needs an Analytics Engineer? Small or Large Businesses?The short answer is it depends. But the general rule follows that while both large and small companies can benefit from having this professional on their staff, there are different things to consider. For example, a small business may be able to find what they need in a single individual. The Analytics Engineer is something of a jack-of-all-trades. Larger businesses, on the other hand, may already have a Data team in place. In this case, an Analytics Engineer adds to your team, something like an additional set of eyes increasing insight drawn from those large swathes of Data we spoke about earlier.So, what’s next for the role of Analytics Engineer? Who knows? The roles of any Data industry professional is constantly evolving. If you’re interested in Analytics Engineering, Machine Learning, Data Science, or Business Intelligence just to name a few, Harnham may have a role for you. Check out our latest Data & Analytics Engineering jobs or contact one of our expert consultants to learn more. For our West Coast Team, contact us at (415) 614 – 4999 or send an email to email@example.com. For our Arizona Team, contact us at (602) 562 7011 or send an email to firstname.lastname@example.org. For our Mid-West and East Coast teams contact us at (212) 796-6070 or send an email to email@example.com.
CAN’T FIND THE RIGHT OPPORTUNITY?
If you can’t see what you’re looking for right now, send us your CV anyway – we’re always getting fresh new roles through the door.