DATA ENGINEER
Düsseldorf, Nordrhein-Westfalen / €60000 - €65000
INFO
€60000 - €65000
LOCATION
Düsseldorf, Nordrhein-Westfalen
Permanent
DATA ENGINEER
LOCATION: Düsseldorf, Germany
SALARY: 60,000 to 65,000 Euro per Annum
We are seeking a Data Engineer in Düsseldorf, Germany. As a Data Engineer, you will be responsible for designing, developing, and maintaining scalable data pipelines and data solutions using Databricks, Spark, and Azure cloud services.
Responsibilities:
- Design, develop, and maintain data pipelines and data solutions using Databricks, Spark, and Azure cloud services.
- Collaborate with cross-functional teams including Data Scientists, Data Analysts, and Business Analysts to understand data requirements and ensure data integrity and quality.
- Optimize and performance-tuned data pipelines and data solutions to ensure optimal performance and scalability.
- Implement data security and data governance best practices to ensure data compliance and privacy.
- Monitor and troubleshoot data pipelines and data solutions to identify and resolve data-related issues.
- Collaborate with DevOps teams to ensure smooth deployment and operation of data solutions in a production environment.
- Stay up-to-date with the latest advancements in data engineering technologies and best practices to drive innovation and continuous improvement.
Requirements:
- Bachelor's degree in Computer Science, Information Technology, or a related field.
- Hands-on experience in designing, developing, and maintaining data pipelines and data solutions using Databricks, Spark, and Azure cloud services.
- Strong programming skills in Scala or Python.
- Experience with Azure cloud services such as Azure Data Factory, Azure Databricks, and Azure Blob Storage.
- Experience with data modeling, data integration, and ETL processes.
- Ability to work in a fast-paced, collaborative environment.
- Strong communication skills in English and German.
If you are a skilled Data Engineer with expertise in Databricks, Spark, and Azure, and are looking for an exciting opportunity to work with cutting-edge technologies in a dynamic environment, then we want to hear from you! Apply now with your updated CV and cover letter, including your salary expectation, to join our team in Düsseldorf.
CONTACT
Adanya Hott
Recruitment Consultant
SIMILAR
JOB RESULTS
Data Engineer Or Software Engineer: What Does Your Business Need? | Harnham US Recruitment post
+
We are in a time in which what we do with Data matters. Over the last few years, we have seen a rapid rise in the number of Data Scientists and Machine Learning Engineers as businesses look to find deeper insights and improve their strategies. But, without proper access to the right Data that has been processed and massaged, Data Scientists and Machine Learning Engineers would be unable to do their job properly. So who are the people who work in the background and are responsible to make sure all of this works? The quick answer is Data Engineers!… or is it? In reality, there are two similar, yet different profiles who can help help a company achieve their Data-driven goals. Data Engineers When people think of Data Engineers, they think of people who make Data more accessible to others within an organization. Their responsibility is to make sure the end user of the Data, whether it be an Analyst, Data Scientist, or an executive, can get accurate Data from which the business can make insightful decisions. They are experts when it comes to data modeling, often working with SQL. Frequently, “modern” Data Engineers work with a number of tools including Spark, Kafka, and AWS (or any cloud provider), whilst some newer Databases/Data Warehouses include Mongo DB and Snowflake. Companies are choosing to leverage these technologies and update their stack because it allows Data teams to move at a much faster pace and be able to deliver results to their stakeholders. An enterprise looking for a Data Engineer will need someone to focus more on their Data Warehouse and utilize their strong knowledge of querying information, whilst constantly working to ingest/process Data. Data Engineers also focus more on Data Flow and knowing how each Data sets works in collaboration with one another. Software Engineers – DataSimilar to a Data Engineers, Software Engineers – Data ( who I will refer to as Software Data Engineers in this article) also build out Data Pipelines. These individuals might go by different names like Platform or Infrastructure Engineer. They have to be good with SQL and Data Modeling, working with similar technologies such as Spark, AWS, and Hadoop. What separates Software Data Engineers from Data Engineers is the necessity to look at things from a macro-level. They are responsible for building out the cluster manager and scheduler, the distributed cluster system, and implementing code to make things function faster and more efficiently. Software Data Engineers are also better programers. Frequently, they will work in Python, Java, Scala, and more recently, Golang. They also work with DevOps tools such as Docker, Kubernetes, or some sort of CI/CD tool like Jenkins. These skills are critical as Software Data Engineers are constantly testing and deploying new services to make systems more efficient. This is important to understand, especially when incorporating Data Science and Machine Learning teams. If Data Scientists or Machine Learning Engineers do not have a strong Software Engineers in place to build their platforms, the models they build won’t be fully maximized. They also have to be able to scale out systems as their platform grows in order to handle more Data, while finding ways to make improvements. Software Data Engineers will also be looking to work with Data Scientists and Machine Learning Engineers in order to understand the prerequisites of what is needed to support a Machine Learning model. Which is right for your business? If you are looking for someone who can focus extensively on pulling Data from a Data source or API, before transforming or “massaging” the Data, and then moving it elsewhere, then you are looking for a Data Engineer. Quality Data Engineers will be really good at querying Data and Data Modeling and will also be good at working with Data Warehouses and using visualization tools like Tableau or Looker. If you need someone who can wear multiple hats and build highly scalable and distributed systems, you are looking for a Software Data Engineer. It’s more common to see this role in smaller companies and teams, since Hiring Managers often need someone who can do multiple tasks due to budget constraints and the need for a leaner team. They will also be better coders and have some experience working with DevOps tools. Although they might be able to do more than a Data Engineer, Software Data Engineers may not be as strong when it comes to the nitty gritty parts of Data Engineering, in particular querying Data and working within a Data Warehouse. It is always a challenge knowing which type of job to recruit for. It is not uncommon to see job posts where companies advertise that they are looking for a Data Engineer, but in reality are looking for a Software Data Engineer or Machine Learning Platform Engineer. In order to bring the right candidates to your door, it is crucial to have an understanding of what responsibilities you are looking to be fulfilled.That’s not to say a Data Engineer can’t work with Docker or Kubernetes. Engineers are working in a time where they need to become proficient with multiple tools and be constantly honing their skills to keep up with the competition. However, it is this demand to keep up with the latest tech trends and choices that makes finding the right candidate difficult. Hiring Managers need to identify which skills are essential for the role from the start, and which can be easily picked up on the job. Hiring teams should focus on an individual’s past experience and the projects they have worked on, rather than looking at their previous job titles. If you’re looking to hire a Data Engineer or a Software Data Engineer, or to find a new role in this area, we may be able to help. Take a look at our latest opportunities or get in touch if you have any questions.
Where Does the Data Engineer Sit in the Data Chain? | Harnham US Recruitment post
+
Data Scientist. Data Architect. Data Engineer. With so many professional titles in the Data and Technology space, it can be difficult to distinguish one from another. You may have an interest in Data, but aren’t sure which field you’d like to move into, and as things become more specialized, it adds another layer of education and experience required to make the move.Every one of the titles above has a place and a responsibility along the Data chain. But some may be more well-known than others. In order to wrangle Data, clean and analyze it, or develop programming from it, you need someone to build and maintain the pipeline systems that give Data Scientists a map to follow when collecting, cleaning, and analyzing the data.Though not interchangeable, the Data Scientist and the Data Engineer work together as two halves of a whole on the Data team. One role crafts the roadmap or blueprint for others to follow while the other gathers insights from the data based on specific datasets requested and designed by the Data Engineer.So, let’s look first at what a Data Engineer does and the skillsets required for the role.WHAT IS A DATA ENGINEER?A Data Engineer takes the blueprints from the Data Architect and creates the pipelines. It sounds simple. But it isn’t. A Data pipeline is just like it sounds. It is the process Data goes through from inception to implementation, and the technologies and frameworks involved at an in-depth level which can involve up to 30 different technologies. So, a Data Engineer is responsible for developing, testing, and maintaining the data pipeline. That’s a lot of wrangling, cleaning, and prepping to ensure reliable information is filtered to the Data Scientist. 3 TYPES OF DATA ENGINEER ROLES1. Generalist – This role is often found on small teams, and though this role may understand processes, but not necessarily systems, it’s a good place to begin if you’re a Data Scientist interested in stepping into a Data Engineer role. The focus here is end-to-end collection to processing of Data.2. Pipeline – You’ll find this role conquering more complicated projects on midsize Analytics team. The Pipeline focused Data Engineer is found in medium to larger-size businesses.3. Database – The Database focused engineer is found most often in larger businesses with distributed systems across several databases. These individuals are responsible for implementing what the Data Architect has created, and collecting the information to inform analytics databases.7 SKILLS REQUIRED FOR DATA ENGINEERData Engineers are the ones who keep everything running smoothly. Even if a technology doesn’t necessarily fall within their scope of responsibilities, they should still understand it, and be able to prepare Data for it. This is particularly the case when it comes to Machine Learning. Though it’s more aligned with Data Scientist, a Data Engineer should know enough about it craft algorithms and gather insights.Below are a few more technical skills a Data Engineer should have to be successful in their role.1. Know and understand the right tools for the job2. Technical Skills include:3. Linux4. SQL5. Python6. Kafka, Flink, and Kudu languages for processing frameworks and storage engines, and which tool is best for which task.7. General understanding of distributed systems and how they’re different from traditional systems.The role of the Data Engineer is unique in that how this person thinks depends on what needs to be done. In some cases, you’ll need to think like an engineer, and in other cases, you’ll need to think like a product manager. This is one of the reasons it’s important to have such deep knowledge of systems, processes, and knowing the right tool, and the right person for the job.If you’re looking for your next role in Big Data, Analytics, Software Engineering, or Computer Vision, Harnham may have a role for you. Check out our current vacancies or contact one of our expert consultants to learn more.For our West Coast Team, contact us at (415) 614 – 4999 or email sanfraninfo@harnham.com.For our Mid-West and East Coast teams, contact us at (212) 796-6070 or email newyorkinfo@harnham.com.
The Evolution Of The Data Engineer | Harnham Recruitment post
+
Every Data Science department worth its salt has at least one engineer on the team. Considered the “master builders,” Data Engineers design, implement and manage Data infrastructure. They lay down digital foundations and monitor performance.At least, that’s what they used to do. Over the last few years, the role has shifted. Data Engineers have gone from mainly designing and building infrastructure, to a much more supportive and collaborative function. Today, a key part of the engineer role is to help their Data Analyst and Data Scientist colleagues process and analyse data. In doing so, they are contributing to improved team productivity and, ultimately, the company’s bottom line.
THE IMPACT OF THE CLOUDIn the past, a Data Engineer would often move data to and from databases. They’d load it in a Data Warehouse, and create Data structures. Engineers would also be on hand to optimise Data while businesses upgraded or installed new servers. And then along came the Cloud. The rapid dominance of cloud computing meant that optimisation was no longer needed. And as the cloud made it easy for companies to scale up and down, there was less need for someone to manage the data infrastructure. The collective adoption of the cloud has had a big impact on the function of Data Engineers. Because, provided a company has the funds, there is no longer the same demand for physical storage.Freed from endless scaling requests, engineers have more time to program and develop. They also spend more time curating data for better analytics.
AUTOMATING THE BORING BITS Less than a decade ago, if start-ups wanted to run a sophisticated analytics program, they’d automatically hire a couple of Data Engineers. Without them, Data Analysts and Data Scientists wouldn’t have any Data. The engineers would extract it from operational systems, before giving analysts and business users access. They might also do some work to make the Data simpler to interpret. In 2019, none of this extraction and transformation work is necessary. Companies can now buy off-the-shelf technology that does exactly what a Data Engineer used to do. As Tristan Handy, Founder and President of Fishtown Analytics, puts it: “Software is increasingly automating the boring parts of Data Engineering.”
STILL SOUGHT-AFTER With automation hot on the Data Engineer’s tail, it can be tempting to ask whether they are still needed at all. The answer is: yes, absolutely.When recruiting engineers, Data Strategist Michael Kaminsky says he looks for people “who are excited to partner with analysts and Data Scientists.” He wants a Data Engineer who knows when to pipe up with, “What you’re doing seems really inefficient, and I want to build something better.”Despite the rise in off-the-shelf solutions, engineers still play a key role in the Data Science team. The difference is simply that their priorities and tasks have shifted. Today, innovation is the watchword. The best engineers are hugely collaborative, helping their teams go further, faster.It’s an exciting time to be a Data Engineer. If you’re interested in this field, we may have a job for you. Take a look at our latest opportunities or get in touch with our expert consultants.
CAN’T FIND THE RIGHT OPPORTUNITY?
STILL LOOKING?
If you can’t see what you’re looking for right now, send us your CV anyway – we’re always getting fresh new roles through the door.