Senio Big Data Architect Lead

San Francisco, California
US$150000 - US$180000 per year + equity

This vacancy has now expired. Please see similar roles below...

Senior Big Data Lead Architect

San Francisco, Bay Area

$1500,000-$180,000 + Equity

Harnham has partnered with a selfless Fintech company that is looking to change the lives of billions of people around the world by allowing financial access to individuals would not have access otherwise. This company partners with telecommunication companies to utilize machine learning/deep learning to create an identity-based relationship with anonymous prepaid users, opening up access to otherwise unattainable mobile financial services.

ROLE OVERVIEW- Senior Lead Architect

As a Senior Lead Architect, you will report to the VP Engineering. You will be hands-on end to end leading the design and development of data solutions providing and establishing efficient, scalable financial platforms that will serve billions of people that have insufficient access to banks.

The Senior Lead Architect position will help establish a bulletproof architecture that will be able to scale as the company continues to skyrocket. You will build and scale internal analytics, reporting, and dynamic decisioning platforms; while helping spearhead technical direction, driving the necessary changes and making appropriate technology choices working collaboratively with the DevOps, Product and Engineering Teams in an AWS environment

REQUIRED SKILLS & EXPERIENCE

Ideal candidates will possess:

  • 4+ years commercial experience in AWS (S3, DynamoDB, EMR, Redshfit, EC2)
  • 2+ years of real-time streaming exposure (Kafka, Spark Streaming, Kinesis)
  • 3+ Spark (Scala/Python)
  • 4+ years working cross-functionally (w/DevOps. Product, Data Science, and Infrastructure teams)
  • 4+ years of experience with ETL workflow (Airflow, Pinball, Luigi)
  • Experience/Examples of creating highly scalable infrastructure
  • Experience with Docker, Kubernetes, Terraform.

BENEFITS

  • Medical, 401k, dental, etc





Send similar jobs by email
33445
San Francisco, California
US$150000 - US$180000 per year + equity

Similar Jobs

Salary

US$160000 - US$200000 per year

Location

New York

Description

Lead Data Manager - Political Organization

Salary

£60000 - £100000 per annum + bonus, pension, benefits

Location

City of London, London

Description

A great opportunity as an MDM Consultant for a leading consultancy, considered the market leaders within data management.

Salary

US$464 - US$504 per day

Location

Seattle, Washington

Description

Join a top-notch consultancy working with exciting clients as a multi-faceted data engineering.

Salary

€7200 - €8400 per annum

Location

Munich, Bayern

Description

Join a project in one of the world´s leading e-commerce platforms and support them with your expertise in Data Engineering.

Salary

500000kr - 600000kr per annum

Location

Stockholm

Description

Are you a motivated Senior Database Administrator, ready for the next challenge?

Harnham blog & news

With over 10 years experience working solely in the Data & Analytics sector our consultants are able to offer detailed insights into the industry.

Visit our Blogs & News portal or check out our recent posts below.

The Evolution Of The Data Engineer

Every Data Science department worth its salt has at least one engineer on the team. Considered the “master builders,” Data Engineers design, implement and manage Data infrastructure. They lay down digital foundations and monitor performance. At least, that’s what they used to do.  Over the last few years, the role has shifted. Data Engineers have gone from mainly designing and building infrastructure, to a much more supportive and collaborative function.  Today, a key part of the engineer role is to help their Data Analyst and Data Scientist colleagues process and analyse data. In doing so, they are contributing to improved team productivity and, ultimately, the company’s bottom line. THE IMPACT OF THE CLOUD In the past, a Data Engineer would often move data to and from databases. They’d load it in a Data Warehouse, and create Data structures. Engineers would also be on hand to optimise Data while businesses upgraded or installed new servers.  And then along came the Cloud.  The rapid dominance of cloud computing meant that optimisation was no longer needed. And as the cloud made it easy for companies to scale up and down, there was less need for someone to manage the data infrastructure.   The collective adoption of the cloud has had a big impact on the function of Data Engineers. Because, provided a company has the funds, there is no longer the same demand for physical storage. Freed from endless scaling requests, engineers have more time to program and develop. They also spend more time curating data for better analytics.  AUTOMATING THE BORING BITS  Less than a decade ago, if start-ups wanted to run a sophisticated analytics program, they’d automatically hire a couple of Data Engineers. Without them, Data Analysts and Data Scientists wouldn’t have any Data. The engineers would extract it from operational systems, before giving analysts and business users access. They might also do some work to make the Data simpler to interpret.  In 2019, none of this extraction and transformation work is necessary. Companies can now buy off-the-shelf technology that does exactly what a Data Engineer used to do. As Tristan Handy, Founder and President of Fishtown Analytics, puts it: “Software is increasingly automating the boring parts of Data Engineering.”  STILL SOUGHT-AFTER  With automation hot on the Data Engineer’s tail, it can be tempting to ask whether they are still needed at all.  The answer is: yes, absolutely. When recruiting engineers, Data Strategist Michael Kaminsky says he looks for people “who are excited to partner with analysts and Data Scientists.” He wants a Data Engineer who knows when to pipe up with, “What you’re doing seems really inefficient, and I want to build something better.” Despite the rise in off-the-shelf solutions, engineers still play a key role in the Data Science team. The difference is simply that their priorities and tasks have shifted.  Today, innovation is the watchword. The best engineers are hugely collaborative, helping their teams go further, faster. It’s an exciting time to be a Data Engineer. If you’re interested in this field, we may have a job for you. Take a look at our latest opportunities or get in touch with our expert consultants.  

A Q&A With Dyson’s Data Governance CDO

A Q&A With Dyson’s Data Governance CDO

Mridul Mathur is a skilled Senior Program Director with more than 15 years of experience working in businesses from Deutschebak to Dyson. He has a proven track record of successfully delivering large and complex cross-functional programs and building high performing teams from scratch. In last five years the main focus of his work has been in the area of Data Management to address the issues and challenges organisations have faced in the wake of various new regulations. Data Management and Data Governance are hot topics at the moment. Do you feel that attitudes have changed towards the fields since the beginning of your career? It’s been a very big shift. Going back to my involvement at Deutsche Bank around 2007, we were managing Data purely because we needed to create a Credit Risk position so that we could explain to the Bank of England and other regulators what we were doing. We didn’t really look beyond that.  But now, if you look at the industry, we want to use Data to not only calculate our Risk position but to derive value out of that Data.  It's something that can give a company a competitive advantage  one of those things that can significantly change a business. I personally feel that the turning point, not just for Deutsche Bank but for everybody was the market crash that happened in 2008.  A lot of the company did not have Data Management skills, or the ability to bring the Data together to understand exposures. Those who had exposure against Lehman, for example, could not recover any of the money they lost. That was the big turning point for all of them, when they actually lost hundreds of millions of dollars’ worth of revenue and loans overnight. They didn’t have the right Data, in the right place, and it cost them. What major issues do you see successful Data Governance facing over the next 12 months? I think we're still going through a phase of understanding and internalizing the issue. By that I mean that we understand that our Data is important and how it can help us not only manage Risk but create value. But, when it comes to actually applying it, we are hamstrung by two things:  One is that we haven't quite grasped the ways in which we can internalise that Data. We understand the value but the actual application is not really out there currently. Secondly, I think that in some places, we have too much activity. I've been in places where there have been competing Data agendas and competing Data Governance ideas. When people are not taking their organisational view and just looking to get ahead, it’s hard to achieve any real success.  If you were advising a company about to commence on a large Data Management transformation project, what advice would you give them? This links to the previous point really, and it’s a bigger issue in large companies. You need to have a business approach to Data Governance, as well as the IP capabilities to deal with a project of that scale. And what you find sometimes is that multiple groups get together and they each have a different view of what good looks like. They end up not communicating throughout the organisation and properly aligning everybody’s roles and responsibilities. These different agendas then end up causing issues because everyone has a different idea of what they want.  We need to be able to plan across the organization to get the right agenda and get the right properties in place. Then you can start the work, as opposed to each team just working where they think the biggest problem lies first.  What would you say are the biggest threats to a successful Data Management program? Obviously the above is one, but it leads to another which is really the lack of Senior Management sponsorship. If you don’t get the right level of sponsorship, then you don’t get the mandate to do what you need. This can cause huge delays and is definitely one of the biggest threats to your program being a success.  In finance, you worked within a highly regulated industry. How have your approaches changed now that you’re in a highly innovative, tech-driven environment? The approach is different. We do have challenges that others don't, but over and above, because we innovate and create things, there is an abundance of new information. Information protection and intellectual property protection is therefore at the top of the agenda. That drives the need for effective Data Governance and it really has to be at the forefront of the approach.  Data breaches have caused widespread reputational damage to companies such as Facebook and Yahoo. Have you found that companies now view Data protection as central to their commercial performance? Absolutely. People realize that they not only need Data to do their business, but they also need to protect that Data. These breaches have resulted in a greater importance being given to this function and every year I see it moving closer to the center of the organisation. There are very few large organisations left that haven’t recognized Data Protection as one of their formal functions. A lot of companies are now looking to build out their Data Protection teams from the ground up, starting with lower levels of analysts, but also management as well. It’s becoming a much greater priority and these big breaches are one of the driving factors.  What do you feel will be the most effective technical advancement within Data Management in 2019? I think, from a technological perspective, we still have some way to go with digital rights management. There’s now one or two solutions that are supposed to be at Enterprise level, but they’re not enough and they’re still not joining the digital rights management side of things with the Big Data Loss Prevention side.  So companies are having to rely on seeing this together with a combination of plugin software and various tools and technology. It’s sticking around the edges of the edges of a fix, but it’s not actually doing the job. I'd like to see these technologies develop because I think we're crying for some help in this area.  What is the biggest risk to their Data that businesses should be aware of? Not knowing where to get hold of Data. It is just mind boggling to me, that there has not been a single company that I have been a part of where we started a program and we knew where to get all our Data from. Obviously we knew where most of it was,  but we didn’t know where else it was and that what we were looking at was a comprehensive set of maps. It just continues to be the same at every business I have worked at.   What role does data governance have to play in protecting a business’ intellectual property? It plays a huge role. Firstly, a company needs to be very clear on their Data policies. This means regularly training teams on the importance of this, much like you would with health and safety. By clearly defining and educating people on the dos and don’ts of data handling you can better protect your intellectual property. I think getting the policy framework right and implementing it using digital rights management is crucial and good Data Governance relies on this.  When hiring for your teams, which traits or skills do you look for in candidates? There are two key parts; one is technical and the other non-technical. In my mind, it’s less about the technical because, ultimately, I just want someone who knows how to use ‘technology x’. They need to be able to make use of Data from a database, or be able to spot Data in an unstructured environment. But, for me, the most important skill is more of a characteristic: tenacity. I use the word tenacity because you have to put yourself out there. You have to ask people questions and you have to educate them. You can’t assume that people just understand Data you’re presenting them and you have to become their friends and learn to speak their language. It also really brings in the skill of being able to work with teams and across teams. Being a team player would absolutely be top of my list. Mridul spoke to Femi Akintoye, a Recruitment Consultant in our Data & Technology function. Take a look at our latest roles or get in touch with Femi.

Recently Viewed jobs