Data Engineer

San Francisco, California
US$120000 - US$160000 per year + Equity

This vacancy has now expired. Please see similar roles below...

Data Engineer
120-160 with Stock options, Work from home and Flex hours
San Francisco, CA

THE COMPANY:

Very mission driven and passionate about providing financial security to developing countries that do not have the access or resources to.

THE ROLE:

The main responsibly for this role is to be the driving force in design and development of data solutions on the AWS hosted platform. They are providing financial security for developing companies. By helping make small transactions to develop better credit and pull people out of poverty.

In specific, you can expect to be involved in the following:

  • Be a huge part in the design and implementation of scalable ETL and data processing systems to go into our big data ecosystem including data collection, cleaning, processing, ETL and the creation of a common data lake.
  • Work alongside architects to work on the engineering and infrastructure to improve data strategy, quality and governance.
  • Work closely with both business and partner stakeholders to quickly deliver high-quality applications.
  • Work closely with architects, product owners, data scientists and test engineers to help bring data science R&D projects into Production.
  • Manage data infrastructure to grow and support the Data Science and Engineering team in relation to the construction of performance data products.
  • Establish the technical direction for the team, driving the necessary changes and making appropriate technology choices working collaboratively with the Data Science, Product and Engineering Teams.
  • Maintain and scale production environments for ML-based data products.

YOUR SKILLS AND EXPERIENCE:

  • Have a BS/MS/PhD in Computer Science or related quantitative fields.
  • Strong experience in enterprise-scale software architecture is preferred
  • Strong experience in data engineering and software development.
  • Strong experience designing data warehouse/data lake and ETL architectures with big data technologies such as Spark, Spark Streaming, Hive, Storm, Sqoop, Kafka, Hbase, and HDFS.
  • Working experience and knowledge with AWS services.
  • Experience with SQL/NoSQL systems such as MongoDB, Cassandra, Solr, Elasticsearch, Redshift, DynamoDB, etc.
  • Fluent in at least one of the following programming languages Scala, Python, Java.
  • Familiar with infrastructure frameworks such as Terraform, Docker, Kubernetes, etc.
  • Experience in Agile processes and scrum.
  • Highly organized, structured work approach and dependable.

THE BENEFITS:

  • Stock options
  • Work from home
  • Flex hours
  • 401k
  • Weekly catered lunches
  • Choice of best and newest tech from headphones, to apple products etc
  • Discount on Samsung products

HOW TO APPLY:

Please register your interest by sending your CV to Kyle Wong via the Apply link on this page.

16823
San Francisco, California
US$120000 - US$160000 per year + Equity

Similar Jobs

Salary

US$110000 - US$130000 per year + Benefits

Location

New York

Description

A fast-paced fin-tech start-up is looking for a Software Engineer to join them in New York! Click below to read more!

Salary

US$120000 - US$150000 per year + Benefits

Location

Boston, Massachusetts

Description

A global technology company are looking for a Big Data Engineer to join them in the heart of Boston! Click below to read more

Salary

US$160000 - US$180000 per year + Benefits

Location

New York

Description

One of the most exciting start-ups in NewYork are looking for a Lead Software Engineer with Ruby on rails experience to join them! Click below to read more!

Salary

US$160000 - US$180000 per year + 1.5% equity

Location

New York

Description

One of the most exciting start-ups in NewYork are looking for a Lead Saas Software Engineer to join them! Click below to read more!

Salary

US$150000 - US$170000 per year + + $50,000 benefits

Location

Boston, Massachusetts

Description

A global technology company are looking for a Big Data Architect to join them in the heart of Boston! Click below to read more!

Harnham blog & news

With over 10 years experience working solely in the Data & Analytics sector our consultants are able to offer detailed insights into the industry.

Visit our Blogs & News portal or check out our recent posts below.

Big Data In Politics – Win, Lose, Or Draw

Big Data In Politics – Win, Lose, Or Draw

In the movie Definitely, Maybe starring Ryan Reynolds, there’s a scene in which he must sell tables for a political campaign dinner fundraiser. He makes call after call with no luck. Finally, in frustration, he speaks plainly and finds a connection between the politician and the prospective donor. In an instant, he understands. Make the connection and you can’t go wrong. This is the 90’s version of micro-targeting. Online advertising today has honed targeted Marketing to an art form and it’s infused every industry from Fisherman’s Wharf to Wall Street to Washington. Messages are crafted on detailed profiles of what makes us unique such as hopes, fears, dreams, emotional triggers, and more which is then taken out of the hands of humans. Enter such deep, personal details into automated technologies and you’ll get automated reactions. How did we get here? Ever since Cicero’s brother, Quintus, who approached politics with a do anything to win mindset, we’ve been working toward this point. But, when it comes to technological advances within politics, George Simmel put it best when he wrote around 1915, “the vast intensive and extensive growth of our technology…entangles us in a web of means, and means toward means, more and more intermediate stages, causing us to lose sight of our real ultimate ends.”  What does this mean? It means we have moved so quickly and with such intensity as we push inwards while reaching outward, we get tangled up in our own systems. Before we know it, it’s difficult to separate the means from their ends, and we lose sight of our purpose. In other words, it can be hard to keep our sense of direction with our constant distraction of tasks, systems, and processes. According to Simmel, this would soon morph into what he called a ‘fragmentary character.’ Like a mosaic, we put the pieces back together and assemble the bits to fit our concept of the world.   The Digitizing of Campaigns Traditional campaigning has traditionally looked much like the movie scene mentioned above with phone banks, whiteboards, and handmade signs. But, today, things are changing. Everyone has at least one smart device which can sync information in real time to a range of devices. Algorithms and predictive modeling help reduce the guesswork, though gut feeling and instinct still prevail. At least, for now. Our machines are learning how to learn about us and define what we believe and wish to see by historical Data, or rather our past behaviors. Where psychographic profiling meets micro-targeting. What was once only seen in the Marketing world has now entered politics. Just like marketers want to know what people are interested in, so to do politicians wish to know what voters think. To do this, both industries will study behavioral and attitudinal profiles to help understand a demographic better or discern a gap in the marketplace. In consumer research, companies rely on psychographic micro-targeting to reach smaller groups and individuals. The key question here is to ask is to what extent are politicians prepared to pass laws that restrict their own opportunities to know more about voters. Just as the next generation of voters are coming, so too are the next generation of tools being developed.  One Final Thought… Over the last 20 years or so, we have built an immense Data structure from mobile devices to social media to modelling processes and more. With this kind of connectivity combined with fragmentary media, the use of Data Analysis has a big role to play going forward. If we seek change in our political and social infrastructures, we will have to reimagine the structures currently in place. From algorithmic modelling to AI and Machine Learning, the possibilities for new ideologies has emerged blurring the lines between context and production in which Data underpins capitalism. As those in Data Analytics continue to pursue an uninterrupted (read: non-fragmentary) vision of the world, we find ourselves at a new stage in history of where both looking back and looking forward at the same time informs our future.   Where would you like to go? If you’re interested in Big Data & Analytics, we may have a role for you. Take a look at our latest opportunities or contact one of our expert consultants to find out more:  For our West Coast Team, call (415) 614 - 4999 or send an email to sanfraninfo@harnham.com.  For our Mid-West and East Coast Teams, call (212) 796 - 6070 or send an email to newyorkinfo@harnham.com.

Going Green With Big Data

Going Green With Big Data

Greta Thunberg sailed the Atlantic to come the UN to talk about climate change. Her mother, a renowned opera singer, has given up air travel to support her daughter’s efforts. There is a zero-waste movement to lessen our trash and help alleviate the carbon footprints from our buying, traveling and more. These are steps humans have made. Yet technological advances may make it possible to flip the script for the environment and Big Data has a big role to play.   What are Some of the Advances Taking Place? Technological advances have brought us breakthroughs in modern science and in every industry. Now, we are at a time and place in where our technologies cam help tackle climate change. From modeling to predictions, we can begin to build not just a map of environmental concerns, but begin to build a road toward a solution. Below are just a few of the ways technology is being used to advance solutions for climate change. AI modeling makes it easier to identify problemsPredictive Analytics models can create different scenarios to see ‘what happens if?’Big Data is used to identify areas which need immediate attention This is just the tip of the iceberg when it comes to using technology to predict and identify climate concerns. While some parts of the world contribute more to the problem than others, Big Data has made it possible to draw conclusions where the hardest hit areas are and is key to addressing the problem. But whatever Data brings, the information is useless if it isn’t used to formulate and put forward better environmental practices and policies.  Ways to Upscale Urban Data Science  Manhattan, Berlin, and New Delhi, as varied as they are, have one thing in common. They’re often sites for case studies when it comes to analyzing our environment. However, our advances continue to improve and we’re able to learn from state-of-the-art Data infrastructures. These can include such things as social media data combined with earth observations to see how they might better integrate. A research publication in Berlin suggest three routes for expanding knowledge. They are: Mainstream Data collectionsAmplify Big Data and Machine Learning to scale solutions and maintain privacyUse computational methods to analyze qualitative Data With these advances in place, there is a chance urban climate solutions could effect change on a global scale. With the proper Data of urban areas in place, including that of related greenhouse gases, socio-economic issues, and climate threats, Data professionals can get a clearer picture of what needs to be done. Building on the advances that are in place with the integrated technologies of AI, Predictive Analytics, and Big Data helps make big strides in combatting climate change. According to reports, only about 100 cities make up 20% of the global carbon footprint. Yet 97% of climate concerns are focused in urban areas. There’s still a lot which remains to be done to combat the greatest issue of our age, but working hand in hand – machine and human – we just might find ourselves on reprieve and the chance to leave the world better than we found it for the next generation. The next Greta Thunbergs of the world. If you’re interested in Big Data & Analytics, we may have a role for you. Check out our current opportunities or get in touch with one of our expert consultants to learn more.  For our West Coast Team, call (415) 614 - 4999 or send an email to sanfraninfo@harnham.com.  For our Mid-West and East Coast Teams, call (212) 796 - 6070 or send an email to newyorkinfo@harnham.com.

Recently Viewed jobs