•
Coordinate with data scientists, data analysts, software engineers and product managers to design and evolve our existing architecture, improving the reliability and performance of our data infrastructure
•
Build, maintain and scale our ETL pipelines to help power data science, data products and analytical needs
•
Design, build, and maintain data infrastructure systems such as but not limited to AWS DMS, HBase, EMR, Redshift, S3, Kafka, Airflow while ensuring scalability, reliability, and security
•
Write new tools, services, and platforms in Python
•
Research new technologies to solve tomorrow’s data infrastructure needs
•
Advocate for the adoption of tools in the data engineering team
•
4+ years of data engineering experience
•
Ability to write high-quality code in Python and PySpark
•
Strong understanding of database fundamentals and software engineering principles
•
Strong understanding of Data Contracts, microservices principles, and treating datasets/platforms as API-driven products
•
Deep understanding of two or more of these technologies: Spark, Airflow, AWS EMR, AWS Glue, Sagemaker and Redshift
•
Experience with batch processing and streaming systems
•
Exposure and/or strong interest in Kafka or other queuing technologies
•
A critical thinker and possess creative problem solving skills
•
Quantitative skills, ability to think analytically, and strong attention to details
•
A desire to contribute to the broader development community