Senior Data Engineer

Jellyfish processes a huge amount of engineering data, and we are investing heavily in the foundations that make that data reliable, governable, and easy to use. We are looking for a Data Engineer to help mature our Databricks-based data platform, establish strong data modeling patterns, and build the systems that move data from raw ingestion to trusted production datasets.

You’ll work across ingestion, transformation, storage, governance, and serving. If you enjoy turning messy data pipelines into durable platform architecture and want to help define how a modern lakehouse should actually operate, you’re the perfect fit.

What you’ll actually be doing:

• Databricks Platform Development - You’ll build and maintain data pipelines and datasets in Databricks and Delta Lake, improving reliability, performance, and operational visibility across the platform.

• Medallion Architecture - You’ll help establish clear Bronze, Silver, and Gold layer responsibilities, including standards for schema evolution, transformation ownership, data retention, and promotion between layers.

• Data Modeling - You’ll design durable canonical models for core Jellyfish entities and relationships. You’ll work with application and analytics teams to ensure downstream datasets are structured around consistent definitions rather than one-off transformations.

• Pipeline Engineering - You’ll build and improve batch and incremental pipelines using technologies like Databricks, Airflow, Spark, and cloud object storage. You’ll focus on idempotency, scalability, observability, and recoverability.

• Data Governance and Quality - You’ll work with our catalog and governance tooling to establish lineage, ownership, schema standards, quality checks, and discoverability across the platform.

• Serving and Egress - You’ll help create reliable patterns for moving curated data from Databricks into systems like ClickHouse and other future serving destinations without tightly coupling the platform to any single database.

You’re a great fit if:

• Databricks Experience - You’ve worked extensively with Databricks, Spark, Delta Lake, or a comparable lakehouse platform and understand how to operate it beyond simply writing notebooks.

• Data Engineering Fundamentals - You understand partitioning, incremental processing, schema evolution, distributed execution, file formats, and the performance characteristics of large analytical datasets.

• Strong Data Modeling Skills - You can reason about canonical entities, relationships, grain, dimensional modeling, and the boundary between platform models and consumer-specific models.

• Pipeline Reliability Mindset - You design pipelines to be observable, retryable, idempotent, and understandable when they fail.

• Cloud Fluency - You understand how object storage, compute, networking, IAM, and managed data services fit together in a modern cloud data architecture.

• Pragmatic Platform Builder - You care about standards and architecture, but you also know when to ship a practical solution and iterate.

Bonus Points:

• You’ve helped build or migrate to a medallion-style lakehouse architecture.

• You’ve worked with Databricks Unity Catalog, OpenMetadata, or another governance and lineage platform.

• You’ve implemented CDC pipelines from PostgreSQL, RDS, or Aurora.

• You’ve worked with Airflow or another production workflow orchestration platform.

• You’ve moved analytical data into serving systems like ClickHouse, Snowflake, BigQuery, or similar platforms.

• You’ve helped introduce data contracts, canonical schemas, or platform-wide data quality standards.

A list of job experiences and qualification requirements is great, but humility, a performance-driven attitude, and a team-player approach are most important to us. We love to have fun and win in the process. We only hire people who have a passion for building great companies in an environment where a sense of humor is a must.

Occasional travel may be required.

Applicants must be authorized to work for any employer in the US. We are unable to sponsor or take over sponsorship of an employment visa at this time.

Let’s talk about us!
This is all about you, but you want to know a little about us. Jellyfish https://jellyfish.co is the leading intelligence platform for AI-Integrated engineering, helping more than 1,000 companies including DraftKings, Keller Williams and Blue Yonder, leverage AI to transform how they build software. By combining the industry’s deepest engineering dataset with context-rich intelligence, Jellyfish helps R&D organizations understand what’s driving impact, adopt proven industry best practices, and make smarter decisions across AI adoption, planning, delivery, and engineering performance.