Senior Data Specialist - R01565550

Lead Data Engineer

• ETL Fundamentals, SQL, SQL (Basic + Advanced), Python, Data Warehousing, Modern Data Platform Fundamentals, Data Modelling Fundamentals, PLSQL, T-SQL, Stored Procedures
• Databricks Engineering: Data Engineer

Job Description – Data Engineer (Databricks)

Company: Brillio Technologies
Role: Data Engineer – Databricks
Experience: 5–10 Years
Location: India (Hybrid/3 days)
Employment Type: Full-time

About Brillio

Brillio is a leading digital transformation partner delivering innovative solutions across Data, Cloud, AI, and Digital Engineering. We help global enterprises unlock value through cutting-edge technology and data-driven insights.

Role Overview

We are looking for a hands-on Data Engineer with strong Databricks expertise to design, build, and scale modern data platforms. The ideal candidate will have experience in PySpark, Delta Lake, and cloud-native data engineering and will work closely with analytics, product, and business teams to deliver high-performance data pipelines.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines using Databricks and PySpark

• Build and optimize data lakehouse architectures (Bronze–Silver–Gold layers)

• Develop data ingestion frameworks from multiple sources (batch & real-time)

• Implement Delta Lake features including schema evolution, ACID transactions, and optimization

• Create and manage Databricks notebooks, workflows, and jobs for production pipelines

• Ensure data quality, governance, and lineage across pipelines

• Optimize Spark jobs for performance (partitioning, caching, tuning)

• Collaborate with cross-functional teams (Data Scientists, Analysts, Architects)

• Support production deployments, monitoring, and troubleshooting of data pipelines

• Contribute to migration of legacy data platforms to modern cloud-based architecture

Technical Skills (Must-Have)

• Strong experience in:

• Databricks (Workflows, Notebooks, Jobs)

• PySpark / Spark SQL

• SQL & Data Warehousing concepts

• Hands-on experience with:

• Delta Lake / Lakehouse architecture

• Data pipeline development & orchestration

• Good understanding of:

• Data modeling, ETL/ELT processes, SCD concepts

Cloud & Tools

• Experience in at least one:

• AWS (S3, Glue, EMR) / Azure (ADF, ADLS, Synapse)

• Workflow tools:

• Airflow / Databricks Workflows

• Version control:

• Git / CI-CD pipelines

Good to Have

• Experience with:

• Streaming frameworks (Kafka / Spark Streaming)

• DBT / Snowflake / Redshift / Fabric

• Exposure to:

• Unity Catalog, Autoloader, Delta Live Tables

Experience Required

• 5–10 years of experience in Data Engineering

• Minimum 2–4 years of hands-on Databricks experience

• Proven experience building enterprise-scale data platforms and pipelines

Soft Skills

• Strong problem-solving and analytical skills

• Ability to work in a fast-paced, agile environment

• Good communication and stakeholder management

Why Join Brillio

• Work on cutting-edge data & AI programs

• Opportunity to build large-scale data platforms

• Collaborative and innovation-driven culture