Applications are closed for this role. It was originally posted 2026-07-28. It’s no longer accepting applicants — see roles ShyftLabs is still hiring for →, or browse the live openings below.
About this role
Technical Leadership
- Lead the architecture, design, and implementation of enterprise-scale data platforms from project inception through production deployment.
- Own technical delivery across multiple client engagements while ensuring high-quality engineering standards.
- Define solution architecture, technical roadmaps, and implementation strategies aligned with client business goals.
- Conduct architecture reviews, code reviews, and establish engineering best practices across project teams.
- Mentor and coach Data Engineers while fostering technical excellence and continuous learning.
- Serve as the primary technical leader for complex engineering initiatives and critical project decisions.
Client Partnership
- Partner directly with Fortune 500 clients to understand business requirements and translate them into scalable technical solutions.
- Lead discovery workshops, architecture sessions, and technical planning meetings with both business and engineering stakeholders.
- Present solution designs, delivery plans, and architectural recommendations to technical leadership and executive audiences.
- Build trusted relationships with client teams while providing technical guidance throughout project execution.
- Support pre-sales activities by contributing technical expertise, solution estimates, and implementation approaches when required.
Data Engineering & Platform Development
- Design, develop, and optimize enterprise-grade data pipelines using the Databricks Unified Analytics Platform.
- Build scalable ETL and ELT frameworks capable of processing large-scale structured and unstructured datasets.
- Design and implement Lakehouse architectures using Delta Lake and Medallion design patterns.
- Develop high-performance Spark applications for batch and real-time data processing.
- Integrate data from enterprise applications, APIs, streaming platforms, and cloud storage solutions.
- Ensure data quality, integrity, and reliability through automated validation, testing, and monitoring.
Cloud & DevOps
- Architect cloud-native data platforms across AWS, Azure, or Google Cloud Platform.
- Implement Infrastructure-as-Code using Terraform or similar technologies.
- Build and maintain CI/CD pipelines supporting automated testing and deployment.
- Optimize cloud infrastructure for scalability, reliability, security, and cost efficiency.
- Monitor platform performance and proactively resolve operational issues.
Data Governance & Security
- Implement enterprise data governance frameworks and security best practices.
- Configure Unity Catalog, metadata management, lineage, and role-based access controls.
- Ensure compliance with organizational security standards and regulatory requirements.
- Promote data observability and operational excellence across production environments
Cross-Functional Collaboration
- Partner closely with Product Managers, Data Scientists, Analytics Engineers, Machine Learning Engineers, and Software Engineers to deliver high-impact data products.
- Enable AI and machine learning initiatives through scalable feature engineering pipelines and production-ready datasets.
- Contribute reusable frameworks, accelerators, and engineering standards that improve delivery across client engagements.
- Bachelor's or Master's degree in Computer Science, Data Engineering, Software Engineering, or a related technical discipline.
- 8+ years of experience designing and building enterprise-scale data platforms.
- 5+ years of hands-on experience with Databricks and Apache Spark.
- Proven experience leading enterprise data engineering projects from architecture through production delivery.
- Strong expertise in Python, SQL, and Spark for large-scale data processing.
- Deep understanding of Delta Lake, Lakehouse architecture, and modern data platform design.
- Experience working with AWS, Azure, or Google Cloud Platform.
- Strong knowledge of ETL/ELT frameworks, distributed computing, and data modeling.
- Experience implementing CI/CD pipelines and Infrastructure-as-Code.
- Strong understanding of data governance, security, metadata management, and data quality practices.
- Experience optimizing distributed data processing workloads for performance and cost.
- Excellent communication and stakeholder management skills with experience working directly with enterprise clients.
- Demonstrated ability to mentor engineers and lead technical initiatives
- Databricks Certified Professional Data Engineer certification.
- Experience with Delta Live Tables, MLflow, Unity Catalog, and Databricks SQL.
- Experience with Kafka, Kinesis, Event Hubs, or other streaming technologies.
- Hands-on experience with Snowflake, dbt, Airflow, or modern data orchestration tools.
- Experience with Kubernetes, Docker, and Terraform.
- Knowledge of AI/ML data platforms, Feature Stores, or Retrieval-Augmented Generation (RAG) architectures.
- Previous consulting or professional services experience delivering solutions for enterprise clients.
- Experience within retail, e-commerce, financial services, logistics, healthcare, or ad-tech environments.
- $140,000 – $180,000 (CAD)
Tech stack
DatabricksSparkAWSAzureTerraformPython