Data Engineer

• Build and maintain ETL/ELT pipelines in Microsoft Fabric Notebooks (Python/PySpark) ingesting data from multiple source systems via REST and SOAP APIs • Implement Bronze layer landing (raw data, no transformation), Silver layer cleansing/typing/deduplication, and Gold layer aggression for business analytics • Design and build custom API connectors with OAuth 2.0/Bearer token authentication, incremental sync, pagination handling, rate-limit/retry logic and error recovery • Configure and manage Fabric Workspaces across Dev/Test/Production environments using Fabric Deployment Pipelines and Git integration for CI/CD • Build and maintain Power BI semantic models (star/snowflake schemas) supporting operational reporting and analytics dashboards • Implement row-level security (RLS) using Azure AD RBAC, warehouse-level DAX filters, and dynamic RLS platforms • Set up monitoring, alerting, and telemetry using Azure Monitor, Log Analytics, and Application insights to track pipeline health and data freshness • Manage API credentials and secrets via Azure Key Vault with automated rotation policies • Embed within the client's development process, working directly with business stakeholders to gather requirements and co-develop solutions • Contribute to data governance: data dictionary creation, lineage tracking, and documentation of transformation logic across all pipeline stages
• 5+ years experience in data engineering or a similar role in a commercial environment • Hands-on experience with Microsoft Fabric (Lakehouses, Notebooks, Data Pipelines, OneLake, SQL Analytics Endpoints) or equivalent depth in Azure Synapse Analytics • Advanced SQL for data transformation, performance tuning, and Delta Lake table management • Proficient in Python/PySpark for data processing, API integration, and pipeline automation • Proven experience building custom API connectors (REST, SOAP/XML) with OAuth 2.0, pagination, rate limiting, and incremental sync patterns • Strong understanding of Medallion Architecture, data lakehouse concepts, and Delta Lake (merge, upsert, soft-delete handling, time travel) • Experience with dimensional modelling (star/snowflake schemas) and Power BI semantic model development • Working knowledge of Azure services: Key Vault, Azure Monitor, Log Analytics, Azure AD/Entra ID • Experience translating business requirements into dimensional models iteratively (not just building pre-designed schemas) • Comfort operating autonomously with minimal supervision in a client-embedded model • Experience implementing row-level security in Power BI (DAX-based RLS, dynamic security models) • Familiarity with CI/CD in a Fabric context: Deployment Pipelines, Git integration, environment promotion workflows • Experience with semi-structured data formats (JSON, XML) and handling schema evolution across pipeline layers
• Prior work in retail, wholesale, distribution, or fresh produce / supply chain data environments • Experience with ERP data extraction and replication tools in a Fabric or Azure context • Knowledge of dbt, Power Query, or similar transformation frameworks • Relevant Azure or Microsoft certifications (DP-600, DP-203, PL-300) • Openness to AI coding assistants (GitHub Copilot, Cursor, Claude) as part of your development workflow
• Competitive salary with mentorship and career growth opportunities • Hands-on work with cloud-first technologies (Microsoft Fabric, Azure, Power BI) • Long-term embedded role with a single client, providing deep domain expertise and continuity • Fast-paced, innovative environment where AI-augmented development is the norm