Job Description
<p><p><b>Job Description : Data Engineer Azure Data Pipelines & Governance</b><br/><br/> <b>Overview : </b><br/><br/> We are seeking a hands-on Data Engineer to develop, optimize, and maintain automated data pipelines supporting data governance and analytics initiatives.<br/><br/> This role will focus on building production-ready workflows for ingestion, transformation, quality checks, lineage capture, access auditing, cost usage analysis, retention tracking, and metadata integration, primarily using Azure Databricks, Azure Data Lake, and Microsoft Purview.<br/><br/> <b>Location : </b> Offshore.<br/><br/> <b>Experience : </b> 4+ years in data engineering, with strong Azure and Databricks experience.<br/><br/> <b>Key Responsibilities : </b><br/><br/> - Pipeline Development Design, build, and deploy robust ETL/ELT pipelines in Databricks (PySpark, SQL, Delta Lake) to ingest, transform, and curate governance and operational metadata from multiple sources landed in Databricks.<br/><br/> - Granular Data Quality Capture Implement profiling logic to capture issue-level metadata (source table, column, timestamp, severity, rule type) to support drill-down from dashboards into specific records and enable targeted remediation.<br/><br/> - Governance Metrics Automation Develop data pipelines to generate metrics for dashboards covering data quality, lineage, job monitoring, access & permissions, query cost, usage & consumption, retention & lifecycle, policy enforcement, sensitive data mapping, and governance KPIs.<br/><br/> - Microsoft Purview Integration Automate asset onboarding, metadata enrichment, classification tagging, and lineage extraction for integration into governance reporting.<br/><br/> - Data Retention & Policy Enforcement Implement logic for retention tracking and policy compliance monitoring (masking, RLS, exceptions).<br/><br/> - Job & Query Monitoring Build pipelines to track job performance, SLA adherence, and query costs for cost and performance optimization.<br/><br/> - Metadata Storage & Optimization Maintain curated Delta tables for governance metrics, structured for efficient dashboard consumption.<br/><br/> - Testing & Troubleshooting Monitor pipeline execution, optimize performance, and resolve issues quickly.<br/><br/> - Collaboration Work closely with the lead engineer, QA, and reporting teams to validate metrics and resolve data quality issues.<br/><br/> - Security & Compliance Ensure all pipelines meet organizational governance, privacy, and security standards.<br/><br/> <b>Required Qualifications : </b><br/><br/> - Bachelors degree in Computer Science, Engineering, Information Systems, or related field.<br/><br/> - 4+ years of hands-on data engineering experience, with Azure Databricks and Azure Data Lake.<br/><br/> - Proficiency in PySpark, SQL, and ETL/ELT pipeline design.<br/><br/> - Demonstrated experience building granular data quality checks and integrating governance logic into pipelines.<br/><br/> - Working knowledge of Microsoft Purview for metadata management, lineage capture, and classification.<br/><br/> - Experience with Azure Data Factory or equivalent orchestration tools.<br/><br/> - Understanding of data modeling, metadata structures, and data cataloging concepts.<br/><br/> - Strong debugging, performance tuning, and problem-solving skills.<br/><br/> - Ability to document pipeline logic and collaborate with cross-functional teams.<br/><br/> <b>Preferred Qualifications : </b><br/><br/> - Microsoft certification in Azure Data Engineering.<br/><br/> - Experience in governance-heavy or regulated environments (e., finance, healthcare, hospitality).<br/><br/> - Exposure to Power BI or other BI tools as a data source consumer.<br/><br/> - Familiarity with DevOps/CI-CD for data pipelines in Azure.<br/><br/> - Experience integrating both cloud and on-premises data sources into Azure.</p><br/></p> (ref:hirist.tech)