We are seeking a highly skilled Senior Data Engineer to lead the migration, optimization, and governance of data pipelines in Azure Data Factory (ADF) and SQL.
The ideal candidate will have extensive experience in Change Data Capture (CDC), performance tuning, data security, and compliance within a cloud-based architecture.
- Architect and optimize CDC pipelines in ADF, Data Sync Services, and SQL to ensure efficient data ingestion.
- Implement performance tuning strategies such as parallel processing, indexing, partitioning, and data archiving.
- Enforce security and compliance by implementing RBAC, data encryption, audit logging, and access control best practices.
- Define and implement data governance models, including roles, metadata management, and compliance policies.
- Develop robust error handling and monitoring with automated alerts, logging strategies, and retry mechanisms for CDC failures.
- Provide technical documentation and best practices for CDC implementation, security, and performance optimization.
Skills Required
Sql, Performance Tuning, rbac