Technical Skills
• Databricks Lakehouse Platform:
o Medallion Architecture o Unity Catalog (data governance, lineage)
o Delta Lake & DLT Pipelines
o PySpark Workbooks o Spark SQL & SQL Warehouse
• Programming: Python, SQL, PySpark
• AWS Cloud Services: IAM, S3, Lambda, EMR, Redshift, Bedrock
• Other:
o Familiarity with DevOps and CI/CD processes
o Experience with any Data Security tool
Key Responsibilities
• Establish, enforce, and monitor Data Engineering governance standards, best practices, and guidelines across enterprise and self-service environments.
• Develop and maintain documentation for data engineering processes
• Define and implement data security policies and role-based access controls (RBAC/ABAC) across all data engineering processes.
• Oversee data classification, lifecycle management, and comprehensive metadata management to ensure transparency, traceability, and compliance.
• Implement and manage robust change control processes for all data engineering activities
• Monitor, maintain, and optimize data pipelines and workflows, ensuring reliability, scalability, and efficiency.
• Continuously monitor and optimize the performance of data engineering processes and resource utilization.
• Promote a culture of data engineering governance, data security awareness, and operational excellence within the team and across the organization.