Role: Databricks Senior Developer
Location: Kolkata
Experience: 8 Plus Years
Role**Databricks Senior Developer
Required Technical Skill Set**Databricks, Spark and Data Migration experience
Desired Competencies (Technical/Behavioral Competency)
Must-Have**
1.
8+ years with strong Databricks, Spark and Data Migration experience
2.
Should have experience of end-to-end migration from AWS Redshift or other RDS to Databricks Lakehouse Platform.
Need to have knowledge in Cloud Data Engineering, ETL Modernization, Spark Optimization and SQL-to-PySpark migration.
3.
Key Responsibilities:
a) Migration Strategy and Execution: analyze the existing DWH/DB architecture, Design & Implement scalable data pipelines in Databricks, SQL Script conversion, etc.
b) Development & Optimization: Build Delta Lake tables and Implement ACID-complaint data models, Manage metadata, schema evolution and lineage for migrated datasets
c) Collaboration & Governance: Work with Data Architect to define Target data models and migration approaches, Collaborate with DevOps teams to implement CI/CD pipelines, Data Governance teams on Security, lineage and cataloging;
Mentor junior members in the team
d) Nice to have experience with Unity Catalog for governance and access control;
Exposure to machine learning/AI pipelines, etc.