<p><p><b>About the job :</b></p><p><br/></p><p>Location : Kochi / Bangalore (initially in Kochi)<br/><br/><b>Role Summary :</b></p><p><br/></p><p>We are seeking a Data Engineering Lead to architect, build, and optimize scalable data pipelines and data platforms that power critical business decisions.
The ideal candidate will have deep expertise in PySpark, ETL frameworks, SQL Stored Procedures, and Data Modeling, along with a strong focus on performance tuning across large datasets.<br/><br/><b>Key Responsibilities : </b></p><p><br/></p><p>- Lead the design and development of scalable data pipelines and ETL workflows using PySpark and SQL<br/><br/>- Develop and maintain Stored Procedures to support batch processes and data transformations<br/><br/>- Design and implement robust data models (Star, Snowflake, and normalized models) across data warehouse and lake environments<br/><br/>- Work closely with BI, Analytics, and Product teams to understand data requirements and ensure data quality, lineage, and integrity<br/><br/>- Optimize data workflows for performance, cost, and scalability across distributed systems<br/><br/>- Manage data ingestion from structured and semi-structured sources (e.g., APIs, files, databases, streams)<br/><br/>- Support CI/CD and DevOps practices for data pipeline deployments<br/><br/>- Collaborate with cross-functional teams on data governance, access controls, and compliance<br/><br/>- Guide and mentor junior data engineers; perform code reviews and technical planning<br/><br/><b>Required Skills & Experience : </b></p><p><br/></p><p>- 5+ years of experience in data engineering and data platform development<br/><br/>- Experience with Azure ADlS Synapse or Microsoft Fabric<br/><br/>- Expert in PySpark / Spark SQL for large-scale distributed data processing<br/><br/>- Solid understanding of ETL/ELT architectures, data ingestion, and transformation strategies<br/><br/>- Experience in data modeling techniques and tools<br/><br/>- Proven experience in performance tuning of queries, jobs, and pipelines<br/><br/>- Experience with tools like Airflow or equivalent for orchestration and transformations<br/><br/>- Hands-on experience with version control, CI/CD, and data testing frameworks<br/><br/><b>Nice To Have : </b></p><p><br/></p><p>- Exposure to Data Virtualization platforms (Denodo, Dremio, etc.)<br/><br/>- Exposure to Purview or any Data Governance tools.</p><br/></p> (ref:hirist.tech)