Data Engineer - Spark/Azure Databricks

<p><p><b>Title :</b> Data Engineer<br/><br/><b>Experience :</b> 4 to 6 years<br/><br/><b>Location :</b> Chennai<br/><br/><b>Must Have :</b><br/><br/></p><p>- Databricks<br/><br/></p><p>- Spark<br/><br/></p><p>- Snowflake Schema<br/><br/></p><p>- Azure<br/><br/><b>Role And Responsibilities :</b><br/><br/></p><p>- Develop, maintain, and monitor data pipelines, ensuring reliability and prompt resolution of failures as they occur.<br/><br/></p><p>- Collaborate with stakeholders to gather requirements, provide updates, and participate in design discussions.<br/><br/></p><p>- Design and implement custom ETL processes and automation scripts using AWS, Databricks, Python, and PySpark.<br/><br/></p><p>- Prepare architecture and design documentation for data engineering projects.<br/><br/></p><p>- Utilize expertise in cloud-based big data integration and infrastructure, with a strong focus on Databricks and the Apache Spark framework.<br/><br/></p><p>- Demonstrate proficiency in Python and other scripting languages for data engineering tasks.<br/><br/></p><p>- Leverage strong SQL and PL/SQL skills to write complex procedures and queries for dataframes and notebooks.<br/><br/></p><p>- Maintain a data engineering mindset to ensure quality, reliability, and efficiency across data systems and processes.<br/><br/><b>Minimum Qualifications :</b><br/><br/></p><p>- B.E. or Masters degree in Computer Science, Information Systems, Computer Engineering, or Systems Engineering.<br/><br/></p><p>- 4+ years of relevant professional experience in data engineering or related fields.<br/><br/></p><p>- At least 2 years of hands-on experience with Databricks, Spark, Databricks workflows, and Autoloader.<br/><br/></p><p>- Proficiency in additional scripting languages such as Scala and JavaScript, as well as experience with API development.<br/><br/></p><p>- Familiarity with data orchestration and ETL tools, including Alteryx, Snowflake, and Azure Data Factory.<br/><br/></p><p>- Experience with business intelligence tools such as Tableau, Power BI, or QuickSight.</p><br/></p> (ref:hirist.tech)