AWS Data Engineer - Python/SQL/ETL

<p><p><b>Profile : </b> AWS Data Engineer <br/><br/><b>Mandate Skills : </b> AWS + Databricks + Pyspark + SQL role<br/><br/><b>Location</b> : Bangalore /Pune /Hyderabad /Chennai /Gurgaon<br/><br/><b>Notice Period : </b> Immediate<br/><br/><b>Key Responsibilities : </b><br/><br/>- Design, build, and maintain scalable data pipelines to collect, process, and store data from multiple datasets<br/><br/>- Optimize data storage solutions for performance, scalability, and cost-efficiency<br/><br/>- Develop and manage ETL/ELT processes with schema transformations and data slicing/dicing<br/><br/>- Collaborate with cross-functional teams to understand requirements and accelerate feature development<br/><br/>- Create curated datasets for downstream consumption and end-user reporting<br/><br/>- Automate deployment and CI/CD processes using GitHub workflows<br/><br/>- Ensure compliance with data governance, privacy regulations, and security protocols<br/><br/>- Work with AWS cloud services and Databricks for data processing with S3 storage<br/><br/>- Utilize big data technologies : Spark, SQL, Delta Lake<br/><br/>- Integrate SFTP for secure data transfer from Databricks to remote locations<br/><br/>- Analyze Spark query execution plans for performance optimization<br/><br/>- Troubleshoot issues in large-scale distributed systems<br/><br/><b>Required Skills : </b><br/><br/>- 3+ years AWS experience (S3, EMR, Glue, Lambda, Redshift)<br/><br/>- Proficiency in Spark, SQL, Delta Lake, and Databricks<br/><br/>- Strong Python/Scala programming skills<br/><br/>- Experience with ETL/ELT processes and data pipelines<br/><br/>- Git/GitHub workflows and CI/CD automation<br/><br/>- Knowledge of data governance and security protocols<br/><br/>- Problem-solving skills in distributed systems<br/><br/><b>Preferred : </b><br/><br/>- AWS certifications<br/><br/>- Real-time data processing experience<br/><br/>- Bachelor's in Computer Science or related field</p><br/></p> (ref:hirist.tech)