<p><p><b>Job Title : </b> Data Engineer Azure Responsibilities : </b></p><p><br/></p>- Build scalable real-time and batch processing workflows using Azure Databricks, PySpark, and Apache Spark.<br/><br/></p><p>- Perform data pre-processing (cleaning, transformation, deduplication, normalization, encoding, scaling) to ensure high-quality inputs for analytics and AI/ML.<br/><br/></p><p>- Design and maintain cloud-based data architectures (data lakes, lakehouses, warehouses) following Medallion Architecture.<br/><br/></p><p>- Deploy and optimize data solutions on Azure (preferred), AWS, or GCP, focusing on performance, security, and scalability.<br/><br/></p><p>- Develop and optimize ETL/ELT pipelines for structured/unstructured data from IoT, MES, SCADA, LIMS, and ERP systems.<br/><br/></p><p>- Automate data workflows using CI/CD and DevOps best practices, ensuring compliance with security standards.<br/><br/></p><p>- Monitor, troubleshoot, and enhance pipelines for high availability and reliability.<br/><br/></p><p>- Utilize Docker and Kubernetes for scalable data processing environments.<br/><br/></p><p>- Collaborate with automation teams, data scientists, and engineers to provide clean, structured data for AI/ML Skills (Must-Have) : Programming : </b> PySpark, Apache Spark</p><p><b><br/></b></p><p><p><b>- Platforms : </b> Azure Databricks (preferred), AWS, or GCP</p><b><p><b><br/></b></p></b><p><b>- Data Management : </b> SQL, ETL/ELT development, data lakes, lakehouses, and DevOps/Automation : </b> CI/CD pipelines, Docker, Kubernetes</p><p><b><br/></b></p><p><b>- Architecture : </b> Medallion architecture, scalable data systems.
Experience with IoT, MES, SCADA, LIMS, ERP data : </b><br/><br/></p><p>- Bachelors/Masters degree in Computer Science, Data Engineering, or related field (IIT graduates only).<br/><br/></p><p>- Proven 4+ years experience as a Data Engineer in enterprise-scale environments.<br/><br/></p><p>- Strong problem-solving and communication skills with a collaborative mindset.</p><br/></p> (ref:hirist.tech)