Data Engineer - Scala/Python

<p><p><b>Roles & Responsibilities : </b><br/><br/>- Lead the design, development, and implementation of scalable big data solutions using Azure Databricks, Spark, and Azure Cloud services.<br/><br/>- Architect end-to-end data pipelines for ingestion, transformation, and storage of structured and unstructured data.<br/><br/>- Collaborate with Data Engineers, Data Scientists, and Business Analysts to translate business requirements into technical solutions.<br/><br/>- Optimize Spark jobs, Databricks clusters, and workflows for performance, cost, and scalability.<br/><br/>- Implement data governance, security, and compliance standards in cloud data platforms.<br/><br/>- Mentor junior team members and provide technical guidance across the team.<br/><br/>- Evaluate and recommend new tools, frameworks, and best practices for big data processing and cloud analytics.<br/><br/><b>Required Skills : </b><br/><br/>- Strong experience with Azure Databricks, PySpark, and Spark SQL.<br/><br/>- Hands-on experience with Azure Data Lake, Azure Synapse, Azure Data Factory, and other Azure cloud services.<br/><br/>- Expertise in data warehousing concepts, ETL processes, and big data architecture.<br/><br/>- Proficient in programming languages : Python, Scala, or Java.<br/><br/>- Knowledge of CI/CD pipelines, version control, and Agile methodology.<br/><br/>- Experience in performance tuning, cluster management, and job scheduling.<br/><br/>- Excellent problem-solving, communication, and leadership skills.<br/><br/><b>Desired Skills : </b><br/><br/>- Familiarity with Snowflake, Databricks Delta Lake, or Apache Kafka is a plus.<br/><br/>- Certification in Azure Data Engineering / Databricks is desirable</p><br/></p> (ref:hirist.tech)