Data Engineer - Azure Databricks

<p><p><b>Job Description :</b></p><p><b><br/></b></p><p><b>Key Responsibilities :</b></p><p><p><b><br/></b></p>- Design, build, and optimize data pipelines and workflows on the Databricks platform using Apache Spark.<br/><br/></p><p>- Develop scalable and efficient ETL/ELT processes to ingest, transform, and process large volumes of structured and unstructured data.<br/><br/></p><p>- Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and deliver solutions.<br/><br/></p><p>- Implement data quality checks, monitoring, and alerting to ensure pipeline reliability and accuracy.<br/><br/></p><p>- Optimize performance of data processing jobs and storage for cost-efficiency and scalability.<br/><br/></p><p>- Work closely with cloud engineering teams to deploy and maintain data infrastructure on cloud platforms such as Azure, AWS, or GCP.<br/><br/></p><p>- Develop and maintain technical documentation for data architecture and pipeline processes.<br/><br/></p><p>- Support data governance and compliance by implementing security and access controls.<br/><br/></p><p>- Troubleshoot and resolve data issues promptly to minimize Skills & Qualifications :</b></p><p><b><br/></b></p>- 5+ years of experience in data engineering or related roles.<br/><br/></p><p>- Strong hands-on experience with Databricks and Apache Spark (PySpark, Scala, or Spark SQL).<br/><br/></p><p>- Proficient in programming languages such as Python, Scala, or Java.<br/><br/></p><p>- Experience with cloud platforms (Azure, AWS, or GCP) and their data services (e.g., Azure Data Lake, S3, BigQuery).<br/><br/></p><p>- Solid understanding of ETL/ELT concepts, data modeling, and data warehousing principles.<br/><br/></p><p>- Experience working with large-scale datasets and distributed computing frameworks.<br/><br/></p><p>- Familiarity with SQL and NoSQL databases.<br/><br/></p><p>- Knowledge of DevOps practices including CI/CD pipelines and infrastructure as code.<br/><br/></p><p>- Strong problem-solving skills and ability to work in a fast-paced environment.<br/><br/></p><p>- Excellent communication and collaboration Qualifications :</b></p><p><b><br/></b></p>- Experience with workflow orchestration tools like Apache Airflow or Azure Data Factory.<br/><br/></p><p>- Knowledge of containerization (Docker, Kubernetes).<br/><br/></p><p>- Understanding of data governance, security, and compliance standards.<br/><br/></p><p>- Familiarity with machine learning pipelines and tools</p><br/></p> (ref:hirist.tech)