Job Responsibilities:
- Design, develop, and implement robust and scalable data pipelines using Azure Data Factory (ADF).
- Efficiently extract, transform, and load (ETL) data from various disparate sources into Azure Data Lake Storage (ADLS).
- Utilize Azure Databricks for advanced data processing, transformation, and analytics.
- Develop complex data solutions and scripts primarily using Python and PySpark.
- Collaborate with data scientists, analysts, and other engineering teams to understand data requirements and deliver high-quality data solutions.
- Monitor, optimize, and troubleshoot data pipelines to ensure performance, reliability, and data integrity.
- Implement best practices for data governance, security, and quality within the Azure ecosystem.
Required Skills:
- Proficiency in designing, developing, and implementing data pipelines using Azure Data Factory.
- Strong experience in extracting, transforming, and loading data from various sources into Azure Data Lake Storage (ADLS).
Good to have skills :
- Hands-on experience with Azure Databricks.
- Strong programming skills in Python.
- Expertise in PySpark for big data processing.
Skills Required
Azure Data Factory, Azure Databricks, Python, Pyspark