Job Description
<p><p><b>Data Engineer (Azure Data Factory and Azure Databricks)</b><br/><br/><b>Job Description :</b><br/><br/>We seek a Data Engineer with at least 5 years of experience in Azure Databricks, Azure Data Lake Storage gen2 (ADLS), and Azure Data Factory (ADF).<br/><br/>The candidate will design, build, and maintain data pipelines using Azure ADF/Databricks.<br/><br/>The candidate should be able to create Data Modelling & Governance, they will also work closely with our data science and engineering teams to develop and deploy machine learning models and other advanced analytics solutions.<br/><br/><b>Primary skills :</b> Azure Data Factory, Azure Data Bricks, SQL, ADLS, Spark-SQL, Python (Pandas), Pyspark or Scala.<br/><br/><b>Secondary Skills :</b> Basics of Azure Security (RBAC, Azure AD), Hadoop, HDFS, ADLS, Azure DBFS, PowerBI, and Tableau visualization tool is a plus.<br/><br/><b>Responsibilities :</b><br/><br/>- Design, build, and maintain data pipelines using Azure ADF/Databricks.<br/><br/>- Strong expertise in Azure Data Factory, Databricks, and related technologies such as Azure Synapse Analytics and Azure Functions.<br/><br/>- Hands-on experience in designing and implementing data pipelines and workflows in Azure Data Factory and Databricks.<br/><br/>- Experience in Data Extraction, Transformation, and Loading of data from multiple data sources into target databases, using Azure Databricks, Spark SQL, Pyspark, and Azure SQL.<br/><br/>- Designing and implementing data ingestion pipelines from multiple sources using Azure Databricks.<br/><br/>- Sound working experience in Cleansing, Transformation, Business Logic, Incremental Transformation of Data, and merging the data with datamart tables,.<br/><br/>- Developing scalable and reusable frameworks for ingesting data sets.<br/><br/>- Integrating the end-to-end data pipeline to take data from source systems to target data repositories ensuring the quality and consistency of data is maintained at all times.<br/><br/>- Interacting with stakeholders and leaders to understand business goals and data requirements.<br/><br/>- Experience in working with Agile (Scrum, Sprint) and Waterfall Methodologies.<br/><br/>- Collaborate with data engineers, data architects, and business analysts to understand and translate business requirements into technical designs.<br/><br/>- Provide technical guidance and support to junior team members.<br/><br/>- Design, develop, and maintain SQL databases, including creating database schemas, tables, views, stored procedures, and triggers.<br/><br/>- Self-starter and team player with excellent communication, problem-solving skills, and interpersonal skills, and a good aptitude for learning.</p><br/></p> (ref:hirist.tech)