<p><p><b>About the job :</b><br/><br/>Location : Remote<br/><br/>Project start : ASAP<br/><br/>Workload : And Qualifications : </b></p><p><br/></p><p>- Advice on architectural discussions and workshops with customers, understand their business and technical requirements to create the desired technical architectures and solutions on Cloud around data engineering, data lakes, data lakehouses, BI and ML/AI.</p><p><br/>- Perform requirements scoping exercises with project and use case stakeholders.</p><br/>- Translate requirements into desired technical solution design.<br/><br/>- Carry out PoCs, prototypes and build MVPs for new innovative solutions and technology scouting on cloud (Azure) and Big Data Technologies.<br/><br/>- Participate in hands-on technical project work including actual project implementation tasks and some production support tasks (i.e. monitoring) as required.<br/><br/>- Evaluate and implement platform cost optimizations<br/><br/>- Create and maintain technical documentation for the use cases, solutions and data platform.<br/><br/>- Perform analysis of best practices and emerging concepts in Cloud based technologies with special focus on Data and Analytics cloud Musts :</b><br/><br/>- In-depth knowledge of Apache Spark and experience in optimizing and performance tuning of Apache Spark data processing jobs<br/><br/>- In-depth knowledge of Delta Lake<br/><br/>- In-depth knowledge of Data Engineering on (Azure) Databricks<br/><br/>- Strong hands-on experience in Python programming and in writing complex SQL queries<br/><br/>- Strong hands-on experience in building complex data pipelines in Azure Data Factory<br/><br/>- Strong hands-on experience in the following Azure Data Services : ADLS Gen2, Synapse Serverless<br/><br/>- Experience in architecting and building enterprise grade data platforms on Cloud and Developing Big Data Solution Architectures Preferably On Azure, Incl.
Gathering requirements and mapping those to technical architectures<br/><br/>- Awareness of best practices for selecting a component mix of Cloud services (i.e. ADF, Databricks, Synapse, etc.)<br/><br/>- Ability to assess pros and cons of architecture variations (i.e. Databricks vs Snowflake vs.
MS Fabric, Synapse vs MS Fabric Lakehouse, Databricks vs.
open-source Spark, Required : </b></p><p><br/></p><p>Multi-year experience working (>6 years) in a data engineer role, incl : </p><p><br/></p><p>- Expertise in designing, building and maintaining large scale data pipelines as well as processing (transforming, aggregating, wrangling) data.</p><p><br/>- Proficient with Streaming technologies like Kafka, Spark Structured Streaming or equivalent cloud services</p><br/>- Good know-how of cloud computing concepts and of Azure cloud platform, networking, security and monitoring aspects<br/><br/>- Hands-on experience in using and applying IAC, CI/CD and DevOps practices in real Data Analytics projects, preferably using Azure DevOps and Terraform.<br/><br/>- Knowledge of Microsoft Fabric will be added advantage.</p><p><br/></p><p>Skills : Azure, Cloud, Spark</p><br/></p> (ref:hirist.tech)