Job Description
<p><p><b>Description : </b><br/><br/>Job Title : Lead Data Consultant / Senior Data Engineer<br/><br/>Experience Level : 10+ Years (with deep Azure experience)<br/><br/>Employment Type : Full-time / Contract<br/><br/><b>Role Overview : </b><br/><br/>As a Lead Data Consultant / Senior Data Engineer, you will design, develop, and lead the delivery of enterprise-scale, cloud-native data platforms for our clients, with a particular focus on Azure, Databricks, and modern Lakehouse architectures.
Working under the direction of the Principal Data Consultant you will help shape go-to-market technical assets and production-ready solutions across data engineering, system integration, governance, and AI/ML-enablement.<br/><br/>This is a hands-on consulting role requiring both strong implementation expertise and the ability to influence design, compliance, and delivery decisions across diverse enterprise environmentsparticularly in Financial Services and Insurance (FSI).<br/><br/><b>Key Responsibilities : </b><br/><br/>- Architect and build Lakehouse solutions with bronze/silver/gold layers using Delta Lake and Databricks<br/><br/>- Implement scalable ETL/ELT pipelines using Azure Data Factory, Airflow, Databricks, and PySpark<br/><br/>- Define enterprise-level data architectures, including lineage, governance, and integration patterns<br/><br/>- Enable agent orchestration and validation pipelines to support dynamic, AIenabled processing flows<br/><br/>- Lead development of data ingestion, transformation, and orchestration frameworks across cloud-native environments<br/><br/>- Guide implementation of data governance and compliance standards specific to FSI, including lineage tracking and access controls<br/><br/>- Collaborate with cross-functional teams (platform, ML, API, compliance) to design integrated data + AI systems<br/><br/>- Develop CI/CD pipelines and IaC (Terraform) modules to automate provisioning and deployment of data infrastructure<br/><br/>- Mentor other engineers and ensure reusable, modular, well-documented assets are delivered<br/><br/><b>Technical Skills & Experience : </b><br/><br/><b>Data Platforms & Engineering : </b><br/><br/>- 10+ years of experience in data engineering, with deep Azure cloud experience<br/><br/>- Proven experience designing Lakehouse architectures and implementing bronze/silver/gold/curated layers<br/><br/>- Hands-on expertise with Databricks, Delta Lake, PySpark, and SQL<br/><br/>- Experience integrating Kafka, RDBMS, and unstructured data into cloud pipelines<br/><br/>- Knowledge of DLT-Meta frameworks, metadata-driven ELT development, and data mesh concepts<br/><br/>- Familiarity with GenAI, and its integration with structured/unstructured data<br/><br/><b>System Architecture : </b><br/><br/>- Ability to design systems spanning data layers, API integrations, and AI orchestration components<br/><br/>- Knowledge of agentic systems, validation workflows, and event-driven orchestration<br/><br/><b>Governance & Compliance : </b><br/><br/>- Strong understanding of data lineage, data quality, and access control models<br/><br/>- Experience applying FSI regulatory standards, including auditing and privacy best practices<br/><br/><b>Cloud & DevOps : </b><br/><br/>- Proficiency in Azure services, especially ADF, Synapse, Blob Storage, Key Vault, and Managed Identity<br/><br/>- CI/CD using GitHub, GitLab, or Azure DevOps, with automated deployment patterns<br/><br/>- Infrastructure automation using Terraform with Git-based workflows<br/><br/>- Development environments : VSCode, Python scripting, Git version control<br/><br/><b>Non-Technical & Consulting Skills : </b><br/><br/>- Agile delivery experience : refinement, estimation, MVP definition, backlog grooming<br/><br/>- Ability to translate technical solutions into high-level architecture artifacts and documentation (Markdown, Confluence, Lucidchart)<br/><br/>- Comfortable reviewing existing codebases and recommending paths for reuse or modernization<br/><br/>- Proven ability to mentor and unblock delivery teams, with experience guiding junior and mid-level engineers<br/><br/>- Strong communication skills for client interaction, executive presentations, and cross-team coordination</p><br/></p> (ref:hirist.tech)