Principal Data Engineer Big Data Technologies

Job description

<p><p><b>Description :</b></p><br/><p><b>Title :</b> Principal Data Engineer</p><br/><p><b>Location :</b> Bangalore, Bhopal, Chennai, Gurugram, Hyderabad, Jaipur, Pune</p><br/><p><b>Mode of Operation :</b> Hybrid</p><br/><p><b>Employment Type :</b> Full-time Permanent</p><br/><p><b>Experience :</b> 10 -14 years</p><br/><p><b>About the Role :</b></p><br/><p>We are seeking an accomplished and highly technical Principal Data Engineer to join our emerging Data Warehouse team across multiple locations.

</p><p><br/></p><p>This role is a critical leadership position that requires deep expertise in cloud-native data architecture, advanced data modeling, and practical delivery using modern tools like Spark, Databricks, and dbt.

</p><p><br/></p><p>As a Principal Engineer, you will be relied upon to lead a team of senior engineers, define the technical roadmap, manage client interactions, and solve the most complex data challenges in building our innovative BNL Data Warehouse :</b></p><br/><p>- <b>Technical Leadership & Architecture :</b> Act as the technical authority in the Data Warehouse team, leading a team of 4-5 Senior Data Engineers and taking ownership of proposing, reviewing, and defining scalable, secure, and cost-efficient data architectures across cloud environments (GCP or AWS).</p><br/><p>- <b>Advanced Data Engineering :</b> Design, develop, and maintain high-performance, resilient data pipelines using Python, PySpark, and JVM languages, ensuring reliability and efficiency at petabyte scale.</p><br/><p>- <b>Modern Data Warehouse Development :</b> Lead the design and development of data warehouses, implementing advanced data models (e.g., Data Vault, Kimball) suitable for the business nature, utilizing modern cloud data warehouses, and ensuring data quality and governance.</p><br/><p>- <b>Orchestration & Workflow Management :</b> Possess high proficiency in architecting and implementing robust workflow management systems, with Airflow expertise being mandatory for defining complex ETL/ELT orchestration.</p><br/><p>- <b>Databricks and Transformation :</b> Drive data transformation strategy using Databricks and proficiency with dbt (data build tool) for managing and deploying analytical data transformations with software engineering rigor.</p><br/><p>- <b>Complex Problem Solving :</b> Solve the most <b>complex situations and data models</b>, ensuring optimal data organization and query performance for downstream analytics and marketing suites.</p><br/><p>- <b>Client Engagement & Strategy :</b> Interact directly with clients for requirement gathering, translating ambiguous business needs into clear, scalable technical solutions, and building a comprehensive, holistic roadmap for feature delivery.</p><br/><p>- <b>Quality & CI/CD :</b> Lead the integration of advanced CI/CD processes into the data pipeline lifecycle, with strong experience in building automated pipelines, experience with GitHub Actions, ArgoCD, or Jenkins is highly valued.</p><br/><p>- <b>Risk Management & Delivery :</b> Proactively identify technical and project risks and develop effective mitigation strategies to ensure the timely and high-quality delivery of features, always maintaining a focus on business value over engineering interest.</p><br/><p><b>Required Experience :</b></p><br/><p>- <b>Overall Experience :</b> 10+ years of total experience in the field of data engineering, with 4+ years experience in a technical leadership or team lead role.</p><br/><p>- <b>Cloud Native Expertise :</b> 5+ years of proven experience building and operating cloud-native, data-intensive applications on either GCP or AWS.</p><br/><p>- <b>Programming & Scripting :</b> 8+ years of hands-on, production experience using Python, JVM languages (Java/Scala), and Shell Scripting.</p><br/><p>- <b>Big Data Processing :</b> Mandatory, hands-on experience building and maintaining high-volume Spark applications (using PySpark or Scala/Java) for large-scale data processing.</p><br/><p>- <b>Architecture & IaC :</b> Mandatory proficiency on an IaC tool such as Terraform, AWS CDK, or CloudFormation, and a proven ability to propose and review detailed cloud architecture.</p><br/><p>- <b>Data Lake Fundamentals :</b> Fundamental understanding of efficient file formats like Parquet, Delta Lake, and other Open Table Formats (OTFs).</p><br/><p>- <b>Testing and Automation :</b> Strong experience with unit testing and test automation frameworks specific to data pipelines and applications.</p><br/><p>- <b>Observability :</b> Experience implementing and utilizing observability systems (such as Datadog or Prometheus/Grafana) for pipeline health, monitoring, and alerting.</p><br/><p><b>Preferred Skills :</b></p><br/><p>- Experience with containerization and orchestration technologies (e.g., Docker, Kubernetes).</p><br/><p>- Prior experience with specialized marketing suite data integrations.</p><br/><p>- Experience implementing Data Mesh or other advanced data governance paradigms.</p><br/><p>- Excellent documentation skills with the ability to provide good quality, readable business documentation in addition to technical specifications.</p><br/><p>- Proactive and service-oriented approach when interacting with internal customers and stakeholders.</p><br/></p> (ref:hirist.tech)

Required Skill Profession

Computer Occupations