<p><p><b>Description :</b><br/><br/><b>Responsibilities :</b></p><p><p><b><br/></b></p>- Building PB Scale Data Pipelines : Building highly performant, large-scale data infrastructure that can scale to 100K+ jobs and handle PB-scale data per day.<br/><br/></p><p>- Cloud-Native Data Infrastructure : Design and implement a robust, scalable data infrastructure on AWS, utilizing Kubernetes and Airflow for efficient resource management and deployment.<br/><br/></p><p>- Intelligent SQL Ecosystem : Design and develop a comprehensive SQL intelligence system encompassing query optimization, dynamic pipeline generation, and data lineage tracking.
</p><p><br/></p><p>- Leverage your expertise in SQL query profiling, AST analysis, and parsing to create a sophisticated engine focused on query performance improvements, building adaptive data pipelines, and implementing granular column-level lineage.<br/><br/><b>Requirements :</b></p><p><p><b><br/></b></p>- Experience in data engineering, with a focus on building scalable data pipelines and systems.<br/><br/></p><p>- Strong proficiency in Python and SQL.<br/><br/></p><p>- Extensive experience with SQL query profiling, optimization, and performance tuning, preferably with Snowflake.<br/><br/></p><p>- Deep understanding of SQL Abstract Syntax Tree (AST) and experience working with SQL parsers (e.
g., sqlglot) for generating column-level lineage and dynamic ETLs.<br/><br/></p><p>- Experience in building data pipelines using Airflow or dbt.<br/><br/></p><p>- [Optional] Solid understanding of cloud platforms, particularly AWS.<br/><br/></p><p>- [Optional] Familiarity with Kubernetes (K8S) for containerized deployments.</p><br/></p> (ref:hirist.tech)