Capco - Senior Data Engineer - Python/Snowflake

<p><p><b>Job Summary : </b><br/><br/>We are seeking a highly experienced Senior Data Engineer with 9+ years of experience, specializing in building advanced Data Engineering solutions on the Snowflake/AI Cloud platform utilizing AWS.

The role demands a minimum of 7+ years of expertise in Python development and over 1 year of experience in building Snowflake applications using the Snowflake AI/Cortex platform (Cortex Agents, Search, and LLM).

The engineer will be responsible for end-to-end data pipeline development, sophisticated data governance, cloud deployment, and implementing context enrichment techniques like RAG for LLM integration.<br/><br/><b>Job Description : </b><br/><br/><b>Python Development and Engineering Excellence : </b><br/><br/>- Apply 7+ years of experience in developing Data Engineering and data science projects using advanced Python development techniques.<br/><br/>- Demonstrate a deep understanding of implementing Object-Oriented Programming (OOP) in Python, leveraging advanced data structures like Pandas data frames to write clean and maintainable engineering code.<br/><br/>- Implement Object-Relational Mapping (ORM) in Python using frameworks like SQLAlchemy or equivalent for robust database interactions.<br/><br/>- Possess a technical understanding of multi-threading concepts and concurrency implementation using Python server-side custom modules to enhance application performance.<br/><br/>- Utilize modern development tools, including VS Code or Anaconda, for development and proficiently use Git or Bitbucket for version control and utilize standard Python unit testing frameworks.<br/><br/><b>Snowflake AI/Cortex and Data Modeling : </b><br/><br/>- Possess mandatory 1+ years of experience in building Snowflake applications utilizing the Snowflake AI/Cortex platform, specifically focusing on Cortex Agents, Cortex Search, and Cortex LLM.<br/><br/>- Demonstrate a strong understanding of context enrichment techniques for LLMs within Snowflake, specifically using Prompts or Retrieval-Augmented-Generation (RAG) methods.<br/><br/>- Good understanding of input query enrichment using Snowflake YAMLs and integrating them with LLMs within Snowflake.<br/><br/>- Apply required experience with different data modeling techniques necessary for designing efficient and scalable data warehouses within Snowflake.<br/><br/>- Demonstrate a good understanding of Relevance search principles and the process of building custom interaction applications with LLMs.<br/><br/><b>Snowflake Governance and Data Management : </b><br/><br/>- Possess a good understanding of Snowflake Account hierarchy models and best practices for defining a robust Account-role-permissions strategy.<br/><br/>- Demonstrate proficiency in Data Governance/Security concepts within Snowflake, including implementing Row/Column level dynamic data masking using Snowflake Tags.<br/><br/>- Possess good technical knowledge of Data sharing, preferably utilizing the Internal Data Marketplace and Data Exchanges for various data listings.<br/><br/>- Demonstrate good skills in developing applications using Snowpipe and Snowpark, effectively moving and processing data from Cloud sources like AWS S3 and handling unstructured data from data lakes.<br/><br/><b>Cloud Deployment and CI/CD : </b><br/><br/>- Exhibit proficiency in developing and deploying Python applications like Lambda on the AWS Cloud platform.<br/><br/>- Demonstrate strong skills in deploying web applications on AWS Cloud using Docker containers or Kubernetes, including practical experience utilizing CI/CD pipelines.<br/><br/>- Work effectively using Agile and SCRUM methodologies, preferably leveraging JIRA for task management.<br/><br/>- Be dedicated to continuously improving functionality through experimentation, performance tuning, and incorporating customer feedback.<br/><br/><b>Required Skills & Qualifications : </b><br/><br/>- Experience : Mandatory 9+ years total experience, with 7+ years in Python development/Data Engineering.<br/><br/>- Snowflake AI : 1+ years of experience in building applications using Snowflake AI/Cortex platform (Agents, Search, LLM, RAG).<br/><br/>- Core Python : Deep understanding of OOP in Python, data structures (Pandas), concurrency, and SQLAlchemy/ORM.<br/><br/>- Cloud/DevOps : Good experience deploying Python apps/web apps on AWS Cloud using Docker/Kubernetes and utilizing CI/CD pipelines.<br/><br/>- Data Governance : Good knowledge of Snowflake Account-role-permissions strategy, data sharing, and dynamic data masking/Snowflake Tags.<br/><br/>- Methodology : Experience using Agile and SCRUM methodologies and Git or Bitbucket for version control.<br/><br/>- Soft Skills : Good analytical skills, strong problem solving ability, and capability to communicate technical concepts clearly.<br/><br/><b>Preferred Skills : </b><br/><br/>- Experience with Snowpark.<br/><br/>- Experience in building Snowflake native applications using Streamlit and deploying them onto AWS Cloud instances.<br/><br/>- Experience in implementing CI/CD pipelines within Snowflake applications.<br/><br/>- Experience with any application Cache implementation within Python web applications.<br/><br/>- Experience with duckdb and Apache arrow.<br/></p><br/></p> (ref:hirist.tech)