CGI - Azure Data Engineer - Python/Pandas

<p><p><b>Work location : </b>Pune (Working from office is mandatory IN office /Hybrid)<br/><br/><b>Experience : </b>Overall IT exp of 6 to 12 years experience, with relevant experience as Data Engineer 4 yrs & above<br/><br/>Join us in building scalable data pipelines using Python- Pandas/Polars, SQL, Airflow, and Azure DevOps.

If you love solving problems across diverse data sources (APIs, PDFs, web scraping) and working hands-on with pandas/polars, SQL, and test automation.<br/><br/><b>Tech stack :</b><br/><br/><b>Main/essential : </b>Python, Pandas and/or Polars Essential, Web Scraping, including using Selenium, SQL, Azure DevOps and Airflow<br/><b><br/>Additional :</b> Databricks, AWS, Jenkins, ADO Pipelines<br/><br/><b>Key Responsibilities :</b><br/><br/>- Design, build, and maintain pipelines in Python to collect data from a wide range of sources (APIs, SFTP servers, websites, emails, PDFs, etc.)<br/><br/>- Deploy and orchestrate workflows using Apache Airflow<br/><br/>- Perform web scraping using libraries like requests, BeautifulSoup, Selenium<br/><br/>- Handle structured, semi-structured, and unstructured data efficiently<br/><br/>- Transform datasets using pandas and/or polars<br/><br/>- Write unit and component tests using pytest<br/><br/>- Collaborate with platform teams to improve the data scraping framework<br/><br/>- Query and analyze data using SQL (PostgreSQL, MSSQL, Databricks)<br/><br/>- Conduct code reviews, support best practices, and improve coding standards across the team<br/><br/>- Manage and maintain CI/CD pipelines (Azure DevOps Pipelines, Jenkins)<br/><br/><b>Required Skills & Experience :</b><br/><br/>- Proficient in Python, with deep experience using pandas or polars<br/><br/>- Strong understanding of ETL development, data extraction, and transformation<br/><br/>- Hands-on experience with SQL and querying large datasets<br/><br/>- Experience deploying workflows on Apache Airflow<br/><br/>- Familiar with web scraping techniques (Selenium is a plus)<br/><br/>- Comfortable working with various data formats and large-scale datasets<br/><br/>- Experience with Azure DevOps, including pipeline configuration and automation<br/><br/>- Familiarity with pytest or equivalent test frameworks<br/><br/>- Strong communication skills and a team-first attitude.<br/><br/>- Experience with Databricks<br/><br/>- Familiarity with AWS services<br/><br/>- Working knowledge of Jenkins and advanced ADO Pipelines</p><br/></p> (ref:hirist.tech)