Senior Python Developer ETL & Data Pipeline Automation

Job description

<p><b>Role overview : </b><br/><br/>- Proficient in Python, with hands-on experience using Pandas for large dataset processing.<br/><br/>- Solid knowledge of ETL frameworks and pipeline automation strategies.<br/><br/>- Experience with Elasticsearch : indexing, querying, and optimizing for performance.<br/><br/>- Strong command of Hive SQL and working with distributed data systems.<br/><br/>- Experience building APIs using Flask or similar Python frameworks.<br/><br/>- Understanding of data modeling, schema design, and workflow orchestration.<br/><br/>- Strong debugging, analytical, and communication skills.<br/><br/><b>What would you do here : </b><br/><br/>- Design, develop, and automate scalable ETL workflows using Python and Pandas.<br/><br/>- Implement and manage data ingestion and transformation processes using Hive SQL for data warehousing.<br/><br/>- Configure, monitor, and optimize Elasticsearch clusters for indexing, searching, and analytics performance.<br/><br/>- Build and maintain RESTful APIs using Flask to facilitate secure data exchange.<br/><br/>- Collaborate with Data Scientists, Analysts, and Engineering teams to support data-driven applications and reporting tools.<br/><br/>- Ensure high standards of data quality, reliability, and integrity throughout the pipeline.<br/><br/>- Monitor performance, debug issues, and tune queries for maximum efficiency.<br/><br/>- Document data flows, pipeline architecture, and automation logic clearly.<br/><br/><b>Senior Level : </b><br/><br/><b>Required Skills & Experience : </b><br/><br/>- 8-12+ years of experience in professional Python development.

<br/><br/>- Proven expertise in designing and building backend systems and APIs at scale.

<br/><br/>- Strong experience with LLMs, GenAI integrations, Agentic workflows and AI-enhanced workflows.

<br/><br/>- Solid understanding of microservices architecture and distributed systems.

<br/><br/>- Proficiency in automated testing and observability best practices.

<br/><br/>- Hands-on experience building and maintaining CI/CD pipelines.

<br/><br/>- Strong problem-solving, communication, and stakeholder management skills.

<br/><br/><b>Must Have : <br/></b><br/>- Python, LLD/HLD, AI/LLM, Unit Testing, Code Reviews, CI/CD pipelines, Frameworks like Fast API, Database like Postgres, Monitoring and visualisation tools like Prometheus and Grafana, Team Leading<br/><br/><b>Key Responsibilities : </b><br/><br/><b>System Architecture & Design : </b><br/><br/>- Architect and develop secure, scalable backend systems using Python.

<br/><br/>- Lead high- and low-level design (HLD/LLD) with a focus on modularity, fault tolerance, and API best practices.

<br/><br/>- Champion performance optimization and drive observability improvements across services.

<br/><br/><b>AI/LLM Integration : </b><br/><br/>- Integrate Large Language Models (LLMs) and GenAI-powered features into core applications.

<br/><br/>- Build robust systems that mitigate hallucinations and improve AI reliability.

<br/><br/>- Leverage agentic frameworks and AI-driven automation for process enhancement <br/><br/>- Practical experience with OpenAI developer ecosystem, including APIs (Completions, Responses, Files) and tools/SDKs (Agents SDK, Web Search).

<br/><br/>- Familiarity with agent debugging/monitoring tools (Phoenix, LangSmith, or similar).

<br/><br/><b>Engineering Excellence : </b><br/><br/>- Define and enforce engineering best practices across testing, code reviews, and quality gates.

<br/><br/>- Ensure strong observability through logging, monitoring, and alerting tools.

<br/><br/>- Write unit, integration, and functional tests to support high-reliability systems.

<br/><br/><b>DevOps & Deployment : </b><br/><br/>- Build and manage CI/CD pipelines for automated testing and smooth deployments.

<br/><br/>- Ensure deployment processes support rollback, scalability, and system health.

<br/><br/><b>Collaboration & Leadership : </b><br/><br/>- Collaborate closely with product, data, business, and DevOps teams to align on solution delivery.

<br/><br/>- Mentor and support junior engineers, fostering a culture of continuous improvement and technical excellence.

<br/><br/><b>Python & Backend Engineering : </b><br/><br/>- Strong Python development skills with experience in frameworks like FastAPI <br/><br/>- Proven experience designing REST APIs, microservices, and backend architectures.

<br/><br/>- Experience integrating with databases (e.g., PostgreSQL) for real-time and batch workloads.

<br/><br/>- Experience working with data pipelines and ETL frameworks (e.g., jobs moving data from GCP - Postgres) and integrating this data into backend services or AI workflows.

</p> (ref:hirist.tech)

Required Skill Profession

Computer Occupations