Data Integration Engineer - Python Frameworks

<p><p><b>About the job</b><br/><br/>Designation : Data & Integration Engineer (Python/TypeScript, Azure, Integrations)<br/><br/>Experience : 4 -8 Years<br/><br/>Location : Cochin<br/><br/><b>Job Summary :</b><br/><br/>Build data pipelines (crawling/parsing, deduplication/delta, embeddings) and connect external systems and interfaces.<br/><br/><b>Key Responsibilities :</b><br/><br/>- Development of crawling/fetch pipelines (API-first; playwright/requests only where permitted)<br/><br/>- Parsing/normalization of job postings & CVs, deduplication/delta logic (seen hash, repost heuristics)<br/><br/>- Embeddings/similarity search (controlling Azure OpenAI, vector persistence in pgvector)<br/><br/>- Integrations: HR4YOU (API/webhooks/CSV import), SerpAPI, BA job board, email/SMTP<br/><br/>- Batch/stream processing (Azure Functions/container jobs), retry/backoff, dead-letter queues<br/><br/>- Telemetry for data quality (freshness, duplicate rate, coverage, cost per 1,000 items)<br/><br/>- Collaboration with FE for exports (CSV/Excel, presigned URLs) and admin configuration<br/><br/><b>Must Have Requirements :</b><br/><br/>- 4+ years of backend/data engineering experience<br/><br/>- Python (FastAPI, pydantic, httpx/requests, Playwright/Selenium), solid TypeScript for smaller services/SDKs<br/><br/>- Azure: Functions/Container Apps or AKS jobs, Storage/Blob, Key Vault, Monitor/Log Analytics<br/><br/>- Messaging: Service Bus/Queues, idempotence & exactly-once semantics, pragmatic approach<br/><br/>- Databases: PostgreSQL, pgvector, query design & performance tuning<br/><br/>- Clean ETL/ELT patterns, testability (pytest), observability (OpenTelemetry)<br/><br/><b>Nice-to-have :</b><br/><br/>- NLP/IE experience (spaCy/regex/rapidfuzz), document parsing (pdfminer/textract)<br/><br/>- Experience with license/ToS-compliant data retrieval, captcha/anti-bot strategies (legally compliant)<br/><br/>- Working method: API-first, clean code, trunk-based development, mandatory code reviews<br/><br/>- Tools/stack: GitHub, GitHub Actions/Azure DevOps, Docker, pnpm/Turborepo (Monorepo), Jira/Linear, Notion/Confluence<br/><br/>- On-call/support: rotating, "you build it, you run it"</p><br/></p> (ref:hirist.tech)