Data Engineer - RAG/Python

<p><p><b>Description :</b><br/><br/></p><p><br/>- Role: Data Engineer (AI Knowledge Base Platform)<br/><br/>- Experience: 4 years to 7 years<br/><br/>- Work location: Anywhere in India (Remote)<br/><br/>- Notice period: Immediate to 15 days<br/><br/>- Shift timing: 3:00 PM to 12:00 AM (IST)<br/><br/>We are seeking a Data Engineer to design and maintain the ingestion, processing, and transformation pipelines that feed AI knowledge bases on GCP.<br/><br/><b>Required Skills & Qualifications :</b><br/><br/>- 4+ years of Data engineering experience<br/><br/>- Strong programming skills in Python and SQL<br/><br/>- Proficiency with GCP data services: BigQuery, Cloud Storage, Dataflow, Pub/Sub, Composer/Airflow<br/><br/>- Experience with ETL/ELT pipeline design and workflow orchestration<br/><br/>- Familiarity with vector databases/embeddings (e.g., Pinecone, FAISS, Vertex AI Matching Engine)<br/><br/>- Knowledge of data modeling, governance, and version control<br/><br/>- Experience with CI/CD, Docker, and Kubernetes for data pipeline deployment<br/><br/>- Strong analytical and debugging skills<br/><br/><b>Key Responsibilities :</b><br/><br/>- Design and implement data ingestion pipelines from GCS, BigQuery, and Google Drive<br/><br/>- Build workflows for knowledge base indexing/vectorization (RAG pipelines)<br/><br/>- Implement incremental updates, targeted revisions, and full rebuild options for AI agents<br/><br/>- Collaborate with ML Engineers to integrate data pipelines with Vertex AI models<br/><br/>- Optimize pipelines for cost, performance, and scalability within GCP<br/><br/>- Ensure data quality, governance, and security with IAM-based access control<br/><br/>- Maintain logging, monitoring, and alerting for pipeline health<br/><br/>- Work with DevOps to enable self-healing and rollback automation<br/></p><br/></p> (ref:hirist.tech)