ETL Data Engineer - Python/PySpark

<p><p><b>Key Responsibilities :</b><br/><br/>- Design, develop, and maintain robust data pipelines and ETL/ELT workflows using PySpark, Python, and SQL.<br/><br/>- Build and manage data ingestion and transformation processes from various sources including Hive, Kafka, and cloud-native services.<br/><br/>- Orchestrate workflows using Apache Airflow and ensure timely and reliable data delivery.<br/><br/>- Work with large-scale big data systems to process structured and unstructured datasets.<br/><br/>- Implement data quality checks, monitoring, and alerting mechanisms.<br/><br/>- Collaborate with cross-functional teams including data scientists, analysts, and product managers to understand data requirements.<br/><br/>- Optimize data processing for performance, scalability, and cost-efficiency.<br/><br/>- Ensure compliance with data governance, security, and privacy Skills & Qualifications : </b></p><p><br/></p><p>- 5+ years of experience in data engineering or related roles.</p><br/>- Strong programming skills in Python and PySpark.<br/><br/>- Proficiency in SQL and experience with Hive.<br/><br/>- Hands-on experience with Apache Airflow for workflow orchestration.<br/><br/>- Experience with Kafka for real-time data streaming.<br/><br/>- Solid understanding of big data ecosystems and distributed computing.<br/><br/>- Experience with GCP (BigQuery, Dataflow, Dataproc)<br/><br/>- Ability to work with both structured (e.g., relational databases) and unstructured (e.g., logs, images, documents) data.<br/><br/>- Familiarity with CI/CD tools and version control systems (e.g., Git).<br/><br/>- Knowledge of containerization (Docker) and orchestration (Kubernetes).<br/><br/>- Exposure to data cataloging and governance tools (e.g., AWS Lake Formation, Google Data Catalog).<br/><br/>- Understanding of data modeling and architecture principles.</p><br/></p> (ref:hirist.tech)