Big Data Engineer - Python/PySpark

<p><p><b>Key Responsibilities :</b><br/><br/></p><p>- Design, develop, and maintain scalable ETL/ELT pipelines using Python and PySpark.<br/><br/></p><p>- Work with large-scale datasets across distributed systems like HDFS, Hive, and Spark.<br/><br/></p><p>- Optimize performance of data ingestion and transformation processes using Delta Lake and Apache Iceberg.<br/><br/></p><p>- Implement and manage file formats such as Avro, ORC, and Parquet.<br/><br/></p><p>- Design external tables and optimize partitioning strategies in Hive/Databricks.<br/><br/></p><p>- Perform DML operations across distributed data platforms ensuring consistency and performance.<br/><br/></p><p>- Collaborate with data architects, analysts, and stakeholders to ensure high-quality data Skills :</b></p><p><br/></p>- Strong programming skills in Python and PySpark.<br/><br/></p><p>- Deep knowledge of HDFS, Hive, and Spark internals and architecture.<br/><br/></p><p>- Proficiency in working with Delta Lake and Apache Iceberg.<br/><br/></p><p>- Hands-on experience with file formats: Avro, ORC, Parquet.<br/><br/></p><p>- Expertise in ETL pipeline design, data ingestion, and performance tuning.<br/><br/></p><p>- Knowledge of Hive Metastore, external tables, and partition strategies.<br/><br/></p><p>- Experience in performing DML operations over distributed data to Have :</b></p><p><br/></p>- Hands-on experience with SAS SPDE tables and migration strategies.<br/><br/></p><p>- Familiarity with Starburst, Databricks, or similar query engines/platforms.<br/><br/></p><p>- Understanding of modern data lakehouse architectures and query optimizations.<br/><br/></p><p>- Exposure to cloud data platforms (AWS, Azure, or GCP) is a plus.</p><p><b><br/></b></p><p><p><b>Why Join Us :</b></p><p><br/></p>- Opportunity to work on cutting-edge data platforms and tools.<br/><br/></p><p>- Flexible hybrid work model based in Hyderabad.<br/><br/></p><p>- High-impact role in a data-driven enterprise environment.<br/><br/></p><p>- Competitive contract-to-hire opportunity with long-term growth prospects.</p><br/></p> (ref:hirist.tech)