<p><p><b>Description : </b> Big Data Functions : </b> </p><p><br/></p><p>- Design and implement data pipelines for migration from HDFS/Hive to cloud object storage (e.g., S3, Ceph).
</p><p><br/></p><p>- Optimize Spark (and optionally Flink) jobs for performance and scalability in a Kubernetes environment.</p><p><br/></p><p>- Ensure data consistency, schema evolution, and governance with Apache Iceberg or equivalent table formats.
</p><p><br/></p><p>- Support migration strategy definition by providing technical input and identifying risks.
</p><p><br/></p><p>- Mentor junior developers and review their code / design decisions.</p><p><br/></p><p>- Collaborate with platform engineers, cloud architects, and product stakeholders to align technical implementation with project goals.
</p><p><br/></p><p>- Troubleshoot complex distributed system issues in data pipelines or storage : </b> </p><p><br/></p><p>- Experience 7 to 12 Years.
</p><p><br/></p><p>- Scala and Python.<br/>- Apache Spark (batch & streaming) must! </p><p><br/></p><p>- Deep knowledge of HDFS internals and migration strategies.
</p><p><br/></p><p>- Experience with Apache Iceberg (or similar table formats like Delta Lake / Apache Hudi) for schema evolution, ACID transactions, and time travel.
</p><p><br/></p><p>- Running Spark and/or Flink jobs on Kubernetes (e.g., Spark-on-K8s operator, Flink-on-K8s).
</p><p><br/></p><p>- Experience with distributed blob storages like Ceph or AWS S3 and similar.
</p><p><br/></p><p>- Building ingestion, transformation, and enrichment pipelines for large-scale datasets.
</p><p><br/></p><p>- Infrastructure-as-Code (Terraform, Helm) for provisioning data infrastructure.
</p><p><br/></p><p>- Ability to work independently while guiding be a plus : </b> </p><p><br/></p></p><p>- Experience with Apache Flink.
</p><p><br/></p><p>- Prior experience in migration projects or large-scale data platform modernization.
</p><p><br/></p><p>- Apple experience preferred (to enable him/her to get up to speed on our tooling set quickly and more independently).<br/></p><br/></p> (ref:hirist.tech)