Big Data Developer

<p><p><b>Description : </b> Big Data Developer<br/><br/><b>Location : </b> Pune, India<br/><br/><b>Experience : </b> 7+ Years<br/><br/><b>About the Role : </b><br/><br/>We are seeking a highly experienced and technically proficient Big Data Developer to join our team in Pune.

In this role, you will be responsible for designing, developing, and optimizing large-scale data processing pipelines and platforms.

The ideal candidate has a deep background in the modern data ecosystem, with expertise in Python, PySpark, and cloud-based data analytics services, and a proven ability to solve complex data transformation challenges.<br/><br/><b>Key Responsibilities : </b><br/><br/>- Big Data Development : Design, build, and maintain robust, scalable, and highly performant data pipelines using Python and PySpark.<br/><br/>- Data Architecture : Work extensively with core Big Data technologies, including Hadoop and Apache Spark, to process vast datasets.<br/><br/>- Platform Expertise : Leverage deep knowledge of the Databricks platform to manage and execute data workloads efficiently.<br/><br/>- Data Structuring : Implement best practices for data storage and querying, with extensive, hands-on experience utilizing Delta Tables and working with various file formats such as JSON and Parquet.<br/><br/>- Cloud Integration : Develop and deploy solutions leveraging AWS data analytics services.<br/><br/>- Problem Solving : Tackle and resolve complex data processing and transformation-related problems, ensuring data quality, accuracy, and performance.<br/><br/>- Database Management : Apply solid knowledge of both NoSQL and RDBMS databases in data ingestion and integration workflows.<br/><br/>- Collaboration : Utilize good communication skills to effectively collaborate with cross-functional teams, including data scientists, analysts, and other engineering teams.<br/><br/><b>Required Qualifications : </b><br/><br/>- Experience : 7+ years of professional experience as a Big Data Engineer or in a similar role.<br/><br/>- Programming Proficiency : Must be proficient with Python & PySpark.<br/><br/>- Core Big Data : In-depth, practical knowledge of Hadoop and Apache Spark.<br/><br/>- Data Lake/Warehouse Technologies : Extensive hands-on experience with Databricks and deep familiarity with Delta Tables.<br/><br/>- File Formats : Must have extensive experience working with and optimizing data stored in JSON and Parquet file formats.<br/><br/>- Database Knowledge : Solid understanding and working knowledge of both NoSQL and RDBMS databases.<br/><br/>- Analytical Skills : Proven ability to analyze and solve complex data processing, transformation, and optimization Qualifications : </b><br/><br/>- Cloud Data Services : Experience with AWS data analytics services such as Athena, Glue, Redshift, and EMR.<br/><br/>- Data Warehousing : Familiarity and practical experience with Data Warehousing concepts and methodologies will be a plus.</p><br/></p> (ref:hirist.tech)