YASH Technologies Data Engineer Hadoop Platform

Job description

<p><p><b>Job Description :</b></p><br/><p><b>Position Title :</b> Data Engineer</p><br/><p><b>SCOPE OF RESPONSIBILITY :</b></p><br/><p>As part of a global, growing team of data engineers, you will collaborate in a DevOps model to enable Clients Life Science business with cutting-edge technology, leveraging data as an asset to support better decision-making.

</p><p><br/></p><p>You will design, develop, test, and support automated end-to-end data pipelines and applications within Life Sciences data management and analytics platform (Palantir Foundry, Hadoop, and other components).

</p><p><br/></p><p>This position requires proficiency in data engineering, distributed computation, and DevOps methodologies, utilizing AWS infrastructure and on-premises data centers to support multiple technology stacks.</p><br/><p><b>PURPOSE OF THE POSITION :</b></p><br/><p>The purpose of this role is to build and maintain data pipelines, develop applications on various platforms, and support data-driven decision-making processes across clients Life Science business.

</p><p><br/></p><p>You will work closely with cross-functional teams, including business users, data scientists, and data analysts, while ensuring the best balance between technical feasibility and business :</b></p><br/><p>- Develop data pipelines by ingesting various structured and unstructured data sources into Palantir Foundry.</p><br/><p>- Participate in end-to-end project lifecycles, from requirements analysis to deployment and operations.</p><br/><p>- Act as a business analyst for developing requirements related to Foundry pipelines.</p><br/><p>- Review code developed by other data engineers, ensuring adherence to platform standards and functional specifications.</p><br/><p>- Document technical work professionally and create high-quality technical documentation.</p><br/><p>- Balance technical feasibility with strict business requirements.</p><br/><p>- Deploy applications on Foundry platform infrastructure with clearly defined checks.</p><br/><p>- Implement changes and bug fixes following clients change management framework.</p><br/><p>- Work in DevOps project setups following Agile principles (e.g., Scrum).</p><br/><p>- Act as third-level support for critical applications, resolving complex incidents and debugging problems across the full stack.</p><br/><p>- Work closely with business users, data scientists, and analysts to design physical data models.</p><br/><p>- Provide support in designing ETL/ELT processes with databases and Hadoop platforms.</p><br/><p><b>EDUCATION :</b></p><br/><p>Bachelors degree or higher in Computer Science, Engineering, Mathematics, Physical Sciences, or related fields.</p><br/><p><b>EXPERIENCE :</b></p><br/><p>- 5+ years of experience in system engineering or software development.</p><br/><p>- 3+ years of experience in engineering with a focus on ETL work involving databases and Hadoop platforms.</p><br/><p><b>TECHNICAL SKILLS :</b></p><br/><p><b>- Hadoop General :</b> Deep knowledge of distributed file system concepts, map-reduce principles, and distributed computing.

Familiarity with Spark and its differences from MapReduce.</p><br/><p><b>- Data Management :</b> Proficient in technical data management tasks such as reading, transforming, and storing data, including experience with XML/JSON and REST APIs.</p><br/><p><b>- Spark :</b> Experience in launching Spark jobs in both client and cluster modes, with an understanding of property settings that impact performance.</p><br/><p><b>- Application Development :</b> Familiarity with HTML, CSS, JavaScript, and basic visual design competencies.</p><br/><p><b>- SCC/Git :</b> Experienced in using source code control systems like Git.</p><br/><p><b>- ETL/ELT :</b> Experience developing ETL/ELT processes, including loading data from enterprise-level RDBMS systems (e.g., Oracle, DB2, MySQL).</p><br/><p><b>- Authorization :</b> Basic understanding of user authorization, preferably with Apache Ranger.</p><br/><p><b>- Programming :</b> Proficient in Python, with expertise in at least one high-level language (e.g., Java, C, Scala).

Must have experience using REST APIs.</p><br/><p><b>- SQL :</b> Expertise in SQL for manipulating database data, including views, functions, stored procedures, and exception handling.</p><br/><p><b>- AWS :</b> General knowledge of the AWS stack (EC2, S3, EBS, etc.).</p><br/><p><b>- IT Process Compliance :</b> Experience with SDLC processes, change control, and ITIL (incident, problem, and change management).</p><br/><p><b>REQUIRED SKILLS :</b></p><br/><p>- Strong problem-solving skills with an analytical mindset.</p><br/><p>- Excellent communication skills to collaborate with both technical and non-technical teams.</p><br/><p>- Experience working in Agile/DevOps teams, utilizing Scrum principles.</p><br/><p>- Ability to thrive in a fast-paced, dynamic environment while managing multiple tasks.</p><br/><p>- Strong organizational skills with attention to detail.</p><br/></p> (ref:hirist.tech)

Required Skill Profession

Computer Occupations