Data Engineer Python/Spark

Job description

<p><b>About The Job : </b><br/><br/>- Develops technical tools and programming to cleanse, organize and transform data and to maintain, protect and update data structures and integrity on an automated basis.<br/><br/>- Applies data extraction, transformation, and loading techniques in order to tie together large data sets from a variety of sources.<br/><br/>- Partners with both internal & external sources to design, build and oversee the deployment and operation of technology architecture, solutions and software.<br/><br/>- Designs, develops and programs methods, processes and systems to capture, manage, store and utilize structured and unstructured data to generate actionable insights and solutions.<br/><br/>- Responsible for the maintenance, improvement, cleaning, and manipulation of data in the business client's operational and analytics databases.<br/><br/>- Proactively analyzes and evaluates the business client's databases in order to identify and recommend improvements and optimization.<br/><br/><b>Essential Job Functions : </b><br/><br/>- Uses knowledge of existing and emerging data science engineering principles, theories, and techniques to inform business decisions; and produce accurate business insights.<br/><br/>- Completes projects and assignments of moderate scope and complexity under normal supervision to ensure customer and business needs are met.<br/><br/>- Applies discretion and independent judgement to interpret data trends and summarize data insights.<br/><br/>- Assists in the preliminary data exploration, data preparation for accurate model development.<br/><br/>- Establishes working relationships with others outside area of Data Science Engineering expertise.<br/><br/>- Prepares presentations of project outputs for external customers with assistance.<br/><br/>- Design, develop, and maintain scalable data pipelines and systems for data processing.<br/><br/>- Utilize Data Lakehouse, Spark on Kubernetes and related technologies to manage large-scale data processing.<br/><br/>- Perform data ingestion from various sources like API's, RDBMS, NoSQL DB's, Kafka, Middleware & Files using Spark and process data into Lakehouse platform.<br/><br/>- Develop and maintain py-spark scripts for automation of data processing tasks.<br/><br/>- Implement full and incremental data loading strategies to ensure data consistency and availability.<br/><br/>- Orchestrate and monitor workflows using Apache Airflow.<br/><br/>- Ensure code quality and version control using GIT.<br/><br/>- Troubleshoot and resolve data-related issues in a timely manner.<br/><br/>- Stay up-to-date with the latest industry trends and technologies to continuously improve our data infrastructure.<br/><br/>- Design, develop, and maintain scalable data pipelines and systems for data processing.<br/><br/>- Utilize Data Lakehouse, Spark on Kubernetes and related technologies to manage large-scale data processing.<br/><br/>- Perform data ingestion from various sources like API's, RDBMS, NoSQL DB's, Kafka, Middleware &<br/><br/>- Files using Spark and process data into Lakehouse platform.<br/><br/>- Develop and maintain py-spark scripts for automation of data processing tasks.<br/><br/>- Implement full and incremental data loading strategies to ensure data consistency and availability.<br/><br/>- Orchestrate and monitor workflows using Apache Airflow.<br/><br/>- Ensure code quality and version control using GIT.<br/><br/>- Troubleshoot and resolve data-related issues in a timely manner.<br/><br/>- Stay up-to-date with the latest industry trends and technologies to continuously improve our data : </b><br/><br/>- Proven experience as a Data Engineer (ETL, data warehousing, data Lakehouse).<br/><br/>- Strong knowledge of Spark on Kubernetes, S3 and Docker Images.<br/><br/>- Proficiency in Data engineering techniques with Py-spark.<br/><br/>- Strong experience in Data warehousing techniques like data mining, data analysis, data profiling.<br/><br/>- Experience with Python scripting for automation.<br/><br/>- Expertise in full and incremental data loading techniques.<br/><br/>- Excellent problem-solving skills and attention to detail.<br/><br/>- Ability to work collaboratively in a team environment and communicate effectively with stakeholders.<br/><br/><b>Good to have :</b><br/><br/>- Understanding of streaming data applications using.<br/><br/>- Hands-on experience with Apache Airflow for workflow orchestration.<br/><br/>- Proficiency with GIT for version control<br/><br/>- Understanding of data engineering integration with LLMs or GEN-AI applications and Vector DB.<br/><br/>- Knowledge on Shell scripting Postgres SQL or SQL server or MSBI.</p> (ref:hirist.tech)

Required Skill Profession

Computer Occupations