<p><p><b>Description :</b></p><p><b><br/></b></p><p>We are seeking a strong Data Engineer with advanced expertise in Databricks and PySpark.
The successful candidate will be a key contributor to critical projects, including migrating Palantir data transformation pipelines to Databricks Notebooks, designing and implementing incremental data pipelines, and orchestrating workflows in Azure Databricks.</p><br/><p><b>Key Responsibilities :</b></p><p><b><br/></b></p><p>- Migrate Palantir data pipelines to Databricks Notebooks, leveraging PySpark for complex transformations.</p><br/><p>- Replace proprietary Palantir libraries with open source or custom Pyspark implementations</p><br/><p>- Design, build, and maintain incremental data load pipelines to handle dynamic updates from various sources, ensuring scalability and efficiency.</p><br/><p>- Develop robust data ingestion pipelines to load data into the Databricks Bronze layer from relational databases, APIs, and file systems.</p><br/><p>- Implement incremental data transformation workflows to update silver and gold layer datasets in near real-time, adhering to Delta Lake best practices.</p><br/><p>- Integrate Airflow with Databricks to orchestrate end-to-end workflows, including dependency management, error handling, and scheduling.</p><br/><p>- Understand business and technical requirements, translating them into scalable Databricks solutions.</p><br/><p>- Optimize Spark jobs and queries for performance, scalability, and cost-efficiency in a distributed environment.</p><br/><p>- Implement robust data quality checks, monitoring solutions, and governance frameworks within Databricks.</p><br/><p>- Collaborate with team members on Databricks best practices, reusable solutions, and incremental loading strategies.</p><br/><p><b>Required Qualifications :</b></p><p><b><br/></b></p><p>- Bachelors degree in computer science, Information Systems, or a related discipline.</p><br/><p>- 6+ years of hands-on experience with Databricks, including expertise in PySpark.</p><br/><p>- Proven experience in incremental data loading techniques into Databricks, leveraging Delta Lake's features (e.g., time travel, MERGE INTO).</p><br/><p>- Strong understanding of data warehousing concepts, including data partitioning, and indexing for efficient querying.</p><br/><p>- Solid knowledge of Azure Cloud Services, particularly Azure Databricks and Azure Data Lake Storage.</p><br/><p>- Familiarity with version control systems (e.g., Git) and CI/CD pipelines for data engineering workflows.</p><br/><p>- Excellent analytical and problem-solving skills with a focus on detail-oriented development.</p><br/><p><b>Preferred Qualifications :</b></p><p><b><br/></b></p><p>- Proficiency in Palantir and experience in migrating Palantir data pipelines to Databricks.</p><br/><p>- Expertise in Airflow integration for workflow orchestration, including designing and managing DAGs.</p><br/><p>- Familiarity with advanced Airflow features, such as SLA monitoring and external task dependencies.</p><br/><p>- Advanced knowledge of Delta Lake optimizations, such as compaction, Z-ordering, and vacuuming.</p><br/><p>- Experience with real-time streaming data pipelines using tools like Kafka or Azure Event Hubs.</p><br/><p>- Experience with building, updating, deploying, finetuning ML models</p><br/><p>- Certifications such as Databricks Certified Associate Developer for Apache Spark or equivalent.</p><br/><p>- Experience in Agile development methodologies.</p><br/></p> (ref:hirist.tech)