Expertini Resume Scoring: Our Semantic Matching Algorithm evaluates your CV/Résumé before you apply for this job role.

Get Resume Score

India Jobs Expertini

Data Engineer - Azure Databricks/PySpark Job Opening In India, India – Now Hiring Codesmith

Data Engineer Azure Databricks/PySpark

Job description

Description : We are seeking a strong Data Engineer with advanced expertise in Databricks and PySpark.

The successful candidate will be a key contributor to critical projects, including migrating Palantir data transformation pipelines to Databricks Notebooks, designing and implementing incremental data pipelines, and orchestrating workflows in Azure Databricks. Key Responsibilities : - Migrate Palantir data pipelines to Databricks Notebooks, leveraging PySpark for complex transformations. - Replace proprietary Palantir libraries with open source or custom Pyspark implementations - Design, build, and maintain incremental data load pipelines to handle dynamic updates from various sources, ensuring scalability and efficiency. - Develop robust data ingestion pipelines to load data into the Databricks Bronze layer from relational databases, APIs, and file systems. - Implement incremental data transformation workflows to update silver and gold layer datasets in near real-time, adhering to Delta Lake best practices. - Integrate Airflow with Databricks to orchestrate end-to-end workflows, including dependency management, error handling, and scheduling. - Understand business and technical requirements, translating them into scalable Databricks solutions. - Optimize Spark jobs and queries for performance, scalability, and cost-efficiency in a distributed environment. - Implement robust data quality checks, monitoring solutions, and governance frameworks within Databricks. - Collaborate with team members on Databricks best practices, reusable solutions, and incremental loading strategies. Required Qualifications : - Bachelors degree in computer science, Information Systems, or a related discipline. - 6+ years of hands-on experience with Databricks, including expertise in PySpark. - Proven experience in incremental data loading techniques into Databricks, leveraging Delta Lake's features (e.g., time travel, MERGE INTO). - Strong understanding of data warehousing concepts, including data partitioning, and indexing for efficient querying. - Solid knowledge of Azure Cloud Services, particularly Azure Databricks and Azure Data Lake Storage. - Familiarity with version control systems (e.g., Git) and CI/CD pipelines for data engineering workflows. - Excellent analytical and problem-solving skills with a focus on detail-oriented development. Preferred Qualifications : - Proficiency in Palantir and experience in migrating Palantir data pipelines to Databricks. - Expertise in Airflow integration for workflow orchestration, including designing and managing DAGs. - Familiarity with advanced Airflow features, such as SLA monitoring and external task dependencies. - Advanced knowledge of Delta Lake optimizations, such as compaction, Z-ordering, and vacuuming. - Experience with real-time streaming data pipelines using tools like Kafka or Azure Event Hubs. - Experience with building, updating, deploying, finetuning ML models - Certifications such as Databricks Certified Associate Developer for Apache Spark or equivalent. - Experience in Agile development methodologies. (ref:hirist.tech)

Required Skill Profession

Computer Occupations