EverestDX - Azure Databricks Administrator - Scala/Python

<p><p>We are seeking a Cloud Databricks Administrator to join our Cloud Engineering team.<br/><br/> In this role, you will monitor and maintain Databricks jobs, troubleshoot issues, ensure platform reliability, and manage access controls.<br/><br/> You will play a key part in operational support, defect management, and proactive issue resolution to keep our data pipelines running smoothly.</p><br/><p><b>Key Responsibilities :</b></p><p><p><b><br/></b></p>- Monitor hourly and daily Databricks jobs, investigate failures, and implement fixes to minimize downtime.<br/><br/></p><p>- Identify, log, and track defects/bugs through ticketing systems, ensuring timely resolution.<br/><br/></p><p>- Manage Databricks access via Azure AD groups with Admin, Edit, and Read permissions.<br/><br/></p><p>- Provide production support for Databricks environments, including cluster operations, job failures, and notebook troubleshooting.<br/><br/></p><p>- Collaborate with data engineers and platform teams to resolve platform-related incidents and performance bottlenecks.<br/><br/></p><p>- Proactively monitor system health, resource utilization, and performance metrics.<br/><br/></p><p>- Implement and enforce archival/retention policies for Databricks storage to optimize costs and performance.<br/><br/></p><p>- Support CI/CD pipelines (Jenkins, Azure Automation) and automate repetitive operational tasks.<br/><br/></p><p>- Maintain technical documentation, SOPs, and runbooks for Databricks operations.<br/><br/></p><p>- Ensure security compliance with RBAC, MFA, and encryption best Qualifications : </b></p><p><br/></p>- 3+ years of hands-on experience with Databricks and Apache Spark in production environments.<br/><br/></p><p>- Strong knowledge of Azure (AWS/GCP acceptable) and cloud-native services.<br/><br/></p><p>- Experience in SRE or production support environments with SLAs and ticketing systems </p><p>(ServiceNow, Jira).<br/><br/></p><p>- Proficiency in Python or Scala for data processing and automation.<br/><br/></p><p>- Familiarity with Power BI/Tableau for building monitoring and cost dashboards.<br/><br/></p><p>- Knowledge of CI/CD tools, version control (Git), and scripting languages (PowerShell, Bash).<br/><br/></p><p>- Understanding of cloud cost optimization and usage tracking.<br/><br/></p><p>- Excellent problem-solving skills, communication, and cross-team collaboration abilities.</p><br/><br/><p><p><b>Nice to Have : </b></p><p><br/></p>- Experience with Databricks Lakehouse/Madelaine architecture.<br/><br/></p><p>- Background in monitoring, logging, and incident response for data platforms.<br/><br/></p><p>- Exposure to Kubernetes, Docker, and Skills : </b></p><p><br/></p>- Bachelors degree in Computer Science, IT, or equivalent professional experience.<br/><br/></p><p>- 5+ years in data engineering, cloud operations, or database administration.<br/><br/></p><p>- Proven ability to troubleshoot, communicate effectively, and collaborate across teams in a </p><p>fast-paced environment.</p><br/></p> (ref:hirist.tech)