Lead Cloud Data Engineer (Azure Databricks)

Job description

Creative Synergies Group Overview

• Creative is a leading Global Technology Services firm delivering innovative solutions that build upon synergies between our Deep Engineering Domain Expertise and Digital Engineering Services
• We work with clients across the Global Transportation, Energy/Process, Industrial Products, Hi-Tech, and EdTech verticals
• Creative’s global team of 1,500+ “Can-Do” engineers collaborate with our 40+ Fortune 500 clients to help them achieve their strategic and operational objectives (Including Google, GM, Samsung, Tesla, Mercedes, Sony, Mitsubishi)
• 95% of our revenue comes from US, Europe, Japan based customers

Service Offerings:

Data, Analytics, and AI Engineering Services

Digital Product and Platform Engineering

Intelligent Connected Product Engineering.

• Vehicle Centric Electronics
• Connected Car Solutions
• Mobile Robotics
• Industrial IoT

Digital Manufacturing

Digital Plant Engineering

Founder / CEO: Dr. Mukesh Gandhi

Former Michigan State University Professor (1984 - 1999)

• Published Ground-Breaking Technical Publications and Books
• Supervised PhD, Masters and Undergraduate Research Students - Who Are Now Leaders and Decision Makers In Industry And Academia
• Awarded Over $15 Million For Pioneering Research from National Science Foundation (NSF), US Army Research Office (ARO), Defense Advanced Research Projects Agency (DARPA), US National Institute of Standards and Technology (NIST)

Founder / CEO of Quantech Global Services

• Pioneering US-Indian Services company to serve major global automotive OEMs (GM, Ford, Nissan, Peugeot) from India-based delivery centers in the early 2000s
• Acquired by Wipro (NYSE:WIT)

Former Wipro, EDS Chief Executive

Founded Creative Synergies Group In 2011

Locations: Headquartered in the U.S., multiple delivery centers in India (Bengaluru, Pune), branch offices in Germany, U.K, Netherlands and Japan

Culture: Creative has a flat organization and an agile culture of positivity, entrepreneurial spirit, customer centricity, teamwork, and meritocracy

Creative Synergies Group 3 Year Vision

• Continue growing overall business at 40% per year for the next 3 years
• Establish Leadership in AI/ML Engineering Services
• Substantially expand revenues in Electronics, Connected Vehicle Solutions, Mobile Robots, Industrial IoT
• Meaningfully expand into EdTech
• Explore niche acquisitions in the U.S.

Job Title: Azure Data Engineer Lead - (Availability to join within 4 weeks is highly preferable)

Experience: 8-12 years

Job Summary:

We are seeking a skilled and motivated Senior Azure Data Engineer with 8-12 years of relevant experience.

Candidate will play a pivotal role in designing, implementing, and optimizing large-scale data solutions on the Microsoft Azure platform.

Candidate will work closely with other cross-functional teams to ensure data solutions are scalable, secure, and high-performance.

The ideal candidate will have a strong background in data engineering, deep expertise in Azure Databricks and proven leadership skills to guide and mentor the team.

Key Responsibilities:

• Architect and Design Data Solutions: Lead the design and implementation of scalable, secure and high-performing data solutions on Azure, ensuring alignment with business goals and technical requirements.
• Data Pipeline Development: Build and optimize end-to-end data pipelines to collect, process and analyze data from various sources using Azure Databricks, Azure Data Factory and other Azure data services.
• Cloud Infrastructure Management: Architect and maintain cloud data infrastructure using Azure Data Lake, Azure Synapse Analytics, Azure SQL Data Warehouse and Azure Blob Storage.
• Collaborate with Cross-functional Teams: Work closely with data architects, business analysts and developers to gather requirements, define data models and ensure smooth integration of data across systems.
• Performance Tuning and Optimization: Continuously monitor and optimize the performance of data workflows and pipelines to ensure efficient data processing.
• Data Governance & Security: Implement best practices for data governance, security and compliance, ensuring that data solutions adhere to company and regulatory standards.
• Mentor and Lead Teams: Provide technical leadership and mentorship to junior data engineers, guiding them in the best practices of data engineering and cloud technologies.

Required Qualifications

• Bachelor's or Master’s degree in Computer Science, Engineering, or a related field

• 8-12 years of experience in data engineering, with at least 5+ years of hands-on experience with Microsoft Azure cloud technologies.
• Strong experience with Azure Databricks, Azure Data Factory, Azure Synapse Analytics, Azure SQL Database, Azure Blob Storage, Azure Data Lake and other related Azure services.
• Hands-on experience using these services: PySpark, Notebooks, Structured Streaming, Unity Catalog, Delta Live Tables, Workflows and SQL Warehouse
• In-depth knowledge of data modeling, ETL/ELT processes, data warehousing and data pipeline architecture.
• Proven experience in building and optimizing large-scale data pipelines in a cloud environment.
• Strong experience with SQL, NoSQL databases and big data processing technologies (e.G., Spark, Hadoop, Kafka).
• Hands-on experience with data transformation, data integration and data quality frameworks.
• Expertise in data security and compliance standards in a cloud-based environment.
• Ability to design and implement data governance strategies for large datasets.
• Strong leadership, communication and collaboration skills, with a proven ability to guide teams and manage complex projects.

Good to have:

• Microsoft Azure Data Engineer Certification
• Databricks Certified Data Engineer Professional
• Familiarity with GenAI Concepts
• Familiar with DevOps Practices

Required Skill Profession

Computer Occupations