Key Responsibilities:
- Requirement gathering and analysis
- Design of data architecture and data model to ingest data
- Experience with different databases like Synapse, SQL DB, Snowflake etc.
- Design and implement data pipelines using Azure Data Factory, Databricks, Synapse
- Create and manage Azure SQL Data Warehouses and Azure Cosmos DB databases
- Extract, transform, and load (ETL) data from various sources into Azure Data Lake Storage
- Implement data security and governance measures
- Monitor and optimize data pipelines for performance and efficiency
- Troubleshoot and resolve data engineering issues
- Hands on experience on Azure functions and other components like realtime streaming etc
- Oversee Azure billing processes, conducting analyses to ensure cost-effectiveness and efficiency in data operations.
- Provide optimized solution for any problem related to data engineering
- Ability to work with verity of sources like Relational DB, API, File System, Realtime streams, CDC etc.
- Strong knowledge on Databricks, Delta tables
Skills Required
Api, File System, cdc, Databricks