Description
Data Engineer: Primary : ADF, SQLSeconday : Databricks, Pyspark / Scala, CICD, Data Security, encryption
Responsibilities: •Work closely with scrum master, technical leads and provide daily progress updates on the tasks assigned. •Provide support in defining the scope and sizing of the work. •Working closely with various enterprise architects, Information security teams, Data management team, to ensure the architected solution meets all the needs of a customer, from a functionality perspective and IT solution engineering perspective. •Translate business requirements into technology solutions. •Lead designing of all aspects of our data solution including artifact creation such as diagrams, playbooks, and other technical documents Skills: •Over all 5 to 8 years of experience with Data Management & Big Data and Analytics. •At least 2 to 3 years of experience in implementing the data solutions on MS Azure using Azure Data Factory. •Must have strong SQL background and ability to tune the SQL for optimized ETL jobs. •one to two years of experience in Azure Synapse Analytics is plus. •Extensive hands-on experience in data warehousing design, tuning and ETL/ELT process development by using cloud native technologies. •Apache Spark experience using Scala or PySpark or pre-packaged tools like Databrick is plus. •At least one year of experience with API based/real time integrations using Kafka or similar technologies. •At least 1 to 2 years of experience with big data using spark and java for data processing. •Developing the CICD pipeline for Azure Infrastructure, version control strategy and Integrate source control (git, GitHub, azure repos). •At least one year experience with unified data governance solution using MS Purview.