Data Engineer (AWS MSK Specialist)

Job Title: Data Engineer - AWS MSK Specialist
Job type: Full time hire
On-site work from office - Hitech City
Work timings: 3 PM/4 PM - 11 PM/12 AM (Flexible as per work requirement, according to US EST Timings)

Job Summary:
We're looking for a skilled Data Engineer with expertise in AWS Managed Streaming for Apache Kafka (MSK) to design, build, and maintain high-performance data pipelines.

You'll work closely with data architects and software engineers to ensure seamless data integration and streaming capabilities using AWS MSK.

Responsibilities:
- Design, develop, and manage real-time data pipelines using AWS MSK
- Integrate AWS MSK with various data sources and consumers
- Monitor and optimize AWS MSK cluster performance and costs
- Collaborate with DevOps to ensure optimal performance and uptime
- Implement security best practices for data streaming and storage
- Document technical processes and best practices
- Work with AWS services like S3, Glue, and Lambda for data processing and integration

Requirements:
- Bachelor’s degree in computer science, Engineering, or a related field
- 3-5 years of experience working with AWS MSK or Apache Kafka in a production environment
- Strong understanding of distributed systems, real-time data processing, and cloud-based data streaming
- Proficiency in programming languages such as Python
- Experience with AWS services like S3, Glue, and Lambda
- Excellent problem-solving and debugging skills

Nice-to-Have Skills:
- Experience with containerization technologies like Docker and Kubernetes
- Familiarity with other messaging systems like RabbitMQ or Apache Pulsar
- Experience with data serialization formats like Avro, Protobuf, or JSON
- Knowledge of Snowflake or other cloud-based data warehousing solutions