Role and Responsibilities:
- Participate in the entire data pipeline lifecycle, focusing on building data lakes, data warehouses and related processes
- Contribute to the design and documentation of our data lake platform
- Partner with Sr staff on the development and maintenance of data governance policies and procedures that ensure data quality, accuracy, consistency, and privacy.
- Mentor other data engineers, providing technical guidance, and ensuring that team members have the necessary skills and resources to perform their duties.
- Conduct code and design reviews with junior and peer developers
- Write clean code to develop data pipelines and necessary infrastructure as code
- Aid in setting coding standards for team members to follow
- Partner with cross functional teams to understand requirements and effectively design appropriate solutions.
- Monitor and troubleshoot performance, scalability, and security issues, making improvements as needed
- Aid in determining and implementing KPIs for performance and stability of platform
- Participate in the review of new technologies and tools for our data teams
- Partner with engineering and other cross functional teams to collect and implement technical and design requirements
- Communicate effectively with technical and non-technical team members to ensure alignment and successful execution of projects.
- Participate in cross-functional initiatives, promoting a culture of data-driven decision-making and continuous improvement.
- Participate in an on call rotation.
Job Requirement : - 8 years of related experience; or 6 years and an advanced degree
- Bachelors degree in engineering, computer science, math, data engineering or another related field or equivalent work experience
- Experience in the development of modern data architecture, analytics, data governance, AI/ML, or related areas.
- Strong Experience with AWS big data technologies including EMR, Glue, S3, EKS, Lambda, Athena, RDS, MKS/Kafka/Kinesis
- Expertise with relational and NoSQL databases such as PostgresQL, Cassandra, DynamoDB, MongoDb etc and data modeling principles
- Knowledge of data pipeline tools and workflow engines, e.g. Apache Airflow and Spark.
- Experience with DevOps, including CI/CD pipelines, containerized deployment/Kubernetes, and infrastructure-as-code/AWS Cloud Formation/Terraform.
- Successfully contributes to the entire development lifecycle of teams products
- Exposure to secure coding practices, access controls, authentication, and audit.
AWS Lake formation preferred.
- Strong experience with SQL (SQL Server, Postgres) and Python, Scala
- Identify opportunities for process improvement as well as efficiency in solutions
- Demonstrated ability to mentor junior team members
- Excellent analytical and time management skills, with a proven ability to deliver value independently
- Strong written and verbal communication skills, with demonstrated experience providing technical input to technical and non-technical stakeholders
- Ability to participate in on-call rotation
Preferred:
- FinTech industry experience
- Understanding of MPP Data Warehouse (Redshift, Greenplum)
- Experience with Graph database