Lead Big Data Engineer Spark

Job description

<p><p>Come work at a place where innovation and teamwork come together to support the most exciting missions in the world!.<br/><br/><b>Description :</b><br/><br/>We are seeking a talented Lead Big Data Engineer to deliver roadmap features of Enterprise TruRisk Platform which would help customers to Measure, Communicate and Eliminate Cyber Risks.<br/><br/>Working with a team of engineers and architects, you will be responsible for prototyping, designing, developing and supporting a highly scalable, distributed SaaS based Security Risk Prioritization product.<br/><br/>This is a fantastic opportunity to be an integral part of a team building Qualys next generation platform using Big Data & Micro-Services based technology to process over billions of transactions data per day, leverage open-source technologies, and work on challenging and business-impacting :</b><br/><br/>- Be the thought leader in data platform and pipeline along with Risk Evaluation.<br/><br/>- Provide technical leadership to the engineering organization on data platform design, roll out and evolution.<br/><br/>- Liason to product teams, professional services and sales engineers on solution and trade-off reviews and represent engineering in such conversations.<br/><br/>- Drive technology explorations and roadmaps.<br/><br/>- Serve as a technical lead on our most demanding, cross-functional departments.<br/><br/>- Ensure the quality of architecture and design of systems.<br/><br/>- Functionally decompose complex problems into simple, straight-forward solutions.<br/><br/>- Fully and completely understand system interdependencies and limitations.<br/><br/>- Possess expert knowledge in performance, scalability, enterprise system architecture, and engineering best practices.<br/><br/>- Leverage knowledge of internal and industry prior art in design decisions.<br/><br/>- Effectively research and benchmark cloud technology against other competing systems in the industry.<br/><br/>- Able to document the details so it will be easy for developers to understand the requirements.<br/><br/>- Assisting developers with proper requirements and directions.<br/><br/>- Assist in the career development of others, actively mentoring individuals and the community on advanced technical issues and helping managers guide the career growth of their team members.<br/><br/>- Exert technical influence over multiple teams, increasing their productivity and effectiveness by sharing your deep knowledge and experience.<br/><br/>- Able to share knowledge and train others.<br/><br/><b>Qualifications :</b><br/><br/>- Bachelors degree in computer science or equivalent.<br/><br/>- 8+ years of total experience.<br/><br/>- 4+ years of relevant experience in design and architecture Big Data solutions using Spark.<br/><br/>- 3+ years experience in working with engineering resources for innovation.<br/><br/>- 4+ years experience in understanding Big Data events flow pipeline.<br/><br/>- 3+ years experience in performance testing for large infrastructure.<br/><br/>- 3+ In depth experience in understanding various search solutions solr/elastic.<br/><br/>- 3+ years experience in Kafka.<br/><br/>- In depth experience in Data lakes and related ecosystems.<br/><br/>- In depth experience of messing queue.<br/><br/>- In depth experience in giving requirements to build a scalable architecture for Big data and Micro-services environments.<br/><br/>- In depth experience in understanding caching components or services.<br/><br/>- Knowledge in Presto technology.<br/><br/>- Knowledge in Airflow.<br/><br/>- Hands-on experience in scripting and automation.<br/><br/>- In depth understanding of RDBMS/NoSQL, Oracle , Cassandra , Kafka , Redis, Hadoop, lambda architecture, kappa , kappa ++ architectures with flink data streaming and rule engines.<br/><br/>- Experience in working with ML models engineering and related deployment.<br/><br/>- Design and implement secure big data clusters to meet many compliances and regulatory requirements.<br/><br/>- Experience in leading the delivery of large-scale systems focused on managing the infrastructure layer of the technology stack.<br/><br/>- Strong experience in doing performance benchmarking testing for Big data technologies.<br/><br/>- Strong troubleshooting skills.<br/><br/>- Experience leading development life cycle process and best practices.<br/><br/>- Experience in Big Data services administration would be added value.<br/><br/>- Experience with Agile Management (SCRUM, RUP, XP), OO Modeling, working on internet, UNIX, Middleware, and database related projects.<br/><br/>- Experience mentoring/training the engineering community on complex technical issue.<br/><br/>- Project management experience.<br/><br/></p><br/></p> (ref:hirist.tech)

Required Skill Profession

Computer Occupations