<p><p><b>Title :</b> AI QA Automation Engineer<br/><br/><b>Location :</b> Remote<br/><br/><b>Job Summary :</b><br/><br/>We are seeking an AI Quality Engineer with a strong automation skillset to ensure the robustness, performance, and reliability of our AI systems and services.
The ideal candidate is tech-savvy, proactive, and passionate about quality at every stepfrom initial design through deployment and ongoing monitoring.
You will play a key role in building and maintaining a highly automated testing infrastructure to support fast, reliable model and pipeline delivery as the company scales.<br/><br/><b>Key Responsibilities :</b><br/><br/><b>Testing Expertise :</b></p><p><p><b><br/></b></p>- Conduct comprehensive testing across all layers, including server load, integration points, and output quality.<br/><br/></p><p>- Apply Test Driven Development (TDD) principlesanticipate, design, and define all necessary tests before the start of feature development.<br/><br/></p><p>- Identify what needs to be tested and proactively communicate requirements before build phases.<br/><br/><b>Automation-First Approach :</b></p><p><p><b><br/></b></p>- Develop, maintain, and extend a fully automated testing suite that covers unit, integration, performance, and end-to-end testing.<br/><br/></p><p>- Emphasize automation to minimize manual intervention and maximize test coverage, reliability, and repeatability.<br/><br/><b>DevOps & CI/CD Integration :</b></p><p><p><b><br/></b></p>- Collaborate closely with DevOps to ensure all tests (including those for model deployment and data pipelines) are tightly integrated with modern CI/CD workflows.<br/><br/></p><p>- Streamline rapid yet safe releases through automation and timely feedback.<br/><br/><b>Automated Testing Frameworks :</b></p><p><p><b><br/></b></p>- Extensive hands-on experience with frameworks such as Pytest (Python testing), Playwright (end-to-end browser testing), Postman (API testing), and Langfuse (LLM output tracking/testing).<br/><br/></p><p>- Implement and maintain robust API contract testing to ensure reliable interactions between services.<br/><br/><b>Manual & LLM Testing :</b></p><p><p><b><br/></b></p>- Execute manual test cases with strong attention to detail, especially for evaluating Large Language Model (LLM) output quality.<br/><br/></p><p>- Flag issues such as hallucinations, factual inaccuracies, or unexpected edge case responses.<br/><br/></p><p>- Continuously update manual testing strategies to adapt to evolving model behaviors and business requirements.<br/><br/><b>Monitoring, Observability & Post-Deploy Quality :</b></p><p><p><b><br/></b></p>- Configure, deploy, and interpret dashboards from monitoring tools like Prometheus, Grafana, and CloudWatch.<br/><br/></p><p>- Track model health, pipeline performance, error rates, and system anomalies after deployment.<br/><br/></p><p>- Proactively investigate and triage quality issues uncovered in production.<br/><br/><b>Core Abilities and Technical Skills :</b></p><p><p><b><br/></b></p>- Deep practical knowledge in test automation, performance, and reliability engineering.<br/><br/></p><p>- In-depth experience integrating tests into CI/CD pipelines, especially for machine learning and AI model workflows.<br/><br/></p><p>- Hands-on proficiency in automated QA tools : Pytest, Playwright, Postman, Langfuse, and similar.<br/><br/></p><p>- Solid foundation in manual exploratory testing, particularly for complex and evolving outputs such as those from LLMs.<br/><br/></p><p>- Expertise in monitoring, APM, and observability tools (e.g., Prometheus, Grafana, CloudWatch).<br/><br/></p><p>- Demonstrated strong problem-solving skillsanticipate, identify, and resolve issues early.<br/><br/></p><p>- Strong communication skills to clearly articulate requirements, quality risks, and advocate for automation-driven quality throughout the organization.<br/><br/><b>Mindset :</b><br/><br/></p><p>- Automation-First : Relentless emphasis on driving automation over manual effort.<br/><br/></p><p>- Proactive : Anticipates issues and testing needs; does not wait to be told what to test.<br/><br/></p><p>- Quality Advocate : Champions testing best practices and designs processes to catch bugs before production.<br/><br/></p><p>- Curious & Continuous Learner : Seeks out new tools, stays current with testing frameworks and industry best practices.<br/><br/></p><p>- Collaborative : Partners effectively with product, engineering, and DevOps teams to deliver high-quality models and features at scale</p><br/></p> (ref:hirist.tech)