Chatbot Quality Analyst

About The Role

We're looking for experienced Chatbot Quality Analysts in São Paulo to evaluate and improve AI-powered chatbots and dialogue systems. You'll assess how well AI maintains context, coherence, and helpfulness across extended conversations — and your insights will directly shape the next generation of conversational products.

* Organization: Alignerr

* Type: Hourly Contract

* Compensation: $20–$40 /hour

* Location: Remote

* Commitment: 10–40 hours/week

What You'll Do

* Conduct structured evaluations of AI chatbot conversations across diverse topics and use cases

* Assess dialogue quality for context retention, state management, coherence, and natural flow

* Identify where and how conversations degrade over extended multi-turn interactions

* Design and execute test scenarios that stress-test conversational AI capabilities

* Document failure patterns, edge cases, and quality issues using structured feedback frameworks

* Rate chatbot responses for accuracy, helpfulness, safety, and tone consistency

* Provide actionable recommendations for improving dialogue system performance

* Collaborate asynchronously with research and product teams to refine evaluation criteria

Who You Are

* 2+ years of experience in conversational AI evaluation, chatbot QA, dialogue system testing, or UX research for conversational products

* Strong understanding of how context, state, and coherence degrade across extended conversations

* Analytical mindset with the ability to identify subtle quality issues in natural language interactions

* Excellent written communication skills in English — you can clearly articulate what went wrong and why

* Detail-oriented and systematic in your approach to testing

* Comfortable working independently and managing your own schedule

* Familiar with conversation design principles and common chatbot failure modes

Nice to Have

* Experience with NLP, LLMs, or machine learning concepts

* Background in linguistics, cognitive science, or human-computer interaction

* Familiarity with evaluation frameworks like BLEU, human preference ratings, or custom rubrics

* Experience writing test plans or QA documentation for conversational products

* Prior work with prompt engineering or red-teaming AI systems

* Proficiency in Portuguese is a plus but not required

Why Join Us

* Work on cutting-edge AI projects shaping the future of conversational technology

* Flexible, fully remote work — set your own schedule and work asynchronously

* Collaborate with top AI research labs and industry leaders

* Autonomy and independence in how you approach your work

* Potential for ongoing work and contract extension

* Build deep expertise in one of the fastest-growing areas of AI

Application Process (Takes 10–15 min)

* Submit your resume

* Complete a short screening

* Project matching and onboarding

PS: Our team reviews applications daily. Please complete your application steps to be considered for this opportunity.