Remote AI/ML Data Annotation Engineer

About Rex.zone

Rex.zone is a recruiting and talent solutions platform supporting modern AI/ML training workflows across the United States. This remote, full-time role focuses on building scalable annotation and evaluation programs that improve training data quality and model performance in production LLM pipelines.

Role Overview

Build and maintain high-quality annotation workflows for AI/ML, including RLHF-style preference data, prompt evaluation, and QA evaluation loops. You will design labeling schemas and write clear annotation guidelines to ensure consistent, measurable outcomes.

Responsibilities

* Own training data quality metrics, error taxonomies, and continuous improvement plans

* Design labeling schemas and write/update annotation guidelines and ambiguity-resolution rules

* Implement QA evaluation programs (review queues, spot checks, gold tasks, inter-annotator agreement)

* Perform RLHF preference labeling and ranking to support alignment and helpfulness

* Run prompt evaluation and regression test sets to track model performance improvement

* Support NLP tasks such as named entity recognition and intent classification

* Support computer vision annotation (e.g., bounding boxes, segmentation) when needed

* Execute content safety labeling for policy-aligned model behavior

* Partner with engineering to integrate tools and datasets into LLM training pipelines

Qualifications

* 3+ years in data operations, ML ops, evaluation, or annotation engineering

* Strong understanding of NLP, LLM evaluation, and human feedback signals (RLHF)

* Experience with QA evaluation methods, sampling strategies, and disagreement analysis

* Familiarity with annotation tools and workflow automation

* Excellent technical writing for guidelines and repeatable processes

Compensation

Competitive hourly base pay: $30–$50/hr.