AI/ML

Multiple Job Openings on ML/AI roles - Toronto, Canada

Please email your resume to [email protected]

Responsibilities:

Own end-to-end QA/validation for ML and Agentic AI solutions: requirements-to-metrics, test planning, execution, defect triage, and release sign-off.

Design and maintain evaluation frameworks for agentic systems: task success rate, tool-call correctness, grounding/citation quality (where used), latency/cost, and regression detection.

Build automated test suites in Python for: data validation (schema, drift, anomalies), feature/label quality checks, model inference correctness, and agent workflow validation (multi-step, tool-using, and memory-based flows).

Implement LLM/agent-specific quality checks: hallucination and factuality testing, prompt injection and jailbreak resistance testing, PII leakage checks, toxicity/safety filters, and policy conformance.

Validate RAG systems end-to-end: document chunking/embedding quality, retrieval accuracy (precision/recall), reranking behavior, and answer faithfulness to retrieved context.

Establish test data and “golden” datasets: curated evaluation sets, adversarial test cases, synthetic data generation (where appropriate), and clear acceptance criteria.

Integrate quality gates into CI/CD: unit/integration tests, evaluation runs, reporting dashboards, and release-blocking thresholds.

Partner with engineers to instrument observability: tracing, structured logs, metrics, error cohorts, and production monitoring for drift, degradation, bias, latency, and cost.

Collaborate with platform teams to run validations at scale (Databricks jobs, Spark pipelines, scheduled workflows) and ensure governance over data/model access.

Document validation approaches, test evidence, and risk assessments; support audits and compliance needs for regulated or high-impact use cases.

help me with keywords for ceipal job posting keyword search