What it means
Also called: pairwise comparison, side-by-side.
Comparing two or more model answers to the same prompt and saying which is better, often with a reason and a rating of how much better. These judgements are the main input to RLHF.
Used in: AI evaluator and rater roles.
In open roles right now
120 of the open roles we list mention “preference ranking” or a word for it in the title or description, on 16 platforms (as of 23 October 2026). Of those that state an hourly rate (27 distinct postings), the median is $30/hr, and the middle half pay $20 to $60.
Most of them on: Prolific (34), Mercor (27), Turing (17), Weekday (10), Anyone AI (8), RWS TrainAI (8), and 10 more.
Data Quality Specialist - Swedish, AGI Data Services - G T Quality (posted today)Amazon AGI Data Services·United Kingdom·Full-time
Pay set per projectToday
Senior SWE - Python / TypeScriptWeekday·United States, Canada·Part-time
$100–150/hr4 days ago
Film & Video Craft Expert (Camera, VFX, Editing & Performance)Mercor·Location not stated·Part-time
$70–100/hr7 days ago
Quality Evaluator – Fluent English (Brazil)Anyone AI·Brazil·Full-time
$7–12/hr7 days ago
AI Agent EvaluatorJT AI Labs·15 countries·Flexible hours
$25–45/hr14 days ago
AI Agent Evaluatormicro1·United States·Flexible hours
$25–45/hr14 days ago
Related terms
EvaluationTesting what a model can do by having it answer a set of questions and scoring the answers, often by people working from a rubric.Fact-checkingChecking each claim in a model's answer against reliable sources and marking what is wrong, unsupported or out of date.RLHFA way of training a model on people's judgements. ← Per approved hourPrevious term, A to Z. Project pause →Next term, A to Z. The whole glossaryEvery term on one page, A to Z.Open →