What happens to my data?
We value your privacy and only store data that is relevant for our research. We act in accordance with the EU GDPR. For detailed information, see the data protection information sheet.
Comparity.ai is an ongoing research project. The full dataset will be publicly released alongside the first publication from this project.
How much can I use Comparity.ai?
Effectively, there are no limit to your usage. However, to protect from malicious intent each registered user account and IP address is limited to 500 requests per 24-hour window – which corresponds to approximately 1775000 tokens per user on average. This is assumed to be sufficient, even for power-users.
How are the scores computed?
There are two leaderboards, and both are scored the same way: a rating shown on the familiar Elo scale, anchored at 1500 — that's a reference point, not a maximum, so models can score above or below it (a 400-point gap is roughly 10× the odds of winning). A Style control switch on the leaderboard decides whether that rating is style-controlled: with the switch on (the default for the overall boards), formatting and length are held constant when we compute the rating — so a model can't climb the board just by adding headers, bold text, and bullet lists, or by writing longer answers; only the underlying quality counts. Switch it off to see the ranking from the raw outcomes, with no presentation adjustment. (The dwell-based Cascading board always accounts for where a response appeared and how long it was — a longer answer simply takes longer to read — so its switch governs only the formatting adjustment.)
The Side-by-Side board is fit on your explicit A/B votes (both-good and both-bad are treated as ties), so a higher rating means the model is preferred more often — with formatting and length accounted for while Style control is on. A raw win rate is shown alongside for reference.
The Cascading board is fit on how long you linger on each response before the model names are revealed — within a single turn, which response held more attention — while controlling for slot position (the response shown first gets more attention regardless of which model wrote it). It also reports an Engagement figure: how much attention a model's answers actually held versus what their slot and length alone predict, shown as a multiplier (1.0× = exactly as engaging as expected, above = holds attention beyond that, below = less).
Why does it look different for different users?
Comparity.ai has two distinct usage modes. One is the standard side by side view, where you see the output of two models, and one is the Cascading mode, where you only see one answer, but can "swipe" through all models.
Questions or interested in collaborating?
