Inter-Rater Reliability Calculator
Two people rate the same items and you need one honest number for how well they agree. This calculator gives you both standard answers: percent agreement for the raw match rate, and Cohen’s kappa from a 2×2 table, which corrects for the agreement two raters would produce by chance alone and labels the result on the Landis and Koch scale.
Percent agreement does not adjust for chance: two raters guessing would still agree sometimes. For publishable reliability, report Cohen’s kappa.
Quick Answer
With 43 agreements and 7 disagreements, percent agreement is 86.00%. In the pre-filled kappa example, two coders sort 165 items (a = 50, b = 10, c = 5, d = 100): observed agreement Po is 90.91%, chance agreement Pe is 54.55%, and Cohen’s kappa is 0.8000, substantial agreement on the Landis and Koch scale. Percent agreement flatters the pair; kappa is the number to report.
Key takeaways
- Report both numbers: percent agreement for readers, kappa for reviewers. One without the other hides half the story.
- Kappa is built from the table marginals, so the same 90% match rate can earn 0.80 in a balanced task and 0.31 in a skewed one.
- Counts are whole items. The tool truncates decimals (43.9 becomes 43), so tally your table in integers before you start.
- This tool covers exactly two raters and two categories. Three or more raters need Fleiss’ kappa, which this page does not compute.
- Quick Answer
- How to use the inter-rater reliability calculator
- The formulas, and where chance agreement comes from
- Worked example, checked against this exact tool
- The kappa paradox: high agreement, low kappa
- What competing calculators miss
- Frequently asked questions
- Add This Calculator to Your Website
- References
How to use the inter-rater reliability calculator
Pick the mode that matches the data you actually have: two summary counts for percent agreement, or the full 2×2 table for kappa. Results stay hidden until you click Calculate.
- Percent agreement mode: enter how many items the pair agreed on and how many they disagreed on. The tool divides agreements by the total and multiplies by 100. That is the whole method, which is also its weakness: nothing is removed for luck.
- Kappa mode, build the table first: rows are Rater 1, columns are Rater 2. Cell a is both said A, cell d is both said B, and cells b and c are the two ways to disagree. Swapping b and c changes nothing in the result, because Pe uses both marginals symmetrically, but swapping a diagonal cell with an off-diagonal cell changes everything.
- Read all three kappa lines: kappa itself, Po (what the pair actually did) and Pe (what chance predicts from their habits). A kappa with no Po and Pe beside it cannot be judged.
- Check the sample size: the tool accepts any total above zero, but a kappa from 12 items swings wildly with one changed rating. Treat 30 items as a floor for a number you plan to publish.
Kappa is an agreement statistic, not a correlation. Two raters can track each other closely yet rarely match exactly, which is why this page sits apart from the correlation coefficient calculator: correlation grades the relationship between two numeric series, kappa grades exact categorical matches between two judges.
The formulas, and where chance agreement comes from
Percent agreement is agreements divided by total items, times 100. Kappa goes one step further: it asks how much of the possible improvement over chance the pair actually achieved.
Observed agreement Po = (a + d) ÷ n
Chance agreement Pe = ((a + b)(a + c) + (c + d)(b + d)) ÷ n²
Cohen’s kappa κ = (Po − Pe) ÷ (1 − Pe)
Pe is the clever part. Each rater has habits: if Rater 1 calls 60 of 165 items A and Rater 2 calls 55 of 165 items A, pure chance already predicts a stack of both-A matches, and the same logic applies to category B. Pe adds those two chance predictions using the marginal totals of your table, following the coefficient Jacob Cohen published in 1960 for nominal scales. Under the hood it is a discrete probability distribution argument: independent raters, multiplied marginal probabilities. Kappa then rescales what is left: a pair at chance level scores 0, a perfect pair scores 1, and a pair that disagrees more than chance scores below 0.
Worked example, checked against this exact tool
Two coders classify 165 survey responses as A (on-topic) or B (off-topic). The pre-filled table is a = 50, b = 10, c = 5, d = 100. Every figure below is the tool’s own output, digit for digit.
| Quantity | Arithmetic | Tool output |
|---|---|---|
| Observed agreement Po | (50 + 100) ÷ 165 | 90.91% |
| Chance agreement Pe | ((60 × 55) + (105 × 110)) ÷ 165² | 54.55% |
| Cohen’s kappa | (0.9091 − 0.5455) ÷ (1 − 0.5455) | 0.8000 |
| Percent agreement run | 43 ÷ 50 × 100 | 86.00% |
The band label matters as much as the number. At exactly 0.8000 the tool still says Substantial, because its top band starts above 0.80. The bar view below uses the tool’s own cut points, the ones Landis and Koch proposed in 1977 and most journals still print.
The kappa paradox: high agreement, low kappa
A kappa far below your percent agreement usually means a skewed table, not sloppy raters. When almost every item lands in one category, Pe climbs toward Po and leaves kappa little room to rise.
Run the table a = 95, b = 2, c = 2, d = 1 in the tool and you get the paradox in pure form: the pair agrees on 96.00% of items, yet kappa is only 0.3127 (Fair), because chance alone predicts 94.18% agreement when both raters say A almost every time. The fix is reporting, not despair: publish Po, Pe and kappa together, name the prevalence problem, and let readers judge. Statisticians have documented this behaviour since Feinstein and Cicchetti’s 1990 papers, and methods guides such as Marcia McHugh’s 2012 primer on the kappa statistic advise exactly this paired reporting. If your write-up needs an uncertainty range around a proportion as well, the confidence interval calculator covers that separate job; this tool deliberately reports the point values only.
What competing calculators miss
The kappa tools ranking for this keyword split into two camps, and each camp drops something a researcher actually needs:
- GraphPad QuickCalcs: a solid, citable kappa from a 2×2 table, but kappa only. There is no percent-agreement mode, and the workflow is one step in a larger Prism sales funnel.
- Social Science Statistics and similar stats portals: they add standard error, z-scores and p-values, which is genuinely useful, but the table entry is bare and there is no raw agreement mode for a quick check before the formal run.
- Generalist pages (Brave Calculator, Savvy Calculator): friendly explainers with both metrics described, yet their worked examples hand you a pre-computed chance agreement instead of deriving Pe from your own marginals, so you cannot see why kappa moved.
- Advanced browser tools (Lensym, Induwara): paste-in rating lists, weighted kappa, bootstrap intervals and tables beyond 2×2. Powerful, but they demand tidy per-item data; if your study records only the finished 2×2 counts, they are slower than this page.
- The gap this page fills: both reporting modes on one page, Pe derived visibly from your table with the full substitution printed under the result, the Landis and Koch label attached to the exact cut points the code uses, and an honest scope note: two raters, two categories, no confidence interval, no weighted kappa.
Frequently asked questions
The simplest way is percent agreement: agreements divided by total items, times 100. The rigorous way is Cohen’s kappa, which adjusts for chance agreement: kappa = (Po – Pe) / (1 – Pe), where Po is observed agreement and Pe is the agreement the raters’ marginal rates predict by chance.
On the Landis and Koch scale this tool uses, 0.61 to 0.80 is substantial and 0.81 to 1.00 is almost perfect. Most journals expect at least 0.60 before coded data is treated as reliable, though the right bar depends on what rides on the ratings.
Percent agreement counts raw matches; kappa subtracts the agreement that would happen by chance. High percent agreement with a low kappa means the raters mostly agree because the task is easy or one category dominates, not because they are consistent.
Yes. A negative kappa means agreement is worse than chance, which usually signals a systematic disagreement, opposite readings of the coding rules, or a data entry error worth investigating before you collect more data.
At least 30 rated items for a reasonably stable kappa; more is better. With tiny samples, one changed rating swings the result, and this tool will still compute a value for very small tables, so check the count before trusting it.
Fleiss’ kappa is the generalization of Cohen’s kappa to three or more raters. This tool covers the two-rater case only, in percent agreement and Cohen’s kappa modes, so use a Fleiss’ kappa tool or a statistics package when three or more raters scored the same items.
This is the kappa paradox: when almost all items fall in one category, chance agreement Pe is very high, so even 96 percent observed agreement can yield a kappa near 0.31. Report Po, Pe and kappa together so readers can see the prevalence effect.
For ordinal scales where near-misses matter, such as 1 to 5 ratings, weighted kappa gives partial credit and is the better choice. For nominal categories with no order, use unweighted kappa as computed here. This tool computes unweighted Cohen’s kappa for a 2 by 2 table.
No. That is its main weakness and the reason kappa exists. Two raters guessing at random still agree on a share of items, and percent agreement counts those lucky matches as reliability. Use it for quick checks, and kappa for anything you will publish or defend.
If all items land in a single cell, chance agreement Pe reaches 100 percent and kappa is mathematically undefined, because there is no variation to measure. The tool says so with the message: Chance agreement is 100%: kappa is undefined for this table.
Add This Calculator to Your Website
Embed this calculator on your own site with the code below. Pick a size to preview it live, then copy the code.
Powered by Calculator Zilo
References
- Cohen, Jacob. A Coefficient of Agreement for Nominal Scales, Educational and Psychological Measurement, 1960. Accessed October 22, 2026.
- Landis, J. Richard, and Koch, Gary G. The Measurement of Observer Agreement for Categorical Data, Biometrics, 1977. Accessed October 22, 2026.
- McHugh, Mary L. Interrater reliability: the kappa statistic, Biochemia Medica, via PubMed Central, 2012. Accessed October 22, 2026.
- scikit-learn developers. sklearn.metrics.cohen_kappa_score documentation. Accessed October 22, 2026.
- lrd package authors. Cohen’s Kappa documentation for the R package lrd, CRAN. Accessed October 22, 2026.