Cohen's kappa | EPFL Graph Search

Are you an EPFL student looking for a semester project?

Work with us on data science and visualisation projects, and deploy your project as an app on top of GraphSearch.

Cohen's kappa coefficient (κ, lowercase Greek kappa) is a statistic that is used to measure inter-rater reliability (and also intra-rater reliability) for qualitative (categorical) items. It is generally thought to be a more robust measure than simple percent agreement calculation, as κ takes into account the possibility of the agreement occurring by chance. There is controversy surrounding Cohen's kappa due to the difficulty in interpreting indices of agreement. Some researchers have suggested that it is conceptually simpler to evaluate disagreement between items. The first mention of a kappa-like statistic is attributed to Galton in 1892. The seminal paper introducing kappa as a new technique was published by Jacob Cohen in the journal Educational and Psychological Measurement in 1960. Cohen's kappa measures the agreement between two raters who each classify N items into C mutually exclusive categories. The definition of is where po is the relative observed agreement among raters, and pe is the hypothetical probability of chance agreement, using the observed data to calculate the probabilities of each observer randomly seeing each category. If the raters are in complete agreement then . If there is no agreement among the raters other than what would be expected by chance (as given by pe), . It is possible for the statistic to be negative, which can occur by chance if there is no relationship between the ratings of the two raters, or it may reflect a real tendency of the raters to give differing ratings. For k categories, N observations to categorize and the number of times rater i predicted category k: This is derived from the following construction: Where is the estimated probability that both rater 1 and rater 2 will classify the same item as k, while is the estimated probability that rater 1 will classify an item as k (and similarly for rater 2). The relation is based on using the assumption that the rating of the two raters are independent.

About this result

This page is automatically generated and may contain information that is not correct, complete, up-to-date, or relevant to your search query. The same applies to every other page on this website. Please make sure to verify the information with EPFL's official sources.

Related publications (1)

Related concepts (13)

Related courses (3)

Related lectures (26)

Cohen's kappa

Inter-rater reliability

In statistics, inter-rater reliability (also called by various similar names, such as inter-rater agreement, inter-rater concordance, inter-observer reliability, inter-coder reliability, and so on) is the degree of agreement among independent observers who rate, code, or assess the same phenomenon. Assessment tools that rely on ratings must exhibit good inter-rater reliability, otherwise they are not valid tests. There are a number of statistics that can be used to determine inter-rater reliability.

Fleiss' kappa

Fleiss' kappa (named after Joseph L. Fleiss) is a statistical measure for assessing the reliability of agreement between a fixed number of raters when assigning categorical ratings to a number of items or classifying items. This contrasts with other kappas such as Cohen's kappa, which only work when assessing the agreement between not more than two raters or the intra-rater reliability (for one appraiser versus themself). The measure calculates the degree of agreement in classification over that which would be expected by chance.

RNA sequencing-based single sample predictors of molecular subtype and risk of recurrence for clinical assessment of early-stage breast cancer

Cathrin Brisken, Ayyakkannu Ayyanan, Rachel Marcone

Multigene assays for molecular subtypes and biomarkers can aid management of early invasive breast cancer. Using RNA-sequencing we aimed to develop single-sample predictor (SSP) models for clinical markers, subtypes, and risk of recurrence (ROR). A cohort of 7743 patients was divided into training and test set. We trained SSPs for subtypes and ROR assigned by nearest-centroid (NC) methods and SSPs for biomarkers from histopathology. Classifications were compared with Prosigna in two external cohorts (ABiM, n = 100 and OSLO2-EMIT0, n = 103). Prognostic value was assessed using distant recurrence-free interval. Agreement between SSP and NC for PAM50 {five subtypes) was high (85%, Kappa = 0.78) for Subtype (four subtypes) very high {90%, Kappa = 0.84) and for ROR risk category high (84%, Kappa = 0.75, weighted Kappa = 0.90). Prognostic value was assessed as equivalent and clinically relevant. Agreement with histopathology was very high or high for receptor status, while moderate for Ki67 status and poor for Nottingham histological grade. SSP and Prosigna concordance was high for subtype (OSLO-EMIT0 83%, Kappa = 0.73 and ABiM 80%, Kappa = 0.72) and moderate and high for ROR risk category (68 and 84%, Kappa = 0.50 and 0.70, weighted Kappa = 0.70 and 0.78). Pooled concordance for emulated treatment recommendation dichotomized for chemotherapy was high (85%, Kappa = 0.66). Retrospective evaluation suggested that SSP application could change chemotherapy recommendations for up to 17% of postmenopausal ER+/HER2-/N0 patients with balanced escalation and de-escalation. Results suggest that NC and SSP models are interchangeable on a group-level and nearly so on a patient level and that SSP models can be derived to closely match clinical tests.

NATURE PORTFOLIO2022

ENG-209: Data science for engineers with Python

Ce cours est divisé en deux partie. La première partie présente le langage Python et les différences notables entre Python et C++ (utilisé dans le cours précédent ICC). La seconde partie est une intro

BIO-322: Introduction to machine learning for bioengineers

Students understand basic concepts and methods of machine learning. They can describe them in mathematical terms and can apply them to data using a high-level programming language (julia/python/R).

CS-233(a): Introduction to machine learning (BA3)

Machine learning and data analysis are becoming increasingly central in many sciences and applications. In this course, fundamental principles and methods of machine learning will be introduced, analy

Generalized Linear Regression: Classification BIO-322: Introduction to machine learning for bioengineers

Explores Generalized Linear Regression, Classification, confusion matrices, ROC curves, and noise in data.

Introduction to Human Rights HUM-485: Data in context: Critical Data Studies I

Introduces key human rights concepts, explores discrimination, social norms, and the impact of algorithms on rights.

Supervised Learning Essentials

Introduces the basics of supervised learning, focusing on logistic regression, linear classification, and likelihood maximization.