Statistics of agreement
September 27, 2012 Leave a comment
I found this formula that calculates the percentage of agreement between two ratings quite interesting and coded the following simple steps using ‘R’. This is called Cohen’s kappa and even though there is nothing original about this entry it is very useful. I wrote the simple R code though because I am learning R.
It was also surprising that I didn’t know about it and our teams are not at all technical enough even to use these foundational principles. As is evident this has wide applications in the fields of percentage agreement calculations when two teams don’t agree or auditors don’t agree with each other. Whither will our antagonistic attitude towards good calculations in technical and project management drive us.
The other point that is a highlight is that I found the description of this formula in a paper dealing with Architecture Trade-off Analysis Method.
The matrix created below shows that two people agree with each other on certain points and disagree on others. The formula to calculate the level of agreement is
Observed percentage of agreement - Expected percentage of agreement
--------------------------------------------------------------
1 - Expected percentage of agreement
R code
kappa<-matrix(c(5,2,1,2),ncol=2)
colnames(kappa)<-c("Disagree","Agree")
rownames(kappa)<-c("Disagree","Agree")
kappa
( I have formatted the output of 'R' as a table )
| Disagree | Agree | |
|---|---|---|
| Disagree | 5 | 1 |
| Agree | 2 | 2 |
kappamargin<-kappa/margin.table(kappa) kappamargin
( I have formatted the output of 'R' which are the percentages as a table )
| Disagree | Agree | |
|---|---|---|
| Disagree | 0.5 | 0.1 |
| Agree | 0.2 | 0.2 |
Observed percentage of agreement = 0.5 + 0.2
Now we want the totals as this table shows. We multiply the total figures of the same color
| Agree | Disagree | Total | |
|---|---|---|---|
| Agree | 0.5 | 0.1 | 0.6 |
| Disagree | 0.2 | 0.2 | 0.4 |
| Total | 0.7 | 0.3 |
So I have just used this line of code to create a matrix of the totals for illustration.
marginals<-matrix(c(margin.table(kappamargin,1),margin.table(kappamargin,2)),ncol=2) marginals
( I have formatted the output of 'R' as a table )
| 0.6 | 0.7 |
| 0.4 | 0.3 |
Expected percentage of agreement = ( 0.6 * 0.7 ) + ( 0.4 * 0.3 )
So final kappa value is
(0.7 - (marginals[1,1] * marginals[1,2]) + (marginals[2,1] * marginals[2,2])) / (1- (marginals[1,1] * marginals[1,2]) + (marginals[2,1] * marginals[2,2]))
0.57
(i.e)
0.7 - (( 0.6 * 0.7 ) + ( 0.4 * 0.3 ))
----------------------------------
1 - (( 0.6 * 0.7 ) + ( 0.4 * 0.3 ))
MindSpace 