Tuesday, October 30, 2007

8 Inter-Rater Reliability

Group members: Gregory Stephens, Joshua Davis, Elliot Pinkus

The thread we chose is a support group on Usenet about people trying to quit smoking. It had 35 total posts, and we rated the first 20. These were our results:




We found that this assignment was really an exercise in experimental design. Although design is important to any scientific field, psychology has a special need for it because there is a delicate balance involved in having empirical evidence to support intangible theories (i.e. a reason why Freud may lose a lot of support). The inter-rater reliability for our team just barely met the requirement of 70%. An inter-rater reliability above 80% would have helped to better support our findings as the quality of our results, though statistically significant, might come into question in any criticism of our analysis. Our ratings showed that esteem and emotional support are closely tied. We found that we had a hard time distinguishing these two characteristics seeing as how we rated nine tenths of the same messages having both characteristics, or the others having neither characteristic. This somewhat subjective rating choice should have been corrected by having multiple graders, but that did not occur in our trials (possibly stemming from our ambiguity of the differences between the two). However, we discovered that, in accordance with Braithwaite, tangible and network support were the least present of all 6 characteristics. This strikes us that in such various forms of support the results seem to hold true and reinforce the findings of Braithwaite. We were surprised how well our results concurred with each other especially since we were skeptical of the inter-rater reliability between our group at the start of the assignment, but we nonetheless found the results reliable and validated them.

From our findings in this experiment, we hypothesized that Wallace's number theory does not really hold up in most online environments. Although many online spaces can create a breeding ground for flame wars and hostility between members, we believe that, depending on where you look, support can be prevalent in the online world. For example, the vast majority of the posts submitted to the Usenet group we examined were supportive and meant to be helpful to the original poster. There were, perhaps, one or two members who left inflammatory or hateful messages, but this amount is dwarfed by the number of people who truly wanted to provide help and support. This lead us to think that many people, at their core, have a fundamental desire or goal to help others, without expecting something (tangible or intangible) in return – as did the people who responded on this support group, who were all posting anonymously (i.e. with a screen name). They were providing their honest support to people they don’t even know in the real world without expecting anything in return, other than the satisfaction of helping others.

Wallace presents the case of Kitty Genovese in the chapter, an example of the number theory. We think that this behavior might exist in a very small percentage of online environments, but certainly not all, or even most of them. The number theory may hold true in some real world environments, but we don’t think this idea applies to online environments to the same extent. As for Walther's Social Distance, Anonymity, Acccess, and Impression Management: we feel that these are more in line with our findings than Wallace’s theory, though our findings are most compatible with Braithwaite’s theory. Looking at individual posts we could find instances of each of these factors. With further research, possibly Braithwaite and Walther could find similar aspects in their theories and form a more cohesive theory suitable for the online world. For now, our findings mainly support Braithwaite as his theory best fits our observations.

No comments: