Automated Peer Review of Writing in Science
In the continuing effort to increase retention among students who begin as STEM majors and to help deepen the conceptual understandings of all STEM students, Sweetland Director Anne Gere and Ginger Schultz from the Department of Chemistry have begun incorporating automated peer review into their ongoing project on writing in science.
The first phase of the project focused on developing writing prompts about key concepts in STEM courses, based on the hypothesis that writing fosters deeper learning, especially for under-represented populations in STEM. With support from a U-M Third Century grant titled Digital Writing to Learn Introductory Chemistry and Physics, Gere and Schultz developed a protocol in which students write in response to these prompts, receive feedback from peers, read and respond to the writing of others, and then revise their own writing using a commercially produced automated system.
Nearly 400 students participated in this first phase, and the data gathered shows a number of encouraging results. Participating female students outperformed non-participating female students and performed as well as non-participating male students. Given the gender imbalance in most STEM courses, this finding suggests a potential benefit of the project. Students’ evaluation of their peers’ writing paralleled that of experts in STEM. Although the student-generated evaluations were higher than those of the experts, students’ evaluations produced the same rank-ordering of responses to prompts. This result confirms that automated peer review can be an effective means of responding to student writing in large classes. In addition, analysis of the writing produced by students in response to prompts showed that it will be possible to identify syntactic structures and uses of terminology that characterize more and less successful pieces of student writing. Analyses like this can provide useful information about misconceptions of key concepts – information that can be valuable to STEM faculty.
The next step in this project will be to develop a CTools-based system that will support automated peer review as well as natural language processing. With this system in place, the project will be able to expand into multiple STEM departments as well as other areas of the curriculum where large introductory courses are offered.