We studied student learning in the MOOC 8.MReV Mechanics ReView, run on the edX.org open source platform. We studied learning in two ways. We administered 13 conceptual questions both before and after instruction, analyzing the results using standard techniques for pre- and posttesting. We also analyzed each week’s homework and test questions in the MOOC, including the pre- and posttests, using item response theory (IRT). This determined both an average ability and a relative improvement in ability over the course. The pre- and posttesting showed substantial learning: The students had a normalized gain slightly higher than typical values for a traditional course, but significantly lower than typical values for courses using interactive engagement pedagogy. Importantly, both the normalized gain and the IRT analysis of pre- and posttests showed that learning was the same for different cohorts selected on various criteria: level of education, preparation in math and physics, and overall ability in the course. We found a small positive correlation between relative improvement and prior educational attainment. We also compared homework performance of MIT freshmen taking a reformed on-campus course with the 8.MReV students, finding them to be considerably less skillful than the 8.MReV students.

Keywords

edX · Item Response Theory · learning gain · online learning