Evaluation of Evidence-Based Practices in Online Learning
A Meta-Analysis and Review of Online Learning Studies
Means, B., Toyama, Y., Murphy, R., Bakia, M., Jones, K., Department of Education (ED), O., & SRI, I. (2009). Evaluation of Evidence-Based Practices in Online Learning: A Meta-Analysis and Review of Online Learning Studies. US Department of Education, Retrieved from EBSCOhost.
http://www.eric.ed.gov/PDFS/ED505824.pdf
Each of you please answer the following questions on the class wiki as indicated, although I want you to “talk/collaborate” with one another in the class as you answer these questions.
1. What were the measures used in this study?
This study compared results from multiple previous studies, attempting to determine if there were statistically significant differences between face-to-face, online and hybrid instruction. There were a total of 50 effects or contrasts (gathered from the previous studies) that were analyzed, including not only the results of the studies, but the structure, design and calculated effectiveness/accuracy of each study. Each of the previous studies was analyzed for variables categorized as practice, moderator or study that were then assigned a contrasting measure. These measures were then statistically analyzed for comparative commonalities using Cohen's average effect size measures.
2. How did the researchers define “better”?
Better was a defined by using Cohen's average effect size when calculated using the 50 contrasts as defined above (including a myriad of variables). The idea is that, by using a measure that can take into account the differences between the studies being analyzed, those differences can be minimized and common traits will come forward and be statistically significant.
3. How did the researchers define “performance?
Performance seemed to be based on fairly objective student learning outcomes as reported by each study. These performance measures were used as contrasts/effects in the meta-analysis.
I should note that better and performance were also used together often when discussing comparisons within a specific study. In such cases, both relate directly to objective measurements (typically grade-based).