1. What were the measures used in this study?

          The meta-analysis was completed on 50 independent effects identified from empirical studies of online learning, found by a systematic search of the research literature from 1996 through July 2008, in hopes of discerning the effectiveness of online learning compared with face-to-face instruction; the presence of improved learning when online instruction is included along with face-to-face instruction; practices, when in conjunction with online instruction, produce more effective learning; and finally, conditions, when present with online instruction, sway learning efficacy.

          A parameter for the meta-analysis was the limitation to studies of Web-based instruction. Studies limited to video and audio-based telecourses or stand-alone, computer-based instruction were not included. The inclusion of only random-assignment or controlled quasi-experimental design studies and the examination of only objective measures of student learning also narrowed the scope of the meta-analysis.

1. How did the researchers define “better”?

          Outcome differences between students participating in online and face-to-face instruction were measured by the division of the difference between treatment and control means by the pooled standard deviation. Findings on average demonstrated a modestly better performance from students in online instruction than students in face-to-face instruction. A larger difference in performance was seen in studies contrasting conditions that blended elements of online and face-to-face instruction with conditions taught entirely face-to-face. Because of extra time and instruction in these blended cases, the better performance cannot be attributed solely to media.

1. How did the researchers define “performance?

          The basis for all effects in the meta-analysis was objective measures of learning. Some examples include scores on standardized tests, scores on researcher-created assessments, grades/scores on teacher-created assessments or assignments, and mid-term or final grades or grade point averages. Exclusion criteria included student/teacher perception of learning or course quality, student/teacher affect, attendance, and level of satisfaction of student/teacher. Student or teacher/instructor self-report of learning was not considered a direct, objective measure. Outcome measurement also had to be obtained in the same manner across study and control groups, by the same test or demonstration method.

One of the most basic characteristics for classifying online activities is its objective—whether the activity serves as a replacement for face-to-face instruction (e.g., a virtual course) or as an enhancement of the face-to-face learning experience (i.e., online learning activities that are part of a course given face-to-face). This distinction is important because the two types of applications have different objectives. A replacement application that is equivalent to conventional instruction in terms of learning outcomes is considered a success if it provides learning online without sacrificing student achievement. If student outcomes are the same whether a course is taken online or face-to-face, then online instruction can be used costeffectively in settings where too few students are situated in a particular geographic locale to warrant an on-site instructor (e.g., rural students, students in specialized courses). In contrast, online enhancement activities that produce learning outcomes that are only equivalent to (not better than) those resulting from face-to-face instruction alone would be considered a waste of time and money because the addition does not improve student outcomes.