Patricia's Meta Analysis


Meta-Analysis Assignment 

 

A.  What were the measures used in this study

 

1.  Sample size

 

 2. Type of knowledge tested

 

3.  Strength of study design

 

4.  Unit of assignment to condition

 

5.  Instructor equivalence across conditions

 

6.  Equivalence of curriculum and instructional approach across conditions

 

7.  Examining the learning effects of blended or online learning vs face-to-face conditions.

 

8.  Independent effects.  [Each effect is the difference between the mean for the treatment group and the mean for the control group, divided by the pooled standard deviation.]

 

Examples of learning outcome measures for teacher learners (in addition to those accepted as student outcomes) included assessments of content knowledge, analysis of lesson plans or other materials related to the intervention, observation (or logs) of class activities, analysis of portfolios, or supervisor’s rating of job performance

 

They were looking for student performance data or overall academic data.  Performance in the sense of achievement is the overriding theme for this study.

 

 

B.  How did the researchers define “better”?

 

Although subtle I derive better based on the following: 

 

  1. Among the 50 individual study effects 11 were significantly positive, favoring the online or blended condition.  11 contrasts associated with online or hybrid instruction exceeds what one would expect by chance (pg xiii).

 

     2.   The meta-analysis found that on average, students in on line learning conditions performed modestly better than those receiving face to face instruction (pgs ix and xiv).

 

     3.  It is reassuring to note that, on average, online learning produced better student learning outcomes than face-to-face instruction… (pg 73)

 

     4.  Because these were not ‘runaway’ numbers the researchers gave a slight edge to online learning:  “the difference between treatment and control means, divided by the standard deviation was LARGER in those studies contrasting blended elements of online and F2F with conditions taught entirely F2F.   

 

     5.  They went on to say that “the POSITIVE EFFECTS associated with blended…”

 

     6.  “…blended instruction has been more EFFECTIVE

 

     7.  “… online learning is much more conducive to the expansion of learning…

 

     8.  “…strong support for blended learning applications,…” (pg xviii)

 

     9.  “…produced observed learning advantages…”

 

     10.  Zhao et al.  (2005) found advantages for blended learning  (combining elements of online and face-to-face communication) over purely online learning experiences, a finding similar to that of this meta-analysis. (pg 74) 

 

Because the numbers statistically speaking were not cut and dried or big/significant enough to highlight, per se, the researchers buried the advantage within the assessment and conclusions or in foot notes.  They did not allow it to stand out.

 

They aren’t confident of the hard outcome because there were so many variables involved.  For me, better is “better” even if based on a point or two differences.  The researchers wanted to know if one group/set performed ‘better’ than the other group/set by comparison.

 

C.  How did the researchers define “performance"?

 

      1.  The individual effect size estimates included in this meta-analysis ranged from a low of –0.80 (tendency for higher performance in the face-to-face condition) to a high of +1.11 (favoring online instruction). (pg 48)

 

     2.  Learning outcome findings were extracted at the individual level. Analysts did not extract group-level learning outcomes (e.g., scores for a group product). Too few group products were included in the studies to support analyses of this variable.

 

     3.  Concerning the learning effects of using alternative or multiple delivery platforms for online learning performance in this case is defined by:

 

The effect of student learner outcomes or enhanced learning, effectiveness of student learning, or bottom line - was learning enhanced by using one of the comparison standards?