Article: Evaluating Online Course Quality: Teaching Evaluation Using First Principles of Instruction

Pre-publication draft, reference information at end of post. 
To access PDF copies of this and other articles, visit my Academia.edu page.

Authors:

Joel Gardner
Max Cropper
Joanne Bentley

 Abstract

In a recent online class teaching the evaluation of online courses, Merrill's First Principles of Instruction were utilized to teach students how to evaluate online courses.  Students were taught how to use multiple online course rating rubrics, including Merrill's 5-Star Instructional Design Rating Form. A description of the course, the unique instructional methods and the outcome of the course is given, including recommendations for teaching effectively in an online environment. 

Background

Effective instruction is important, and because this is as true in online courses as it is in the classroom, many have recently taken a close look at the quality in online courses (Hirumi, 2005; Sherry, 2005). One important method for identifying the quality of online courses is by evaluating the effectiveness of the instructional strategy.  This article will discuss how the instructors utilized First Principles of Instruction (Merrill, 2002, 2007b) to teach a hybrid course on evaluating online courses.  The benefits of using First Principles of Instruction based on this case are included.
 

First Principles of Instruction

Recognizing the plethora of instructional design theories, Dr. David Merrill determined to establish basic principles of effective instruction.  To do this, he reviewed a number of instructional theories to identify and incorporate instructional principles found common among those theories. Merrill then abstracted from these theories what he calls First Principles of Instruction, a set of interrelated prescriptive instructional design principles (Merrill, 2002).

These First Principles of Instruction are outlined briefly below. Merrill asserts that learning is facilitated when:

• learners are engaged in solving real-world problems and real-world whole tasks. 
• existing knowledge is activated as a foundation for new knowledge. 
• new knowledge is demonstrated to the learner. 
• new knowledge is applied by the learner. 
• new knowledge is integrated into the learner's world (Merrill, 2002). See Figure 1.
  Figure 1. First Principles of Instruction diagram (Merrill, 2002).

Since first writing First Principles, several authors have written on its validity and usefulness as an instructional strategy (Collis, 2005a, 2005b; Romiszowski, 2006).  Instructors chose this strategy as the method for the course based on A detailed description of each of these principles follows, including a description of how these principles were utilized in teaching the course.

Teaching Instructional Evaluation With First Principles

The course was designed to teach students to evaluate online courses using a variety of rubrics, with an emphasis on Merrill’s 5 Star Instruction rating form.  It was taught in a hybrid format, approximately 30% of the course in-class, with the majority of the instruction online through Blackboard Vista and Macromedia Breeze. The goal for the course was to teach students to effectively evaluate online course quality.

Task-Centered

In designing this course, we decided from the beginning to make this course problem- or task-centered(Merrill, 2002).  Merrill describes these problems or tasks as specific, authentic, complete real-world tasks. If, possible they should be personal to the learner.  Merrill is careful, however, to point out that problem-centered instruction is not the same as problem-based learning (Merrill, 2007b). Although the kinds of problems utilized in PBL are also those that are valued in the real world (Savery, 2006), the way that those problems are presented and solved is quite different.  Merrill emphasizes the presentation and solving of increasingly complex problems (Merrill, 2006b), problem-based learning deemphasizes the demonstration of the problem and advocates the collaborative solving of ill-structured problems (Barrows, 1996; Savery, 2006).

In this case, the instructors were very careful to follow Merrill's lead and base the course primarily on the real-world task of evaluating actual online courses, a task which was very useful to the students.

In keeping our instructional strategy task-centered, we attempted to present what Merrill calls a progression of tasks, moving from simpler to more complex tasks (Merrill, 2006b). The students began by using the easier rubrics and later moved to the more difficult and central 5 Star Instructional Design Rating form and finished by giving a full evaluation report. This presentation fulfilled Merrill's recommendation of "going public with new knowledge," (2005) a key part of Integration, discussed later in this article.

Activation
Merrill recommends beginning a course by activating students prior learning regarding the topic or task (Merrill, 2002).  Activation takes place with students recall, describe or demonstrate relevant prior knowledge (Merrill, 2006a). To activate students’ prior knowledge in this course, instructors began by discussing the basics of evaluation and allowed students to discuss their prior learning and experience with instructional design and evaluation. Because the students had had several courses on designing instruction, this principle was accomplished with ease. 

Demonstration

Merrill asserts that learning is facilitated with new knowledge is demonstrated to the learners (Merrill, 2002). Proper demonstration enables learners to observe a demonstration of the skills to be learned (Merrill, 2006a). To facilitate the training process, the instructors for this course taught the first day of class face to face. After activation of students' prior learning and understanding of instructional evaluation, instructors immediately began demonstrating to the students how to evaluate a course. Instructors showed students a course to be evaluated, went through the process of evaluating that course and explained why specific decisions were made in that process.  Students were also allowed to ask questions for clarification of decision-making rules. This focus on demonstrating the whole task enabled students to see how the whole task was to be accomplished.

Application

The third principle of instruction is application, wherein students apply their newly acquired knowledge or skill (Merrill, 2006a). Merrill recommends that instructors provide successively less guidance with each subsequent task until learners are completing the tasks on their own (Merrill, 2006a). After instructor demonstration of how to evaluate a course, students were given the opportunity to evaluate a second course using the assigned evaluation rubric.  To help facilitate this process, the instructors gave them some assistance at the beginning of the second course evaluation and allowed the students to do more on their own until the students were rating courses without any instructor assistance. The students then worked with each other to achieve interrater reliability.  This process of students evaluating a course and then achieving interrater reliability was repeated yet again with little to no instructor guidance to ensure reliability.

Throughout the remainder of the class, students were assigned to rate 10 courses on their own. These students worked with each other throughout the course to ensure that they still had interrater reliability for each of the ratings. Finally, in a teleconference held a few weeks before the end of the course, Dr. Merrill trained the students on how to use of his latest 5-Star Instruction rating form, based on First Principles of Instruction (Merrill, 2007a).

Integration
Finally, Merrill recommends having learners integrate their new knowledge into everyday life (Merrill, 2006a). Since the completion of the course, some of the students have already begun using the skills and knowledge the gained from the course.  One student began evaluating courses for the company where he was interning, and now works for the company full time.  Another student used the skill to redevelop a university grant-writing course.

Another aspect of integration includes having students reflect on and publicly demonstrate their new knowledge or skill. (Merrill, 2006a). To facilitate this integration, students were required to read and summarize several articles on online course evaluation. And on the final day of class, each student presented an analysis of a selected course they evaluated during the class, along with with a written report of their evaluation.  

Discussion
Something interesting occurred when the students had learned about First Principles of Instruction near the end of the course. After students had just been trained by Dr. Merrill on how to use the 5 Star Rating form, they began commenting about how the 5 Star rubric seemed to be much more central and meaningful than the others. Having read Merrill's latest articles, and participated in the teleconference, they concluded that Merrill's rubric, which focuses entirely upon core instructional strategies and sub strategies, should be valued more highly than the peripheral polish associated with traditional standards of quality associated with the other rubrics.

The differences in emphasis among the rubrics was of interest to the instructors as well. The Southeast Regional Education Board (SREB) rubric, Texas IQ, and WebCT evaluation rubrics are those used for this particular study, along with Merrill’s 5 Star Rating. While Merrill’s rating provides balanced focus across his five principles, other rubrics provide extensive focus in certain areas. For example, the Texas IQ rubric has 9 questions about the syllabus and course requirements of a course. The WebCT rubric has six questions about collaboration. The SREB rubric includes four questions on collaboration five questions on practice and feedback and very few questions about other topics. See Figure 2.

This article discusses the development and pedagogical decisions that went into creating an evaluation course within the field of Instructional Technology. Like Evans, Beyer, & Todd, (1988) we have found that “evaluators have too often taken for granted the tacit assumptions underlying…” [teaching evaluation], “using these to determine the criteria for evaluating success” of a particular course. We would add that the teachers of evaluation should also be more explicit about the values they use to determine course content.

Worthen & Sanders suggest that evaluation differs from traditional research in that evaluation is trying to assess the value or social utility of something rather than just discover knowledge about it (pg, 30, 1987). In teaching this course, we attempted to balance our acknowledged bias towards Merrill's approach by having the students use a variety of instruments to evaluate the same courses. To a large extent, each of the evaluation instruments chosen for use in this class emphasized instructional strategies. However, each valued instructional methods differently as evidenced by the types and frequency of questions asked about different instructional methods. 

Using a variety of instruments with different values allowed us to demonstrate to the students how differently value-based evaluation of online course quality can be conducted depending on the instrument used.  Psychometricians are well aware that no matter how carefully constructed, the creators’ philosophical paradigm and personal values creep into instrument construction. While the public may naively think that evaluation rubrics are neutral or unbiased. As students reflected on their experience using each instrument they could compare and contrast the judgment of quality/merit as determined by each instrument. 

Summary 

Based on this case, the instructors recommend using a task-centered approach for teaching instructional evaluation. Students seem more engaged in the subject matter and better able to integrate the new knowledge to practice.  In addition, when Merrill's 5 Star Rating form is compared with the other rubrics, his focus dramatically eclipses the peripheral criteria of the other rubrics.  Making instructional strategies that support your instructional goal your primary focus will bring about greater student learning.

The use of First Principles of Instruction also appears to increase the effectiveness of online instruction. Where possible, instructors think it more effective to teach evaluation using real world tasks as the center of the instructional strategy.

This is a pre-publication draft of an article previously published in the Midwest Journal of Educational Communications and Technology in 2008. Feel free to refer to and use these materials, just be sure to use the reference below when citing the publication:

Gardner, J., Bentley, J., & Cropper, M. (2008). Evaluating On line Course Quality: Teaching Evaluation Using First Principles of Instruction. Midwest Journal of Educational Communication and Technology, 2(2), pp. 1-7. Accessed online at http://www.wiu.edu/users/iaect/MJECT/MJECT_V2_N2.pdf.