Pre-publication draft, reference information at end of post.
To access PDF copies of this and other articles, visit my Academia.edu page.
Authors:
Joel Gardner
Max Cropper
Joanne Bentley
Abstract
In a recent
online class teaching the evaluation of online courses,
Merrill's First Principles of Instruction were utilized to teach
students how to evaluate online courses. Students were taught
how to use multiple online course rating rubrics, including Merrill's
5-Star Instructional Design Rating Form. A
description of the course, the unique instructional methods and the
outcome of the course is given, including recommendations for
teaching effectively in an online environment.
Background
Effective
instruction is important, and because this is as true in online
courses as it is in the classroom, many have recently taken a close
look at the quality in online courses (Hirumi, 2005; Sherry, 2005).
One important method for identifying the quality of online courses is
by evaluating the effectiveness of the instructional strategy.
This article will discuss how the instructors utilized First
Principles of Instruction (Merrill, 2002, 2007b) to teach a hybrid
course on evaluating online courses. The benefits of using
First Principles of Instruction based on this case are included.
First
Principles of Instruction
Recognizing the
plethora of instructional design theories, Dr. David Merrill
determined to establish basic principles of effective instruction.
To do this, he reviewed a number of instructional theories to
identify and incorporate instructional principles found common among
those theories. Merrill then abstracted from these theories what he
calls First Principles of Instruction, a set of interrelated
prescriptive instructional design principles (Merrill, 2002).
These First
Principles of Instruction are outlined briefly below. Merrill asserts
that learning is facilitated when:
Since first
writing First Principles, several authors have written on its
validity and usefulness as an instructional strategy (Collis, 2005a,
2005b; Romiszowski, 2006). Instructors chose this strategy as
the method for the course based on A detailed description of each of
these principles follows, including a description of how these
principles were utilized in teaching the course.
Teaching
Instructional Evaluation With First Principles
The course was
designed to teach students to evaluate online courses using a variety
of rubrics, with an emphasis on Merrill’s 5 Star Instruction rating
form. It was taught in a hybrid format, approximately 30%
of the course in-class, with the majority of the instruction online
through Blackboard Vista and Macromedia Breeze. The goal for the
course was to teach students to effectively evaluate online course
quality.
Task-Centered
In designing
this course, we decided from the beginning to make this course
problem- or task-centered(Merrill, 2002). Merrill describes
these problems or tasks as specific, authentic, complete real-world
tasks. If, possible they should be personal to the learner.
Merrill is careful, however, to point out that problem-centered
instruction is not the same as problem-based learning (Merrill,
2007b). Although the kinds of problems utilized in PBL are also those
that are valued in the real world (Savery, 2006), the way that those
problems are presented and solved is quite different. Merrill
emphasizes the presentation and solving of increasingly complex
problems (Merrill, 2006b), problem-based learning deemphasizes the
demonstration of the problem and advocates the collaborative solving
of ill-structured problems (Barrows, 1996; Savery, 2006).
In this case,
the instructors were very careful to follow Merrill's lead and base
the course primarily on the real-world task of evaluating actual
online courses, a task which was very useful to the students.
In keeping our
instructional strategy task-centered, we attempted to present what
Merrill calls a progression of tasks, moving from simpler to more
complex tasks (Merrill, 2006b). The students began by using the
easier rubrics and later moved to the more difficult and central 5
Star Instructional Design Rating form and finished by giving a full
evaluation report. This presentation fulfilled Merrill's
recommendation of "going public with new knowledge," (2005)
a key part of Integration, discussed later in this article.
Activation
Merrill
recommends beginning a course by activating students prior learning
regarding the topic or task (Merrill, 2002). Activation takes
place with students recall, describe or demonstrate relevant prior
knowledge (Merrill, 2006a). To activate students’ prior knowledge
in this course, instructors began by discussing the basics of
evaluation and allowed students to discuss their prior learning and
experience with instructional design and evaluation. Because the
students had had several courses on designing instruction, this
principle was accomplished with ease.
Demonstration
Merrill
asserts that learning is facilitated with new knowledge is
demonstrated to the learners (Merrill, 2002). Proper demonstration
enables learners to observe a demonstration of the skills to be
learned (Merrill, 2006a). To facilitate the training process, the
instructors for this course taught the first day of class face to
face. After activation of students' prior learning and understanding
of instructional evaluation, instructors immediately began
demonstrating to the students how to evaluate a course. Instructors
showed students a course to be evaluated, went through the process of
evaluating that course and explained why specific decisions were made
in that process. Students were also allowed to ask questions
for clarification of decision-making rules. This focus on
demonstrating the whole task enabled students to see how the whole
task was to be accomplished.
Application
The third
principle of instruction is application, wherein students apply their
newly acquired knowledge or skill (Merrill, 2006a).
Merrill recommends that instructors provide successively less
guidance with each subsequent task until learners are completing the
tasks on their own (Merrill, 2006a). After instructor demonstration
of how to evaluate a course, students were given the opportunity to
evaluate a second course using the assigned evaluation rubric.
To help facilitate this process, the instructors gave them some
assistance at the beginning of the second course evaluation and
allowed the students to do more on their own until the students were
rating courses without any instructor assistance. The students then
worked with each other to achieve interrater reliability. This
process of students evaluating a course and then achieving interrater
reliability was repeated yet again with little to no instructor
guidance to ensure reliability.
Throughout the
remainder of the class, students were assigned to rate 10 courses on
their own. These students worked with each other throughout the
course to ensure that they still had interrater reliability for each
of the ratings. Finally, in a teleconference held a few weeks
before the end of the course, Dr. Merrill trained the students on how
to use of his latest 5-Star Instruction rating form, based on First
Principles of Instruction (Merrill, 2007a).
Integration
Finally,
Merrill recommends having learners integrate their new knowledge into
everyday life (Merrill, 2006a). Since the completion of the course,
some of the students have already begun using the skills and
knowledge the gained from the course. One student began
evaluating courses for the company where he was interning, and now
works for the company full time. Another student used the skill
to redevelop a university grant-writing course.
Another aspect
of integration includes having students reflect on and publicly
demonstrate their new knowledge or skill. (Merrill, 2006a). To
facilitate this integration, students were required to read and
summarize several articles on online course evaluation. And on
the final day of class, each student presented an analysis of a
selected course they evaluated during the class, along with with
a written report of their evaluation.
Discussion
Something
interesting occurred when the students had learned about First
Principles of Instruction near the end of the course. After students
had just been trained by Dr. Merrill on how to use the 5 Star Rating
form, they began commenting about how the 5 Star rubric seemed to be
much more central and meaningful than the others. Having read
Merrill's latest articles, and participated in the teleconference,
they concluded that Merrill's rubric, which focuses entirely upon
core instructional strategies and sub strategies, should be valued
more highly than the peripheral polish associated with traditional
standards of quality associated with the other rubrics.
The differences
in emphasis among the rubrics was of interest to the instructors as
well. The Southeast Regional Education Board (SREB) rubric, Texas IQ,
and WebCT evaluation rubrics are those used for this particular
study, along with Merrill’s 5 Star Rating. While Merrill’s rating
provides balanced focus across his five principles, other rubrics
provide extensive focus in certain areas. For example, the Texas IQ
rubric has 9 questions about the syllabus and course requirements of
a course. The WebCT rubric has six questions about collaboration. The
SREB rubric includes four questions on collaboration five questions
on practice and feedback and very few questions about other topics.
See Figure 2.
This article
discusses the development and pedagogical decisions that went into
creating an evaluation course within the field of Instructional
Technology. Like Evans, Beyer, & Todd, (1988) we have found that
“evaluators have too often taken for granted the tacit assumptions
underlying…” [teaching evaluation], “using these to determine
the criteria for evaluating success” of a particular course. We
would add that the teachers of evaluation should also be more
explicit about the values they use to determine course content.
Worthen &
Sanders suggest that evaluation differs from traditional research in
that evaluation is trying to assess the value or social
utility of something rather than just discover knowledge about
it (pg, 30, 1987). In teaching this course, we attempted to
balance our acknowledged bias towards Merrill's approach by having
the students use a variety of instruments to evaluate the same
courses. To a large extent, each of the evaluation instruments chosen
for use in this class emphasized instructional strategies. However,
each valued instructional methods differently as evidenced by the
types and frequency of questions asked about different instructional
methods.
Using a variety
of instruments with different values allowed us to demonstrate to the
students how differently value-based evaluation of online course
quality can be conducted depending on the instrument used.
Psychometricians are well aware that no matter how carefully
constructed, the creators’ philosophical paradigm and personal
values creep into instrument construction. While the public may
naively think that evaluation rubrics are neutral or unbiased. As
students reflected on their experience using each instrument they
could compare and contrast the judgment of quality/merit as
determined by each instrument.
Summary
Based on this
case, the instructors recommend using a task-centered approach for
teaching instructional evaluation. Students seem more engaged in the
subject matter and better able to integrate the new knowledge to
practice. In addition, when Merrill's 5 Star Rating form is
compared with the other rubrics, his focus dramatically eclipses the
peripheral criteria of the other rubrics. Making instructional
strategies that support your instructional goal your primary focus
will bring about greater student learning.
The use
of First Principles of Instruction also appears to increase the
effectiveness of online instruction. Where possible, instructors
think it more effective to teach evaluation using real world tasks as
the center of the instructional strategy.
This is a pre-publication draft of an article previously published
in the Midwest Journal of Educational Communications and Technology in
2008. Feel free to refer to and use these materials, just be sure to use
the reference below when citing the publication:
Gardner, J., Bentley, J., & Cropper, M. (2008). Evaluating On
line Course Quality: Teaching Evaluation Using First Principles of
Instruction. Midwest Journal of Educational Communication and Technology, 2(2), pp. 1-7. Accessed online at http://www.wiu.edu/users/iaect/MJECT/MJECT_V2_N2.pdf.
