The Ideal in Evaluating and Improving Teaching Effectiveness

Quality teaching yields quality learning, and important step in improving teaching is evaluating teaching success and changing teaching strategies based on that evaluation. But what is the best way to evaluate a teacher's effectiveness? And what data sources can illuminate the path forward? the I recently had a conversation with my good friend John Louviere at USU, and through our mutual brainstorming, we generated the following data sources for effective teaching evaluation. The idea here is to utilize the principle of triangulation - the more data points you can use in your analysis, the more realistic picture you are able to paint of the situation. This in turn gives one a better process and method for moving forward.

The ideal data sources for effective teaching evaluation, particularly in higher education.

Teaching Evaluation Data Points
To make a truly holistic evaluation of a faculty member's teaching effectiveness, the following data points should be analyzed:

• Teacher Observations
• Student Teaching Evaluations 
• Formative Evaluation Tools
• Course Design Principles Rubric
• Course Design Quality Rubric
• LMS Data Analytics

Creating and Implementing Teaching Evaluation Tools

To create and implement effective data analytics tools, the following general phases should be followed. These phases should be used for each of the six data points described above. 

1. Identify and Synthesize Standards of Excellence - Determine based on the literature what the ideal is for each tool. For example, what should an observer find in the ideal teacher? This phase should include a thorough review of literature to find the research-based best-practices and should also include a clear synthesis of those practices into a coherent whole.
2. Develop Reports, Tools, and Rubrics Based on Standard - Create methods for gathering data on how well the standards are being used by the teacher. These can be rubrics, checklists, data analytics reports, and other tools. The goal is to have a way to efficiently and effectively gather data for each data point. For example, a teaching observation form would be developed to guide the observation of a teacher.
3. Gather Formative Feedback and Perform Validity Testing - Get sufficient expert feedback on the drafted tool or report and refine it based on the feedback. For example, have an expert or a scholar of teaching review the teaching observation form and give you feedback on how to refine it.
4. Pilot Testing - Conduct a pilot test of the tool and gather formative feedback on how well it functions and on the quality of the data being generated. For example, have someone use the teaching observation rubric to observe teaching and gather data on how well it worked and on whether the data is useful.
5. Implementation - After the tool has been refined and improved, implement it. This should inherently include the gathering of data on how it is being used and on the outcomes of its use.
6. Summative Evaluation - After the tool has been implemented, gather data on its effectiveness and draw conclusions from its use. Publish the results of the findings and create plans to move forward based on the results of the tool.  

Steps for establishing, implementing and evaluation tools, rubrics, and reports for improving teaching effectiveness.

Interactions of Data 

There will likely be a strong correlation between the many of the measures in each of these tools. For example, the use of effective design principles could correlate strongly to positive reports in other measures. The key is to create the tools, gather data on their use and results, and look for ways to further analyze and improve teaching effectiveness. As data is gathered, we will likely be able to determine what measure(s) are the strongest predictors of successful student learning, and this in turn gives us increased power to develop and improve future teaching evaluation and effectiveness. And in the increasingly competitive higher education environment, the universities who use data to help students succeed will both survive and thrive.

So, what do you think? Am I missing something? What else could be included in this? I believe that the effective establishment and use of these strategies will lay the foundation for increased and continued excellence in teaching at any university.

ADDIE Process- Evaluation

This post is part 5 in a 5-part series on the Phases of the ADDIE Process of Instructional Design. 
<< Previous post: Implementation

A critical phase in the ADDIE Process of instructional design is evaluation. To evaluate means to judge effectiveness.  This "phase" is a little misleading, because in reality you should be continually evaluating your work at every phase through formative evaluation, described below 

 Summative evaluation is performed to determine how effective a given piece of instruction is at improving student learning and usually takes place after a piece of instruction has been implemented.

Gathering Formative Feedback
A formative evaluation is performed to solicit feedback and guidance during the design process with the purpose of improving and refining the instruction. Formative feedback is gathered during this process, and there are several important things to remember: 

• Gather data iteratively and at every step in the design process. You should make sure you are on the right track as you move through the design phases.
• Focus on refining and improving instruction. Your goal should be to make the instruction more effective, more efficient, and more engaging for the students.  Gather feedback from many different sources, including:

1. Experts. Ask people who know the content to review your instruction. This will help ensure that the content is accurate.
2. Designers. Ask other instructional designers to review your instruction and give you feedback on how you could make it more clear and effective. Instructional design is the creative application of research-based principles, and creativity can be enhanced through collaboration and brainstorming.
3. Editors. Have a course editor review your materials to help improve their quality. This could be seen as part of the design process, but I mention it here because it is formative in nature and improves the quality of the instruction.
4. Learners. Test your instruction on your learners to see how it is received. Your instruction should be tailored to the needs of your target audience, and their feedback will help you improve the quality of your instruction.

In our MS Degree at Franklin University, we follow a stronger formative evaluation process to improve the quality of each of our courses. Each course undergoes at least two revisions based on data gathered from each of the sources described above. We see a dramatic increase in student satisfaction with our courses as we continually refine and improve them.


Here is an excellent video providing more ideas on evaluation in instructional design:

The Problem of Perfection in Instructional Design

Many instructional designers want to create the most effective, engaging product possible. I know I do, but deep down somewhere in my psyche, I believe that I subconsciously want to create a "perfect" unit of instruction, something that will help each and every student learn, something that manages cognitive load effectively, and help students acquire knowledge and skills that are fully transferred to real-world situations. Unfortunately, I have seen some instructional designers become obsessive about the quality of their work, to the point of losing sleep and generally becoming unhappy and unpleasant people.

I have realized over several years of teaching, training and designing instruction that instructional perfection is not possible. I may be burned at the scholarly stake for saying this, but here are a few considerations:

1. We design under constraints. We all have limitations of time, resources, money, expertise, tools, etc. Without any one of these, we will likely never have all we need to obtain instructional perfection.
2. All learners are different. Even if we design and develop a fabulous piece of instruction, our learners all have different backgrounds, different experiences, and different skill-levels. This means that what works "perfectly" for one learner likely won't for another.
3. Perfection is a mirage. The idea of perfection is not necessarily a reality- even when we meet 100% of the criteria for an instructional product, there is still always more that could be done. Perfection is, from a design perspective, literally unattainable.

Now, having made this (perhaps heretical) assertion, I believe that as designers, we should strive for excellence in our work, but must learn to be satisfied with what is possible given our constraints. The classic Whinston Churchill quote rings true: "Do what you can with what you have where you are."

In striving for excellence, it should be noted great deal of joy can be derived from continually refining and improving a piece of instruction. As we develop our curriculum in the IDPT Master's Program at Franklin University, we formatively evaluate each course and determine how we can refine and improve them. This means restructuring sequence and assignments, eliminating or clarifying confusing materials, providing worked examples, supportive multimedia, templates, and tips to guide students toward learning success.

Working to improve a course should be seen as a pleasure, done for the love of instruction and not as the result of some compulsion to work for instructional perfection. Instructional design is a creative endeavor- it should be seen as the creative application of research-based principles toward the goal of helping learners learn.

Let us move in our design toward excellence, avoid the mirage of instructional perfection, and find satisfaction in the principles and practices of instructional design.