The Ideal in Evaluating and Improving Teaching Effectiveness

Quality teaching yields quality learning, and important step in improving teaching is evaluating teaching success and changing teaching strategies based on that evaluation. But what is the best way to evaluate a teacher's effectiveness? And what data sources can illuminate the path forward? the I recently had a conversation with my good friend John Louviere at USU, and through our mutual brainstorming, we generated the following data sources for effective teaching evaluation. The idea here is to utilize the principle of triangulation - the more data points you can use in your analysis, the more realistic picture you are able to paint of the situation. This in turn gives one a better process and method for moving forward.

The ideal data sources for effective teaching evaluation, particularly in higher education.

Teaching Evaluation Data Points
To make a truly holistic evaluation of a faculty member's teaching effectiveness, the following data points should be analyzed:

• Teacher Observations
• Student Teaching Evaluations 
• Formative Evaluation Tools
• Course Design Principles Rubric
• Course Design Quality Rubric
• LMS Data Analytics

Creating and Implementing Teaching Evaluation Tools

To create and implement effective data analytics tools, the following general phases should be followed. These phases should be used for each of the six data points described above. 

1. Identify and Synthesize Standards of Excellence - Determine based on the literature what the ideal is for each tool. For example, what should an observer find in the ideal teacher? This phase should include a thorough review of literature to find the research-based best-practices and should also include a clear synthesis of those practices into a coherent whole.
2. Develop Reports, Tools, and Rubrics Based on Standard - Create methods for gathering data on how well the standards are being used by the teacher. These can be rubrics, checklists, data analytics reports, and other tools. The goal is to have a way to efficiently and effectively gather data for each data point. For example, a teaching observation form would be developed to guide the observation of a teacher.
3. Gather Formative Feedback and Perform Validity Testing - Get sufficient expert feedback on the drafted tool or report and refine it based on the feedback. For example, have an expert or a scholar of teaching review the teaching observation form and give you feedback on how to refine it.
4. Pilot Testing - Conduct a pilot test of the tool and gather formative feedback on how well it functions and on the quality of the data being generated. For example, have someone use the teaching observation rubric to observe teaching and gather data on how well it worked and on whether the data is useful.
5. Implementation - After the tool has been refined and improved, implement it. This should inherently include the gathering of data on how it is being used and on the outcomes of its use.
6. Summative Evaluation - After the tool has been implemented, gather data on its effectiveness and draw conclusions from its use. Publish the results of the findings and create plans to move forward based on the results of the tool.  

Steps for establishing, implementing and evaluation tools, rubrics, and reports for improving teaching effectiveness.

Interactions of Data 

There will likely be a strong correlation between the many of the measures in each of these tools. For example, the use of effective design principles could correlate strongly to positive reports in other measures. The key is to create the tools, gather data on their use and results, and look for ways to further analyze and improve teaching effectiveness. As data is gathered, we will likely be able to determine what measure(s) are the strongest predictors of successful student learning, and this in turn gives us increased power to develop and improve future teaching evaluation and effectiveness. And in the increasingly competitive higher education environment, the universities who use data to help students succeed will both survive and thrive.

So, what do you think? Am I missing something? What else could be included in this? I believe that the effective establishment and use of these strategies will lay the foundation for increased and continued excellence in teaching at any university.

Effective Strategies for Making Decisions

We all make decisions. At work, within our families, and in our communities, the decisions we make and the actions we take have a profound impact on our own success and on the success of the organizations in which we work. There are several approaches to decision-making that are promoted in popular literature. But which is the best way to make decisions? In this post I will describe what I see as the three major categories or types of decision-making and will show how these can be used to make good decisions.

As I discussed in a previous post, there are three general approaches to decision-making: (1) systematic process approaches, (2) intuitive approaches, and (3) principle-based approaches.

In instructional design, systematic process approaches include things like the ADDIE Process, The Dick and Carey Model, and other systematic processes for creating instruction. Intuitive approaches are more creative in nature and employ what has been called The Creative Spirit of Design. Principles-based approaches assume that systematic processes produce generalizable results, and that we can base our design on the assumptions embedded in the principles. (I describe instructional principles in detail in this previous post).

Combine these Approaches

I would submit that effective decision-makers intuitively combine each of these approaches in the decisions they make. Thinking systematically enables the individual to use scientific forms of thinking and reasoning, and without this the decision will likely be flawed. For example, when I redesign a course in my work at Franklin University, I often review and analyze data related to that course - student feedback, faculty comments and concerns, the course materials, and trends in the field - so that I can have a sound basis upon which to redesign the course. But in that redesign, I can also rely on principles to guide my work. I assume that applying these research-based instructional principles will produce the optimal learning environment, so I apply them to the course I am redesigning. Finally, in any given course redesign, I find that I must use my intuition and creativity to make some of the decisions that go into the design and development of the course. If there is no clear way to develop an assignment or a piece of multimedia, I do my best to creatively apply what I believe will help students learn effectively.

Previous Post: 6 Principles for Using Data to Make Decisions

Goals and Decisions
To make a good decision, you must have a goal or an outcome. If you know what you want to happen, then you are already on the right track. Sometimes the decision is to simply decide what your goal is. If you can't agree on a goal, then you are probably in trouble.

Context is Crucial
It should be noted that these three decision-making approaches all occur within a specific context. The nation, state, city, organization, college, department, and workspace a person is working in can all significantly influence the decisions being made. A good decision-maker is very aware of these contexts and considers them when applying all three decision-making strategies.

Decisions are Often Not "Right and Wrong"
In making decisions at work and in life, there is often no "right" answer.* There may be literally hundreds of ways to address a problem or work toward a goal, and you eventually have to make a decision and see what happens. But these decisions can be measured and held against a standard of success, and if the decisions (and their actions) lead to the desired result or goal, then it can be held up as a good decision. As a bonus, you can often learn from your experiences adjust your actions and decisions later to move you closer to your goal.

Previous Post: Leadership and Effective Decision Making

Use These Strategies!
I recommend using each of these strategies. The magic, of course, lies in when an individual should use which strategy, and through time an experience, you will develop your own approach. As long as you are thinking systematically, identifying and using principles effectively, and creatively applying your knowledge, you will make effective decisions that will positively impact your life and your work.

*I want to make clear that I am not promoting the notion that there is no right or wrong, or no good or evil. There are some decisions (and their subsequent actions) that I believe are most certainly wrong. I refer here to those decisions which do not have clearly right or wrong paths.

Decision-Making in Instructional Design (and Life)

How Do You Make Decisions?
How do you make your decisions? What do you do before you decide something? This is an important question, because the decisions we make determine our destiny in life. And if we want to enjoy all of the blessings that life has to offer, we must examine how we approach decision-making so that we can improve our decision-making.

In previous posts, I have written about principles for using data to make decisions, as well as leadership and decision-making. In this post, I will continue my discussion on decision-making and will relate it to instructional design and to life.

Decisions can be grouped as lower-level decisions (reactive) and higher-level decisions (proactive). Based on my experience, within these levels are several different approaches, which I describe below.

Lower-level Decision-Making (Reactive)

• Reaction - the decision is based solely on the compulsive nature of the environment. Design changes only occur when the designer is compelled to design.
• Rule-following - the designer uses a checklist approach to design. He does what others have done before him and thinks little.
• Social decision-making - the designer allows social influences to determine the design. This is heavily related to Reaction approach. 

In each of these lower-level decision-making approaches, the designer actually does very little design thinking.

Higher-Level Decision-Making (Proactive)

• Systematic - in this approach, the designer gathers data to support each phase of the design process. This approach is more scientific in nature and relies on critical thinking.
• Principles-based - in this approach, the designer bases decisions on principles of instruction. These principles are formed through their own instructional practice as well as through their exposure to research.
• Intuition - this approach uses the creativity of the designer. The designer creatively applies or ignores systematic or principles-based approaches. 

In my experience, when good design occurs, a lovely combination of all three of these approaches emerges. A good designer knows when to be more systematic and when to use principles instead. A good designer knows how to add his or her creativity to the work, and this creativity is best used when it employs principles in unique, tailored ways.

Use the Higher Levels
We should make our decisions based on the higher levels of decision-making. Very often, these approaches are more difficult, time-consuming, and are sometimes unpopular. With the ever-constant pressure to "get the work done," we are often tempted to revert to the more reactive approaches.

Progression in Decision-Making Approaches
People can progress in their decision-making approaches. We all have a tendency in our youth to employ the reactionary approach. But over time, we can develop the capacity to subdue the reactionary approach and develop disciplined habits. These habits enable use to continue on up the progression until we are creatively combining higher-level approaches to achieve success.

A Note on Habits and Decision-Making
The old axiom states that "excellence is a habit." I extend this by saying that "Design Excellence is achieved through excellent habits." Habits can empower us in our work, or they can drag us down. The lower-level, reactive approaches are easy to form because they are the path of least resistance and require little thinking. The higher-level, proactive approaches are more difficult to form because they are difficult and require lots of thinking and feeling.

Organizations and Decision-making
 It should be noted that these kinds of decision-making approaches are evident in organizations. Have you ever worked in an organization that is totally reactive? That does what it does because that is how it does what it does? That encourages its people to follow the rules, even when it makes absolutely no sense to do so? What about a proactive organization that gathers data to make decisions? That runs based on solid principles and allows its employees to use their creativity to bring about success? Yes, these decision-making approaches certainly apply to organizations.

Designing Life
So, where do you fall in your own decision-making as an instructional designer? What about your decisions in life? Either way, you are designing. You design instruction by making effective decisions. You design your life the same way. And the fundamental truth is that proactive approaches will always yield greater success that reactive.This can apply to organizations, as well.

6 Principles for Using Data to Make Decisions

Using Data to Make Decisions
If you want to succeed, you must be willing to base your decisions on reality. But work in the knowledge society can be so complex that making the right decision becomes extremely difficult.

In my experience in the world of competitive education, we must be goal-oriented and must be as efficient and effective as possible. But how is this accomplished? How can we ensure that our activity will actually produce the results we desire? I believe that appropriate use of data is the single-most important component to be used in making effective decisions. (And no, I do not mean the fictional Star Trek character Data, though it would be awesome if we could use him to make decisions...).

Related Post: Leadership and Effective Decision-Making

Below are 6 principles for making effective decisions based on data:

1. Have goals. What does your organization want to accomplish? These goals should be measurable, something that you can know you have accomplished. They should be concrete enough to align with your organization's mission but should be flexible enough to adapt to the environment in which you operate.
2. Identify performance measures that indicate (1) progress toward your goals (sometimes called leading indicators), and (2) accomplishment of your goals (lagging indicators). Lagging indicators will likely not change often, for example, the number of persons served, number of students graduating, amount of money made, etc. No matter the purpose of your organization, there are specific goals and these goals must be measurable and measured. How you reach your goals might change, and therefore your leading indicators might change as well.
3. Automate measurement of performance measures. Decisions are most effective when based on reality, so it is crucial to make the data readily available. This automated measurement can be done through technology or through people, and often it is a combination of both.
4. Organize the data in meaningful, relevant ways so that they can be interpreted, understood, and linked to our goals and to our decisions. Data is only useful if it appropriately represents important aspects of reality (i.e. your goals) and enables effective decision-making.
5. Use the data. You can use data to perform several key functions, including (but not limited to) the following actions:
1. Identify whether you are moving toward and reaching your goals.
2. Identify where you are having performance issues. These issues can be articulated as performance gaps, meaning the gap between desired performance and actual performance.
3. Identify what is causing the gap. This data should come from many sources and should be comprehensive in nature. You will often have to gather additional data at this stage to get a real picture of what is happening. This data can be gathered through observation, interviews, focus groups, data mining, existing reports, etc. Be sure to gain data from all relevant sources and triangulate all sources to get a holistic, realistic picture of what is happening.
4. Identify what can be done to close the gap. This part is usually a pretty easy step because it should be based on the cause of the gap. Again, you may need to gather additional data from the sources identified above.
6.  Make a decision and implement it based on the results of your data analysis. Again, if you do not base what you do on thoughtfully acquired, thoroughly analyzed data, then you will be much more likely to make poor decisions that could have potentially disastrous consequences.

As I wrote about in a previous post, humans have the tendency to jump to conclusions, to do what seems best based on instinct or previous experience. And while this approach can often be effective, it does not always work. We must be willing to gather and use sound data to make our decisions, or we will be left in the proverbial dust by those who are willing to do the work required to make good decisions.

Principle-Based and Data-based Decision-Making
I want to be clear that I am not advocating an arduous process of data-gathering and analysis every time a decision must be made, which would be rather counter-intuitive. As human beings, we have the capacity to generalize our experiences into principles and best practices, which can guide effective decision-making. Experienced leaders and professionals can often intuitively discern effective decisions based on their experiences in similar prior situations.

Perhaps, then, we might use principles to guide our use of the data. Indeed, our mission, goals, and strategies can be based on principles and values and informed and refined by data. A combination of principle-based and data-based decision-making, therefore, becomes most powerful.

Thinking Systematically
Making decisions based on data aligns with the idea of thinking systematically, and it requires discipline and hard work to focus efforts on the systematic activities that will bring the results we desire. This systematic process can be applied in any setting and should be used when specific results are desired. For examples of how to apply systematic thinking, consider the following previous posts in which I share examples of thinking systematically:

1. The ADDIE Process (a systematic process for designing effective instruction)
2. Setting Professional Goals (I describe a systematic process for doing this)
3. A post describing the ISPI-adopted HPT Model (a systematic process for improving performance in business) 

You will note that these processes all follow the same basic pattern: (1) identify where you are and where you want to be; (2) plan out how to get where you want to be; (3) do your plan; (4) see how well your plan worked; and (5) adjust your plan and keep trying. This process is powerfully effective, and when data guides the decisions made at every step, success becomes much more likely.