| |
Evaluation for
Learning, Competency and Performance
Part I: Why
does ADDIE have a silent
E?
For several decades
learning and performance
technologies have been grounded in Quality methods
(that's the
big Q Quality) such as Instructional Systems
Development (ISD)
and Performance Centered Design (PCD). These
methods evolved from
a number of sources, not the least of which was the
United States Department
of Defense, which recognized the value of
consistency, uniformity, addressing
root cause and the need to compress the time to
competency in relation
to achieving the mission. W. Edwards Deming
arguably led the Quality
revolution, transferring knowledge of how
organizations become greater
than the sum of their parts, seizing crises as
opportunities, continuously
improving, getting to root causes, providing tools
to address them then
encouraging and empowering organizations to excel
through practice.
The re-engineering revolution of the 1990s applied
Deming's manufacturing
successes to information technology, and our own
Performance Centered
Design methodology is a manifestation of Deming qua
knowledge management
(in many respects).
Critical to all
Quality methods are evaluation
protocol - formative, summative, continuous
evaluation, gap analysis
and the like. "In God we trust, all others
bring data," said
Deming. But when pressed for
details we find organizations
woefully deficient in measurement tools, techniques
and a commitment
to evaluation that ensures competency and continuous
improvement.
It is as if the quality process known as "ADDIE" -
Analysis, Design,
Development, Implementation and Evaluation - has a
silent E.
You would think that
project managers
would be eager to ensure the presence of a line item
and appropriate
budget for evaluation. Wasn't it Peter Drucker
who, in our lifetime,
pointed out that lack of attention to business
purpose and business
mission are the most important causes of business
frustration and failure?
How on this grey earth can you pay proper attention
to such things without
measuring? How can you roll out a training
program or a performance
enhancing initiative bent on reducing time to
competency without measuring
that which informs competency and putting a stop
watch to tasks to determine
"how long?" And what is the cost to
business of each
hour or day that employees - performers - are unable
to perform
and thus the organization is unable to achieve the
mission?
What is defined as
"good" in training
and performance initiatives too often reflect
secondary or irrelevant
criteria - like how authentic the simulation
appears, how "usable"
the interface is, how many employees checked the
course out of the LMS,
how many hours were "covered" or how the performers
filled out the
smile sheets. What about the organizational
mission and goal?
What about measuring the number of transactions that
were completed
successfully, how quickly and accurately proposals
are generated and
delivered by the sales force, how the close and
revenue rates improved
or how the call queue decreased, how many fewer
employees quit and
exactly how similar are employees accomplishing
tasks as compared with
the highest performers? As I said, the E in
ADDIE is silent, for
the most part.
The good news is that
tools and techniques
are evolving and finding their way to the
marketplace that focus on
process measurements in computer-mediated work.
This is precisely
what is necessary to properly apply the big Q to
knowledge work.
Was, or is, evaluation really a lost art?
Hardly. But our
heuristics and naive tools are simply insufficient
to gather and analyze
data at the rate required to keep our eye on
the business purpose
and mission. Companies have applied capture
technologies to model-driven
development tools for a few years now, with good
results. Now
they are taking a step back and realizing that more
depth is required
at the very tip of the capability, namely data
gathering and analysis,
now and live. We are finally at a
point where we can make
quick and accurate measurements that show a clear
relationship between
cause and affect, between evaluation and
ROI.
In part 2 of this
series we will examine
the emerging tools, look at case studies and perhaps
take a few small
steps toward turning some very bad habits
around. Until then just
imagine the possibilities!
Regards,
Gary J.
Dickelman
|