Evaluation for Learning, Competency and
Performance
Part II: Questions from our
infoREADERs
I
promised a follow-on to last month?s general
remarks concerning Quality
measurements that foster performance and
competency. To this end, let me
respond at the same time to a number of readers who
made inquiries similar to
the following:
"Dear EPSScentral - I have not
been able to find much in the way of measurement
with respect to performance
other than return-on-investment (ROI).
Although ROI is important to document
and critical to success, I need more than comparing
pre- and post-tests to
justify true performance improvement. Any
suggestions?"
EPSScentral's
Response:
A
number of tools and techniques have emerged around
measurement for performance
support. Such measurements are really nothing
new: They are the
"Q"uality measurements a la Deming, such
as unit time and cost,
cycle time / cost and resource utilization.
For the manufacturing world,
such measurements were fairly straightforward using
the standard Quality tool
set (Check Sheets, Pareto Charts, Histograms,
Ichikawa diagrams (for
determining root cause), and more. Depending
on the performance issue,
you would measure things like total time to produce
a complete widget, unit
times for completing the sub-processes that comprise
the widget (unit times),
the costs involved, and examine these against
resource utilization
(percentage) and evaluate problems such as
bottlenecks, broken processes and
tasks that consume resources but are of no
consequence (e.g., "checkers
checking checkers"...busy work).
All
of this is fine for manufacturing, but in the world
of enterprise systems and
computer-mediated work, the corresponding units,
cycles and the like happen at
a much faster rate and in situations where the
business rules change
frequently. The same quality measures and
principles apply, but until
recently we have not seen tools capable of measuring
even the basics for
knowledge work within computer-mediated tasks.
More recently a set of Deming/Quality-based tool sets
have emerged for making proper measurements and
analyzing such work. They
include:
The
common principle is to capture system metadata at a
very granular level while
performers go about their business. Process /
workflow is inferred from
actual action while the data required to perform the
Quality measurements are
captured (what processes / sub-processes are
completed, aborted or broken;
associated times; costs; cycle stats
etc). Far superior to any
pre-and-post testing, these tools capture and
calculate actual performance
metrics, real-time, and analyze them against
expectations (criterion
referenced) or based on inferred best practices from
actual work being performed
(norm referenced). Statistics such as Rasch
can be applied to
"performance items" (in a manner similar
to ?test items?) and we
obtain similar distributions (e.g., standard normal
distribution of tasks in
the 47-70% range, associated with "perpetual
intermediaries."
We are seeing some truly useful and powerful
measurements and patterns emerging.
Just a few MeasureLive? results are shown
below based on remotely capturing
system usage across a group of users and
applications:
MeasureLive(TM)
Results. (Used with permission
www.measurelive.com.)
The
next generation of such tools would include coupling
the measurement tools with real-time
"sensing" tools that push appropriate
support to users as the quality
measures are being collected and analyzed. In
real time, you would then
create an adaptive performance-centered
system that self-corrects,
continuously increasing performance and
competency. In many respects, such
knowledge delivery is the Semantic Web vision.
Stay tuned!
Regards,
Gary
Dickelman
EPSScentral Announces Recipients of 2007 PCD Awards
