Yet more on evaluation

The ILS Coalition conference was last week, and it kind of got me going, so there may be a few posts this week. I promised earlier to talk in more depth about evaluation for system and the topic came up at the conference. Folks have waited avidly and long enough for the answers. Hating to disappoint anyone, here they are.

I guess the first question is why we need an evaluation. Here are three reasons:
1. To improve the quality of the system,
2. To eliminate waste in the system,
3. To learn how policy decisions are connected to outcomes.

California's statutes which set out the goals and methods of the system set some criteria for how quality is defined. The system succeeds when it:
A. Provides support for the choices clients make as to how they live and which goals they pursue,
B. Allows people with developmental disabilities to live lives similar to those experienced by their non-disabled peers (this is typically, but not necessarily described as integration of clients into the broader community),
C. Preserves the health and maximize the safety of clients.

Since evaluation becomes the set of incentives that professional participants face, the system that works for California should respond to the needs numbered 1-3, while incenting the mission of the system described in A-C. To accomplish all of this, the following should be elements of a statewide system for evaluation:

Client-centeredness: While the evaluation has to begin with valid outcomes which will make sense in the aggregate, the value of the measurement should depend on the stated goals of the individual client. Some of the waste in our system comes from supporting individuals to accomplish goals that don't interest the client. Sure, I agree, everyone should work and participate in the greater community but its awfully easy to lose an unwanted job or get kicked out of a club you hate to see. The statewide evaluations can and should aggregate the outcomes the clients choose to pursue rather than measure the success of clients in achieving statewide policy goals. I bet I write a better explanation of this another day.

Reasonable Expectations: A baseline level of accomplishment should be predicted and the outcomes judged against that baseline. This doesn't suggest low expectations, it refers to the idea that clients who face greater challenges should be as desirable to serve as people who are more typical of the population at large. No-one can defend planning for people based on traits like measured IQ, assessed functional abilities or behavioral history. Nonetheless to extend proper credit and reward audacity, baseline expectations can and should be adjusted with standard measures of ability. This can not only help make more challenged clients more attractive to serve, but may ultimately allow clients and their family members to identify agencies that may succeed especially well (or fail spectacularly often) with similar challenges to theirs.

Universality: Evaluations should focus on every client every year and reflect on every program they used and the regional center they're with. In order to identify successful strategies, every program should be measured for success and diverse cohorts should be identified. With 200,000 participants an annual survey of client outcomes can quickly separate what is working with whom from what is waste. Furthermore, the potenial exists to make service and agency selection an informed choice which it isn't today. The faster successful programs and strategies are identified and connected to real people, the less waste and failed agencies will tie up scarce resources.

Transparency and availability: The means of assessing success and the scores achieved by agency, program type and regional center need to be accessible to everyone. This is where there's a pragmatic difference between California's private sector system and other states' self-run systems. In a single system, the goal of evaluation is to measure the success of the overall system. California's system depends on the private sector to provide and clients to select the services and supports which will succeed for each individual. The best system for eliminating waste would be to insure that people considering services have the means to identify the right supports and providers through clear, transparent and easily available information.

OK. That was probably arcane and boring even for this weblog. If you've read this far, I probably owe you lunch.