Evaluation and Accountability

So, now that POSS standards have been so thoroughly discredited that they probably won't re-emerge as a proposal until August, it seems like this window can be used to discuss meaningful reform which might meet one-or-both criteria of improving the lives of people with developmental disabilities and improving the taxpayer's return on their investment.

The biggest omission in California's system is a meaningful, useful (or even non-fictional) system of evaluation and accountability. In human-service systems around the world, governmental or private it is standard practice to measure outcomes. In other words, since we know why we do this work its worth asking if we're succeeding and how well. Measuring outcomes, when done correctly, can establish causal relationships between the work that is paid for and the results which are obtained. The system, the people the system serves, service providers and the state's taxpayers can all benefit by:
-Making available objective information about which agencies and programs may best serve an individual;
-Allowing for informed budgeting so that more money goes where it helps and less where it doesn't;
-Improving advocacy by allowing consituencies to demonstrate what is lost when funds are unavailable; and
-Providing a basis for all system stakeholders to recognize what works, what doesn't and how to improve.
-Plus, one Director of a small ILS agency will sleep better at night, providing for better management of that agency and less crankiness at meetings.

Although California's state government hasn't done much to make evaluation happen, that isn't because a lot of smart people haven't put good thinking to the task. Julie Jackson, Deputy Director at California's Department of Developmental Services has thought about, worked on and advocated for evaluating services systemwide. Dr. Barbara Wheeler at USC/UAP has worked on devising an evaluation model which fully integrates the concept of choice into outcome measurement (this work is ongoing due to the distraction and unreliability of a colleague of hers who nonetheless found the time to start a blog.) California Community Action Network (the predecessor to CDCAN, mentioned below) has often focused attention on the need for outcomes for those who receive services. A national Core Indicators Project is years underway and the Service Delivery Reform committee which met a few years back picked some domains and agreed to set-up a learning model of evaluation as long as there was nothing complicated about it. Assemblymember Keith Richman (R-Northridge) included a first step toward evaluation in a bill he sponsored (AB 2775)this session which is being held in committee.

So why is there no real evaluation or useful accountability in the system? One problem we're up against is the difficulty of making long-term investment during a budget crisis (can you call it a crisis in it's fourth year?) I suspect another is that during the Davis Administration, the state Department of Finance fell in love with the idea of standardizing and set that agenda. Schwarzenegger's DOF is carrying the same torch. Yet another challenge is that service providers, who have tended to be the organizers of much of the system's advocacy tend to feel pretty strongly that measuring the result of their work is a poor substitute for funding it better. Hopefully, if real organizing continues at the grassroots level, the people who deserve the best possible system will focus more on their need for support that really helps than the historic concentration on provider rates.

Mind you, doing evaluation correctly or even well takes a lot of work. One old saw is, "What gets measured gets improved" and it can be challenging to make sure you actually measure the things you want to improve rather than the things you wish to avoid. In my experience, Unicorns are rarer than well-designed evaluation systems which are rarer than trout. In the case of California's system of support for people with developmental disabilities you can reverse the first two terms.

Later posts will discuss some thinking about what would make an evaluation system useful and meaningful to this system.