Monday, July 26, 2004

Evaluation and Accountability

So, now that POSS standards have been so thoroughly discredited that they probably won't re-emerge as a proposal until August, it seems like this window can be used to discuss meaningful reform which might meet one-or-both criteria of improving the lives of people with developmental disabilities and improving the taxpayer's return on their investment.

The biggest omission in California's system is a meaningful, useful (or even non-fictional) system of evaluation and accountability. In human-service systems around the world, governmental or private it is standard practice to measure outcomes. In other words, since we know why we do this work its worth asking if we're succeeding and how well. Measuring outcomes, when done correctly, can establish causal relationships between the work that is paid for and the results which are obtained. The system, the people the system serves, service providers and the state's taxpayers can all benefit by:
-Making available objective information about which agencies and programs may best serve an individual;
-Allowing for informed budgeting so that more money goes where it helps and less where it doesn't;
-Improving advocacy by allowing consituencies to demonstrate what is lost when funds are unavailable; and
-Providing a basis for all system stakeholders to recognize what works, what doesn't and how to improve.
-Plus, one Director of a small ILS agency will sleep better at night, providing for better management of that agency and less crankiness at meetings.

Although California's state government hasn't done much to make evaluation happen, that isn't because a lot of smart people haven't put good thinking to the task. Julie Jackson, Deputy Director at California's Department of Developmental Services has thought about, worked on and advocated for evaluating services systemwide. Dr. Barbara Wheeler at USC/UAP has worked on devising an evaluation model which fully integrates the concept of choice into outcome measurement (this work is ongoing due to the distraction and unreliability of a colleague of hers who nonetheless found the time to start a blog.) California Community Action Network (the predecessor to CDCAN, mentioned below) has often focused attention on the need for outcomes for those who receive services. A national Core Indicators Project is years underway and the Service Delivery Reform committee which met a few years back picked some domains and agreed to set-up a learning model of evaluation as long as there was nothing complicated about it. Assemblymember Keith Richman (R-Northridge) included a first step toward evaluation in a bill he sponsored (AB 2775)this session which is being held in committee.

So why is there no real evaluation or useful accountability in the system? One problem we're up against is the difficulty of making long-term investment during a budget crisis (can you call it a crisis in it's fourth year?) I suspect another is that during the Davis Administration, the state Department of Finance fell in love with the idea of standardizing and set that agenda. Schwarzenegger's DOF is carrying the same torch. Yet another challenge is that service providers, who have tended to be the organizers of much of the system's advocacy tend to feel pretty strongly that measuring the result of their work is a poor substitute for funding it better. Hopefully, if real organizing continues at the grassroots level, the people who deserve the best possible system will focus more on their need for support that really helps than the historic concentration on provider rates.

Mind you, doing evaluation correctly or even well takes a lot of work. One old saw is, "What gets measured gets improved" and it can be challenging to make sure you actually measure the things you want to improve rather than the things you wish to avoid. In my experience, Unicorns are rarer than well-designed evaluation systems which are rarer than trout. In the case of California's system of support for people with developmental disabilities you can reverse the first two terms.

Later posts will discuss some thinking about what would make an evaluation system useful and meaningful to this system.

Tuesday, July 20, 2004

Statewide Purchase of Service Standards

I'll start with a couple of apologies. First, that my laptop was in the shop and I haven't updated this blog in awhile. Second, that this post will be even more arcane than most and probably indecipherable to anyone not well acquainted with California's community-based system of support to people with developmental disabilities. As always, I'll be happy to answer any questions by email.

One proposal for reform which keeps coming up is the one for statewide purchase of service standards (POSS.) This post will explain why this is a bad proposal unlikely to produce the desired outcomes of improving equity or lowering costs while reducing some of the virtues of the current system.

POSS seems at the surface like a reasonable proposal. For those not aware of how our system operates, State law provides an entitlement to people with developmental disabilities, including the criteria by which a person is eligible and the standards that service providers (Regional Centers and their vendors) must meet. The state then provides the Department of Developmental Services with a budget to fund Regional Centers, highly-regulated (in truth semi-governmental, but officially private non-profit) agencies which contract out to vendors to provide services. Those services are required to pursue the individual client's aspirations and reflect that client's needs and preferences. Each Regional Center (RC) has a local board of Directors and a monopoly on providing services to clients in the catchment area.

The intellectual basis for the POSS proposals have focused on concern about significant disparity in how much funding different RCs provide on a per-client basis. This has led some policy-makers to reach two conclusions: The first is that there are major inequities between how clients are served across regions and the second is that the high spending regional centers must be frivolous.

The conclusion was that statewide standards, essentially limitations on what any RC could offer to clients would improve equity and lower cost growth. Laudable goals. Absolutely. They fit with all three basic assumptions in my original post.

The problems come in when one considers how the standards would operationalize. A few more facts about this system before the analysis.

1. The Social Security Act, with which the State must comply to retain the 38% of system funding which comes through our Medicaid Waiver, requires the State to allow for fair hearings when a client feels that they have been wrongly denied a service or support. This requirement is not waivable as part of the Medicaid Waiver program.
2. There are two ways money gets wasted in this system- POSS is designed to prevent the waste that comes from state funds being spent on services or supports which are excessive, don't help or are otherwise inappropriate. The other is when services or supports are denied which would have allowed a client to live in a less costly, less restrictive setting.
3. The challenge that all RC staff face is that there is rarely clarity as to how the risks mentioned (in 2.) above are balanced.
4. California's regulations require an exception process for most RC purchasing policies to allow for unusual conditions in which the standard policies will fail to account for a legitimate need.

As a consequence of the four facts above, most client's receive services within their RC policy guidelines but some do not. When policies are applied which a client or their family/guardian/conservator feel should be waived due to health and safety needs or in order to allow the client to live in a less-restrictive environment, there are two processes (exceptions to policy and fair hearings) which allow the interested party to appeal for a different service decision.

Although the POSS proposal creates an exception when service limitations would cause someone to move to a more restrictive setting, the question is: How will that be determined. At the more liberal RCs, that finding will often occur. At the more restrictive ones, pretty much, the finding will occur only for board members and their families. As a result, actual behavior needn't change at any of the regional centers. The one difference will be that in the rare occassion that a client files an appeal, the likelihood of an RC decision being overturned is reduced because the RC will have been following state regulations rather than its own policy in denying services.

In other words, the likely result of POSS may be a small increase in the inequity of the system as liberal RCs retain the tools to remain so and restrictive RCs are emboldened to be more so.

So, POSS is unlikely to achieve it's objective vis-a-vis greater equity. What about cost?

That really depends on whether the funding wasted on inappropriate, ineffective or unhelpful supports will be reduced by the money wasted when decisions are made which lead to clients in more restrictive settings. Neither the sponsors of POSS, the writers of it, or I have any credible information to make that judgement. What information exists is interesting, though:

The RCs in LA County generally considered to be the most liberal in their POS policies also happen to have the lowest expenditures on a per client basis. Although there are a lot of possible explanations for the differences other than good policy-making, it is compelling that the evidence doesn't support the principle that for our system, conservatism and stewardship go together.

Ultimately, POSS would have reduced the individualization that is the moral core of this system with little promise of lower cost or greater equity. It may represent the best thinking of California's Department of Finance but we do better thinking in the community. And at DDS. The next post will be about some reforms with more promise being worked on at DDS and in the community.