How Many Researchers Does It Take to Screw in a Transformation?

February 28, 2008

This page contains links to documents that can be downloaded. The links include a document type logo and the document’s file size. Visit our Accessibility page, if you have trouble opening them.

Editor's note: This is the first of a two-part series looking at research and evaluation issues associated with the Mental Health Transformation Project.

Every carpenter in the land knows that you "measure twice, cut once." But it's not always easy to know where to cut when it's results of research that you're measuring.

That's exactly the problem that Ron Jemelka and his Evaluation Team with the Mental Health Transformation Project (MHTP) have faced. A recent presentation to a national group of mental-health services researchers, Using Data, Transforming Practice: Evaluating Mental Health Transformation in Washington State [ PowerPoint slideshow 1.7MB], details some of the unique strategies they have adopted.

Big Picture = Diverse Viewpoints

"When you think about how do you evaluate the transformation of a system, my thinking was that I don't know if any one person knows how to do that," says Jemelka. "So rather than hire one evaluator, which is what you usually do, we decided to put together a team of evaluators. Let's get people who have different kinds of expertise."

Some of the players Jemelka assembled include:

Mental-health services consumers and their families
Youth representatives
Department of Social and Health Services (DSHS) and MHTP staff
University and DSHS researchers

Making the Introductions Between Water and Oil

"I'm not sure yet if it was the most brilliant or the stupidest thing I ever did," Jemelka says, discussing the difficulties involved in bringing together the various groups.

Consumers and researchers, in particular, often approach the same questions and problems with widely varying strategies.

"They speak completely different languages," says Jemelka. "Consumers have no patience for discussion of sampling design or sample size or any of the nuances of what kind of correlation coefficient we ought to be calculating." Similarly, researchers can lose sight of the day-to-day challenges faced by consumers and family members.

The process became something like what you see with an athlete willing to suffer a poor year in order to successfully reinvent his or her game.

"We sat together and the first year we had to learn how to talk to each other, and that was not easy," says Jemelka. "But in the end it was well worth it. We have a perspective that is unique, a breadth of expertise to bring to bear on evaluation that's very different from anything anyone else is doing."

Why It Was Worth It

If putting together the evaluation team had its trials, the emerging payoffs are abundantly apparent:

More range and wider relevance of topics under consideration
More attention paid to innovative and unusual services and programs
Improved evaluation protocols with enhanced design measures and "test drive" approaches
Validity of conclusions better grounded
Criteria for success now includes what's actually important to consumers, youth, and families
Likelihood that results will be used are vastly improved

Clarity And Unity

With all its constituent parts tuned and ready, the MHTP Evaluation Team is moving confidently toward the contributions it's uniquely qualified to make in a transformation of mental-health services in Washington State that is effective and lasting.

Dr. Eric Bruns of the University of Washington now leads the team, coordinating its efforts moving forward.

"My sense is that Washington is in the top 10 among all states in terms of what it's accomplishing," Bruns says. "There's a lot going on here. We're the only one that's prioritized the needs of consumers, family, and youth."

The MHTP Evaluation Team is in a very good position to focus attention on policy recommendations, synthesizing and broadcasting analyses of relevant studies. As a result, new and innovative approaches stand a better chance of fair and serious consideration. With consumers included so prominently in the mix, the approach stands an even better chance of being widely accepted and effective.

Top of Page

Your Comments

Did you find this article of interest? Do you have a comment you'd like to make? If you'd like to comment on this article, click on the "Send A Comment" button above.