Outcome-Based Evaluation
A Working Model for Arts Projects

Measuring Joy: Evaluation at Baltimore Clayworks

By Deborah Bedwell, Executive Director

Samual Wallace with student Sherrice Rogers at McKim Center Summer Camp, 1999, Baltimore Clayworks.

It wasn’t so long ago that when I would see the words "measurable outcomes" on a grant proposal, I would experience a wave of nausea and anxiety. I would be required, the grant stated, to prove to the prospective funder that our programs and activities had created a better life for those who touched clay, and for the rest of the city – and maybe the rest of humanity. Wow! A tall order. Not only were we to prove how a small clay airplane can save the world, but save it by what percent over last year, and how airplanes versus cups, for instance, can have a significant impact on a community.

I have a personal bias against lying. I especially dislike lying to funders who could hold the artmaking future of some kids or of some Clayworks’ activities in their unpredictable hands. So, I decided that I’d better set to the task at hand and figure out how to evaluate joy, how to measure creativity, and how to quantify that "I get it!" moment that makes weeks of hard work worth the effort.

First, I had to look at what kinds of evaluations we were already doing at Clayworks and why we were doing them.

Most of the evaluations that we were and are doing at Baltimore Clayworks, we are doing for ourselves. (I’ve since learned that "ourselves" is called an internal audience.) These exercises include wrap-up sessions with volunteers after fundraising events, surveys with our on-site students after their semester’s clay classes at Clayworks, meetings with our community arts teachers and partners about their programs in community centers, annual performance reviews with staff, tracking direct mail fundraising appeals for their effectiveness, and other means of trying to find out if and how Baltimore Clayworks was making a difference. Furthermore, most of these surveys, wrap-ups, etc. were preceded by planning sessions. If little or none of this information was of interest to funders, why were we taking the time and making the effort to meet, write, and ask all of these questions? We were trying to strengthen our programs and trying to maximize the use of our resources — both very sound reasons to conduct evaluations.

Our community arts program is one of the most crucial programs to our mission, the program experiencing the most rapid growth, the program requiring the most accountability to funders, and yet the hardest program for us to assess. . In February 2000, Clayworks organized a weekend retreat to identify the elements of success as well as the challenges for this program as it grows. Artists who had been working with us in community arts within the last decade, community partners, funders, and a few board and staff members put their ideas and experiences together for two days. There were many stories, lists of mutual expectations from participants, suggestions for organizational tools, and much more that made the weekend a rich and valuable experience for Baltimore Clayworks. In the last hours on Sunday, we again grappled with the core questions — how do you know when the artistic experience is authentic? How can you measure joy? The consensus was you know it when you see it."

Enter Maryland Association for Nonprofit Organizations (MANO) in early April with a two-day Program Planning and Evaluation workshop led by two researchers from Brandeis University. Out of my desire for Clayworks’ scant professional development hours and dollars to be used wisely, I queried the presenters by phone before enrolling. "Will I learn how to measure children’s creativity and joy in this workshop? Will you talk about the arts as well as human services?" When I was assured that my evaluation questions would be addressed, I wrote the check and blocked out the two days.

Eureka! What a revelation! Formal, methodological evaluation, I learned, was not just for rounding out grant proposals and following up funding, it was for strengthening programs and allocating resources! And what’s more, at Clayworks, we were already doing many evaluative procedures and doing them properly. But what I learned that was of enormous value, is that by instituting a down-to-earth, commonsense, understandable framework for evaluation, we could achieve a level of consistency and determine our programs’ effectiveness with far more accuracy than before.

This framework, called the "Theory-Based Logic Model Evaluation," is the model that The Kellogg Foundation uses to assess its own programs, and Kellogg encourages its grantees to try it as well. It is based on a series of "If-Then" assumptions that are tested by conducting activities (programs — making those clay airplanes, cups, etc.) that have early, interim, and final benchmarks (observed by evaluators) that are either met or not. The meeting of benchmarks are the measurable outcomes, and the outcomes determine the ultimate impact of the program. The logic model can be used for both internal (ourselves) audiences and an external (funders) audiences, but the assumptions, or what exactly you are trying to evaluate, can be tweaked for the particular audience. Sounds simple?

It may be, but as we begin to use this system we’re finding some challenges. First, it takes time.

• We are holding staff training sessions of 3-5 hours on the method; that may not seem like many hours, but to pull half the staff together for training can be complicated.
• The programs’ planning sessions must include getting all of the people involved to discuss and agree upon the assumptions as well as the program activities.
• We must find evaluators to attend the programs at the beginning, middle, and end, and they need to be able to collect data in multiple ways — holding focus groups, conducting individual interviews, distributing and retrieving questionnaires, etc. If the evaluators are program staff or volunteers, they must also be trained.
• Finally, the program team needs to be able to analyze the data, and make decisions concerning the programs’ effectiveness and ultimate impact.

At Clayworks, we are gradually moving into logic model evaluations. As part of staff training, we are putting some existing programs and events through the model, and using the exercise as a way of familiarizing ourselves with the language and thinking. We are also choosing to measure only those things we deem critical, not just what it would be nice to know about.

I don’t foresee a day in the near future where all of Clayworks programs will be evaluated by using logic models, but we’re making a start. In those areas where it is harder to measure success, and where gains and positive steps in children’s creative activities are more incremental, and where the measurable outcomes are more ephemeral, we’re moving to institute logic model evaluation. Can we measure joy? Yes we can. And by knowing where and how to look for it, how to document what we see and hear, and how to communicate that, Baltimore Clayworks can create opportunities for more joy more often.

Index

National Endowment for the Arts
Contact the Web Manager