When I'm on campus, I'll often find myself having conversations with faculty members about various theories and approaches to education evaluation. Sometimes it's because Burke is on his tour of offices and pops into mine with a question he's been mulling over. Other times it's because Jim and I are working on a project that spurs an idea or a reflection. The best way I've been able to describe my initial experiences here are that they've been like those moments, only more frequent.
The best example of this was a conversation I had with a colleague here about the problems with defining the museum experience in terms of "outcomes" and "learning". From his perspective, when we distill the museum experience to these constructs and begin developing "learning outcomes", we are defining the "right" way to experience a museum. Are we limiting the information we think is worthy of investigation to only those things we thought were important? Are we communicating to the people who come to those museums that those learning outcomes are what makes an experience legitimate? What if the people visiting don't learn a new skill or fact, but feel a sense of wonder? What if the experience generates some feeling or outcome that we didn't know to look for?
While I'm not ready to throw out the concept or use of outcomes in evaluation, I do think this conversation is worthy of consideration. When we develop our lovely logic models, are we leaving room for the unknown? Are we so focused on investigating the the outcomes identified during program planning that we forget to look for the unexpected?
This conversation also made me reflect on how what we do as evaluators can communicate to our audience what we believe is important and how this can trickle into programmatic choices and participant experiences. I believe that this is true for both formal and informal learning experiences. There has been significant discussion in the K-12 arena about the increase in school time spent on areas assessed with high stakes testing (math and reading) to the detriment of other areas like music and art. In many ways, this shift in resources was predictable. If students do not do well in art classes, there are few institutional consequences. If students do not do well on a high-stakes math achievement test, people can get fired, schools can be closed, etc. (that's why it's high stakes, right?). This has impacted how resources are allocated within schools. But what does it mean when we tell our students through our actions that math matters, but not music?
With the movement toward inserting outcome assessment into the informal learning space, we should consider the lessons learned from the formal learning sphere about how assessment and evaluation can impact resource allocation and unspoken messages. When someone visits an art museum, have they "failed" if they felt revulsion instead of wonder? If a student visits an exhibit about Abraham Lincoln and doesn't remember the facts we've presented but thinks to himself "That guy was pretty funny looking. I think I'm funny looking too. Maybe that's ok." has the student had an unimportant experience?
I think that in many cases, outcome assessment is essential. In evaluation, we often want to know whether we're throwing away money or if it's having some sort of impact. However, it's worth the time and energy to consider the potential consequences of narrowing our focus. In the end, evaluators have significant influence in determining what outcomes matter. And we should take that seriously.