Wednesday, July 18, 2012

The Role of Artifact Reflections in Participatory Assessment

By Rebecca Itow and Dan Hickey

On June 7, 2012, we hosted Bloomington’s first Hackjam in conjunction with the Monroe County Public Library. In our initial recount of the day’s events, we mentioned that we used artifact reflection and digital badges as ways of gauging, evaluating, and rewarding progress in each activity. In this post, we will explain how and why we chose to use reflection and badges as forms of assessment. To read more about the theory of badges as Transformative Assessment, read our June 10 blog post. 

Assess Reflections Rather than Artifacts
We have been struggling for several years to refine practices for assessing artifacts that students create.  It seems pretty clear that badges are going to highlight a problem that teachers and proponents of portfolio assessment deal with all the time: rubrics.  If you attach consequences to the quality of student artifacts, there is a natural tendency to demand detailed rubrics and individualized feedback as to whether the artifact matches what is demanded by the rubric.  Most learning environments are more concerned with the learning embodied by the artifact than by the artifact itself.  So focusing so much on the artifact and the rubric can be quite problematic. 

We suggest eliminating rubrics entirely.  Instead, provide guidelines for creating artifacts, then invest in helping students reflect on how the artifacts reveal that they have participated in something meaningful.  Evaluate those reflections instead.  As you can see from the examples below, the main strategy in our reflections is getting students to reflect on how their own version of the activity impacted what they learned about the concepts we were teaching them.  This notion of “context x concept” reflection is central to all of our efforts to use assessment to foster participatory learning.  It turns out that in order to reflect on the intersection of your (more concrete) context and the (more abstract) targeted concept, you need to have or get some understanding of that concept.  Since you never directly teach students the definition or even the abstract meaning of the concept, any evidence of it in the reflection is a good sign that students have learned about the concept.  This is an example of how participatory approaches balance formative and summative assessment functions.

Assessing Reflections in the Hackjams
The activities in the Hackjam were posted to individual wiki pages created by the students using They ranged from informal and “light” hacking of an article in the local paper to creating a webpage from scratch using Mozilla’s Thimble App, and within each activity hackers could engage to varying degrees. For simply completing an activity and demonstrating the skills outlined in the badge requirements, hackers received a One Star Maker Badge. For thinking about their learning process and composing a reflection on how that activity affected the way they understood the web literacies specified, they could receive a Two Star Reflector Badge. 

These first two levels of badges were awarded by the Hackjam mentors, and a picture of their completed activity or reflection was attached to the badge as evidence using ForAllSystems’ newly developed app available in the iTunes store. Three Star Participator Badges were peer-awarded. Hackers read and reviewed each other’s hacks and reflections, and could choose to award their peer with a Three Star Badge, indicating that the author had impacted the reader’s knowledge in some significant way. Badge awarders were required to include a statement of why they felt the peer deserved this badge as the evidence attached to the badge itself. They were also asked to comment on the awardee’s wiki page and tell them why a Three Star Badge was awarded.

ForAllSystems’ Toby Kavukattu was instrumental in making the badge system successful. He helped us rework the existing badges in the Hacktivity Kit – which only have one level of each type of badge – to reflect the principles behind the development of our curriculum, and to represent different levels of engagement. He drove down from Chicago to help implement the badge system, and he even taught the advanced webmaking section of the Jam. 

What Did We Learn?
While the overall implementation of badging was successful, our first attempt at awarding badges has left us with clear goals and adjustments for future implementations. The first thing we noticed is that we had too many badges – or learning outcomes – than could possibly be covered in one six-hour period. For example, we determined that, while the hackers all enjoyed making their own webpage from scratch, they literally only scratched the surface of what could have been covered, and really needed a day devoted just to that activity.