The Role of Artifact Reflections in Participatory Assessment

By Rebecca Itow and Dan Hickey

On June 7, 2012, we hosted Bloomington’s first Hackjam in conjunction with the Monroe County Public Library. In our initial recount of the day’s events, we mentioned that we used artifact reflection and digital badges as ways of gauging, evaluating, and rewarding progress in each activity. In this post, we will explain how and why we chose to use reflection and badges as forms of assessment. To read more about the theory of badges as Transformative Assessment, read our June 10 blog post. 

Assess Reflections Rather than Artifacts

We have been struggling for several years to refine practices for assessing artifacts that students create.  It seems pretty clear that badges are going to highlight a problem that teachers and proponents of portfolio assessment deal with all the time: rubrics.  If you attach consequences to the quality of student artifacts, there is a natural tendency to demand detailed rubrics and individualized feedback as to whether the artifact matches what is demanded by the rubric.  Most learning environments are more concerned with the learning embodied by the artifact than by the artifact itself.  So focusing so much on the artifact and the rubric can be quite problematic. 

We suggest eliminating rubrics entirely.  Instead, provide guidelines for creating artifacts, then invest in helping students reflect on how the artifacts reveal that they have participated in something meaningful.  Evaluate those reflections instead.  As you can see from the examples below, the main strategy in our reflections is getting students to reflect on how their own version of the activity impacted what they learned about the concepts we were teaching them.  This notion of “context x concept” reflection is central to all of our efforts to use assessment to foster participatory learning.  It turns out that in order to reflect on the intersection of your (more concrete) context and the (more abstract) targeted concept, you need to have or get some understanding of that concept.  Since you never directly teach students the definition or even the abstract meaning of the concept, any evidence of it in the reflection is a good sign that students have learned about the concept.  This is an example of how participatory approaches balance formative and summative assessment functions.

Assessing Reflections in the Hackjams

The activities in the Hackjam were posted to individual wiki pages created by the students using wikispaces.com. They ranged from informal and “light” hacking of an article in the local paper to creating a webpage from scratch using Mozilla’s Thimble App, and within each activity hackers could engage to varying degrees. For simply completing an activity and demonstrating the skills outlined in the badge requirements, hackers received a One Star Maker Badge. For thinking about their learning process and composing a reflection on how that activity affected the way they understood the web literacies specified, they could receive a Two Star Reflector Badge. 

These first two levels of badges were awarded by the Hackjam mentors, and a picture of their completed activity or reflection was attached to the badge as evidence using ForAllSystems’ newly developed app available in the iTunes store. Three Star Participator Badges were peer-awarded. Hackers read and reviewed each other’s hacks and reflections, and could choose to award their peer with a Three Star Badge, indicating that the author had impacted the reader’s knowledge in some significant way. Badge awarders were required to include a statement of why they felt the peer deserved this badge as the evidence attached to the badge itself. They were also asked to comment on the awardee’s wiki page and tell them why a Three Star Badge was awarded.

ForAllSystems’ Toby Kavukattu was instrumental in making the badge system successful. He helped us rework the existing badges in the Hacktivity Kit – which only have one level of each type of badge – to reflect the principles behind the development of our curriculum, and to represent different levels of engagement. He drove down from Chicago to help implement the badge system, and he even taught the advanced webmaking section of the Jam. 

What Did We Learn?

While the overall implementation of badging was successful, our first attempt at awarding badges has left us with clear goals and adjustments for future implementations. The first thing we noticed is that we had too many badges – or learning outcomes – than could possibly be covered in one six-hour period. For example, we determined that, while the hackers all enjoyed making their own webpage from scratch, they literally only scratched the surface of what could have been covered, and really needed a day devoted just to that activity. 

The One Star Badges were effective in that they provided enough motivation to help students push on as the fourth hour of the day came about, but were not the sole reason students were engaged. As evidenced by their reflections, the students were genuinely invested in each hack, and wanted to make their current project the best it could be. These One Star Badges were easy to award, as there were clear goals and tasks to be completed that could be easily seen and photographed for evidence.

Two Star Badges gave hackers more of a challenge, and participation in pursuing these badges dipped a little. However, many of the students did complete reflections, and they are very telling of the participant’s engagement and comfort in the particular skills. We did not specify a length requirement, as this would lead to the participants trying to “figure out what we wanted” in the reflection, and we found that, while the initial reflections were short, they became longer and more detailed as the activities became more involved. This validated our initial assumption that participants can and will engage deeply when they feel the activities hold some meaning and usefulness for them.

Three Star Badges presented the most challenges but they were challenges we had anticipated. For example, some of the participants knew each other before coming to the Hackjam and wanted to award their friends as many Three Star Badges as possible. When this began happening, the mentors reminded students that they needed to provide a statement of evidence as to why this badge was deserved, and deleted invalid badges. The random awarding slowed considerably at that point, and most of the subsequent Three Star Badges awarded were given with supporting evidence and for good reasons. The participants who sought and awarded Three Star Badges were fewer than those who pursued the Two Star Badges, and tended to be those who were deeply invested in their hacks, as was evidenced by the detail in their hacks and reflections.

How Did We Wrap it Up?

At the end of the day, the hackers counted up the number of stars they had received, and were allowed to pick a prize from the prize table. These were trinkets, ranging from glow sticks to a Grow Your Own Boyfriend sponge. Each participant also left with a coupon for a free sandwich from WhichWich and a comic book from Phoenix Comics. These prizes were not so substantial that they kept any one of the participants engaged when they otherwise would have been disengaged, but served as a small token of appreciation of their engagement. 

We know that these kinds of incentive practices are controversial.  But people are going to do things like this all the time anyway and we want to study it.  In particular we are pretty confident that the consequential engagement afforded by the new learning environments will counter or even reverse the “overjustification effect” that leads extrinsic incentives to undermine intrinsic motivation.  It seems pretty unlikely that attaching such prizes would keep students from engaging with Hackasaurus next time because they were not again offered a prize.  Cognitive motivation theorists have long argued against giving rewards when the activity was already intrinsically motivating.  So they would likely argue against using them here. 

Before students walked out the door, they were shown how to “push” their badges into the Badge Backpack and share them on Facebook, Twitter, and Google+. As they exited the library many participants asked when the next Hackjam would be, and were excited to share their creations with their friends and family. Throughout the day, hackers wanted to make sure they could get the tools we were using in the activities at home so they could continue to hack more sites and create more pages.

Future goals include finding out if and how participants in these Hackjams continue to hack, and making continual refinements to the Hackjam curriculum.