Sunday, September 9, 2012

Intended Purposes Versus Actual Functions of Digital Badges


By Daniel Hickey
On September 4th and 5th, there was a meeting at the National Science Teachers Association in Arlington, VA.  Al Byers of NSTA and Kyle Peck of Penn State organized the meeting to discuss the online NSTA Learning Center for science educator professional development.  I was only able to make it to the second day of the meeting where Kyle discussed the pilot work with the site and his use of digital badges from the Teacher Learning Journeys project.  In the afternoon, Sunny Lee and Erin Knight (Mozilla Foundation) and Brian Mulligan (Sligo Institute of Technology, Ireland) and I did a panel on digital badges that Kyle moderated.. 

One of the questions about badges that came up seems like a crucial issue as we grapple with different ways of characterizing and describing badges.  This post aims to add the category of badge functions to other badge taxonomies like the one by Carla Casilli. Because these issues are complex, this post ended up being rather long.  You may wish to jump directly to the summary at the bottom.  You may also wish to read a condensed version at the HASTAC website.



Badge Purposes Versus Badge Functions
In that panel, I elaborated on a point that I introduced in an earlier post and further in some conversations with Kyle and Eastern Michigan doctoral student Angela Elkordy on Mozilla’s Open Badges discussion list.  This point concerns the difference between the purposes we intend to use digital badges for and the actual functions that they ultimately serve.  I raised this point earlier in my discussions with Kyle when he suggested that Angela concentrate on using badges to recognize (i.e., accredit) learning, but suggested caution in using badges to motivate learning (because of the concerns over extrinsic incentives and other issues).  My response was that if Angela was going to use badges to recognize learning, then those badges will almost certainly serve some motivational functions.  And those functions would depend on how she used badges to recognize learning.

Zipporah Miller 
After the panel, NSTA Associate Executive  Director Zipporah Miller asked me to elaborate on this distinction.  I used the same example in the context of the NSTA Learning Center about how any use of badges to recognize or reward teachers for completing the online activities at the center was going to have some impact on motivation, and that different ways of recognizing learning would lead to different motivation functions.  For example, giving a badge automatically when somebody completed some activity would likely motivate differently than only giving that badge when an expert or a peer reviewed what the student turned in.  To complicate things more, having an expert versus a peer recognize that accomplishment could ultimately have different motivational functions.  As the break was ending and my explanation was getting complicated, I suggested to Zipporah that it might make more sense to continue the conversation as a blog post. 

Purposes Versus Functions in Assessment
Dylan Wiliam
I got interested in this distinction several years ago when thinking about assessment and transfer.  Jim Pellegrino (assessment guru and my doctoral advisor) asked me to co-author a chapter with him on assessment and transfer.  Many leading assessment scholars have followed Paul Black and Dylan Wilam’s lead in categorizing assessments in terms of assessment purposes.  In their groundbreaking work on formative assessment, they argued that assessment should fundamentally be put to formative purposes, which they defined as assessment for learning.  This was an important distinction and helped distinguish formative assessment from summative assessment, which they defined as assessment of learning.  This distinction is now widely embraced, along with the further distinction between evaluative purposes, which is assessment collected primarily for evaluating curricula or programs.   This focus on purposes was particularly helpful at the time because it got people to stop categorizing the assessments themselves as formative, summative, or evaluative.  This helped the field realize that ostensibly formative assessments were often used for summative purposes, and vice versa.  Doing so is always problematic and sometimes disastrous.

In the chapter I eventually wrote with Jim, (available at academia.edu)  I argued that focusing on purposes was also problematic, because it overlooked that fact that the same assessment could be summative for some kinds of learning and formative for other kinds of learning.  Because Jim has a narrower view of learning than I do, he was a bit skeptical at first.  The example I used in the chapter drew from the formative assessment work I had been doing with the GenScope software for teaching introductory genetics.  That research explored the use of multiple levels of assessment:  

We had very informal assessments that served a summative function of the way groups of students completed the inquiry-oriented activities in GenScope, but were formative for each student's understanding of the concepts introduced in those activities. 

We also had semi-formal performance assessments that were summative of each student’s understanding of the concepts, but formative for the teacher’s learning how to use the informal assessments.

Finally we had formal achievement tests that were summative of the teacher’s use of the informal and semiformal assessments, but were formative for our research effort in learning whether we were improving outcomes from one year to the next. 

Once I adopted a broader sociocultural view of cognition that recognizes all three types of learning, I was pretty much forced to adopt the notion of assessment functions to capture this distinction between different functions for the same assessment.  This was partly a response to the concerns that Jim and others had raised in in the 2001 National Research Council Report Knowing What Students Know.  In that, they argued that using the same assessment for multiple purposes was problematic. 

The difference in perspectives stems from our underlying view of learning.  If you view learning as primarily individual conceptual development (as is  in Knowing What Students Know), then it makes sense to focus on assessment purposes.  The intended and unintended consequences can then be treated more as policy or technical issues than learning outcomes.  But if you take a broader view of learning that also includes how a whole class learns to talk about something like genetics and policy makers learning whether their policies are impacting achievement, then it makes more sense to talk about functions.  Thus, the NCLB-induced explosion of test-prep, the narrowing of the curriculum, and the bizarre focus on “bubble kids” just above or below proficiency benchmarks are all ways that teachers and schools learned to accommodate test-driven accountability in the last decade.  I elaborated on this issue in much more detail in a paper about the GenScope research that I wrote with Steven Zuiker that will appear in the next issue of The Journal of the Learning Sciences.

The Varied Functions of Badges
My interest in the  function of badges was spurred along when the MacArthur Foundation asked me to help document the design principles for using digital badges that emerge across the 30 projects underway by the awardees in their Badges for Lifelong Learning project.  We need to come up with a manageable number of categories; I almost always consider “manageable” to be at least three and no more than five.  After reading a bunch of stuff and talking to Barry Joseph at Global Kids and Carla Casilli at Mozilla, Connie Yowell at MacArthur, and MacArthur Scholar Mimi Ito, my team and I settled on the following four categories of functions for digital badges to shape our study:

Recognizing Learning.  This is the most obvious and arguably the primary function of badges.  I sometimes extend it to Recognizing/Accrediting Learning to acknowledge both informal and formal learning recognition processes.  David Wiley has argued cogently that this should be the primary purpose of badges.  If we focus only on purposes, then he may well be right.  His point is that badges are credentials and not assessments.  This is also consistent with the terrifically concise definition in Seven Things You Should Know About Badges by Erin Knight and Carla Casilli.

Assessing Learning.  Nearly every application of digital badges includes some form of assessment.  These assessments have either formative or summative functions and likely have both.  In some cases, these are simply an assessment of whether somebody clicked on a few things or made a few comments.  In other cases, there might be a project or essay that was reviewed and scored, or a test that was graded.  In still other cases, peers might assess an individual, group, or project as badgeworthy. 

Much of the discussion of using badges in the NSTA Learning Center concerned the sorts of assessments associated with digital badges.  Some argue that all badges must be associated with some sort of formal assessment if the badges are to be meaningful. But doing so requires significant infrastructure, scorers, formal assessments, online quizzes, security measures, etc. This echoed a similar discussion in August around how the Department of Education might award badges for teachers participating in Connected Educators Month.  So yes, badges are not assessments in and of themselves.  But nearly every use of badges has some assessment function associated with it.

Motivating Learning.  This is where the controversy comes in.  Much of the debate over badges concerns the well-documented negative consequences of extrinsic incentive on intrinsic motivation and free choice engagement. As first demonstrated by Lepper and Malone in 1973 and reiterated by Mitch Resnick and Henry Jenkins, when you give somebody an arbitrary extrinsic reward for something they already enjoy doing, they will do it less when the reward is no longer offered.  This rich topic deserves another post all to itself.  But for now, it appears that badges are going to happen.  And if we use badges to recognize and assess learning, they are certainly going to impact motivation.  So we might as well harness this crucial function of badges and study these functions carefully while searching for both their positive and negative consequences for motivation.  Perhaps more importantly, we should recognize that badges that offer new opportunities and empowerment and are less likely to leave learners disempowered. (You can read more about these issues at this online handbook entry about motivation) 

Some of the DML awardees articulated specific motivational functions for their badges in their proposal.  We are now interviewing awardees about how those plans are being enacted. Many of them are uncovering entirely new motivational functions, and some are changing their plans because of motivational concerns.  What is particularly interesting to us is how project are realizing that their plans for using badges to recognize and assess learning are pushing them to rethink their expectations regarding motivation.  We are really excited about the many new ideas bubbling to the surface.  I hope that some of the proposals for the new round of research of the DML projects will explore these issues in some detail.  This would be a nice complement to our study of the design principles across projects.  Also check out some of the new scholarly considerations about the motivational functions of badges, including this one by Judd Antin.

Evaluating Learning.  The final category of badge functions we are working with concerns evaluating learning (which we sometime extend to Evaluating/Researching Learning).  Only a few of the DML proposals articulated formal evaluation and research plans.  But the awardees who care about evaluating programs and researching learning are quickly uncovering new ways of using digital badges to help accomplish those goals.   For example, Jim Diamond at the Educational Development Center is working with the Who Built America? project.  They are exploring ways of collecting data for their project and are experimenting with giving participants badges for completing surveys about the project.  These will be different from the badges that they award teachers for completing well-designed lesson plans approved by experts in American history or expert peers.  And they will ultimately need to consider whether the nature of the survey (i.e., the assessment function) interacts with the evaluation function.

Perhaps even more far reaching are the many ways that programs might use the data contained in the badges to evaluate and study learning.  Relative to grades or transcripts, badges provide a
TON of information about learning.  Just the eight metadata tags in the OBI specification include stuff like date, issuer, recipient, and accomplishment.  For many of the programs that are using badges, having a system that tracks just this data will be a huge step forward for evaluating and studying learning.  And much of the information in the actual badge will be hyperlinked back to things like artifacts, testimonials, rubrics, course descriptions, etc.  In the case of the NSTA, much of the data they need to evaluate the impact of their online learning center should be obtained just by compiling the information about all of the stamps and badges that they award.  The potential value of this information was nicely articulated by Stacy Kruse of Pragmatic Solutions.  Stacy is working with two awardees, the Corporation for Public Broadcasting (who are incorporating badges into three different CPB initiatives) and the Digital Onramps Project with Arun Prabhakaran of the Philadephia Urban Affairs Coalition to use badges with comprehensive programs for 21st Century education and workforce training.  To paraphrase Stacy, she pointed out that Pragmatics Solutions is fundamentally a learning analytics firm, so these projects are a dream come true!

Clearly there are additional functions of badges that don’t fall into these categories.  In contrast to Carla Casilli's taxonomy which aimed to be expansive, our goal of documenting a manageable set of design principles from the DML badges projects has pushed us to be concise. We might just need to add a category called “other.”  We have two years to sort it out, and it will be interesting to see where we end up.  In a subsequent post, we will elaborate some on the DML Design Principles Documentation project and say more about how we are using these categories in our project.   We welcome your suggestions and concerns.

Summary of Badge Functions
Okay, this has gotten pretty long.  But these are not simple issues.  These long posts remind me of the first chapter of Daniel Koretz’s 2008 book Measuring Up: What Educational Testing Really Tells Us. Koretz described the frustration that parents, business leading, and policy makers express when assessment and measurement experts insist on giving complicated answers to seemingly simple questions. There is not a simple answer to the question "what are the functions of digital badges?"

The following table summarizes the basic point of this post.  While the categories of badging functions are likely to evolve, I think the basic point will hold that there are a definable set of ways to use badges to support learning, and that functions are a good way to sort these things out and their interactions.  Recognition appears to be an essential to digital badges. Most conceivable uses of badges will serve some assessment function, while many will also serve some motivational function.  Finally while badges offer potential for evaluating and researching learning, these are potential functions which need to be specifically pursued.  


Functions of Badges
Recognizing/Credentialing Learning
Assessing Learning
Motivating Learning
Evaluating/Researching Learning
Nature of Functions
Essential
Functions
Probable Functions
Likely
Functions
Potential
Functions




6 comments:

  1. Great posting Dan. Thanks for sharing your thoughts. I could see a redoring where Motivating Learning comes second (since it is likely, where Assessing Learning is only probable?). I think on a scale of easy to harder to do, the left to right arrangement you have is a good way to think about it. I also like the subtle overlapping of functions. One thing I notice that might add a dimension to the "purpose" is to point out "whose purpose" - if it is the main actor-agent "self" that can use the feedback for self-improvement, then its formative; and if it is an "other" or "self" for making a judgment, categorization, or decision, then its summative.

    Another way I'm thinking about the issues these days is that if the data record is rich, thickly textured, time-series sensitive, and more like a music score than a score card, then it has more formative power, because it can pin point when, where and how some dynamic relationship occurred in the performance. And if you can fit the summary on a single piece of paper, or if it is a snapshot, ignores time and sequence, and aggregates or collapses information (somewhat like a "badge") then its more likely to be summative. A formative badge system will have several badges or achievements, points, and other visible mechanisms to help the "self" leading up to a big badge and in that sense, will be a series of summative signposts along a pathway toward a summative certification.

    ReplyDelete
  2. David--
    thanks!. That was the kind of feedback I was looking for. That makes me wonder if I choose the right works. From my perspective, the assessment function is pretty inevitable. Even if there is no "assessment" something is being recognized. If the system automatically detects somebody submitted something and awards a badge for that, the system has "learned" that something was submitted and assessed whether or not it was. But that is a consequence of my view of learning. Likewise, it is hard for me to imagine a situation where a badge does not have a motivation function, even though it might be quite modest.

    Stacy helped me appreciate that there is going to be all of this data out there for evaluation and research. The more I think about it the more I am coming to realize that the easy access to data regarding learning that badges should generate will be the most transformative aspect of badges. But certainly some will choose not to use that data so in those cases it will not have that function.

    ReplyDelete
  3. As to your point about the scope of learning, that is really an important point. I have been thinking about that in terms of time scales (following from Jay Lemke's insights). The smaller elements represent learning that occurs on a shorter time scale and respond to a very different type of feedback that the bigger elements.

    The original version of this post included some of Kyles discussion of "stamps" which are mini-badges. I decided to hold off on that in order to focus more directly on it. It is a really big issue for us all to consider. Thanks for the comments!

    ReplyDelete
  4. I have enjoyed reading (and re-reading) this blog post as well as the one you listed about Sociocultural Theories of Motivation. It has taken me some time to absorb all the information, but very useful information that I am passing along at work as we start to introduce badges on our campus.

    One comment: while much of what you discuss regarding motivation seems to be geared towards those creating badges and their thoughts about how badges are perceived by those engaging in the activities, I am interested to know your thoughts about the motivation from the perspective of the person taking a class online. Will their motivation be driven by their thirst for knowledge? whether or not badges are accepted in the business world (ie. a supplement to a resume)? etc. I suppose my thoughts are more of a macro vision while your excellent discussion is more on a micro level, but I'm curious to your thoughts on this subject.
    NancyJ

    ReplyDelete
  5. Great question Nancy--
    Yes, I was talking about motivation from the perspective of folks issuing badges, but primarily to make the point that how people use badges to motivate may have nothing to do with how badges actually function to motivate learners.

    I suspect that many of the badging systems that people are now racing to set up will make people feel less motivated to learn deeply and less motivated to learn about those topics in the future when badges are no longer offered. And I certainly think this will be the case on online course and MOOCs where (a) badges are likely to a substitute for points and grades, and (b) where some may find it hard to award badges for peer judgement or for meaningful engagement, and (c) where it will be hard to award badges that grant the earner added privileges or powers.

    We had a fascinating conversation with Jonathan Finkelstein and others at LearningTimes about their collaboration with Cooper-Hewitt and the National Design Museum. They are in the midst of a big event working with youth to specify and design (as in professionally design) the badges that they will be awarding. Many of the badges will be associated with significant opportunities and authority in that learning ecosystem. It is likely that those badges will be strongly associated with much deeper levels of engagement, higher levels of intrinsic motivation, and substantially enhanced sense of empowerment.

    In the coming weeks we hope to get out additional posts about the practices associated with all four functions that we are uncovering in our interviews with DML awardees. We will also start working in some of the relevant research literature in each area with folks just like you in mind. Those will gradually grow into sections of a report and review papers presumably for publication.

    Does this sound like it is going to help answer the questions you and your colleagues are asking?

    ReplyDelete
  6. A thought provoking post, with many facets to consider. It's interesting, that the context, learning content and learning trajectory will perhaps shape the digital badges' purpose and also its capacity to motivate. For example, a well thought out learning path, which is adequately supported (perhaps a design component -- method of instruction or collaborative learning) may be more motivating than the "idea" of a badge. Similarly, a learning path which does not engage the learner may result in lack of motivation (do I want this badge really if it is so simple to earn?).

    Furthermore, the perception of the intrinsic value (by the intended community of practice) of a badge will impact its capacity to function as intended. It's quite convoluted!

    As more badges roll out, I think interesting questions in terms of functions and purposes may be similar to reliability and validity. How is mastery learning defined and measured? how might badge design (learning objectives, learning trajectories and end products demonstrating learning) scaffold learning -- so the "working" purpose is formative assessment with a function to evaluate learning. Importantly, do the badges actually measure learning in the competency (validity) and can the learning be reproduced (validity).

    great post!
    Angela (Angela Elkordy)

    ReplyDelete