Monday, May 20, 2013

Digital Badge Design Principles for Recognizing Learning

Cross-posted at HASTAC

by Andi Rehak and Daniel Hickey


This post introduces the design principles for recognizing learning that are emerging  from the Design Principles Documentation Project (DPD).  A previous post summarized how the DPD project derived these principles. This is the first of four posts, to be followed by posts outlining the principles for using badges to assess, motivate, and study learning.


First and foremost, digital badges serve to recognize some learning or accomplishment.  As succinctly put by Mozilla's Carla Casilli and Erin Knight in their EDUCAUSE Brief, "badges signify accomplishments such as completion of a project, mastery of a skill, or marks of experience."  An important distinction here was made by David Wiley when he pointed out that "badges are not assessments…badges are things we award to people who pass assessments.”

In more formal educational contexts, this practice is usually called "credentialing."  We are using a more general term of “recognizing” to emphasize that many of the practices for recognizing learning with badges are more informal.  For example, while a Girl Scout badge is technically a credential, few would call it that.  The different DML projects recognize various types of learning.  The way that learning is recognized and who recognizes it also varies. These differences then have consequences for assessing, motivating, and studying learning.

Our Process for Identifying these Principles
We first identified the intended practices for recognizing learning in each project's proposal.  As project teams began implementing those intentions, we interviewed them to uncover their enacted practices.  The 5-10 enacted practices for each project were then sorted into a manageable set of more general principles.  We debated categories and subcategories ourselves. We then discussed and refined them at a half-day workshop before the DML conference with 45 badge project team members.

This resulted in the preliminary set of design principles listed below, starting with the ones that were most widely represented.  These principles are not set in stone.  The principles and (eventually) sub-principles will continue to evolve as we work with them, identify resources that are relevant to each, share them with projects, and begin sharing them more generally.  We seek your questions and suggestions.  Please post comments directly on the blog.

Nine Principles for Recognizing Learning with Digital Badges
The following principles are ordered by prevalence in the current badging practices of the DML Competition 4: Badges for Lifelong Learning awardees. While these principles were developed based on the practices of the DML awardees, these principles should be useful in helping organize efforts to design and refine any badging system. The usefulness of the principles will (hopefully) increase as they are connected to existing literature on credentialing of learning experiences as the DPD team delves into that phase of the project.

1. Use badges to map learning trajectories.  Most of the projects used badges to organize curriculum and learning experiences by either determining levels of badges or offering meta-badges. Alternatively, some projects allow for a more learner-directed process that encourages students to create their own trajectory.

2. Align badges to standards.  Many of the projects used national or international standards to increase the external value of the badge. Alignment to standards is presumed to improve transparency of the credential and help to facilitate better communication of earner knowledge and skills. Some of these standards were more formal such as, the Common Core State Standards, while others were the less formal such as "21st Century Skills." Sometimes the alignment was very formal but other times it was very informal.  Highlighting the relationship between recognition and assessment, the formality of this alignment was usually defined by the formality of the assessment practices involved. 

3.  Have experts issue badges.  Having experts issue badges increases the credibility of the badge and likely influences the usefulness of the credential outside of the issuing community.  At some level, some expert is associated with issuing badges.  But the nature and role of this expert varied quite a bit, as did that way that the expert was him or herself credentialed.  Sometimes the expert held an external credential, while other times the expert was credentialed by the community; some projects include both. 

4. Seek external backing. External backing is presumed to increase the usefulness of the badge as name recognition is a driving force in getting schools or employers to recognize the badge. In the projects that sought external backing, this seemed different than just using badges as a means of external communication. Whether or not the badge is actually externally endorsed, existing formal relationships can increase its external value. (For those wishing to have formal credit granted for a badge, this is the first step.) Partnerships increase communication of the learning recognized in the badge and thus increase the importance/usefulness of the badge for earners. In some cases, a badge is formally endorsed and carries the insignia of the endorsing institution.

5.  Recognize diverse learning.  Credentialing a broad spectrum of experiences helps to legitimize these areas and recognizes knowledge and skills which would otherwise only be implicitly noticed or not at recognized. While this principle could be uncovered in nearly all of the projects at some level, we highlighted several projects that embraced it explicitly. These projects recognized skills and learning outside of what is traditionally recognized in formal learning environments, giving badges for both "hard" and "soft" skills.

6.  Use badges as a means of external communication of knowledge and/or skills.   As with the previous principle, most projects did this at some level.  But some projects really made a concerted effort to increase communication of the learning or accomplishment that the badges represent.

7. Make badges permanent.  While many projects did not explicitly discuss whether or not their badges would expire or require upgrading, a few made strong cases for learners being able to have permanent credentials that will always exist to recognize that specific skill, knowledge, or experience.

8.  Recognize educator learning as well.  Some of the projects awarded badges specifically to educators.  This is a different principle that relates to the several projects where additional badges were included alongside student badges.  These were sometimes that same as the badges for students and other times they were specific to the educators.  Generally speaking, these badges were used to recognize the educators’ participation in the broader learning ecosystem.

9.  Award formal academic credit for badges.  In a few projects, badges were used as a supplement to a formal grade for in-school experiences. Currently, only a couple of projects have successfully created partnerships that allow a badge to directly result in formal academic credit. This of course greatly increases the value of the badge for badge earners.

A Design Implication
We conclude by introducing a dilemma we faced in this process.  This dilemma was embodied in the distinction between (a) integrating badges into an existing curriculum and (b) creating a badge system and a curriculum at the same time.  For a while we treated this as a design principle for recognizing learning.  But we came to see it more as a “design implication.”  This difference seems to have great impact for the design of the badging practices (and likely for the learning ecosystem that results).  But this distinction was largely a result of the context in which badges were being introduced, rather than a deliberate choice.  As such it did not seem like this qualified as an actual design principle. When badges are being added to a pre-existing curriculum, the curriculum will constrain the way learning is recognized. For example, if an existing curriculum is not aligned to standards, it is very difficult to align a badge to standards. Alternatively, when the curriculum is being developed alongside badges, the options for both may seem limitless and overwhelming. A pre-existing curriculum can importantly help to structure design decisions. There are specific advantages for either approach.
We would love to hear what people think about this distinction or about our design principles for recognition of learning with digital badges.

13 comments:

  1. The blog uses the phrase "creating a badge system and a curriculum" and I think you might want to broaden from curriculum to "digital engagement or learning experience." Two reasons. first in a curriculum, someone other than the learner decides what needs to be learned and pursued and I wonder if this is OK in all forms of learning. Second a "curriculum" does not seem to track well with open exploratory learning.

    Am I being too harsh on formal schooling? too narrowly defining "curriculum"?

    ReplyDelete
    Replies
    1. Sorry for the delay David. I was on holiday when these went up and Andi was away for health reasons.

      I agree with the concern that you raise. We were given a rather narrow charge to focus on the 30 projects funded by the DML 2012 competition. This is partly because Mozilla was funded to work with the much larger open badges community. It turns out that none of the 30 projects we are looking at is focusing directly on open and exploratory learning (at least not from the start). But I think that badges are the ideal recognition practice for moving the open learning community forward. Many of the folks looking at this are part of the Mozilla Open Badges community. I am a bit out of the loop in the last couple of months, but I bet that some of the things being offered this summer in the Chicago Summer of Learning are good examples. Anyone come to mind?

      Delete
  2. Second in a series of thoughts...

    I think it is a little odd to say recognize educator learning "as well" (sort of like an after-thought?) If we want these to be general principles for all kinds of badge-based recognition, then anyone can be a recipient - they just have to be a person using the digital system and doing things that earn them badges.

    ReplyDelete
    Replies
    1. Again, this comes back to the constraints of our project. We wanted to have a way to help connect the specific projects that are focusing specifically on teacher learning (which were funded by Gates). Our goal is that someone who is interested in teacher learning and digital badges can zero in on the projects that enacted this principle. It is a pretty complicated info management thing in the end but I think it will make sense in the end.

      Delete
  3. Third in a series of thoughts...

    On the item "make them permanent" it might be better to say that badges have some kind of time-sensitive span of relevance that has to be considered as part of the ecosystem niche of the badge issuer. For example, while my knot-tying badge from summer camp might be permanent, I will not use it in my adult job applications. And my "flying license for Cessna's" might be a badge that I would use in an adult resume, but unless I retrain as the plane evolves (new flight characteristics, new instruments) then my badge is no longer relevant for getting a pilot's job.

    ReplyDelete
    Replies
    1. Completely agree. I think the principle will be renamed "consider the life span of the recognition"

      Delete
  4. Fourth in a series of thoughts...

    The experts issuing badges seems more closely aligned to formal recognition of learning. I think peer-assigned badges (as in community reputation) also have a role - and even an increasingly formal role in the assessment of learning, so I'd recommend discussing both of these concepts.

    ReplyDelete
    Replies
    1. But we can only do that if we have examples of peer-awarded badges to point to. Interestingly, several of the projects proposed peer-awarded badges but did not follow through. Our challenge is how do we transform their experience into a principle that can guide others? For example, one of the projects could not make it work in their Drupal platform so they dropped it. More broadly speaking, how do we deal with the fact that as that badges become more valuable, the harder it is to do peer awarded badge? I have been dealing with this in my own work by using peer-awarded "stamps" which add up to teacher-awarded badges. But there are lots of other ways people are dealing with peer awarded badges out there that we can learn from. I hope that eventually we can connect our principles to those efforts

      Delete
  5. Another in a series of thoughts...

    On number 9, I'm assuming you want these principles to be very broad and applicable even outside of formal education, so I would call this one "Make the rewards clear" and talk about a variety of rewards for earning badges, some of which might open new trajectories, open new doors, can be cashed in for prizes, etc. as well as put on a grade sheet or transcript.

    ReplyDelete
    Replies
    1. No we are getting at something very specific here. Some of the project are awarding formal academic credit. In particular the Providence After School Alliance has made tremendous progress. This is a much bigger challenge than many people realize. We want to help people find the specific projects that are doing this and help them see the contextual factors discourage or support efforts at doing so. Katie Davis at University of Washington was funded to study PASA and it should be quite interesting.

      Delete
  6. I will enter a few comments as separate items here to see if we can get extended discussions on them, so this is the first in a short series...

    After glancing at all the principles, I began to wonder if we need psychometric validity as well as the validity of the expert community, in which case there should be a mention of how reliability is monitored and improved. What if one expert scores a work with an x but the next one scores it with a y - is this OK? An old rule of thumb used to be that if the stakes are low, then don't worry so much about reliability, but I'm not so sure that rule is a great one. (the broken watch is reliable, after all, but fails on validity; but if we do NOT have reliable measures, then the validity suffers).

    ReplyDelete
    Replies
    1. Thanks for your awesome comments. These last two were also posted at the cross posted at the discussion of the assessment principles at HASTAC so I think I will direct you and others over to the nice discussion that his fostered. Always great to discuss these issues with you.

      Delete
  7. I am reminded of the triad of models needed for assessment ( see "Knowing what students know" - Pelligrino etc.) While you might have the principles covering these, you might want to elaborate.

    (1) You need a task that elicits the knowledge and skill you want to measure (and this is tricky - you sort of have that with the levels, standards, hard and soft skills, and relevant context);

    (2) you need a model of what a performance of that task looks like when done very well, not so well, etc.(this overlaps with your levels and trajectory concepts and relies on expert judgment - you had mentioned auto-methods in the recognition blog, maybe bring those back here again); and

    (3) you need a model of how you are going to use the evidence of the specific performance of someone to make a judgment about how much of the knowledge and skill they showed you (you have the rubrics, but there are many other ways to build an evidence chain).

    One point you might want to make is that if the badges are going to be signs that a valid assessment has taken place (as opposed to say just motivating people to keep going), then the system has to have all three of these components.

    ReplyDelete