Design Principles for Assessing Learning with Digital Badges

This post is cross-posted at HASTAC

by Rebecca. C. Itow and Daniel T. Hickey

This post introduces the emerging design principles for assessing learning with digital badges. This is the second of four posts that will introduce the Design Principles Documentation Project’s (introduced in a previous post) emerging design principles around recognizing, assessing, motivating and evaluating learning.

At their core, digital badges recognize some kind of learning. But if one is going to recognize learning, there is usually some kind of assessment of that learning so that claims about learning can be substantiated by evidence. Over the course of the last year, we have tracked the way that assessment practices have unfolded across the 30 DML Badges for Lifelong Learning competition winners. We have categorized these practices into ten more general principles for assessing learning with digital badges. These principles are not presented as “best practices.” Rather, these principles are meant to represent appropriate practices that seemed to work for particular projects as they designed and refined their badge systems.

No one project embodies all of these principles, and the principles mean somewhat different things to different projects. The general principles have been broken down into specific practice categories, but they are still being refined. Some of these categories are discussed here. We seek input from individuals and projects at this time as we attempt to firm up these principles and categories.

Design Principles for Assessing Learning

The following principles are ordered by their prevalence among the different projects. The first principles are employed by many badging projects, and the last principles are employed by just a few. These principles are fluid and are being refined as we enter the second year of our project. Feedback and comments on these principles are welcome and encouraged.

Use leveled badge systems. A majority of badging projects use some kind of “leveling system” that learners work through as they complete activities and projects. For example, some projects use competency levels to structure their system so that learners earn badges as they master each level of the content (e.g. learning to add fractions before learning to multiply them). Some projects have set up a system where learners earn smaller recognition like stars or points as they complete smaller goals, and these add up to a larger badge that learners can push out to their backpack (we are currently calling these metabadges). Other projects have categories of badges such as leadership badges and collaboration badges; these categories may not be levels, but they are included in this principle because they act as levels that learners can master.

Enhance validity with expert judgment. In an effort to substantiate the claims made about learning, many badging projects are using some kind of expert to assess learners and judge the artifacts they create. Projects are doing this in a variety of ways; some are using human experts like teachers or field experts; others are using some kind of computer scoring system or artificially-intelligent tutors. Several projects reported that a combination of human and computer experts provide a useful balance of nuance and automaticity. Interestingly, some badging projects are actually giving their human experts badges. In this way, the assessors can earn credentials that show off their expertise, and in some cases badges may give them privileges to be a more prominent leader in a community.

Align assessment activities to standards: Create measurable learning objectives. Almost all of the badging projects have decided to align their activities to some established standard. In some cases these are established state or national standards; in other cases they are internal standards developed by their organization. By aligning assessment activities to standards, projects are better positioned to make and warrant particular claims about the kinds of learning that takes place within their badging system.

Use performance assessments in relevant contexts. There are many different types of assessments. Projects are naturally thinking carefully about the claims they wish to make about learning and the type of learning they are supporting when selecting assessment formats. Some projects are using performance assessments. In some cases these are summative or final assessments that ask learners to use the knowledge they have practiced with in previous activities in a new and sometimes complex context. In other cases these are more formative assessments that share some features with portfolio assessment.

Use e-portfolios. Portfolio assessment can be cumbersome, but when implemented well, portfolios trace a learner’s growth over time and extensive and valuable feedback conversations can occur around them. Some projects encourage the learning community to provide feedback on the portfolios to the learners, while others make the portfolios public and encourage family and local community members to provide insights and feedback so that learners benefit from an array of opinions and points of view. Still other projects use portfolios primarily to track each learner’s growth. In these cases, experts evaluate and provide feedback, although this interaction may be limited to a small number of exchanges between the learner and expert.

Use formative functions of assessment. Formative feedback was sufficiently incorporated into enough of the projects to qualify as a specific principle. Many projects use some kind of formative assessment; this means that they create opportunities to provide feedback to the learner that shapes their ongoing and future work rather than just providing a score at the completion of an activity. Some projects do this in the form of expert feedback on artifacts. Some encourage peer feedback and collaboration, although in many cases, the peer feedback is informal and does not directly influence formal assessment.

Use mastery learning. Some projects’ goals for learners involve mastering very specific skills. These projects use conventional feedback techniques where learners practice a skill in a particular context until they master it. Many of the projects using mastery learning do so in conjunction with some kind of assessment that provides the learners with more general formative feedback about their overall progress.

Use rubrics. Some projects have invested substantial effort in identifying or developing rubrics for assessing student work. Rubrics provide both the learner and the assessor a clear idea of expected levels of mastery. Some projects create their own rubrics while others use rubrics created by schools, districts, states, or organizations. Some rubrics are developed for specific activities, while others are developed more generally to be used for a variety of activities.

Promote "hard" and "soft" skill sets. In addition to learning specific content, some projects encourage the development of skills such as leadership and collaboration. Different projects have designed creative ways for learners to demonstrate these difficult-to-assess skills, such as designing activities so that they cannot be completed without collaborating with peers. Some projects are building in places for learners to acknowledge those who helped them reach certain goals, which in turn is helping to build collegiality and encourage collaboration within their communities.

Involve students at a granular level. A few projects have decided to involve their community in the design and assessment processes. They encourage learners to think about the kinds of activities that they think would best demonstrate desired skills, and involve them in the basic design of the badge system. They ask for learner input on different aspects, from the particulars of the activities, to the kinds of assessments that will be used, to the design the badges and their specifications.

We Want Your Feedback!

Once again, not every badge system will use all of these general principles in their design. These principles outline assessment practices that should be considered when designing a badge system. Which principles inform a project will depend on the context in which they are being implemented and the goals of the project. We welcome your feedback on these general principles, and hope that this work is useful as you move your badging project forward.