Research Design Principles for Studying Learning with Digital Badges

Web-enabled digital badges are quickly transforming the way that learning is recognized in schools and in informal learning contexts. But there are few examples or models for studying digital badges. This post introduces six design principles for studying learning with digital badges that are emerging in the Design Principles Documentation Project. These principles distinguish between summative, formative, and “transformative” research, and between using conventional forms of evidence and using the evidence contained in digital badges.

This is cross-posted at HASTAC.  Commenting is more likely there but you are required to log in to leave comments at HASTAC.  Comments are moderated here but you do not need to log in.

PS.  There are dozens of comments across several threads at that post.

Background

The Design Principles Documentation Project is capturing the design principles for using digital badges that are emerging across the 30 projects funded to develop badges in the 2012 Digital Media and Learning (DML) competition. Previous posts introduced the principles the project uncovered for using badges to recognize learning, assess learning, and motivate learning. This post introduces the principles for studying learning. While the more inclusive term “study” is used here, this is really about what is often referred to as “research and evaluation.”

Of course, research and evaluation are contentious topics in education. One of the reasons for this is that people disagree on what counts as “evidence” and what methods count as “scientific.” A 2001 report by the National Research Council laid out the argument that the “gold standard” of scientific educational research is randomized experimental trials. But the NRC also recognized that many of the most important ideas that might be tested in experimental research are unlikely to be discovered in experimental studies. This seems certain to be the case with digital badges in education. 

The principles summarized in the other three posts provide a nice context for thinking about where the science behind badges is going to come from.  These principles are more general descriptions of the specific practices that emerged as the various project figured out how to use digital badges in their contexts.  We expect these principles to be quite useful for others who wish to use badges—particularly when they are linked to examples in project and to relevant research resources.  But most of these principles are not framed as “hypothesis” that could be tested in an experiment.  Even if they did present testable hypotheses, the results would probably not generalize from the context where the experiment was conducted to other badging contexts where those findings might be applied.

Research and Evaluation of Digital Badges

Thanks to the DML competition and extensive media coverage, many schools and programs are considering using digital badges. This means that many are also beginning to ask about the research evidence concerning the effectiveness of digital badges for student outcomes. Digital badges are so new that there are very few published studies and just a handful have yet to make it through the peer review process.Sheryl Grant’s annotated bibliography provides  a nice summary of what badges research there is, along with a lot of other relevant resources from other contexts.  After the initial badges competition, HASTAC announced a separate research competition to study digital badges and made awards to five badges research projects. Some of these will be discussed below.

The DML 2012 badge awardees are still tweaking their badge systems. Our interviews confirmed that most projects started with a pretty clear idea of the learning they wanted to recognize with digital badges. But many are still sorting out how to assess that learning, and most are just beginning to consider how their recognition and assessment practices would impact motivation.

Few of the projects included any formal research or evaluation studies in their original proposals. Notably, the DML 2012 competition did not require that proposal include detailed evaluation plans. I think this was a wise decision. It is my opinion that requiring detailed evaluation plans would have led projects to prematurely search for “summative” evidence that badges were effective before they had a chance to figure out how to best support learning with badges. But our interviews with project leaders revealed that many were starting to think quite seriously about the sorts of studies they might conduct now that their badge systems were taking shape. However, many of them were also unclear as to where they might start, and few of them had even begun to grapple with the far-reaching idea of using the evidence contained in the digital badges in their research.

Important Distinctions for Studying Digital Badges

In our initial efforts to understand the studies of digital badges that were taking place or might be carried out, we uncovered three dimensions that seem helpful:

Systematicity. Arguably, the distinguishing feature of “research” is that it is systematic. Research involves systematically gathering some sort of evidence and attempting to document things in a way that could inform others. The design principles that the DPD project is identifying for recognizing, assessing, and motivating learning are mostly not coming out of systematic studies. Rather the project is attempting to capture more informal knowledge as it emerges as teams get their badge systems up and running. This fourth strand of the DPD project is concerned with more systematic efforts to create new knowledge concerning digital badges.

Purpose. Building on the assessment literature, one can distinguish between summative studies “of badges” and formative studies “for badges.” Summative studies are more naturalistic examination of the way the world is, while formative studies are more interventionist efforts to change things. While most summative studies are intended to be formative, they do so less directly. One can also distinguish transformative research that examines how entire learning ecosystems are changed or created around badges.

Evidence. We are distinguishing between studies that don’t use the evidence of learning contained in digital badges and studies that do use this evidence. What makes digital badges unique is that they contain the actual evidence (or links to evidence such as artifacts produced by learners) to support particular claims of proficiency or accomplishment. There is usually a lot of negotiation involved in deciding what learning should be recognized with badges and how that learning will be assessed. As such, the evidence contained in badges will embody the values of the program or organization that issued them. This means that the database of issued badges has enormous potential for studying learning.

Focusing only on systematic studies and crossing purposes and evidence yields six research design principles: 

The following descriptions of each research design principle draw on selected examples from the DML competition as well as the studies being conducted the awardees in the 2013 HASTAC Badges Research Competition. This will be updated over the next year as addition systematic studies get underway.

1.  Research OF Badges 

Summative studies of digital badges are likely to be the largest category of badges research. Some will rely more on interpretive methods and qualitative evidence. For example, HASTAC Badges Research awardee Katie Davis (University of Washington) will study how students and teachers in the Providence After School Alliance (PASA) experience the badges used to give high school credit for expanded learning opportunities. Katie will use interviews, questionnaires, and observations to explore (a) how badges fit in the academic and peer culture, (b) the role that badges play in motivation and achievement, and (c) whether badges connect in-school and after-school experience. Likewise, one of the studies being carried out by HASTAC Badges Research awardee Jan Plass (New York University) falls in this category. Plass and colleagues will video record game play in publicly available games with and without digital badges. They will then analyze those recordings for trends and insights into participants’ perceptions and valuations of badges, and for changes in gameplay patterns due to badges.

Other summative studies of badges might rely more on correlational methods and focus on individual differences and variables. In one of the first published peer-reviewed studies of digital badges, Abramovich, Schunn, and Higashi (2013) explored mastery-based and participation-based badges in an intelligent tutoring system for teaching proportional reasoning in mathematics. They measured self-reported motivation toward mathematics before and after the game, pre-achievement of proportional reasoning, and opinion toward badges. Correlational analyses revealed both positive and negative effects of badges on learner motivation, and that these finding interacted in turn with student ability and types of badges. The Badge Impact Survey (BIS) that Jan Plass is planning to develop based on the results of the initial observational study promises to be quite useful in this class of studies.

Other studies of the impact of digital badges will use experimental methods, such as creating different versions of the same types of badges issued. For example, the final study that Jan Plass has proposed will modify a geometry game to examine the impact of two different types of badges.  They will compare mastery badges (based on players’ own progress mastering learning goals) and performance badges (based on players’ performance relative to others). They will examine impact of the different badges on a range of individual outcomes, including motivation and learning. This study promises to provide generalizable principles about the impact of these two common types of badges in game-based learning environments. Other summative studies will be more consistent with typical program evaluations. While DML awardees were not required to include formal evaluations of their badging programs, some of them are evaluating the programs as part of their larger organizational mission.

2.  Research FOR Badges

Other studies will formatively intervene more directly in badge system design. One distinctly formative effort is the study proposed by HASTAC Badges Research Awardee Jim Diamond of the Educational Development Center. Jim has already been working intensively with the DML/Gates 2012 Awardees Who Built America? (WBA) teacher mastery badge system. Jim’s study is asking some of the same questions as Katie Davis’ study of PASA. For example Jim is asking about the role that WBA badges play in teacher professional development, and examining the ways that badge-related activities influence the development of an online teacher professional development community. What pushes this research into the formative category is that Jim is asking these questions while directly participating in efforts to build the badging system and the online professional development network.

Studying things as they are changing gets messy really quickly. And studying one’s own practice makes it hard to be “objective.” Jim certainly recognized this in his proposal. This is why he is using design-based research (DBR) methods. As articulated by Paul Cobb and colleagues in 2003, DBR builds “local” theories in the context of iterative refinements of practice. Generally speaking, DBR studies start with some relatively general design principles for getting from the current state of affairs to the desired state of affairs. The back and forth process of translating the general principles into specific features yields specific design principles. Importantly, this process also reveals the key aspects of the learning context that support the specific design principles. It is this “embodiment” of the design principles in learning contexts that is presumed to generate useful insights that others can readily build on (Sandoval, 2004).

Two ongoing explosions of badging efforts should offer numerous opportunities for systematic formative research of digital badges. A number or researchers and graduate students are involved in efforts to design badge systems for the 2013 Summer of Learning in Chicago and ongoing efforts of Hive NYC.While it is beyond the scope or timeframe of the DPD project to track of all of these efforts, it appears certain that new models of practice for formative studies of digital badge systems will emerge from these efforts.

3.  Research FOR Ecosystems

Of course, many projects are using digital badges to create new learning ecosystems or transform existing ones. Some of the projects are beginning to study this process systematically. Consider the pilot study carried out by Global Kids of a new badging system for their youth programs. A DML award paired them with DML Badge System awardee Learning Times to implement BadgeStack in Global Kids’ Race to the White House and Virtual Video Project programs. The report of the pilot study provides some examples of what this might look like. The report of the pilot study describes how badges impacted the educational programs that Global Kids had already developed. For example, they found that:     

       

Global Kids youth leaders received confirmation 48 times that evidence submitted of their work met the requirement of one of thirteen different educational objectives in their programs. At the same time, youth leaders received confirmation ten times that their evidence did not meet the requirements. Both took extra time—for the youth to submit the evidence and the GK staff to review and evaluate—but the goal of providing formative assessment was significantly advanced (page 6).

The report explains that this sort of assessment had never been carried out in the educational programs that Global Kids offer.

Other systematic studies of the transformational effects of badges on ecosystems are likely to emerge in the Summer of Learning and various Hive projects. One ambitious example is the dissertation research of Global Kids Alumni and Indiana Learning Sciences student Rafi Santo. A grant from the New York Community Trust is supporting Rafi’s extended study of the diffusion of innovations in the Hive NYC. Rafi’s study is not focusing specifically on digital badges.  But a DML award to Global Kids should help ensure that badges are systematically implemented across the Hive NYC community. This and other such efforts promise to provide more specific research design principles for studying the creation and transformation of learning ecosystems via badges and other specific innovations.

Formative studies of entire learning ecosystems are incredibly complex. There are many variables to consider, numerous principles and features to be refined, and many methods that might be used. And there is the incredibly complex relationship between mentor/teacher learning and mentee/student learning. While Jim Diamond’s study certainly has some of these characteristics, it seems like he made a wise decision to tame some of that complexity by staying within the DBR framework. However, as the badges community matures, it is certainly going to need to tackle this complexity. Fortunately, a new strand of DBR known as Design-Based Implementation Research (DBIR, Penuel et al., 2011) aims to address these additional challenges. In particular, DBIR explicitly addresses (a) the existence of multiple stakeholders with different perspectives, (b) the crucial and unique role of educators and mentors in DBR, and (c) a concern with developing capacity for sustaining change in systems.

4.  Research WITH Badges & OF Badges

 Using the evidence contained in badges offers new opportunities for summative research of badges. This includes studies of the credibility of claims made in badges. This question naturally has come up a lot around digital badges. A 2012 article in US News & World Report suggested badges might someday overturn the monopoly that colleges currently hold on formal credentials—but “only if badges are proven credible.” As badges begin to function as more formal credentials, employers and college admissions officers are wondering about the reliability of the assessments behind the badges and validity of the claims made in badges. Some have noted that the credibility of conventional credentials (grades and transcripts) are seldom systematically scrutinized. Nonetheless, more formal badges are likely to trigger studies using conventional criteria from educational and psychological testing (e.g., internal reliability, construct validity, generalizability, etc.). Mozilla’s Carla Casilli has written convincingly that the fact that badges are web-enabled means that the validity of the claims made in any badges will ultimately be crowdsourced. This means that evidence from formal reliability and validity studies might be meaningless if relevant personal or professional networks collectively ignore or dismiss that evidence.

The evidence contained in digital badges has many other potential uses. The aforementioned pilot study of badges at Global Kids provides initial examples of the how programs can use the evidence to evaluate and study their programs. Before Global Kids introduced badges, their primary evidence of learning in program evaluations were summaries of blog entries that students were asked (but not really required) to make. With BadgeStack, it was simple to link to a detailed description of the badges that were offered to program participants. Additionally, the details of who earned what badges provide a surprisingly comprehensive complete picture of the learning that was supported by the program. Examining the order in which badges were earned also allowed Global Kids to begin studying the paths that learners took through their programs. Given the challenges that many schools and programs face in evaluating and studying learning, the introduction of digital badges seems poised to unlock enormous potential in this regard.

5.  Research WITH Badges & FOR Badges

The evidence contained in digital badges also has the potential for systemic efforts to formatively improve badge systems. Consider for example, the work of Stacy Kruse, Creative Director of DML 2012 awardee Pragmatic Solutions. Stacy and a team including assessment guru David Gibson are collaborating with the Digital On-Ramps project in Philadelphia and several educational initiatives at the Corporation for Public Broadcasting (e.g.,Road Trip Nation). As Stacy put it, “before I started working with digital badges, I was working on learning analytics.” This kind of experience has left Stacy and colleagues quite enthusiastic about building learning analytics directly into the badging systems they are building, and using those results to dynamically refine what badges are available, how they are displayed, etc.

Our interviews with other DML awardees uncovered some other promising efforts to use the evidence in badges to transform badging systems. GoGoLabs CEO Lisa Dawley and the Planet Stewards project are using badges to connect educational content from the National Oceanic and Atmospheric Administration to the Next Generation Science Standards. One of their challenges is mapping the game-like curricular “quests” to the standards. Such mapping is notoriously difficult and a major obstacle to standards-based reform. Curricular activities naturally touch on multiple standards, and systems need redundancy so that students and teachers can select from multiple activities. Because badges can be more specific and because they contain actual evidence of learning, they open up entirely new formative possibilities for mapping. This same evidence can then be used summatively to examine the learning trajectories that students take.

6.  Research WITH Badges & FOR Ecosystems

Eventually researchers are likely to begin using the evidence in digital badges to systematically study and improve entire learning ecosystems. In this way it seems possible that digital badges might ultimately transform the entire learning analytics movement. But this seems unlikely to even get started until clear research design principles for summative and formative studies using the evidence in badges emerges.

References

Cobb, P., Confrey, J., Lehrer, R., & Schauble, L. (2003). Design experiments in educational research. Educational Researcher, 32(1), 9-13.

Penuel, W. R., Fishman, B. J., Cheng, B. H., & Sabelli, N. (2011). Organizing research and development at the intersection of learning, implementation, and design. Educational Researcher, 40(7), 331-337.

Acknowledgements

This work was supported by a grant from the John D. and Catherine T. MacArthur Foundation’s Digital Media and Learning Initiative.  Thanks to all the DML awardees for their continued participation in this project.  Katarina Schenke, Cathy Tran, and Rebecca Itow contributed to this work and this report.