Traditional Approaches to Validity in Assessment Innovation (Part 2: Consequential Validity)

This is the second post in a series on the topic of validity in educational assessment. In my first post, I described the traditional characterization of content-related, criterion-related, and construct-related evidence as they are relevant to educators and credentialing innovators who use and design assessments. This post summarizes traditional characterizations of “consequential validity.” This aspect of validity concerns the broader consequences of administering assessments to learners and using the resulting scores. It is a complex idea that is really crucial to many assessment and credentialing innovators (because broader change is their goal). Many measurement professionals have long argued that it an "unsanctioned" aspect of validity.  Before I write about how that is changing, I want to describe how consequential validity has traditionally been written about and why I have long disagreed.  

Traditional Approaches to Validity in Classroom Assessment and Innovative Credentialing (Part 1)

By Daniel Hickey

In my work with the Participatory Assessment Lab at Indiana University and in my graduate education courses, I spend a lot of time helping people understand validity in the context of educational assessment.  In this post, I describe validity as it has traditionally been presented to educators. I summarize what one leading textbook has long said educators should know about validity when assessing learning in their own classes, and I extend that to credentialing innovators who are developing digital badge systems, micro-credentials, and competency-based educational programs.  In subsequent posts, I will explore traditional views of “face validity” and “consequential validity.” Together, these posts will lay the groundwork for a final post that will explore several new developments in validity theory that I believe are important for these two communities.

2016 AAEEBL Midwest Meeting Keynote: Open Badges + ePortfolios: Searching for and Supporting Synergy

By Dan Hickey

This is a brief report and link to a slideshare from the keynote address at the 2016 Midwest Meeting of the Association for Authentic, Experiential, and Evidence Based Learning.  It summarizes potential synergy between these two important educational technologies, as well as progress towards this synergy by the seven leading eportfolio platforms and between AAEEBL, the Open Badges in Higher Education Project, and the Badge Alliance.

Keynote Address at AAEEBL Midwest Regional Meeting at Notre Dame

Once I get this post up I will get to the several dozen emails awaiting me in my inbox with headers suggesting that we have indeed made some progress towards increased synergy.

A Situative "Roadmap" for Synergy in Motivating Family STEM Engagement

By Dan Hickey

This is the third post about my collaboration with Iridescent Inc., a science education non-profit in LA. This new post describes how a key assumption in newer "situative" theories of motivation can resolve the tensions between prior empiricist and constructivist approaches. When combined with Design Based Research methods, this assumption can result in a coherent "roadmap" towards synergy across the three approaches. I contend that such a roadmap can help Iridescent and other informal STEM innovators find a route that takes them from current levels of engagement to much higher levels of engagement, both in terms of quantity and quality.

This post could use some work and some trimming but I need to get it up for my class and colleagues and get on to other things. Will try to clean it up soon

Motivating STEM Engagement in Children, Families, and Communities

By Daniel Hickey

 This extended post is the second in a series of post about my work with non-profit STEM education start up called Iridescent.  In a previous post I describe their Curiosity Machine and an evaluation I carried out of a program designed to encourage families to complete Engineering Design Challenges on that website from home, after attending two-hour weekly evening workshops. In this post I explore the challenges that organizations like Iridescent and science museums face in attempting to motivate inquiry learning in informal contexts. I am also using this post as my weekly contribution to the graduate education class on motivation that I am currently teaching.

The Data of Learning: A Response to Martin Kurzweil's "Responsible Use of Student Data"

by James Willis

In mid-April, Stanford University hosted the "Learning Summit 2016: Inventing the Future of Higher Education." For those of us who study how the newer processes and protocols of using student data have ethical and legal consequences, one session in particular should be of interest: Marco Molinaro (UC Davis) moderated a panel on the "Responsible Use of Student Data for Individual and Organizational Improvement" which included speakers Martin Kurzweil (ITHAKA S+R), Mitchell Stevens (Stanford), and Kent Wada (UCLA). Kurzweil provided a recent blog posting summarizing the panel discussion, raising some important points.

Engaging Children and Families in Informal Science Learning

By Dan Hickey 

In this post, I share some things I have learned about the design of "semi-formal" science learning environments that I learned working with a remarkable science education non-profit in Los Angeles called Iridescent.

AERA and Open Digital Badges

by James Willis

The 2016 conference of the American Educational Research Association (AERA) will happen next week (April 8 - April 12) in Washington, D.C. There are some events related to open digital badges, so I'll discuss them briefly along with links to additional information. There is also an interesting disparity between the growing embrace of badges and the relatively little coverage at AERA.

Recognizing, Assessing, and Motivating Entrepreneurial Mindsets

By Dan Hickey

In this longer post, I explore some of the issues around recognizing and motivating an entrepreneurial "mindset" using digital badges. I am collaborating with Rebecca DeVasher at Rose-Hulman Institute of Technology and Helen Chen at Stanford University. They and their colleagues are working with the Kern Entrepreneurial Engineering Network (KEEN) on programs to help engineering students develop the dispositions needed to be a successful entrepreneur alongside their more conventional technical skills and problem solving ability.

2016 Developments at the Badge Alliance

by James Willis and Dan Hickey in conjunction with Nate Otto

At the end of 2015 and beginning of 2016, there have been a number of changes at the Badge Alliance. In a recent blog posting, Nate Otto, Director of the Badge Alliance, sets out an ambitious agenda for 2016. Beyond the on-going work of supporting productive and evidence-rich badging projects, the Badge Alliance is also concentrating efforts on communicating across the various communities and providing technical support to the ecosystem. We summarize some of those initiatives here.

So maybe open badges can get you into Harvard (or Yale or Dartmouth) after all

By Daniel Hickey and James Willis

As summarized in Education Dive and reported in the Boston Globe, Harvard and 80 representatives from other Ivy League schools released a report arguing that admissions officers should give more attention to service-learning and volunteer activities. They recommended:

1. "Promoting more meaningful contributions to others, community service and engagement with the public good.
2. Assessing students' ethical engagement and contributions to others in ways that reflect varying types of family and community contributions across race, culture and class.
3. Redefining achievement in ways that both level the playing field for economically diverse students and reduce excessive achievement pressure."

Those of you who follow RMA can likely appreciate how much this means to us and where this post is going. Yup. Badges and assessment. That second recommendation is going to be a really tough one to implement. As we will elaborate in some length, open digital badges are intended to provide valid evidence of accomplishment outside of accredited contexts. Some new developments may allow them to serve precisely this function.

The Possibilities and Challenges of Dual Enrollment for Assessment and Beyond

By Dan Hickey
My colleague James Willis was kind enough to introduce me to Emily Swafford who is the Programs Manager at the American History Association.  I emailed Emily about our several promising forays into history education, including our work with Chris Hitchcock to develop new participatory online courses at Indiana University High School and our initial discussion of the Dual Enrollment courses supervised by Indiana University's Advance College Project.  It just so happens that the current issue of the AHA's news magazine Perspectives on History just published a timely special issue on dual enrollment that was full of ideas we care about here at RMA.  In particular, it illustrated the complex challenges that emerge for assessment around dual enrollment in general and in particular when states mandate things like DE and get directly involved in school curricula.

Resource and Repository for Open Badges

by James Willis

We are pleased to announce a repository for open badges projects, articles, case studies, and links to additional resources. Our website, Open Badges in Higher Ed, has launched and is now searchable on Google and other web browsers. This site is an on-going effort by the collaborators of the OBHE project at Indiana University.

Calculus PLAnet: Promising First steps in Participatory Supplemental Instruction in Mathematics

by Suraj Uttamchandani, Tristan Tager, and Daniel Hickey

We discuss our recent pilot of networked online approaches for providing supplemental instruction for learners in Calculus. This work responds to the raging debate over how to respond to underprepared college entrants who are at high risk for failing "gateway" introductory courses, particularly in STEM and composition. Placing such students in remedial Developmental Education (DE) courses generates tuition revenue and fits within the existing course structure. But decades of research show that this approach does not work and that many of these student never move from this "remedial ghetto" into credit-bearing courses, and fewer still go on to graduate. Most now argue for providing peer-assisted Supplemental Instruction (SI) to support all students in challenging gateway courses. But because SI does not generate tuition, it draws from existing instructional resources. Additionally, SI requires space, scheduling, and supervision that many schools struggle to sustain. In this blog post we introduce an alternative approach to supplemental instruction that draws from recent advances on participatory and networked approaches to online learning.

Digital Badges on Resumes and College Applications

by James Willis 

Our recent article, "Where Badges Work Better," published as an ELI Brief in July 2015 by the EDUCAUSE Learning Initiative as a different version, notes the difficulty in getting organizations to value badges. Translating badges to real-world outcomes like obtaining employment or entry into college is one of the next major goals of open digital badge development. This is no small task: even as badges have gained traction in educational technology, they remain a bit more remote to businesses and some college admissions departments. The Sprout Fund, a Pittsburgh, PA-based organization that "supports innovative ideas that are catalyzing change...making our community a better place to live, work, play, and raise a family" is helping move badges into the workplace and college admissions practices.

Digital Badges and the Workplace: Some Recent Resources

by James Willis 

The use and interest in open digital badges continues to grow. This is especially true for the workplace. Seen as dynamic evidence of learning and skills, badges are becoming better known and accepted by employers. This trend is expected to continue as badges become better understood in the wider conversation of credentialing and in alternative models like competency-based education. I've gathered some recent articles (with links) that discuss badges in the workplace.

Connecting Students and Teachers with Open Digital Badges

by James Willis, Dan Hickey, and John Foster*

John Foster, Ph.D. (guest blogger) is President/CEO of NOCTI and its subsidiary, Nocti Business Solutions. 

The Open Badges in Higher Education project is collaborating with a number of entrepreneurs who are expanding into digital badges. Recognizing prior learning assessment (PLA) has been difficult to work into the curricula of numerous programs. This is due to a number of reasons which may include lack of uniformity, difficulty of assessment, and inability to situate prior learning into an existing curriculum. That is not to say, however, that learning hasn't happened. As those who work to connect prior learning, alternative credentialing, and skills assessment can attest, prior learning can provide compelling evidence for learner capability in job and academic skills.

IBL Studios Issues an Open Source Badging Platform

by James Willis

We worked with Michael Amigot at IBL Studios in a previous project to launch the first instance of open badges in Open edX in Lorena Barba's Python MOOC at George Washington University. The code to issues badges is now available at GitHub as an open source tool for those interested in issuing their own Open Badges Infrastructure (OBI)-compliant badges. IBL designed this to be "[a] platform to award your own institution's badges. The badges you create and earn with this server are compatible with the specifications of the OpenBadges project."

3600 US Cerfication Bodies Lacking Third Party Validation

By Dan Hickey

Lumina Foundation just released a report with some surprising data about the manner in which most of the bodies that award professional certification validate their credentials.  Make me wonder if all of the concern over validity of badges and other evidence-rich digital credentials is focused too narrowly.

Is Self-Paced Participatory Learning Possible?

by Suraj Uttamchandani and Daniel Hickey

In this post, we discuss current efforts to offer the flexibility of self-paced learning with the interactive social engagement of participatory learning. We describe two new features in the Big Open Online Course (BOOC) on Educational Assessment that allow current learners to interact with prior learners and let learners proceed at their own pace.

Competency-Based Education, Badges, and Professional Development

by Benjamin Roome (guest blogger) and James Willis 

*Benjamin Roome, Ph.D., is Chief Product Officer for Badge List and Ethics Consultant at Ethical Resolve. 

While competency-based education (CBE) has been around for many years, a number of forces are now advancing CBE to the forefront of the educational reform. Major initiatives include the U.S. Department of Education, the Lumina Foundation, the Bill and Melinda Gates Foundation and many others. This, in turn, is transforming how students, institutions, and employers think about education. Moving away from the traditional metric of “seat time,” proponents of CBE suggest representing learning through the lens of specific competencies. This has re-ignited a debate that has been simmering for decades, which helps highlight one of the many ways digital badges may serve educational reform more broadly.

2015 Learning Impact Awards Announced!

by Gina Howard and James Willis

This year saw the 9th annual worldwide Learning Impact Awards competition put on by IMS Global Consortium. The Learning Impact Awards program encourages and recognizes the exploration and use of technology in diverse contexts within educational institutions where observable and measurable results have been achieved. A panel of expert evaluators select the winners using a series of rubrics to evaluate eight dimensions for improving teaching and learning impact. Given the recent partnership between IMS and Mozilla Open Badge, we were pleased to see that badges were central to the efforts of at least one of the winners.

Hollywood Airbrush Tanning Academy Issues Digital Badges

By Dan Hickey

My web crawler just picked up an intriguing new use of open badges as digital credentials for workplace competencies.  Hollywood Airbrush Tanning Academy partnered with Credly to offer digital badges for their graduates.  It was interested enough that I called Academy owner Simone Emmons to learn more. Sure enough, this example highlights some key points about alternative digital credentials.

IMS Global Seeks Members for Its Newest Initiative with Mozilla Foundation

By Gina Howard and James Willis 

On April 21, 2015, the IMS Global Learning Consortium (IMS/ IMS Global) announced the creation of the IMS Digital Credentialing Initiative. This new initiative will aim to accelerate the adoption, integration, and transferability of digital credentials and badges in schools, institutions, and corporations. With this new initiative, they are now seeking to form a group of IMS Digital Credentialing charter members.

Amanda Opperman on Competency-Based Badging

by Gina Howard and James Willis

In a recent meeting with Amanda Opperman, a blogger for Wonderlic, James Willis and Dan Hickey discussed the implications of awarding competency-based, open digital badges in the evolving workplace. Wonderlic is a private company that has been offering businesses and schools with a comprehensive library of assessments and surveys for each phase of the hiring and student selection process for the last 75 years. Wonderlic is beginning to explore the potential of competency-based, open digital badges associated with their assessments of vocational competencies, starting with criminal justice and health care.