The "Design Knowledge Evaporation Problem" and the Design of Complex Digital Badge Systems

By Dan Hickey
I am crunching out the final report of the Open Badges Design Principles Document Program and it pushed me to dig more deeply into the research on "knowledge evaporation" in the design of complex software architectures.  It makes me wonder if current efforts to build badges into the larger, more complex learning management systems are about to run into the wall that complex software systems always run into.
The DPD project set out to capture the new insight generated by efforts of the thirty 2012 DML awardees to build new badging systems.  Early on, Carla Casilli reminded us in one of her prescient 2012 blog posts why this was going to be difficult:

"Regardless of where you start, it’s more than likely you’ll end up somewhere other than your intended destination. That’s okay. Systems are living things, and your badge system needs to be flexible. You must embrace a bit of chaos in its design."

Somehow we had to capture the knowledge across these thirty projects. We made a lot of progress and while it was not perfect, we are about to start sharing the overall findings, and give public access, to our extensive database or the intended, enacted, and formal practices for using digital badges to recognize, assess, motivate, and study learning.

Philippe Kruchten

It was a struggle for me to design this project. We had been proscribed from directly assisting projects (that was HASTAC's job and doing so would bias our project). Mimi Ito helped us flesh out a project design that was essentially an ethnography of the emergence of the badge design principles. When I was writing the original proposal I could not locate examples of any other project attempting to distill insights from such a diverse set of innovations. However, I did uncover a strand of research within efforts to capture the knowledge that emerged in the design of complex software systems.  Scholars like Philippe Kruchten remind us that much of the knowledge that is generated in the trial and error design of complex software architectures “evaporates” as software features evolve and development teams dissolve and reform around new features and problems. Highlighting the challenge that the DPD project faced, most efforts to address the “design knowledge evaporation problem” have focused on capturing knowledge as it concerns the development of a single complex software architecture as it evolves over time.  The DPD project attempted to capture the more general principles that emerged across thirty projects.

Are Complex Badge Systems Facing a Wall?
Our new project is attempting to incorporate open digital badges in advanced, worthwhile uses in all the major course management systems across higher education. It has been a struggle. We spent months trying to connect Mozilla's BadgeKit with Open edX; we recently engineered a workaround in our collaboration with Lorena Barba's Python MOOC involving Achievery Inc and IBL Studies. It looks pretty sure that we will have initial badges to share with everybody at the Open edX Conference on Nov 18th and 19th, 2014. I am also working with Indiana University to get the Badge Safe extension for Canvas up and running and tested before Accreditrust Inc. releases their open source extension for presumed wide use and further refinement.

All of these efforts remind me that I am not a programmer and just how complicated it is to design complex software architectures. It seems that as the badge system designs get more and more complex, efforts to derive the knowledge needed to manage that complexity can be drawn from this research on complex software architectures.  A 2009 special issue of IEEE Software on Capturing Design Knowledge by Uwe Zdun was a provocative read. Zdun's introduction to the special issue included a nice sidebar summarizing Lehman and Belady's law of increasing complexity (entropy) and the law of continuing change from 1995. The first law highlights the way that knowledge of the original architecture design gets lost through bug fixes and feature enhancements. Complexity gets out of hand, but retroactive redesign requires major re-engineering which can be enormously costly. The second law posits that systems used in real-world environments must change to, or become progressively less relevant in, that environment.  So quick fixes are necessary to keep the system useful. But these quick fixes lead to further knowledge evaporation, making it progressively harder to fix the design problems.  

So here is my question: are current efforts to incorporate digital badges into the major learning management systems already facing a "tipping point” where they can no longer be refined outside of a major redesign of the entire learning management system?