re-mediating assessment

Another Misuse of Standardized Tests: Color Coded ID Cards?

2011-12-17T21:09:00.003-05:00

An October 4, 2011 Orange County Register article that reports a California high school’s policy to color code student ID cards based on their performance on state exams raises several real concerns, including student privacy. Anthony Cody in his blog post “Color Coded High School ID Cards Sort Students By Test Performance” published on October 6, 2011 in Education Week Teacher writes that “[s]tudents [at a La Palma, CA high school] who perform at the highest levels in all subjects receive a black or platinum ID card, while those who score a mix of proficient and advanced receive a gold card. Students who score "basic" or below receive a white ID card.” These cards come with privileges and are meant to increase motivation to perform well on state standardized exams. Followers’ comments and concerns posted to the blog address “fixing identity” and that testing conveys the idea that “learning and achievement isn't reward in itself. … You're not worth anything unless WE tell you are based on this one metric.” These are valid concerns, but the larger issue being highlighted here is the misuse and misapplication of the standardized tests themselves.

True, the La Palma school policy of color coding does move students into specific identities, but this is already being done, though perhaps not as explicitly as color coded ID cards; students are separated into different leveled classes and groups within those classes, and even in a “mixed level” class, the students tend to divide amongst themselves. While diversity of achievement level does exist within social groups, many times students of similar ability and experience flock together, whether or not they are given color coded ID cards. The concern of individual motivation is valid as well, on both ends of the spectrum; however, this, too, has been prevalent for years. There are those who are motivated to do well on these tests independently, those who are not interested at all, and those who need external motivation. Color coding will likely not have such a great effect on these students.

So let us turn our attention to the tests themselves, and their use and application – or misuse and misapplication. Standardized tests are useful in comparing skill sets among bodies of students. In tests on the scale of these state standardized tests, the information about performance skill among subgroups within a school, performance between schools in a particular district, and performance between districts across the state can be quite informative and helpful in assessing and guiding steps for improvement at each level. They are even helpful in designing curriculum, if used appropriately, because they set a standard of skill to be reached.

However, state standardized tests are not useful when they become the curriculum. The tests are meant to test large skill sets; they are meant to examine a student’s ability to apply a skill learned in one context to a new context. When teachers choose or are forced to exchange developing critical thinking skills for test preparation, the students and the community lose. All test prep does is teach students to take tests. It does not help them learn to be critical, thinking members of society who can analyze a book or film, weed out the spin in a political ad or news source, or understand the gravity of an oil spill or nuclear disaster halfway around the world. These critical thinking skills can and should be honed in every domain, but all too often teachers focus on memorizing vocabulary and formulas and what Romeo said in Act II, Scene i, line 2. (Incidentally, it is “Turn back, dull earth, and find thy centre out.” But does knowing that enhance your life?)

State standardized tests are not useful when they determine a teacher’s teaching schedule or bonus money. Every year I wished our school faculty would be open to sharing test scores with one another. I could see where my students fell short, and I wanted to share that with my colleagues so they would be able to work on those skills next year. I wanted to know how that teacher taught this standard…maybe I could learn something. I wanted to know where this year’s students would need extra help, and work with my colleagues to determine the best strategies to move forward. But no one wanted to share with me. Test scores have become part of a teacher’s personal pride, and often they also determine the types of classes one teaches the following year. A merit system may sound good in theory, but this is not what the tests were meant to determine, so it is not likely to work.

State standardized tests are not useful when they are the determining factor of a student’s acceptance or rejection into a higher level course. While they may give a general picture of a student’s performance – assuming they took the test seriously, were focused, and had an overall “good” testing experience – generalities do not tell the entire story. More than test scores are needed to accurately determine the potential success of a student in a particular class; one must take into account specific skills necessary to achieve, as well as the fact the scores used to place students in a 10^th grade English class often come from their 8^th grade scores, since the 9^th grade scores will not be available in time for registration and the making of the master schedule. This is not what the tests are for.

State standardized tests can be harmful when they determine whether a school will keep its funding. It seems backward to take funding away from an underperforming school and make them pay for their students to be bussed somewhere else. How will they improve? How will they fund professional development and bring in more student assistance? A lack of funding means that materials cannot be updated in a timely fashion and that stressed teachers need to focus more on getting their students to prepare for a test than developing the learning strategies and critical thinking skills that would actually help them learn and grow and perform well on exams and in life.

State standardized tests have their function, and they are good at measuring what they were intended to measure. But when the tests are misused – when a school tells students through the use of color coded ID cards that standardized tests are valued over their development of critical and analytical thinking skills, we have a problem.

Introducing Rebecca

2011-12-13T11:41:00.000-05:00

It has been just about six months since I closed up my classroom in sunny Southern California, picked up my life, and moved to Bloomington, Indiana to pursue my PhD in Learning Sciences. I can say that a year ago I certainly did not think I would be posting on a blog about Re-Mediating Assessment. I didn't think I would writing up my research or be helping teachers develop and discuss curriculum that fosters more participation and learning in their classroom. But here I am.

In fact, a year ago I was celebrating Banned Books Week with my AP Language and Composition and Honors 9 English classes, preparing my Mock Trial team for another year of success, starting a competitive forensics team, chairing the AP department, and generally trying to convince my colleagues that my lack of “traditional” tests and use of technology in my almost-paperless classroom were not only good ideas, but actually enhanced learning. A year ago I was living life in sunny Southern California as normal ... then I decided to take the GRE. And I am so glad I did. It has been an interesting journey getting to this moment.

I never thought I would become a teacher. I have an AA in Dance, an AA in Liberal Arts, and a BA in Theatre Directing, but I found that working in the top 99 seat theatre in Los Angeles left me wanting more. When I went back for my MAEd and teaching credential, I was the only one who was surprised. Teaching students, I learned, is very much like directing actors - we want them to come to conclusions, but they need to come to them in their own way in order for the outcome to be authentic.

I have worked as a choreographer, director, and actor. I have taught 10 minute playwriting and directed festivals, as well as developed curriculum around this theme. I studied Tourette Syndrome under Dr. David Commings at the Beckman Research Institute at City of Hope, and informally counseled TS students at the high school. I am a classical dancer and recently picked up circus arts as a hobby.

Each of these very different interests contributed to my teaching. We explored literature through discussion, and often took on the roles of the characters to discuss what a piece said about the society in which it was written and its relevance today. Quite often administrators walked in while students were debating the ethics of the latest redaction of Adventures of Huckleberry Finn or discussing Fitzgerald’s symbolism while dressed to the nines at a Gatsby picnic. Still, I came up against resistance when presenting my methods and ideas to my colleagues; they didn't think that I was teaching if I wasn't giving traditional tests. I had too many A's and too few F's. I knew that I could affect greater change, but I wasn't sure how. Then the opportunity to come to Indiana University and work with Dan Hickey arose, and I had to take it.

Now I am in Bloomington, reflecting on a semester of writing, learning, studying, and creating curriculum. I have immersed myself in the school and culture and work here, and have found smaller networks of people with whom I can engage, play, think, debate, and grow. I am excited and encouraged by the adventures that await in this chapter, and am looking forward.

RMA is back!

2011-12-12T18:19:00.000-05:00

After an extended hiatus, Re-Mediating Assessment is back. In the meantime, lots has happened. Michelle Honeyford completed her PhD and joined the faculty at the University of Manitoba in Winnipeg. Jenna McWilliam has moved on to Joshua Danish's lab and is focusing more directly critical theory in new media contexts. She renamed her blog too.

Lots of other things have happened that my student and I will be writing about. I promise to write shorter posts and focus more on commentary regarding assessment-related events. I have a bunch of awesome new doctoral students and collaborations who are lined up to start posting regularly about assessment-related issues.

For now I want to let everybody know that today is the official release day of a new volume on formative assessment that Penny Noyce and I edited. It has some great chapters. On the Harvard Education Press website announcing the book, my assessment hero Dylan Wiliam said:

"This is an extraordinary book. The chapters cover practical applications of formative assessment in mathematics, science, and language arts, including the roles of technology and teachers’ professional learning. I found my own thinking about formative assessment constantly being stretched and challenged. Anyone who is involved in education will find something of value in this book."

Lorrie Shepard's foreword is a nice update on the state of assessment. David Foster writes about using the tools from Mathematics Assessment Resources Services in the Silicon Valley Mathematics Initiative Dan Damelin and Kimberle Koile from the Concord Consortium write about using formative assessment with cutting edge technology. (And we appreciate that the Concord Consortium is featuring their book on their website.

For me the best part was the chapter from Paul Horwitz of the Concord Consortium. Paul wrote a nice review of his work with Thinker Tools and GenScope and the implications of that work for assessment. Paul's chapter provided a nice context for me to summarize my ten year collaboration with him around GenScope. That chapter is perhaps the most readable description of participatory assessment that I have managed to write. A much more detailed account of our collaboration was just accepted for publication by the Journal of the Learning Sciences and will appear in 2011.

I promise you will be hearing from us regularly starting in the new year. We hope you will comment and share this with others. And if you have posts or links that you think we should comment on, please let us know. I will let the rest of the team introduce themselves and add their bios to the blog as they start posting.

short-sighted and socially destructive: thoughts on Ning's decision to cut free services

2010-04-15T17:09:00.003-04:00

Lord knows I'm not a huge fan of Ning, the social networking tool that allows users to create and manage online networks. I find the design bulky and fairly counterintuitive, and modifying a network to meet your group's needs is extremely challenging, and Ning has made it extremely difficult or impossible for users to control, modify, or move network content. Despite the popularity of Ning's free, ad-supported social networks among K-16 educators, the ads that go along with the free service have tended toward the racy or age-inappropriate.

But given the Ning trifecta--it's free, getting students signed up is fast and fairly easy, and lots of teachers are using it--I've been working with Ning with researchers and teachers for the last two years. So the recent news that Ning will be switching to paid-only membership is obnoxious for two reasons.

The first reason is the obvious: I don't want to pay--and I don't want the teachers who use Ning to have to pay, either. One of the neat things about Ning is the ability to build multiple social networks--maybe a separate one for each class, or a new one each semester, or even multiple networks for a single group of students. In the future, each network will require a monthly payment, which means that most teachers who do decide to pay will stick to a much smaller number of networks. This means they'll probably erase content and delete members, starting fresh each time. The enormous professional development potential of having persistent networks filled with content, conversations, and student work suddenly disappears.

Which brings me to my second point: That anyone who's currently using Ning's free services will be forced to either pay for an upgrade or move all of their material off of Ning. This is tough for teachers who have layers upon layers of material posted on various Ning sites, and it's incredibly problematic for any researcher who's working with Ning's free resources. If we decide to leave Ning for another free network, we'll have to figure out some systematic way of capturing every single thing that currently lives on Ning, lest it disappear forever.

Ning's decision to phase out free services amounts to a paywall, pure and simple. Instead of putting limits on information, as paywalls for news services do, this paywall puts limits on participation. In many ways, this is potentially far worse, far more disruptive and destructive, far more short-sighted than any information paywall could be.

If Ning was smart, it would think a little more creatively about payment structures. What about offering unlimited access to all members of a school district, for a set fee paid at the district level? What about offering an educator account that provides unlimited network creation for a set (and much lower) fee? What about improving the services Ning provides to make it feel like you'd be getting what you paid for?

More information on Ning's decision to go paid-only will be released tomorrow. For now, I'm working up a list of free social networking tools for use by educators. If you have any suggestions, I'd love to hear them.

Update, 4/15/10, 6:48 p.m.: Never one to sit on the sidelines in the first place, Alec Couros has spearheaded a gigantic, collaborative googledoc called "Alternatives to Ning." As of this update, the doc keeps crashing because of the number of collaborators trying to help build this thing (the last time I got into it, I was one of 303 collaborators), so if it doesn't load right away, keep trying.

Diane Ravitch Editorial on the Failure of NCLB

2010-04-02T19:55:00.001-04:00

I have long admired Diane Ravitch. While I have disagreed with her on fundamental philosophical grounds, her arguments have always been grounded in the realities of schooling--even if those were the realities of conservative parents and stakeholders.

Now the evidence has shown what some of us predicted and what many of us have known for years: that external tests of basic skills and punitive sanctions were just going to lead to illusory gains (if any) and undermine other value outcomes. Her editorial in today's (April 2) Washington Post is very direct. While I disagree with her on where to go from here, I applaud her for using her audience and her reputation to help convince a lot of stakeholders who have found one reason or another to ignore the considerable evidence against continuing NCLB. Like Jim Popham has been saying for years, all of the improvement schools could make with test scores already happened between 1990 and 2000, once newspapers began publishing test scores.

Certainly this will factor into the pending NCLB reauthorization. Perhaps Indiana's Republican leadership will read this and think twice about going forward with their two core ideas for their Race to the Top reform proposal, even though it was not funded. The twin shells in their reform shotgun is "Pay for Performance" merit pay for Indiana teachers based on basic skills test scores, and "Value Added" growth modeling that ranks teachers based on how much "achievement" they instilled in their kids. For reasons Ravitch summarizes and other concerns outlined in a recent letter and report by the National Academy, the recoil from pulling these two triggers at once might be just enough to blow our schools and our children pretty far back into the 20th century.

Video of Barry McGaw on Assessment Strategies for 21st Century Skills (Measurement Working Group)

2010-03-09T17:48:00.004-05:00

I just came across a video of a keynote by Barry McGaw at last month’s Learning and World Technology Forum. McGaw heads the Intel/Microsoft/Cisco initiative known as Assessment and Teaching of 21st Century Skills. This high-powered group is aiming to transform the tests used in large-scale national comparisons and education more broadly. Their recent white papers are a must read for anyone interested in assessment or new proficiencies. McGaw’s video highlights aspects of this effort that challenge conventional wisdom about assessment. In this post I focus on McGaw’s comments on the efforts of the Measurement working group. In particular they point to (1) the need to iteratively define assessments and the constructs they aim to capture, and (2) the challenge of defining developmental models of these new skills.

Iterative Design of Assessments and Constructs
McGaw highlighted that the Measurement Working Group (led by Mark Wilson) emphasized the need for iterative refinement in the development of new measures. Various groups spent much of the first decade of the 21st century debating how these proficiencies should be defined and organized. In this abstract context, this definition process could easily consume the second decade as well. Wilson’s group argues that the underlying constructs being assessed must be defined and redefined in the context of the assessment development process. Of this, McGaw said

You think about it first, you have a theory about what you want those performances to measure. You then begin to develop ways of capturing information about that skills. But the data themselves give you information about the definition, and you refine the definition. This is the important point of pilot work with these assessment devices. And not just giving the tests to students, but giving them to students and seeing what their responses are, and discovering why they gave that response. And not just in the case where it is the wrong response but in the case where it is the correct response, so that you get a better sense of the cognitive processes underlying the solution to the task.

In other words, you can’t just have one group define standards and definitions and then pitch them to the measurement group when dealing with these new proficiencies. Because of their highly contextualized nature, we can’t just pitch standards to testing companies as has been the case with hard skills for years. This has always nagged at me in previous consideration, in that they seemed to overlook both the issue and the challenge that it presents (e.g., the Partnership for 21st Century Skills). Maybe now we can officially decide to stop trying to define what assessment scholar Lorrie Shepard so aptly labeled “21st Century Bla Bla Bla.”

The Lack of Learning Progression Models
McGaw also reiterated the concerns of the Measurement Working Groups over the lack of consensus about the way these new proficiencies develop. There is a strong consensus about the development of many of the hard skills in math, science, and literacy, and these insights are crucial for developing worthwhile assessments. I learned about this first hand developing a performance assessment for introductory genetics working with Ann Kindfield at ETS. Ann taught me the difference between the easier cause-to-effect reasoning (e.g., completing the Punnett square) and the more challenging effect-to-cause reasoning (e.g., using a pedigree chart to infer mode of inheritance). We used these and other distinctions she uncovered in her doctoral studies to create a tool that supported tons of useful studies on teaching inheritance in biology classes. Other more well known work on “learning progressions” include Ravit Duncan’s work in molecular genetics and Doug Clements’ work in algebra. In each case it took multiple research teams many years reach consensus about the way that knowledge typically developed.

Wilson and McGaw are to be commended for reminding us how difficult it is going to be to agree on the development of these much softer 21st century proficiencies. They are by their very definition situated in more elusive social and technological contexts. And those contexts are evolving. Quickly. Take for example judging credibility of information on the Internet. In the 90s this meant websites. In the past decade it came to mean blogs. Now I guess it includes Twitter. (There is a great post about this at MacArthur’s Spotlight Blog, as well as a recent CBC interview about fostering new media literacies, featuring my student Jenna McWilliams.)

Consider that I taught my 11-year-old son to look at the history page on Wikipedia to help distinguish between contested and uncontested information in a given entry. He figured out on his own how to verify the credibility of suggestions for modding his Nerf guns at nerfhaven.com and YouTube. Now imagine you are ETS, where it inevitably takes a long time and buckets of money to produce each new test. They already had to replace their original iSkills test with the iCritical Thinking test. From what I can tell, it is still a straightforward test of information from a website. Lots of firms are starting to market such tests. Some places (like Scholastic’s Expert21) will also sell you curriculum and classroom assessments that will teach students to pass the test—without ever actually going on the Internet. Of course ETS know that they can’t sell curriculum if they want to maintain their credibility. But I am confident that as soon as organizations start attaching meaningful consequences to the test, social networks will spring up telling students exactly how to answer the questions.

There is lots of other great stuff in the Measurement white paper. Much if it is quite technical. But I applaud their sobering recognition of the many challenges that these new proficiencies pose for large scale measurement. And they only get harder when these new tests are used for accountability purposes.

Next up: McGaw’s comments about the Classroom Environments and Formative Evaluation working group.

Can We Really Measure "21st Century" Skills?

2010-01-22T07:29:00.002-05:00

The members of the 21st Century Assessment Project were asked a while ago to respond to four pressing questions regarding assessment of “21st Century Skills.” These questions had come via program officers at leading foundations, including Connie Yowell at MacArthur’s Digital Media and Learning Initiative, which funds our Project. I am going to launch my efforts to blog more during my much-needed sabbatical by answering the first question, with some help from my doctoral student Jenna McWilliams.

Question One: Can critical thinking, problem solving, collaboration, communication and "learning to learn" be reliably and validly measured?

As Dan Koretz nicely illustrated in the introduction to his 2008 book, Measuring Up: What Educational Testing Really Tells Us, the answers to questions about educational testing are never simple. We embrace strongly situative and participatory view of knowing and learning, which is complicated to explain to those who do not embrace it. But I have training in psychometrics (and completed a postdoc at ETS) and have spent most of my career refining a more pragmatic stance that treats educational accountability as inevitable. When it comes to assessment, I am sort of a born-again situativity theorist. Like folks who have newly found religion and want to tell everybody how Jesus helped them solve all of the problems they used to struggle with, I am on a mission to tell everyone how situative approaches measurement can solve some nagging problems that they have long struggled with.

In short, no, we don’t believe we can measure these things in ways that are reliable and yield scores that are valid evidence of what individuals are capable of in this regard. These are actually “practices” that can most accurately be interpreted using methods accounting for the social and technological contexts in which they occur. In this sense, we agree with skeptics like Jim Greeno and Melissa Gresalfi who argued that we can never really know what students know. This point riffs on the title of the widely cited National Research Council report of the same name that Jim Pellegrino (my doctoral advisor) led. And as Val Shute just reminded me, Messick has reminded us forever that measurement never really gets directly at what somebody knows, but instead provides evidence about what the seem to know. My larger point here is my concern about what happens with these new proficiencies in schools and in tests when we treat them as individual skills rather than social practices. In particular I worry what happens to both education and evidence when students, teachers, and schools are judged according to tests of these new skills.

However, there are lots of really smart folks who have a lot of resources at their disposal who think you can measure them. This includes most of my colleagues in the 21st Century Assessment Project. For example, check out Val Shute’s great article in the International Journal of Learning and Media. Shute also has an edited volume on 21st Century Assessment coming out shortly. Likewise Dan Schwartz has a tremendous program of research building on his earlier work with John Bransford on assessments as preparation for future learning. Perhaps the most far reaching is Bob Mislevy’s work on evidence-centered design. And of course there is the new Intel-Microsoft-Cisco partnership which is out to change the face of national assessments and therefore the basis of international comparisons. I will elaborate on these examples in my next post, as that is actually the second question we were asked to answer. But first let me elaborate on why I believe that the assessment (of what individuals understand) and the measurement (of what groups of individuals have achieved) of 21st Century skills is improved if we assume that we can never really know what students know.

To reiterate, from the perspective of contemporary situated views of cognition, all knowledge and skills are primarily located in the social context. This is easy to ignore when focusing on traditional skills like reading and math that can be more meaningfully represented as skills that individuals carry from context to context. This assumption is harder to ignore with these newer ones that everyone is so concerned with. This is expecially the case with explicity social practices like collaborating and communicating, since these can't even practiced in isolated contexts. As we argued in our chapter in Val’s book, we believe it is a dangerously misleading to even use the term skills in this regard. We elected to use the term proficiencies because that term is broad enough to capture the different ways that we think about them. As 21st Century Assessment project leader Jim Gee once put it

Abstract representations of knowlege, if they exist at all, reside at the end of long chains of situated activity.

However, we also are confident that that some of the mental “residue” that gets left behind when people engage meaningfully in socially situated practices can certainly be assessed reliably and used to make valid interpretations about what individuals know. While we think these proficiencies are primarily social practices, it does not exclude recognizing the secondary “echoes” of participating in these practices. This can be done with performance assessments and other extended activities that provide some of that context and then ask individuals or groups to reason, collaborate, communicate, and learn. If such assessments are created carefully, and individuals have not been directly trained to solve the problems on the assessments, it is possible to obtain reliable scores that are valid predictions of how well individuals can solve, communicate, collaborate, and learn in new social and technological contexts. But this continues to be difficult and the actual use of such measures raises serious validity issues. Because of these issues (as elaborated below), we think this work might best be characterized as “guessing what students know.”

More to the point of the question, we believe that only a tiny fraction of the residue from these practices can be measured using conventional standardized multiple-choice tests that provide little or no context. For reasons of economy and reliability, such tests are likely to remain the mainstay of educational accountabiity for years to come. Of course, when coupled with modern psychometrics, such tests can be extremely reliable, with little score variation across testing time or version. But there are serious limitations in what sorts of interpretations can be validly drawn from the resulting scores. In our opinion, scores on any standardized test of these new skills are only valid evidence of proficiency when they are

a) used to make claims about aggregated proficiencies across groups of individuals;
b) used to make claims about changes over longer times scales, such as comparing the consequences of large scale policy decisions over years; and
c) isolated from the educational environment which they are being used to evaluate.

Hence, we are pretty sure that national and international assessments like NAEP and PISA should start incorporating such proficiencies. But we have serious concerns about using these measures to evaluate individual proficiencies in an high-stakes sorts of ways. If such tests are going to continue to be used on any high stakes decisions, they may well best be left to more conventional literacies, numeracies, and knowledge of conventional content domains, which are less likely to be compromised.

I will say that I am less skeptical about standardized measures of writing. But they are about the only standardized assessments left in wide use that actually requires students to produce something. Such tests will continue to be expensive and standardized scoring (by humans or machines) requires very peculiar writing formats. But I think the scores that result are valid for making inferences about individual proficiency in written communication more broadly, as was implied by the original question. They are actually performance assessments and as such can bring in elements of different contexts. This is particularly true if we can relax some of the needs for reliability (which requires very narrowly defined prompts and typically gets compromized and writers get creative and original). Given that I think my response to the fourth question will elaborate on my belief that written communication is probably the single most important “new proficiency” needed for economic, civic, and intellectual engagement, I think that improved testing of written communication will be the one focus of assessment research that yields the most impact on learning and equity.

To elaborate on the issue of validity, it is worth reiterating that validity is a property of the way the scores are interpreted. Unlike reliability, validity is never a property of the measure. In other words, validity always references the claims that are being supported by the evidence. As Messick argued in the 90s, the validity of any interpretation of scores also depends on the similarity between prior education and training contexts and the assessment/measurement context. This is where things get messy very quickly. As Kate Anderson and I argued in a chapter in an NSSE Yearbook on Evidence and Decision Making edited by Pam Moss, once we attach serious consequences to assessments or tests for teachers or students, the validity of the resulting scores will get compromised very quickly. This is actually less of a risk with traditional proficiencies and traditional multiple choice tests. This is because these tests can draw from massive pools of items that are aligned to targeted standards. In these cases, the test can be isolated from any preparation empirically, by randomly sampling from a huge pool of items. As we move to newer performance measures of more extended problem solving and collaboration, there necessarily are fewer and fewer items and the items become more and more expensive to develop and validate. If teachers are directly teaching students to solve the problems, then it becomes harder and harder to determine how much of an individual score is real proficiency and how much is familiarity with the assessment format (what Messick called construct-irrelevant variance). The problem is that it is impossible to ever know how much of the proficiency is “real.” Even in closely studied contexts, different observers are sure to differ in the validity—a point made most cogently in Michael Kane’s discussions of validity as interpretive argument.

Because of these validity concerns, we are terrified that the publishers of these tests of “21^st Century Skills” are starting marketing curricula and test preparation materials of those same proficiencies. Because of the nature of these new proficiencies, these new “integrated” systems raise even more validity issues than the ones that emerged under NCLB for traditional skills. Another big validity issue we raised in our chapter concerns the emergence of socially networked cheating. Once these new tests are used for high-stakes decisions (especially for college entrance), social networks will emerge to tell students how to solve the kinds of problems that are included on the tests. (This has already begun to happen, as in the "This is SPARTA!" prank on the English Advanced Placement test that we wrote about in our chapter and in a more recent "topic breach" wherein students in Winnipeg leaked the essay topic for the school's 12th grade English exam.)

Of course, proponents of these new tests will argue that learning how to solve the kinds of problems that appear on their tests is precisely what they want students to be doing. And as long as you adopt a relatively narrow view of cognition and learning, there is some truth to that assumption. Our real concern is that this unbalanced focus in addition to new standards and new tests will distract from the more important challenge of fostering equitable, ethical, and consequential participation in these new skills in schools.

That is it for now. We will be posting my responses to the three remaining questions over the next week or so. We would love to hear back from folks about their responses to the first question.

Questions remaining:
2) Which are the most promising research initiatives?
3) Is it or will it be possible to measure these things in ways that they can be scored by computer? If so, how long would it take and what sorts of resources would be needed?
4) If we had to narrow our focus to the proficiencies most associated with economic opportunity and civic participation, which ones do we recommend? Is there any evidence/research specifically linking these proficiencies to these two outcomes? If we further narrowed our focus to only students from underserved communities, would this be the same list?

Join this discussion on Grading 2.0

2009-11-16T15:08:00.001-05:00

Over at the HASTAC forum, a conversation has begun around the role of assessment in 21st-century classrooms.

The hosts of this discussion, HASTAC scholars John Jones, Dixie Ching, andMatt Straus, explain the impetus for this conversation as follows:

As the educational and cultural climate changes in response to new technologies for creating and sharing information, educators have begun to ask if the current framework for assessing student work, standardized testing, and grading is incompatible with the way these students should be learning and the skills they need to acquire to compete in the information age. Many would agree that its time to expand the current notion of assessment and create new metrics, rubrics, and methods of measurement in order to ensure that all elements of the learning process are keeping pace with the ever-evolving world in which we live. This new framework for assessment might build off of currently accepted strategies and pedagogy, but also take into account new ideas about what learners should know to be successful and confident in all of their endeavors.

Topics within this forum conversation include:

Technology & Assessment ("How can educators leverage the affordances of digital media to create more time-efficient, intelligent, and effective assessment models?");
Assignments & Pedagogy ("How can we develop assignments, projects, classroom experiences, and syllabi that reflect these changes in technology and skills?");
Can everything be graded? ("How important is creativity, and how do we deal with subjective concepts in an objective way, in evaluation?"); and
Assessing the assessment strategies ("How do we evaluate the new assessment models that we create?").

The conversation has only just started, but it's already generated hundreds of visits and a dozen or so solid, interesting comments. If you're into technology, assessment and participatory culture, you should take a look. It's worth the gander.

Here's the link again: Grading 2.0: Assessment in the Digital Age.

The Void Between Colleges of Education and the University Teaching and Learning

2009-10-27T15:36:00.005-04:00

In this post, I consider the tremendous advances in educational research I am seeing outside of colleges of education and ponder the relevance of mainstream educational research in light of the transformation of learning made possible by new digital social networks.

This weekend, the annual conference of the International Society for the Scholarship of Teaching and Learning took place at Indiana University. ISSOTL is the home of folks who are committed to studying and advancing teaching and learning in university settings. I saw several presentations that are directly relevant to what we care about here at Re-Mediating Assessment. These included a workshop on social pedagogies organized by Randy Bass, the Assistant Provost for Teaching and Learning at Georgetown, and several sessions on open education, including one by Randy and Toru Iiyoshi, who heads the Knowledge Media Lab at the Carnegie Foundation. Toru co-edited the groundbreaking volume Opening up Education, of which we here at RMA are huge fans. (I liked it so much I bought the book, but you can download all of the articles for free—ignore the line at the MIT press about sample chapters).

I presented at a session about e-Portfolios with John Gosney (Faculty Liaison for Learning Technologies at IUPUI) and Stacy Morrone (Associate Dean for Learning Technologies at IU). John talked about the e-Portfolio efforts within the Sakai open source collaboration and courseware platform; Stacy talked about e-Portfolio as it has been implemented in OnCourse, IU’s instantiation of the Sakai open source course collaboration platform. I presented about our efforts to advance participatory assessment in my classroom assessment course using newly available wikis and e-Portfolio tools in Oncourse (earlier deliberation on those efforts are here; more posted here soon). I was flattered that Maggie Ricci of IU’s Office of Instructional Consulting interviewed me about my post on positioning assessment for participation and promised to post the video this week (I will update here when I find out).

I am going to post about these presentations and how they intersect with participatory assessment as time permits over the next week or so. In the meantime, I want to stir up some overdue discussion over the void between the SOTL community and my colleagues in colleges of education at IU and elsewhere. In an unabashed effort to direct traffic to RMA and build interest in past and forthcoming posts, I am going to first write about this issue. I think it raises issues about the relevance of colleges of education and suggests a need for more interdisciplinary approaches to education research.

I should point out that I am new to the SOTL community. I have focused on technology-supported K-12 education for most of my career (most recently within the Quest Atlantis videogaming environment). I have only recently begun studying my own teaching in the context of developing new core courses for the doctoral program in Learning Sciences and in trying to develop online courses that take full advantage of new digital social networking practices (initial deliberations over my classroom assessment course are here). I feel sheepish about my late arrival because I am embarrassed about the tremendous innovations I found in the SOTL community that have mostly been ignored by educational researchers. My departmental colleagues Tom Duffy, who has long been active in SOTL here at IU, and Melissa Gresalfi have recently gotten seriously involved as well. The conference was awash with IU faculty, but I only saw a few colleagues from the School of Education. One notable exception was Melissa’s involvement on a panel on IU’s Interdisciplinary Teagle Colloquium on Inquiry in Action. I could not go because it conflicted with my own session, but this panel described just the sort of cross-campus collaboration I am aiming to promote here. I also ran into Luise McCarty from the Educational Policy program who heads the school’s Carnegie Initiative on the Doctorate for the school.

My search of the program for other folks from colleges of education revealed another session that was scheduled against mine and that focused on the issue I am raising in this post. Karen Swanson of Mercer University and Mary Kayler of George Mason reported on the findings of their meta-analysis of the literature on the tensions between colleges of education and SOTL. The fact that there is enough literature on this topic to meta-analyze points out that this issue has been around for a while (and suggests that I should probably read up before doing anything more than blogging about this issue.) From the abstract, it looks like they focused on the issue of tenure, which I presume refers to a core issue in the broader SOTL community: that SOTL researchers outside of schools of education risk being treated as interlopers by educational researchers, while treated as dilettantes by their own disciplinary communities. This same issue was mentioned in other sessions I attended as well. But significantly from my perspective, it looks like Swanson and Kayler looked at this issue from the perspective of Education faculty, which is what I want to focus on here. I have tenure, but I certainly wonder how my increased foray into the SOTL community will be viewed when I try to get promoted to full professor.

I will start by exploring my own observations about educational researchers who study their own university teaching practices. I am not in teacher education, but I know of a lot of respected education faculty who seem to be conducting high quality, published research about their teacher education practices. However, there is clearly a good deal of pretty mediocre self-study taking place as well. I review for a number of educational research journals and conferences. When I am asked to review manuscripts or proposals for educational research carried out in classrooms in the college of education, I am quite suspect. Because I have expertise in motivation and in formative assessment, I get stacks of submissions of studies of college of education teaching that seem utterly pointless to me. For example, folks love to study whether self______ is correlated with some other education relevant variables. The answer is always yes, (unless their measures are unreliable), and then there is some post hoc explanation of the relationships with some tenuous suggestions for practice. Likewise, I review lots of submissions that examine whether students who get feedback on learning to solve some class of problems learn to solve those problems better than students whose feedback is withheld. Here the answer should be yes, since this is essentially a test of educational malpractice. But the studies often ignore the assessment maxim that feedback must be useful and used, and instead focus on complex random assignment so that their study can be more “scientific.” I understand the appeal, because they are so easy to conduct and there are enough examples of them actually getting published to provide some inspiration (while dragging down the over effect size of feedback in meta-analytic studies). While it is sometimes hard to tell, these “convenience” studies usually appear to be conducted in the author’s own course or academic program. So, yes, I admit that when that looks to be the case, I do not expect to be impressed. I wonder if other folks feel the same way or if perhaps I am being overly harsh.

Much of my interest in SOTL follows from my efforts to help my college take better advantage of new online instructional tools and to help take advantage of social networking tools in my K-12 research. While my colleagues in IU Bloomington and IUPUI are making progress, I am afraid that we are well behind the curve. While I managed to attend a few SOTL sessions, I saw tremendous evidence of success that I will write about in subsequent posts. Randy Bass and Heidi Elmendorf (also of Georgetown) showed evidence of deep engagement on live discussion forums that simply can’t be faked; here at IU, Phillip Quirk showed some very convincing self-report data about student engagement in our new interdisciplinary Human Biology Program, which looks like a great model of practice for team-teaching courses. These initial observations reminded me of the opinion of James Paul Gee, who leads the MacArthur Foundation’s 21st Century Assessment Project (which partly sponsors my work as well). He has stated on several occasions that “the best educational research is no longer being conducted in colleges of education.” That is a pretty bold statement, and my education colleagues and I initially took offense to it. Obviously, it depends on your perspective; but in terms of taking advantage of new digital social networking tools and the movement towards open education and open-source curriculum, it seems like it may already be true.

One concern I had with SOTL was the sense that the excesses of “evidence-based practice” that has infected educational research was occurring in SOTL. But I did not see many of the randomized experimental studies that set out to “prove” that new instructional technology “works.” I have some very strong opinions about this that I will elaborate on in future posts; for now I will just say that I worry that SOTL researchers might get are too caught up in doing controlled comparison studies of conventional and online courses that they completely miss the point that online courses offer an entirely new realm of possibilities for teaching and learning. The “objective” measures of learning normally used in such studies are often biased in favor of traditional lecture/text/practice models that train students to memorize numerous specific associations; as long as enough of those associations appear on a targeted multiple-choice exam, scores will go up. The problem is that such designs can’t capture the important aspects of individual learning and any aspects of the social learning that is possible in these new educational contexts. Educational researchers seem unwilling to seriously begin looking at the potential of these new environments that they have “proven” to work. So, networked computers and online courses end up being used for very expensive test preparation…and that is a shame.

Here at RMA, we are exploring how participatory assessment models can foster and document all of the tremendous new opportunities for teaching and learning made possible by new digital social networks, while also producing convincing evidence on these “scientific” measures. I will close this post with a comment that Heidi Elmendorf made in the social pedagogies workshop. I asked her why she and the other presenters were embracing the distinction between “process” and “product.” In my opinion, this distinction is based on outdated individual models of learning; it dismisses the relevance of substantive communal engagement in powerful forms of learning, while privileging individual tests as the only “scientific” evidence of learning. I don’t recall Heidi’s exact response, but she immediately pointed out that her disciplinary colleagues in Biology leave her no choice. I was struck by the vigorous nods of agreement from her colleagues and the audience. Her response really brought be me back down to earth and reminded me how much work we have to do in this regard. In my subsequent posts, I will try to illustrate how participatory assessment can address precisely the issue that Heidi raised.

Positioning Portfolios for Participation

2009-10-01T11:09:00.010-04:00

Much of our work in our 21st Century Assessment project this year has focused on communicating participatory assessment to broader audiences whose practices we are trying to inform. This includes:

classroom teachers whose practices we are helping reshape to include more participation (like those we are working with in Monroe County right now);

other assessment researchers who seem to dismiss participatory accounts of learning as “anecdotal” (like my doctoral mentor Jim Pellegrino who chaired the NRC panel on student assessment);

instructional innovators who are trying to support participation while also providing broadly convincing accounts of learning (like my colleagues Sasha Barab and Melissa Gresalfi whose Quest Atlantis immersive environment has been a testbed for many of our idea about assessment);

faculty in teacher education who are struggling to help pre-service teachers build professional portfolios while knowing that their score on the Praxis will count for much more (and whose jobs are being threatened by efforts in Indiana to phase out teacher education programs and replace them with more discipline-based instruction);

teachers in my graduate-level classroom assessment course who are learning how to do a better job assessing students in their classrooms, as part of their MA degree in educational leadership.

It turns out that participatory approaches to assessment are quite complicated, because they must bridge the void between the socially-defined views of knowing and learning that define participation, and the individually-defined models of knowing and learning that have traditionally been taken for granted by the assessment and measurement communities. As our project sponsor Jim Gee has quite succinctly put: Your challenge is clarity.

As I have come to see most recently, clarity is about entry. Where do we start introducing this comprehensive new approach? Our approach itself is not that complicated really. We have it boiled down to a more participatory version of Wiggins' well known Understanding by Design. In fact we have taken to calling our approach Participation by Design (or if he sues us, Designing for Participation). But the theory behind our approach is maddeningly complex , because it has to span the entire range of activity timescales (from moment-to-moment classroom activity to long-term policy change) and characterizations of learning (from communal discourse to individual understanding to aggregated achievement).

Portfolios and Positioning
Now it is clear to me that the best entry point is the familiar notion of the portfolio. Portfolios consist of any artifacts that learners create. Thanks to Melissa Gresalfi, I have come to realize that the portfolio, and the artifacts that they contain, are ideal for explaining participatory assessment. This is because portfolios position (where position is used as a verb). Before I get to the clarity part, let me first elaborate on what this means.

It turns out that portfolios can be used to position learners and domain content in ways that bridges this void between communal activity and aggregated attainment. In a paper with Caro Williams about the math project that Melissa and I worked on together, Melissa wrote that

“positioning, as a mechanism, helps bridge the space between the opportunities that are available for participation in particular ways and what individual participants do”

Building on the ideas of her doctoral advisor Jim Greeno (e.g., Greeno and Hull, 2002) Melissa explained that positioning refers to how students are positioned relative to content (called disciplinary positioning) and how they are positioned relative to others (called interpersonal positioning). As I will add below, positioning also refer to how instructors are positioned relative to the students and the content (perhaps called professorial positioning). This post will explore how portfolios can support all three types of positioning in more effective and in less effective ways.

Melissa further explained that positioning occurs at two levels. At the more immediate level positioning concerns the moment-to-moment process in which students take up opportunities that they are presented with. Over the longer term, students become associated with particular ways of participating in classroom settings (these ideas are elaborated by scholars like Dorothy Holland and Stanton Wortham). This post will focus on identifying two complementary functions for portfolios helps them support both types of positioning.

Portfolios and Artifacts
Portfolios are collections of artifacts that students created. Artifacts support participation because they are where students apply what they are learning in class to something personally meaningful. In this way they make new meanings. In our various participatory assessment projects, artifacts have included

the “Quests” that students complete and revise in Quest Atlantis’ Taiga world where they explain, for example, their hypothesis for why the fish in the Taiga river are in decline;
the remixes of Moby Dick and Huck Finn that students in Becky Rupert’s class at Aurora Alternative High School create in their work with the participatory reading curricula that Jenna McWilliams is creating and refining.
the various writing assignments that the English teachers in Monroe and Greene County have their students complete in both their introductory and advanced writing classes;
the wikifolio entries that my students in my graduate classroom assessment course complete where they draft examples of different assessment items for a lesson in their own classrooms, and state which of the several item writing guidelines in the textbook they found most useful.

Problematically Positioned Portfolios
Probably the most important aspect of participatory assessment has to do with the way portfolios are positioned in the classroom. We position them so they serve as a bridge between the communal activities of participatory classroom and the individual accountability associated with compulsory schooling. If portfolios are to serve as a bridge, they must be firmly anchored. On one side they must be anchored to the enactment of classroom activities that support students’ creation of worthwhile portfolios. On the other side they must be anchored to the broader accountability associated with any formal schooling.

To keep portfolio practices from falling apart (as they often do) it is crucial that they rest on these two anchors. If accountability is placed on the portfolio, the portfolio practice will collapse. In other words, don’t use the quality of the actual portfolio artifacts for accountability. Attaching consequences to the actual artifacts means that learners will expect precise specifications regarding those artifacts, and then demand exhausting feedback on whether the artifacts meet particular criteria. And if an instructor’s success is based on the quality of the artifacts, that instructor will comply. Such classrooms are defined by an incessant clamor from learners asking “Is this what you want???”

When portfolios are positioned this way (and they often are), they may or may not represent what students actually learned and are capable of. When positioned this way, the portfolio is more representative of of (a) the specificity of the guidelines, (b) their ability to follow those guidelines, and (3) the amount of feedback they get from the instructor. Accountability-oriented portfolios position disciplinary knowledge as something to be competitively displayed rather than something to be learned and shared, and portfolios position students as competitors rather the supporters. Perhaps most tragically, attaching consequences to artifacts positions instructors (awkwardly) as both piano tuners and gatekeepers. As many instructors (and ex-instructors) know, doing so generates massive amounts of work. This is why it seems that many portfolio-based teacher education programs rely so heavily on doctoral students and adjuncts who may or may not be qualified to teach courses. The more knowledgeable faculty members simply don’t have the time to help students with revision after revision of their artifacts as students struggle to create the perfect portfolio. This is the result of positioning portfolios for production.

Productive Positioning Within Portfolios
Portfolio are more useful when they are positioned to support reflection. Instead of grading the actual artifacts that students create, any accountability should be associated with student reflection on those artifacts. Rather than giving students guidelines for producing their artifact, students need guidelines for reflecting on how that artifact illustrates their use of the “big ideas” of the course. We call these relevant big ideas, or RBIs. The rubrics we provide students for their artifacts essentially ask them to explain how their artifact illustrates (a) the concept behind the RBI, (b) the consequences of the RBI for practice, and (c) what critiques others might have of this characterization of the RBI. For example:

Students in my classroom assessment course never actually “submit” their wikifolios of example assessments. Rather, three times a semester they submit a reflection that asks them to explain how they applied the RBIs of the corresponding chapter.

Students in Taiga world in Quest Atlantis submit their quests for review by the Park Ranger (actually their teacher but they don’t know that). But the quest instructions (the artifact guidelines) also include a separate reflection section that asks students to reflect on their artifact. The reflection prompts are designed to indirectly cue them what their quest was supposed to address.

Students in Becky Rupert’s English class are provided a rubric for their remixes that ask them to explain how that artifact illustrates how an understanding of genre allows a remix to be more meaningful to particular audiences.

Assessing the resulting reflections positions portfolios, students, and teachers in ways that strongly support participation. For example, if the particular student’s artifact actually does not lend itself to applying the RBIs, my classroom assessment students can simply indicate that in their assignment. This is important for at least three reasons:

it allows full individualization for students and avoids a single ersatz assignment that is only half-meaningful to some students and mostly meaningless to the rest;
understanding if and how ideas from a course do not apply is a crucially important part of that expertise.
The reflection itself provides more valid evidence of learning, precisely because it can include very specific guidelines. We give students very specific guidelines asking them to reflect on the RBIs conceptually, consequentially, and critically.

For example, the mathematics teachers in the classroom assessment course are going to discover that it is very difficult to create portfolio assessments for their existing mathematical practices. Rather than forcing them to do so anyways (and giving them a good grade for an absurd example), they can instead reflect on what it is about mathematics that makes it so difficult, and gain some insights into how they might more readily incorporate project-based instruction into their classes. The actual guidelines for creating good portfolios are in the book when they need them; reflecting on those guidelines more generally will set them up to use them more effectively and meaningfully in the future.

Another huge advantage of this way of positioning portfolios is that it greatly eliminate a lot of the grading busywork and allows more broadly useful feedback. In the Quest Atlantis example, our research teacher Jake Summers of Binford Elementary discovered that whenever the reflections were well written and complete, the actual quest submission would also be well done. In the inevitable press for time, he just started looking at the artifacts. Similarly in my classroom assessment course, I will only look need to go back and look at the actual wikifolio entries when a reflection is incomplete or confusing. Given that the 30 students each have 8 entries, it is impossible to carefully review all 240 entries and provide meaningful feedback. Rather throughout the semester, each of the students have been getting feedback from their group members and from me (as they specifically request and as time permits). Because the artifacts are not graded, students understand the feedback they get as more formative than summative, and not as instructions for revision. While some of the groups in class are still getting the hang of it, many of the entries are getting eight or nine comments along with comments on comments. Because the entries are wikis it is simple for the originator go in and revise as appropriate. These students are starting to send me messages that, for me, suggest that the portfolio has indeed been positioned for participation: “Is this what you meant?” (emphasis added). This focus on meaning gets at the essence of participatory culture.

In a subsequent post, I will elaborate on how carefully positioning portfolios relative to (a) the enactment of classroom activities and (b) external accountability can further foster participation.

Participation versus Compulsion

2009-10-01T10:19:00.000-04:00

In Sleeping Alone and Starting Out Early, Jenna McWilliams offers up a concise summary of the value of blogging for schools. Her post got me reflecting on the complex intersection of participation (in public persistent discourse as you have described) and compulsion (as in the inevitable way that compulsory attendance compels students to attend but not necessarily participate). I am thinking today context of the graduate-level education course we are teaching. We are trying to coax some busy teachers who are getting graduate degrees in educational leadership to participate in meaningful semi-public discourse around improving their classroom assessment practices. We have students building and sharing wikifolios where they apply what they are learning about assessment to their own classroom practice, and using forums to discuss the big ideas in the text. The resistance to the participatory aspects from some students is remarkably strong. We have agonized over the various design features that will compel all students to participate more than they would otherwise. While we are finding success, we are in part doing so by linking their participation to a grade. It seems effective but bizarre, for example, to motivate students to engage consequentially and critically in a discussion forum by pointing out that doing so will prepare them to engage conceptually on an exam at the end of the semester.

When I say we are finding success, it is because I see a strong level of engagement emerging across the class, and that the most reluctant participants are indeed engaging in ways that for me meet the level of accountability associated with this required course. If somebody is going to have a graduate degree in educational leadership, then they need to be able to engage in meaningful discussions around that aspect of practice. And in case I drop the ball, there are faculty members in charge of the graduate programs looking over my shoulder (and to some extent watching my back); if we drop the ball there is an accreditation agency out there looking over our collective shoulders (but probably not watching our backs).

So in this regard I do believe we are finding success. But I worry that we are not supporting the handful of students who are disposed (as in have the disposition, a carefully chosen term) to becoming 21st century educational leaders. For example, am I sacrificing the chance to help a couple of these students who might end up keeping a blog that critiques and unpacks local and state accountability practices that are buffeting teachers and administrators in their district in exchange for passing exam scores and adequate teaching evaluations? [Anybody who teaches required courses knows that the way to get great evaluations is go really easy on students and emphasize what they already know and make them think they have learned a lot.]

So this is what it boils down to: Compulsory attendance versus scaffolded participation. For me, this is the major issue facing education today. I do think that blogging is a bridge too far for many novices. But I do think that well structured discussion forums can give beginners the opportunity to try on the identities and try out the discourses of participatory culture. I also want to second Jenna’s shoutout for the collection of participatory media activities that Sam Rose and Howard Rheingold have provided at sociamediaclassroom. It is the best collection out there, and a great starting place for anybody looking to refine a more specific set for particular educational contexts. We are putting some together for our teachers in our project with teachers in Monroe and Eastern Green Counties, and they should be available on a site at ning.com soon.

And finally, what I would not give to be able to put everything else aside to blog. I have four of five posts that wake me up every morning. But then I remember I still have an overdue annual report for the National Science Foundation that I have been working on for a week. I will be lucky if my project officer reads it. But I have to crank it out to keep the grant money coming in so my graduate students can eat and have a place to sleep.

Q & A with Henry Jenkins' New Media Literacies Seminar

2009-09-09T13:13:00.007-04:00

New media scholar Henry Jenkins is teaching a graduate seminar on new media literacies at the University of Southern California's Annenberg School for Communication. The participants had raised the issues of assessment and evaluation, especially related to educational applications of new media. Henry invited Dan Hickey to skype into their class to field questions about this topic. They perused some of the previous posts here at re-mediating assessment and proceeded to ask some great questions. Over the next few weeks, Dan and other members of the participatory assessment team will respond to these and seek input and feedback from others.

The first question was one they should have answered months ago:

Your blog post on what is not participatory assessment critiqued prevailing assessment and testing practices. So what is participatory assessment?

The answer to this question has both theoretical and practical elements. Theoretically, participatory assessment is about reframing all assessment and testing practices as different forms of communal participation, embracing the views of knowledgeable activity outlined by media scholars like Henry Jenkins, linguists like Jim Gee, and cognitive scientists like Jim Greeno. We will elaborate on that in subsequent posts, hopefully in response to questions about this post. But this first post will focus more on the practical answer.

Our work in participatory assessment takes inspiration from the definition of participatory culture in the 2006 white paper by Project New Media Literacies:

not every member must contribute, but all must believe they are free to contribute when ready and that what they contribute will be appropriately valued.

As Henry, Mimi Ito, and others have pointed out, such cultures define the friendship-driven and interest-driven digital social networks that most of our youth are now immersed in. This culture fosters tremendous levels of individual and communal engagement and learning. Schools have long dreamed of attaining such levels but have never even come close. Of course, creating (or even allowing) such a culture in compulsory school settings requires new kinds of collaborative activities for students. Students like those in Henry’s class, and students in our Learning Sciences graduate program are at the forefront of creating such activities. Participatory assessment is about creating guidelines to help students and teachers use those activities to foster both conventional and new literacy practices. Importantly, these guidelines are also intended to produce more conventional evidence of the impact of these practices on understanding and achievement that will always be necessary in any formal educational context. Such evidence will also always be necessary if there is to be any sort of credentialing offered for learning that takes place in less formal contexts.

Because successful engagement with participatory cultures depends as much on ethical participation (knowing how) as it does on information proficiency (knowing what), At the most basic practical level participatory assessment is intended to foster both types of know-how. More specifically, participatory assessment involves creating and refining informal discourse guidelines that students and teachers use to foster productive communal participation in collaborative educational activities, and then in the artifacts that are produced in those activities. Our basic idea is that before we assess whether or not individual students understand X (whatever we are trying to teach them), they must first be invited to collectively “try on” the identities of the knowledge practices associated with X. We do this by giving ample opportunities to “try out” discourse about X, by aggressively focusing classroom discourse towards communal engagement in X, and discouraging a premature focus on individual students’ understanding of X (or even their ability to articulate the concept of X). Premature focus on individual understanding leaves the students who are struggling (or have perhaps not even been trying) self-conscious and resistant to engagement. This will make them resist talking about X. Even more problematically, they will resist even listening to their classmates talk about X. Whatever the reason the individual is not engaging, educators must help all students engage with increased meangingfulness.
To do participatory assessment for activity A, we first define the relevant big ideas (RBIs) of the activity (i.e., X, Y, and perhaps, Z). We then create two simple sets of Discourse Guidelines to ensure that all students enlist (i.e., use) X, Y, and Z in the discourse that defines the enactment of that activity. Event reflections encourage classrooms to reflect on and critique their particular enactment of the activity. These are informal prompts that are seamlessly embedded in the activities. A paper we just wrote for the recent meeting of the European Association for Research on Learning and Instruction in Amsterdam discussed examples from our implementation of Reading in a Participatory Culture developed by Project New Media Literacies. That activity Remixing and Appropriation used new media contexts to conventional literary notions like genre and allusion. One of the Event Reflection prompts was

How is the way we are doing this activity helping reveal the role of genre in the practice of appropriation?

Given that the students had just begun to see how this notion related to this practice, the students struggled to make sense of such questions. But it set the classroom up to better appreciate how genre was just as crucial to Melville’s appropriation of the Old Testament in Moby-Dick as it was to the music video "Ahab" by nerdcore pioneer MC Lars. The questions are also worded to introduce important nuances that will help foster more sophisticated discourse (such as the subtle distinction between a concept like genre and a practice like appropriation)
Crucially, the event guidelines were aligned to slightly more formal Activity Reflections. These come at the end of the activity, and ask students to reflect on and critique the way the particular activities were designed, in light of the RBIs:

How did the way that the designers at Project New Media Literacies made this activity help reveal the role of genre in the practice of appropriation?

Note that the focus of the reflection and critique has shifted from the highly contextualized enactment of the activity, the more fixed design of the activity. But we are still resisting the quite natural tendency to begin asking ourselves whether each student can articulate the role of genre in appropriation. Rather than ramping up individual accountability, we first ramp up the level of communal discourse by moving from the rather routine conceptual engagement in the question above, and into the more sophisticated consequential and critical engagement. While these are not the exact questions we used, these capture the idea nicely:

Consequential Reflection: How did the decision to focus on both genre and appropriation impact the way this activity was designed?

Critical Reflection: Can you think of a different or better activity than Moby-Dick or Ahab to illustrate genre and appropriation?

We are still struggling to clarify the nature of these prompts, but have found a lot of inspiration in the work of our IU Learning Sciences colleagues Melissa Gresalfi and Sasha Barab, who have been writing about consequential engagement relative to educational video games.

The discourse fostered by these reflections should leave even the most ill-prepared (or recalcitrant) participant ready to meaningfully reflect on their own understanding of the RBIs. And yet, we still resist directly interrogating that understanding, in order to continue fostering discourse. Before jumping to assess the individual, we first focus on the artifacts that the individual is producing in the activity. This is done with Reflective Rubrics that ask the students to elaborate on how the artifact they are creating in the activity (or activities) reflects consequential and critical engagement with the RBI. As will be elaborated in a subsequent post, these are aligned to formal Assessment Rubrics of the sort that teachers would use to formally assess and (typically) grade the artifacts.

Ultimately, participatory assessment is not about the specific reflections or rubrics, but the alignment across these increasingly formal assessments. By asking increasingly sophisticated versions of the same questions, we can set remarkably high standards for the level of classroom discourse and the quality of student artifacts. In contrast to conventional ways of thinking about how assessment drive curriculum, former doctoral student Steven Zuiker help us realize that we have to thing impact of these practices using the anthropological notion of prolepsis. It helps us realize that anticipation of the more formal assessments motivates communal engagement in the less formal reflective process. By carefully refining the prompts and rubrics over time, we can attain such high standards for both that any sort of conventional assessment of individual understanding or measure of aggregated achievement just seems…well…. ridiculously trivial.
So the relevant big idea here is that we should first focus away from individual understanding and achievement if we want to confidently attain it with the kinds of participatory collaborative activities that so many of us are busily trying to bring into classrooms.

putting the "our" in "open source"

2009-08-04T18:47:00.000-04:00

on the dearth of women in the open source programming movement

In case you haven't seen it yet, I wanted to link you to Kirrily Robert's keynote at this year's O'Reilly Open Source Convention. Robert's keynote, "Standing Out in the Crowd," focused on the dearth of female developers in the open source movement. She offers this image from the 2008 Linux Kernel Summit:

Image credit: Jonathan Corbet, lwn.net

Robert writes:

This is a normal sort of open source project. I’ll give you a minute to spot the women in the picture. Sorry, make that woman. She’s on the right. Can you see her?

While women are a minority in most tech communities, Robert explains, the gender disparity in open source development is more pronounced than in other technology disciplines. While women make up between 10-30% of the tech community in general, they comprise about 5% of Perl developers, about 10% of Drupal developers, and (according to an EU-funded survey of open source usage and development, called FLOSSPOLS) about 1.5% of open source contributors in general.

Robert surveyed female developers to find out why women seem to be so reluctant to contribute to open source projects; the most common reason was some variation of "I didn't feel welcome." She points to a pair of innovative projects whose members have actively worked to recruit women. One is the Organization for Transformative Works' (OTW) Archive of Our Own (or AO3); the other is Dreamwidth, a blogging and community platform forked from the LiveJournal codebase. Both projects focused on recruiting women, not to be inclusive but because they felt it was essential for the success of the projects.

The entire talk is worth a read-through or a listen, but I want to highlight one key point from the set of strategies she offers for recruiting diverse candidates: Find potential users of the application and teach them programming, instead of recruiting good programmers and teaching them about the value of the application. She says:

If you’re working on a desktop app, recruit desktop users. If you’re writing a music sharing toolkit, recruit music lovers. Don’t worry about their programming skills. You can teach programming; you can’t teach passion or diversity.

I've been thinking about this very aspect of the open education movement since the Sakai 2009 Conference I attended last month. Sakai offers an open source collaborative learning environment for secondary and higher education institutions, emphasizing openness of knowledge, content, and technology. This embrace of openness was evident in every aspect of the conference, except for one: The notable lack of educators in the panels and audience.

If you want a good open education resource, you need to start by recruiting open source-friendly educators. Otherwise, you run the risk of developing a highly robust, highly functional tool that's limited only in its ability to offer the features educators actually want.

What Participatory Assessment is NOT

2009-07-22T16:48:00.002-04:00

Obviously a blog devoted to participatory assessment should explain what that means. And try to do so in simple every day terms. This is the first in a series of posts that attempts to do so. Quite specifically, participatory assessment is first about assessing and improving a communities social participation knowledgeable activity, with the added bonus of fostering the understanding and achievement of the individuals in that community. To explain what this means, this post introduces some basic ideas and terms from educational assessment help explain what this does not mean. It was authored by Dan Hickey and Jenna McWilliams.

What is Assessment?
It makes sense to start first by saying what we mean by assessment. For us, assessment is about documenting outcomes. Most of the time we are interested in assessing what someone knows, or can do. Traditionally, educational assessment has been linked to documenting what students know and what they have learned. Often it has been associated with what teachers do in classrooms, at the end of a unit or a term. In the language of modern cognitive science, this view is well represented in the title of the 2001 National Research Council report entitled Knowing What Students Know: The Science and Design of Educational Assessment. This report (whose editors include Dan’s doctoral advisor Jim Pellegrino) nicely describes the mainstream view of educational assessment in terms of conceptual structures called “schema.” We here at re-mediating assessment often use the phrase “assessing individual understanding” to represent this commonly held view.

Assessment vs. Measurement
It is helpful to distinguish assessment from measurement. Most people use the term measurement to emphasize the use of sophisticated psychometric techniques. While these techniques are also useful for student assessment, they are typically associated with large-scale achievement testing. These techniques make it possible to figure out how hard one test item is compared to other test items. When you know this, you can estimate a person’s chance of getting a particular item correct after taking a few other items. This is how standardized tests come up with such precise scores that don’t change from test to test (this means the scores are reliable). As long as you think the items get at the thing you want to measure, you can efficiently and accurately compare how much of that thing different people have. This is most often used to measure achievement of educational standards, by creating large numbers of items that target those standards. Hence we here at re-mediating assessment often use the phrase “measurement of aggregated achievement” to describe how achievement tests are designed and used. By combining psychometrics, large pools of items of known difficulty, and computer-based testing, we get a big industry devoted to assessing aggregated achievement of lots of students. Because these tests are developed by institutions (usually companies) rather than teachers or schools, they are often called external tests.

Assessment, Measurement, and Educational Reform
Most (but certainly not all) efforts to improve education are concerned with raising scores on achievement tests. Because achievement tests target educational standards, they can be used to compare different textbooks or techniques that teach to those standards. Among those who care about achievement tests, there is a lot of debate about how directly education should target achievement tests. With the No Child Left Behind act, many educational reforms focused very directly on achievement tests. Sometimes people call this “drill and practice” but many teachers give students drills and then practice on things that are not directly included on achievement tests. The best example is what we call “test-prep.” These are computer-based programs that essentially train students in the very specific associations between words that will help them do well on a particular tests. Under No Child Left Behind, schools that were not making “adequate yearly progress” were required to take money out of the classrooms and use it to provide tutoring. Under NCLB, this often involves (and sometimes requires) schools paying private companies to deliver that tutoring, on the assumption that the teachers had failed. In practice, this meant that companies would get paid to use school computer labs to deliver computer-based test preparation. While the impact of these efforts on targeted achievement tests is a subject of fierce debate, one thing is for sure: It remains a very lucrative industry. (If you are interested see the column on Supplemental Educational Services by Edward Rotherham, aka Eduwonk).

As you might have guessed, we here at re-mediating assessment feel pretty strongly about test-prep practices. We also feel pretty strongly about what test mean, and have lots of debates about it. While Dan believe that achievement tests of literacy, numeracy, and domains like sciences and history measure stuff that is useful. Therefore as long as the tests are not directly taught to, higher scores are better. Jenna, like many, thinks that tests and the testing industry are so bad for education that they should be abandoned. But we both agree that (1) any viable current educational reforms must impact achievement tests and (2) that the transformation of knowledge into achievement tests makes that new representation of that knowledge practically useless for anything else. This is where the name of our blog comes from. When we transform knowledge of school subjects to make achievement tests, we re-mediate that knowledge. For example, consider the simple case of determining which brand of laundry detergent is the cheapest per ounce. Figuring this out while in the cleaning supplies aisle at the supermarket is very different from sitting at a table, with story problem, a pencil and a blank sheet of paper. Voila: Re-mediation. It is also worth noting that we also have some disagreements about the use of this term.

Re-mediation of knowledge for achievement tests usually means making multiple-choice items—lots of them. And this is where the trouble begins for many of us. A multiple choice item usually consists of a question (called a stem) and five answers (called responses). Most assume that test items have one “correct” answer while the rest are wrong. But it is usually more a matter of one response being “more” correct. When we take multiple choice tests we are as much ruling out wrong responses as we are recognizing the most correct one. Because the stem and five response make up five associations, we characterize achievement testing as the process of “guessing which of four or five loosely related associations is least wrong.” The more a student knows about the stuff being tested, the better they are at doing so.

What is the Problem with Test Prep?
Test prep programs raise achievement scores by training students to recognize hundreds or even thousands of specific associations that might appear on tests. Because of the way our brain works, we don’t need to “understand” an association to recognize it. All test prep programs have to do is help students recognize a few more associations as being “less wrong” or “more correct” to raise scores. Because of the way tests are designed, getting even a handful of the more difficult items correct can raise scores. A lot. And this is the root of the problem this blog is dedicated to solving. We believe that the way knowledge is remediated for tests makes that knowledge entirely worthless for teaching, and mostly worthless for classroom assessment. Specifically, we believe that training kids to recognize a bunch of isolated associations is mostly worthless for anything other than raising scores on the targeted tests. Test preparation practices and the politically motivated lowering of passing scores (“criteria”) on state achievement tests is why scores on state tests have gone up dramatically under No Child Left Behind, while scores on non-targeted tests (like the National Assessment of Educational Progress) and lots of other educational outcomes (like college readiness) have declined. Here is an article referencing some of the earlier studies. We are particularly distressed the so many schools find their computer laboratories locked up and their technology budgets locked down by computer based test preparation and interim “formative” testing. Despite a decade of e-rate funding, many students in many schools still don’t have access to networked computers to engage in networked technology practices that are actually useful.

There is a lot of debate about consequences of test preparation for achievement and its impact on other outcomes. We think that any programs that directly train students in specific associations on targeted tests is educational malpractice, because that knowledge is useless for any other purpose. This is because we think that knowledge is more about successful participation in social practices. And these practices have very little to do with tests scores. So, in summary, test preparation is the epitome of what participatory assessment is not. Our next post will try to explain what it is.

I'm bringing sexyback: some thoughts on formative assessment

2009-07-22T12:44:00.008-04:00

Immersed as I am lately in the world of participatory assessment, I go through cycles of forgetting and then remembering and then forgetting again that not everybody in educational research thinks assessment is sexy.

I was reminded of this again recently while reading Lorrie Shepard's excellent 2005 paper, "Formative Assessment: Caveat Emptor." The piece argues that the notion of "formative assessment" has been twisted in unfortunate ways as a result of the excessive hammering kids get from high-stakes standardized tests.

I helpfully plugged the entire paper into the wordle machine for you and got this:

In theory, then, assessment should be easy to understand: All of the most frequently used words in Shepard's paper are fairly common and comprehensible. In practice, though, assessment research is complicated by the impulse to put a fine point on things. Here's a sample paragraph from Shepard's piece, which starts out okay but descends into chaos before the end:

“Everyone knows that formative assessment improves learning,” said one anonymous test maker, hence the rush to provide and advertise “formative assessment” products. But are these claims genuine? Dylan Wiliam (personal communication, 2005) has suggested that prevalent interim and benchmark assessments are better thought of as “early-warning summative” assessments rather than as true formative assessments. Commercial item banks may come closer to meeting the timing requirements for effective formative assessment, but they typically lack sufficient ties to curriculum and instruction to make it possible to provide feedback that leads to improvement.

I'm not saying the language is unnecessary; I'm not saying that assessment types are putting too fine a point on things. What I will argue here is that assessment research has, for lots of good and not-so-good reasons, been divorced so thoroughly from other aspects of educational research that it's decontextualized itself right into asexuality. It's like that guy in the corner booth at the bar on Friday night who wants to talk about Marxism when everybody else just wants to make sure everybody gets the same amount of beer before closing time.

Think about that guy for a second. Let's call him Jeff. Jeff has been single for a long time now, and he's spent a lot of that time reading. Maybe he's grown nostalgic for the early days before his girlfriend cheated on him and then moved in with some guy she met in her Econ class. His friends miss those days, too, mainly because he was so much goddamn fun back then. They're nice enough; they want to take him out and help him snap out of it. But the minute the beers come he's back on the Marxism soapbox again and NOBODY. FREAKING. CARES. It's Friday night, late July, and everybody just wants to get stupid drunk. They drop him some hints. Sully slaps him on the back and asks him to tell that one joke he told last week.

"In a minute," Jeff says. "I'm explaining where Marxism went wrong."

Eventually his friends will tell him to either cut it out or go home. If he wants to keep hanging out with these guys, he'll shut up. Or maybe he'll tell that one joke he executes so well. If the girls around him laugh, he might tell another one. Girls like funny guys, he'll suddenly remember. They don't necessarily like Marxists.

All of this is what we might call "formative assessment." This guy wants to be accepted by his friends, which means he needs to pay attention to his behavior. He learns (or re-learns) how to act at the bar on Friday night by paying attention to the feedback he gets from his friends, from other people at the bar, from his memories of having a social life all those years ago.

If we wanted to, we could spend some time talking about better ways to help Jeff learn the social skills he needs. For example, his friends could have sat him down before they went out and explained that his primary goal was to be the funniest guy in the room. "Because girls like funny guys," his buddy Rufus might remind him. They might also set deadlines: By 11:30 you better have told at least three jokes. Then, over the course of the evening, they could check in with him and get a joke-count.

The point is that everybody's on board with the evening's goals. Everybody--Jeff, his friends--wants Jeff to have a good time, and they want to have a good time with him.

Haha! I tricked you into caring about formative assessment.

This is what assessment is, even if it doesn't always feel that way to students, teachers, or researchers. There is an end goal, an objective, and formative assessment is a way of getting everyone on board with this goal and keeping them on board. When it works right, everybody involved actually wants to achieve the objective and the assessment is valuable because it helps them get where they want to go.

But as Shepard's piece points out, too often the insanity of NCLB substitutes test scores for real, intrinsic motivation. Too often and too easily, students learn skills it takes to attain high test scores without actually learning anything. Though "(the) idea of being able to do well on a test without really understanding the concepts is difficult to grasp," Shepard writes, she gives as evidence a 1984 study performed by M.L. Koczor, which focused on two groups of children learning about Roman numerals:

One group learned and practiced translating Roman to Arabic numerals. The other group learned and practiced Arabic to Roman translations. At the end of the study each group was randomly subdivided again (now there were four groups). Half of the subjects in each original group got assessments in the same format as they had practiced. The other half got the reverse. Within each instructional group, the drop off in performance, when participants got the assessment that was not what they had practiced, was dramatic. Moreover, the amount of drop-off depended on whether participants were low, middle, or high achieving. For low-achieving students, the loss was more than a standard deviation. Students who were drilled on one way of translation appeared to know the material, but only so long as they were not asked to translate in the other direction.

Because NCLB and other insane policies that mandate high-stakes testing for accountability have pushed assessment out of its natural home--as Jim Gee explains it, "in human action"--assessment researchers have themselves been backed into a separate corner of the room.

This is not okay. It doesn't help anybody to take the sexy out of assessment by tossing it into a corner. What we need, more than anything, is to push assessment back where it belongs: inside of the participation structures that support authentic learning.

Participatory assessment is, at its core, about social justice, about narrowing the participation gap that keeps our society stratified by race and class, about motivating learners to achieve real goals and overcome real obstacles to their own learning. Participatory assessment, if we do it right, can make almost anything possible for almost anyone.

making universities relevant: the naked teaching approach

2009-07-20T12:10:00.007-04:00

I feel sorry for college deans, I really do*. They face the herculean task of proving that the brick-and-mortar college experience offers something worth going into tens of thousands of dollars of debt for, a task made even more difficult by the realities of a recession that's left nearly a quarter of Americans either unemployed or underemployed.

Then there's the added challenge of proving colleges have anything other than paper credentials to offer in a culture where information is free and expert status is easily attainable. Only in a participatory culture, for example, would it be possible for time-efficiency guru Timothy Ferriss to offer a set of instructions on "How to Become a Top Expert in 4 Weeks." "It's time to obliterate the cult of the expert," Ferriss writes in his mega-bestseller, The Four-Hour Workweek. He argues that the key is to accumulate what he calls "credibility indicators." It is possible, he writes,

to know all there is to know about a subject--medicine, for example--but if you don't have M.D. at the end of your name, few will listen.... Becoming a recognized expert isn't difficult, so I want to remove that barrier now. I am not recommending pretending to be something you're not... In modern PR terms, proof of expertise in most fields is shown with group affiliations, client lists, writing credentials, and media mentions, not IQ points or Ph.D.s.

Ferriss then offers five tips for becoming a "recognized expert" in your chosen field. None of them include earning the credential through formal education.

Just like that, we've gone from the position that expertise takes a decade, at minimum, to develop, to the argument that a person can become an expert in just four weeks.

In the face of this qualitative shift in how we orient to expertise, colleges--the educational institutions that have made their bones on offering a sure path to credentialing--are struggling to remain viable. One strategy--and the one chosen by José A. Bowen, dean of the Meadows School of the Arts--is to offer "naked teaching." Bowen's approach, as described in a recent piece in the Chronicle of Higher Education, is to actually remove networked technologies from the classroom. The article makes it clear that Bowen is not anti-technology; he just thinks technologies are being misused by faculty who overrely on PowerPoint and technology-supported lecturing techniques. He favors using technologies like podcasting for delivering lecture materials outside of the classroom, then using the class itself to foster group discussion and debates.

To support this approach, all faculty were recently given laptops and support for creating podcasts and videos.

According to the Chronicle piece, the group that's most upset about the shift away from the traditional lecture format is...students. According to Kevin Heffernan, an associate professor in the school's division of cinema and television, students

are used to being spoon-fed material that is going to be quote unquote on the test. Students have been socialized to view the educational process as essentially passive. The only way we're going to stop that is by radically refiguring the classroom in precisely the way José wants to do it.

For all the griping we do about No Child Left Behind, test-centered accountability practices, and high-stakes assessment practices, the roaring success of decontextualized accountability structures is their astounding ability to keep formal education relevant. "Success" at the primary and secondary level means high achievement on high-stakes tests; and, achievement depends on the learner's ability to internalize the value systems and learning approaches implicit in the approach of this kind of testing structure. Do well on a series of state-mandated tests and you'll probably also do well on the SAT; do well on the SAT and you're well positioned for the lecture-style, knowledge-transfer and, in general, highly decontextualized experience of most undergraduate-level classes. We gravitate toward the kind of experience that make us feel successful, which means the testing factory churns out its own customer base.

While Bowen's experiment (one that he's been moving toward for years; see this 2006 piece in the National Teaching and Learning Forum) may garner attention for an apparent anti-technology stance, the impetus behind his "naked teaching" approach is an effort to reshape the role of institutions of higher education. In truth, learning can happen anywhere, and Bowen's embrace of this truth through his embrace of technologies for supporting out-of-class information transfer seems like a low-risk and high-yield slant on the role of the university.

If learning can happen anywhere, then the physical community of learners gathered together within four walls, engaged in the act of collaborative knowledge-building: That's the rare commodity. In a world where everyone can be an expert, the promise of credentials become just another strategy for bringing that community together.

*jk I really don't.

getting students off of Maggie's farm

2009-07-17T09:39:00.004-04:00

I stumbled across an interesting cross-blog conversation about Social Media Classroom and similar Learning Management Systems (LMS's). I have been, and continue to be, a strong and vocal supporter of Social Media Classroom (SMC), Howard Rheingold's Drupal-based, open-source educational technology intended to support participatory practices in formal learning settings.

Most significantly for me, it was participation in SMC that led to my passion for all things open-source. This is not a trivial thing: If participation in an LMS fosters a disposition toward increased openness, collaboration, and sharing, then it's clearly putting its money where its mouth is.

Blogger and computer scientist Andre Malan writes that he recently took SMC for a spin around the block and found it impressive in some ways and lacking in others. He writes:

It seems to be closed off and private by default (although this may have just been the system I used). If outsiders can participate (as has been shown by Jon Beasley-Murray, Jim Groom and D’Arcy Norman) magic can happen. We need to let the world see what students are doing in university.

The “Social Media Classroom” is missing one little word in the title. A game changer would rather be a “Social Network Media Classroom”. Although students can edit their own profiles in the Social Media Classroom, there is no way to form groups or to add people to their network. The network is often the most powerful part of any social media applications and it is a terrible oversight to not include it.

The training wheels don’t come off. This application is great for students who do not know of, or use social media tools. However, it sucks for those that do. They are not able to use their current networks or applications. Most people who have blogs would want to use their own blogs for a class. Or use their own social bookmarking service. These people (the ones who would be very useful in this environment as they could guide their peers and instructors in the use of social media) will feel alienated and resent having to use the Social Media Classroom. If an education-based social media application is ever to be successful it has to provide an easy way for experienced students to show others the tricks of the trade and for novice students to take the wheels off of the bicycle and use real tools when they are ready for it.

D'Arcy Norman, writing from the University of Calgary, responded to the above points first in the comments section and then in a full post on his own blog. Norman doesn't have a problem with fostering student engagement within "walled gardens"--he writes:

The goal isn’t to publish content to the open internet. The goal is to engage students, in creation, discussion, and reflection. If they need a walled garden to do that effectively (and there are several excellent reasons for needing privacy for a community) then so be it. If they’d like to do it in the open, that’s just a checkbox on a settings page.

And, in the most spectacular finish to a post I've so far read anywhere, by anyone, Norman ends with this:

That option isn’t available for users of The Big Commercial LMS Platform. If it’s in an LMS, it’s closed. End of discussion. And people only gain experience in using the LMS, in farming for Maggie.

Norman is right and he's wrong. A closed LMS that lacks the capacity for open participation in a larger community turns learners into day laborers reduced to carting bushels of cognitive work from the fields to the barn and taking home only what they can hide away in their pockets. But in many ways, a "walled garden" isn't much better. Not to overstretch the metaphors here, but legend has it that Prince Siddhartha spent his youth inside of a walled garden. The kind of participation his surroundings supported was absolutely voluntary, and probably felt authentic, in the main. But when he left the garden, everything he knew to be true was true no longer.

One of the big failings of educational institutions is that they too often offer a beautiful walled garden. Inside the garden, food is abundant, and everybody eats equally well. (Well, that depends on the garden you've walked into, how you got there, how long you can stay, and whether you have comparable walled garden experience in your past.)

Sure, participation in a closed system engages students "in creation, discussion, and reflection." This is, I agree, a necessary component of higher education. But I disagree with Norman that this type of participation is sufficient. In fact, creation, discussion and reflection are only useful learning experiences insofar as they support learners' ability and willingness to engage with wider, more public, and less protected communities of practice. This means that publishing content on the open internet should--indeed, must--be a key curricular element. The internet isn't a garden; it's an ecosystem complete with backlots, busted glass, some ragged sunflowers and lots of rich material ripe for harvesting--but only if you've learned what it takes to grow and then harvest that material.

on the community-source model for open educational software design

2009-07-13T10:07:00.006-04:00

For all my fascination with all things open-source, I'm finding that the notion of open source software (OSS) is one that's used far too broadly, to cover more categories than it can rightfully manage. Specifically, the use of this term to describe collaborative open education resource (OER) projects seems problematic. The notion of OSS points to a series of characteristics and truths that do not apply, for better or worse, to the features of collaborative learning environments developed for opening up education.

While in general, open educational resources are developed to adhere to the letter of the OSS movement, what they miss is what we might call the spirit of OSS, which for my money encompasses the following:

A reliance on people's willingness to donate labor--for love, and not for money.
An embrace of the "failure for free" model identified by Clay Shirky in Here Comes Everybody.
A loose collaboration across fields, disciplines, and interest levels.

Open educational resources are not, in general, developed by volunteers; they are more often the product of extensive funding mechanisms that include paying participants for their labor.

There are good reasons for this. As Christopher J. Mackie points out in Opening Up Education, while the OSS movement has produced some "runaway successes" (Perl, Linux, and Firefox), the moveent has less success at tackling certain types of projects, including development of products designed for widespread institutional use (instead of adoption by individuals). There are good reasons for this, he argues; and his explanation points to both the weaknesses and the strengths of the open education movement:

This limitation may trace to any of several facotrs: the number of programmers having the special expertise required to deliver an enterprise information system may be too small to sustain a community; the software may be inherently too unglamorous or uninteresting to attract volunteers; the benefits of the software may be too diffuse to encourage beneficiaries to collaborate to produce it; the software may be too complex for its development to be coordinated on a purely volunteer basis; the software may require the active, committed participation of specific firms or institutions having strong disincentives to participate in OSS; and so on.

Perhaps the two most significant weak spots Mackie points to are the unglamorous nature of developing OERs and the strong disincentives against institutional participation in developing and circulating these resources. OERs require sustained, consistent dedication at all levels, from programmers all the way up to administrators and funders; and this type of dedication is difficult to attain for the following reasons:

While OSS is primarily affiliated with the movement itself, OERs are by their nature affiliated first with an institution or funder; as project affiliates change institutions or roles, their commitment to developing the OER can shift or disappear.
OERs require institutional buy-in, and the notion of openness, on its surface at least, appears at odds with institutional goals. (Universities survive by offering something unique, something you can only get by paying your money and walking through the gates.)

Mackie suggests an alternate term for OERs designed in keeping with the open source ideals: community source software (CSS). He identifies the following characteristics as key to the CSS movement:

Multiple institutions band together to design software that meets their collective needs, with the ultimate goal of releasing the software as open source eventually;
Development of the software is conducted virtually, with employees from each institution collaborating;
The collaboration aligns with a corporate, even sometimes hierarchical, structure, with project leaders, paid staff, and experts in a range of design and development categories;
Everybody is compensated for their expertise, and this supports a systematic, targeted approach to software development that is often lacking in OSS projects.

Embracing the notion of community source software instead of open source is more than a semantic choice, in my view. It opens up new avenues for participation and the possibility for new affiliation structures across institutions of higher education. Just as higher education institutions have historically affiliated around various community markers (cf. The Associated Writers and Writing Programs, HASTAC member institutions, the Doctoral Consortium in Rhetoric and Composition), colleges and universities--and their affiliates--might unite around the notion of opening up education by opening up technologies, access, and information.

After all, let's take our heads out of the clouds for a second and think about what sorts of factors might motivate a university to align with the open educational movement. Asking institutions to relinquish their monopoly on whatever they think makes them unique (cf. the college ranking system at U.S. News and World Report) requires that we offer them something in exchange. "For the good of humankind" is a sweet notion, but you can't take it to the bank.

Participatory Assessment for Bridging the Void between Content and Participation.

2009-07-09T15:06:00.004-04:00

Here at Re-Mediating Assessment, we share our ideas about educational practices, mostly as they relate to innovative assessment practices and mostly then as they relate to new media and technology. In this post, I respond to an email from a colleague about developing on-line versions of required courses in graduate-level teacher education courses.

My colleague and I are discussing how we ensure coverage of “content” in proposed courses that focuses more directly on “participation” in the actual educational practices. This void between participation (in meaningful practices) and content (as represented in textbooks, standards, and exams) is a central motivation behind Re-Mediating Assessment. So it seems worthwhile to expand my explanation of how participatory assessment can bridge this void and post it here.

To give a bit of context, note that the course requirements of teacher education programs are constantly debated and adjusted. From my perspective it is reasonable to assume that someone with a Master’s degree in Ed should have taken a course on educational assessment. But it also seems reasonable to have also had a course on, say, Child Development. But it simply may not be possible to require students to take both classes. Because both undergraduate and graduate teacher educator majors have numerous required content area courses (i.e., math, English, etc.), there are few slots left for other courses that most agree they need. So the departments that offer these other required courses have an obvious obligation to maintain accountability over the courses that they offer.

I have resisted teaching online because previous courseware tools were not designed to foster participation in the meaningful discourse that is what I think is so important to a good course. Without a classroom context for discourse (even conversations around a traditional lecture), students have few cues for what matters. Without those cues, assessment practices become paramount in communicating the instructor values. And this is a lot to ask of an assessment.

This is why, in my observation, online instruction heretofore has mostly consisted of two equally problematic alternatives. The first is the familiar on-line tools for pushing content out to students: “Here is the text, here are some resources, and here is a forum where you can post questions, and here is the exam schedule.” The instructors log on to the forums regularly and answer any questions, students take exams, and that is it. Sometimes these courses are augmented with papers and projects and perhaps with collaborative projects; hopefully students get feedback, and they might even use that feedback to learn more. But many many on-line course are essentially fancy test prep. My perceptions are certainly biased by my experiences back in the 90s in the early days of on-line instruction. The Econ faculty where I was working could not figure out why the students who took the online version of Econ 101 always got higher exam scores than the face-to-face (FTF) students, but almost always did far worse in the FTF Econ 201. This illustrates the problem with instruction that directly preparing students to pass formal exams. Formal exams are just proxies for prior learning, and framing course content entirely around tests (especially multiple choice ones) is just a terrible idea. Guessing which of four associations is least wrong is still an efficient way of reliably comparing what people know about a curriculum or a topic. But re-mediating course content to fit into this format makes it nearly useful for teaching.

The other extreme of on-line instruction is “project based” classes that focus almost entirely on developing a portfolio of course-related projects. These approaches seem particularly popular in teacher education programs. The problem with on-line portfolios is that the lack of FTF contact requires the specifications for the portfolios to be excruciatingly detailed. Much of the learning that occurs tends to be figuring out what the instructor wants in order to get a good grade. The most salient discourse in these classes often surrounds the question “Is this what you want?” These classes are usually extremely time-consuming to teach because the accountability associated with the artifacts leads students to demand, and instructors to provide, tons of detailed feedback on each iteration of the artifacts. So much so that the most qualified faculty can’t really afford to teach many of these courses. As such, these courses are often taught by graduate students and part-time faculty who may not be ideal for communicating the “Relevant Big Ideas” (RBIs, or what a learning scientist might call “formalisms") behind the assignments, and instead just focus on helping students create the highest quality artifacts. This creates a very real risk that students in these classes may or may not actually learn the underlying concepts, or may learn them in a way that they are so bound to the project that they can’t be used in other contexts. In my observation, such classes seldom feature formal examinations. Without careful attention, lots of really good feedback, and student use of feedback, students may come away from the class with a lovely portfolio and little else. Given the massive investment in e-Portfolios in e-learning platforms like Sakai, this issue demand careful attention. (I will ask my friend Larry Mikulecky in Indiana’s Department of Culture, Communication, and Language Education who I understand has been teaching non-exam online courses for years and has reportedly develops considerable evidence of student’s enduring understanding.)

A Practical Alternative
I am teaching on-line for the first time this summer. The course is P540, Cognition and Learning, a required course for many M. Ed programs. I am working like crazy to take full advantage of the new on-line resources for social networking that are now available in OnCourse, IU’s version of Sakai (an open-source collaborative learning environment designed for higher education). In doing so I am working hard to put into place an on-line alternative that balances participation and content. I also plan to use some of the lessons I am learning in my Educational Assessment course this Fall—which is partly what prompted that aforementioned conversation with my colleague. I want to put some of my ideas as they are unfolding in that class out there and seek input and feedback, including from my current students who are (so far) patiently hanging with me as I refine these practices as I go.

In particular I am working hard to incorporate the ideas about participatory culture that I have gained from working with Henry Jenkins and his team at Project New Media Literacies over the last year. Participatory assessment assumes that you can teach more "content" and gather more evidence that students “understand” that content by focusing more directly on participation and less directly on content. Theoretically, these ideas are framed by situative theories of cognition that say participation in social discourse is the most important thing to think about, and that individual cognition and individual behavior are “secondary” phenomena. These ideas come to me from three Jims: Greeno (whose theorizing has long shaped my work) Gee (who also deeply influences my thinking about cognition and assessment and whose MacArthur grant funded the aforementioned collaboration and indirectly supports this blog) and Pellegrino (with whom I did my doctoral studies of assessment, transfer, and validity with but who maintains an individual differences approach to cognition).

Per the curriculum committee that mandated a cognition and learning course for most masters degrees for teachers, my students are just completing ten tough chapters on memory, cognition, motivation, etc. I use Roger Bruning’s text because he make is quite clear and puts 5-7 “implications for teaching” at the end of each chapter. But it is a LOT of content for these students to learn, especially if I just have them read the chapters.

I break students up into domain groups (math science, etc.) and in those groups they go through the 5-7 implications for teaching. Each group must use the forum to generate a specific example of that implication, and then rank order the implications in terms of relevance and warrant those rankings and post them to the OnCourse wiki. The level of discourse in the student-generated forums around the content is tremendous. Then the lead group each week synthesizes the postings of all five groups to come up with a single list. I also have now asked them to do the same with “things worth being familiar with” in the chapter (essentially the bolded items and any highlighted research studies). What I particularly like about the discussions is the way that the discourse around agreeing that an implication or topic is less relevant actually leads to a pretty deep understanding of that implication or idea. This builds on ideas I have learned from my colleague Melissa Gresalfi about “consequential engagement.” By struggling to conclude that the implication is least likely to impact practice makes it more likely that they will remember that implication if they find themselves is a situation that makes it more relevant.

This participatory approach to content is complemented by four other aspects of my class. Illustrating my commitment to content, I include three formal exams that are timed and use traditional MC and short answer items. But I prioritize the content that the class has deemed most important, and don't even include the content they deem least important.

The second complement is the e-Portfolios each student has to post each week in OnCourse. Students have to select the one implication they think is most relevant, warrant the selection, exemplify and critique it, and then seek feedback on that post from their classmates. Again following Melissa’s lead, the e-Portfolio asks students for increasingly sophisticated engagement with the implication relative to their own teaching practice: procedural engagement (Basically explain the implication in your own words), conceptual engagement (give an example that illustrates what this implication means), consequential engagement (what are the consequence of this implication for your teaching practice, what should you do differently now that you understand this aspect of cognition?) and critical engagement (why might someone disagree with you and what would happen if you took this implication too far?). I require them to request feedback from their classmates. While this aspect of the new on-Course e-Portfolio tools is still quite buggy, I am persevering because the mere act of knowing that a peer audience is going to read it pushes them to engage more deeply. Going back to my earlier point, it is hard for me to find time to review and provide detailed feedback on 220 indivdiual submissions across the semester. When I do review them (students submit them for formal review after five submissions), I can just look at the feedback from other students and the students' own reflection on what they have learned for pretty clear evidence of consequential and critical engagement.

The third complement is the e-Portfolio that each student completes during the last five weeks of class. While each of the groups leads the class in the chapter associated with their domain (literacy, comprehension, writing, science and math), students will be building an e-portfolio in which they critique and refine at least two web-based instructional resources (educational videogames, webquests, the kind of stuff teachers increasingly are searching out and using in their classes). They select two or more of the implications from that chapter to critique the activities and provide suggestions for how it should be used (or if it should be avoided), along with one of the implications from the chapter on instructional technology, and one of the implications from the other chapters on memory and learning. If I have done my job right, I don’t need to prompt them to the consequential and critical engagement at this stage. This is because they should have developed what Melissa calls a “disposition” towards these important forms of engagement. All I have to do is include the requirement that they justify why each implication was selected, the feedback from their classmates, and their reflection on what they learned from feedback. It turns out the consequential and critical engagement is remarkably easy to recognize in discourse. That seems partly because it is so much more interesting and worthwhile to read than the more typical class discourse that is limited to procedural and conceptual engagement. Ultimately, that is the point.

Five tips for seeding and feeding your educational community

2009-07-07T10:34:00.007-04:00

Dan Hickey's recent post on seeding, feeding, and weeding educators' networks got me thinking, for lots of reasons--not least of which being that I will most likely be one of the research assistants he explains will “work with lead educators to identify interesting and engaging online activities for their students.”

This got me a-planning. I started thinking about how I would seed, feed, and weed a social network if (when) given the chance to do so. As David Armano, the author of "Debunking Social Media Myths, the article that suggests the seeding, feeding, and weeding metaphor, points out, building a social media network is more difficult than people think—this is not a “if we build it, they will come” sort of thing. Designing, promoting, and growing a community takes a lot of work. People will, given the right motives, participate in the community for love and for free, but you have to start out on the right foot. This means offering them the right motivations for giving up time they would otherwise be spending on something else.

A caveat
First, know that I am a True Believer. I have deep faith in the transformative potential of participatory media, not because I see it as a panacea to all of our problems but because participatory media supports disruption of the status quo. A public that primarily consumes media primarily gets the world the media producers decide they want to offer. A public that produces and circulates media expressions gets to help decide what world it wants.

Social media, because of its disruptive and transformative potential, is both essential and nigh on impossible to get into the classroom. This is precisely why it needs to happen, and the sooner it happens, the better.

But integrating participatory media and the participatory practices they support into the field of education is not a simple matter. Too often people push for introduction of new technologies or practices (blogging, wikis, chatrooms and forums) without considering the dispositions required to use them in participatory ways. A blog can easily be used as an online paper submission tool; leveraging its neatest affordances--access to a broad, engaged public, joining a web of interconnected arguments and ideas, offering entrance into a community of bloggers--takes more effort and different, often more time-consuming, approaches.

Additionally, while social networks for educators hold a great deal of promise for supporting the spread of educational practices, designing, building, and supporting a vibrant community of educators requires thinking beyond the chosen technology itself.

Five Tips for Seeding and Feeding your Community
With these points in mind, I offer my first shot at strategies for seeding and beginning to feed a participatory educational community. (Weeding, the best part of the endeavor, comes later, once my tactics have proven to work.)

1. Think beyond the classroom setting.
In the recently published National Writing Project book, Teaching the New Writing, the editors point out that for teachers to integrate new media technologies into their classrooms, they "need to be given time to investigate and use technology themselves, personally and professionally, so that they can themselves assess the ways that these tools can enhance a given curricular unit."

The emerging new media landscape offers more than just teaching tools--it offers a new way of thinking about communication, expression, and circulation of ideas. We would do well to remember this as we devise strategies for getting teachers involved in educational communities online. After all, asking a teacher who's never engaged with social media to use it in the classroom is like asking a teacher who's never used the quadratic equation to teach Algebra.

Anyone who knows me knows what a fan of blogging I am. I proselytize, prod, and shame people into blogging--though, again, not because I think blogging is the best new practice or even necessarily the most enjoyable one. Blogging is just one type of practice among a constellation of tools and practices being adopted by cutting edge educators, scholars, and Big Thinkers across all disciplines. Blogging was, for me, a way in to these practices and tools, and I do think blogging is one of the most accessible new practice for teacherly / writerly types. The immediacy and publicness of a blogpost is a nice preparation for increased engagement with what Clay Shirky calls the “publish, then filter” model of participatory media. This is a chaotic, disconcerting, and confusing model in comparison to the traditional “filter, then publish” model, but getting in synch with this key element of participatory culture is absolutely essential for engaging with features like hyperlinking, directing traffic, and identifying and writing for a public. In a larger sense, connecting with the publish, then filter approach prepares participants to join the larger social networking community.

2. Cover all your bases--and stop thinking locally
One of the neatest things about an increasingly networked global community is that we're no longer limited to the experts or expertises of the people who are within our physical reach. Increasingly, we can tap into the knowledge and interests of like-minded folks as we work to seed a new community.

Backing up a step: It helps, in the beginning for sure but even more so as a tiny community grows into a small, then medium-sized, group, to consider all of the knowledge, experience, and expertises you would like to see represented in your educational community. This may include expertise with a variety of social media platforms, experience in subject areas or in fields outside of teaching, and various amounts of experience within the field of education.

3. In covering your bases, make sure there's something for everyone to do.
Especially in the beginning, people participate when they feel like they a.) have something they think is worth saying, b.) feel that their contributions matter to others, and c.) can easily see how and where to contribute. I have been a member of forums where everybody has basically the same background and areas of expertise; these forums usually start out vibrant, then descend into one or two heavily populated discussion groups (usually complaining or commiserating about one issue that gets up in everyone's craw) before petering out.

Now imagine you have two teachers who have decided to introduce a Wikipedia-editing exercise into their classrooms by focusing on the Wikipedia entry for Moby-Dick. Imagine you have a couple of Wikipedians in your network who have extensive experience working with the formatting code required for editing; and you have a scholar who has published a book on Moby-Dick. This community has the potential for a rich dialogue that supports increasing the expertise of everybody involved. Everybody feels valued, everybody feels enriched, and everybody feels interested in contributing and learning.

4. Use the tool yourself, and interact with absolutely everybody.
Caterina Fake, the founder of Flickr, says that she decided to greet the first ten thousand Flickr users personally. Assuming ten thousand users is several thousand more than you want in your community, you might have the time to imitate Fake's example. It also helps to join in on forums and other discussions, especially if one emerges from the users themselves. Students are not the only people who respond well to feeling like someone's listening.

Use the tool. Use the tool. Use the tool. I can't emphasize enough how important this is. You should use it for at least one purpose other than seeding and feeding your community. You should be familiar enough with it to be able to answer most questions and do some troubleshooting when necessary. You should be able to integrate new features when they become available and relevant, and you should offer a means for other users to do the same.

5. Pick a tool that supports the needs of your intended community, and then use the technology's features as they were designed to be used.
Though I put this point last, it's the most important of all. You can't--you cannot--build the right community with the wrong tools. Too often, community designers hone in on a tool they have some familiarity with or, even worse, a tool that they've heard a lot about. This is the wrong tack.

What you need to do is figure out what you want your community to do first, then seek out a tool that supports those practices. If you want your community to refine an already-established set of definitions, approaches, or pedagogical tenets, then what you're looking for is a wiki. If you want the community to discuss key issues that come up in the classroom, you want a forum or chat function. If you want them to share and comment on lesson plans, you need a blog or similar text editing function.

Once you've decided on the functions you want, you need to stick with using them as god intended. Do not use a wiki to post information that doesn't need community input. Don't use a forum as a calendar. And don't use a blog for forum discussions.

It's not easy to start and build a community, offline or online. It takes time and energy and a high resistance to disappointment and exhaustion. But as anybody who's ever tried and failed (or succeeded) to start up a community knows, we wouldn't bother if we didn't think it was worth the effort.

Weeding, Seeding, and Feeding Social Educational Designs

2009-07-07T06:59:00.003-04:00

This post examines the implications of a post at the Harvard Business blog by David Armano titled Debunking Social Media Myths about social business design. He points to three labor-intensive activities that are necessary for a profitable social network: weeding, seeding, and feeding. We examine these three considerations for social education design, and how they are necessary for a worthwhile social network for educators.

First some background and context. One of our primary interests here at Re-Mediating Assessment is how innovative classroom assessment practices can be shared over digital social networks. By assessment practices, we mean both particular assessments for particular activities, as well as expertise associated with those practices. Of course, we know that most efforts to create collaborative networks for educators don’t take hold (Check out the 2004 book Designing Virtual Communities in the Service of Learning by Barab, Kling, and Gray, and a special issue of The Information Society they edited for a good discussion of some pioneering efforts).

We have previously written about the value of insights out of media scholarship for thinking about the sharing of educational practices. In particular Henry Jenkins’ notions of "spreadable" practices have prompted us to launch serial posts at Project New Media Literacies introducing the idea of Spreadable Educational Practices (SEP) and to juxtapose them with the doomed distribution of centrally defined and "scientifically" validated scripts, what we label Disseminated Instructional Routines.

We are currently outlining several new proposals to expand the nascent networks that are forming around various efforts. We also want to design and test strategies for helping other nascent networks succeed by helping to foster spread of effective practices and the necessary social bonds.

Media scholars inevitably consider the for-profit nature of commercial media. Of course, not all media scholars care about markets and eyeballs, and the nature of media markets are undergoing tremendous change. In our prior posts at Project NML, Henry's descriptions of failed corporate efforts to create "viral" messages and "sticky" websites seemed to describe some of the failed efforts to create educator social networks. The point here is that educational social design can be informed by business social design.

Armano’s talk at the Conversational Marketing Summit pointed out something that is easy to underestimate: "Being social means having real people who actively participate in your initiatives." Because educators tend to be so overwhelmed by the daily press of teaching, worthwhile social education design must find ways to get teachers actively involved. And this takes resources. Building a network will require significant support at the outset, likely sponsoring leading participants to welcome newcomers and foster effective practices. As Clay Shirky wrote in Here Comes Everybody, the founder of Flickr said that she learned early on that "you have to greet the first ten thousand users personally.”

Here are three things that Armano says that successful networks must plan for, and what they might look like in an educational network:

Seeding: Someone has to seed a network with resources and practices that your particular users need. In the Participatory Activities and Assessment Network we are trying to build out, we will budget quite a bit of time for research assistants to work with lead educators to identify interesting and engaging on-line activities for their students. However, the participatory assessments that educators can use to implement those activities in their classrooms and refine them over time will likely have to be constructed by the research effort. So we are budgeting for that too. We will work with heavily subsidized and fully-supported lead teachers for the first year to seed the network with useful activities before bringing on less-subsidized and partly-supported teachers. Only then do we think that the network will be sufficiently seeded to expect unsupported users to start participating in large numbers.

Feeding. The network needs a steady stream of content. By content, what we mean is information--information that other participants will find useful. In our case, the most useful information will be the anecdotes and guidelines for implementing participatory activities with actual students, and sharing the "low stakes" evidence obtained from participatory assessments for improving success. For example, we view the posting of accounts, videos, and artifacts from the enactment of successful implementation as crucial content that needs to be fed to the network. It is our job to make sure that there is both a source and an audience. Our lead teachers will need help posting accounts of enactments to the network. For example, most teachers know that they themselves can't post video of their students to YouTube (as researchers we are forbidden from even thinking about doing so because of Human Subjects constraints). But there is nothing to stop us from giving the lead teachers several inexpensive flip cams and letting students post accounts of themselves to YouTube (if the school allows access; they may be better off with SchoolTube which is less likely to be blocked).

Once accounts of practice are up, it is also our job to ensure that there is an audience. In this case, we will have paired teachers up to select activities to complete with their classrooms, and then asking one of them to implement first. This first lead teachers’ posted accounts and informal guidelines will be immediately useful for to second lead teacher. One or two simple successes like this will create a powerful social bond between two otherwise isolated participants. Because this interaction will take place via public and persistent discourse in the network, the accounts will be immediately useful for other participants wishing to use the activities; this discourse will be crucial for helping that newcomer locate and access the informal expertise that is now spread across the network in those two lead teachers.

Weeding. Armano points out that productive social business design must prune content that inhibits growth. This might be the most challenging aspect of a productive social education design. This partly refers to getting rid of problematic content. One of the lessons we learned in our collaboration with Project New Media Literacies working with Becky Rupert at Aurora High School is that need for involving students in helping keep offensive or objectionable material off of school related networks. If the students find the networking activities an enjoyable alternative to traditional activities the quickly become a powerful ally in minimizing transgressions. This is crucial, as teachers simply won’t have time to do it, and will be overwhelmed with the nuanced decisions between creative expressions and those that are patently offensive.

Another important part of weeding is getting rid of stuff that does not work. As educators we have a tendency to hang on to everything and make it available to all. Thus we create a huge obstacle for other educators who have to weed through endless list of resources looking for the right one, and then implement it and hope it succeeds. Our network assumes that most web-based educational resources are not very good at fostering worthwhile classroom participation. This is not because the resources are inherently bad, but because participatory classroom culture is so challenging to attain. Our Participatory Activities and Assessment Network will start with a carefully catalogued and tagged set of activities that have been initially vetted and aligned to one or two Relevant Big Ideas (or RBIs, which in turn can be easily aligned to content standards). As lead teachers select activities for further consideration, they will be tested by research assistants before participatory assessments are created and released along with the activity. If the activities and assessments don’t foster worthwhile participation for the first two lead teachers, it will be tagged as such.

Importantly, the network will contain useful information about the nature of that “productive failure” that will be useful for others. Consider that it may well be that the activity turned out to be too easy or too hard or required too much background knowledge for the particular students. Rather than labeling the activity as “useless” it should be tagged in a way that another teacher who works with students for whom it might be “perfect” can find the activity, along with the information and distributed expertise for using it. It looks to us like building information systems for accomplishing this will be one of our major challenges.

On collaborative platforms for sharing educational practices

2009-07-05T22:23:00.002-04:00

I've been in conversation with lots of educators recently about strategies for developing and supporting collaborative communities of teachers within various social networks online. Most recently I am talking with IU Mathematics Education Professor Cathy Brown about the lovely site that she has created in Moodle to support the math teachers who are teaching at the New Tech High Schools in Indiana. We are going to meet to see if some of the ideas we have been developing about participatory activities and assessment might help NewTech teachers use the site to do what they are doing--Helping integrate mathematics into interesting and engaging projects. Because Indiana is now rolling out End of Course assessments in Algebra (along with English and Biology) I assume that these teachers are under significant pressure to show not only that thier students are passing (required to get credit for the course) but exceling. This creates an important tension that gets at the heart of what we care about here at Re-Mediating Assessment.

Though I'd like to say otherwise, there is unfortunately no perfect tool--no single network that magically fosters community, cooperation, and collaboration. Part of this is due to the fact that all platforms are designed to support only certain kinds of engagement and therefore have benefits and drawbacks inherent to them; the other factor is that too often, people try to bend a community to the affordances of the technology instead of finding a tool or set of tools that align most closely to the needs of the community.As for platforms, I have bounced around a lot from several which have distinct advantages and disadvantages. I want to take a minute and share my experiences and them make the point I want to make.

I used SocialMediaClassroom for my graduate classes in Spring 2009 and that was very informative and help.. One of the great things about using it was that it hooked us up with it sponsor, social networking pioneer Howard Rheingold and his deep and interesting community who kibbutz at his installation of SMC at http://socialmediaclassroom.com/. It also hooks you up with the open-source Drupal community, which also has a lot of potential. It was a bit buggy, which was not surprising at it was an early stage open source program. Sam Rose did a tremendous job setting it up and was really helpful both in getting it installed and then working out the many bugs that resulted from my ignorance. MacArtur’s Digitial Media and Learning initiative funded the initial development, and are using it in the DML hub which is also important

This summer I have been using Indiana University's OnCourse CL, an online collaborative learning environment designed through the open-source Sakai Project. OnCourse brings the whole Sakai community and is very stable. Now that it has e-portfolios and wikis it has a lot of potential for the kinds of participatory activites and assessments that are so important to me. Stacy Morrone has pushed hard on the e-Portfolio features and they really have tremendous untapped potential. A big personal advantage for me in using OnCourse is the tremendous support that I get from the IU staff who are quite committed to it. The Learning Sciences graduate program just got a grant to expand our online course offerings, and we aim to use this to build a strong community of scholars around these courses, and will be using OnCourse.
The big drawback with OnCourse is that it is so closed--it only supports participation from IU affiliates and therefore restricts participation across multiple institutions. Case in point, I was planning on having my students in my Cognition and Learning course seek feedback from at least one outside expert or peer on the e-Portfolios that each of the students are drafting. The author of our textbook Roger Bruning has even agreed to review some. But for non-IU folks to do so they have to register for guest accounts. I have to do the same all the time so I can view my class as a student (another hassle of OnCourse) and I know it is a huge hassle. I have to get a new password every time. So I really can't include that in the course requirements as it will cause a revolt and a lot of headaches. Of course, the beauty of the Sakai platform is that I should be able to build and mount my own version for this. I will keep you posted!

For the last year, we have been working with an ELA curriculum designed by Project New Media Literacies, a project headed by media scholar Henry Jenkins and funded by the MacArthur Foundation's Digital Media and Learning Initiative. Our collaboration with Project NML revolved around a site in Ning which, like Moodle, is very popular with teachers. (Ning has dominated the "best educational use of a social networking service" category of the Edublogs Awards for the last two years: In 2008, 9 out of the 10 finalists were Ning-based, and in 2007 all ten finalists were based in Ning.) Our thoughts are influenced as usual by Clay Shirky. In Here Comes Everybody he pointed out that "there are no generically good tools, only tools that are good for certain purposes."

The point I want to make here is that focusing too much on the actual hub ends up as technological determinism--and leads to efforts to squeeze the community into the tool instead of using the tool to support the community. We must be much more focused on the participatory cultures and practices that the networks support. Often, this means supporting layered use of various technologies, according to the interests, needs, and dispositions of community members. In fact, the most important evidence that you have established a participatory culture around a network is that the practices you are fostering in your network spread to other networks. In other words, if you lurk on other networks, you should see reference to your network and practice.

opening up scholarship: generosity among grinches

2009-07-05T19:25:00.003-04:00

why academic research and open exchange of ideas are like that bottle of raspberry vinaigrette salad dressing you've had in the back of your fridge since last summer

The folks over at Good Magazine are tossing up a series of blogposts under the heading "We Like to Share."

The articles are actually a series of interviews with creative types in a variety of fields who share one characteristic: they believe that sharing of ideas and content is valuable and important. The edited interviews are being posted by Eric Steuer, the Creative Director of Creative Commons--a project which, though I admittedly don't fully understand it, I find deeply ethical and innovative with respect to offering new approaches to sharing and community.

So far, two posts have gone up, the first with Chris Hughes, a co-founder of Facebook and the former online strategist for the Obama presidential campaign, and the second second with Flickr founder Caterina Fake. Talking about how much we've changed in our attitudes toward sharing, Fake explains that

[i]f you go online today you will see stories about Obama sharing his private Flickr photos. So this is how far the world has come: our president is sharing photos of his life and experiences with the rest of the world, online. Our acceptance of public sharing has evolved a lot over the course of the past 15 years. And as people became increasingly comfortable sharing with each other—and the world—that lead to things that we didn’t even anticipate: the smart mob phenomenon, people cracking crimes, participatory media, subverting oppressive governments. We didn’t know these things were going to happen when we created the website, but that one decision—to make things public and sharable—had significant consequences.

Hughes' interview is less overtly about sharing as we typically think of the term, but he points out that the Obama campaign was successful because it focused on offering useful communications tools that lowered barriers to access and then

getting out of the way of the grassroots supporters and organizers who were already out there making technology the most efficient vehicle possible for them to be able to organize. That was a huge emphasis of our program: with people all over the place online—Facebook, MySpace, and a lot of other different networks—we worked hard to make sure anyone who was energized by the campaign and inspired by Barack Obama could share that enthusiasm with their friends, get involved, and do tangible things to help us get closer to victory. The Obama campaign was in many ways a good end to the grassroots energy that was out there.

Both interviews, for as far as they go, offer interesting insights into how sharing is approached by innovators within their respective spheres. But though these posts present their subjects as bold in their embrace of sharing and community, their ideas about what sharing means and how it matters are woefully...limited. Fake uses the Obama example to point out how far we've come; but really, does Obama's decision to make public photos of his adorable family mean much more than that he knows how to maintain his image as the handsome, open President who loves his family almost to a fault? I don't imagine we'd be very surprised to learn that Obama's advisors counseled him to make these photos widely available.

Indeed, the Flickr approach, in general, is this: These photos are mine and I will let you see them, but you have to give them back when you're done. It's a version of sharing, yes, but only along the lines of the sharing we learned to do as children.

The same is true of the picture Hughes paints of a campaign that successfully leveraged social networking technologies. The Obama campaign's decision to use participatory technologies was a calculated move: Everybody knows that a.) More young, wired and tech-savvy people supported Obama than McCain; and b.) those supporters required a little extra outreach in order to line up at the polls on election day. You can bet that if Republicans outnumbered Democrats on Facebook, you can bet Obama's managers would have been a little less quick to embrace these barrier-dropping communication tools.

What we're not seeing so far among these innovators is an innovative approach to sharing--one that opens up copyright-able and patent-able and, therefore, economically valuable ideas and content to the larger community.

I've been thinking about this lately because of my obsession with open education and open access. In particular, educational researchers--even those who embrace open educational resources--struggle with the prospect of making their work available to other interested researchers.

This makes sense to anyone who's undertaken ed research--prestige, funding, and plum faculty positions (what little there is of any of these things) are secured through the generation of innovative, unique scholarship and ideas, and ideas made readily available are ideas made readily stealable. As a fairly new addition to the field, even I have been a victim of intellectual property theft. It's enough to give a person pause, even if, like me, you're on open education like Joss Whedon on strong, feminist-type leading ladies.

But, come on, we all know there's no point to hiding good research from the public. As Kevin Smith writes in a recent blogpost on a San Jose State University professor who accused a student of copyright violation for posting assigned work online,

[t]here are many reasons to share scholarship, and very few reasons to keep it secret. Scholarship that is not shared has very little value, and the default position for scholars at all levels ought to be as much openness as is possible. There are a few situations in which it is appropriate to withhold scholarship from public view, but they should be carefully defined and circumscribed. After all, the point of our institutions is to increase public knowledge and to put learning at the service of society. And there are several ways in which scholars benefit personally by sharing their work widely.

Smith is right, of course, and the only real issue is figuring out strategies for getting everybody on board with the pro-sharing approach to scholarship. The "I made this and you can see it but you have to give it back when you're done" model is nice in theory but, in practice, limits innovation and progress in educational research. A more useful approach might be along the lines of: "I made this and you can feel free to appropriate the parts that are valuable to you, but please make sure you credit my work as your source material." This is a key principle at the core of the open education approach and of what media scholar Henry Jenkins calls "spreadability."

The problem is that there are enough academics who subscribe to the "share your toys but take them back when you're done playing" approach to research that anybody who embraces the free-appropriation model of scholarship ends up getting every toy stolen and has to go home with an empty bag. This is why the open education movement holds so much promise for all of academia: Adherents to the core values of open education agree that while we may not have a common vocabulary for the practice of sharing scholarship, we absolutely need to work to develop one. For all my criticisms of the OpenCourseWare projects at MIT and elsewhere, one essential aspect of this work is that it opens up a space to talk about how to share materials, and why, and when, and in what context. The content of these projects may be conservative, but the approach is wildly radical.

Applying the abundance model to the classroom

2009-06-24T09:08:00.007-04:00

In a recent Wired article called "Tech is Too Cheap to Meter: It's Time to Manage for Abundance, Not Scarcity," Chris Anderson considers the difference between a scarcity management model and an abundance model. His point is linked to management of technology resources; he writes that

[i]f you're controlling a scarce resource, like the prime-time broadcast schedule, you have to be discriminating. There are real costs associated with those half-hour chunks of network time, and the penalty for failing to reach tens of millions of viewers with them is calculated in red ink and lost careers. No wonder TV executives fall back on sitcom formulas and celebrities—they're safe bets in an expensive game.

But if you're tapping into an abundant resource, you can afford to take chances, since the cost of failure is so low. Nobody gets fired when your YouTube video is viewed only by your mom.

Anderson's point is that when resources--in this case, willing content producers with cheap production tools--are abundant, we need to rethink how we structure, market, and make money off of content.

The point, though linked to media marketing models, might easily be applied to the domain of education. The following graphic accompanies Anderson's piece:

Clearly, the abundance model as presented here aligns with the spirit of participatory culture, at its heart an egalitarian, anti-hierarchical movement wherein cultural decisions become crowdsourced. Here's where many school policies confuse scarcity and abundance: They block participatory media (including YouTube, many social networking sites, and sometimes Google and Wikipedia) and evaluate students based on their ability to repeat back to the teacher (or testmaker) the big ideas of the class. Knowledge, in this case, is treated as a scarce resource, when in a participatory culture knowledge is almost the most abundant thing we have.

What would it look like to apply an abundance model to the classroom? What new roles can and should teachers and students play in an egalitarian classrom in which "everything is permitted unless it is forbidden"? What's the difference, practically speaking, between a "command and control" classroom and a class without that type of control?

Important questions to chew on. More soon.

position paper abstract: embracing open education, open source, open technologies

2009-06-22T15:55:00.002-04:00

This post is intended as an abstract / scaffold for a longer position paper on the role of open education, the Free / Libre / Open source software (FLOSS) movement, open access and open technologies in re-mediating assessment. In this post, I identify some key principles of what, for the sake of brevity, I'll label the "open resource(s) movement" (ORM). In general, while FLOSS, open educational resources, and open access movements focus on different aspects of culture, they are linked theoretically to one key tenet: Making materials available to all benefits everybody.

Abstract: Relevant Big Ideas of the open resource(s) movement
We embrace the ethos of the open education movement. The ethos of open education also helps us to approach all three of the concerns identified in NML’s white paper: The participation gap, the transparency problem, and the ethics issue. The key tenet, that education can be improved by making educational assets visible and accessible and by harnessing the collective wisdom of a community of practice and reflection. The work of integrating participatory practices and accompanying assessment approaches into the formal classroom is an effort to address the three key concerns, and opening up access and assets to all learners and teachers can help in this regard.

We embrace the design approach inherent in the open education movement. As Casserly and Smith point out, open access is the most obvious but not the sole key feature of open educational resources:

Fully open educational resources provide a license that grants permission to users not only to read the material but also to download, modify, and post it for reuse. Users are empowered to change the materials to meet their own needs. They can mix and remix. The capacity and right to reuse materials is an important step in providing users all over the world the opportunity to actively participate…. Reuse also makes possible continuous cycles of improvement of educational materials as users quickly provide critical reactions and evaluations to developers of the quality and effectiveness of the materials. These fast feedback loops of users and developers create an environment for the improvement of content similar to the environment of open source software.

In designing a social network that presents a participatory, spreadable, and open approach to learning, knowing, and teaching, we intend to leverage the affordances of open educational resources to support innovative design and circulation of innovative teaching practices.

We embrace an undergirding principle of the F/OSS Movement that Clay Shirky identifies in Here Comes Everybody: That the F/OSS movement is so powerful because it relies on a “failure for free” model. As I explain "what open source can teach us about failure",

It's not, after all, that most open source projects present a legitimate threat to the corporate status quo; that's not what scares companies like Microsoft. What scares Microsoft is the fact that OSS can afford a thousand GNOME Bulgarias on the way to its Linux. Microsoft certainly can't afford that rate of failure, but the OSS movement can, because, as Shirky explains, “open systems lower the cost of failure, they do not create biases in favor of predictable but substandard outcomes, and they make it simpler to integrate the contributions of people who contribute only a single idea.”

Anyone who's worked for a company of reasonable size understands the push to keep the risk of failure low. "More people," Shirky writes, "will remember you saying yes to a failure than saying no to a radical but promising idea." The higher up the organizational chart you go, the harder the push will be for safe choices. Innovation, it seems, is both a product of and oppositional to the social contract.

Working from a “failure for free” approach to designing and spreading “successful” teaching practices offers a useful place to consider the role of high-stakes testing and other accountability measures alongside of the issue of how best to prepare learners for the new valued mindsets and skillsets afforded and supported by participatory media.

We need to consider how the notion of spreadability as described by Jenkins et al. aligns and conflicts with the ideals of the F/OSS ethos. In some significant ways, the notion of “spreadability” as described by Jenkins et al. seems at odds with at least some of the tenets and ideals of the F/OSS and open education movements. (Again, I’ve written some about this in a recent blogpost, "what open source can teach us about spreadability". Specifically, Jenkins et al. describe a tension between a “commodity culture” and a “gift economy.” Briefly, the a gift economy is the phenomenon of building social structures and social capital around the giving and receiving of gifts, whereas commodity culture considers the cash value of all goods and services.

As Jenkins et al. point out, in a culture where commodity culture and the gift economy collide (they give the example of the language of "file sharing" vs. "software piracy"),

[f]ocusing on...spreadability may thus offer us some tentative first steps towards renegotiating the social contract between media producers and consumers in a way which may be seen as legitimate and mutually rewarding to all involved.

In general, despite the relatively obvious conflicting interests of these two value systems, both define themselves in terms of “success”: A successful gift is one the giver values, and it therefore “buys” you cultural capital. A successful commodity is one the buyer wants, and is therefore willing to spend money on. The notion of "spreadability" relies on an assumption that content spreads when it is of value to a community--that is, when a person thinks other people like her will enjoy a certain link, commercial, song, product, and so on. Producers of content are hard at work analyzing that inscrutable kernel of a thing that makes it valuable.

This is a fantastic way of thinking through the issues tied to sticky and spreadable media, but not quite so applicable to the open source ethos where you get "failure for free." Indeed, OSS is premised on the foundational need for failure in order to arrive at success.

Additionally, in the F/OSS movement spreadability matters very little or not at all, because--and this is essential--there is no money to be made, and therefore no conflict between producers and consumers. In fact, as Shirky explains, the most successful OSS product of all time, Linux, succeeded precisely because its key developer, Linus Torvalds, made it clear from the beginning that he did not intend to make money off of the result.

We embrace the notion of a “teaching commons” and the struggles inherent in developing and circulating the knowledge offered therein. As Huber and Hutchings describe this in “What’s Next for Open Knowledge?”, a teaching commons is “’an emergent conceptual space for exchange and community among faculty, students, administrators, and all others committed to learning as an essential activity of life in contemporary democratic society’ (Huber and Hutchings, 2005, p. 1). In this commons, one can find a growing set of resources for and about teaching and learning, produced not only by pedagogical specialists, but by teachers and learners of all kinds.”

A key concern for the future of the teaching commons, as Huber and Hutchings explain, is the issue of “open knowledge in an era of accountability.” The authors are concerned about

how to maintain a space for educational experimentation and exchange in a period that seems headed for increasingly bottom-line forms of accountability, with its concomitant calls for institutions to make evidence of student learning outcomes available to the public….

At one level, the value of evidence is something that any responsible educator would share. Faculty care about their students, and they want to know that the resources they find in the teaching commons will serve those students well. The danger comes when high stakes constrict people’s ability or willingness to explore new pedagogical ideas (Shulman, 2007).

The authors here are focused on accountability in higher education, and we would argue that the problem is even more acute at the secondary education level, where high-stakes testing and NCLB can leave innovative teachers feeling hamstrung by the need to meet accountability demands. Our work on re-mediating assessment practices to align with the new valued literacy practices supported by participatory media is an effort to provide teachers with the support they need to integrate participatory practices into their classroom and meet the real needs of the formal classroom.

open education, open source, open access: some definitions

2009-06-19T09:59:00.003-04:00

Pretty soon, you'll be seeing a position paper on the open education movement on this blog. The paper is in the works, but I wanted to toss up a glossary of some key terms that will inform that paper. This is my Friday present to you.

Open Source: Open source is an approach to the design, development, and distribution of software, offering practical accessibility to a software's source code. Some consider open source as one of various possible design approaches, while others consider it a critical strategic element of their operations. Before open source became widely adopted, developers and producers used a variety of phrases to describe the concept; the term open source gained popularity with the rise of the Internet, which provided access to diverse production models, communication paths, and interactive communities. (http://en.wikipedia.org/wiki/Open_source)

Open Source Software (OSS): computer software for which the source code and certain other rights normally reserved for copyright holders are provided under a software license that meets the Open Source Definition or that is in the public domain. This permits users to use, change, and improve the software, and to redistribute it in modified or unmodified forms. It is very often developed in a public, collaborative manner. Open source software is the most prominent example of open source development and often compared to user-generated content. The term open source software originated as part of a marketing campaign for free software.

Free Software (vs. Open Source Software): The term “free software” was coined by Richard Stallman, who explains that

When we call software “free,” we mean that it respects the users' essential freedoms: the freedom to run it, to study and change it, and to redistribute copies with or without changes. This is a matter of freedom, not price, so think of “free speech,” not “free beer.” (http://www.gnu.org/philosophy/open-source-misses-the-point.html)

Briefly, the difference in the terms highlights different ethical approaches to software development. In general, the OSS movement emphasizes the collective engagement with source code in order to develop, and sometimes to market, powerful and efficient software. The free software movement identifies as a social movement. Stallman explains:

Nearly all open source software is free software; the two terms describe almost the same category of software. But they stand for views based on fundamentally different values. Open source is a development methodology; free software is a social movement. For the free software movement, free software is an ethical imperative, because only free software respects the users' freedom. By contrast, the philosophy of open source considers issues in terms of how to make software “better”—in a practical sense only. It says that non-free software is a suboptimal solution. For the free software movement, however, non-free software is a social problem, and moving to free software is the solution.

Many adherents to these movements, to avoid this issue, simply refer to the Free/Open Source Software (F/OSS) Movement.

Community Source Software (CSS): Community Source Software differs from OSS in that institutions devote paid employees to the project, with the intention of collaboratively developing a product that embraces the open source ethos. From the Wikipedia article on Community source,

An important distinctive characteristic of community source as opposed to plain open source is that the community includes some organizations or institutions that are committing their resources to the community, in the form of human resources or other financial elements. In this way, the open source project will have both more solid support, rather than purely volunteer efforts as found in other open source communities, and will possibly be shaped by the strategic requirements of the institution committing the resource.

Examples of CSS include: the Sakai Project, Kuali Foundation, and Open Source Portfolio.

Open Access (OA): From http://www.earlham.edu/~peters/fos/overview.htm, “open access (OA) literature is digital, online, free of charge, and free of most copyright and licensing restrictions. The goal of adopting OA policies is to remove barriers to information. Many higher education institutions have adopted an open access policy, as for example the Massachusetts Institute of Technology, which explains that it adopted an OA policy because “The Faculty of the Massachusetts Institute of Technology is committed to disseminating the fruits of its research and scholarship as widely as possible.”

Open Education Movement and Open Educational Resources (OERs): From Opening Up Education, a key tenet of this movement is that education can be improved by making educational assets visible and accessible and by harnessing the collective wisdom of a community of practice and reflection. The open education movement embraces a shift away from a scarcity-based model of higher education, which bases its value on limiting access. As Batson, Paharia, and Kumar explain (in chapter 6, “A Harvest Too Large? A Framework for Educational Abundance”), open education works within a “knowledge ecology characterized by unfettered access to educational resources, choice, and change in the context and clientele of higher education.” In the open, “abundance-based” learning framework, we see the following shifts, with the “trend indicators” column showing features of higher education that point to the shift.

Recursive Publics: This term was coined by Christopher Kelty, who describes it at length in Two Bits (available for download, online browsing, and modulation for free online):

A recursive public is a public that is vitally concerned with the material and practical maintenance and modification of the technical, legal, practical, and conceptual means of its own existence as a public; it is a collective independent of other forms of constituted power and is capable of speaking to existing forms of power through the production of actually existing alternatives.

More to the point, a recursive public is a group of people who exist outside of traditional institutions (governments, churches, schools, corporations) and, when necessary, use this outsider status to hold these entities in check. The engagement of these publics goes far beyond simply protesting decisions or stating their opinions. Kelty, writing about geek culture as a recursive public, explains it thus:

Recursive publics seek to create what might be understood, enigmatically, as a constantly “self-leveling” level playing field. And it is in the attempt to make the playing field self-leveling that they confront and resist forms of power and control that seek to level it to the advantage of one or another large constituency: state, government, corporation, profession. It is important to understand that geeks do not simply want to level the playing field to their advantage—they have no affinity or identity as such. Instead, they wish to devise ways to give the playing field a certain kind of agency, effected through the agency of many different humans, but checked by its technical and legal structure and openness. Geeks do not wish to compete qua capitalists or entrepreneurs unless they can assure themselves that (qua public actors) that they can compete fairly. It is an ethic of justice shot through with an aesthetic of technical elegance and legal cleverness.

a no-win proposition: how small schools hurt the big-school bottom line

2009-06-17T22:25:00.006-04:00

Recently, the crew here at re-mediating assessment has been thinking and talking about the affordances of small schools. Specifically, we've been talking about alternative schools, with their smaller populations, greater flexibility, and targeted instructional techniques.

When I say "alternative," I want you to think alternative as in "an option other than the big public high school" and not as in "an option other than going to juvenile detention." In particular, we've been working with Becky Rupert, a teacher at Aurora Alternative High School, in Bloomington, IN. We've had roaring success working with her and her students, and we've been talking about expanding our work to multiple alternative or small schools in Indiana.

Now, however, comes a report out of New York City that the small schools movement there is causing larger schools to struggle to accommodate the needs of a shifting student population. According to the report, which was developed by New York's Center for New York City Affairs, the small schools movement was a key initiative embraced by Mayor Bloomberg as an effort to serve students at failing high schools. The city has closed over two dozen large high schools in the last seven years, replacing them with smaller schools.

While students at those smaller schools showed strong increases on achievement, many students from the closed schools simply moved to other large public high schools in the area. Whereas the smaller schools were able to provide individualized attention and target learning deficiencies, the larger schools--often already ill equipped to meet the needs of its student body--was unable to accommodate the increased population of struggling learners. As a result, attendance and graduation rates have dropped at the larger schools.

People are going to be tempted to use this study to prove one of the following assertions:

Small schools are meeting student needs more successfully, and we should therefore try to replace as many larger schools as possible with the smaller, more personalized alternative;
Small schools are hurting more students than they are helping, and we should therefore close them down and invest that money in larger public schools.

Neither of these arguments is right. This is not an either/or proposition, and turning it into a black-and-white issue does a disservice to the complicated field of teaching and learning. The truth is that while small schools may raise achievement on accountability measures--standardized test scores, attendance and graduation rates--there are indications that this comes at the expense of some other key learning opportunities. Small schools cannot, for example, offer the variety of courses that larger schools can provide; they often lack extracurricular opportunities such as sports teams, clubs, and academic organizations; and while they may offer comparable technological resources to those offered at larger schools, small schools are less likely to contain the range of adult expertises that allow the technologies to be leveraged for a range of participatory experiences.

At the same time, large schools that come equipped with the above still often lack the ability to meet the needs of the broad population that they serve. Instruction tends to target a "typical learner"--the type of student who can do well on state-mandated test, given proper instruction; who will do moderately well in one or two AP classes; who will graduate with a 3.4 grade point average, one internship, and plans for college. Large schools can't direct instruction toward the specific interests, needs, and values of its student body, and all the extracurricular opportunities and Zulu classes in the world won't make a difference to the overlooked and underserved students of these bigger institutions.

The solution is somewhere in the middle, though don't ask me where the "middle" is. The answer is not to simply close down both large and small schools and replace them with "medium" schools. The answer is not to open up city-sponsored afterschool programs for all learners. The change that must occur is something much deeper.

In truth, as the product of a sprawling public high school that served 5,000 students, I lean more toward the value systems embraced by and possibilities inherent in many small schools. A more careful, more measured, more personalized approach to instruction is strides away from the factory model of education that lingers from the early days of compulsory education in America. As I think we can all agree, the last thing we need is more high school graduates equipped for a career in the factories that, for the most part, no longer even exist.

the harrison bergeron approach to education: how university rankings stunt the social revolution

2009-06-14T09:44:00.006-04:00

I've been thinking some lately about the odd and confusing practice of comparing undergraduate and graduate programs at American colleges and universities and producing a set of rankings that show how the programs stack up against each other.

One of the most widely cited set of rankings comes from U.S. News and World Report, which offers rankings in dozens of categories, for both undergraduate and graduate-level programs. Here, the magazine offers its altruistic rationale behind producing these rankings:

A college education is one of the most important—and one of the most costly—investments that prospective students will ever make. For this reason, the editors of U.S. News believe that students and their families should have as much information as possible about the comparative merits of the educational programs at America's colleges and universities. The data we gather on America's colleges—and the rankings of the schools that arise from these data—serve as an objective guide by which students and their parents can compare the academic quality of schools. When consumers purchase a car or a computer, this sort of information is readily available. We think it's even more important that comparative data help people make informed decisions about an education that at some private universities is now approaching a total cost of more than $200,000 including tuition, room, board, required fees, books, transportation, and other personal expenses.

(To access the entire rankings, developed and produced selflessly by U.S. News and World Report, you need to pay. Click here to purchase the Premium Online Edition, which is the only way to get complete rankings, for $14.95.)

The 2009 rankings, released in April, are in the news lately because of questions related to how the magazine gathers data from colleges. As Carl Bialik points out in a recent post at the Wall Street Journal, concerns over how Clemson University set about increasing its rank point to deeper questions about the influence of rankings numbers on university operations. Clemson President James F. Barker reportedly shot for cracking the top 20 (it was ranked 38th nationally in 2001) by targeting all of the ranking indicators used by U.S. News. Bialik writes:

While the truth about Clemson’s approach to the rankings remains elusive, the episode does call into question the utility of a ranking that schools can seek to manipulate. “Colleges have been ‘rank-steering,’ — driving under the influence of the rankings,” Lloyd Thacker, executive director of the Education Conservancy and a critic of rankings, told the Associated Press. “We’ve seen over the years a shifting of resources to influence ranks.”

Setting aside questions of the rankings' influence on university operations and on recruiting (both for prospective students and prospective faculty), and setting aside too the question of how accurate any numbers collected from university officials themselves could possibly be when the stakes are so high, one wonders how these rankings limit schools' ability to embrace what appear to be key tenets emerging out of the social revolution. A key feature of some of the most vibrant, energetic, and active online communities is what Clay Shirky labels the "failure for free" model. As I explained in a previous post on the open source movement, the open source software (OSS) movement embraces this tenet:

It's not, after all, that most open source projects present a legitimate threat to the corporate status quo; that's not what scares companies like Microsoft. What scares Microsoft is the fact that OSS can afford a thousand GNOME Bulgarias on the way to its Linux. Microsoft certainly can't afford that rate of failure, but the OSS movement can, because, as Shirky explains,
open systems lower the cost of failure, they do not create biases in favor of predictable but substandard outcomes, and they make it simpler to integrate the contributions of people who contribute only a single idea.

Anyone who's worked for a company of reasonable size understands the push to keep the risk of failure low. "More people," Shirky writes, "will remember you saying yes to a failure than saying no to a radical but promising idea." The higher up the organizational chart you go, the harder the push will be for safe choices. Innovation, it seems, is both a product of and oppositional to the social contract.

The U.S. News rankings, and the methodology behind them, runs completely anathema to the notion of innovation. Indeed, a full 25 percent of the ranking system is based on what U.S. News calls "peer assessment," which comes from "the top academics we consult--presidents, provosts, and deans of admissions" and, ostensibly, at least, allows these consultants

to account for intangibles such as faculty dedication to teaching. Each individual is asked to rate peer schools' academic programs on a scale from 1 (marginal) to 5 (distinguished). Those who don't know enough about a school to evaluate it fairly are asked to mark "don't know." Synovate, an opinion-research firm based near Chicago, in spring 2008 collected the data; of the 4,272 people who were sent questionnaires, 46 percent responded.

Who becomes "distinguished" in the ivory-tower world of academia? Those who play by the long-established rules of tradition, polity, and networking, of course. The people who most want to effect change at the institutional level are often the most outraged, the most unwilling to play by the rules established by administrators and rankings systems, and therefore the least likely to make it into the top echelons of academia. Indeed, failure is rarely free in the high-stakes world of academics; it's safer to say no to "a radical but promising idea" than to say yes to any number of boring but safe ideas.

So what do you do if you are, say, a prospective doctoral student who wants to tear wide the gates of academic institutions? What do you do if you want to go as far in your chosen field as your little legs will carry you, leaving a swath of destruction in your wake? What do you do if you want to bring the social revolution to the ivory tower, instead of waiting for the ivory tower to come to the social revolution?

You rely on the U.S. News rankings, of course. It's what I did when I made decisions about which schools to apply to (the University of Wisconsin-Madison [ranked 7th overall in graduate education programs, first in Curriculum & Instruction, first in Educational Psychology] the University of Texas-Austin [tied at 7th overall, 10th in Curriculum & Instruction], the University of Washington [12th overall, 9th in Curriculum & Instruction], the University of Michigan [14th overall, 7th in Curriculum & Instruction, and 3rd in Educational Psychology] the University of Indiana [19th overall, out of the top 10 in individual categories], and Arizona State University [24th overall, out of the top 10 in individual categories]). Interestingly, though, the decision to turn down offers from schools ranked higher than Indiana (go hoosiers) wasn't all that difficult. I knew that I belonged at IU (go hoosiers) almost before I visited, and a recruitment weekend sealed the deal.

But I had an inside track to information about IU (go hoosiers) via my work with Dan Hickey and Michelle Honeyford. I also happen to be a highly resourceful learner with a relatively clear sense of what I want to study, and with whom, and why. Other learners--especially undergraduates--aren't necessarily in such a cushy position. They are likely to rely heavily on rankings in making decisions about where to apply and which offer to accept. This not only serves to reify the arbitrary and esoteric rankings system (highest ranked schools get highest ranked students), but also serves to stunt the social revolution in an institution that needs revolution, and desperately.

In this matter, it's turtles all the way down. High-stakes standardized testing practices and teacher evaluations based on achievement on these tests limits innovation--from teachers as well as from students--at the secondary and, increasingly, the elementary level. But the world that surrounds schools is increasingly ruled by those who know how to innovate, how to say yes to a radical but promising idea, how to work within a "failure for free" model. If schools can't learn how to embrace the increasingly valued and valuable mindsets afforded by participatory practices, it's failing to prepare its student body for the world at large. The rankings system is just another set of hobbles added on to a system of clamps, tethers, and chains already set up to fail the very people it purports to serve.

on social networking guidelines for teachers

2009-06-02T14:10:00.001-04:00

I was recently directed to a recent post on a blog called "Blogg-ed Indetermination" offering a first pass at a set of guidelines for using social networking tools in the K-12 classroom.

The blog's author, Steve Taffee, points out that while young people are taking to social networking "like ducks to water," adults are more conflicted about the appropriate uses for social networks in schools. He offers up a set of nine guidelines, not intended to be the final word but intended to start a conversation "in the best of social networking tradition." With this impulse in mind, I'll repeat the proposed set of guidelines and offer my suggestions for refinement.

Proposed Guidelines for Use of Social Networks by School Faculty and Staff*

New technologies, such as social networking tools, provide exciting new ways to collaborate and communicate. Nevertheless we must exercise care to be sure we use such tools with students in ways that are both age-appropriate and consistent with the mission of the school.

School faculty and staff are expected to behave honorably in both real and virtual (online) spaces. Activities which are improper, unethical, illegal, or which cause undue discomfort for students, employees, parents, or other members of the school community should be judiciously avoided in both physical space and cyberspace.

To that end, we offer the following guidelines for school employees who use online social networking applications which may be frequented by current or former students.

1. COURSE USE OF SOCIAL NETWORKING: In order to provide equal, age-appropriate access for students to course materials, faculty should limit class activities to school-sanctioned online tools. New social networking tools and features are being continually introduced which may or may not be appropriate for course use. The same care must be taken in choosing such tools as other tools and support materials.

2. MODEL APPROPRIATE BEHAVIOR: Exercise appropriate discretion when using social networks for personal communications (friends, colleagues, parents, former students, etc.) with the knowledge that adult behavior on social networks may be used as a model by our students.

3. FRIENDING ALUMNI: Accept social network friend requests only with alumni over the age of 18. Do not initiate friend contacts with alumni.

4. UNEQUAL RELATIONSHIPS: Understand that the uneven power dynamics of the school, in which adults have authority over former students, continues to shape those relationships.

5. OTHER FRIENDS: Remind all other members of your network of your position as an educator whose profile may be accessed by current or former students, and to monitor their posts to your network accordingly. Conversely, be judicious in your postings to all friends sites, and act immediately to remove any material that may be inappropriate from your site whether posted by you or someone else.

6. GROUPS IN YOUR SOCIAL NETWORK: Associate with social networking groups consistent with healthy, pro-social activities and the mission and reputation of the school, acting with sensitivity within context of a diverse educational environment in which both students and adults practice tolerance and accept competing views.

7. PRIVACY SETTINGS AND CONTENT: Exercise care with privacy settings and profile content. Content should be placed thoughtfully and periodically reviewed to maintain this standard.

8. MISREPRESENTATION: Faculty who use social networks should do so using their own name, not a pseudonym or nickname.

9. PUBLIC INFORMATION: Recognize that many former students have online connections with current students, and that information shared between school adults and former students is likely to be seen by current students as well.

===

*Some of the ideas for this list come from a Facebook group I belong to, Faculty Ethics on Facebook. It is geared towards higher education, and so if you stumbled upon this post and really want to read about colleges and universities, head on over to Facebook. I also appreciate colleague Matt Montagne’s feedback via Google Docs on an earlier draft of these ideas.

In general, these guidelines offer a strong starting point for discussing the ethical dimensions of participation in social networking sites, both in the classroom and outside of it. The drive toward modeling honest, responsible networking activities makes good sense, especially in a world where faculty can lose their jobs and their careers for the material they post online. But these guidelines present strategies that have the potential to limit teacher and student access to authentic participation in online social spaces. Specifically, the slant against "misrepresentation" and toward using only approved social networking sites in schools present significant participation concerns. For teachers, the issue is about their right to engage meaningfully in a public sphere that may offer the potential for inappropriate or damaging material. For students, the issue is more drastic: It's a matter of social justice. Students who don't have access to new media technologies and can't experience the authentic online social spaces in the classroom will be ill equipped to experience those spaces when they leave school.

On "Misrepresentation"
The push toward "honesty" goes perhaps a few steps too far, overlooking the fact that engagement with media platforms that are increasingly persistent, searchable, and replicatable call for new approaches to disclosure. I'm pointing here to guideline 8, which Taffee labels "misrepresentation."

Anonymity and its close cousin, pseudonymity, have a long and storied relationship with the politics of identity performance. We've come a long way (we have, haven't we?) from the time when speaking up against a tyrant could lead to personal, financial, or social ruin. (We have, haven't we?)

But until recently, "misrepresentation" was generally viewed as the domain of the whistleblower, and members of everyday culture were expected to act in their own names. In a participatory culture, however, where people can increasingly engage with identity play in a wide range of online spaces, psuedonyms, nicknames, and even complete anonymity serve as a buffer against repercussion. Indeed, it may be the case that a teacher wants to use Facebook or a similar site to engage in NSFW conversations, photo sharing, and precisely the kind of social networking that these sites afford. In that case, the teacher might choose to design a "fake" profile in order to prevent students or students' parents from encountering this material. It's not "misrepresentation" so much as it's a version of protected self-presentation.

As our social lives increasingly occupy online spaces in addition to offline, in-person relationships, we need to offer new strategies for engagement with these sites--strategies that afford full participation in addition to protecting people from the risk of having material intended for one audience dragged into the public light of a different, unintended audience.

On Course Use of Social Networking
The impulse driving guideline #1 is a valid one. It is, as Lynn Sykes, a teacher and friend, pointed out to me, a great big social networking world out there, and the minute we introduce social media into the classroom we also introduce the risk that learners will stumble upon material that is inappropriate for the classroom setting.

But ignoring this risk doesn't make it go away; indeed, it leaves many students ill-equipped to make intelligent decisions about what to do when they encounter this kind of material in real life, as they are certain to do. Learners who have access to social media and adult support for reflecting on their engagement with it in their homes will be prepared, of course. It's the learners with less access and less extracurricular support--in other words, the poor, the disadvantaged, the learners who have historically been left behind in school, in work, in life--who can most benefit from the experience of engaging with social media in the classroom.

This isn't to say that the concerns about inappropriate material aren't valid concerns. This is why we need to work in two distinct directions:

Working at the policy level to develop regulations that allow for safe and guided access to the authentic social media experiences that will prepare learners for engagement with the participatory media, practices, and cultures that are becoming increasingly essential to success outside of school;
Working in the classroom to establish norms that can govern students' ethical participation in social media, such that they can immediately identify, and know how to respond to, material that's inappropriate for the school context.

Steve, I would recommend including the above guidelines into a revised version of these guidelines. I'm looking forward to continuing this important conversation.

smacking down the Boston Globe

2009-05-30T11:50:00.003-04:00

Originally posted May 5 at sleeping alone and starting out early

It's too bad the Boston Globe is staying open, because now I have to smack it down big-time for this editorial arguing that we shouldn't standardize and measure achievement on so-called 21st-century skills. The op-ed offers further proof--as if we needed it--that the Globe's editorial board has no idea how the playing field has been utterly transformed by participatory culture.

The impetus behind the op-ed is a move by the state Department of Elementary and Secondary Education to put its money where its mouth is. The department recently awarded a $146 million contract to the designer of the MCAS, the standardized test mandated in the commonwealth of Massachusetts by No Child Left Behind, and part of that money is earmarked for integration of 21st-century skills assessment. This is a problem, as the Globe's editorial board will point out momentarily.

But first, it uses state MCAS scores as proof of public school rigor. As it explains,

Massachusetts stands apart in public education precisely because it created high academic standards, developed an objective measure of student performance and progress through the MCAS test, and required a passing grade in order to graduate. Students, as a result, rank at or near the top of standardized testing not just nationally but on tough international achievement tests in math and science. Any retreat from this strategy would be a profound mistake.

So to summarize: Massachusetts students are among the top in the nation because their achievement on standardized tests prepares them to...score well on standardized tests. It's like the iconic example of circular reasoning: The MCAS is useful because it prepares them for future learning. How do you know? Because Massachusetts students do well on other standardized tests. What prepares them to do well on those tests? Doing well on standardized tests, of course.

Given the Globe's wholehearted genuflection at the altar of bubble tests, one wonders why this editorial might oppose integrating assessment of 21st-century skills in addition to traditional subjects. It turns out their concern is less about whether we should measure 21st century skills than it is about how doing so on the MCAS will affect test scores in general. As the editorial points out,

[s]tate education officials have done a generally poor job of defining 21st-century skills - which can include interdisciplinary thinking and media literacy - or explaining how to test them statewide.

The problem for the Globe, it turns out, is that if we develop mediocre assessment strategies it'll ruin the MCAS for all of us. Because 21st-century skills can only be measured subjectively, the Globe argues, an "objective" test like the MCAS is an inappropriate place to assess achievement. Instead,

MCAS testers should concentrate on accurately measuring math ability and reading comprehension, which surely correlate with a student's success in the workplace.

Let's leave, just for now, the outrageous assumption that a standardized test could conceivably be considered "objective." Let's leave the assumption that a standardized test could "accurately" measure student ability in anything other than the ability to engage in the weird and peculiar game of test-taking. Which leaves just one last question:

In what world can anybody make the argument that achievement in math and reading without the accompanying facility with 21st-century proficiencies prepares any learning for any workplace worth the energy of applying for employment in the first place?

It's such a weird argument to make, that literacy practices like reading, writing, and doing math can be somehow isolated from the 21st-century contexts that make them meaningful. It's like asking someone if she knows how to tie her shoe, then making her
prove it by writing a detailed step-by-step description of how to do it. It's like asking someone to prove he can build a fire: But is the fire for warmth, for signaling for help, or for burning the whole house down?

Same with math: Knowing how to "do" fractions doesn't mean a learner is equipped to, say, resize a .jpg for a blogpost.

Arguing that we should keep 21st-century skills out of standardized tests in order to keep the tests objective is as lame as the argument that standardized tests are objective in the first place. Neither one makes any logical sense. Neither one gets you anywhere.

why you should invite me to your next party

2009-05-29T16:15:00.002-04:00

(hint: because I will entertain your guests with talk of the social revolution)
(cross-posted from sleeping alone and starting out early).

I was at a party last week when someone asked me what I do for a living. I used the opportunity to engage in what, in retrospect, may have been an ill-timed impromptu pronouncement about the status of the social revolution.

It turns out I'll need to rethink how I use that phrase "social revolution," at least in mixed company, because a tubby drunk man wearing a confusing hat walked up to me and tried to steer the conversation toward war atrocities.

"You can't tell me," he bellowed, "that the atrocities that are happening during the Iraq War are any different from the ones that happened during World War II. It's just that we have more media coverage now."

As I wrote in an earlier post, this is what I've decided to call the Space Odyssey mistake. This particular kind of error is explained by Clay Shirky, who describes a scene from 2001 in which

space stewardesses in pink miniskirts welcome the arriving passenger. This is the perfect, media-ready version of the future--the technology changes, hemlines remain the same, and life goes on much as today, except faster, higher, and shinier.

Lately I've been finding Christopher Kelty's notion of a "recursive public" useful in thinking about what, other than hemlines, have changed. As Kelty describes it in Two Bits (available for download, online browsing, and modulation for free online),

A recursive public is a public that is vitally concerned with the material and practical maintenance and modification of the technical, legal, practical, and conceptual means of its own existence as a public; it is a collective independent of other forms of constituted power and is capable of speaking to existing forms of power through the production of actually existing alternatives.

Recursive publics seek to create what might be understood, enigmatically, as a constantly “self-leveling” level playing field. And it is in the attempt to make the playing field self-leveling that they confront and resist forms of power and control that seek to level it to the advantage of one or another large constituency: state, government, corporation, profession. It is important to understand that geeks do not simply want to level the playing field to their advantage—they have no affinity or identity as such. Instead, they wish to devise ways to give the playing field a certain kind of agency, effected through the agency of many different humans, but checked by its technical and legal structure and openness. Geeks do not wish to compete qua capitalists or entrepreneurs unless they can assure themselves that (qua public actors) that they can compete fairly. It is an ethic of justice shot through with an aesthetic of technical elegance and legal cleverness.

This is precisely the difference between 1945 and 2009. It's not just that we have more media coverage but that, as Shirky proclaims, everybody is a potential media outlet--everyone has the potential to join a recursive public, whether impromptu or planned.

In fact, the notion that we can all engage in reportage is perhaps a bit too simplistic, at least until we can adjust what we mean by "journalism." When Facebook users joined up in opposition to a change in Facebook's terms of service and successfully pressed administrators to rethink and reword the terms of service agreement, that was the work of a recursive public, loosely banded and easily disbanded once their purpose had been achieved (if necessary, they will quickly gather again in their virtual space and just as quickly disband). We don't recognize this as journalism, often don't even recognize it as civic engagement--but for those who joined this Facebook knotwork, it's certainly some kind of engagement. And what could be more civic-minded than fighting to define the uses of a public space?

The atrocities of war are approximately the same (though, as always, new technologies mean new modes of torture and murder). What's different is the following:

Citizens within and outside of the territories in question have the capacity to keep mainstream media outlets honest: Biased, incomplete, and corrupt journalism will no longer stand in for the official story.
Citizens can engage in consequential debate, which is to say that serious public conversation about the merits of a particular conflict can gather enough voices to sway public opinion and result in real policy changes more quickly than ever before. This is Vietnam to the power of Internet.
Citizens without access to a wide range of new media technologies can still have their experiences broadcast to the world by other members of their recursive public. A cellphone picture of a soldier torturing a civilian, for example, can be quickly distributed for maximum effect.

All in all, it was a good party. Near the end, someone produced a Donald Rumsfeld piñata. We were going to hoist it up and smash it, but it seemed kind of...irrelevant.

figuring out "how to go on"

2009-05-29T00:46:00.004-04:00

In his paper "Human Action and Social Groups as the Natural Home of Assessment: Thoughts on 21st Century Learning and Assessment," Jim Gee describes what at first glance appears to be two opposing uses of assessment in informal online spaces. As Gee explains,

Assessment for most social groups is both a form of mentoring and policing. These two are, however, not as opposed to each other as it might at first seem (and as they often are in school). Newcomers want to “live up to” their new identity and, since this is an identity they value, they want that identity “policed” so that it remains worth having by the time they gain it more fully. They buy into the “standards.” Surely this is how SWAT team members, scientists, and Yu-Gi-Oh fanatics feel.

At its best, assessment in formal education serves the same dual role; yet something is most assuredly different. What's different is not the degree to which students "buy into" the value of assessment; they see assessment in school as important, just as they would argue that the mentoring and policing of the online spaces they inhabit is an essential element to keeping those spaces alive.

What's different is not the degree of investment; what's different is the degree of relevance. The Yu-Gi-Oh fanatics Gee references want--sometimes desperately--to be accepted into the groups they join, and so they agree to the terms of this belonging, even if it requires being held to at times impossibly high standards of participation. The same is true of novice SWAT team members and scientists; it's less true of 11th graders reading Moby-Dick in an English classroom. To what purpose? they might ask, and they would be right to do so. Until we can align the goals, roles, and assessment practices of the formal classroom--until, that is, we can transform the domain to meet the needs of a participatory culture--investment exists without relevance. Students want A's to get into college; they want A's because that's what their parents tells them equals success; they want A's (or D's, or F's) because their friends tell them they should.

We know that practices are mediated by the tools we use to engage in those practices. We know that writing with a pencil is different from writing with a computer is different from writing with a Blackberry. The notion of "re-mediation" is intended to point to another level of mediation: That the tools that mediate traditional literacy practices get re-mediated by new media, which then re-mediates the practices that we bring to the tools. It's all very meta.

All of this is by way of introduction to re-mediating assessment, a new blog emerging out of a clever little partnership between a plucky crew of assessment-oriented researchers out of Indiana University and MIT. The plucky researchers include:

Daniel T. Hickey, Associate Professor of Learning Sciences at Indiana University, and our intrepid leader. Dan's research focuses on participatory approaches to assessment and motivation, design-based educational research, and program evaluation. He is particularly interested in how new participatory approaches can advance nagging educational debates over things like assessment formats and the use of extrinsic incentives. His work is funded by the National Science Foundation, NASA, and the MacArthur foundation, and has mostly been conducted in the context of digital social networks and videogames. He teaches graduate courses on cognition & instruction, assessment, and motivation, and undergraduate classes on educational psychology.

Michelle Honeyford, a Ph.D. Candidate in the Literacy, Culture, and Language Education Department at Indiana University and the cool head behind this operation. Michelle is a Graduate Research Assistant on the MacArthur-funded 21st Century Assessment Project for Situated and Sociocultural Approaches to Learning, working on a participatory assessment model for new media literacies. Her broader research interests include identity, cultural citizenship, and new literacy studies. Michelle is a former middle and high school English Language Arts teacher, and has taught courses in the teaching of writing at IU.

Jenna McWilliams, the little engine that could. Jenna is a prolific blogger who is working on mastering the art of being both smart and lucky, sometimes simultaneously. She recently got picked up by the Guardian's online site, Comment is Free, and was interviewed about the future of newspapers on the BBC's News Hour program. She currently works as an educational researcher for Project New Media Literacies, a MacArthur-funded research project based at MIT; prior to that, she taught English composition, literature, and creative writing at Suffolk University, Bridgewater State College, and at Newbury College and at Colorado State University, where she earned her MFA in Creative Writing. In Fall 2009, she will begin doctoral study in the Learning Sciences Program at Indiana University, under the tutelage of Dan Hickey, who is her sensei.

Together, this merry band will start working out the simple matter of "how to go on" and how to align classroom practices with the proficiencies called for--indeed, demanded--by a participatory culture.

We're bringing the smart. Wish us luck.