Positioning Portfolios for Participation

Much of our work in our 21st Century Assessment project this year has focused on communicating participatory assessment to broader audiences whose practices we are trying to inform. This includes:

• classroom teachers whose practices we are helping reshape to include more participation (like those we are working with in Monroe County right now);


• other assessment researchers who seem to dismiss participatory accounts of learning as “anecdotal” (like my doctoral mentor Jim Pellegrino who chaired the NRC panel on student assessment);


• instructional innovators who are trying to support participation while also providing broadly convincing accounts of learning (like my colleagues Sasha Barab and Melissa Gresalfi whose Quest Atlantis immersive environment has been a testbed for many of our idea about assessment);


• faculty in teacher education who are struggling to help pre-service teachers build professional portfolios while knowing that their score on the Praxis will count for much more (and whose jobs are being threatened by efforts in Indiana to phase out teacher education programs and replace them with more discipline-based instruction);


• teachers in my graduate-level classroom assessment course who are learning how to do a better job assessing students in their classrooms, as part of their MA degree in educational leadership.

It turns out that participatory approaches to assessment are quite complicated, because they must bridge the void between the socially-defined views of knowing and learning that define participation, and the individually-defined models of knowing and learning that have traditionally been taken for granted by the assessment and measurement communities. As our project sponsor Jim Gee has quite succinctly put: Your challenge is clarity.

As I have come to see most recently, clarity is about entry. Where do we start introducing this comprehensive new approach? Our approach itself is not that complicated really. We have it boiled down to a more participatory version of Wiggins' well known Understanding by Design. In fact we have taken to calling our approach Participation by Design (or if he sues us, Designing for Participation). But the theory behind our approach is maddeningly complex , because it has to span the entire range of activity timescales (from moment-to-moment classroom activity to long-term policy change) and characterizations of learning (from communal discourse to individual understanding to aggregated achievement).

Portfolios and Positioning
Now it is clear to me that the best entry point is the familiar notion of the portfolio. Portfolios consist of any artifacts that learners create. Thanks to Melissa Gresalfi, I have come to realize that the portfolio, and the artifacts that they contain, are ideal for explaining participatory assessment. This is because portfolios position (where position is used as a verb). Before I get to the clarity part, let me first elaborate on what this means.

It turns out that portfolios can be used to position learners and domain content in ways that bridges this void between communal activity and aggregated attainment. In a paper with Caro Williams about the math project that Melissa and I worked on together, Melissa wrote that

“positioning, as a mechanism, helps bridge the space between the opportunities that are available for participation in particular ways and what individual participants do”

Building on the ideas of her doctoral advisor Jim Greeno (e.g., Greeno and Hull, 2002) Melissa explained that positioning refers to how students are positioned relative to content (called disciplinary positioning) and how they are positioned relative to others (called interpersonal positioning). As I will add below, positioning also refer to how instructors are positioned relative to the students and the content (perhaps called professorial positioning). This post will explore how portfolios can support all three types of positioning in more effective and in less effective ways.

Melissa further explained that positioning occurs at two levels. At the more immediate level positioning concerns the moment-to-moment process in which students take up opportunities that they are presented with. Over the longer term, students become associated with particular ways of participating in classroom settings (these ideas are elaborated by scholars like Dorothy Holland and Stanton Wortham). This post will focus on identifying two complementary functions for portfolios helps them support both types of positioning.

Portfolios and Artifacts
Portfolios are collections of artifacts that students created. Artifacts support participation because they are where students apply what they are learning in class to something personally meaningful. In this way they make new meanings. In our various participatory assessment projects, artifacts have included

• the “Quests” that students complete and revise in Quest Atlantis’ Taiga world where they explain, for example, their hypothesis for why the fish in the Taiga river are in decline;
• the remixes of Moby Dick and Huck Finn that students in Becky Rupert’s class at Aurora Alternative High School create in their work with the participatory reading curricula that Jenna McWilliams is creating and refining.
• the various writing assignments that the English teachers in Monroe and Greene County have their students complete in both their introductory and advanced writing classes;
• the wikifolio entries that my students in my graduate classroom assessment course complete where they draft examples of different assessment items for a lesson in their own classrooms, and state which of the several item writing guidelines in the textbook they found most useful.

In each case, various activities scaffold the student learning as they create their artifacts and make new meanings in the process. As a caveat, this means that participatory assessment is not really much use in classrooms where students are not asked to create anything. More specifically, if your students are merely being asked to memorize associations and understand concepts in order to pass a test, stop reading now. Participatory assessment won’t help you. [I learned this the hard way trying to do participatory assessment with the Everyday Mathematics curriculum. Just do drill and practice. It works.]

Problematically Positioned Portfolios
Probably the most important aspect of participatory assessment has to do with the way portfolios are positioned in the classroom. We position them so they serve as a bridge between the communal activities of participatory classroom and the individual accountability associated with compulsory schooling. If portfolios are to serve as a bridge, they must be firmly anchored. On one side they must be anchored to the enactment of classroom activities that support students’ creation of worthwhile portfolios. On the other side they must be anchored to the broader accountability associated with any formal schooling.



To keep portfolio practices from falling apart (as they often do) it is crucial that they rest on these two anchors. If accountability is placed on the portfolio, the portfolio practice will collapse. In other words, don’t use the quality of the actual portfolio artifacts for accountability. Attaching consequences to the actual artifacts means that learners will expect precise specifications regarding those artifacts, and then demand exhausting feedback on whether the artifacts meet particular criteria. And if an instructor’s success is based on the quality of the artifacts, that instructor will comply. Such classrooms are defined by an incessant clamor from learners asking “Is this what you want???”

When portfolios are positioned this way (and they often are), they may or may not represent what students actually learned and are capable of. When positioned this way, the portfolio is more representative of of (a) the specificity of the guidelines, (b) their ability to follow those guidelines, and (3) the amount of feedback they get from the instructor. Accountability-oriented portfolios position disciplinary knowledge as something to be competitively displayed rather than something to be learned and shared, and portfolios position students as competitors rather the supporters. Perhaps most tragically, attaching consequences to artifacts positions instructors (awkwardly) as both piano tuners and gatekeepers. As many instructors (and ex-instructors) know, doing so generates massive amounts of work. This is why it seems that many portfolio-based teacher education programs rely so heavily on doctoral students and adjuncts who may or may not be qualified to teach courses. The more knowledgeable faculty members simply don’t have the time to help students with revision after revision of their artifacts as students struggle to create the perfect portfolio. This is the result of positioning portfolios for production.

Productive Positioning Within Portfolios
Portfolio are more useful when they are positioned to support reflection. Instead of grading the actual artifacts that students create, any accountability should be associated with student reflection on those artifacts. Rather than giving students guidelines for producing their artifact, students need guidelines for reflecting on how that artifact illustrates their use of the “big ideas” of the course. We call these relevant big ideas, or RBIs. The rubrics we provide students for their artifacts essentially ask them to explain how their artifact illustrates (a) the concept behind the RBI, (b) the consequences of the RBI for practice, and (c) what critiques others might have of this characterization of the RBI. For example:

• Students in my classroom assessment course never actually “submit” their wikifolios of example assessments. Rather, three times a semester they submit a reflection that asks them to explain how they applied the RBIs of the corresponding chapter.

• Students in Taiga world in Quest Atlantis submit their quests for review by the Park Ranger (actually their teacher but they don’t know that). But the quest instructions (the artifact guidelines) also include a separate reflection section that asks students to reflect on their artifact. The reflection prompts are designed to indirectly cue them what their quest was supposed to address.

• Students in Becky Rupert’s English class are provided a rubric for their remixes that ask them to explain how that artifact illustrates how an understanding of genre allows a remix to be more meaningful to particular audiences.

Assessing the resulting reflections positions portfolios, students, and teachers in ways that strongly support participation. For example, if the particular student’s artifact actually does not lend itself to applying the RBIs, my classroom assessment students can simply indicate that in their assignment. This is important for at least three reasons:

1. it allows full individualization for students and avoids a single ersatz assignment that is only half-meaningful to some students and mostly meaningless to the rest;
2. understanding if and how ideas from a course do not apply is a crucially important part of that expertise.
3. The reflection itself provides more valid evidence of learning, precisely because it can include very specific guidelines. We give students very specific guidelines asking them to reflect on the RBIs conceptually, consequentially, and critically.

For example, the mathematics teachers in the classroom assessment course are going to discover that it is very difficult to create portfolio assessments for their existing mathematical practices. Rather than forcing them to do so anyways (and giving them a good grade for an absurd example), they can instead reflect on what it is about mathematics that makes it so difficult, and gain some insights into how they might more readily incorporate project-based instruction into their classes. The actual guidelines for creating good portfolios are in the book when they need them; reflecting on those guidelines more generally will set them up to use them more effectively and meaningfully in the future.

Another huge advantage of this way of positioning portfolios is that it greatly eliminate a lot of the grading busywork and allows more broadly useful feedback. In the Quest Atlantis example, our research teacher Jake Summers of Binford Elementary discovered that whenever the reflections were well written and complete, the actual quest submission would also be well done. In the inevitable press for time, he just started looking at the artifacts. Similarly in my classroom assessment course, I will only look need to go back and look at the actual wikifolio entries when a reflection is incomplete or confusing. Given that the 30 students each have 8 entries, it is impossible to carefully review all 240 entries and provide meaningful feedback. Rather throughout the semester, each of the students have been getting feedback from their group members and from me (as they specifically request and as time permits). Because the artifacts are not graded, students understand the feedback they get as more formative than summative, and not as instructions for revision. While some of the groups in class are still getting the hang of it, many of the entries are getting eight or nine comments along with comments on comments. Because the entries are wikis it is simple for the originator go in and revise as appropriate. These students are starting to send me messages that, for me, suggest that the portfolio has indeed been positioned for participation: “Is this what you meant?” (emphasis added). This focus on meaning gets at the essence of participatory culture.

In a subsequent post, I will elaborate on how carefully positioning portfolios relative to (a) the enactment of classroom activities and (b) external accountability can further foster participation.

Participatory Assessment for Bridging the Void between Content and Participation.

Here at Re-Mediating Assessment, we share our ideas about educational practices, mostly as they relate to innovative assessment practices and mostly then as they relate to new media and technology. In this post, I respond to an email from a colleague about developing on-line versions of required courses in graduate-level teacher education courses.

My colleague and I are discussing how we ensure coverage of “content” in proposed courses that focuses more directly on “participation” in the actual educational practices. This void between participation (in meaningful practices) and content (as represented in textbooks, standards, and exams) is a central motivation behind Re-Mediating Assessment. So it seems worthwhile to expand my explanation of how participatory assessment can bridge this void and post it here.

To give a bit of context, note that the course requirements of teacher education programs are constantly debated and adjusted. From my perspective it is reasonable to assume that someone with a Master’s degree in Ed should have taken a course on educational assessment. But it also seems reasonable to have also had a course on, say, Child Development. But it simply may not be possible to require students to take both classes. Because both undergraduate and graduate teacher educator majors have numerous required content area courses (i.e., math, English, etc.), there are few slots left for other courses that most agree they need. So the departments that offer these other required courses have an obvious obligation to maintain accountability over the courses that they offer.

I have resisted teaching online because previous courseware tools were not designed to foster participation in the meaningful discourse that is what I think is so important to a good course. Without a classroom context for discourse (even conversations around a traditional lecture), students have few cues for what matters. Without those cues, assessment practices become paramount in communicating the instructor values. And this is a lot to ask of an assessment.

This is why, in my observation, online instruction heretofore has mostly consisted of two equally problematic alternatives. The first is the familiar on-line tools for pushing content out to students: “Here is the text, here are some resources, and here is a forum where you can post questions, and here is the exam schedule.” The instructors log on to the forums regularly and answer any questions, students take exams, and that is it. Sometimes these courses are augmented with papers and projects and perhaps with collaborative projects; hopefully students get feedback, and they might even use that feedback to learn more. But many many on-line course are essentially fancy test prep. My perceptions are certainly biased by my experiences back in the 90s in the early days of on-line instruction. The Econ faculty where I was working could not figure out why the students who took the online version of Econ 101 always got higher exam scores than the face-to-face (FTF) students, but almost always did far worse in the FTF Econ 201. This illustrates the problem with instruction that directly preparing students to pass formal exams. Formal exams are just proxies for prior learning, and framing course content entirely around tests (especially multiple choice ones) is just a terrible idea. Guessing which of four associations is least wrong is still an efficient way of reliably comparing what people know about a curriculum or a topic. But re-mediating course content to fit into this format makes it nearly useful for teaching.

The other extreme of on-line instruction is “project based” classes that focus almost entirely on developing a portfolio of course-related projects. These approaches seem particularly popular in teacher education programs. The problem with on-line portfolios is that the lack of FTF contact requires the specifications for the portfolios to be excruciatingly detailed. Much of the learning that occurs tends to be figuring out what the instructor wants in order to get a good grade. The most salient discourse in these classes often surrounds the question “Is this what you want?” These classes are usually extremely time-consuming to teach because the accountability associated with the artifacts leads students to demand, and instructors to provide, tons of detailed feedback on each iteration of the artifacts. So much so that the most qualified faculty can’t really afford to teach many of these courses. As such, these courses are often taught by graduate students and part-time faculty who may not be ideal for communicating the “Relevant Big Ideas” (RBIs, or what a learning scientist might call “formalisms") behind the assignments, and instead just focus on helping students create the highest quality artifacts. This creates a very real risk that students in these classes may or may not actually learn the underlying concepts, or may learn them in a way that they are so bound to the project that they can’t be used in other contexts. In my observation, such classes seldom feature formal examinations. Without careful attention, lots of really good feedback, and student use of feedback, students may come away from the class with a lovely portfolio and little else. Given the massive investment in e-Portfolios in e-learning platforms like Sakai, this issue demand careful attention. (I will ask my friend Larry Mikulecky in Indiana’s Department of Culture, Communication, and Language Education who I understand has been teaching non-exam online courses for years and has reportedly develops considerable evidence of student’s enduring understanding.)

A Practical Alternative
I am teaching on-line for the first time this summer. The course is P540, Cognition and Learning, a required course for many M. Ed programs. I am working like crazy to take full advantage of the new on-line resources for social networking that are now available in OnCourse, IU’s version of Sakai (an open-source collaborative learning environment designed for higher education). In doing so I am working hard to put into place an on-line alternative that balances participation and content. I also plan to use some of the lessons I am learning in my Educational Assessment course this Fall—which is partly what prompted that aforementioned conversation with my colleague. I want to put some of my ideas as they are unfolding in that class out there and seek input and feedback, including from my current students who are (so far) patiently hanging with me as I refine these practices as I go.

In particular I am working hard to incorporate the ideas about participatory culture that I have gained from working with Henry Jenkins and his team at Project New Media Literacies over the last year. Participatory assessment assumes that you can teach more "content" and gather more evidence that students “understand” that content by focusing more directly on participation and less directly on content. Theoretically, these ideas are framed by situative theories of cognition that say participation in social discourse is the most important thing to think about, and that individual cognition and individual behavior are “secondary” phenomena. These ideas come to me from three Jims: Greeno (whose theorizing has long shaped my work) Gee (who also deeply influences my thinking about cognition and assessment and whose MacArthur grant funded the aforementioned collaboration and indirectly supports this blog) and Pellegrino (with whom I did my doctoral studies of assessment, transfer, and validity with but who maintains an individual differences approach to cognition).

Per the curriculum committee that mandated a cognition and learning course for most masters degrees for teachers, my students are just completing ten tough chapters on memory, cognition, motivation, etc. I use Roger Bruning’s text because he make is quite clear and puts 5-7 “implications for teaching” at the end of each chapter. But it is a LOT of content for these students to learn, especially if I just have them read the chapters.

I break students up into domain groups (math science, etc.) and in those groups they go through the 5-7 implications for teaching. Each group must use the forum to generate a specific example of that implication, and then rank order the implications in terms of relevance and warrant those rankings and post them to the OnCourse wiki. The level of discourse in the student-generated forums around the content is tremendous. Then the lead group each week synthesizes the postings of all five groups to come up with a single list. I also have now asked them to do the same with “things worth being familiar with” in the chapter (essentially the bolded items and any highlighted research studies). What I particularly like about the discussions is the way that the discourse around agreeing that an implication or topic is less relevant actually leads to a pretty deep understanding of that implication or idea. This builds on ideas I have learned from my colleague Melissa Gresalfi about “consequential engagement.” By struggling to conclude that the implication is least likely to impact practice makes it more likely that they will remember that implication if they find themselves is a situation that makes it more relevant.

This participatory approach to content is complemented by four other aspects of my class. Illustrating my commitment to content, I include three formal exams that are timed and use traditional MC and short answer items. But I prioritize the content that the class has deemed most important, and don't even include the content they deem least important.

The second complement is the e-Portfolios each student has to post each week in OnCourse. Students have to select the one implication they think is most relevant, warrant the selection, exemplify and critique it, and then seek feedback on that post from their classmates. Again following Melissa’s lead, the e-Portfolio asks students for increasingly sophisticated engagement with the implication relative to their own teaching practice: procedural engagement (Basically explain the implication in your own words), conceptual engagement (give an example that illustrates what this implication means), consequential engagement (what are the consequence of this implication for your teaching practice, what should you do differently now that you understand this aspect of cognition?) and critical engagement (why might someone disagree with you and what would happen if you took this implication too far?). I require them to request feedback from their classmates. While this aspect of the new on-Course e-Portfolio tools is still quite buggy, I am persevering because the mere act of knowing that a peer audience is going to read it pushes them to engage more deeply. Going back to my earlier point, it is hard for me to find time to review and provide detailed feedback on 220 indivdiual submissions across the semester. When I do review them (students submit them for formal review after five submissions), I can just look at the feedback from other students and the students' own reflection on what they have learned for pretty clear evidence of consequential and critical engagement.

The third complement is the e-Portfolio that each student completes during the last five weeks of class. While each of the groups leads the class in the chapter associated with their domain (literacy, comprehension, writing, science and math), students will be building an e-portfolio in which they critique and refine at least two web-based instructional resources (educational videogames, webquests, the kind of stuff teachers increasingly are searching out and using in their classes). They select two or more of the implications from that chapter to critique the activities and provide suggestions for how it should be used (or if it should be avoided), along with one of the implications from the chapter on instructional technology, and one of the implications from the other chapters on memory and learning. If I have done my job right, I don’t need to prompt them to the consequential and critical engagement at this stage. This is because they should have developed what Melissa calls a “disposition” towards these important forms of engagement. All I have to do is include the requirement that they justify why each implication was selected, the feedback from their classmates, and their reflection on what they learned from feedback. It turns out the consequential and critical engagement is remarkably easy to recognize in discourse. That seems partly because it is so much more interesting and worthwhile to read than the more typical class discourse that is limited to procedural and conceptual engagement. Ultimately, that is the point.