Major Highlights of the 2013 Educational Assessment BOOC

by Tara Kelley and Dan Hickey

This post summarizes the high levels of engagement and learning that occurred in the Big Open Online Course (BOOC) on Educational Assessment in fall 2013.  The course will be offered again during summer 2014, starting May 13.  You can earn a certificate, digital badge, credit, or even just learn.  For more information and to register, visit here.

With a grant from Google, a few of us at Indiana University offered a BOOC (Big Open Online Course) on Educational Assessment. Google encouraged us to explore highly interactive kinds of of using its Course Builder platform.  The course scaled up innovations that Dan had previously refined in his small conventional versions of the online course.  This encouraged participants to interact with and discuss topics related to the practices, policies, and procedures of educational assessment. This post will discuss the major accomplishments of the fall 2013 Educational Assessment BOOC, which includes the following:

- High retention rates
- Substantial wikifolio posts
- High levels of engagement
- Good average exam scores
- Enthusiastic sharing of badges

As recent studies have confirmed, most of the massive "xMOOCs" offered by Coursera, Udacity, edX typically have very low persistence rates (i.e., single digits); even for students who persist and complete, social engagement declines over time (because the discussion forums become so unwieldy).  To confront these issues, the BOOC serves as an intermediate development stage between a typical college-level online course and a massive open online course.

This first BOOC was capped at 500 registrants and was designed around the five principles of a framework called Participatory Learning and Assessment. These principles are intended to efficiently foster larger amounts interactive engagement with the disciplinary knowledge of the course. The course proved to be quite successful, with five major accomplishments.

I. High retention rates

Table 1: Initial Registrants

The course began with 460 registrants who were initially organized into seventeen professional networking groups based on the information they provided when the registered (Table 1). Students are organized into these groups and are expected to interact weekly with peers inside and outside of their networking groups.

Over 160 registrants (35%) completed the first assignment, and 60 of them (37%) completed the course.  This is a big improvement in retention.  Eight students paid IU tuition and took the course for credit. They were required to complete the required and optional parts of all of the weekly assignments, and expected to more regularly with the peers, the teaching assistant, and the instructor.

II. Substantial wikifolio posts

Table 2. Words/Wikifolio by Week.

The course consisted of eleven weekly units. Each week, participants were asked to post a wikifolio where they applied the concepts of their course to personalized curricular context and aim. The average number of words per wikifolio post increased after the first few weeks of the course and then slowly decreased at the end of the course, but never to a point lower than the first week of the course.

The enrolled students (for credit) wrote substantially more than the completers (open non-credit students who completed the course), who wrote slightly more than posters (open students who did not complete the course (Table 3). Given that all of the wikifolios required students to make sense of topics in assessment around a personally meaningful context, this was a lot of disciplinary engagement just in posting the weekly wikifolios.

III. High levels of engagement

Table 3.  Engagement in Peer Endorsement, Comments, and Promotions

BOOC are designed to support high levels of interactive social engagement among peers and instructors. This was accomplished by including peer comments (in threads directly on the wikifolio), peer endorsement (at least three per week, for being complete) and peer promotion (only one per week, for being "exemplary").

While the number of posters each week declined gradually from 160 to 60, the level of social engagement for three kinds of interaction was stable or increased increased among posters each week. Table 3 shows that the open (non-credit) students average about 3 comments per week while  that for credit students averaged around 6 comments per week.  The open students average about 5 endorsements per week while the for credit students averaged about seven comments per week.  Most importantly, the crucial peer endorsement levels were quite high.  You can see that the for credit student were promoted more more often than the open students.  But overall about 70% of the posters engaged in peer promotion at first, and this increased to almost 90% by the last week.  In other word, only about 10% of participants elected to not endorse one peer for being exemplary by the end of the course.

IV. Good average exam scores

Table 4.  Average Exam Scores

In fall 2013, we originally stated that participants must score a 70% or higher on the exams in order to receive a badge for that part of the course (three parts: practices, principles, and policies). However, many participants had trouble getting the textbook and, therefore, did poorly on the exams, despite high engagement in the course. Other participants got very close to a score of 70%, but did not "pass." Despite the issues with the exam in the fall 2013 course, the average exam scores for each exam, including the final, were above 70%, with the students enrolled in the course for graduate level credit scoring, on average, higher than the open participants (Table 4).  In the summer 2014 coruse, there will be two tiers of badges, one that recognizes participation alone and one that recognize higher levels of engagement and passing exam scores.


V. Enthusiastic sharing of badges

Table 5.  Badge Shared Per Day.

Many students openly and enthusiastically shared their badges via social networking channels. Approximately 30 participants shared their badges each time the first three badges were awarded (these badges were the Practices, Principles, and Policies badges respectively). Fewer participants shared the final badge (Table 5).

In summary, our first BOOC got off to a good start.  We have refined many of the features to foster even more interaction for students and automated some things so it will be less work for us.  Consider joining us in summer 2015.