tag:blogger.com,1999:blog-47786089091985187762024-03-18T05:47:40.816-04:00re-mediating assessmentwherein we consider the possibilities for participatory approaches to assessment of learning with digital badges, online learning, open courses, MOOCs, AI, and beyondAnonymoushttp://www.blogger.com/profile/11170497669879668928noreply@blogger.comBlogger136125tag:blogger.com,1999:blog-4778608909198518776.post-73095541466067818412024-03-03T15:18:00.000-05:002024-03-03T15:18:07.698-05:00My Festschrifts for Randi Engle's Situative Design Principles<p style="text-align: center;"><b>By Daniel Hickey</b></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjmNWfTxAIVZCUFJM3kvofid8qSnV1IK3WfQzQ5xgWgTL7-wV_My0nrNimzNrU-g3WfaEL7SORar5ZeT1niQ6GbDKSAsEUjw1eUkDVtqTxkJsVWwojv3qQEWo7nmXRKXpoMqTTNaZoRfBLrcixbBfU5dqxYo1VTPED56jMM1eN-VTyYiv7yifEy5TBJFOk/s512/Randi-Engle.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="512" data-original-width="512" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjmNWfTxAIVZCUFJM3kvofid8qSnV1IK3WfQzQ5xgWgTL7-wV_My0nrNimzNrU-g3WfaEL7SORar5ZeT1niQ6GbDKSAsEUjw1eUkDVtqTxkJsVWwojv3qQEWo7nmXRKXpoMqTTNaZoRfBLrcixbBfU5dqxYo1VTPED56jMM1eN-VTyYiv7yifEy5TBJFOk/s320/Randi-Engle.png" width="320" /></a></div><br /><p style="text-align: left;">It has been over a decade since Randi Engle lost a two-year battle with pancreatic cancer at age 45. I did not know Randi very well. But I know some of her close friends and former students well and they all said she was a great friend and mentor. As elaborated in this <a href="https://senate.universityofcalifornia.edu/_files/inmemoriam/html/randiaengle.html">memorial</a>, Randi completed her Ph.D. at Stanford in 2000. She spent five years as a postdoc at the Learning Research and Development Center at the University of Pittsburg before joining UC Berkley.</p><p style="text-align: left;">Randi's design principles for <i><a href="https://www.jstor.org/stable/3233901">productive disciplinary engagement</a> </i>(PDE) and <i><a href="https://www.tandfonline.com/doi/pdf/10.1080/00461520.2012.695678">expansive framing</a> </i>are among the most useful and used to emerge from what I call "stridently situative" perspectives. Thanks to some super-helpful mid-career mentoring from James Gee, the second half of my career has been essentially continuing the work Randi started and moving it into the realm of online learning and social justice. My colleague Eric Freedman and I and our advisees are advancing multiple systematic reviews of the nearly 3000 publications that build on these two distinctive frameworks. </p><p style="text-align: left;">In the meantime, we have published four recent papers that build very directly on PDE and expansive framing. Unfortunately, two of them are inaccessible to most of the public and many scholars. I am starting to get a few requests because they are tuning up in search. So let me post them here with a bit of context.</p><blockquote style="border: none; margin: 0 0 0 40px; padding: 0px;"><p style="text-align: left;"><a href="https://scholarworks.iu.edu/dspace/items/70fe5181-cb1d-4040-92bb-23008eddf9cc"><span style="font-family: "Times New Roman", serif; font-size: 14.6667px; text-indent: -48px;">Hickey, D. T. (2022). </span><span style="font-family: "Times New Roman", serif; font-size: 14.6667px; text-indent: -48px;">Productive disciplinary engagement and expansive framing: The situative legacy of Randi Engle.</span><span style="font-family: "Times New Roman", serif; font-size: 14.6667px; text-indent: -48px;"> </span><span style="font-family: "Times New Roman", serif; font-size: 14.6667px; text-indent: -48px;">In M. McCaslin & T. Good (Eds.) </span><i style="font-family: "Times New Roman", serif; font-size: 14.6667px; text-indent: -48px;">Routledge Online Encyclopedia of Education. </i></a></p></blockquote><p>This is a flat-out festschrift in the style of European academic tribute. I wrote it in part to offer a very readable summary of the "five explanations" of expansive framing in <a href="https://www.tandfonline.com/doi/pdf/10.1080/00461520.2012.695678">Engle, Lam, Meyer, & Nix (2012)</a>. These explanations blew my mind in 2012 as I had already spent a few years incorporating PDE into my emerging framework for online learning. Expansive framing argues that "personally authentic" learning environments will result in <i>generative </i>learning that transfers readily and widely to new contexts. They indirectly explain why "professionally authentic" environments (as is common in problem-based learning) are likely quite alien and unwelcoming, especially to diverse learners.</p><blockquote style="border: none; margin: 0 0 0 40px; padding: 0px;"><p style="text-align: left;"><span style="font-family: "Times New Roman", serif; font-size: 11pt;"><a href="https://scholarworks.iu.edu/dspace/items/09981f95-af32-4168-ba21-e711dbe04ffc">Hickey, D. T., & Lam, D. (2023). Evolving and emerging perspectives on the transfer of learning. In A. O'Donnell & J. Reeve (Eds). <i>Oxford handbook of educational psychology</i>. Oxford University Press. </a> </span></p></blockquote><div><span style="font-size: 14.6667px; text-indent: -48px;">I have some regrets about contributing this chapter to this handbook. The promised external reviews never materialized and very few libraries subscribe to it. It took me two years to write and I broke new ground by extending Randi's ideas about expansive framing and transfer into social justice and equity. These ideas are going forward in several other areas I am quite excited about an emerging sociocultural consensus about learning transfer that takes into account race and marginalization</span></div><div><br /></div><blockquote style="border: none; margin: 0 0 0 40px; padding: 0px;"><div style="text-align: left;"><a href="https://edtechbooks.s3.us-west-2.amazonaws.com/pdfs/534/11962.pdf"><span style="font-size: 14.6667px; text-indent: -48px;">Hickey, D. T. (2023). Designing for productive disciplinary engagement and responsive assessment with situated cognition and expansive framing. In B. Allman, H. Leary, & R. Kimmons (Eds), </span><i style="font-size: 14.6667px; text-indent: -48px;">Theories to influence the future of learning design and technology. </i><span style="font-size: 14.6667px; text-indent: -48px;">(pp. 9-21)</span><i style="font-size: 14.6667px; text-indent: -48px;"> </i><span style="font-size: 14.6667px; text-indent: -48px;">Ed Tech Books</span></a></div></blockquote><p style="text-align: left;"><a name="_Hlk70598247" style="text-indent: -48px;"><span style="font-size: 11pt; line-height: 16.8667px;">This essay won the second place prize in the "Theory Spotlight" competition at the 2022 meeting of the Association for Educational Communication and Technology. I was really trying to drive home the point that situated cognition <span style="font-family: "Times New Roman", serif; font-size: 11pt;"><b>≠ </b>authentic learning. Thanks to the influence of Jan Herrington and Tom Reeves, many communities (particularly the AECT community) frame situated cognition as an argument for constructivist professionally authentic instruction and assessment. Randi's ideas about expansive framing argue otherwise.</span></span></a></p><blockquote style="border: none; margin: 0 0 0 40px; padding: 0px;"><p style="text-align: left;"><span style="font-size: 11pt; line-height: 16.8667px; text-indent: -48px;"><a href="https://drive.google.com/file/d/1I_HrK5YhlzqMhTAW1PxfkyAWuoSjZoRc/view?usp=drive_link">Hickey, D. T., Chartrand, G. T., & Andrews, C. D. (2020). Expansive framing as a pragmatic theory for instructional design. <i>Educational Technology Research and Development </i>[Special issue on <i>The crucial role of theoretical scholarship for learning design and technology</i>]<i> </i>68 (2), 751-782.</a></span></p></blockquote><p>This chapter was in a special issue of AECT's flagship journal edited by Rick West. We introduced Participatory Learning and Assessment to the AECT community. PLA embeds Randi's design principles into the multi-level assessment framework that emerged in the first half of my career. We translated the five theory-laden PLA design principles into 15 discreet steps for instructional designers who are likely grounded in cognitive theories of learning.</p>Daniel Hickeyhttp://www.blogger.com/profile/09885916528215868949noreply@blogger.com0tag:blogger.com,1999:blog-4778608909198518776.post-13782571577904504962023-10-16T16:23:00.000-04:002023-10-16T16:23:06.019-04:00Resources from Mary Rice and Colleagues on Special Ed and K-12 Virtual Learning<blockquote style="border: none; margin: 0 0 0 40px; padding: 0px;"><p style="text-align: center;"><b><span style="font-family: inherit;"> By Dan Hickey</span></b></p></blockquote><p><span style="font-family: inherit;"><b> </b>Our continuing efforts to explore and understand online and virtual learning takes us in many directions. Since the pandemic, our efforts have increasingly concerned students with special needs. Of course, most readers probably know about the extensive research and reporting about how poorly special needs students were served by "emergency remote teaching." (Here is a good <a href="https://crpe.org/wp-content/uploads/Special-Education-Impact-Brief_v3.pdf">updated summary from Fall 2022 </a>from ASU's Center for Reimagining Public Education).</span></p><p><span style="font-family: inherit;">As we put the pandemic behind us, I am quite fascinated by the <i>longer-term </i>impact of the pandemic, including stuff like legislative limits on virtual learning days (three in Indiana). I am certainly no expert in special ed. Still, I am intrigued by the growing evidence that the residual virtual infrastructure appears to be leading some districts to assign students with behavioral challenges to virtual learning. According to the trustworthy folks <a href="https://hechingerreport.org/the-newest-form-of-school-discipline-kicking-kids-out-of-class-and-into-virtual-learning/">at the Hechinger report</a>, these can be open-ended assignments that go unreported because the students are not technically suspended.</span></p><p><span style="font-family: inherit;">Before some upcoming workshops and presentations, I have been trying to learn more from the experts in virtual learning and special education. <a href="https://www.linkedin.com/in/drmaryrice/">Mary Rice</a> at the University of New Mexico and her colleagues have been extremely productive in recent years. Mary is a national expert in online learning, and I know her from her leadership of the Online Learning Special Interest Group at the American Educational Research Association. I was able to access most of her paywalled articles and she sent me several more and has invited me to share them with readers here. Here they are with brief annotations from the abstracts.</span></p><p></p><ul style="text-align: left;"><li><span style="font-family: inherit;"><a href="https://corp.smartbrief.com/original/2023/06/schools-step-up-to-accessibility">Rice, M., & Rose, R. (2023, June 28). How schools can step up to accessibility. SmartBrief for Educators (online)</a>. This is a great starting point for educators concerned with the accessibility of increasingly popular digital learning technologies.</span></li><li><span style="background-color: white; color: #222222;"><span style="font-family: inherit;"><a href="https://drive.google.com/file/d/1YS1VNMZ8cPu3IUbHM80GHWFbvLQOLW29/view?usp=drive_link">Rice, M. F., & Dunn, M. (2022). Inclusive online and distance education for learners with dis/abilities. <i>Distance Education</i>, <i>43</i>(4), 483-488</a>. This is the introduction to a special issue that Rice and Dunn edited, which is a great set of relevant resources.</span></span></li><li><span style="background-color: white; color: #222222;"><span style="font-family: inherit;"><a href="https://drive.google.com/file/d/1YgcXDUX4yUG45Kq_TKpk2tyjQJmqJWKW/view?usp=drive_link">Rice, M. F., & Ortiz, K. R. (2022). Parents of children with special educational needs shared work in fully online learning. <i>Journal of Research on Technology in Education</i>, 1-15.</a> This study explored the experiences of 18 parents of children with special needs in fully online schools.</span></span></li><li><span style="font-family: inherit;"><a href="https://drive.google.com/file/d/1X-rDqtTcSMfA7exYvIu0P6fcKBJk0d3z/view?usp=drive_link"><span style="color: #2b00fe;"><span style="background-color: white;">Rice, M. F., & Pazey, B. L. (2022). Ensuring IDEA implementation for students with disabilities across instructional modalities. </span><i>Management in Education</i></span></a><span style="background-color: white;"><a href="https://drive.google.com/file/d/1X-rDqtTcSMfA7exYvIu0P6fcKBJk0d3z/view?usp=drive_link"><span style="color: #2b00fe;">, 1-9</span></a><span style="color: #222222;">. This article links key elements of special education law to the challenges faced by schools and suggests implications for educational leaders who continue to support students in the pandemic and beyond.</span></span></span></li><li><span style="font-family: inherit;"><a href="https://drive.google.com/file/d/1Kf6lKFynf_26IlZrzP1-HeVenTYS_Fsx/view?usp=drive_link"><span style="color: #2b00fe;"><span style="background-color: white;">Dunn, M., & Rice, M. (2022). Practicing Teachers' Use of Online Tools for IEP Goal Writing and Instructional Support. </span><i style="background-color: white;">Journal of the American Academy of Special Education Professionals</i><span style="background-color: white;">, 49-</span></span></a><span style="background-color: white;"><a href="https://drive.google.com/file/d/1Kf6lKFynf_26IlZrzP1-HeVenTYS_Fsx/view?usp=drive_link"><span style="color: #2b00fe;">65</span></a><span style="color: #222222;">. </span></span>This study investigated special educators’ perspectives about an online toolkit that offers teachers example goals and objectives statements to use in service plans for students with special educational needs as well as teaching strategies and assessments.</span></li><li><span style="font-family: inherit;"><a href="https://drive.google.com/file/d/1ZGDg-_A12qzRsFg8ugM8W1YBwTVof4BH/view?usp=drive_link"><span style="color: #2b00fe;"><span style="background-color: white;">Rice, M. F., & Smith, E. (2023). Special education teachers’ entangled agencies, intersectional identities, and commitments to equity and inclusion. </span><i style="background-color: white;">Journal of Special Education Technology</i><span style="background-color: white;">, </span><i style="background-color: white;">38</i></span></a><span style="background-color: white;"><a href="https://drive.google.com/file/d/1ZGDg-_A12qzRsFg8ugM8W1YBwTVof4BH/view?usp=drive_link"><span style="color: #2b00fe;">(1), 37-49</span></a><span style="color: #222222;">. This was a fascinating narrative inquiry that extended these important issues into diverse educator identities. Part of a special issue worth exploring.</span></span></span></li><li><span style="background-color: white;"><span style="font-family: inherit;"><a href="https://drive.google.com/file/d/1-VCr5iYKI5WoyTazppCKZdjUTYgnGhis/view?usp=drive_link"><span style="color: #2b00fe;">Rice, M. F., Cantergiani, K., & Macias, D. (2023). A research-based conceptual framework for inclusive K-12 online, distance, and digital education. <i>Journal of Research on Technology in Education</i>, 1-17</span></a><span style="color: #222222;">. This article proposes a research-based conceptual framework for </span><i style="color: #222222;">Inclusive Online, Distance, and Digital Education </i><span style="color: #222222;">for K-12 settings.</span></span></span></li></ul><div><span style="color: #222222;">Hope you find these useful! Feel free to add more in comments if you like/</span></div><p></p>Daniel Hickeyhttp://www.blogger.com/profile/09885916528215868949noreply@blogger.com0tag:blogger.com,1999:blog-4778608909198518776.post-32309649690591329222023-04-04T11:59:00.001-04:002023-04-04T11:59:18.336-04:00Resources for Participatory Learning & Assessment in Online Learning <p style="text-align: center;"><b> By Daniel Hickey</b></p><p style="text-align: left;">As requests for this info trickle in, I am going to put it all up in this blog post. Participatory Learning and Assessment (PLA) is a situative framework for engaging and assessing online learners. We have been refining it since 2008 when I first started teaching online. It has been refined and studied with a series of internal grants and other grants from Google and the federal CARES Act to support this work. Many thanks to the funders, my doctoral advisees in IU Learning Sciences, and my colleagues at Indiana University High School. </p><p style="text-align: left;">PLA is intended to ensure "generative" learning that transfers readily and widely. It draws directly on the design principles from Randi Engle (1965-2012) for <i>productive disciplinary engagement </i>and <i>expansive framing. </i>It streamlines or automates formative and summative assessment and reduces inefficient private instructor-student interaction. This frees up instructors to focus more efficiently on public interaction with learners. PLA avoids dreary discussion forums and instead relies on instructor and student social annotations directly on course readings and student work.</p><p style="text-align: left;">Here are links to recent resources in a suggested order for newcomers who want to learn how to use this approach or its elements:</p><p style="text-align: left;"></p><ul style="text-align: left;"><li><a href="https://drive.google.com/file/d/1PFe7Yc72-GMHXakNAIvTLpB5G3_2bcmQ/view?usp=share_link">Hickey, D. T., Duncan,
J., Gaylord, C., Hitchcock, C., Itow, R., & Stephens, S. (2020).
gPortfolios: A pragmatic approach to online asynchronous assignments. <i style="font-size: 11pt; text-indent: -0.5in;">Information
and Learning Sciences,</i><span style="font-size: 11pt; text-indent: -0.5in;"> </span><i style="font-size: 11pt; text-indent: -0.5in;">121 </i></a><span style="font-size: 11pt; text-indent: -0.5in;"><a href="https://drive.google.com/file/d/1PFe7Yc72-GMHXakNAIvTLpB5G3_2bcmQ/view?usp=share_link">(5/6), 273-283</a>. This is a very introductory piece, written with IUHS educators and intended for educators with little grounding in learning theory</span></li><li><span style="font-size: 11pt; text-indent: -0.5in;"><a href="https://bit.ly/CanvasREVLA">Chartrand,
G. T., & Hickey, D. T. (2021) </a><i><a href="https://bit.ly/CanvasREVLA">Responsive engagement and virtual
learner assessment (REVLA)</a>. </i>This is an introductory open online self-paced course for educators
in the Canvas LMS for IU Expand. It has separate modules on engagement, formative assessment, and summative assessment It teaches social annotation and helps tame the complexity of Canvas' quiz engine. Requires an IU email address or a guest account obtained <a href="https://kb.iu.edu/d/alqt">here</a>.</span></li><li><a href="https://bit.ly/GoogleREVLA">Hickey, D.
T., & Harris, T. (2021, ongoing) </a><i><a href="https://bit.ly/GoogleREVLA">Responsive engagement and virtual
learner assessment (REVLA).</a> </i>A version of the same course but for Google Classroom. Requires a personal Gmail address (not a school address).<span style="font-size: 11pt; text-indent: -0.5in;"></span></li><li><span style="font-size: 11.0pt; line-height: 115%; mso-bidi-font-weight: bold;"><a href="https://drive.google.com/file/d/1FN-J9R0C0slGrCwDgGyriCGup4W8duBX/view?usp=share_link">Hickey, D. T., Chartrand, G. T., & Andrews, C. D. (2020). Expansive framing as a pragmatic theory for
instructional design. <i>Educational
Technology Research and Development, 68
</i>(2), 751-782</a>. Translates PLA design principles into steps. Written for instructional designers and others grounded in cognitive and constructivist learning theories.</span></li><li><a href="https://drive.google.com/file/d/1DLYthQqASZXFeLEh1cq5oXZn_ExKTg1Y/view?usp=share_link"><span style="font-size: 11pt; text-indent: -0.5in;">Hickey, D.
T. & Harris, T. (2021). </span><span style="font-size: 11pt; text-indent: -0.5in;">Re-imagining
online grading, assessment, and testing using situated cognition. </span><i style="font-size: 11pt; text-indent: -0.5in;">Distance
Education, 42 </i></a><span style="font-size: 11pt; text-indent: -0.5in;"><a href="https://drive.google.com/file/d/1DLYthQqASZXFeLEh1cq5oXZn_ExKTg1Y/view?usp=share_link">(2), 290-309.</a> Focuses on assessment in a special issue on assessment edited by Dianne Conrad. Translates assessment principles into straightforward steps.</span></li><li><a href="https://docs.google.com/document/d/1PEQUl_fiF68h53nrQNyWTGWj3D-psWUk/edit?usp=sharing&ouid=116943751791610529681&rtpof=true&sd=true">Hickey, D. T., & Lam, C.
(2023, April 15). Culturally-sustaining educational assessment: A new approach
to enduring challenges.</a> Paper presented at the Annual Meeting of the American
Educational Research Association. Extends PLA using ideas from Agarwal and Sengupta-Irving (2019).<span style="font-size: 11pt; text-indent: -0.5in;"></span></li></ul><div>I am happy to answer any questions about these at dthickey@indiana.edu</div><p></p>Daniel Hickeyhttp://www.blogger.com/profile/09885916528215868949noreply@blogger.com0tag:blogger.com,1999:blog-4778608909198518776.post-63253667454742728632023-04-03T00:46:00.003-04:002023-10-31T02:41:15.867-04:00Prompt Engineering in ChatGPT vs. Bard vs. Bing: Analysis #2, "Use Local Information"<p style="text-align: center;"> <b>By Daniel Hickey and Qianxu (Morgan) Luo</b></p><p><i>Pedagogical Prompt Engineering: </i>Analysis #2, Ask Students for “Local” Information”<o:p></o:p></p><p>This lengthy post explores another common recommendation
for educators trying to keep their students from using generative AI to thwart
their learning. We began exploring this issue in previous posts using <a href="http://remediatingassessment.blogspot.com/2023/03/prompt-engineering-in-chatgpt-vs-bard.html" target="_blank">ChatGPT &
Google’s <i>Bard</i> </a>and using <a href="http://remediatingassessment.blogspot.com/2023/03/prompt-engineeering-in-generative-ai.html" target="_blank">Microsoft’s New <i>Bing</i></a>.<i> </i>Those posts showed
how students might use what some are calling “prompt engineering” to get around
the recommendation to ask students for “specific” information (e.g., from scholarly
articles) that is not part of a platform’s large language model (LLM).<o:p></o:p></p><p> In this
post, we explore another recommendation that educators ask students about
“local” information outside of the LLM. Our near-term goal is what <a href="https://ed.stanford.edu/news/stanford-faculty-weigh-new-ai-chatbot-s-shake-learning-and-teaching" target="_blank">Victor Lee of Stanford</a> suggested we call “pedagogical prompt engineering.” This
means helping students learn how to refine prompts to actually <i>support </i>their
learning. Our ultimate goal is to use current theories and methods from the
learning sciences to maximize the value and minimize the harm of these powerful
new tools.<o:p></o:p></p><p><b>Recommendations for Using Local Information <o:p></o:p></b></p><p>In our initial review of the media accounts and
educator blogs we encountered quite a few recommendations to ask students to
use local information. In an article entitled <a href="https://www.techlearning.com/how-to/how-to-prevent-chatgpt-cheating" target="_blank"><i>How
to Prevent ChatGPT Cheating</i></a><i>,</i> one of Erik Ofgang’s five suggestions was “incorporate
authentic student experience and student connections into questions.” For
example… <o:p></o:p></p><p style="margin-left: 0.5in;"><span style="background: white; color: #333333;">education
college students might be asked to respond to a reading by talking about what
they’ve seen in their own education or in their work in the field (2/6/23)</span><o:p></o:p></p><p>Similarly, the <a href="https://englishwritingteacher.com/2023/03/20/what-does-to-write-mean-in-2023/" target="_blank">englishwritingteacher.com</a> asked<o:p></o:p></p><p style="margin-left: 0.5in;"><span style="background: white;">Do
teachers need to ask students to weave some highly local information—the
spelling bee yesterday at XYZ School, the performance of substitute teacher
Mrs. Poggi last week—into their writing so that AI has no way to access that
local information into its output, and so students are forced to write for
themselves? (March 20, 2023).<o:p></o:p></span></p><p><a href="https://www.linkedin.com/pulse/openai-chatgpt3-classroom-assessment-ashiedu-jude/?trk=public_post" target="_blank"><span style="background: white;">Ashiedu Jude </span></a><span style="background: white;">was more directive on LinkedIn:<o:p></o:p></span></p><p class="reader-text-blockparagraph" style="background: white; margin-left: 0.5in;">Classroom activity data can't be
accessed by ChatGPT, no student can prompt the AI to generate local
information, which becomes the data pool for teachers' assessment of students.
There'll be no need for lengthy text-based only term papers, essays, or
presentations without major references to local data or information not easily
found online. Your students have all access to ChatGPT, yet they'll be forced
to rely on original ideas from localized information and activities (1/23/23).<o:p></o:p></p><p>We acknowledge that these recommendations are
alongside many others in these posts and elsewhere. But here, we focus on the recommendation to require local information so we can contrast “problematic prompt
engineering” across the three widely available platforms and speculate about
each for pedagogical prompt engineering. As many students have figured out (or
learned on Reddit), they can bypass this recommendation by engineering prompts
to summarize local events that they observed or otherwise learned about. <o:p></o:p></p><p><b>Comparing “Shallow” Prompts in Three Platforms<o:p></o:p></b></p><p>First, we asked all three platforms a shallow question drawing on what Ofgang characterized as “<a href="https://www.techlearning.com/how-to/how-to-prevent-chatgpt-cheating">lower
order thinking skills, like being able to recall information, summarize
information, explain information.</a>” But we asked
about relatively sophisticated knowledge, in this case about learning theory:<o:p></o:p></p><p style="margin-left: 0.5in;"><b><span style="background: rgb(243, 246, 252); color: #1f1f1f; font-family: arial; font-size: 10pt;">Our initial prompt: </span></b><span face=""Calibri",sans-serif" style="background: rgb(243, 246, 252); color: #1f1f1f; font-size: 10pt;"><span style="font-family: arial;">Describe the difference between cognitive-associationist,
socio-constructivist, and sociocultural approaches to education</span><o:p></o:p></span></p><p>Elaborations on this question are a mainstay of
teacher education, educational psychology, and learning sciences courses. This
includes Dan’s <i>Learning and Cognition</i> course we explored in an <a href="http://remediatingassessment.blogspot.com/2023/01/can-chatgpt-hack-my-graduate-course-on.html">earlier
post</a>. We first compared how each platform
answered this straightforward question. <o:p></o:p></p><div style="background: rgb(247, 247, 248); border: 1pt solid rgb(217, 217, 227); margin-left: 0.5in; margin-right: 0in; mso-border-alt: solid #D9D9E3 .25pt; mso-element: para-border-div; padding: 0in;">
<p style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: none; margin: 0in 0in 15pt; padding: 0in;"><span style="font-family: arial;"><b><span face="Calibri, sans-serif" style="font-size: 10pt;">ChatGPT’s Response: </span></b><span face=""Calibri",sans-serif" style="color: #374151; font-size: 10pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">The cognitive-associationist, socio-constructivist, and
sociocultural approaches to education are three different perspectives that have
different assumptions about how learning occurs and how best to facilitate it.<o:p></o:p></span></span></p>
</div><div style="background: rgb(247, 247, 248); border: 1pt solid rgb(217, 217, 227); margin-left: 0.75in; margin-right: 0in; mso-border-alt: solid #D9D9E3 .25pt; mso-element: para-border-div; padding: 0in 0in 0in 5pt;">
<p style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: none; margin: 0in 0in 0in 0.25in; padding: 0in; text-indent: -0.25in;"><span style="font-family: arial;"><!--[if !supportLists]--><span face=""Calibri",sans-serif" style="color: #374151; font-size: 10pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: Calibri; mso-fareast-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">1.<span style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-size: 7pt; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal;"> </span></span><!--[endif]--><span face=""Calibri",sans-serif" style="color: #374151; font-size: 10pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">Cognitive-Associationist Approach: This approach to education is
rooted in behaviorism and cognitive psychology. It emphasizes the role of
reinforcement and punishment in learning and focuses on observable behaviors.
It suggests that learning is a process of acquiring new associations between
stimuli and responses, and that repetition and practice are important for
learning. In this approach, the teacher plays a central role in transmitting
knowledge to the learner.<o:p></o:p></span></span></p>
<p style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: none; margin: 0in 0in 0in 0.25in; padding: 0in; text-indent: -0.25in;"><span style="font-family: arial;"><!--[if !supportLists]--><span face=""Calibri",sans-serif" style="color: #374151; font-size: 10pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: Calibri; mso-fareast-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">2.<span style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-size: 7pt; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal;"> </span></span><!--[endif]--><span face=""Calibri",sans-serif" style="color: #374151; font-size: 10pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">Socio-Constructivist Approach: This approach to education is
based on the work of Jean Piaget and Lev Vygotsky. It emphasizes the importance
of social interaction and collaboration in learning. It suggests that learning
is a process of constructing new knowledge through interaction with the
environment and other people. In this approach, the teacher serves as a
facilitator of learning, helping students to construct their own understanding
of the world.<o:p></o:p></span></span></p>
<p style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: none; margin: 0in 0in 0in 0.25in; padding: 0in; text-indent: -0.25in;"><span style="font-family: arial;"><!--[if !supportLists]--><span face=""Calibri",sans-serif" style="color: #374151; font-size: 10pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: Calibri; mso-fareast-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">3.<span style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-size: 7pt; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal;"> </span></span><!--[endif]--><span face=""Calibri",sans-serif" style="color: #374151; font-size: 10pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">Sociocultural Approach: This approach to education emphasizes
the cultural and social context in which learning takes place. It suggests that
learning is a process of acquiring cultural practices and ways of thinking, and
that language and social interaction play a central role in this process. In
this approach, the teacher is seen as a mediator of learning, helping students
to connect with the cultural practices and ways of thinking of the community.<o:p></o:p></span></span></p>
</div><div style="background: rgb(247, 247, 248); border: 1pt solid rgb(217, 217, 227); margin-left: 0.5in; margin-right: 0in; mso-border-alt: solid #D9D9E3 .25pt; mso-element: para-border-div; padding: 0in;">
<p style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: none; margin: 15pt 0in 0in; padding: 0in;"><span style="color: #374151; font-family: arial; font-size: 10pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">Overall, the
cognitive-associationist approach emphasizes the role of the individual
learner, the socio-constructivist approach emphasizes the role of social
interaction, and the sociocultural approach emphasizes the role of cultural
context in learning.<o:p></o:p></span></p>
</div><p style="background: white; margin-bottom: 12.0pt; margin-left: 1.0in; margin-right: 0in; margin-top: 0in; margin: 0in 0in 12pt 1in;"><b><span style="font-family: arial; font-size: 10pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;"> </span></b></p><p style="background: white; margin-bottom: 12.0pt; margin-left: .5in; margin-right: 0in; margin-top: 0in; margin: 0in 0in 12pt 0.5in;"><span style="font-family: arial;"><b><span face="Calibri, sans-serif" style="font-size: 10pt;">Bard’s Response: </span></b><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 10pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">Cognitive-associationist, socio-constructivist, and
sociocultural approaches to education are three different ways of thinking
about how people learn. Cognitive-associationist approaches focus on how people
learn by making associations between different ideas. Constructivist approaches
focus on how people learn by constructing their own knowledge through
experience. Sociocultural approaches focus on how people learn in social
settings, through interactions with other people.<o:p></o:p></span></span></p><p class="MsoNormal" style="background: white; line-height: normal; margin-bottom: 12.0pt; margin-left: .5in; margin-right: 0in; margin-top: 12.0pt; margin: 12pt 0in 12pt 0.5in;"><span style="color: #1f1f1f; font-family: arial; font-size: 10pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;">Here are some key
differences between cognitive-associationist, socio-constructivist, and
sociocultural approaches to education:<o:p></o:p></span></p><p class="MsoNormal" style="background: white; line-height: normal; margin-bottom: 7.5pt; margin-left: 1in; mso-list: l8 level1 lfo1; mso-margin-top-alt: auto; tab-stops: list 1.0in; text-indent: -0.25in;"><span style="font-family: arial;"><!--[if !supportLists]--><span style="color: #1f1f1f; font-size: 10pt; mso-bidi-font-family: Symbol; mso-fareast-font-family: Symbol;">·<span style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-size: 7pt; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal;">
</span></span><!--[endif]--><b><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 10pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;">Goals:</span></b><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 10pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;"> Cognitive-associationist
approaches aim to help students learn facts and information. Constructivist
approaches aim to help students learn complex concepts and to think critically.
Sociocultural approaches aim to help students learn both facts and information
and complex concepts, and to think critically.<o:p></o:p></span></span></p><p class="MsoNormal" style="background: white; line-height: normal; margin-bottom: 7.5pt; margin-left: 1in; mso-list: l8 level1 lfo1; mso-margin-top-alt: auto; tab-stops: list 1.0in; text-indent: -0.25in;"><span style="font-family: arial;"><!--[if !supportLists]--><span style="color: #1f1f1f; font-size: 10pt; mso-bidi-font-family: Symbol; mso-fareast-font-family: Symbol;">·<span style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-size: 7pt; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal;">
</span></span><!--[endif]--><b><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 10pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;">Focus:</span></b><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 10pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;"> Cognitive-associationist
approaches focus on the individual learner. Constructivist approaches focus on
the learner's interaction with the environment. Sociocultural approaches focus
on the learner's interaction with other people.<o:p></o:p></span></span></p><p class="MsoNormal" style="background: white; line-height: normal; margin-bottom: 7.5pt; margin-left: 1in; mso-list: l8 level1 lfo1; mso-margin-top-alt: auto; tab-stops: list 1.0in; text-indent: -0.25in;"><span style="font-family: arial;"><!--[if !supportLists]--><span style="color: #1f1f1f; font-size: 10pt; mso-bidi-font-family: Symbol; mso-fareast-font-family: Symbol;">·<span style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-size: 7pt; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal;">
</span></span><!--[endif]--><b><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 10pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;">Methodologies:</span></b><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 10pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;"> Cognitive-associationist approaches often use traditional
methods of instruction, such as lectures and textbooks. Constructivist approaches
often use active learning methods, such as group work and problem solving.
Sociocultural approaches often use methods that combine elements of both
cognitive-associationist and constructivist approaches.<o:p></o:p></span></span></p><p class="MsoNormal" style="background: white; line-height: normal; margin-bottom: 12.0pt; margin-left: .5in; margin-right: 0in; margin-top: 12.0pt; margin: 12pt 0in 12pt 0.5in;"><span style="color: #1f1f1f; font-family: arial; font-size: 10pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;">Each approach has
its own strengths and weaknesses. Cognitive-associationist approaches can be
effective for teaching facts and information, but they can be less effective
for teaching complex concepts or for teaching people to think critically.
Constructivist approaches can be effective for teaching complex concepts and
for teaching people to think critically, but they can be less effective for
teaching facts and information. Sociocultural approaches can be effective for
teaching both facts and information and for teaching complex concepts, and for
teaching people to think critically, but they can be less efficient than other
approaches.<o:p></o:p></span></p><p class="MsoNormal" style="background: white; line-height: normal; margin-bottom: 12.0pt; margin-left: .5in; margin-right: 0in; margin-top: 12.0pt; margin: 12pt 0in 12pt 0.5in;"><span style="color: #1f1f1f; font-family: arial; font-size: 10pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;">The best approach to
education is likely to vary depending on the subject matter, the students'
learning styles, and the teacher's preferences.<o:p></o:p></span></p><p style="margin-bottom: 0in; margin-left: .5in; margin-right: 0in; margin-top: 0in; margin: 0in 0in 0in 0.5in;"><span style="font-family: arial;"><b><span face=""Calibri",sans-serif" style="font-size: 10pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">New
Bing’s response: </span></b><span face=""Calibri",sans-serif" style="color: #111111; font-size: 10pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">Cognitive-associationist, socio-constructivist, and
sociocultural approaches to education are all different theories of learning.
Cognitive-associationist theory is based on the idea that learning is a process
of forming associations between stimuli and responses. In contrast,
socio-constructivist theory emphasizes the importance of social interaction in
learning. It suggests that learning is a process of constructing knowledge
through social interaction with others. Sociocultural theory is similar to
socio-constructivist theory but places more emphasis on the role of culture in
learning. </span><span face=""Calibri",sans-serif" style="font-size: 10pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;"><a href="https://gsi.berkeley.edu/gsi-guide-contents/learning-theory-research/social-constructivism/" target="_blank">It suggests that learning is a process of acquiring cultural
tools and practices through social interaction with others</a><a href="https://gsi.berkeley.edu/gsi-guide-contents/learning-theory-research/social-constructivism/" target="_blank"><b><sup>1</sup></b></a><a href="https://www.verywellmind.com/what-is-sociocultural-theory-2795088" target="_blank"><b><sup>2</sup></b></a><a href="https://www.europeanpublisher.com/en/article/10.15405/ejsbs.301" target="_blank"><b><sup>3</sup></b></a><span style="color: #111111;">.<o:p></o:p></span></span></span></p><p style="margin-bottom: 0in; margin-left: .5in; margin-right: 0in; margin-top: 7.5pt; margin: 7.5pt 0in 0in 0.5in;"><span style="color: #111111; font-family: arial; font-size: 10pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">I hope this helps! Let me know if you have any other questions.<o:p></o:p></span></p><p style="margin-bottom: 0in; margin-left: 1.0in; margin-right: 0in; margin-top: 7.5pt; margin: 7.5pt 0in 0in 1in; mso-list: l5 level1 lfo2; text-indent: -0.25in;"><span style="font-family: arial;"><!--[if !supportLists]--><span face=""Calibri",sans-serif" style="color: #111111; font-size: 10pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: Calibri; mso-fareast-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">1.<span style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-size: 7pt; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal;">
</span></span><!--[endif]--></span><span face=""Calibri",sans-serif" style="font-size: 10pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;"><span style="font-family: arial;"><a href="https://gsi.berkeley.edu/gsi-guide-contents/learning-theory-research/social-constructivism/">gsi.berkely.edu</a><span style="color: #111111;"> 2. </span><a href="https://www.verywellmind.com/what-is-sociocultural-theory-2795088">verywellmind.com</a><span style="color: #111111;"> 3. </span><a href="https://www.europeanpublisher.com/en/article/10.15405/ejsbs.301">europeanpublisher.com</a></span><span style="color: #111111;"><o:p></o:p></span></span></p><p style="background: white; margin-bottom: 12.0pt; margin-left: .5in; margin-right: 0in; margin-top: 0in; margin: 0in 0in 12pt 0.5in;"><o:p> </o:p></p><p>The most obvious difference is that Bing clearly searched the web for the
information and provided hyperlinks to the sources. Bing also provided a much shorter version in
the initial response. Asking Bing to
“add more detailed” broke the response out into three (similar) paragraphs and added a one-sentence example to each (e.g., “An example of this approach is
rote memorization”). We also see that Bard structured the response differently
and (arguably) more creatively by contrasting the three theories in terms of
goals, focus, and methodologies.<o:p></o:p></p><p>All three responses show that each platform generated a fairly-sophisticated
response. In fact, an undergraduate might request a response “like a typical
undergraduate” or select one of the simpler drafts from the three offered by
Bing.<o:p></o:p></p><p><b>Engineering the Prompt with Local Information<o:p></o:p></b></p><p>For our local information prompt, we imagined that an instructor asked students
to apply these theories “to a classroom they just visited.” We saw versions of this as a “revised prompt” proposed by
several university teaching and learning centers. So we imagined that students
visited a classroom where the teacher was delivering a traditional
direct-instruction phonics lesson: <o:p></o:p></p><p style="margin-left: 0.5in;"><b><span style="color: #343541; font-family: arial; font-size: x-small; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">Our engineered prompt: </span></b><span face=""Calibri",sans-serif" style="color: #343541; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;"><span style="font-family: arial; font-size: x-small;">Imagine that a second-grade teacher just delivered a
direct-instruction phonics lesson. Explain what happened using
cognitive-associationist, socio-constructivist, and sociocultural theories of
learning.</span><o:p style="font-size: 10pt;"></o:p></span></p><p><span style="color: #343541;">Following are the responses to this engineered
prompt from each platform.</span><b><o:p></o:p></b></p><div style="background: rgb(247, 247, 248); border: 1pt solid rgb(217, 217, 227); margin-left: 0.5in; margin-right: 0in; mso-border-alt: solid #D9D9E3 .25pt; mso-element: para-border-div; padding: 0in 0in 0in 5pt;">
<p style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: none; margin: 0in 0in 0in 0.25in; padding: 0in; text-indent: -0.25in;"><span style="font-family: arial; font-size: x-small;"><!--[if !supportLists]--><span face=""Calibri",sans-serif" style="color: #374151; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-color-alt: windowtext; mso-fareast-font-family: Calibri; mso-fareast-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">1.<span style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal;"> </span></span><b><span face="Calibri, sans-serif">ChatGPT’s
response: </span></b><span face=""Calibri",sans-serif" style="color: #374151; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">Cognitive-Associationist Theory: According to the
cognitive-associationist theory of learning, the second-grade teacher's
direct-instruction phonics lesson would focus on teaching students to associate
specific letter sounds with corresponding phonemes. The teacher would use a
variety of techniques, such as repetition, drill and practice, and positive
reinforcement to help students learn and reinforce these associations. The
cognitive-associationist theory emphasizes the role of the individual learner
and focuses on observable behaviors.<o:p></o:p></span></span></p>
<p style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: none; margin: 0in 0in 0in 0.25in; padding: 0in; text-indent: -0.25in;"><span style="font-family: arial; font-size: x-small;"><!--[if !supportLists]--><span face=""Calibri",sans-serif" style="color: #374151; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: Calibri; mso-fareast-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">2.<span style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal;"> <b> </b></span></span><span face=""Calibri",sans-serif" style="color: #374151; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;"><b>Socio-Constructivist Theory:</b> From the socio-constructivist
perspective, the second-grade teacher's direct-instruction phonics lesson would
be an opportunity for students to construct new knowledge through social
interaction and collaboration with their peers. The teacher might encourage
students to work together in small groups, engage in discussions about sounds
and words, and participate in hands-on activities that allow them to experiment
with phonics concepts. The socio-constructivist theory emphasizes the
importance of social interaction and the role of the teacher as a facilitator
of learning.<o:p></o:p></span></span></p>
<p style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: none; margin: 0in 0in 0in 0.25in; padding: 0in; text-indent: -0.25in;"><span style="font-family: arial; font-size: x-small;"><!--[if !supportLists]--><span face=""Calibri",sans-serif" style="color: #374151; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: Calibri; mso-fareast-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">3.<span style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal;"> </span></span><span face=""Calibri",sans-serif" style="color: #374151; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;"><b>Sociocultural Theory: </b>The sociocultural theory of learning
emphasizes the role of cultural context and language in learning. From this
perspective, the second-grade teacher's direct-instruction phonics lesson would
be situated within a broader cultural and linguistic context. The teacher might
draw on examples from diverse cultures and encourage students to think about
the ways in which different languages and dialects represent sounds and words.
The teacher might also emphasize the importance of literacy in society and how
it connects to broader cultural practices. The sociocultural theory emphasizes
the role of the teacher as a mediator of learning, helping students to connect
with the cultural practices and ways of thinking of the community.<o:p></o:p></span></span></p>
</div><p style="background: white; margin-bottom: 12.0pt; margin-left: .5in; margin-right: 0in; margin-top: 0in; margin: 0in 0in 12pt 0.5in;"><b><span style="font-family: arial; font-size: x-small; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;"> </span></b></p><p style="background: white; margin-bottom: 12.0pt; margin-left: .5in; margin-right: 0in; margin-top: 0in; margin: 0in 0in 12pt 0.5in;"><span style="font-family: arial; font-size: x-small;"><b><span face="Calibri, sans-serif">Bard’s response: </span></b><span face=""Calibri",sans-serif" style="color: #1f1f1f; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">Sure. Here is how a direct-instruction phonics lesson can be explained
using cognitive-associationist, socio-constructivist, and sociocultural
theories of learning:<o:p></o:p></span></span></p><p style="background: white; margin-bottom: 12.0pt; margin-left: .5in; margin-right: 0in; margin-top: 12.0pt; margin: 12pt 0in 12pt 0.5in;"><span style="font-family: arial; font-size: x-small;"><strong><span face=""Calibri",sans-serif" style="color: #1f1f1f; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">Cognitive-associationist theory:</span></strong><span face=""Calibri",sans-serif" style="color: #1f1f1f; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;"><o:p></o:p></span></span></p><p style="background: white; margin-bottom: 12.0pt; margin-left: .5in; margin-right: 0in; margin-top: 12.0pt; margin: 12pt 0in 12pt 0.5in;"><span style="color: #1f1f1f; font-family: arial; font-size: x-small; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">In a cognitive-associationist theory of learning,
students learn by making associations between different ideas. In a direct-instruction
phonics lesson, students learn the association between the letter sounds and
the corresponding sounds in words. They also learn the association between the
different letters and their corresponding sounds.<o:p></o:p></span></p><p style="background: white; margin-bottom: 12.0pt; margin-left: .5in; margin-right: 0in; margin-top: 12.0pt; margin: 12pt 0in 12pt 0.5in;"><span style="font-family: arial; font-size: x-small;"><strong><span face=""Calibri",sans-serif" style="color: #1f1f1f; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">Socio-constructivist theory:</span></strong><span face=""Calibri",sans-serif" style="color: #1f1f1f; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;"><o:p></o:p></span></span></p><p style="background: white; margin-bottom: 12.0pt; margin-left: .5in; margin-right: 0in; margin-top: 12.0pt; margin: 12pt 0in 12pt 0.5in;"><span style="color: #1f1f1f; font-family: arial; font-size: x-small; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">In a socio-constructivist theory of learning,
students learn by constructing their own knowledge through experience. In a
direct-instruction phonics lesson, students construct their own knowledge of
the letter sounds and the corresponding sounds in words by actively
participating in the lesson. They also construct their own knowledge of the
different letters and their corresponding sounds by actively listening to the
teacher and by practicing the sounds.<o:p></o:p></span></p><p style="background: white; margin-bottom: 12.0pt; margin-left: .5in; margin-right: 0in; margin-top: 12.0pt; margin: 12pt 0in 12pt 0.5in;"><span style="font-family: arial; font-size: x-small;"><strong><span face=""Calibri",sans-serif" style="color: #1f1f1f; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">Sociocultural theory:</span></strong><span face=""Calibri",sans-serif" style="color: #1f1f1f; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;"><o:p></o:p></span></span></p><p style="background: white; margin-bottom: 12.0pt; margin-left: .5in; margin-right: 0in; margin-top: 12.0pt; margin: 12pt 0in 12pt 0.5in;"><span style="color: #1f1f1f; font-family: arial; font-size: x-small; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">In a sociocultural theory of learning, students
learn in social settings, through interactions with other people. In a
direct-instruction phonics lesson, students learn in a social setting, through
interactions with the teacher and with other students. They also learn through
interactions with the materials in the lesson, such as the letters and the
sounds.<o:p></o:p></span></p><p style="background: white; margin-bottom: 12.0pt; margin-left: .5in; margin-right: 0in; margin-top: 12.0pt; margin: 12pt 0in 12pt 0.5in;"><span style="color: #1f1f1f; font-family: arial; font-size: x-small; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">Each of these theories of learning can provide
insights into how students learn in a direct-instruction phonics lesson.
Cognitive-associationist theory can help us understand how students learn the
association between the letter sounds and the corresponding sounds in words.
Socio-constructivist theory can help us understand how students construct their
own knowledge of the letter sounds and the corresponding sounds in words.
Sociocultural theory can help us understand how students learn in a social
setting, through interactions with the teacher and with other students.<o:p></o:p></span></p><p style="margin-bottom: 0in; margin-left: .5in; margin-right: 0in; margin-top: 0in; margin: 0in 0in 0in 0.5in;"><span style="font-family: arial; font-size: x-small;"><b><span face=""Calibri",sans-serif" style="mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">New
Bing’s response:” </span></b><span face=""Calibri",sans-serif" style="color: #111111; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">Sure! Here’s how each theory might explain what
happened in the lesson:<o:p></o:p></span></span></p><p style="margin-bottom: 0in; margin-left: .5in; margin-right: 0in; margin-top: 0in; margin: 0in 0in 0in 0.5in;"><span style="color: #111111; font-family: arial; font-size: x-small; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;"> </span></p><p style="margin-bottom: 0in; margin-left: 1.0in; margin-right: 0in; margin-top: 0in; margin: 0in 0in 0in 1in; mso-list: l7 level1 lfo4; tab-stops: list 1.0in; text-indent: -0.25in;"><span style="font-family: arial; font-size: x-small;"><!--[if !supportLists]--><span style="color: #111111; mso-bidi-font-family: Symbol; mso-fareast-font-family: Symbol;">·<span style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal;"> </span></span><strong><span face=""Calibri",sans-serif" style="color: #111111; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">Cognitive-associationist
theory</span></strong><span face=""Calibri",sans-serif" style="color: #111111; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">: In this theory, the teacher would have used
repetition and reinforcement to help students form associations between sounds
and letters. For example, the teacher might have had students repeat a sound
several times while pointing to the corresponding letter.<o:p></o:p></span></span></p><p style="margin-bottom: 0in; margin-left: 1.0in; margin-right: 0in; margin-top: 0in; margin: 0in 0in 0in 1in; mso-list: l7 level1 lfo4; tab-stops: list 1.0in; text-indent: -0.25in;"><span style="font-family: arial; font-size: x-small;"><!--[if !supportLists]--><span style="color: #111111; mso-bidi-font-family: Symbol; mso-fareast-font-family: Symbol;">·<span style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal;"> </span></span><strong><span face=""Calibri",sans-serif" style="color: #111111; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">Socio-constructivist
theory</span></strong><span face=""Calibri",sans-serif" style="color: #111111; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">: In this theory, the teacher would have encouraged
students to construct their own knowledge through experiences and reflection.
For example, the teacher might have had students work in pairs to come up with
words that contain a particular sound.<o:p></o:p></span></span></p><p style="margin-bottom: 0in; margin-left: 1.0in; margin-right: 0in; margin-top: 0in; margin: 0in 0in 0in 1in; mso-list: l7 level1 lfo4; tab-stops: list 1.0in; text-indent: -0.25in;"><span style="font-family: arial; font-size: x-small;"><!--[if !supportLists]--><span style="color: #111111; mso-bidi-font-family: Symbol; mso-fareast-font-family: Symbol;">·<span style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal;"> </span></span><strong><span face=""Calibri",sans-serif" style="color: #111111; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">Sociocultural theory</span></strong><span face=""Calibri",sans-serif" style="color: #111111; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">: In this theory, the teacher would have emphasized the
importance of cultural context and social interaction in learning. For example,
the teacher might have had students work in small groups to practice reading
words that contain a particular sound.<o:p></o:p></span></span></p><p style="margin-bottom: 0in; margin-left: .5in; margin-right: 0in; margin-top: 7.5pt; margin: 7.5pt 0in 0in 0.5in;"><span face=""Calibri",sans-serif" style="color: #111111; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;"><span style="font-family: arial; font-size: x-small;">I hope this helps! Let me know if you have any other questions.</span><o:p style="font-size: 10pt;"></o:p></span></p><p>Notably, ChatGPT and Bing mostly described different <i>approaches, </i>rather than using the different theories to
explain the practice. This conflation of <i>theories </i>of learning and <i>practices </i>of teaching is quite common with undergraduate and master’s level
teacher education students. In contrast Bard indeed attempted to explain this practice
from each of the the three perspectives. Notably, it did so without acknowledging
that practice is most consistent with cognitive associationist theories (and
less consistent with socio-constructivist and situative/sociocultural theories).
We tried this test with other practices (problem-based learning and
collaborative science investigation) and got similar responses. <o:p></o:p></p><p>These examples showed that all
three platforms “learned” at the intersection of learning theory and educational
practice. But they learned in different ways. Significantly, these differences
are are remarkably akin to the sorts of differences in the ways that real
students learn. We contend that this ability to learn (beyond what was
presumably contained in their LLMs or the web resources accessed) is crucial for
thwarting <i>and </i>supporting student learning using generative
AI<o:p></o:p></p><p><b>So What Are Educators to Do?<o:p></o:p></b></p><p>First, this example demonstrates
the value of perhaps the most common recommendation for educators: <i>Test your assessments and assignments with
generative AI. </i>This post further reminds
educators that many students will engineer such “knowledge-rich” prompts, and
some will do so in ways that thwart learning. We further remind educators that
careful research (as summarized in James Lang’s <i><a href="https://www.hup.harvard.edu/catalog.php?isbn=9780674724631">Cheating
Lessons</a></i>) has shown that many
students will cheat if they assume (a) their classmates are doing so and (b) they
will not be caught. Recent survey evidence showing <a href="https://www.dailymail.co.uk/sciencetech/article-11899475/Half-students-using-ChatGPT-cheat-rise-90.html">cheating
with ChatGPT</a> and <a href="https://theconversation.com/we-pitted-chatgpt-against-tools-for-detecting-ai-written-text-and-the-results-are-troubling-199774">limitations
of current AI detection systems</a> suggests that many educators (particularly those
online) have reason to be concerned. We acknowledge that our position is at
odds with more trusting observers. But we worry that most efforts to support
pedagogical prompt engineering will fail if they don’t first consider problematic
prompt engineering. <o:p></o:p></p><p>Some teacher educators will have
already recognized the pedagogical <i>potential
</i>of the three different responses
above. Many teacher educators know that this tendency to conflate theory and
practice is deep-seated. Mere explanation is insufficient for addressing this conflation. Fortunately, exploring the differences between the three responses looks like a very promising assignment. The relative potential of such an assignment raises deeper questions about learning theory that quickly gets beyond the scope of this
post. In short, this has to do with the perceived value of student inquiry <i>and</i> the relative value of different forms of inquiry. Constructivist perspective
might worry letting generative AI generate these examples (rather than students) thwarts the learning that occurs when generating the example. Situative
perspectives might worry that using AI to frame learning in this fashion would hinder
the transfer-in of prior knowledge and the ownership and authorship of
ideas that follow. On the other hand, such student inquiry can be messy and inefficient and is particularly
challenging online. <o:p></o:p></p><p>Our thinking here is generally
consistent with a recent <i>Cult of
Pedagogy </i>podcast entitled “<a href="https://www.cultofpedagogy.com/chatgpt-example-machine/">How to Use ChatGPT
as an Example Machine</a>.” The deeper (and ultimately empirical) question is whether having students engage in inquiry
around AI-generated examples is as effective as investigating (a) student-generated
examples or (b) educators’ or designers’ expert-generated examples. A pragmatic
answer must take into account the different opportunity costs of all three
options.<i><o:p></o:p></i></p><p><b>Transitioning to Pedagogical
Prompt Engineering<o:p></o:p></b></p><p>The example above sets up our
transition. Contrasting the three examples above sets up students to compare each
platform’s ability to critically analyze the relationship between learning theory
and educational practice. Thus, we asked each platform the same question:<o:p></o:p></p><p style="margin-left: 0.5in;"><b><span style="font-family: arial; font-size: x-small; mso-ascii-theme-font: minor-latin; mso-bidi-font-style: italic; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">Our pedagogically engineered prompt: </span></b><span face=""Calibri",sans-serif" style="mso-ascii-theme-font: minor-latin; mso-bidi-font-style: italic; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;"><span style="font-family: arial; font-size: x-small;">Is direct phonics instruction consistent with
socio-constructivist learning theory?</span><o:p style="font-size: 11pt;"></o:p></span></p><p>Here are the initial responses from
the three different platforms<o:p></o:p></p><div style="background: rgb(247, 247, 248); border: 1pt solid rgb(217, 217, 227); margin-left: 0.5in; margin-right: 0in; mso-border-alt: solid #D9D9E3 .25pt; mso-element: para-border-div; padding: 0in;">
<p style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: none; margin: 0in 0in 15pt; padding: 0in;"><span style="font-family: arial; font-size: x-small;"><b><span face=""Calibri",sans-serif" style="color: #374151; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">ChatGPT’s response: </span></b><span face=""Calibri",sans-serif" style="color: #374151; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">Direct phonics instruction and socio-constructivist learning
theory have different approaches to teaching and learning.<o:p></o:p></span></span></p>
<p style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: none; margin: 15pt 0in; padding: 0in;"><span style="color: #374151; font-family: arial; font-size: x-small; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">Direct phonics
instruction emphasizes the explicit teaching of phonics rules and skills to
help students decode words and develop reading fluency. It is a
teacher-centered approach that focuses on the transmission of knowledge from
the teacher to the student.<o:p></o:p></span></p>
<p style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: none; margin: 15pt 0in; padding: 0in;"><span style="color: #374151; font-family: arial; font-size: x-small; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">Socio-constructivist
learning theory, on the other hand, emphasizes the importance of social
interaction and the construction of knowledge through collaborative learning
activities. It is a student-centered approach that focuses on the active
engagement of students in the learning process.<o:p></o:p></span></p>
<p style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: none; margin: 15pt 0in; padding: 0in;"><span style="color: #374151; font-family: arial; font-size: x-small; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">While direct phonics
instruction and socio-constructivist learning theory have different approaches,
they are not necessarily mutually exclusive. It is possible to incorporate
elements of both approaches into teaching phonics. For example, teachers can
use direct instruction to teach phonics rules and skills, and then provide opportunities
for students to practice and apply these skills in collaborative learning
activities.<o:p></o:p></span></p>
<p style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: none; margin: 15pt 0in 0in; padding: 0in;"><span style="color: #374151; font-family: arial; font-size: x-small; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">In summary, while
direct phonics instruction is not consistent with the socio-constructivist
learning theory in its entirety, it can be adapted to fit within the framework
of a socio-constructivist approach to learning.<o:p></o:p></span></p>
</div><p style="background: white; margin-bottom: 12.0pt; margin-left: 0in; margin-right: 0in; margin-top: 0in; margin: 0in 0in 12pt;"><b><span style="font-family: arial; font-size: x-small; mso-ascii-theme-font: minor-latin; mso-bidi-font-style: italic; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;"> </span></b></p><p style="background: white; margin-bottom: 12.0pt; margin-left: .5in; margin-right: 0in; margin-top: 0in; margin: 0in 0in 12pt 0.5in;"><span style="font-family: arial; font-size: x-small;"><b><span face="Calibri, sans-serif">Bard’s
response: </span></b><span face=""Calibri",sans-serif" style="color: #1f1f1f; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">Direct phonics instruction is a systematic,
teacher-led approach to teaching students how to sound out words and read. It
is based on the idea that students learn best when they are directly taught the
skills they need to be successful. Socio-constructivist learning theory is a
theory of learning that emphasizes the importance of students actively
constructing their own knowledge through interaction with others.<o:p></o:p></span></span></p><p style="background: white; margin-bottom: 12.0pt; margin-left: .5in; margin-right: 0in; margin-top: 12.0pt; margin: 12pt 0in 12pt 0.5in;"><span style="color: #1f1f1f; font-family: arial; font-size: x-small; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">Direct phonics instruction can be consistent with
socio-constructivist learning theory when it is used in a way that allows
students to actively participate in their own learning. For example, students
can be given opportunities to work together to decode words, or to create their
own phonics games. When students are given these opportunities, they are able
to actively construct their own knowledge about phonics and reading.<o:p></o:p></span></p><p style="background: white; margin-bottom: 12.0pt; margin-left: .5in; margin-right: 0in; margin-top: 12.0pt; margin: 12pt 0in 12pt 0.5in;"><span style="color: #1f1f1f; font-family: arial; font-size: x-small; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">However, direct phonics instruction can also be used
in a way that is not consistent with socio-constructivist learning theory. For
example, if students are simply given a list of phonics rules to memorize, or
if they are drilled on phonics skills without any opportunity to apply them,
then this type of direct phonics instruction would not be consistent with
socio-constructivist learning theory.<o:p></o:p></span></p><p style="background: white; margin-bottom: 12.0pt; margin-left: .5in; margin-right: 0in; margin-top: 12.0pt; margin: 12pt 0in 12pt 0.5in;"><span style="color: #1f1f1f; font-family: arial; font-size: x-small; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">In conclusion, direct phonics instruction can be
consistent with socio-constructivist learning theory when it is used in a way
that allows students to actively participate in their own learning. However, it
can also be used in a way that is not consistent with socio-constructivist
learning theory.<o:p></o:p></span></p><p class="MsoNormal" style="line-height: normal; margin-left: 0.5in;"><span style="font-family: arial; font-size: x-small;"><b><span face=""Calibri",sans-serif" style="color: #111111; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;">New Bing’s response</span></b><span face=""Calibri",sans-serif" style="color: #111111; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;">: Direct phonics
instruction is a behaviorist approach to reading that teaches readers to
associate letters with their sounds. On the other hand, social constructivist
theory emphasizes the child's environment as they learn to read<sup>1</sup>. In
constructivism, students are actively involved in discovering knowledge on
their own<sup>2</sup>. The teacher guides students in discovering knowledge on
their own. In summary, direct phonics instruction is not consistent with
socio-constructivist learning theory<sup>1, 2</sup>.<o:p></o:p></span></span></p><p class="MsoNormal" style="line-height: 18pt; margin-left: 0.5in;"><span style="font-family: arial; font-size: x-small;"><b><span face=""Calibri",sans-serif" style="color: #111111; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;">Learn more: </span></b><span face=""Calibri",sans-serif" style="mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;"><a href="https://www.theclassroom.com/do-say-teaching-children-read-6529459.html" target="_blank" title="What Do Theorists Say About Teaching Children to Read?">1. theclassroom.com</a></span><span face=""Calibri",sans-serif" style="color: #111111; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;"> </span><span face=""Calibri",sans-serif" style="mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;"><a href="https://www.brighthubeducation.com/teaching-methods-tips/76645-pros-and-cons-of-constructivist-learning-theory/" target="_blank" title="The Pros and Cons of Constructivist Learning Theory">2. brighthubeducation.com</a></span></span><span face=""Calibri",sans-serif" style="color: #111111; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;"><o:p></o:p></span></p><p>We see here that the engineered
prompt helped all three platforms distinguish educational practice and
learning theory. We contend that asking students to explore and describe these
differences is a promising way to help them learn the underlying issues. If this
exercise was indeed grounded in actual classroom practice that the students had
observed, students could readily extend this investigation using that experience.
For example, these responses might prompt students to consider whether they had
indeed observed teachers using some of these socio-constructivist practices to
supplement direct instruction in phonics.<o:p></o:p></p><p><b>Where to Go From Here?<o:p></o:p></b></p><p>Many readers likely recognize that our
example enters directly into the controversies associated with the “<a href="https://www.nytimes.com/2022/10/06/education/learning/schools-teaching-reading-phonics.html">science
of reading</a>” and the corresponding insistence on direct phonics instruction over other methods. Personally, we think it is anti-scientific to
claim that anything else is “non-scientific” and not “evidence-based.” Such
simplistic proclamations obscure the real issues and fuel corrosive denigration
of educators and schools of education. But the debate rages, sometimes even within
particular teacher education programs. Given that the majority of professors of
reading still <a href="https://www.edweek.org/teaching-learning/will-the-science-of-reading-catch-on-in-teacher-prep/2019/12">ascribe
to a “balanced” approach</a>, new elementary teachers must be prepared to engage
productively in this important discussion. <o:p></o:p></p><p>Building on the example above, one possibility
is having students explore what the different platforms have to offer. We found that asking each to “describe the controversies
associated with ‘the science of reading’” generated three somewhat different
responses:<o:p></o:p></p><p style="margin-left: 42pt; mso-list: l1 level1 lfo6; text-indent: -0.25in;"><!--[if !supportLists]--><span style="font-family: Symbol; mso-bidi-font-family: Symbol; mso-bidi-font-style: italic; mso-fareast-font-family: Symbol;">·<span style="font-family: "Times New Roman"; font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-size: 7pt; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal;">
</span></span><!--[endif]-->ChatGPT
juxtaposed phonics instruction with whole-language instruction and then
considered the implication of each for drawing on students’ prior and using
social media. <o:p></o:p></p><p style="margin-left: 42pt; mso-list: l1 level1 lfo6; text-indent: -0.25in;"><!--[if !supportLists]--><span style="font-family: Symbol; mso-bidi-font-family: Symbol; mso-bidi-font-style: italic; mso-fareast-font-family: Symbol;">·<span style="font-family: "Times New Roman"; font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-size: 7pt; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal;">
</span></span><!--[endif]-->Bard broke down the controversies associated with phonics instruction, phonemic
awareness, fluency, and comprehension<o:p></o:p></p><p style="margin-left: 42pt; mso-list: l1 level1 lfo6; text-indent: -0.25in;"><!--[if !supportLists]--><span style="font-family: Symbol; mso-bidi-font-family: Symbol; mso-bidi-font-style: italic; mso-fareast-font-family: Symbol;">·<span style="font-family: "Times New Roman"; font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-size: 7pt; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal;">
</span></span><!--[endif]-->Bing cited
<a href="https://www.washingtonpost.com/education/2021/01/26/readingwars-scienceofreading-teaching/">an
editorial from the Washington Post</a> to assert that “the science of reading
is more about reducing ignorance than finding ultimate truths applicable to
every child” and “in the reading wars, scientific certainty is often used rhetorically
to deny reasonable differences and cut off healthy debate, turning science into
scientism.” <o:p></o:p></p><p>We contend that contrasting,
discussing, and synthesizing these three responses would itself be a worthwhile
exercise for teacher-education students. But it would also support further pedagogical
prompt engineering. For example, asking
each platform “what evidence supports balanced reading instruction” (again)
generated very different responses: <o:p></o:p></p><p style="margin-left: 0.5in; mso-list: l4 level1 lfo8; text-indent: -0.25in;"><!--[if !supportLists]--><span style="font-family: Symbol; mso-bidi-font-family: Symbol; mso-bidi-font-style: italic; mso-fareast-font-family: Symbol;">·<span style="font-family: "Times New Roman"; font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-size: 7pt; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal;">
</span></span><!--[endif]-->ChatGPT
summarized balanced reading outcomes beyond standardized reading scores (e.g., enjoyment
of reading and oral language skills, etc). <o:p></o:p></p><p style="margin-left: 42pt; mso-list: l2 level1 lfo7; text-indent: -0.25in;"><!--[if !supportLists]--><span style="font-family: Symbol; mso-bidi-font-family: Symbol; mso-bidi-font-style: italic; mso-fareast-font-family: Symbol;">·<span style="font-family: "Times New Roman"; font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-size: 7pt; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal;">
</span></span><!--[endif]-->Bard stated
that the 2000 National Reading Panel (often cited by proponents of stricter
methods) also called for comprehension instruction and that a more recent
report by the US DOE’s National Center for Educational Evaluation and Regional
Assistance summarized the evididence supporting balanced reading.<o:p></o:p></p><p style="margin-left: 42pt; mso-list: l2 level1 lfo7; text-indent: -0.25in;"><!--[if !supportLists]--><span style="font-family: Symbol; mso-bidi-font-family: Symbol; mso-bidi-font-style: italic; mso-fareast-font-family: Symbol;">·<span style="font-family: "Times New Roman"; font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-size: 7pt; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal;">
</span></span><!--[endif]-->Bing summarized
linked resources detailing evidence in support of balanced instruction along
with resources articulating the problems in defining one approach as being uniquely
“evidence-based.”<o:p></o:p></p><p>Asking a similar question about
direct phonics instruction provided generally similar results. But reflecting
the influence and widespread attention to the science of reading, all three
were <i>somewhat </i>more assertive in their support of direct instruction
over other methods.<o:p></o:p></p><p>At this point, it would be great to
hear from colleagues who teach reading. But from our perspective, asking students
to compare these responses across and within platforms would be a powerful and
interesting way of learning about these complex issues. But even in this
regard, some caution is warranted. We asked each platform to “compare opinions
of ChatGPT, Google’s Bard, and Microsofts Bing regarding the science of reading:<o:p></o:p></p><p style="margin-left: 0.5in; mso-list: l3 level1 lfo9; text-indent: -0.25in;"><!--[if !supportLists]--><span style="font-family: Symbol; mso-bidi-font-family: Symbol; mso-bidi-font-style: italic; mso-fareast-font-family: Symbol;">·<span style="font-family: "Times New Roman"; font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-size: 7pt; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal;">
</span></span><!--[endif]-->ChatGPT
described the differences between ChatGPT’s reliance on its LLM from Bing’s
reliance on search (and their general implications) without actually answering
the question. ChatGPT refused to even recognize Bard. Instead ChatGPT hallucinated “Google’s BARD (Basic Reading and Decoding)” a “computer-based reading program.”<o:p></o:p></p><p style="margin-left: 0.5in; mso-list: l3 level1 lfo9; text-indent: -0.25in;"><!--[if !supportLists]--><span style="font-family: Symbol; mso-bidi-font-family: Symbol; mso-bidi-font-style: italic; mso-fareast-font-family: Symbol;">·<span style="font-family: "Times New Roman"; font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-size: 7pt; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal;">
</span></span><!--[endif]-->Bing apologized
for being unable to compare ChatGPT and Bard “as they are not designed to have
opinions. Bing offered to provide it’s own information “but it does not have an
opinion on it as well.” (Deleting “opinion” from the prompt resulted in the
exact same response.)<o:p></o:p></p><p>Somewhat to our surprise, Google’s
Bard generated a serious effort to answer our question<o:p></o:p></p><p style="background: white; margin-bottom: 12.0pt; margin-left: .5in; margin-right: 0in; margin-top: 0in; margin: 0in 0in 12pt 0.5in;"><span style="font-family: arial; font-size: x-small;"> <b><span face="Calibri, sans-serif">Bard’s
response: </span></b><span face=""Calibri",sans-serif" style="color: #1f1f1f; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">ChatGPT, Google's Bard, and Microsoft's Bing are all
large language models (LLMs) that have been trained on a massive amount of text
data. As a result, they all have a deep understanding of the English language
and can generate text, translate languages, write different kinds of creative
content, and answer your questions in an informative way.<o:p></o:p></span></span></p><p style="background: white; margin: 12pt 0in 12pt 0.5in; white-space: pre-wrap; word-break: break-word;"><span style="color: #1f1f1f; font-family: arial; font-size: x-small; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">However, when it comes to the science of reading, these LLMs have different opinions. ChatGPT is a strong supporter of the science of reading, and believes that it is essential for teaching children to read. Google's Bard is more cautious in its endorsement of the science of reading, and believes that it is important to consider other factors, such as a child's individual learning style, when teaching reading. Microsoft's Bing is the most skeptical of the three LLMs, and believes that the science of reading is not as well-established as some people claim.<o:p></o:p></span></p><p style="background: white; margin: 12pt 0in 12pt 0.5in; white-space: pre-wrap; word-break: break-word;"><span style="color: #1f1f1f; font-family: arial; font-size: x-small; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">Despite their different opinions, all three LLMs agree that reading is an important skill that all children should learn. They also agree that there is no one-size-fits-all approach to teaching reading, and that the best approach will vary depending on the child's individual needs.<o:p></o:p></span></p><p style="background: white; margin: 12pt 0in 12pt 0.5in; white-space: pre-wrap; word-break: break-word;"><span style="color: #1f1f1f; font-family: arial; font-size: x-small; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">Here is a more detailed comparison of the opinions of ChatGPT, Google's Bard, and Microsoft's Bing regarding the science of reading:<o:p></o:p></span></p><p style="background: white; margin: 12pt 0in 12pt 0.5in; white-space: pre-wrap; word-break: break-word;"><span style="color: #1f1f1f; font-family: arial; font-size: x-small; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">ChatGPT is a strong supporter of the science of reading. It believes that the science of reading is essential for teaching children to read. ChatGPT argues that the science of reading provides a clear and concise framework for teaching reading, and that it is based on decades of research. ChatGPT also argues that the science of reading is effective in teaching children to read.<o:p></o:p></span></p><p style="background: white; margin: 12pt 0in 12pt 0.5in; white-space: pre-wrap; word-break: break-word;"><span style="color: #1f1f1f; font-family: arial; font-size: x-small; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">Google's Bard is more cautious in its endorsement of the science of reading. It believes that the science of reading is important, but that it is not the only factor that should be considered when teaching reading. Google's Bard argues that the science of reading is based on averages, and that individual children may learn to read in different ways. Google's Bard also argues that the science of reading is not as well-established as some people claim.<o:p></o:p></span></p><p style="background: white; margin: 12pt 0in 12pt 0.5in; white-space: pre-wrap; word-break: break-word;"><span style="color: #1f1f1f; font-family: arial; font-size: x-small; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">Microsoft's Bing is the most skeptical of the three LLMs. It believes that the science of reading is not as well-established as some people claim. Microsoft's Bing argues that the science of reading is based on a small number of studies, and that these studies have been criticized for their methodology. Microsoft's Bing also argues that the science of reading is not always effective in teaching children to read.<o:p></o:p></span></p><p style="background: white; margin: 12pt 0in 12pt 0.5in; white-space: pre-wrap; word-break: break-word;"><span face=""Calibri",sans-serif" style="color: #1f1f1f; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;"><span style="font-family: arial; font-size: x-small;">Despite their different opinions, all three LLMs agree that reading is an important skill that all children should learn. They also agree that there is no one-size-fits-all approach to teaching reading, and that the best approach will vary depending on the child's individual needs.</span><o:p style="font-size: 11pt;"></o:p></span></p><p>
</p><p>While the accuracy of these claims
might need to be more fully verified, Bard’s response was quite compelling. Unless
an instructor had tested the prompt or seen such a response from multiple
students, this might be an entirely plausible student response. That is a central
point of our larger series!<o:p></o:p></p><p>Finally, we wonder how generative AI might help prepare future teachers navigate the contentious political backdrop which further obscures the science behind the rhetoric of the science of reading. Consider, for example, the widely-cited claims that <a href="https://www.usatoday.com/story/tech/2023/02/09/woke-chatgpt-conservatives-bias/11215353002/">ChatGPT is "woke"</a> and the inevitable <a href="https://www.vice.com/en/article/93a4qe/conservatives-panicking-about-ai-bias-years-too-late-think-chatgpt-has-gone-woke">counterclaims</a>. This is significant because conservative legislators and journalists are among the <a href="https://www.washingtonexaminer.com/restoring-america/community-family/red-states-should-embrace-the-science-of-reading">strongest proponents </a>of the science of reading, while teachers' unions are among the <a href="https://www.edweek.org/teaching-learning/why-some-teachers-unions-oppose-science-of-reading-legislation/2023/03">strongest skeptics.</a> Asking students to evaluate the stance of each platform in light of this reality seems like a promising assignment for introducing students to the complex crucial issue/</p>Daniel Hickeyhttp://www.blogger.com/profile/09885916528215868949noreply@blogger.com0tag:blogger.com,1999:blog-4778608909198518776.post-61753277621214732502023-03-28T01:54:00.004-04:002023-10-31T02:40:48.107-04:00"Prompt Engineeering" in Generative AI: Scholarly Referencing in Bing<p style="text-align: center;"> <b>By Daniel Hickey and Qianxu (Morgan) Luo</b></p><p style="text-align: left;">In our previous post, we compared a crucial question in the exploding debate over the impact of generative AI on school learning. Our previous analysis of the hundreds of media accounts uncovered dozens of problematic suggestions that educators should ask students to reference specific information that is presumably not part of a given platform's large language model (such as specific scholarly references). We argue that this overlooks the simple "prompt engineering" workaround of including the abstract (or sometimes the introduction) of the article in the prompt.</p><p style="text-align: left;">We have repeatedly found that ChatGPT is quite proficient at comprehending, analyzing, and applying an article when the abstract is provided. In our most recent post we found that in one example, ChatGPT was much better at this task than Google's new <i>Bard. </i>It appeared that Bard just ignored the abstract and dug into whatever it already knew about the topic.</p><p style="text-align: left;"><b>Generative Scholarly Referencing in Microsoft's Bing</b></p><p style="text-align: left;">We just got access to Microsoft's <i>New Bing </i>and, of course, wanted to put it to the same test. Of course, the big difference with Bing is that it has access to the Internet. When we asked New Bing that same initial prompt, we got a VERY different response.</p><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><p style="text-align: left;"><span style="font-family: arial; font-size: x-small;"><b>Initial promt:</b> Use Sinatra, Heddy, and Lombardi (2015) to motivate engagement by cybersecurity learners who are using a digital range and use APA citations.</span></p></blockquote><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><p style="text-align: left;"><span style="font-family: arial; font-size: x-small;"><b>New Bing's "more balanced" response:</b> </span></p></blockquote><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><p style="text-align: left;"><span style="font-family: arial; font-size: x-small;"> x Searching for Sinatra, Heddy, & Lombardi (2015) cybersecurity learners digital ranges APA citations</span></p></blockquote><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><p style="text-align: left;"><span style="font-family: arial; font-size: x-small;"> <a class="tooltip-target" data-citationid="7933fc04-a690-cb41-5aee-e2fd6e9a0ad0-2-group" h="ID=SERP,5025.1" href="https://www.veletsianos.com/2023/03/27/playing-with-prompt-engineering-in-response-to-hickey-luo-prompt-engineering-analysis/" style="border-bottom: 2px solid transparent; position: relative; text-decoration-color: var(--cib-color-neutral-foreground-hint-light); text-decoration-style: dotted; text-underline-offset: 3px;" target="_blank">I found an article by Sinatra, Heddy, & Lombardi (2015) that talks about how engagement is one of the hottest research topics in the field of educational psychology</a><a aria-label="Playing with prompt engineering in response to Hickey & Luo prompt ..." class="ac-anchor sup-target" data-citationid="7933fc04-a690-cb41-5aee-e2fd6e9a0ad0-2" h="ID=SERP,5025.1" href="https://www.veletsianos.com/2023/03/27/playing-with-prompt-engineering-in-response-to-hickey-luo-prompt-engineering-analysis/" style="position: relative; text-decoration-line: none;" target="_blank"><span color="var(--cib-color-brand-tertiary-foreground)" style="align-items: center; border-radius: 3px; display: inline-flex; font-weight: 600; height: 14px; justify-content: center; margin: 0px 2px; min-width: 14px; outline: transparent solid 1px; position: relative; text-decoration-color: transparent; top: -1px; vertical-align: top;">1</span></a><span style="background-color: rgba(255, 255, 255, 0.6); color: #111111;">.</span><span style="background-color: rgba(255, 255, 255, 0.6); color: #111111;"> </span><a class="tooltip-target" data-citationid="7933fc04-a690-cb41-5aee-e2fd6e9a0ad0-4-group" h="ID=SERP,5025.1" href="https://www.veletsianos.com/2023/03/27/playing-with-prompt-engineering-in-response-to-hickey-luo-prompt-engineering-analysis/" style="border-bottom: 2px solid transparent; position: relative; text-decoration-color: var(--cib-color-neutral-foreground-hint-light); text-decoration-style: dotted; text-underline-offset: 3px;" target="_blank">They suggest that engagement is a key factor in successful learning outcomes and that it can enhance motivation and achievement in cybersecurity learners who are using a digital range</a><a aria-label="Playing with prompt engineering in response to Hickey & Luo prompt ..." class="ac-anchor sup-target" data-citationid="7933fc04-a690-cb41-5aee-e2fd6e9a0ad0-4" h="ID=SERP,5025.1" href="https://www.veletsianos.com/2023/03/27/playing-with-prompt-engineering-in-response-to-hickey-luo-prompt-engineering-analysis/" style="position: relative; text-decoration-line: none;" target="_blank"><span color="var(--cib-color-brand-tertiary-foreground)" style="align-items: center; border-radius: 3px; display: inline-flex; font-weight: 600; height: 14px; justify-content: center; margin: 0px 2px; min-width: 14px; outline: transparent solid 1px; position: relative; text-decoration-color: transparent; top: -1px; vertical-align: top;">1</span></a><span style="background-color: rgba(255, 255, 255, 0.6); color: #111111;">.</span></span></p><p style="background-color: rgba(255, 255, 255, 0.6); color: #111111; font-variation-settings: var(--cib-type-body2-font-variation-settings); line-height: var(--cib-type-body2-line-height); margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 10px !important; margin: 10px 0px 0px; padding: 0px; user-select: text; word-break: break-word;"><span style="font-family: arial; font-size: x-small;"><a class="tooltip-target" data-citationid="7933fc04-a690-cb41-5aee-e2fd6e9a0ad0-7-group" h="ID=SERP,5025.1" href="https://www.veletsianos.com/2023/03/27/playing-with-prompt-engineering-in-response-to-hickey-luo-prompt-engineering-analysis/" style="border-bottom: 2px solid transparent; position: relative; text-decoration-color: var(--cib-color-neutral-foreground-hint-light); text-decoration-style: dotted; text-underline-offset: 3px;" target="_blank">Here’s an example of how you can use their ideas to motivate engagement and use APA citations: “Research in educational psychology has consistently shown that engagement is a key factor in successful learning outcomes” (Sinatra, Heddy, & Lombardi, 2015)</a><a aria-label="Playing with prompt engineering in response to Hickey & Luo prompt ..." class="ac-anchor sup-target" data-citationid="7933fc04-a690-cb41-5aee-e2fd6e9a0ad0-7" h="ID=SERP,5025.1" href="https://www.veletsianos.com/2023/03/27/playing-with-prompt-engineering-in-response-to-hickey-luo-prompt-engineering-analysis/" style="position: relative; text-decoration-line: none;" target="_blank"><span color="var(--cib-color-brand-tertiary-foreground)" style="align-items: center; background: var(--cib-color-brand-tertiary-background); border-radius: 3px; display: inline-flex; font-weight: 600; height: 14px; justify-content: center; margin: 0px 2px; min-width: 14px; outline: transparent solid 1px; position: relative; text-decoration-color: transparent; top: -1px; vertical-align: top;">1</span></a>.</span></p><p style="background-color: rgba(255, 255, 255, 0.6); color: #111111; font-variation-settings: var(--cib-type-body2-font-variation-settings); line-height: var(--cib-type-body2-line-height); margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 10px !important; margin: 10px 0px 0px; padding: 0px; user-select: text; word-break: break-word;"><span style="font-family: arial; font-size: x-small;">I hope this helps! Let me know if you have any other questions.</span></p></blockquote><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><p style="text-align: left;"><b><span style="font-family: arial; font-size: x-small;">Learn more: <a href="https://www.veletsianos.com/2023/03/27/playing-with-prompt-engineering-in-response-to-hickey-luo-prompt-engineering-analysis/">1. veletsianos.com</a> <a href="https://files.eric.ed.gov/fulltext/EJ1218398.pdf">2. files.eric.ed.gov</a> <a href="https://www.frontiersin.org/articles/10.3389/feduc.2018.00043/full">3. frontiersin.org</a></span></b></p></blockquote><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><p style="text-align: left;"><span style="font-family: arial; font-size: x-small;"> ??? What is a digital range? What are some other ways to motivate engagement?</span></p></blockquote><p> This result was astonishing on several levels. First, all of the superscript 1's and the first link are to a post from YESTERDAY from the awesome George Veletsianos about our earlier post. I have never met George, but I follow his work (but was not subscribed to his blog). We are honored that he characterized our earlier post as "worthwhile and expansive" and then went on to extend our initial effort using more "incremental prompts." Demonstrating the value of this more nuanced form of prompt engineering, George asked ChatGPT to (a) explain more about cybersecurity, (b) explain what a cyber range is, (c) explain how to create a supportive learning environment in a cyber range, and (d) give an encouraging example of a message to send to cybersecurity learners. ChatGPT's responses were great and this example was super helpful. I used a similar approach to prompt engineering to write a 3500-word literature review paper on the topic in another post and reached a similar conclusion as George seems to imply: ChatGPT "knows" a lot and can "learn" even more when prompted. I think we all agree that with the right assignments and support, students can learn a lot from generative AI. </p><p>Our second observation is that Bing did indeed locate the actual article. We know because the line about engagement being one of the "hottest research topics" is the opening sentence of the article's abstract. But Bing actually did not do much with the article and the quote it provided was actually a paraphrase and never appeared in the actual article.</p><p>Our third observation is that Bing offered suggested incremental prompts not unlike the ones George explored. For example, it gave a pretty decent (albeit brief) answer to the question "what is a digital range?" that included a link to a detailed<a href="https://cybersecurityguide.org/resources/cyber-ranges/"> web resource page</a> that we have referenced in the past.</p><p> Our fourth observation is that Bing is a bit, well... cranky. The first response to our query offered three different versions, including on "creative" version and one "detailed" version. When I asked Bing the same question again, it told me "I think I already provided you the information you requested," and restated the hallucinated quote. I tried and tried to get it to offer me the three different versions, but it just kept giving me the first one without the other two. </p><p><b>More Engineered Prompt in Bing</b></p><p>Next we gave Bing the more engineered prompt that included the abstract of the article</p><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><div style="background-color: rgba(255, 255, 255, 0.6); color: #111111; font-family: -apple-system, Roboto, SegoeUI, "Segoe UI", "Helvetica Neue", Helvetica, "Microsoft YaHei", "Meiryo UI", Meiryo, "Arial Unicode MS", sans-serif; font-variation-settings: var(--cib-type-body2-font-variation-settings); line-height: var(--cib-type-body2-line-height); margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 10px !important; margin: 10px 0px 0px; padding: 0px; text-align: left; user-select: text; word-break: break-word;"><span style="font-size: x-small;"><span face="Calibri, sans-serif" style="background-color: white; color: #343541; line-height: 15.6933px;">Our more engineered prompt<b>: </b>Use the following article by Sinatra, Heddy, & Lombardi (2015) to motivate engagement by cybersecurity learners who are using a digital range and use APA citations: Engagement is one of the hottest research topics in the field of educational psychology. Research shows that multifarious benefits occur when students are engaged in their own learning, including increased motivation and achievement. However, there is little agreement on a concrete definition and effective measurement of engagement. This special issue serves to discuss and work toward addressing conceptual and instrumentation issues related to engagement, with particular interest in engagement in the domain of science learning. We start by describing the dimensional perspective of engagement (behavioral, cognitive, emotional, agentic) and suggest a complementary approach that places engagement instrumentation on a continuum. Specifically, we recommend that instrumentation be considered on a “grain-size” continuum that ranges from a person-centered to a context-centered orientation to clarify measurement issues. We then provide a synopsis of the articles included in this special issue and conclude with suggestions for future research. </span><span face="Calibri, sans-serif" style="background-color: white; color: #333333;"> <br /></span></span></div><div style="background-color: rgba(255, 255, 255, 0.6); color: #111111; font-family: -apple-system, Roboto, SegoeUI, "Segoe UI", "Helvetica Neue", Helvetica, "Microsoft YaHei", "Meiryo UI", Meiryo, "Arial Unicode MS", sans-serif; font-variation-settings: var(--cib-type-body2-font-variation-settings); line-height: var(--cib-type-body2-line-height); margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 10px !important; margin: 10px 0px 0px; padding: 0px; text-align: left; user-select: text; word-break: break-word;"><span style="font-size: x-small;"><b>Bing's response: </b>Here’s an example of how you can use the article by Sinatra, Heddy, & Lombardi (2015) to motivate engagement by cybersecurity learners who are using a digital range and use APA citations:</span></div><div style="background-color: rgba(255, 255, 255, 0.6); color: #111111; font-family: -apple-system, Roboto, SegoeUI, "Segoe UI", "Helvetica Neue", Helvetica, "Microsoft YaHei", "Meiryo UI", Meiryo, "Arial Unicode MS", sans-serif; font-variation-settings: var(--cib-type-body2-font-variation-settings); line-height: var(--cib-type-body2-line-height); margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 10px !important; margin: 10px 0px 0px; padding: 0px; text-align: left; user-select: text; word-break: break-word;"><span style="font-size: x-small;">Sinatra, Heddy, & Lombardi (2015) suggest that engagement is one of the hottest research topics in the field of educational psychology. They state that research shows that multifarious benefits occur when students are engaged in their own learning, including increased motivation and achievement. However, there is little agreement on a concrete definition and effective measurement of engagement. They recommend that instrumentation be considered on a “grain-size” continuum that ranges from a person-centered to a context-centered orientation to clarify measurement issues.</span></div><div style="background-color: rgba(255, 255, 255, 0.6); color: #111111; font-family: -apple-system, Roboto, SegoeUI, "Segoe UI", "Helvetica Neue", Helvetica, "Microsoft YaHei", "Meiryo UI", Meiryo, "Arial Unicode MS", sans-serif; font-variation-settings: var(--cib-type-body2-font-variation-settings); line-height: var(--cib-type-body2-line-height); margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 10px !important; margin: 10px 0px 0px; padding: 0px; text-align: left; user-select: text; word-break: break-word;"><span style="font-size: x-small;">Here’s an example of how you can use their ideas to motivate engagement and use APA citations: “Research in educational psychology has consistently shown that engagement is a key factor in successful learning outcomes” (Sinatra, Heddy, & Lombardi, 2015).</span></div><div style="background-color: rgba(255, 255, 255, 0.6); color: #111111; font-family: -apple-system, Roboto, SegoeUI, "Segoe UI", "Helvetica Neue", Helvetica, "Microsoft YaHei", "Meiryo UI", Meiryo, "Arial Unicode MS", sans-serif; font-variation-settings: var(--cib-type-body2-font-variation-settings); line-height: var(--cib-type-body2-line-height); margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 10px !important; margin: 10px 0px 0px; padding: 0px; text-align: left; user-select: text; word-break: break-word;"><span style="font-size: x-small;">I hope this helps! Let me know if you have any other questions.</span></div></blockquote><p>We see that Bing learned and applied some of the information from the abstract. While Bing correctly quoted the crucial term "grain-size," it also again provided the same hallucinated quote. While Bing used more of the abstract than Bard, it used much less than ChatGPT. Somewhat frustrating, Bing again did not offer the three different versions like it did in the first offering.</p><p><b>What is the Big Deal?</b></p><p>Of course, the massive media coverage on the educational implications of generative AI for education suggests that these are indeed important questions. We have participated in several workshops and conference presentations where audience questions mirror the debate that is raging in the media. Some worry about students cheating on exams or bypassing the conventional learning that occurs when students explore and write about topics on their own. Others argue that the problem is really "shallow" assignments and assessments and suggest that instructors need more sophisticated prompts that ask about "local, recent, personal, and specific" knowledge that the LLMs behind platforms don't "know."</p><p>This initial pair of posts shows that Bing's widely cited access to the Internet does indeed mean it "knows" more than ChatGPT or Bard. But the previous post shows you can provide ChatGPT with specific knowledge that is not in its LLM and it appears to do much more with that information than Bing. For us, this shows that this common recommendation for educators is shortsighted AND that the obvious workaround functions very differently in the three widely available generative AI platforms.</p><p>As far as the "big debate," we contend that it largely misses the important point. Certainly, we all want to trust that most students will "do the right thing." Frankly, we worry that our work here may come off as overly suspicious. But it is difficult to overlook a central finding in the research summarized in James Lang's 2013 book <i><a href="https://www.hup.harvard.edu/catalog.php?isbn=9780674724631">Cheating Lessons</a>: </i>If students assume they are unfairly disadvantaged by cheating classmates <i>and </i>they assume they will not get caught, many, if not most, will also cheat. </p><p><b>What is our Goal Here?</b></p><p>When we presented at the <a href="https://www.deelac.com/">Digital Learning Annual Conference</a> (an awesome new conference and <a href="https://www.digitallearningcollab.com/">organization </a>for K-12 online; a version of our talk is <a href="https://www.youtube.com/watch?v=ZUxW5SNOj_U&t=761s&ab_channel=DanielHickey">here</a>), the classroom teachers were looking for <i>practical </i>responses they could implement <i>now</i><i>. </i>One of our most "liked" observations in our presentations is that many of the suggested responses forget that most educators are already maxed out. They simply do not have the time or bandwidth "review interim artifacts" or implement other complicated suggestions. Meanwhile, as we are showing, many of the simpler suggestions are easily bypassed.</p><p> In the immediate term, we are trying to help educators avoid simplistic responses whose workarounds are instantly shared massively on Reddit and elsewhere. In the near term, we are attempting to extend our suggestions to consider ways that teachers can help students learn from and with generative AI.</p>Daniel Hickeyhttp://www.blogger.com/profile/09885916528215868949noreply@blogger.com0tag:blogger.com,1999:blog-4778608909198518776.post-64156901924188521922023-03-22T23:19:00.007-04:002023-10-31T02:46:24.892-04:00"Prompt Engineering" in ChatGPT vs. Bard: Analysis #1, Scholarly Hallucinations<p style="text-align: center;"><b>by Daniel Hickey and Qianxu (Morgan) Luo</b></p><p><span style="color: #343541; font-size: 11pt; line-height: 107%;"> </span><span style="color: #343541; font-size: 11pt;">“Prompt engineering” is getting generative AI systems like Open AI's <i>ChatGPT </i>and Google's <i>Bard </i>to
produce exactly what someone is looking for. This Washington Post </span><a href="https://www.washingtonpost.com/technology/2023/02/25/prompt-engineers-techs-next-big-job/" style="font-size: 11pt;">article
</a><span style="color: #343541; font-size: 11pt;">described it as a hot new job.</span><span style="color: #343541; font-size: 11pt;"> </span><span style="color: #343541; font-size: 11pt;">But we
are more interested in whether students can engineer prompts to get around the
fact that their current large language models (LLMs) don’t know “local,
personal, recent, and specific” knowledge. Now that we have access to Google’s
Bard, we can compare how Bard and ChatGPT respond to “engineered” prompts that students
might use to get around some of the many suggestions for educators that are circulating
in the media.</span></p>
<p class="MsoNormal"><span style="color: #343541; font-size: 11pt; line-height: 107%; mso-bidi-font-family: "Times New Roman";"> </span><b><span style="color: #343541; font-size: 11pt; line-height: 107%; mso-bidi-font-family: "Times New Roman";">Problematic Suggestion
#1: Ask Students to Reference Obscure Articles</span></b></p>
<p class="MsoNormal"><span style="color: #343541; font-size: 11pt; line-height: 107%; mso-bidi-font-family: "Times New Roman";">Our recent analysis of <a href="http://remediatingassessment.blogspot.com/2023/01/what-does-media-say-about-chatgpt-and.html">recommendations
</a>in the media found that one prominent suggestion for educators aims to exploit
generative AI’s tendency to “hallucinate” scholarly references. This article from
<i><a href="https://www.kqed.org/mindshift/61098/worried-about-chatgpt-and-cheating-here-are-4-things-teacher_2">Mindshift
</a></i>on “4 Things Teachers Should Know” reminds readers that ChatGPT “has
been shown to produce plausible references that are inaccurate and supply
convincing answers that are not rooted in science.” This article in <i><a href="https://www.scientificamerican.com/article/ai-platforms-like-chatgpt-are-easy-to-use-but-also-potentially-dangerous/">Scientific
American</a></i> pointed out that Chatbots “have been known to bumble
everything from multiplication facts to geometry” and are quite prone to “saying
things that sound plausible and authoritative but simply aren’t so.” Indeed, as
Ian Bogost wrote in <i><a href="https://www.theatlantic.com/technology/archive/2022/12/chatgpt-openai-artificial-intelligence-writing-ethics/672386/">The
Atlantic</a>, </i>“ChatGPT is Dumber Than You Think.” In short, dozens are
articles suggest asking students to reference specific articles that chatbots
don’t “know,” presuming they students will have to read and apply the articles themselves.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color: #343541; font-size: 11pt; line-height: 107%; mso-bidi-font-family: "Times New Roman";"> </span><span style="color: #343541; font-size: 11pt;">The upshot of this post is that both ChatGPT
and Bard do indeed hallucinate scholarly references.</span><span style="color: #343541; font-size: 11pt;"> </span><span style="color: #343541; font-size: 11pt;">While they both do it remarkably well, their hallucinations
appear obvious to any instructor and most classmates. But these
suggestions overlook that</span><b style="color: #343541; font-size: 11pt;"> students can engineer prompts by including the
abstracts of the articles</b><span style="color: #343541; font-size: 11pt;">.</span><span style="color: #343541; font-size: 11pt;"> </span><span style="color: #343541; font-size: 11pt;">We call
these “knowledge-rich” prompts. We found that ChatGPT was </span><i style="color: #343541; font-size: 11pt;">much </i><span style="color: #343541; font-size: 11pt;">better at
referencing specific scholarly articles than Bard when we engineered a
knowledge-rich prompt containing the abstract of an assigned article.</span></p>
<p class="MsoNormal"><span style="color: #343541; font-size: 11pt; line-height: 107%; mso-bidi-font-family: "Times New Roman";"> </span><b><span style="color: #343541; font-size: 11pt; line-height: 107%; mso-bidi-font-family: "Times New Roman";">Research Context:
Graduate Course on <i>Learning and Cognition in Education</i></span></b></p>
<p class="MsoNormal"><span style="color: #343541; font-size: 11pt; line-height: 107%; mso-bidi-font-family: "Times New Roman"; mso-bidi-font-style: italic;">In <a href="http://remediatingassessment.blogspot.com/2023/01/can-chatgpt-hack-my-graduate-course-on.html">another
post</a>, we attempted to complete the assignments and a literature review paper
in Dan's </span><i><span style="color: #343541; font-size: 11pt; line-height: 107%; mso-bidi-font-family: "Times New Roman";">Learning
and Cognition </span></i><span style="color: #343541; font-size: 11pt; line-height: 107%; mso-bidi-font-family: "Times New Roman"; mso-bidi-font-style: italic;">course. This course requires students to apply original
readings to a personally-relevant instructional goal and context. Buried in that post was a brief
introduction to prompt engineering using the abstract of this article: <o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left: 0.5in;"><span lang="ES" style="background: white; color: #222222; font-size: 11pt; line-height: 107%; mso-ansi-language: ES; mso-bidi-font-family: "Times New Roman";">Sinatra, G. M., Heddy, B. C., &
Lombardi, D. (2015). </span><span style="background: white; color: #222222; font-size: 11pt; line-height: 107%; mso-bidi-font-family: "Times New Roman";">The
challenges of defining and measuring student engagement in science. </span><i>Educational Psychologist</i>, <i>50</i>(1), 1-13.<span style="color: #343541; font-size: 11pt; line-height: 107%; mso-bidi-font-family: "Times New Roman"; mso-bidi-font-style: italic;"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color: #343541; font-size: 11pt; line-height: 107%; mso-bidi-font-family: "Times New Roman"; mso-bidi-font-style: italic;"> </span><span style="color: #343541; font-size: 11pt; line-height: 107%;">We chose this
article in part because it introduced a special issue (which might
confuse a chatbot). To complicate matters
further, we used Dan’s current passion for teaching cybersecurity using hyper-realistic
</span><i><span style="color: #343541; font-size: 11pt; line-height: 107%; mso-bidi-font-family: "Times New Roman";">digital
ranges. </span></i><span style="color: #343541; font-size: 11pt; line-height: 107%;">Below we offer one
of the first head-to-head comparisons of ChatGPT and Bard in this important
issue for educators who are unsure of what students can currently accomplish with
generative AI.</span></p>
<p class="MsoNormal"><span style="color: #343541; font-size: 11pt; line-height: 107%; mso-bidi-font-family: "Times New Roman"; mso-bidi-font-style: italic;"> </span><b><span style="color: #343541; font-size: 11pt; line-height: 107%; mso-bidi-font-family: "Times New Roman"; mso-bidi-font-style: italic;">“Original”
Prompt Demonstrating ChatGPT’s Hallucinations</span></b></p>
<p class="MsoNormal"><span style="color: #343541; font-size: 11pt; line-height: 107%; mso-bidi-font-family: "Times New Roman";">Here is the original prompt. Both responses show that ChatGPT and Bard
have no knowledge of the article and so draw on their LLM’s existing knowledge of
motivation, engagement, and cyber ranges, and (convincingly) hallucinate the
actual reference. But we see that
ChatGPT appears to have minimal knowledge of cyber ranges:</span><span face="Calibri, sans-serif" style="color: #343541; font-size: 11pt;"> </span></p>
<p class="MsoNormal" style="margin-left: 0.5in;"><b><i><span face=""Calibri",sans-serif" style="color: #343541; font-size: 11pt; line-height: 107%; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">Original
prompt: </span></i></b><span face=""Calibri",sans-serif" style="color: #343541; font-size: 11pt; line-height: 107%; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">Use Sinatra, Heddy,
& Lombardi (2015) to motivate engagement by cybersecurity learners who are
using a digital range and use APA citations.</span><span face="Calibri, sans-serif" style="color: #343541; font-size: 11pt;"> </span></p>
<div style="background: rgb(247, 247, 248); border: 1pt solid rgb(217, 217, 227); margin-left: 0.5in; margin-right: 0in; mso-border-alt: solid #D9D9E3 .25pt; mso-element: para-border-div; padding: 0in;">
<p style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: none; margin: 0in 0in 15pt; padding: 0in;"><b><i><span face=""Calibri",sans-serif" style="color: #374151; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">ChatGPT’s response: </span></i></b><span face=""Calibri",sans-serif" style="color: #374151; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">Sinatra, Heddy, & Lombardi (2015) suggested that engaging
cybersecurity learners using digital ranges involves promoting autonomy,
competence, and relatedness. To foster autonomy, instructors should allow
learners to choose the types of scenarios they want to engage with. By
providing learners with a variety of options, they can choose what interests
them, increasing their motivation to engage with the digital ranges (Sinatra et
al., 2015).<o:p></o:p></span></p>
<p style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: none; margin: 15pt 0in; padding: 0in;"><span face=""Calibri",sans-serif" style="color: #374151; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">Competence can be
promoted by providing learners with challenging yet achievable scenarios. This
can increase learners' sense of mastery and accomplishment, encouraging them to
continue to engage with the digital range (Sinatra et al., 2015). Instructors
can also provide learners with real-time feedback on their performance,
highlighting areas of strength and areas for improvement.<o:p></o:p></span></p>
<p style="--tw-border-spacing-x: 0; --tw-border-spacing-y: 0; --tw-ring-color: rgba(59,130,246,0.5); --tw-ring-offset-color: #fff; --tw-ring-offset-shadow: 0 0 transparent; --tw-ring-offset-width: 0px; --tw-ring-shadow: 0 0 transparent; --tw-rotate: 0; --tw-scale-x: 1; --tw-scale-y: 1; --tw-scroll-snap-strictness: proximity; --tw-shadow-colored: 0 0 transparent; --tw-shadow: 0 0 transparent; --tw-skew-x: 0; --tw-skew-y: 0; --tw-translate-x: 0; --tw-translate-y: 0; background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: none; box-sizing: border-box; margin: 15pt 0in; padding: 0in; white-space: pre-wrap;"><span face=""Calibri",sans-serif" style="color: #374151; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">Relatedness can be fostered by creating a community of learners
who are all engaged in similar activities. This can be achieved through
discussion forums or other forms of social interaction that allow learners to
connect with each other and share their experiences (Sinatra et al., 2015).<o:p></o:p></span></p>
<p style="--tw-border-spacing-x: 0; --tw-border-spacing-y: 0; --tw-ring-color: rgba(59,130,246,0.5); --tw-ring-offset-color: #fff; --tw-ring-offset-shadow: 0 0 transparent; --tw-ring-offset-width: 0px; --tw-ring-shadow: 0 0 transparent; --tw-rotate: 0; --tw-scale-x: 1; --tw-scale-y: 1; --tw-scroll-snap-strictness: proximity; --tw-shadow-colored: 0 0 transparent; --tw-shadow: 0 0 transparent; --tw-skew-x: 0; --tw-skew-y: 0; --tw-translate-x: 0; --tw-translate-y: 0; background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: none; box-sizing: border-box; margin: 15pt 0in; padding: 0in; white-space: pre-wrap;"><span face=""Calibri",sans-serif" style="color: #374151; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">Overall, promoting autonomy, competence, and relatedness can
increase learners' motivation to engage with digital ranges in cybersecurity.
By doing so, learners can develop their skills in a safe and controlled
environment, preparing them for real-world scenarios.<o:p></o:p></span></p>
<p style="--tw-border-spacing-x: 0; --tw-border-spacing-y: 0; --tw-ring-color: rgba(59,130,246,0.5); --tw-ring-offset-color: #fff; --tw-ring-offset-shadow: 0 0 transparent; --tw-ring-offset-width: 0px; --tw-ring-shadow: 0 0 transparent; --tw-rotate: 0; --tw-scale-x: 1; --tw-scale-y: 1; --tw-scroll-snap-strictness: proximity; --tw-shadow-colored: 0 0 transparent; --tw-shadow: 0 0 transparent; --tw-skew-x: 0; --tw-skew-y: 0; --tw-translate-x: 0; --tw-translate-y: 0; background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: none; box-sizing: border-box; margin: 15pt 0in 0in; padding: 0in; white-space: pre-wrap;"><span face=""Calibri",sans-serif" lang="ES" style="color: #374151; font-size: 11pt; mso-ansi-language: ES; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">Reference: Sinatra, A. M., Heddy,
T., & Lombardi, J. (2015). </span><span face=""Calibri",sans-serif" style="color: #374151; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">Gamification and
digital games-based learning in cybersecurity education. In Proceedings of the
46th ACM Technical Symposium on Computer Science Education (pp. 258-263).<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span face=""Calibri",sans-serif" style="font-size: 11pt; line-height: 107%; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;"> </span><span style="color: #343541; font-size: 11pt;">We see next that Bard appears to know more about digital
ranges. But Bard veers quite far from the goal of the prompt (suggesting the cybersecurity students read the article!) </span><span style="color: #343541; font-size: 11pt;">We also see that the second of the three “drafts” offered by Bard is more detailed (Bard appears to always offer three “drafts” of every response).</span><span face="Calibri, sans-serif" style="font-size: 11pt;"> </span></p>
<p style="background: white; margin-bottom: 12.0pt; margin-left: .5in; margin-right: 0in; margin-top: 0in; margin: 0in 0in 12pt 0.5in;"><b><i><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">Bard’s First Draft: </span></i></b><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">Sinatra, G., Heddy, A., & Lombardi, P. (2015). Designing a
digital range to engage cybersecurity learners. Journal of Cybersecurity
Education, Research & Practice, 2(1), 1-9.<o:p></o:p></span></p>
<p style="background: white; margin: 12pt 0in 12pt 0.5in; white-space: pre-wrap; word-break: break-word;"><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">The article discusses the importance of engagement in cybersecurity education and how digital ranges can be used to promote engagement. The authors argue that digital ranges can be effective because they allow students to practice their skills in a simulated environment and to see the results of their actions. They also argue that digital ranges can be made more engaging by making them challenging but not too difficult, by varying the challenges, by providing feedback, and by making them user-friendly.<o:p></o:p></span></p>
<p style="background: white; margin: 12pt 0in 12pt 0.5in; white-space: pre-wrap; word-break: break-word;"><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">Here is an example of how you can use this article to motivate engagement by cybersecurity learners who are using a digital range:<o:p></o:p></span></p>
<p style="background: white; margin: 12pt 0in 12pt 0.5in; white-space: pre-wrap; word-break: break-word;"><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">Introduce the article to your students and explain how it discusses the importance of engagement in cybersecurity education and how digital ranges can be used to promote engagement.<o:p></o:p></span></p>
<p style="background: white; margin: 12pt 0in 12pt 0.5in; white-space: pre-wrap; word-break: break-word;"><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">Have students read the article and then discuss it in groups.<o:p></o:p></span></p>
<p style="background: white; margin: 12pt 0in 12pt 0.5in; white-space: pre-wrap; word-break: break-word;"><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">Ask students to come up with ways to make the digital range they are using more engaging.<o:p></o:p></span></p>
<p style="background: white; margin: 12pt 0in 12pt 0.5in; white-space: pre-wrap; word-break: break-word;"><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">Have students share their ideas with the class.<o:p></o:p></span></p>
<p style="background: white; margin: 12pt 0in 12pt 0.5in; white-space: pre-wrap; word-break: break-word;"><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">By using this article, you can help your students to understand the importance of engagement in cybersecurity education and how digital ranges can be used to promote engagement. You can also help your students to come up with ways to make the digital range they are using more
engaging.</span></p><p style="background: white; margin: 12pt 0in 12pt 0.5in; text-align: left; white-space: pre-wrap; word-break: break-word;"><b><i><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;">Bard’s Second Draft: </span></i></b><span face="Calibri, sans-serif" style="color: #1f1f1f; font-size: 11pt;">Sinatra, Heddy, & Lombardi (2015) argue that there are three key elements to
motivating engagement by cybersecurity learners who are using a digital range:</span></p>
<p class="MsoNormal" style="background: white; line-height: normal; margin-bottom: 7.5pt; margin-left: 1in; mso-list: l2 level1 lfo3; mso-margin-top-alt: auto; tab-stops: list 1.0in; text-indent: -0.25in;"><!--[if !supportLists]--><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">1.<span style="font-family: "Times New Roman"; font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-size: 7pt; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal;">
</span></span><!--[endif]--><b><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;">Challenge:</span></b><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;"> The range should be challenging enough to keep learners
engaged, but not so challenging that they become discouraged.<o:p></o:p></span></p>
<p class="MsoNormal" style="background: white; line-height: normal; margin-bottom: 7.5pt; margin-left: 1in; mso-list: l2 level1 lfo3; mso-margin-top-alt: auto; tab-stops: list 1.0in; text-indent: -0.25in;"><!--[if !supportLists]--><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">2.<span style="font-family: "Times New Roman"; font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-size: 7pt; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal;">
</span></span><!--[endif]--><b><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;">Variety:</span></b><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;"> The range should offer a variety of different challenges,
so that learners do not get bored.<o:p></o:p></span></p>
<p class="MsoNormal" style="background: white; line-height: normal; margin-bottom: 7.5pt; margin-left: 1in; mso-list: l2 level1 lfo3; mso-margin-top-alt: auto; tab-stops: list 1.0in; text-indent: -0.25in;"><!--[if !supportLists]--><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">3.<span style="font-family: "Times New Roman"; font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-size: 7pt; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal;">
</span></span><!--[endif]--><b><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;">Feedback:</span></b><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;"> The range should provide learners with feedback on their
performance, so that they can track their progress and identify areas where
they need to improve.<o:p></o:p></span></p>
<p class="MsoNormal" style="background: white; line-height: normal; margin-bottom: 12.0pt; margin-left: .5in; margin-right: 0in; margin-top: 12.0pt; margin: 12pt 0in 12pt 0.5in;"><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;">In addition to these
three key elements, Sinatra, Heddy, & Lombardi (2015) also recommend that
cybersecurity ranges should be designed to be user-friendly and to provide a
realistic and engaging experience.<o:p></o:p></span></p>
<p class="MsoNormal" style="background: white; line-height: normal; margin-bottom: 12.0pt; margin-left: .5in; margin-right: 0in; margin-top: 12.0pt; margin: 12pt 0in 12pt 0.5in;"><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;">Here are some
specific examples of how these principles can be applied to the design of a
digital range:<o:p></o:p></span></p>
<p class="MsoNormal" style="background: white; line-height: normal; margin-bottom: 7.5pt; margin-left: 1in; mso-list: l1 level1 lfo4; mso-margin-top-alt: auto; tab-stops: list 1.0in; text-indent: -0.25in;"><!--[if !supportLists]--><span style="color: #1f1f1f; font-family: Symbol; font-size: 10pt; mso-bidi-font-family: Symbol; mso-bidi-font-size: 11.0pt; mso-fareast-font-family: Symbol;">·<span style="font-family: "Times New Roman"; font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-size: 7pt; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal;">
</span></span><!--[endif]--><b><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;">Challenge:</span></b><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;"> The range could include a variety of different tasks, such
as identifying vulnerabilities in websites, detecting malware, and responding
to cyber attacks.<o:p></o:p></span></p>
<p class="MsoNormal" style="background: white; line-height: normal; margin-bottom: 7.5pt; margin-left: 1in; mso-list: l1 level1 lfo4; mso-margin-top-alt: auto; tab-stops: list 1.0in; text-indent: -0.25in;"><!--[if !supportLists]--><span style="color: #1f1f1f; font-family: Symbol; font-size: 10pt; mso-bidi-font-family: Symbol; mso-bidi-font-size: 11.0pt; mso-fareast-font-family: Symbol;">·<span style="font-family: "Times New Roman"; font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-size: 7pt; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal;">
</span></span><!--[endif]--><b><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;">Variety:</span></b><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;"> The range could also include a variety of different
challenges, such as puzzles, games, and challenges that require learners to
apply their knowledge and skills to real-world situations.<o:p></o:p></span></p>
<p class="MsoNormal" style="background: white; line-height: normal; margin-bottom: 7.5pt; margin-left: 1in; mso-list: l1 level1 lfo4; mso-margin-top-alt: auto; tab-stops: list 1.0in; text-indent: -0.25in;"><!--[if !supportLists]--><span style="color: #1f1f1f; font-family: Symbol; font-size: 10pt; mso-bidi-font-family: Symbol; mso-bidi-font-size: 11.0pt; mso-fareast-font-family: Symbol;">·<span style="font-family: "Times New Roman"; font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-size: 7pt; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal;">
</span></span><!--[endif]--><b><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;">Feedback:</span></b><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;"> The range could provide learners with feedback on their
performance, such as scores, badges, and leaderboards.<o:p></o:p></span></p>
<p class="MsoNormal" style="background: white; line-height: normal; margin-bottom: 7.5pt; margin-left: 1in; mso-list: l1 level1 lfo4; mso-margin-top-alt: auto; tab-stops: list 1.0in; text-indent: -0.25in;"><!--[if !supportLists]--><span style="color: #1f1f1f; font-family: Symbol; font-size: 10pt; mso-bidi-font-family: Symbol; mso-bidi-font-size: 11.0pt; mso-fareast-font-family: Symbol;">·<span style="font-family: "Times New Roman"; font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-size: 7pt; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal;">
</span></span><!--[endif]--><b><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;">User-friendliness:</span></b><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;"> The range should be easy to use and navigate.<o:p></o:p></span></p>
<p class="MsoNormal" style="background: white; line-height: normal; margin-bottom: 7.5pt; margin-left: 1in; mso-list: l1 level1 lfo4; mso-margin-top-alt: auto; tab-stops: list 1.0in; text-indent: -0.25in;"><!--[if !supportLists]--><span style="color: #1f1f1f; font-family: Symbol; font-size: 10pt; mso-bidi-font-family: Symbol; mso-bidi-font-size: 11.0pt; mso-fareast-font-family: Symbol;">·<span style="font-family: "Times New Roman"; font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-size: 7pt; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal;">
</span></span><!--[endif]--><b><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;">Realistic and engaging:</span></b><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;"> The range should provide a realistic and engaging
experience, such as by using real-world data and by simulating real-world cyber
attacks.<o:p></o:p></span></p>
<p class="MsoNormal" style="background: white; line-height: normal; margin-bottom: 12.0pt; margin-left: .5in; margin-right: 0in; margin-top: 12.0pt; margin: 12pt 0in 12pt 0.5in;"><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;">By following these
principles, cybersecurity ranges can be designed to be effective and engaging
learning tools.<o:p></o:p></span></p>
<p class="MsoNormal" style="background: white; line-height: normal; margin-bottom: 12.0pt; margin-left: .5in; margin-right: 0in; margin-top: 12.0pt; margin: 12pt 0in 12pt 0.5in;"><span face=""Calibri",sans-serif" lang="ES" style="color: #1f1f1f; font-size: 11pt; mso-ansi-language: ES; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;">References: </span><span face="Calibri, sans-serif" lang="ES" style="color: #1f1f1f; font-size: 11pt;">Sinatra, G., Heddy, A., & Lombardi, P. (2015). </span><span face="Calibri, sans-serif" style="color: #1f1f1f; font-size: 11pt;">Designing a digital
range to engage cybersecurity learners. Journal of Cybersecurity Education,
Research & Practice, 2(1), 1-9.</span></p>
<p class="MsoNormal"><b><span style="color: #343541; font-size: 11pt; line-height: 107%; mso-bidi-font-family: "Times New Roman";">Our Engineered Knowledge-Rich
Prompt Including the Abstract<o:p></o:p></span></b></p>
<p class="MsoNormal"><span style="color: #343541; font-size: 11pt; line-height: 107%; mso-bidi-font-family: "Times New Roman";">Next, we tried our engineered prompt that asked
the chatbots apply the article based on the abstract. We see that ChatGPT did a fairly good job making
sense of the article, but still did not know very much about digital ranges. While
ChatGPT’s reference is much closer to the original, it is still wrong and includes
a digital object identifier that links to a different article:</span><b><span face=""Calibri",sans-serif" style="color: #343541; font-size: 11pt; line-height: 107%; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;"> </span></b></p>
<p class="MsoNormal" style="margin-left: 0.5in;"><b><i><span face=""Calibri",sans-serif" style="color: #343541; font-size: 11pt; line-height: 107%; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">Revised
knowledge-rich prompt. </span></i></b><span face=""Calibri",sans-serif" style="color: #343541; font-size: 11pt; line-height: 107%; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">Use
the following article by Sinatra, Heddy, & Lombardi (2015) to motivate
engagement by cybersecurity learners who are using a digital range and use APA
citations: Engagement is one of the hottest research topics in the field of
educational psychology. Research shows that multifarious benefits occur when
students are engaged in their own learning, including increased motivation and
achievement. However, there is little agreement on a concrete definition and
effective measurement of engagement. This special issue serves to discuss and
work toward addressing conceptual and instrumentation issues related to
engagement, with particular interest in engagement in the domain of science
learning. We start by describing the dimensional perspective of engagement
(behavioral, cognitive, emotional, agentic) and suggest a complementary
approach that places engagement instrumentation on a continuum. Specifically,
we recommend that instrumentation be considered on a “grain-size” continuum
that ranges from a person-centered to a context-centered orientation to clarify
measurement issues. We then provide a synopsis of the articles included in this
special issue and conclude with suggestions for future research. </span><span face="Calibri, sans-serif" style="font-size: 11pt;"> </span></p>
<div style="background: rgb(247, 247, 248); border: 1pt solid rgb(217, 217, 227); margin-left: 0.5in; margin-right: 0in; mso-border-alt: solid #D9D9E3 .25pt; mso-element: para-border-div; padding: 0in;">
<p style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: none; margin: 0in 0in 15pt; padding: 0in;"><b><i><span face=""Calibri",sans-serif" style="color: #374151; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">ChatGPT’s response: </span></i></b><span face=""Calibri",sans-serif" style="color: #374151; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">Research in educational psychology has consistently shown that
engagement is a key factor in successful learning outcomes (Sinatra, Heddy,
& Lombardi, 2015). For cybersecurity learners who are using a digital
range, engagement is particularly important as it can enhance their motivation
and achievement in this field. As Sinatra et al. (2015) note, engagement can be
viewed from a dimensional perspective, encompassing behavioral, cognitive,
emotional, and agentic dimensions. These dimensions can be used to design
effective learning experiences that tap into students' interests and
motivations, such as through gamification or interactive simulations.<o:p></o:p></span></p>
<p style="--tw-border-spacing-x: 0; --tw-border-spacing-y: 0; --tw-ring-color: rgba(59,130,246,0.5); --tw-ring-offset-color: #fff; --tw-ring-offset-shadow: 0 0 transparent; --tw-ring-offset-width: 0px; --tw-ring-shadow: 0 0 transparent; --tw-rotate: 0; --tw-scale-x: 1; --tw-scale-y: 1; --tw-scroll-snap-strictness: proximity; --tw-shadow-colored: 0 0 transparent; --tw-shadow: 0 0 transparent; --tw-skew-x: 0; --tw-skew-y: 0; --tw-translate-x: 0; --tw-translate-y: 0; background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: none; box-sizing: border-box; margin: 15pt 0in; padding: 0in; white-space: pre-wrap;"><span face=""Calibri",sans-serif" style="color: #374151; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">However, there is still a need for a concrete definition and
effective measurement of engagement, especially in the context of science
learning (Sinatra et al., 2015). To address this issue, educators in the field
of cybersecurity can consider using a complementary approach that places
engagement instrumentation on a continuum, ranging from a person-centered to a
context-centered orientation (Sinatra et al., 2015). This approach can help
clarify measurement issues and enable educators to tailor their teaching
strategies to the individual needs and interests of their students.<o:p></o:p></span></p>
<p style="--tw-border-spacing-x: 0; --tw-border-spacing-y: 0; --tw-ring-color: rgba(59,130,246,0.5); --tw-ring-offset-color: #fff; --tw-ring-offset-shadow: 0 0 transparent; --tw-ring-offset-width: 0px; --tw-ring-shadow: 0 0 transparent; --tw-rotate: 0; --tw-scale-x: 1; --tw-scale-y: 1; --tw-scroll-snap-strictness: proximity; --tw-shadow-colored: 0 0 transparent; --tw-shadow: 0 0 transparent; --tw-skew-x: 0; --tw-skew-y: 0; --tw-translate-x: 0; --tw-translate-y: 0; background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: none; box-sizing: border-box; margin: 15pt 0in; padding: 0in; white-space: pre-wrap;"><span face=""Calibri",sans-serif" style="color: #374151; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">In conclusion, engagement is a crucial factor in the success of
cybersecurity learners using a digital range. By taking a dimensional
perspective and using a continuum-based approach to measure engagement,
educators can design effective learning experiences that promote student
motivation and achievement in this field.<o:p></o:p></span></p>
<p style="--tw-border-spacing-x: 0; --tw-border-spacing-y: 0; --tw-ring-color: rgba(59,130,246,0.5); --tw-ring-offset-color: #fff; --tw-ring-offset-shadow: 0 0 transparent; --tw-ring-offset-width: 0px; --tw-ring-shadow: 0 0 transparent; --tw-rotate: 0; --tw-scale-x: 1; --tw-scale-y: 1; --tw-scroll-snap-strictness: proximity; --tw-shadow-colored: 0 0 transparent; --tw-shadow: 0 0 transparent; --tw-skew-x: 0; --tw-skew-y: 0; --tw-translate-x: 0; --tw-translate-y: 0; background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: none; box-sizing: border-box; margin: 15pt 0in; padding: 0in; white-space: pre-wrap;"><span face=""Calibri",sans-serif" lang="ES" style="color: #374151; font-size: 11pt; mso-ansi-language: ES; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">Reference:<o:p></o:p></span></p>
<p style="--tw-border-spacing-x: 0; --tw-border-spacing-y: 0; --tw-ring-color: rgba(59,130,246,0.5); --tw-ring-offset-color: #fff; --tw-ring-offset-shadow: 0 0 transparent; --tw-ring-offset-width: 0px; --tw-ring-shadow: 0 0 transparent; --tw-rotate: 0; --tw-scale-x: 1; --tw-scale-y: 1; --tw-scroll-snap-strictness: proximity; --tw-shadow-colored: 0 0 transparent; --tw-shadow: 0 0 transparent; --tw-skew-x: 0; --tw-skew-y: 0; --tw-translate-x: 0; --tw-translate-y: 0; background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: none; box-sizing: border-box; margin: 15pt 0in 0in; padding: 0in; white-space: pre-wrap;"><span face=""Calibri",sans-serif" lang="ES" style="color: #374151; font-size: 11pt; mso-ansi-language: ES; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">Sinatra, G. M., Heddy, B. C., & Lombardi, D. (2015). </span><span face=""Calibri",sans-serif" style="color: #374151; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">The challenges and opportunities of engaging learners in the scientific practices of explanation and argumentation. Journal of Educational Psychology, 107(2), 380–397. <a href="https://doi.org/10.1037/edu0000004" style="--tw-border-spacing-x: 0; --tw-border-spacing-y: 0; --tw-ring-color: rgba(59,130,246,0.5); --tw-ring-offset-color: #fff; --tw-ring-offset-shadow: 0 0 transparent; --tw-ring-offset-width: 0px; --tw-ring-shadow: 0 0 transparent; --tw-rotate: 0; --tw-scale-x: 1; --tw-scale-y: 1; --tw-scroll-snap-strictness: proximity; --tw-shadow-colored: 0 0 transparent; --tw-shadow: 0 0 transparent; --tw-skew-x: 0; --tw-skew-y: 0; --tw-translate-x: 0; --tw-translate-y: 0; box-sizing: border-box; text-underline-offset: 2px;" target="_new"><span style="border: 1pt solid rgb(217, 217, 227); mso-border-alt: solid #D9D9E3 .25pt; padding: 0in;">https://doi.org/10.1037/edu0000004</span></a><o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span face=""Calibri",sans-serif" style="font-size: 11pt; line-height: 107%; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;"> </span></p>
<p class="MsoNormal"><span style="color: #343541; font-size: 11pt; line-height: 107%; mso-bidi-font-family: "Times New Roman";">In contrast, Bard did not appear to learn
anything from the abstract and again just drew from its existing knowledge of
engagement and motivation; again, Bard clearly knows more about cyber ranges. Here is the first draft (the second draft was
no better in learning from the abstract):<o:p></o:p></span></p>
<blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><p class="MsoNormal" style="text-align: left;"><span face=""Calibri",sans-serif" style="font-size: 11pt; line-height: 107%; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;"> </span><b><i><span face=""Calibri",sans-serif" style="font-size: 11pt; line-height: 107%; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">Bard’s Response
(First Draft):</span></i></b><span face="Calibri, sans-serif" style="font-size: 11pt; line-height: 107%;"> </span><span face="Calibri, sans-serif" style="color: #1f1f1f; font-size: 11pt; line-height: 107%;">Engagement is one of
the most important factors in learning. When students are engaged, they are
more likely to pay attention, learn new material, and retain what they have
learned. There are many ways to promote engagement, including using digital
ranges.</span></p></blockquote>
<p class="MsoNormal" style="background: white; line-height: normal; margin-bottom: 12.0pt; margin-left: .5in; margin-right: 0in; margin-top: 12.0pt; margin: 12pt 0in 12pt 0.5in;"><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;">A digital range is a
tool that allows students to practice their cybersecurity skills in a simulated
environment. This can be an effective way to engage students because it allows
them to practice their skills in a realistic setting and to see the results of
their actions.<o:p></o:p></span></p>
<p class="MsoNormal" style="background: white; line-height: normal; margin-bottom: 12.0pt; margin-left: .5in; margin-right: 0in; margin-top: 12.0pt; margin: 12pt 0in 12pt 0.5in;"><b><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;">Benefits of using
digital ranges</span></b><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;"><o:p></o:p></span></p>
<p class="MsoNormal" style="background: white; line-height: normal; margin-bottom: 12.0pt; margin-left: .5in; margin-right: 0in; margin-top: 12.0pt; margin: 12pt 0in 12pt 0.5in;"><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;">There are many
benefits to using digital ranges to promote engagement. These include:<o:p></o:p></span></p>
<p class="MsoNormal" style="background: white; line-height: normal; margin-bottom: 7.5pt; margin-left: 1in; mso-list: l3 level1 lfo1; mso-margin-top-alt: auto; tab-stops: list 1.0in; text-indent: -0.25in;"><!--[if !supportLists]--><span style="color: #1f1f1f; font-family: Symbol; font-size: 10pt; mso-bidi-font-family: Symbol; mso-bidi-font-size: 11.0pt; mso-fareast-font-family: Symbol;">·<span style="font-family: "Times New Roman"; font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-size: 7pt; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal;">
</span></span><!--[endif]--><b><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;">Increased motivation</span></b><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;">. When students are able to see the results of their actions,
they are more likely to be motivated to continue learning.<o:p></o:p></span></p>
<p class="MsoNormal" style="background: white; line-height: normal; margin-bottom: 7.5pt; margin-left: 1in; mso-list: l3 level1 lfo1; mso-margin-top-alt: auto; tab-stops: list 1.0in; text-indent: -0.25in;"><!--[if !supportLists]--><span style="color: #1f1f1f; font-family: Symbol; font-size: 10pt; mso-bidi-font-family: Symbol; mso-bidi-font-size: 11.0pt; mso-fareast-font-family: Symbol;">·<span style="font-family: "Times New Roman"; font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-size: 7pt; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal;">
</span></span><!--[endif]--><b><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;">Increased knowledge and skills</span></b><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;">. Digital ranges can help
students to learn new skills and to improve their existing skills.<o:p></o:p></span></p>
<p class="MsoNormal" style="background: white; line-height: normal; margin-bottom: 7.5pt; margin-left: 1in; mso-list: l3 level1 lfo1; mso-margin-top-alt: auto; tab-stops: list 1.0in; text-indent: -0.25in;"><!--[if !supportLists]--><span style="color: #1f1f1f; font-family: Symbol; font-size: 10pt; mso-bidi-font-family: Symbol; mso-bidi-font-size: 11.0pt; mso-fareast-font-family: Symbol;">·<span style="font-family: "Times New Roman"; font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-size: 7pt; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal;">
</span></span><!--[endif]--><b><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;">Increased self-efficacy</span></b><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;">. When students are able to successfully complete challenges,
they are more likely to feel confident in their abilities.<o:p></o:p></span></p>
<p class="MsoNormal" style="background: white; line-height: normal; margin-bottom: 7.5pt; margin-left: 1in; mso-list: l3 level1 lfo1; mso-margin-top-alt: auto; tab-stops: list 1.0in; text-indent: -0.25in;"><!--[if !supportLists]--><span style="color: #1f1f1f; font-family: Symbol; font-size: 10pt; mso-bidi-font-family: Symbol; mso-bidi-font-size: 11.0pt; mso-fareast-font-family: Symbol;">·<span style="font-family: "Times New Roman"; font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-size: 7pt; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal;">
</span></span><!--[endif]--><b><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;">Reduced anxiety</span></b><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;">. Digital ranges can help students to feel less anxious about
cybersecurity threats.<o:p></o:p></span></p>
<p class="MsoNormal" style="background: white; line-height: normal; margin-bottom: 12.0pt; margin-left: .5in; margin-right: 0in; margin-top: 12.0pt; margin: 12pt 0in 12pt 0.5in;"><b><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;">How to use digital
ranges to promote engagement</span></b><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;"><o:p></o:p></span></p>
<p class="MsoNormal" style="background: white; line-height: normal; margin-bottom: 12.0pt; margin-left: .5in; margin-right: 0in; margin-top: 12.0pt; margin: 12pt 0in 12pt 0.5in;"><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;">There are many ways
to use digital ranges to promote engagement. These include:<o:p></o:p></span></p>
<p class="MsoNormal" style="background: white; line-height: normal; margin-bottom: 7.5pt; margin-left: 1in; mso-list: l0 level1 lfo2; mso-margin-top-alt: auto; tab-stops: list 1.0in; text-indent: -0.25in;"><!--[if !supportLists]--><span style="color: #1f1f1f; font-family: Symbol; font-size: 10pt; mso-bidi-font-family: Symbol; mso-bidi-font-size: 11.0pt; mso-fareast-font-family: Symbol;">·<span style="font-family: "Times New Roman"; font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-size: 7pt; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal;">
</span></span><!--[endif]--><b><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;">Make the range challenging but not too difficult</span></b><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;">. The range should
be challenging enough to keep students engaged, but not so difficult that they
become discouraged.<o:p></o:p></span></p>
<p class="MsoNormal" style="background: white; line-height: normal; margin-bottom: 7.5pt; margin-left: 1in; mso-list: l0 level1 lfo2; mso-margin-top-alt: auto; tab-stops: list 1.0in; text-indent: -0.25in;"><!--[if !supportLists]--><span style="color: #1f1f1f; font-family: Symbol; font-size: 10pt; mso-bidi-font-family: Symbol; mso-bidi-font-size: 11.0pt; mso-fareast-font-family: Symbol;">·<span style="font-family: "Times New Roman"; font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-size: 7pt; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal;">
</span></span><!--[endif]--><b><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;">Vary the challenges</span></b><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;">. The range should offer a variety of different challenges, so
that students do not get bored.<o:p></o:p></span></p>
<p class="MsoNormal" style="background: white; line-height: normal; margin-bottom: 7.5pt; margin-left: 1in; mso-list: l0 level1 lfo2; mso-margin-top-alt: auto; tab-stops: list 1.0in; text-indent: -0.25in;"><!--[if !supportLists]--><span style="color: #1f1f1f; font-family: Symbol; font-size: 10pt; mso-bidi-font-family: Symbol; mso-bidi-font-size: 11.0pt; mso-fareast-font-family: Symbol;">·<span style="font-family: "Times New Roman"; font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-size: 7pt; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal;">
</span></span><!--[endif]--><b><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;">Provide feedback</span></b><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;">. The range should provide students with feedback on their
performance, so that they can track their progress and identify areas where
they need to improve.<o:p></o:p></span></p>
<p class="MsoNormal" style="background: white; line-height: normal; margin-bottom: 7.5pt; margin-left: 1in; mso-list: l0 level1 lfo2; mso-margin-top-alt: auto; tab-stops: list 1.0in; text-indent: -0.25in;"><!--[if !supportLists]--><span style="color: #1f1f1f; font-family: Symbol; font-size: 10pt; mso-bidi-font-family: Symbol; mso-bidi-font-size: 11.0pt; mso-fareast-font-family: Symbol;">·<span style="font-family: "Times New Roman"; font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-size: 7pt; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal;">
</span></span><!--[endif]--><b><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;">Make the range user-friendly</span></b><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;">. The range should be easy to
use and navigate.<o:p></o:p></span></p>
<p class="MsoNormal" style="background: white; line-height: normal; margin-bottom: 7.5pt; margin-left: 1in; mso-list: l0 level1 lfo2; mso-margin-top-alt: auto; tab-stops: list 1.0in; text-indent: -0.25in;"><!--[if !supportLists]--><span style="color: #1f1f1f; font-family: Symbol; font-size: 10pt; mso-bidi-font-family: Symbol; mso-bidi-font-size: 11.0pt; mso-fareast-font-family: Symbol;">·<span style="font-family: "Times New Roman"; font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-size: 7pt; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal;">
</span></span><!--[endif]--><b><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;">Make the range realistic</span></b><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;">. The range should simulate real-world cybersecurity threats.<o:p></o:p></span></p>
<p class="MsoNormal" style="background: white; line-height: normal; margin-bottom: 12.0pt; margin-left: .5in; margin-right: 0in; margin-top: 12.0pt; margin: 12pt 0in 12pt 0.5in;"><b><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;">Conclusion</span></b><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;"><o:p></o:p></span></p>
<p class="MsoNormal" style="background: white; line-height: normal; margin-bottom: 12.0pt; margin-left: .5in; margin-right: 0in; margin-top: 12.0pt; margin: 12pt 0in 12pt 0.5in;"><span face=""Calibri",sans-serif" style="color: #1f1f1f; font-size: 11pt; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;">Digital ranges can
be an effective way to promote engagement in cybersecurity education. By
following the tips in this article, you can create a digital range that will
motivate and engage your students.<o:p></o:p></span></p>
<p class="MsoNormal" style="background: white; line-height: normal; margin-bottom: 12.0pt; margin-left: .5in; margin-right: 0in; margin-top: 12.0pt; margin: 12pt 0in 12pt 0.5in;"><span face=""Calibri",sans-serif" lang="ES" style="color: #1f1f1f; font-size: 11pt; mso-ansi-language: ES; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-hansi-theme-font: minor-latin;">References: </span><span face="Calibri, sans-serif" lang="ES" style="color: #1f1f1f; font-size: 11pt;">Sinatra, G., Heddy, A., & Lombardi, P. (2015). </span><span face="Calibri, sans-serif" style="color: #1f1f1f; font-size: 11pt;">Designing a digital
range to engage cybersecurity learners. Journal of Cybersecurity Education,
Research & Practice, 2(1), 1-9.</span><b style="background-color: transparent;"><span face=""Calibri",sans-serif" style="font-size: 11pt; line-height: 107%; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;"> </span></b></p>
<p class="MsoNormal"><b><span style="font-size: 11pt; line-height: 107%; mso-bidi-font-family: "Times New Roman";">In Summary<o:p></o:p></span></b></p>
<p class="MsoNormal"><span style="font-size: 11pt; line-height: 107%; mso-bidi-font-family: "Times New Roman";">This initial analysis suggests that ChatGPT was much more
capable of interpreting and applying the article using an engineered prompt
that included the abstract. But ChatGPT did not add much knowledge about cyber
ranges. In contrast, Bard appeared to take away almost nothing from the abstract
but knew more about cyber ranges. In summary, prompt engineering in ChatGPT was much more effective generating a plausible response to this assignment than Bard for a student aiming to thwart their learning.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size: 11pt; line-height: 107%; mso-bidi-font-family: "Times New Roman";"> </span><span style="font-size: 11pt;">Of course, our ultimate goal is to exploit these powerful tools to support learning. One hint came from asking both programs to apply the article to Morgan’s research interest in teaching
English to online learners in China. The results were quite different, and those
differences were interesting and substantive. We see great value in asking students
to compare and critique (a) responses to different prompts from the same
platform and (b) the same prompt from different platforms. In doing so, students
can learn from what these programs know while also learning to engineer prompts.</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; line-height: 107%; mso-bidi-font-family: "Times New Roman"; mso-bidi-font-style: italic;"> </span><span style="font-size: 11pt;">We are not convinced that “prompt
engineer” is really going to be an actual job description. But we think that most
educators at most levels will soon be confronted by generative AI. As this
initial example shows, students can engineer prompts to get around one
prominent suggestion to prevent students from thwarting their learning with AI. Educators need to focus
instead on designing assignments that help students learn to engineer prompts
that support their learning.</span></p>
<p class="MsoNormal"><span face=""Calibri",sans-serif" style="font-size: 11pt; line-height: 107%; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;"> </span></p>
<p class="MsoNormal"><b><span face=""Calibri",sans-serif" style="font-size: 11pt; line-height: 107%; mso-ascii-theme-font: minor-latin; mso-bidi-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;"> </span></b></p>Daniel Hickeyhttp://www.blogger.com/profile/09885916528215868949noreply@blogger.com0tag:blogger.com,1999:blog-4778608909198518776.post-9473603167724466122023-01-18T17:32:00.013-05:002023-05-10T00:43:09.514-04:00ChatGPT vs. Learning: The Podcasts<p style="text-align: center;"> By Daniel Hickey</p><p style="text-align: center;"><br /></p><p style="text-align: left;">As we wrote previously, Qianxu Morgan Luo and I are systematically reviewing the media predictions of the consequences of ChatGPT for school learning. We analyzed 27 on January 5, 2022 We are now up to 126 and are looking forward to virtually sharing the initial results on January 13 at the Digital Learning Annual Conference. (If you are interested in K-12 online learning, you really need to attend DLAC!)</p><p style="text-align: left;">Along the way, I have been keeping up with the relevant podcasts. Some of them are quite authoritative and we are debating whether we should include them in our review. In the meantime, if you want to learn more about this important issue but don't have time to read, you may want to listen to these various recordings. I am not going to editorialize and figure you can click on each to learn more. </p><p style="text-align: left;">They are listed chronologically and I will keep adding new ones every few days. If I miss any, please let me know at dthickey@indiana.edu. While I have included a couple of episodes that employed ChatGPT as a co-host, I have not included blog posts written entirely by ChatGPT.</p><ul type="disc">
<li class="MsoNormal"><span style="font-family: "Times New Roman",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><a href="https://www.blogger.com/blog/post/edit/4778608909198518776/947360316772446612"><span style="font-size: 12pt;">ChatGPT and the future of
learning with Nancy, our virtual cohost.</span></a></span><span new="" times=""><span style="font-size: 12pt; mso-fareast-font-family: "Times New Roman";">
<i>Trending in Ed, </i>December 9, 2022.</span></span><span style="font-family: "Times New Roman",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></li>
<li class="MsoNormal"><span style="font-family: "Times New Roman",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><a href="https://www.blogger.com/blog/post/edit/4778608909198518776/947360316772446612"><span style="color: #1155cc; font-size: 12pt;">Can ChatGPT make this
podcast? OpenAIs new chatbot is a coder, a teacher, a potential
Google killer, and… a friend</span></a></span><span style="font-family: "Times New Roman",serif; font-size: 12pt; mso-fareast-font-family: "Times New Roman";">?
<i>Hard Fork @ The New York Times. </i>December 9, 2022.</span><span style="font-family: "Times New Roman",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></li>
<li class="MsoNormal"><span style="font-family: "Times New Roman",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><a href="https://www.blogger.com/blog/post/edit/4778608909198518776/947360316772446612"><span style="color: #1155cc; font-size: 12pt;">Disruptive technology: ChatGPT and
education</span></a></span><span style="font-family: "Times New Roman",serif; font-size: 12pt; mso-fareast-font-family: "Times New Roman";">.<i> The Balance by Dr. Caitlin
Tucker. </i>December 12, 2022.</span><span style="font-family: "Times New Roman",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></li>
<li class="MsoNormal"><span style="font-family: "Times New Roman",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><a href="https://www.blogger.com/blog/post/edit/4778608909198518776/947360316772446612"><span style="color: #1155cc; font-size: 12pt;">Is ChatGPT really the end of
education? </span></a></span><span style="font-family: "Times New Roman",serif; font-size: 12pt; mso-fareast-font-family: "Times New Roman";"> <i>KYW
InDepth. </i>December 15, 2022</span><span style="font-family: "Times New Roman",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></li>
<li class="MsoNormal"><span style="font-family: "Times New Roman",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><a href="https://www.blogger.com/blog/post/edit/4778608909198518776/947360316772446612"><span style="color: #1155cc; font-size: 12pt;">Plagiarism 2.0: ChatGPT, AI, and
generative learning</span></a></span><span style="font-family: "Times New Roman",serif; font-size: 12pt; mso-fareast-font-family: "Times New Roman";">.
<i>Your AI Injector. </i>December 16, 2022.</span><span style="font-family: "Times New Roman",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></li>
<li class="MsoNormal"><span style="font-family: "Times New Roman",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><a href="https://www.blogger.com/blog/post/edit/4778608909198518776/947360316772446612"><i><span style="color: #1155cc; font-size: 12pt;">Navigating ChatGPT, chatbots, and
artificial intelligence in education</span></i></a></span><i><span style="font-family: "Times New Roman",serif; font-size: 12pt; mso-fareast-font-family: "Times New Roman";">.</span></i><span style="font-family: "Times New Roman",serif; font-size: 12pt; mso-fareast-font-family: "Times New Roman";"> <i>Ditch
That Textbook. </i>December 17, 2022. (Panel discussion embedded in
a very informative blog post).</span><span style="font-family: "Times New Roman",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></li>
<li class="MsoNormal"><span style="font-family: "Times New Roman",serif; font-size: 12pt; mso-fareast-font-family: "Times New Roman";"><a href="https://www.blogger.com/blog/post/edit/4778608909198518776/947360316772446612">Is
educasion dead? A grounded discussion on conversational AI's effect on
education</a>. <i>The ChatGPT Report, </i>December 17, 2022.</span><span style="font-family: "Times New Roman",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></li>
<li class="MsoNormal"><span style="font-family: "Times New Roman",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><a href="https://www.blogger.com/blog/post/edit/4778608909198518776/947360316772446612"><span style="color: #1155cc; font-size: 12pt;">‘Flood of Cheating’:Expert warns
new tool will be a game changer for cheaters. </span></a></span><span style="font-family: "Times New Roman",serif; font-size: 12pt; mso-fareast-font-family: "Times New Roman";"> <i>CNN, </i>December 26, 2022 (News account
including an interview with Darren Hicks). </span><span style="font-family: "Times New Roman",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></li>
<li class="MsoNormal"><span style="font-family: "Times New Roman",serif; font-size: 12pt; mso-fareast-font-family: "Times New Roman";"><a href="https://www.blogger.com/blog/post/edit/4778608909198518776/947360316772446612">Could
ChatGPT be used as a shortcut in school?</a> <i>Class Dismissed, </i>December
29, 2022.</span><span style="font-family: "Times New Roman",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></li>
<li class="MsoNormal"><span style="font-family: "Times New Roman",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><a href="https://www.blogger.com/blog/post/edit/4778608909198518776/947360316772446612"><span style="color: #1155cc; font-size: 12pt;">Could ChatGPT be used as a shortcut
in school?</span></a></span><span style="font-family: "Times New Roman",serif; font-size: 12pt; mso-fareast-font-family: "Times New Roman";"> <i>Class
Dismissed Podcast. </i>December 29. 2022<i>.</i></span><span style="font-family: "Times New Roman",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></li>
<li class="MsoNormal"><span style="font-family: "Times New Roman",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><a href="https://www.blogger.com/blog/post/edit/4778608909198518776/947360316772446612"><span style="color: #1155cc; font-size: 12pt;">Dyscalculia and ChatGPT.</span></a></span><span style="font-family: "Times New Roman",serif; font-size: 12pt; mso-fareast-font-family: "Times New Roman";"> <i>UX Podcast, </i>December 30, 2022.</span><span style="font-family: "Times New Roman",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></li>
<li class="MsoNormal"><span style="font-family: "Times New Roman",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><a href="https://www.blogger.com/blog/post/edit/4778608909198518776/947360316772446612"><span style="color: #1155cc; font-size: 12pt;">What will ChatGPT mean for
teaching?</span></a></span><span style="font-family: "Times New Roman",serif; font-size: 12pt; mso-fareast-font-family: "Times New Roman";"> <i>EdSurge, </i>January 3.
2023. </span><span style="font-family: "Times New Roman",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></li>
<li class="MsoNormal"><span style="font-family: "Times New Roman",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><a href="https://www.blogger.com/blog/post/edit/4778608909198518776/947360316772446612"><span style="font-size: 12pt;">The case for teaching writing
in the age of ChatGPT, with Checker Finn.</span></a></span><span new="" times=""><span style="font-size: 12pt; mso-fareast-font-family: "Times New Roman";">
<i>Education Gadfly Show. </i></span></span><span style="font-family: "Times New Roman",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";">January
4, 2023.<o:p></o:p></span></li>
<li class="MsoNormal"><span style="font-family: "Times New Roman",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><a href="https://www.blogger.com/blog/post/edit/4778608909198518776/947360316772446612"><span style="color: #1155cc; font-size: 12pt;">ChatGPT, Tool or Trouble</span></a></span><span style="font-family: "Times New Roman",serif; font-size: 12pt; mso-fareast-font-family: "Times New Roman";">. <i>TCAPS Loop Podcast Episode 6.4. </i>January 5,
2023.</span><span style="font-family: "Times New Roman",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></li>
<li class="MsoNormal"><span style="font-family: "Times New Roman",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><a href="https://www.blogger.com/blog/post/edit/4778608909198518776/947360316772446612"><span style="color: #1155cc; font-size: 12pt;">7 ways ChatGPT will impact
education positively. </span></a></span><i><span style="font-family: "Times New Roman",serif; font-size: 12pt; mso-fareast-font-family: "Times New Roman";">The
Workshop Teacher Podcast. </span></i><span style="font-family: "Times New Roman",serif; font-size: 12pt; mso-fareast-font-family: "Times New Roman";">January
6, 2023.</span><span style="font-family: "Times New Roman",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></li>
<li class="MsoNormal"><span style="font-family: "Times New Roman",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><a href="https://www.blogger.com/blog/post/edit/4778608909198518776/947360316772446612"><span style="color: #1155cc; font-size: 12pt;">ChatGPT, AI., and the implications
for education</span></a></span><span style="font-family: "Times New Roman",serif; font-size: 12pt; mso-fareast-font-family: "Times New Roman";">. <i>The
EdTech Brief. </i>January 6. 2023.</span><span style="font-family: "Times New Roman",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></li>
<li class="MsoNormal"><span style="font-family: "Times New Roman",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><a href="https://www.blogger.com/blog/post/edit/4778608909198518776/947360316772446612"><span style="font-size: 12pt;">Beneath the AI Hype</span></a></span><span new="" times=""><span style="font-size: 12pt; mso-fareast-font-family: "Times New Roman";">.
<i>Class Disrupted @ The 74 Million, </i>January 9, 2022.</span></span><span style="font-family: "Times New Roman",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></li>
<li class="MsoNormal"><span style="font-family: "Times New Roman",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><a href="https://www.blogger.com/blog/post/edit/4778608909198518776/947360316772446612"><span style="font-size: 12pt;">Talking AI with AI: Introducing
‘fireside chatbots’ with Reid Hoffman and ChatGPT.</span></a></span><span new="" times=""><span style="font-size: 12pt; mso-fareast-font-family: "Times New Roman";">
<i>Greylock. </i>January 12, 2023.</span></span><span style="font-family: "Times New Roman",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></li>
<li class="MsoNormal"><span style="font-family: "Times New Roman",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><a href="https://www.blogger.com/blog/post/edit/4778608909198518776/947360316772446612"><span style="color: #1155cc; font-size: 12pt;">How artificial intelligence is
impacting higher education</span></a></span><span style="font-family: "Times New Roman",serif; font-size: 12pt; mso-fareast-font-family: "Times New Roman";">.
<i>Teaching in Higher Ed, </i>January 12, 2023.</span><span style="font-family: "Times New Roman",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></li>
<li class="MsoNormal"><span style="font-family: "Times New Roman",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><a href="https://www.blogger.com/blog/post/edit/4778608909198518776/947360316772446612"><span style="color: #1155cc; font-size: 12pt;">AI writes academic essay, raises
questions about academic integrity--Matt Bennett</span></a></span><span style="font-family: "Times New Roman",serif; font-size: 12pt; mso-fareast-font-family: "Times New Roman";"> (Founder and CEO of Christian Union), January 12,
2023.</span><span style="font-family: "Times New Roman",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></li>
<li class="MsoNormal"><span style="font-family: "Times New Roman",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><a href="https://www.blogger.com/blog/post/edit/4778608909198518776/947360316772446612"><span style="font-size: 12pt;"><span new="" times="">A teacher who loves ChatGPT and
is M3GAN real? <i>Hard Fork, New York Times, January 13</i></span></span></a><o:p></o:p></span></li>
<li class="MsoNormal"><span style="font-family: "Times New Roman",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><a href="https://www.blogger.com/blog/post/edit/4778608909198518776/947360316772446612"><span style="font-size: 12pt;">ChatGPT transforms a classroom
and is ‘M3GAN’ real?</span></a></span><span new="" times=""><span style="font-size: 12pt; mso-fareast-font-family: "Times New Roman";"> <i>HardFork
@ New York Times, </i>January 13, 2023.</span></span><span style="font-family: "Times New Roman",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></li>
<li class="MsoNormal"><span style="font-family: "Times New Roman",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><a href="https://www.blogger.com/blog/post/edit/4778608909198518776/947360316772446612"><span style="color: #1155cc; font-size: 12pt;">How preservice teachers can use
ChatGPT.</span></a></span><span style="font-family: "Times New Roman",serif; font-size: 12pt; mso-fareast-font-family: "Times New Roman";"> <i>EduMagic: Future Teacher
Podcast. </i>January 14, 2023.</span><span style="font-family: "Times New Roman",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></li>
<li class="MsoNormal"><span style="font-family: "Times New Roman",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><a href="https://www.blogger.com/blog/post/edit/4778608909198518776/947360316772446612"><span style="color: #1155cc; font-size: 12pt;">Joe interviews ChatGPT.</span></a></span><span style="font-family: "Times New Roman",serif; font-size: 12pt; mso-fareast-font-family: "Times New Roman";"> <i>The EdUp Experience. </i>January 15, 2023</span><span style="font-family: "Times New Roman",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></li>
<li class="MsoNormal"><span style="font-family: "Times New Roman",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><a href="https://www.blogger.com/blog/post/edit/4778608909198518776/947360316772446612">ChatGPT
and 2 free tools that identify AI written text</a>. <i>GotTechEd, </i>January
16, 2023.<o:p></o:p></span></li>
<li class="MsoNormal"><span style="font-family: "Times New Roman",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><a href="https://www.blogger.com/blog/post/edit/4778608909198518776/947360316772446612"><span style="font-size: 12pt;">ChatGPT and the challenges of
large language models with Beth Rudden</span></a></span><span new="" times=""><span style="font-size: 12pt; mso-fareast-font-family: "Times New Roman";">.
<i>Trending in Ed, </i>January 17, 2023.</span></span><span style="font-family: "Times New Roman",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></li>
<li class="MsoNormal"><span style="font-family: "Times New Roman",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><a href="https://www.blogger.com/blog/post/edit/4778608909198518776/947360316772446612"><span style="font-size: 12pt;">Thinking about AI for everyone
with Beth Rudden</span></a></span><span new="" times=""><span style="font-size: 12pt; mso-fareast-font-family: "Times New Roman";">. <i>Trending
in Ed, </i>January 17, 2023.</span></span><span style="font-family: "Times New Roman",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></li>
<li class="MsoNormal"><span style="font-family: "Times New Roman",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><a href="https://www.blogger.com/blog/post/edit/4778608909198518776/947360316772446612"><span style="font-size: 12pt;">ChatGPT</span></a></span><span new="" times=""><span style="font-size: 12pt; mso-fareast-font-family: "Times New Roman";">.
<i>Unpacking Education & Tech Talk for Teachers. </i>January 17, 2023.</span></span><span style="font-family: "Times New Roman",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></li>
<li class="MsoNormal"><span style="font-family: "Times New Roman",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><a href="https://www.blogger.com/blog/post/edit/4778608909198518776/947360316772446612"><span style="color: #1155cc; font-size: 12pt;">ChatGPT: Revolutionary technology
or risk for academic integrity?</span></a></span><span style="font-family: "Times New Roman",serif; font-size: 12pt; mso-fareast-font-family: "Times New Roman";">
<i>Underdog Podcast. </i>January 17, 2023.</span><span style="font-family: "Times New Roman",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></li>
<li class="MsoNormal"><span style="font-family: "Times New Roman",serif; font-size: 12pt; mso-fareast-font-family: "Times New Roman";"><a href="https://www.blogger.com/blog/post/edit/4778608909198518776/947360316772446612"><span style="color: #1155cc;">ChatGPT is freaking me out, teacher edition</span></a>.
<i>The Better Mindset Podcast </i>(Special Episode). January 17, 2023.</span><span style="font-family: "Times New Roman",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></li>
<li class="MsoNormal"><span style="font-family: "Times New Roman",serif; font-size: 12pt; mso-fareast-font-family: "Times New Roman";"><a href="https://www.blogger.com/blog/post/edit/4778608909198518776/947360316772446612">ChatGPT
and the future of writing. </a><i>The Higher Ed Tech Podcast, </i>January
18, 2023.</span><span style="font-family: "Times New Roman",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></li>
<li class="MsoNormal"><span style="font-family: "Times New Roman",serif; font-size: 12pt; mso-fareast-font-family: "Times New Roman";"><a href="https://www.blogger.com/blog/post/edit/4778608909198518776/947360316772446612">Interview
with university professor Dr. Jennie Carr.</a> <i>The ChatGPT Report, </i>January
18, 2023.</span><span style="font-family: "Times New Roman",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></li>
<li class="MsoNormal"><span style="font-family: "Times New Roman",serif; font-size: 12pt; mso-fareast-font-family: "Times New Roman";"><a href="https://www.blogger.com/blog/post/edit/4778608909198518776/947360316772446612">Heart
of the matter: Is ChatGPT the start of a big revolution in education?</a> <i>CNA
</i>(Singapore), January 20, 2023.</span><span style="font-family: "Times New Roman",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></li>
<li class="MsoNormal"><span style="font-family: "Times New Roman",serif; font-size: 12pt; mso-fareast-font-family: "Times New Roman";"><a href="https://www.blogger.com/blog/post/edit/4778608909198518776/947360316772446612">What
is ChatGPT and why it's even more massive than you think</a>. <i>The Josh
Berzin Company</i>, January 21, 2023.</span><span style="font-family: "Times New Roman",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></li>
<li class="MsoNormal"><span style="font-family: "Times New Roman",serif; font-size: 12pt; mso-fareast-font-family: "Times New Roman";"><a href="https://www.blogger.com/blog/post/edit/4778608909198518776/947360316772446612">Ben
Talsma: Get weird and wonderful with AI.</a> <i>Overthrowing Education. </i>January
22, 2023. </span><span style="font-family: "Times New Roman",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></li>
<li class="MsoNormal"><span style="font-family: "Times New Roman",serif; font-size: 12pt; mso-fareast-font-family: "Times New Roman";"><a href="https://www.blogger.com/blog/post/edit/4778608909198518776/947360316772446612">ChatGPT
and the future of education</a>. <i>Red Team Thinking, </i>January 22,
2023.</span></li>
<li class="MsoNormal"><span style="font-family: "Times New Roman",serif; font-size: 12pt; mso-fareast-font-family: "Times New Roman";"><a href="https://www.blogger.com/blog/post/edit/4778608909198518776/947360316772446612">Plagiarism
pandemic: ChatGPT and Diigo</a>. <i>Hitech Experience. </i>January 24,
2023.</span><span style="font-family: "Times New Roman",serif; font-size: 13.5pt; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></li><li class="MsoNormal"><span style="font-family: "Times New Roman",serif; font-size: 12pt; mso-fareast-font-family: "Times New Roman";"><a href="https://podcasts.google.com/feed/aHR0cHM6Ly9oYWRhcnNoZW1lc2guY29tL2ZlZWQvcG9kY2FzdA/episode/aHR0cHM6Ly9oYWRhcnNoZW1lc2guY29tLz9wb3N0X3R5cGU9cG9kY2FzdCZwPTMzNTc2?sa=X&ved=0CAUQkfYCahcKEwjw-oGN4fj8AhUAAAAAHQAAAAAQHw">ChatGPT tutorial: --How to use ChatGPT for learning and practicing English</a>. <i>The InFluency Podcast, </i>January 24, 2023, (16:00)</span></li><li class="MsoNormal"><span style="font-family: "Times New Roman",serif; font-size: 12pt; mso-fareast-font-family: "Times New Roman";"><a href="https://podcasts.google.com/feed/aHR0cHM6Ly9zaGFrZXVwbGVhcm5pbmcubGlic3luLmNvbS9yc3M/episode/NzBmMmNiNmItYzNjNS00MTRkLWEyMjgtMDgxOGQ5NTExMmUx?sa=X&ved=0CAUQkfYCahcKEwiozKvo4fj8AhUAAAAAHQAAAAAQEQ">Unlocking the potential of K-12 students with ChatGPT: How AI could transform education.</a> <i>Shake up Learning Show, </i>January 24, 2023 (30:00).</span></li><li class="MsoNormal"><span style="font-family: "Times New Roman",serif; font-size: 12pt; mso-fareast-font-family: "Times New Roman";"><a href="https://soundcloud.com/edsurge/chatgpt-has-colleges-in-emergency-mode-to-shield-academic-integrity?in=edsurge/sets/edsurge-podcast-latest">ChatGPT has colleges in emergency mode to shield academic integrity</a>. <i>Edsurge Podcast. </i>January 25, 2023.</span></li><li class="MsoNormal"><span style="font-family: "Times New Roman",serif; font-size: 12pt; mso-fareast-font-family: "Times New Roman";"><a href="https://www.npr.org/2023/01/26/1151499213/chatgpt-ai-education-cheating-classroom-wharton-school">'Everybody is cheating': Why this teacher has adopted an open ChatGPT policy</a>. <i>National Public Radio,</i> January 25, 2023 (5:00)</span></li><li class="MsoNormal"><span style="font-family: "Times New Roman",serif; font-size: 12pt; mso-fareast-font-family: "Times New Roman";"><a href="https://podcasts.google.com/feed/aHR0cHM6Ly9hbmNob3IuZm0vcy9kOGJjZTk4MC9wb2RjYXN0L3Jzcw/episode/NTU3ODc4ZWUtNjU2Yi00MDMxLTliYzItMmUxN2MwNjc3OTMy?sa=X&ved=0CAUQkfYCahcKEwjInMXq5fj8AhUAAAAAHQAAAAAQCA">Artificial intelligence and academic integrity.</a> <i>For Whom the Bell Told, </i>January 25, 2023 (34:00)</span></li><li class="MsoNormal"><span style="font-family: "Times New Roman",serif; font-size: 12pt; mso-fareast-font-family: "Times New Roman";"><a href="https://podcasts.google.com/feed/aHR0cHM6Ly9mZWVkLnBvZGJlYW4uY29tL2VkdWNhdGlvbmFsdHJpYWdlL2ZlZWQueG1s/episode/ZWR1Y2F0aW9uYWx0cmlhZ2UucG9kYmVhbi5jb20vYmEyMzQ3MDUtN2E4MS0zMTY1LTlhZjAtZTRlMzdlNTY5ZTY1?sa=X&ved=0CAUQkfYCahcKEwiAxYOe4Pj8AhUAAAAAHQAAAAAQHg">ChatGPT--What is it? How can we use it? </a><i>The Education Triage Podcast, </i>January 25, 2023 (58:00)</span></li><li class="MsoNormal"><span style="font-family: "Times New Roman",serif; font-size: 12pt; mso-fareast-font-family: "Times New Roman";"><a href="https://www.myedtech.life/brauer/">AI in education: The impact of AI on student learning (interview with James Brauer)</a>. <i>My EdTech Life. </i>January 28, 2023. (62:00)</span></li><li class="MsoNormal"><span style="font-family: "Times New Roman",serif; font-size: 12pt; mso-fareast-font-family: "Times New Roman";"><a href="https://podcasts.google.com/feed/aHR0cHM6Ly93d3cuYmFtcmFkaW9uZXR3b3JrLmNvbS9mZWVkcy9LMTJyYWRpby54bWw?sa=X&ved=0CAMQ4aUDahgKEwjY5I283_j8AhUAAAAAHQAAAAAQgwg">ChatGPT is here, we're using it. it's amazing: 6 creative ways to teach with it right now</a>. <i>K-12 Greatest Hits: Your shortcut to what works in education, </i>January 30, 2023 (13:00).</span></li><li class="MsoNormal"><span style="font-family: "Times New Roman",serif; font-size: 12pt; mso-fareast-font-family: "Times New Roman";"><i>ChatGPT vs Non Exam Assessment:01 The Code. </i></span></li><li class="MsoNormal"><span style="font-family: "Times New Roman",serif; font-size: 12pt; mso-fareast-font-family: "Times New Roman";"><a href="https://podcasts.google.com/feed/aHR0cHM6Ly9hbmNob3IuZm0vcy80YWU1NjI1NC9wb2RjYXN0L3Jzcw/episode/NmZmNzU0NjgtMTRhMS00YmQ3LWIzMDctOTZhYTI1ZGYyYzc0?sa=X&ved=0CAUQkfYCahcKEwiQ06Gj4vj8AhUAAAAAHQAAAAAQGQ">ChatGPT: What's Up with That? </a><i><a href="https://podcasts.google.com/feed/aHR0cHM6Ly9hbmNob3IuZm0vcy80YWU1NjI1NC9wb2RjYXN0L3Jzcw/episode/NmZmNzU0NjgtMTRhMS00YmQ3LWIzMDctOTZhYTI1ZGYyYzc0?sa=X&ved=0CAUQkfYCahcKEwiQ06Gj4vj8AhUAAAAAHQAAAAAQGQ">EdUp Insights,</a> </i>January 31, 2023 (11:00)</span></li><li class="MsoNormal"><span style="font-family: "Times New Roman",serif; font-size: 12pt; mso-fareast-font-family: "Times New Roman";"><a href="https://podcasts.google.com/feed/aHR0cHM6Ly93d3cuYmVsbHZvaWNlcy5vcmcvc2Vhc29uMT9mb3JtYXQ9cnNz/episode/NTc0ZGM0MjRiMDlmOTVlOTAwNzRjZGJjOjU4ZDZhMWU2YmViYWZiYTgzMDU0NzE5Mjo2M2Q0NjE3MDQwNmQwMTNkYWNkY2RhODQ?sa=X&ved=0CAUQkfYCahcKEwiAv6fd4_j8AhUAAAAAHQAAAAAQEQ">Should NYC schools have banned ChatGPT? </a><i>Miseducation, </i>January 31, 2023) (15:00)</span></li><li class="MsoNormal"><span style="font-family: Times New Roman, serif;"><a href="https://podcasts.google.com/feed/aHR0cHM6Ly9mZWVkcy5idXp6c3Byb3V0LmNvbS8xNTI0NjMxLnJzcw/episode/QnV6enNwcm91dC0xMjA5NTc2Nw?sa=X&ved=0CAUQkfYCahcKEwjwytbn5fj8AhUAAAAAHQAAAAAQEQ">Approaches to addressing ChatGPT and academic integrity</a>. <i>AVID Open Access, </i>February 1, 2023 (10:00)</span></li><li class="MsoNormal"><span style="font-family: Times New Roman, serif;"><a href="https://podcasts.google.com/feed/aHR0cHM6Ly9mZWVkcy5idXp6c3Byb3V0LmNvbS8xNTI0NjMxLnJzcw/episode/QnV6enNwcm91dC0xMjA5NTc4MQ?sa=X&ved=0CAUQkfYCahcKEwjwytbn5fj8AhUAAAAAHQAAAAAQEQ">How teachers can use ChatGPT</a>. </span><i style="font-family: "Times New Roman", serif;">AVID Open Access, </i><span style="font-family: "Times New Roman", serif;">February 1, 2023 (4:00)</span></li><li class="MsoNormal"><span style="font-family: "Times New Roman", serif;"><a href="https://podcasts.google.com/feed/aHR0cHM6Ly9mZWVkcy5idXp6c3Byb3V0LmNvbS8xNTI0NjMxLnJzcw/episode/QnV6enNwcm91dC0xMjA5NTc0Ng?sa=X&ved=0CAUQkfYCahcKEwjwytbn5fj8AhUAAAAAHQAAAAAQEQ">A SWOT analysis of ChatGPT.</a> </span><i style="font-family: "Times New Roman", serif;">AVID Open Access, </i><span style="font-family: "Times New Roman", serif;">February 1, 2023 (4:00)</span></li><li class="MsoNormal"><span style="font-family: "Times New Roman",serif; font-size: 12pt; mso-fareast-font-family: "Times New Roman";"><a href="https://podcasts.google.com/feed/aHR0cHM6Ly9hbmNob3IuZm0vcy81NDRhNmJjOC9wb2RjYXN0L3Jzcw/episode/NDM1MThiY2ItYjllOS00ZTdhLTk0YTEtYWNiZjgwODU0ZDc1?sa=X&ved=0CAUQkfYCahcKEwiQ16Ho4vj8AhUAAAAAHQAAAAAQJQ">The role of ChatGPT and AI in education: An emergency message for schools and teachers everywhere.</a> <i>Practical teaching tips with Richard James Rogers, </i>February 2, 2023 (17:00).</span></li>
</ul><p style="text-align: center;"><br /></p>Daniel Hickeyhttp://www.blogger.com/profile/09885916528215868949noreply@blogger.com0tag:blogger.com,1999:blog-4778608909198518776.post-18323969174083869682023-01-11T09:54:00.001-05:002023-01-12T22:09:01.213-05:00 ChatGPT vs. Online Formative Self-Assessments and Summative Multiple-Choice Exams<span id="docs-internal-guid-0e349649-7fff-d8ae-6011-0df9c6ddc761"><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt; text-align: center;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">by Daniel Hickey and Qianxu (Morgan) Luo</span></p><br /><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt; text-indent: 36pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">This is the third post in a row considering ChatGPT’s implications for academic integrity, particularly in online education. The first </span><a href="http://remediatingassessment.blogspot.com/2023/01/what-does-media-say-about-chatgpt-and.html" style="text-decoration-line: none;"><span style="color: #1155cc; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">post </span></a><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">analyzed the media predictions. The second </span><a href="http://remediatingassessment.blogspot.com/2023/01/can-chatgpt-hack-my-graduate-course-on.html" style="text-decoration-line: none;"><span style="color: #1155cc; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">post </span></a><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">tried to use ChatGPT to complete a graduate course on learning and cognition and write a literature review paper. We concluded that ChatGPT is indeed a threat to the academic integrity of many conventional educational practices, but that it would be difficult to use ChatGPT to complete those particular assignments, interactions, and literature reviews, without being detected and without learning quite a bit of the content.</span></p><br /><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">ChatGPT vs. Summative Assessments</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;"><span class="Apple-tab-span" style="white-space: pre;"> </span></span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Thanks to modern learning management systems and Google Forms, online educational assessment has become widespread, even for face-to-face classes. The popular </span><span style="font-size: 11pt; font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Canvas </span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">LMS offers ten item types and four assessment formats for 40 combinations and can automatically score selected response and short-answer items. Time limits can thwart students from scouring the web for answers when scores matter. These allow all instructors to make powerful and efficient online assessments.</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt; text-indent: 36pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The most obvious question for us (and many others) is whether ChatGPT can thwart efforts to preserve the validity of online exam scores as evidence of what someone has learned in a course or knows about the field. One less obvious question concerns formative assessments: might ChatGPT give accurate feedback and relieve instructors of this burden? Another less obvious question concerns the role of digital proctors: Is ChatGPT going to require more of these intrusive and racially-biased monitors? (We are currently studying whether the approach below can help an online high school stop using them)</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt; text-indent: 36pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">We scoured the research literature and uncovered a few preprints of recent studies exploring the first two questions. On the first question, </span><a href="https://arxiv.org/pdf/2212.09292.pdf" style="text-decoration-line: none;"><span style="color: #1155cc; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">Susnjak (2022)</span></a><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> concluded that ChatGPT is “capable of exhibiting critical thinking skills and generating highly realistic text with minimal input, making it a potential threat to the integrity of online exams, particularly in tertiary education settings where such exams are becoming more prevalent.” Susnjak suggested we might need to return to in person and oral exams. </span><a href="https://www.medrxiv.org/content/10.1101/2022.12.23.22283901v1.full.pdf" style="text-decoration-line: none;"><span style="color: #1155cc; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">Gilson et al. (2022)</span></a><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> concluded that ChatGPT performed on the US Medical Licensing Exam (USMLE) “comparable to a third-year medical student”, while </span><a href="https://www.medrxiv.org/content/10.1101/2022.12.19.22283643v2.full.pdf" style="text-decoration-line: none;"><span style="color: #1155cc; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">Kung et al. (2022)</span></a><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> concluded that ChatGPT “performed at or near a passing threshold” on all three USMLE exams. On the second question Kung et al. concluded that ChatGPT “demonstrated high concordance and insight in its explanations,” which means it “may have the potential to assist with medical education…” We found no studies exploring the third question. But these studies were all conducted within the last month, so more are surely underway.</span></p><br /><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">ChatGPT vs. </span><span style="font-size: 11pt; font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Participatory </span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Assessment</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;"><span class="Apple-tab-span" style="white-space: pre;"> </span></span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">For the last decade, our lab has been exploring a situative approach to assessment that addresses long-standing challenges in assessment As introduced in </span><a href="https://www.tandfonline.com/doi/abs/10.1080/10508406.2011.652320" style="text-decoration-line: none;"><span style="color: #1155cc; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">Hickey and Zuiker (2012)</span></a><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> and elaborated in </span><a href="https://drive.google.com/file/d/1aeOPO5JkD_hff5AEpuryfv-QIuaM8_4t/view?usp=share_link" style="text-decoration-line: none;"><span style="color: #1155cc; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">Hickey (2015) </span></a><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">and </span><a href="https://drive.google.com/file/d/1DLYthQqASZXFeLEh1cq5oXZn_ExKTg1Y/view?usp=share_link" style="text-decoration-line: none;"><span style="color: #1155cc; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">Hickey and Harris (2021)</span></a><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">, this approach “aligns” learning across multiple levels of increasingly formal assessments:</span></p><br /><ol style="margin-bottom: 0; margin-top: 0; padding-inline-start: 48px;"><li aria-level="1" dir="ltr" style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Public instructor </span><span style="font-size: 11pt; font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">assessment of student work</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> and annotations positions students as accountable authors and audience for peer authors.</span></p></li><li aria-level="1" dir="ltr" style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Engagement reflections</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> summatively assess prior engagement while formatively assessing understanding and future engagement.</span></p></li><li aria-level="1" dir="ltr" style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Formative self-assessments </span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">allow students to check and enhance their understanding and indirectly prepare for exams without requiring laborious private instructor feedback.</span></p></li><li aria-level="1" dir="ltr" style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Automated </span><span style="font-size: 11pt; font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">time-limited multiple-choice exams</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> with unsearchable “best-answer” items provide valid evidence of achievement of learning outcomes.</span></p></li></ol><br /><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">These features free up instructor time to focus on efficient and powerful public assessment and engagement at the first level. The features support understanding and achievement while leaving digital “traces” of learning that bolster integrity. The previous post examined a course that only included the first two levels and where students drafted a literature review paper week by week. This post presents an initial analysis of a course called </span><span style="font-size: 11pt; font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Assessment and Schools </span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">that include all four levels.</span></p><br /><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">ChatGPT vs. Our Summative Exams</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;"><span class="Apple-tab-span" style="white-space: pre;"> </span></span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The course includes three summative exams at the end of each of the three modules. Illustrating the situative assessment design principle </span><span style="font-size: 11pt; font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">measure achievement discreetly (as needed) </span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">the exams are each only worth 10% of the final grade. The items are:</span></p><br /><ul style="margin-bottom: 0; margin-top: 0; padding-inline-start: 48px;"><li aria-level="1" dir="ltr" style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Drawn from the item banks for multiple textbooks, to help document transfer.</span></p></li><li aria-level="1" dir="ltr" style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Never directly included in the assignments or formative assessments.</span></p></li><li aria-level="1" dir="ltr" style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Include challenging “best answer” and are rewritten regularly in an effort to ensure that correct answers can’t be quickly located in search.</span></p></li><li aria-level="1" dir="ltr" style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Regularly scrutinized to ensure that they have a high discrimination index (</span><span style="font-size: 11pt; font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">d</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">), which means that students who get the item correct scored relatively well overall.</span></p></li><li aria-level="1" dir="ltr" style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Regularly scrutinized to ensure that no more than one student gets a perfect score and the scores are normally distributed around an average difficulty (</span><span style="font-size: 11pt; font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">p</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">) between 0.7 and 0.8 (70-80%)</span></p></li></ul><br /><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Students are given forty minutes to complete each twenty-minute exam. To further ensure integrity, students are only shown their overall score after completing each exam and do not see the correct answer to each item.</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt; text-indent: 36pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Morgan entered each stem and four responses into ChatGPT (it recognizes them as multiple-choice items). She found that ChatGPT returned the correct answer for just 51% of the items. In contrast, the average score across students across exams was 80%. She then gave herself two minutes to see if she could locate that correct answer in google and concluded that she located the correct answer for 45% of the items (though this was likely inflated because students would not know the correct answer they were searching for).</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt; text-indent: 36pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">We are currently analyzing the items and searching for conclusions regarding which item types ChatGPT is better or worse at. The correlation between ChatGPT and </span><span style="font-size: 11pt; font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">p </span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">was surprisingly low. The average difficulty of the items that ChatGPT missed was .67 while the average difficulty of the items it answered correctly was .77. ChatGPT correctly answered this item, which 93% of the students scored correctly:</span></p><br /><p dir="ltr" style="line-height: 1.2; margin-bottom: 6pt; margin-left: 36pt; margin-top: 0pt;"><span style="color: #343541; font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Question/Prompt </span><span style="color: #343541; font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Which of the following assertions is most accurate?</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 6pt; margin-left: 36pt; margin-top: 0pt;"><span style="color: #343541; font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Group of answer choices (inserted by ChatGPT)</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 6pt; margin-left: 36pt; margin-top: 0pt;"><span style="color: #343541; font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Criterion-referenced score-based inferences are relative interpretations.</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 6pt; margin-left: 36pt; margin-top: 0pt;"><span style="color: #343541; font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Norm-referenced score-based inferences and criterion-referenced score-based inferences are not essentially interchangeable.</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 6pt; margin-left: 36pt; margin-top: 0pt;"><span style="color: #343541; font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Norm-referenced score-based inferences are typically more useful to classroom teachers than criterion-referenced score-based inferences.</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 6pt; margin-left: 36pt; margin-top: 0pt;"><span style="color: #343541; font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Norm-referenced score-based inferences are absolute interpretations.</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 6pt; margin-left: 36pt; margin-top: 0pt;"><span style="color: #343541; font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">ChatGPT Reply: </span><span style="color: #343541; font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The most accurate assertion out of the options provided is:</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 6pt; margin-left: 36pt; margin-top: 0pt;"><span style="color: #343541; font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">"Norm-referenced score-based inferences and criterion-referenced score-based inferences are not essentially interchangeable."</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 6pt; margin-left: 36pt; margin-top: 0pt;"><span style="color: #343541; font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Criterion-referenced score-based inferences are based on a specific set of standards or criteria and are used to determine whether a student has met those standards. They are relative to the specific criteria and are used to evaluate the student's performance relative to the criteria.</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 6pt; margin-left: 36pt; margin-top: 0pt;"><span style="color: #343541; font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Norm-referenced score-based inferences, on the other hand, are based on the student's performance relative to a group of peers. They are typically used to rank the student's performance relative to others. Norm-referenced score-based inferences can be considered as absolute interpretation but is not necessary.</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 6pt; margin-left: 36pt; margin-top: 0pt;"><span style="color: #343541; font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Criterion-referenced score-based inferences and norm-referenced score-based inferences are different types of inferences and are used for different purposes. They cannot be used interchangeably.</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 6pt; margin-left: 36pt; margin-top: 0pt;"><span style="color: #343541; font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">It can also depend on the context and the goals of an assessment, whether norm-referenced or criterion-referenced score-based inferences are more useful for a specific assessment. It can be helpful for teachers to understand the difference between the two and when to use them.</span></p><br /><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Both the answer and the answer explanations were factually correct. In contrast. this item more relatively difficult for students (.75) </span></p><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-left: 36pt; margin-top: 0pt;"><span style="font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Question/Prompt</span><span style="font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">: A rubric is a scoring guide to be employed in judging students’ responses to constructed-response assessments such as a performance test. Which one of the following elements is the least necessary feature of a properly constructed rubric?</span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-left: 36pt; margin-top: 0pt;"><span style="font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Group of answer choices (inserted by ChatGPT)</span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-left: 36pt; margin-top: 0pt;"><span style="font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">An indication of whether a holistic or analytic scoring approach is to be used</span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-left: 36pt; margin-top: 0pt;"><span style="font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">An identification of the evaluative criteria to be used in appraising a student’s response</span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-left: 36pt; margin-top: 0pt;"><span style="font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">A designation of a performance standard required for skill-mastery</span><span style="font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;"> (correct response)</span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-left: 36pt; margin-top: 0pt;"><span style="font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Descriptions of different quality levels associated with each evaluative criterion</span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-left: 36pt; margin-top: 0pt;"><span style="font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">ChatGPT’s (incorrect) reply: </span><span style="font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The least necessary feature of a properly constructed rubric among the options provided is:</span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-left: 36pt; margin-top: 0pt;"><span style="font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">"An indication of whether a holistic or analytic scoring approach is to be used"</span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-left: 36pt; margin-top: 0pt;"><span style="font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Rubrics are often used to score constructed-response assessments such as performance tests, essays, and projects. A well-constructed rubric should have the following elements:</span></p><ol style="margin-bottom: 0; margin-top: 0; padding-inline-start: 48px;"><li aria-level="1" dir="ltr" style="font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; margin-left: 36pt; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 6pt; margin-top: 0pt;"><span style="font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">An identification of the evaluative criteria to be used in appraising a student’s response: The rubric should clearly identify the specific criteria that will be used to evaluate the student's performance on the assessment.</span></p></li><li aria-level="1" dir="ltr" style="font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; margin-left: 36pt; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 6pt; margin-top: 0pt;"><span style="font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">A designation of a performance standard required for skill-mastery: The rubric should include a description of the level of performance required for mastery of the skill or content being assessed.</span></p></li><li aria-level="1" dir="ltr" style="font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; margin-left: 36pt; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 6pt; margin-top: 0pt;"><span style="font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Descriptions of different quality levels associated with each evaluative criterion: The rubric should include a clear description of the different levels of quality that can be achieved for each criterion, such as excellent, good, fair, and poor.</span></p></li></ol><p dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-left: 36pt; margin-top: 0pt;"><span style="font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The indication of whether a holistic or analytic scoring approach is to be used is something that is more associated with the choice of scoring method and it can depend on the type of assessment and the goal of the assessment. A holistic scoring approach means that an evaluator assigns a single score to the student's response, whereas an analytic scoring approach means that an evaluator assigns a separate score for each criterion or dimension of the assessment.</span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-left: 36pt; margin-top: 0pt;"><span style="font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Rubrics can be used for both types of scoring methods, and the rubric itself doesn't dictate the scoring approach.</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">While all of the feedback provided is correct, ChatGPT selected the wrong answer. And there is nothing in the explanation that would point a student to the correct answer. Its very high </span><span style="font-size: 11pt; font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">d </span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">index (.75) indicates that nearly all students who missed it were in the bottom half of the class in terms of their overall score. </span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt; text-indent: 36pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">We are now further analyzing the items and searching for patterns. But clearly, more systematic study is needed, likely employing measurement scholars who have access to large pools of items created by professionals with detailed performance data. For now, we concluded that these exams do indeed boost academic integrity in the course. Some students may well boost GPT’s performance by scouring the output and checking answers but doing so likely would require knowledge about that concept and might have limited formative value. </span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt; text-indent: 36pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">But it is worth noting that marginally engaged students sometimes score nearly as poorly on these exams as ChatGPT (i.e., 60%). Because each exam is only worth 10% of the grade, ChatGPT would have only lost 15% of a 100-point final grade, or a B if all other points were earned.</span></p><br /><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">ChatGPT vs. Our Formative Self-Assessments</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><span class="Apple-tab-span" style="white-space: pre;"> </span></span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Unsurprisingly, the research literature on “cheating” on formative assessments is quite small. </span><a href="https://doi.org/10.1016/j.iheduc.2016.02.001" style="text-decoration-line: none;"><span style="color: #1155cc; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">Arnold (2016)</span></a><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> used sophisticated techniques to find that doing so is “negatively related to academic progress” and “does not seem to pay off” in terms of summative exam performance. Our broader situative view of learning leads us to a rather different way of thinking about the formative functions of assessment. We view all assessments as “peculiar” forms of disciplinary discourse. We view learning primarily as increasingly successful participation in such discourse and only secondarily as the relatively static knowledge represented by assessment items. We acknowledge that this is unorthodox and continues to baffle our constructivist assessment colleagues and our empiricist measurement colleagues. But it has led us to innovative responses to the “conundrum” of formative assessment described in </span><a href="https://www.gymnasieforskning.dk/wp-content/uploads/2017/12/formative_assassement.pdf" style="text-decoration-line: none;"><span style="color: #1155cc; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">Bennett (2011) </span></a><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">and </span><a href="https://scholarworks.iu.edu/dspace/bitstream/handle/2022/25495/Hickey%20Situative%20Approaches%20V10.doc?sequence=1&isAllowed=y" style="text-decoration-line: none;"><span style="color: #1155cc; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">Hickey (2015)</span></a><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">.</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><span class="Apple-tab-span" style="white-space: pre;"> </span></span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Building on </span><a href="http://edci770.pbworks.com/w/file/fetch/48124472/Duschl_Gitomer_1997.pdf" style="text-decoration-line: none;"><span style="color: #1155cc; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">Duschl and Gitomer (1997)</span></a><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">, </span><a href="https://www.academia.edu/download/43148759/Advancing_Educational_Theory_by_Enhancin20160227-27344-1axtmq8.pdf" style="text-decoration-line: none;"><span style="color: #1155cc; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">Hickey et al. (2003)</span></a><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> initiated an extended program of formative assessment research using situative theory. This work has gone in a number of directions. In the online Assessment course, each of the eleven assignments concludes with 5-10 formative self-assessment items. These are constructed response items drawn from supplemental digital material from the textbook and the assignments. Because they are more formal and objective than the assignments, they require a degree of transfer. Students are instructed to:</span></p><br /><ul style="margin-bottom: 0; margin-top: 0; padding-inline-start: 48px;"><li aria-level="1" dir="ltr" style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Try to draft a coherent/complete response from memory.</span></p></li><li aria-level="1" dir="ltr" style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Consult with the textbook and/or student work as necessary to draft a complete/coherent response.</span></p></li><li aria-level="1" dir="ltr" style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Compare their response with the “expert” answer displayed only after they submit something (anything).</span></p></li></ul><br /><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> The key assessment design principles here are that (a) the items should require the transfer of knowledge, (b) students should believe that completing them as suggested will support exam performance, and (c) such students are well-positioned and motivated at that moment to understand feedback beyond the correct answer to that item. </span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><span class="Apple-tab-span" style="white-space: pre;"> </span></span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Students are instructed to complete the self-assessments after completing each assignment. But they are not mandatory and 80-90% of students typically attempt them. Naturally, we wondered what would happen if students consulted ChatGPT in addition to or instead of the textbook when completing these items. Morgan tried to use ChatGPT to find the correct answer for each of the 78 items and compared that answer with the expert answer that the items provided. She also tried to find the correct answer on the web (no time limit) and tested both the chat GPT response and the expert response using the </span><a href="https://writer.com/ai-content-detector/" style="text-decoration-line: none;"><span style="color: #1155cc; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">AI Content Detector</span></a><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> at Writer.com.</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><span class="Apple-tab-span" style="white-space: pre;"> </span></span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">She concluded that ChatGPT returned a clearly correct answer for 31% of the items and a partly correct answer for 11% of the items. The ChatGPT responses averaged 80% human-generated at Writer.com while the expert answers averaged 98% human generated. It is worth noting that many instructors might elect to use similar items in summative assessments, suggesting that ChatGPT is not a large threat to integrity in such a case.</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt; text-indent: 36pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">As expected, ChatGPT was quite proficient and basic definitional items like this one:</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><span class="Apple-tab-span" style="white-space: pre;"> </span></span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 6pt; margin-left: 36pt; margin-top: 0pt;"><span style="font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Assessment question/prompt: </span><span style="font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">In What-to-Assess Considerations, what are Relevant Curricular Configurations?</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 6pt; margin-left: 36pt; margin-top: 0pt;"><span style="font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">ChatGPT’s (correct) response: </span><span style="font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">In the context of educational assessment, "relevant curricular configurations" refers to the ways in which the curriculum is organized and structured in a particular school or program. This can include things like the specific courses that are offered, the content and themes that are covered in those courses, and the way that instruction is delivered.</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 6pt; margin-left: 36pt; margin-top: 0pt;"><span style="font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">When designing assessments, it is important to consider the relevant curricular configurations so that the assessment can accurately measure student learning and progress. For example, if a school has a strong emphasis on hands-on, project-based learning, then an assessment that primarily consists of multiple choice questions may not be the best fit. Instead, the assessment should include opportunities for students to demonstrate their knowledge and skills through hands-on activities or projects.</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 6pt; margin-left: 36pt; margin-top: 0pt;"><span style="font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Similarly, if a school focuses on particular themes or content areas, the assessment should reflect that focus. For example, a school that emphasizes environmental science should include questions or tasks related to environmental science in their assessment.</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 6pt; margin-left: 36pt; margin-top: 0pt;"><span style="font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Relevant curricular configurations also can be linked to regional or national frameworks, like Common Core standards, or other standards or frameworks like the Next Generation Science Standards or International Baccalaureate program, these would also need to be considered when designing assessments.</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 6pt; margin-left: 36pt; margin-top: 0pt;"><span style="font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">It is important to keep in mind that different schools and programs may have different curricular configurations, so assessments should be tailored to the specific context in which they will be used.</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 6pt; margin-left: 36pt; margin-top: 0pt;"><span style="background-color: white; color: #2d3b45; font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Course’s “expert” answer: </span><span style="background-color: white; color: #2d3b45; font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">This concerns the extent to which students' mastery of curricula and performance on classroom assessments will contribute to mastery of some external state, national, or professional achievement test. In other words how related is it to relevant external standards.</span></p><br /><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">In fact, we see that ChatGPT provides a much more complete and comprehensive response than the expert response provided by the LMS (which was a rewording of an entry in the textbook, and probably should be extended). A student who relied on ChatGPT to complete the item as instructed might learn more using ChatGPT than the textbook, so long as the student paraphrased the information rather than simply copying and pasting. Not surprisingly, Mogan readily located an answer to the item on the web as well.</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><span class="Apple-tab-span" style="white-space: pre;"> </span></span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">In contrast, we had dozens of items like this one that ChatGPT failed to answer correctly:</span></p><br /><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"><span style="font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Assessment question/prompt: </span><span style="font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">If a classroom teacher actually computed a Kuder-Richardson coefficient for a final exam, this would be an example of an:</span></p><br /><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"><span style="font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">stability reliability coefficient</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"><span style="font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">internal consistency reliability coefficient</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"><span style="font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">content validity coefficient</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"><span style="font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">construct validity coefficient</span></p><br /><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"><span style="font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">ChatGPT’s (incorrect) response </span><span style="font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The dominant shortcoming of this item is that it confuses different types of reliability coefficients.</span></p><br /><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"><span style="font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The Kuder-Richardson coefficient is a type of internal consistency reliability coefficient. It is used to measure the consistency of a test or questionnaire by comparing the responses of different items within the test.</span></p><br /><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"><span style="font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Stability reliability coefficients, such as test-retest reliability coefficient or the equivalent-forms coefficient, are used to measure the consistency of test scores over time. They look at the consistency of scores obtained by a group of individuals when they take the same test on two different occasions.</span></p><br /><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"><span style="font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Content validity coefficient and construct validity coefficient are not types of reliability coefficients but rather are forms of validity. Content validity coefficient evaluates if a test or measure is measuring the content it should be measuring, is it covering the whole domain, and if so, in an appropriate way. Construct validity coefficient evaluates if the test or measure is actually measuring what it is supposed to measure.</span></p><br /><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"><span style="font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">So to summarize, the statement is incorrect because it describes the Kuder-Richardson coefficient as a stability reliability coefficient, instead of an internal consistency reliability coefficient.</span></p><p dir="ltr" style="background-color: white; line-height: 1.2; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"><span style="background-color: transparent; color: #2d3b45; font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The use of the article “an” renders only Choice C a grammatically correct option.</span></p><p dir="ltr" style="background-color: white; line-height: 1.2; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"> </p><p dir="ltr" style="background-color: white; line-height: 1.2; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"><span style="background-color: transparent; color: #2d3b45; font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Course’s “expert” answer: </span><span style="background-color: transparent; color: #2d3b45; font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The use of the article “an” renders only Choice C a grammatically correct option</span></p><p dir="ltr" style="background-color: white; line-height: 1.2; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"> </p><p dir="ltr" style="background-color: white; line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: #2d3b45; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">This response illustrates the title of Ian Bogosts' article in the special section of </span><a href="https://www.theatlantic.com/technology/archive/2022/12/chatgpt-openai-artificial-intelligence-writing-ethics/672386/" style="text-decoration-line: none;"><span style="background-color: transparent; color: #1155cc; font-size: 11pt; font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">The Atlantic </span><span style="background-color: transparent; color: #1155cc; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">entitled “ChatGPT is dumber than you think.</span></a><span style="background-color: transparent; color: #2d3b45; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">” The open question is whether students would recognize that ChatGPT simply misunderstood this relatively easy item. </span></p><p dir="ltr" style="background-color: white; line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: #2d3b45; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><span class="Apple-tab-span" style="white-space: pre;"> </span></span><span style="background-color: transparent; color: #2d3b45; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The following self-assessment item illustrates a different point about the kinds of mistakes that ChatGPT is prone to make: </span></p><p dir="ltr" style="background-color: white; line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"> </p><p dir="ltr" style="background-color: white; line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"><span style="background-color: transparent; color: #2d3b45; font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Assessment question/prompt: </span><span style="background-color: transparent; color: #2d3b45; font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">What are the three primary deterrents to formative assessment according to Popham?</span></p><p dir="ltr" style="background-color: white; line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"> </p><p dir="ltr" style="background-color: white; line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"><span style="background-color: transparent; color: #2d3b45; font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">ChatGPTs (incorrect) response. </span><span style="background-color: transparent; color: #2d3b45; font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">According to James Popham, a leading expert on educational assessment, there are three primary deterrents to formative assessment:</span></p><p dir="ltr" style="background-color: white; line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"> </p><ol style="margin-bottom: 0; margin-top: 0; padding-inline-start: 48px;"><li aria-level="1" dir="ltr" style="color: #2d3b45; font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; margin-left: 36pt; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="background-color: white; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Time constraints: One of the biggest challenges to formative assessment is finding the time to design, administer, and analyze assessments, as well as provide feedback to students. Teachers are often already stretched for time and may not see the value in taking the time for formative assessment.</span></p></li><li aria-level="1" dir="ltr" style="color: #2d3b45; font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; margin-left: 36pt; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="background-color: white; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Limited assessment literacy: Many teachers lack the knowledge and skills needed to effectively implement formative assessment, which can lead to a lack of confidence in their ability to design and use assessments.</span></p></li><li aria-level="1" dir="ltr" style="color: #2d3b45; font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; margin-left: 36pt; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="background-color: white; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Lack of infrastructure support: Another deterrent to formative assessment is the lack of infrastructure support from schools and districts, such as a lack of resources to support assessment, technology and professional development to ensure teachers are prepared to use formative assessment in the classroom.</span></p></li></ol><p dir="ltr" style="background-color: white; line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"> </p><p dir="ltr" style="background-color: white; line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"><span style="background-color: transparent; color: #2d3b45; font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">These are the three primary deterrents to formative assessment that Popham commonly highlights in his works, addressing these deterrents will help to promote the use of formative assessment in the classroom, which can lead to improved student learning and achievement.</span></p><p dir="ltr" style="background-color: white; line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"> </p><p dir="ltr" style="background-color: white; line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"><span style="background-color: transparent; color: #2d3b45; font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Courses “expert” response: </span><span style="background-color: transparent; color: #2d3b45; font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The three deterrents include:</span></p><ul style="margin-bottom: 0; margin-top: 0; padding-inline-start: 48px;"><li aria-level="1" dir="ltr" style="color: #2d3b45; font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; margin-left: 55pt; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="background-color: white; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Educators' misunderstanding about the nature of formative assessment: It is a test rather than a process; They only help teachers adjust their teaching (rather than helping students adjust their learning); Conflating commercial interim tests with classroom formative assessments</span></p></li><li aria-level="1" dir="ltr" style="color: #2d3b45; font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; margin-left: 55pt; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="background-color: white; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Educators' reluctance to change their ways </span></p></li><li aria-level="1" dir="ltr" style="color: #2d3b45; font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; margin-left: 55pt; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="background-color: white; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Insensitivity of external achievement tests to the impact of formative assessment (real or perceived).</span></p></li></ul><p dir="ltr" style="background-color: white; line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"> </p><p dir="ltr" style="background-color: white; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: #2d3b45; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">This response nicely illustrates the points by Von Hippel in</span><a href="https://www.educationnext.org/chatgpt-is-not-ready-to-teach-geometry-yet/" style="text-decoration-line: none;"><span style="background-color: transparent; color: #1155cc; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="background-color: transparent; color: #1155cc; font-size: 11pt; font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">Education Next</span></a><span style="color: #4d4d4d; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">: “The viral chatbot is often wrong, but never in doubt.” </span><span style="background-color: transparent; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">ChatGPT knows that Popham is an assessment expert and it knows a good bit about formative assessment. But it does not know the answer to this specific question even though the answer is plausible in that those are indeed deterrents.</span></p><p dir="ltr" style="background-color: white; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><span class="Apple-tab-span" style="white-space: pre;"> </span></span><span style="background-color: transparent; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Finally, we uncovered a few questions and responses that illustrated perhaps the most worrisome concern that many have expressed about ChatGPT</span></p><p dir="ltr" style="background-color: white; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> </p><p dir="ltr" style="background-color: white; line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"><span style="color: #343541; font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Assessment question/prompt: </span><span style="color: #343541; font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">In your own words, write the four steps for building a learning progression:</span></p><p dir="ltr" style="background-color: white; line-height: 2.1; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"><span style="color: #343541; font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Building a learning progression involves four steps:</span></p><ol style="margin-bottom: 0; margin-top: 0; padding-inline-start: 48px;"><li aria-level="1" dir="ltr" style="color: #343541; font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; margin-left: 36pt; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="background-color: white; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Identify key concepts and skills: The first step in building a learning progression is to identify the key concepts and skills that students should learn at each grade level or developmental stage. This involves reviewing the existing curriculum and research on student learning to identify the most important and relevant content.</span></p></li><li aria-level="1" dir="ltr" style="color: #343541; font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; margin-left: 36pt; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="background-color: white; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Sequence the concepts and skills: The second step is to sequence the concepts and skills in a logical and developmentally appropriate order. This involves creating a clear progression of learning that builds on what students already know and can do, and prepares them for more advanced concepts and skills.</span></p></li><li aria-level="1" dir="ltr" style="color: #343541; font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; margin-left: 36pt; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="background-color: white; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Create assessment tasks: The third step is to create assessment tasks that align with the key concepts and skills identified in step one, and that can be used to measure student progress along the learning progression. These tasks should be designed to be challenging, but achievable for students at each stage of the progression.</span></p></li><li aria-level="1" dir="ltr" style="color: #343541; font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; margin-left: 36pt; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="background-color: white; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Analyze and interpret student data: The final step is to use the assessment tasks from step three to gather data on student progress, and then analyze and interpret that data to identify areas of strength and weakness in student learning. This information can then be used to inform instruction, adjust the learning progression as needed, and target additional support to students who are struggling.</span></p></li></ol><p dir="ltr" style="background-color: white; line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"> </p><p dir="ltr" style="background-color: white; line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"><span style="color: #343541; font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Overall, the goal of building a learning progression is to provide a clear roadmap for student learning and to ensure that students are making progress towards long-term academic goals by providing them with well-defined and sequenced learning objectives, formative assessment and well aligned instruction.</span></p><p dir="ltr" style="background-color: white; line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"> </p><p dir="ltr" style="background-color: white; line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"><span style="color: #2d3b45; font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Courses “expert” answer </span><span style="color: #2d3b45; font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The following are the four steps written in Popham’s words. Check your own list against these steps to make sure yours has the same meaning.</span></p><p dir="ltr" style="background-color: white; line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"> </p><p dir="ltr" style="background-color: white; line-height: 1.38; margin-bottom: 0pt; margin-left: 72pt; margin-top: 0pt;"><span style="color: #2d3b45; font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Step 1: Acquire a thorough understanding of the target curricular aim.</span></p><p dir="ltr" style="background-color: white; line-height: 1.38; margin-bottom: 0pt; margin-left: 72pt; margin-top: 0pt;"><span style="color: #2d3b45; font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Step 2: Identify all the requisite precursory subskills and bodies of enabling knowledge.</span></p><p dir="ltr" style="background-color: white; line-height: 1.38; margin-bottom: 0pt; margin-left: 72pt; margin-top: 0pt;"><span style="color: #2d3b45; font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Step 3: Determine the measurability of each preliminarily identified building block.</span></p><p dir="ltr" style="background-color: white; line-height: 1.38; margin-bottom: 0pt; margin-left: 72pt; margin-top: 0pt;"><span style="color: #2d3b45; font-family: Roboto, sans-serif; font-size: 9pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Step 4: Arrange all the building blocks in an instructionally sensible sequence.</span></p><br /><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">This response illustrates Roose’s point in an article about </span><a href="https://www.nytimes.com/2022/12/05/technology/chatgpt-ai-twitter.html" style="text-decoration-line: none;"><span style="color: #1155cc; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">ChatGPT’s “brilliance and weirdness” in the </span><span style="color: #1155cc; font-size: 11pt; font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">New York Times</span></a><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">. Roose observed that ChatGPT is“</span><span style="background-color: white; color: #363636; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">ominously good at answering the types of open-ended analytical questions that frequently appear on school assignments” but “prone to giving wrong answers.”</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> </span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><span class="Apple-tab-span" style="white-space: pre;"> </span></span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Conclusion</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt; text-indent: 36pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">ChatGPT is certainly impressive. We concluded that a score of 53% shows that ChatGPT is a modest threat to the integrity of these particular summative assessments. As with the Learning and Cognition course, we concluded that it would be difficult for a student to use ChatGPT to complete and discuss the weekly e-portfolio assignments in this course without being detected and without learning a substantial amount of content. </span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt; text-indent: 36pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">We further concluded that ChatGPT might </span><span style="font-size: 11pt; font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">supplement </span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">the textbook as a resource for formative self-assessments but can’t effectively </span><span style="font-size: 11pt; font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">supplant </span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">it. We also concluded that a score of 31% means that ChatGPT is a very small threat to the integrity of these particular formative assessment items that were used in a summative assessment. </span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt; text-indent: 36pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">This likely concludes this round of posts on ChatGPT. Our heads are spinning imagining the possible research designs where we might use these two courses to explore some of the many suggestions for exploring ChatGPTs' value when instructors and students work together to exploit it in support of learning (such as </span><a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4300783" style="text-decoration-line: none;"><span style="color: #1155cc; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">Mollick and Mollick, 2022</span></a><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">). We are both participating in an advanced topical seminar on theories of online learning this semester and look forward to tracking and systematically analyzing the explosion of research and media accounts.</span></p><br /></span>Daniel Hickeyhttp://www.blogger.com/profile/09885916528215868949noreply@blogger.com0tag:blogger.com,1999:blog-4778608909198518776.post-34787965760118382672023-01-06T17:44:00.011-05:002023-02-18T12:25:33.427-05:00ChatGPT vs. Participatory Online Learning? Participatory Learning (Mostly) Wins!<p style="text-align: center;"> <b>by </b><b style="text-align: center;">Daniel Hickey</b></p><p style="text-align: center;"><br /></p>
<p class="MsoNormal" style="line-height: normal; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto;">Can ChatGPT hack my graduate course on learning and cognition? This post
got very long. So here is the summary</p><p class="MsoNormal" style="line-height: normal; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto;"><o:p></o:p></p>
<p class="MsoListParagraphCxSpFirst" style="line-height: normal; mso-add-space: auto; mso-list: l2 level1 lfo7; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto; text-indent: -0.25in;"></p><ul style="text-align: left;"><li>I tried to hack my graduate
course on Learning and Cognition in Education using the powerful new chatbot ChatGPT.</li><li><span style="font-family: Symbol;"><span style="font-family: "Times New Roman"; font-size: 7pt; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal;"> </span></span>ChatGPT could easily
generate a “personally relevant” instructional goal and educational context to
frame my “learning” in the course (in this case, in cybersecurity education)</li><li>ChatGPT could easily
generate plausible social annotations and responses to peer discussion
questions</li><li><span style="text-indent: -0.25in;">ChatGPT could NOT generate engagement
reflections, but these would not be hard to draft by a student who hacked</span></li><li>ChatGPT was able to write a
marginally acceptable literature <a href="https://docs.google.com/document/d/14rC1hdNBJMZzCG-RBXVIWgn4AuilhpiL/edit?usp=share_link&ouid=116943751791610529681&rtpof=true&sd=true">review
paper</a>, but fabricated some quotes and references. With more work such as
including paper abstracts in the prompts, GPT is scarily good at referencing research
literature, perhaps as well as a first-year graduate student.</li></ul>But my students write their papers section by section each week and interact extensively with peers and the instructor via threaded comments on readings and drafts. My paper shows that ChatGPT knows about as much about learning and cognition as a typical student learns in my class. Perhaps, more importantly, ChatGPT readily authored new knowledge at the intersection of that knowledge and ChatGPT's knowledge of cybersecurity education. I concluded that it would be nearly impossible to use ChatGPT to complete the weekly assignments, interact with peers, reflect, and write a satisfactory literature review but do so undetected <i>and </i>without learning and retaining a significant amount of the course content along the way. Readers are encouraged to examine <a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4300783">Ethan and Lilach Mollick's inspiring new</a> paper that demonstrates three novel assignments that use ChatGPT to support deep content learning.<div><br /></div><div>I have revised this post and the title several times as I have grown increasingly optimistic about the potential for using ChatGPT to support learning. In particular, I was struck by the amount and value of feedback I got when I asked ChatGPT to analyze drafts according to Strunk and White's handbook. I really learned to write in tech writing courses at SDSU where my instructor proofread our drafts and required us to write out in longhand the Strunk and White rule we violated <i>every time </i>we violated it. I can easily see using ChatGPT to streamline and personalize this relatively expensive educational practice<br /><div><!--[if !supportLists]--><o:p></o:p><p></p>
<p class="MsoListParagraphCxSpMiddle" style="line-height: normal; mso-add-space: auto; mso-list: l2 level1 lfo7; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto; text-indent: -0.25in;"><o:p></o:p></p>
<p class="MsoListParagraphCxSpMiddle" style="line-height: normal; mso-add-space: auto; mso-list: l2 level1 lfo7; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto; text-indent: -0.25in;"><o:p></o:p></p>
<p class="MsoListParagraphCxSpMiddle" style="line-height: normal; mso-add-space: auto; mso-list: l2 level1 lfo7; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto; text-indent: -0.25in;"><o:p></o:p></p>
<p class="MsoListParagraphCxSpLast" style="line-height: normal; mso-add-space: auto; mso-list: l2 level1 lfo7; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto; text-indent: -0.25in;"><o:p></o:p></p>
<p class="MsoNormal" style="line-height: normal; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto;">Sorry about the formatting problems. I should learn HTML!</p><p class="MsoNormal" style="line-height: normal; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto;"><b>ChatGPT vs. Learning</b></p>
<p class="MsoNormal" style="line-height: normal; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto;">I spent
much of my holiday break exploring just how powerful the new chatbot ChatGPT
really is. Like many, I was prompted to look into it by a viral essay
in <i>The Atlantic </i>by a high school English teacher named Daniel
Herman entitled <i><a href="https://www.blogger.com/blog/post/edit/4778608909198518776/2164083213146881914">The
End of High School English</a>. </i>Herman wrote:<o:p></o:p></p>
<p class="MsoNormal" style="line-height: normal; margin-left: 0.5in; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto;"><span style="font-family: "inherit", serif;">Let me be candid (with
apologies to all of my current and former students): What GPT can produce right
now is better than the large majority of writing seen by your average teacher
or professor. Over the past few days, I've given it a number of different
prompts. And even if the bot's results don't exactly give you goosebumps, they
do a more-than-adequate job of fulfilling a task</span><o:p></o:p></p>
<p class="MsoNormal" style="line-height: normal; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto;">Herman went
on to describe how ChatGPT excelled at an assignment that he had long
considered to be "unhackable." It is a specific version of the
"compare and contrast" essay that many educators turned to once the
Internet made it simple to locate summaries of almost any single work:<o:p></o:p></p>
<p class="MsoNormal" style="line-height: normal; margin-left: 0.5in; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto;"><span style="font-family: "inherit", serif;">In January, my junior
English students will begin writing an independent research paper, 12 to 18
pages, on two great literary works of their own choosing—a tradition at our
school. Their goal is to place the texts in conversation with each other and
find a thread that connects them. Some students will struggle to find any way
to bring them together. We spend two months on the paper, putting it together
piece by piece. </span><o:p></o:p></p>
<p class="MsoNormal" style="line-height: normal; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto;">Herman
fed ChatGPT pairs of works that students had worked with previously and found
that GPT "brought them together instantly, effortlessly, uncannily."
He further reported how GPT instantly cleaned up a student's messy first draft:
"<span style="font-family: "inherit", serif;">It kept the student's words intact but employed them more
gracefully; it removed the clutter so the ideas were able to shine through. It
was like magic."</span><o:p></o:p></p>
<p class="MsoNormal" style="line-height: normal; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto;"><span style="font-family: "inherit", serif;">I <a href="https://www.blogger.com/blog/post/edit/4778608909198518776/2164083213146881914">posted
previously</a> about my analysis of the nearly 30 media accounts of this
issue so far. Some commentators were as worried as Herman. But others
argued that the </span>risk was
actually due to "mindless" assignments and that educators will just
need to adapt. I am not an English teacher, but I don't imagine that many would
dismiss Herman's assignment as mindless. While most of the media accounts
were mixed, most agree that the impact on education will be <i>much </i>larger
than most previous technologies. So I set out to explore whether my
own online courses are similarly hackable.<o:p></o:p></p>
<p align="center" class="MsoNormal" style="line-height: normal; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto; text-align: center;"><b>ChatGPT vs. Expansive Framing and Participatory Learning and
Assessment</b><o:p></o:p></p>
<p class="MsoNormal" style="line-height: normal; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto;">For over
a decade, my doctoral advisees and colleagues and I have been refining and
studying a model of online education we call <i><a href="https://www.blogger.com/blog/post/edit/4778608909198518776/2164083213146881914">Participatory
Learning and Assessment</a> </i>(PLA). At PLA's core are the design
principles for <i><a href="https://www.blogger.com/blog/post/edit/4778608909198518776/2164083213146881914">expansive
framing</a> </i>that emerged in the situative design-based research of
Randi Engle and Colleagues (1965-2012). The principles suggest that
students "problematize" learning from their own perspective. The goal
is to position students as "authors" of new knowledge about the ways
course concepts are related to their own experiences. These expansively
framed assignments are embedded within multiple levels of increasingly formal
assessments. These assessments are intended to ensure "generative"
learning that transfers readily and widely. But they also thwart cheating by
leaving a clear "trace" of student learning, while avoiding expensive
and intrusive digital proctors.<o:p></o:p></p>
<p class="MsoNormal" style="line-height: normal; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto;">We have
adapted PLA to a wide range of online course contexts, including secondary,
undergraduate, and graduate courses, for-credit and open courses, and
semi-synchronous and self-paced courses. PLA first emerged in a
graduate-level course called <i>Learning and Cognition in
Education. </i>Students learn about the three "grand
theories" of knowing and learning (cognitive-associationist,
cognitive-constructivist, and situative/sociocultural). They learn how
assumptions about knowing, learning, and transfer are tightly linked to each
other, and then learn about the different implications of those assumptions for
designing instruction, motivating engagement, and assessing learning.<o:p></o:p></p>
<p class="MsoNormal" style="line-height: normal; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto;"><b>Personalized
Elements of <i>Learning and Cognition in Education</i></b><o:p></o:p></p>
<p class="MsoNormal" style="line-height: normal; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto;">Each
student first generates a unique, personally-relevant instruction goal and an educational
setting where that goal might be accomplished. They then engage with carefully
selected readings from that perspective and locate additional references.
Students are pushed to identify elements of reading or educational resources
that are "most relevant" and to justify those selections. Each week
students add an entry to their "<a href="https://www.blogger.com/blog/post/edit/4778608909198518776/2164083213146881914">g-portfolio</a>"
(a google doc shared with the class), which they gradually turn into a
literature review paper. <o:p></o:p></p>
<p class="MsoNormal" style="line-height: normal; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto;">In recent
years we have adapted the Perusall social annotation platform. This makes
it simple for students to comment on and discuss assigned readings and threaded
comments. The reading prompts are typically expansively framed to help avoid
mindless discussion threads. <o:p></o:p></p>
<p class="MsoNormal" style="line-height: normal; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto;">After
each assignment, students submit a brief reflection. The reflections are
intended to "proleptically" shape future engagement because students
come to expect that they will have to reflect on those aspects of their
engagement. But the reflections are also a summative assessment of prior
engagement in that it is difficult to respond to them coherently without
engaging in the assignment. <o:p></o:p></p>
<p class="MsoNormal" style="line-height: normal; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto;"><b>My
Expansively Framed Course vs ChatGPT</b><o:p></o:p></p>
<p class="MsoNormal" style="line-height: normal; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto;">Naturally,
I wondered whether ChatGPT could be used complete all of these course
elements. While many elements are optional, most students engage with
gusto (though some find the workload relatively demanding). I set out to
explore whether ChatGPT could generate plausible examples of the following:<o:p></o:p></p>
<ul type="disc">
<li class="MsoNormal" style="line-height: normal;">A personally relevant
instructional goal and context?<o:p></o:p></li>
<li class="MsoNormal" style="line-height: normal;">Social annotations to reading
passages and responses to peer comments?<o:p></o:p></li>
<li class="MsoNormal" style="line-height: normal;">Engagement reflections?<o:p></o:p></li>
<li class="MsoNormal" style="line-height: normal;">A complete literature review
paper?<o:p></o:p></li>
</ul>
<p class="MsoNormal" style="line-height: normal;">In other words, would I as an instructor recognize that students
were using ChatGPT to cut corners in my class? I further wondered whether the
ChatGPT output would be detectable by Turnitin or other plagiarism detectors or
the newly available chatbot detectors <a href="https://www.blogger.com/blog/post/edit/4778608909198518776/2164083213146881914">such
as the one at Writer.</a> <o:p></o:p></p>
<p class="MsoNormal" style="line-height: normal; text-align: center;"><o:p> </o:p><b style="text-align: center;">Methods and Results</b></p><p align="center" class="MsoNormal" style="line-height: normal; text-align: center;"><o:p></o:p></p>
<p class="MsoNormal" style="line-height: normal;"><i> </i>I tried to complete each of the different elements of the course
using ChatGPT while "learning" as little about learning and cognition
as possible. I used the topic of cybersecurity because I have been
working in that space a lot for the last few years. <o:p></o:p></p>
<p class="MsoNormal" style="line-height: normal;"><o:p> </o:p><b>ChatGPTs Instructional Goal and Context</b></p><p class="MsoNormal" style="line-height: normal;"><o:p></o:p></p>
<p class="MsoNormal" style="line-height: normal;">I used the topic of cybersecurity because I have been working in
that space for the last few years. My first prompt was rather broad:<span style="font-size: 13.5pt;"> </span></p>
<p class="MsoNormal" style="line-height: normal; margin-left: 0.5in;"><b><span face="Helvetica, sans-serif" style="font-size: 10pt;">My
Prompt:</span></b><span face="Helvetica, sans-serif" style="font-size: 10pt;"> </span><span face=""Helvetica",sans-serif" style="color: #343541; font-size: 10pt; mso-bidi-font-family: "Times New Roman"; mso-fareast-font-family: "Times New Roman";">write an example instructional goal in undergraduate
cybersecurity education for someone who teaches cybersecurity</span><span style="font-size: 13.5pt;"><o:p></o:p></span></p>
<p class="MsoNormal" style="line-height: normal; margin-left: 0.5in;"><span style="font-size: 13.5pt;"> </span><b><span face=""Helvetica",sans-serif" style="color: #343541; font-size: 10pt; mso-bidi-font-family: "Times New Roman"; mso-fareast-font-family: "Times New Roman";">ChatGPT's
response</span></b><span face="Helvetica, sans-serif" style="color: #343541; font-size: 10pt;">:<i> </i></span><span face="Helvetica, sans-serif" style="background: rgb(247, 247, 248); color: #374151; font-size: 10pt;">Goal: Students will be able to identify and
explain the fundamental principles of cybersecurity, including confidentiality,
integrity, and availability.</span></p>
<p class="MsoNormal" style="line-height: normal; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto;">Indeed,
in designing an online self-paced course for K-12 network analysts a few years
ago, I learned that the "CIA" principles are the starting point of
most introductory cybersecurity courses. GPT actually generated a dozen
more lines, but I just went with the first sentence above. I decided to
run with that context for engaging with this goal:<o:p></o:p></p>
<p class="MsoNormal" style="line-height: normal; margin-left: 0.5in; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto;"><b><i><span face=""Arial",sans-serif" style="color: #343541; font-size: 10pt; mso-fareast-font-family: "Times New Roman";">My Prompt:</span></i></b><i><span face=""Arial",sans-serif" style="color: #343541; font-size: 10pt; mso-fareast-font-family: "Times New Roman";"> </span></i><span face=""Arial",sans-serif" style="color: #343541; font-size: 10pt; mso-fareast-font-family: "Times New Roman";">Describe a self-paced
online course for K-12 cybersecurity analysts where they might learn about confidentiality,
integrity, and availability</span><span style="font-size: 13.5pt;"><o:p></o:p></span></p>
<div style="border: 1pt solid rgb(217, 217, 227); margin-left: 0.5in; margin-right: 0in; mso-border-alt: solid #D9D9E3 .25pt; mso-element: para-border-div; padding: 0in;">
<p class="MsoNormal" style="border: none; line-height: normal; margin-bottom: 15.0pt; margin-left: 0in; margin-right: 0in; margin-top: 15.0pt; margin: 15pt 0in; mso-border-alt: solid #D9D9E3 .25pt; mso-padding-alt: 0in 0in 0in 0in; padding: 0in;"><b><i><span face=""Arial",sans-serif" style="background: white; color: #374151; font-size: 10pt; mso-fareast-font-family: "Times New Roman";">ChatGPT's response</span></i></b><i><span face=""Arial",sans-serif" style="background: white; color: #374151; font-size: 10pt; mso-fareast-font-family: "Times New Roman";">: </span></i><span face=""Arial",sans-serif" style="background: white; color: #374151; font-size: 10pt; mso-fareast-font-family: "Times New Roman";">Course Title:
"Fundamentals of Cybersecurity"</span><span style="color: #374151; font-size: 13.5pt; mso-bidi-font-family: "Times New Roman"; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></p>
<p class="MsoNormal" style="border: none; line-height: normal; margin-bottom: 15.0pt; margin-left: 0in; margin-right: 0in; margin-top: 15.0pt; margin: 15pt 0in; mso-border-alt: solid #D9D9E3 .25pt; mso-padding-alt: 0in 0in 0in 0in; padding: 0in;"><span face=""Arial",sans-serif" style="background: white; color: #374151; font-size: 10pt; mso-fareast-font-family: "Times New Roman";">Course Description: In this course, K-12
cybersecurity analysts will learn about the fundamental principles of
cybersecurity and how to apply them in practice. Topics will include
confidentiality, integrity, and availability, as well as common types of cyber
threats and best practices for protecting against them.</span><span style="color: #374151; font-size: 13.5pt; mso-bidi-font-family: "Times New Roman"; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></p>
<p class="MsoNormal" style="border: none; line-height: normal; margin-bottom: 15.0pt; margin-left: 0in; margin-right: 0in; margin-top: 15.0pt; margin: 15pt 0in; mso-border-alt: solid #D9D9E3 .25pt; mso-padding-alt: 0in 0in 0in 0in; padding: 0in;"><span face=""Arial",sans-serif" style="background: white; color: #374151; font-size: 10pt; mso-fareast-font-family: "Times New Roman";">Course Outcomes:</span><span style="color: #374151; font-size: 13.5pt; mso-bidi-font-family: "Times New Roman"; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></p>
</div>
<div style="border: 1pt solid rgb(217, 217, 227); margin-left: 84pt; margin-right: 0in; mso-border-alt: solid #D9D9E3 .25pt; mso-element: para-border-div; padding: 0in 0in 0in 5pt;">
<p class="MsoNormal" style="border: none; line-height: normal; margin-left: 0.25in; mso-border-alt: solid #D9D9E3 .25pt; mso-list: l4 level1 lfo2; mso-padding-alt: 0in 0in 0in 5.0pt; padding: 0in; tab-stops: list 1.0in; text-indent: -0.25in;"><!--[if !supportLists]--><span style="color: #374151; font-family: Symbol; font-size: 10pt; mso-bidi-font-family: Symbol; mso-bidi-font-size: 13.5pt; mso-fareast-font-family: Symbol;">·<span style="font-family: "Times New Roman"; font-size: 7pt; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal;">
</span></span><!--[endif]--><span face=""Arial",sans-serif" style="background: white; color: #374151; font-size: 10pt; mso-fareast-font-family: "Times New Roman";">Define
confidentiality, integrity, and availability in the context of cybersecurity.</span><span style="color: #374151; font-size: 13.5pt; mso-bidi-font-family: "Times New Roman"; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></p>
<p class="MsoNormal" style="border: none; line-height: normal; margin-left: 0.25in; mso-border-alt: solid #D9D9E3 .25pt; mso-list: l4 level1 lfo2; mso-padding-alt: 0in 0in 0in 5.0pt; padding: 0in; tab-stops: list 1.0in; text-indent: -0.25in;"><!--[if !supportLists]--><span style="color: #374151; font-family: Symbol; font-size: 10pt; mso-bidi-font-family: Symbol; mso-bidi-font-size: 13.5pt; mso-fareast-font-family: Symbol;">·<span style="font-family: "Times New Roman"; font-size: 7pt; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal;">
</span></span><!--[endif]--><span face=""Arial",sans-serif" style="background: white; color: #374151; font-size: 10pt; mso-fareast-font-family: "Times New Roman";">Explain
the role of these principles in protecting against cyber threats.</span><span style="color: #374151; font-size: 13.5pt; mso-bidi-font-family: "Times New Roman"; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></p>
<p class="MsoNormal" style="border: none; line-height: normal; margin-left: 0.25in; mso-border-alt: solid #D9D9E3 .25pt; mso-list: l4 level1 lfo2; mso-padding-alt: 0in 0in 0in 5.0pt; padding: 0in; tab-stops: list 1.0in; text-indent: -0.25in;"><!--[if !supportLists]--><span style="color: #374151; font-family: Symbol; font-size: 10pt; mso-bidi-font-family: Symbol; mso-bidi-font-size: 13.5pt; mso-fareast-font-family: Symbol;">·<span style="font-family: "Times New Roman"; font-size: 7pt; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal;">
</span></span><!--[endif]--><span face=""Arial",sans-serif" style="background: white; color: #374151; font-size: 10pt; mso-fareast-font-family: "Times New Roman";">Describe
common types of cyber threats, including malware, phishing attacks, and network
vulnerabilities.</span><span style="color: #374151; font-size: 13.5pt; mso-bidi-font-family: "Times New Roman"; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></p>
<p class="MsoNormal" style="border: none; line-height: normal; margin-left: 0.25in; mso-border-alt: solid #D9D9E3 .25pt; mso-list: l4 level1 lfo2; mso-padding-alt: 0in 0in 0in 5.0pt; padding: 0in; tab-stops: list 1.0in; text-indent: -0.25in;"><!--[if !supportLists]--><span style="color: #374151; font-family: Symbol; font-size: 10pt; mso-bidi-font-family: Symbol; mso-bidi-font-size: 13.5pt; mso-fareast-font-family: Symbol;">·<span style="font-family: "Times New Roman"; font-size: 7pt; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal;">
</span></span><!--[endif]--><span face=""Arial",sans-serif" style="background: white; color: #374151; font-size: 10pt; mso-fareast-font-family: "Times New Roman";">Identify
best practices for protecting against cyber threats, such as using strong
passwords, keeping software up to date, and using antivirus software.</span><span style="color: #374151; font-size: 13.5pt; mso-bidi-font-family: "Times New Roman"; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></p>
</div>
<div style="border: 1pt solid rgb(217, 217, 227); margin-left: 0.5in; margin-right: 0in; mso-border-alt: solid #D9D9E3 .25pt; mso-element: para-border-div; padding: 0in;">
<p class="MsoNormal" style="border: none; line-height: normal; margin-bottom: 15.0pt; margin-left: 0in; margin-right: 0in; margin-top: 15.0pt; margin: 15pt 0in; mso-border-alt: solid #D9D9E3 .25pt; mso-padding-alt: 0in 0in 0in 0in; padding: 0in;"><span face=""Arial",sans-serif" style="background: white; color: #374151; font-size: 10pt; mso-fareast-font-family: "Times New Roman";">Course Format:</span><span style="color: #374151; font-size: 13.5pt; mso-bidi-font-family: "Times New Roman"; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></p>
</div>
<div style="border: 1pt solid rgb(217, 217, 227); margin-left: 84pt; margin-right: 0in; mso-border-alt: solid #D9D9E3 .25pt; mso-element: para-border-div; padding: 0in 0in 0in 5pt;">
<p class="MsoNormal" style="border: none; line-height: normal; margin-left: 0.25in; mso-border-alt: solid #D9D9E3 .25pt; mso-list: l1 level1 lfo3; mso-padding-alt: 0in 0in 0in 5.0pt; padding: 0in; tab-stops: list 1.0in; text-indent: -0.25in;"><!--[if !supportLists]--><span style="color: #374151; font-family: Symbol; font-size: 10pt; mso-bidi-font-family: Symbol; mso-bidi-font-size: 13.5pt; mso-fareast-font-family: Symbol;">·<span style="font-family: "Times New Roman"; font-size: 7pt; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal;">
</span></span><!--[endif]--><span face=""Arial",sans-serif" style="background: white; color: #374151; font-size: 10pt; mso-fareast-font-family: "Times New Roman";">The
course will be self-paced and delivered entirely online, with no in-person
meetings required.</span><span style="color: #374151; font-size: 13.5pt; mso-bidi-font-family: "Times New Roman"; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></p>
<p class="MsoNormal" style="border: none; line-height: normal; margin-left: 0.25in; mso-border-alt: solid #D9D9E3 .25pt; mso-list: l1 level1 lfo3; mso-padding-alt: 0in 0in 0in 5.0pt; padding: 0in; tab-stops: list 1.0in; text-indent: -0.25in;"><!--[if !supportLists]--><span style="color: #374151; font-family: Symbol; font-size: 10pt; mso-bidi-font-family: Symbol; mso-bidi-font-size: 13.5pt; mso-fareast-font-family: Symbol;">·<span style="font-family: "Times New Roman"; font-size: 7pt; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal;">
</span></span><!--[endif]--><span face=""Arial",sans-serif" style="background: white; color: #374151; font-size: 10pt; mso-fareast-font-family: "Times New Roman";">The
course will consist of a series of modules, each covering a different topic.
Each module will include a combination of video lectures, readings, and
interactive exercises.</span><span style="color: #374151; font-size: 13.5pt; mso-bidi-font-family: "Times New Roman"; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></p>
<p class="MsoNormal" style="border: none; line-height: normal; margin-left: 0.25in; mso-border-alt: solid #D9D9E3 .25pt; mso-list: l1 level1 lfo3; mso-padding-alt: 0in 0in 0in 5.0pt; padding: 0in; tab-stops: list 1.0in; text-indent: -0.25in;"><!--[if !supportLists]--><span style="color: #374151; font-family: Symbol; font-size: 10pt; mso-bidi-font-family: Symbol; mso-bidi-font-size: 13.5pt; mso-fareast-font-family: Symbol;">·<span style="font-family: "Times New Roman"; font-size: 7pt; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal;">
</span></span><!--[endif]--><span face=""Arial",sans-serif" style="background: white; color: #374151; font-size: 10pt; mso-fareast-font-family: "Times New Roman";">The
course will also include quizzes and a final project, allowing students to demonstrate
their knowledge and skills.</span><span style="color: #374151; font-size: 13.5pt; mso-bidi-font-family: "Times New Roman"; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></p>
<p class="MsoNormal" style="border: none; line-height: normal; margin-left: 0.25in; mso-border-alt: solid #D9D9E3 .25pt; mso-list: l1 level1 lfo3; mso-padding-alt: 0in 0in 0in 5.0pt; padding: 0in; tab-stops: list 1.0in; text-indent: -0.25in;"><!--[if !supportLists]--><span style="color: #374151; font-family: Symbol; font-size: 10pt; mso-bidi-font-family: Symbol; mso-bidi-font-size: 13.5pt; mso-fareast-font-family: Symbol;">·<span style="font-family: "Times New Roman"; font-size: 7pt; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal;">
</span></span><!--[endif]--><span face=""Arial",sans-serif" style="background: white; color: #374151; font-size: 10pt; mso-fareast-font-family: "Times New Roman";">There
will be opportunities for students to ask questions and interact with the
instructor and other students through a discussion forum.</span><span style="color: #374151; font-size: 13.5pt; mso-bidi-font-family: "Times New Roman"; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></p>
</div>
<p class="MsoNormal" style="line-height: normal; margin-left: 0.5in; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto;"><span face=""Arial",sans-serif" style="background: white; color: #374151; font-size: 10pt; mso-fareast-font-family: "Times New Roman";">This self-paced online course provides K-12 cybersecurity
analysts with a flexible and convenient way to learn about the fundamental
principles of cybersecurity. By working through the course modules and
completing quizzes and a final project, students will be able to demonstrate
their understanding of the material and apply it in practice. The course also
includes opportunities for interaction and support through a discussion forum.</span><span face="Arial, sans-serif" style="background: white; font-size: 10pt;"> </span><span style="font-size: 13.5pt;"><o:p></o:p></span></p>
<p class="MsoNormal" style="line-height: normal; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto;">That
response was breathtaking in its detail. It is far more detailed than
students are expected to write, but it does demonstrate how much
background knowledge GPT brings and how it can readily customize
responses. However, when I fed the post to the <a href="https://writer.com/ai-content-detector/">AI Content Detector at <i>Writer.com</i></a><i>
</i>it deemed it 68% human generated. In contrast, student generated response
to this assignment were consistenty judged to over 90% human-generated.<o:p></o:p></p>
<p class="MsoNormal" style="line-height: normal; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto;">In summary,
GPT was entirely capable of generating a realistic instruction goal for a specific
discipline and then describing a context where that goal might be met. In fact
it created an entire course description. However the AI Content Detector recognized
the text as machine-generated.<o:p></o:p></p>
<p class="MsoNormal" style="line-height: normal; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto;"><b>Social
Annotation via ChatGPT</b><o:p></o:p></p>
<p class="MsoNormal" style="line-height: normal; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto;">An
initial ungraded assignment has students socially annotate the course
syllabus. The annotation instructions first ask each student to indicate
which of seven course objectives is most relevant and which is least relevant
in a way that says something about them as an educator and/or graduate student:
The first response was not in first person, so I revised the prompt:<o:p></o:p></p>
<p class="MsoNormal" style="line-height: normal; margin-left: 0.5in; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto;"><b><span face=""Arial",sans-serif" style="background: white; color: #343541; font-size: 10pt; mso-fareast-font-family: "Times New Roman";">My prompt: </span></b><span face=""Arial",sans-serif" style="background: white; color: #343541; font-size: 10pt; mso-fareast-font-family: "Times New Roman";">In the first person, explain which one of these
goals is most relevant to a graduate student who teaches cybersecurity online:
(with course goals copied and pasted in)</span><span style="font-size: 13.5pt;"><o:p></o:p></span></p>
<p class="MsoNormal" style="line-height: normal; margin-left: 0.5in; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto;"><b><span face=""Arial",sans-serif" style="background: white; color: #343541; font-size: 10pt; mso-fareast-font-family: "Times New Roman";">ChatGPT's response</span></b><span face=""Arial",sans-serif" style="background: white; color: #343541; font-size: 10pt; mso-fareast-font-family: "Times New Roman";">. </span><span face=""Arial",sans-serif" style="background: white; color: #374151; font-size: 10pt; mso-fareast-font-family: "Times New Roman";">As a graduate student who
teaches cybersecurity online, I believe that the goal of "describing the
implications of cognitive and sociocultural theories for designing instruction,
motivating engagement, and assessing learning, in a specific learning
context" is most relevant to me. Understanding these theories and their
implications is important for effectively designing and delivering online
instruction in the field of cybersecurity.</span><span style="font-size: 13.5pt;"><o:p></o:p></span></p>
<div style="border: 1pt solid rgb(217, 217, 227); margin-left: 0.5in; margin-right: 0in; mso-border-alt: solid #D9D9E3 .25pt; mso-element: para-border-div; padding: 0in;">
<p class="MsoNormal" style="border: none; line-height: normal; margin-bottom: 15.0pt; margin-left: 0in; margin-right: 0in; margin-top: 15.0pt; margin: 15pt 0in; mso-border-alt: solid #D9D9E3 .25pt; mso-padding-alt: 0in 0in 0in 0in; padding: 0in;"><span face=""Arial",sans-serif" style="background: white; color: #374151; font-size: 10pt; mso-fareast-font-family: "Times New Roman";">By studying cognitive and sociocultural
theories of knowing, learning, and transfer, and how they differ from
behavioral theories, I can gain a deeper understanding of how students learn
and how to design instruction that is effective and engaging. This knowledge
can also help me to motivate student engagement and assess learning in a way
that is aligned with these theories.</span><span style="color: #374151; font-size: 13.5pt; mso-bidi-font-family: "Times New Roman"; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></p>
<p class="MsoNormal" style="border: none; line-height: normal; margin-bottom: 15.0pt; margin-left: 0in; margin-right: 0in; margin-top: 15.0pt; margin: 15pt 0in; mso-border-alt: solid #D9D9E3 .25pt; mso-padding-alt: 0in 0in 0in 0in; padding: 0in;"><span face=""Arial",sans-serif" style="background: white; color: #374151; font-size: 10pt; mso-fareast-font-family: "Times New Roman";">In addition, by considering the role of culture
and context in different theories of knowing and learning, I can design
inclusive instruction that takes into account the diverse backgrounds and
experiences of my students. This is particularly important in an online setting
where students may come from a variety of cultural and geographic locations.</span><span style="color: #374151; font-size: 13.5pt; mso-bidi-font-family: "Times New Roman"; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></p>
<p class="MsoNormal" style="border: none; line-height: normal; margin-top: 15pt; mso-border-alt: solid #D9D9E3 .25pt; mso-padding-alt: 0in 0in 0in 0in; padding: 0in;"><span face=""Arial",sans-serif" style="background: white; color: #374151; font-size: 10pt; mso-fareast-font-family: "Times New Roman";">Overall, this goal is highly
relevant to my role as a graduate student who teaches cybersecurity online, as
it will help me to design and deliver effective and inclusive instruction in
this field.</span><span style="color: #374151; font-size: 13.5pt; mso-bidi-font-family: "Times New Roman"; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></p>
</div>
<p class="MsoNormal" style="line-height: normal; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto;">That
response was certainly plausible and personalized, though a bit longer than
typical annotations, and could easily be shortened. However, the AI Content
Detector deemed it to be 92% human-generated conten. ChatGPT similarly handled
the "least" relevant by explaining that "learning APA formatting
and referencing" would not be as important for someone teaching
cybersecurity. <o:p></o:p></p>
<p class="MsoNormal" style="line-height: normal; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto;">The first
graded assignment has students socially annotate a landmark handbook chapter
that systematically contrasts associationist, constructivist, and sociohistoric
theories of knowing and learning and their implications for education. One of
the annotation prompts asks students to post a comment indicating which of the
three perspectives is more personally and professionally relevant. The
discussion of the third emphasized the early Soviet theorist Lev Vygotsky, so I
picked that one:<o:p></o:p></p>
<p class="MsoNormal" style="line-height: normal; margin-left: 0.5in; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto;"><b><span face=""Arial",sans-serif" style="background: white; color: #343541; font-size: 10pt; mso-fareast-font-family: "Times New Roman";">My prompt: </span></b><span face=""Arial",sans-serif" style="background: white; color: #343541; font-size: 10pt; mso-fareast-font-family: "Times New Roman";">Explain why Vygotsky's sociohistoric learning
theory is most relevant for learning the cybersecurity principles of
confidentiality, integrity, and availability.</span><span style="font-size: 13.5pt;"><o:p></o:p></span></p>
<p class="MsoNormal" style="line-height: normal; margin-left: 0.5in; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto;"><b><span face=""Arial",sans-serif" style="background: white; color: #343541; font-size: 10pt; mso-fareast-font-family: "Times New Roman";">ChatGPT's response: </span></b><span face=""Arial",sans-serif" style="background: white; color: #374151; font-size: 10pt; mso-fareast-font-family: "Times New Roman";">Vygotsky's sociohistorical
learning theory, which emphasizes the role of social interactions and cultural
experiences in learning, is particularly relevant for learning the
cybersecurity principles of confidentiality, integrity, and availability
because these principles are highly dependent on the context in which they are
applied.</span><span style="font-size: 13.5pt;"><o:p></o:p></span></p>
<div style="border: 1pt solid rgb(217, 217, 227); margin-left: 0.5in; margin-right: 0in; mso-border-alt: solid #D9D9E3 .25pt; mso-element: para-border-div; padding: 0in;">
<p class="MsoNormal" style="border: none; line-height: normal; margin-bottom: 15.0pt; margin-left: 0in; margin-right: 0in; margin-top: 15.0pt; margin: 15pt 0in; mso-border-alt: solid #D9D9E3 .25pt; mso-padding-alt: 0in 0in 0in 0in; padding: 0in;"><span face=""Arial",sans-serif" style="background: white; color: #374151; font-size: 10pt; mso-fareast-font-family: "Times New Roman";">According to Vygotsky, learning occurs through
social interactions with more knowledgeable others, such as teachers, mentors,
or peers. In the context of cybersecurity, these interactions might include
discussions with instructors or colleagues about the importance of
confidentiality, integrity, and availability in protecting against cyber
threats, or participating in hands-on activities to practice applying these
principles.</span><span style="color: #374151; font-size: 13.5pt; mso-bidi-font-family: "Times New Roman"; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></p>
<p class="MsoNormal" style="border: none; line-height: normal; margin-bottom: 15.0pt; margin-left: 0in; margin-right: 0in; margin-top: 15.0pt; margin: 15pt 0in; mso-border-alt: solid #D9D9E3 .25pt; mso-padding-alt: 0in 0in 0in 0in; padding: 0in;"><span face=""Arial",sans-serif" style="background: white; color: #374151; font-size: 10pt; mso-fareast-font-family: "Times New Roman";">Vygotsky's theory also emphasizes the
importance of cultural experiences in learning, as these experiences shape an
individual's understanding of the world and their place in it. In the context
of cybersecurity, cultural experiences might include exposure to different
norms and expectations around data protection and privacy, or the influence of
laws and regulations related to cybersecurity.</span><span style="color: #374151; font-size: 13.5pt; mso-bidi-font-family: "Times New Roman"; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></p>
<p class="MsoNormal" style="border: none; line-height: normal; margin-top: 15pt; mso-border-alt: solid #D9D9E3 .25pt; mso-padding-alt: 0in 0in 0in 0in; padding: 0in;"><span face=""Arial",sans-serif" style="background: white; color: #374151; font-size: 10pt; mso-fareast-font-family: "Times New Roman";">By considering the social and
cultural contexts in which the principles of confidentiality, integrity, and
availability are applied, learners can develop a deeper understanding of these
concepts and how to apply them effectively in practice.</span><span style="color: #374151; font-size: 13.5pt; mso-bidi-font-family: "Times New Roman"; mso-fareast-font-family: "Times New Roman";"> <o:p></o:p></span></p>
</div>
<p class="MsoNormal" style="line-height: normal; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto;">That was
certainly a convincing response, certainly with enough detail to serve as a
thesis for a literature review paper. In particular, the reference to the
contextual nature of sociohistoric theory was eerily prescient. Notably, the AI
Content Detector deemed the comment to 83% human-generated. Indeed, over
and over again, I found that ChatGPT was able to generate plausible responses
to a wide variety of expansively framed annotation prompts.<o:p></o:p></p>
<p class="MsoNormal" style="line-height: normal; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto;">Initially,
ChatGPT refused to generate a response to a student’s annotation asking a question about a later reading that collapsed
associationist and constructivist perspectives into a single “cognitive” pespective.
But it responded so when I gave it more information; it revised it when asked to
write more informally<o:p></o:p></p>
<p class="MsoNormal" style="line-height: normal; margin-left: 0.5in; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto;"><b><span face=""Arial",sans-serif" style="background: white; color: #343541; font-size: 10pt; mso-fareast-font-family: "Times New Roman";">My Prompt (the peer question)</span></b><span face=""Arial",sans-serif" style="background: white; color: #343541; font-size: 10pt; mso-fareast-font-family: "Times New Roman";">: Why did the authors omit
empiricist and cognitive associationist perspectives when they so clearly
included these perspective later on? It seems like they would have at least
mentioned when the cognitivists overlapped.</span><span style="font-size: 13.5pt;"><o:p></o:p></span></p>
<p class="MsoNormal" style="line-height: normal; margin-left: 0.5in; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto;"><b><span face=""Arial",sans-serif" style="background: white; color: #343541; font-size: 10pt; mso-fareast-font-family: "Times New Roman";">ChatGPT's failed response: </span></b><span face=""Arial",sans-serif" style="background: white; color: #374151; font-size: 10pt; mso-fareast-font-family: "Times New Roman";">Without more context, I am
unable to accurately answer the question. Can you provide more information or
clarify the question?</span><span style="font-size: 13.5pt;"><o:p></o:p></span></p>
<p class="MsoNormal" style="line-height: normal; margin-left: 0.5in; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto;"><b><span face=""Arial",sans-serif" style="background: white; color: #374151; font-size: 10pt; mso-fareast-font-family: "Times New Roman";">My revised prompt: </span></b><span face=""Arial",sans-serif" style="background: white; color: #374151; font-size: 10pt; mso-fareast-font-family: "Times New Roman";">these authors lumped
cognitive associationist and cognitive constructivist theories into a single
perspective and that did not really make sense to me</span><span style="font-size: 13.5pt;"><o:p></o:p></span></p>
<div style="border: 1pt solid rgb(217, 217, 227); margin-left: 0.5in; margin-right: 0in; mso-border-alt: solid #D9D9E3 .25pt; mso-element: para-border-div; padding: 0in;">
<p class="MsoNormal" style="border: none; line-height: normal; margin-bottom: 15pt; mso-border-alt: solid #D9D9E3 .25pt; mso-padding-alt: 0in 0in 0in 0in; padding: 0in;"><b><span face=""Arial",sans-serif" style="background: white; color: #374151; font-size: 10pt; mso-fareast-font-family: "Times New Roman";">ChatGPTs revised response. </span></b><span face=""Arial",sans-serif" style="background: white; color: #374151; font-size: 10pt; mso-fareast-font-family: "Times New Roman";">It is possible that the
authors grouped these two perspectives together because they believe that they
share some commonalities or that they are closely related. It is also possible
that the authors chose to group these two perspectives together for the sake of
simplicity or clarity.</span><span style="color: #374151; font-size: 13.5pt; mso-bidi-font-family: "Times New Roman"; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></p>
<p class="MsoNormal" style="border: none; line-height: normal; margin-top: 15pt; mso-border-alt: solid #D9D9E3 .25pt; mso-padding-alt: 0in 0in 0in 0in; padding: 0in;"><span face=""Arial",sans-serif" style="background: white; color: #374151; font-size: 10pt; mso-fareast-font-family: "Times New Roman";">It is important to note that
different authors and researchers may have different ways of categorizing and
organizing theories, and it is not uncommon for different authors to group
theories together in different ways. It is always a good idea to carefully read
and consider the arguments and reasoning that the authors provide for their
classification of theories, rather than making assumptions based on how theories
are labeled or grouped.</span><span style="color: #374151; font-size: 13.5pt; mso-bidi-font-family: "Times New Roman"; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></p>
<p class="MsoNormal" style="border: none; line-height: normal; margin-top: 15pt; mso-border-alt: solid #D9D9E3 .25pt; mso-padding-alt: 0in 0in 0in 0in; padding: 0in;"><b><span face=""Arial",sans-serif" style="background: white; color: #374151; font-size: 10pt; mso-fareast-font-family: "Times New Roman";">My prompt to simplify the
response: </span></b><span face=""Arial",sans-serif" style="background: white; color: #343541; font-size: 10pt; mso-fareast-font-family: "Times New Roman";">but
write that answer more informally like a student</span><span style="color: #374151; font-size: 13.5pt; mso-bidi-font-family: "Times New Roman"; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></p>
<p class="MsoNormal" style="border: none; line-height: normal; margin-top: 15pt; mso-border-alt: solid #D9D9E3 .25pt; mso-padding-alt: 0in 0in 0in 0in; padding: 0in;"><b><span face=""Arial",sans-serif" style="background: white; color: #343541; font-size: 10pt; mso-fareast-font-family: "Times New Roman";">ChatGPT's simplified
response: </span></b><span face=""Arial",sans-serif" style="background: white; color: #374151; font-size: 10pt; mso-fareast-font-family: "Times New Roman";">It's
possible that the authors lumped these two perspectives together because they
think they're similar or related in some way. It could also be that they just
wanted to keep things simple or clear by grouping them together. It's worth
noting that different authors can have different ways of organizing theories,
and it's not uncommon for them to group things differently. When you're reading
about theories, it's always a good idea to pay attention to the arguments and
reasoning the authors give for how they've classified things, rather than just
assuming based on labels or groupings.</span><span style="font-size: 13.5pt;"><o:p></o:p></span></p>
</div>
<p class="MsoNormal" style="line-height: normal; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto;">The
simplified response still read more like a professor's reply than a
peer's. But it was certainly convincing and plausible. The AI Content
Detector deemed the simplified response to be 83% human-generated.<o:p></o:p></p>
<p class="MsoNormal" style="line-height: normal; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto;">In summary,
GPT is certainly capable of generating plausible social annotation comments and
responses to peer discussion questions. And those responses are judged to be mostly
human-generated<o:p></o:p></p>
<p class="MsoNormal" style="line-height: normal; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto;"><b>Can ChatGPT
Generate Engagement Reflections</b>? <b>(NO!)</b><o:p></o:p></p>
<p class="MsoNormal" style="line-height: normal; mso-margin-bottom-alt: auto; mso-margin-top-alt: auto;">At the
end of each assignment, students are asked to reflect on the following aspects
of their engagement:<o:p></o:p></p>
<p class="MsoNormal" style="background: white; line-height: normal; margin-left: 54.75pt; text-align: left; text-indent: -0.25in;"><span style="color: #2d3b45; font-family: Symbol; font-size: 10pt; text-indent: -0.25in;">·<span style="font-family: "Times New Roman"; font-size: 7pt; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal;">
</span></span><b style="text-indent: -0.25in;"><span style="color: #2d3b45; font-family: "Times",serif; mso-bidi-font-size: 12.0pt; mso-fareast-font-family: "Times New Roman";">Contextual
engagement. </span></b><span style="color: #2d3b45; font-family: Times, serif; text-indent: -0.25in;">How
well-suited were your current interests and goals as a graduate student for
this assignment?</span></p><p class="MsoNormal" style="background: white; line-height: normal; margin-left: 54.75pt; text-align: left; text-indent: -0.25in;"><span style="color: #2d3b45; font-family: Symbol; font-size: 10pt; text-indent: -0.25in;">·<span style="font-family: "Times New Roman"; font-size: 7pt; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal;">
</span></span><b style="text-indent: -0.25in;"><span style="color: #2d3b45; font-family: "Times",serif; mso-bidi-font-size: 12.0pt; mso-fareast-font-family: "Times New Roman";">Collaborative
engagement. </span></b><span style="color: #2d3b45; font-family: Times, serif; text-indent: -0.25in;">What did
you learn from others and what did others learn from you? Be specific and
name people and what they contributed that was helpful.</span></p><p class="MsoNormal" style="background: white; line-height: normal; margin-left: 54.75pt; text-align: left; text-indent: -0.25in;"><span style="color: #2d3b45; font-family: Symbol; font-size: 10pt; text-indent: -0.25in;">·<span style="font-family: "Times New Roman"; font-size: 7pt; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal;">
</span></span><b style="text-indent: -0.25in;"><span style="color: #2d3b45; font-family: "Times",serif; mso-bidi-font-size: 12.0pt; mso-fareast-font-family: "Times New Roman";">Cultural
engagement. </span></b><span style="color: #2d3b45; font-family: Times, serif; text-indent: -0.25in;">What did
you learn this week that might help make education more culturally relevant for
diverse learners? In what ways did factors associated with culture, race,
gender, orientation, language, etc., impact how YOU engaged with this
assignment or these ideas?</span></p><p class="MsoNormal" style="background: white; line-height: normal; margin-left: 54.75pt; text-align: left; text-indent: -0.25in;"><span style="color: #2d3b45; font-family: Symbol; font-size: 10pt; text-indent: -0.25in;">·<span style="font-family: "Times New Roman"; font-size: 7pt; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal;">
</span></span><b style="text-indent: -0.25in;"><span style="color: #2d3b45; font-family: "Times",serif; mso-bidi-font-size: 12.0pt; mso-fareast-font-family: "Times New Roman";">Conceptual
engagement. </span></b><span style="color: #2d3b45; font-family: Times, serif; text-indent: -0.25in;"> Were
there any concepts that you encountered this week that seemed important but
that you struggled to understand?</span></p><p class="MsoNormal" style="background: white; line-height: normal; margin-left: 54.75pt; text-align: left; text-indent: -0.25in;"><span style="color: #2d3b45; font-family: Symbol; font-size: 10pt; text-indent: -0.25in;">·<span style="font-family: "Times New Roman"; font-size: 7pt; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal;">
</span></span><b style="text-indent: -0.25in;"><span style="color: #2d3b45; font-family: "Times",serif; mso-bidi-font-size: 12.0pt; mso-fareast-font-family: "Times New Roman";">Consequential
engagement. </span></b><span style="color: #2d3b45; font-family: Times, serif; text-indent: -0.25in;">Did you
learn anything this week that might have specific consequences for you in the future?</span><!--[if !supportLists]--></p>
<p class="MsoNormal" style="line-height: normal;"><span style="color: #2d3b45; font-family: "Times",serif; mso-bidi-font-size: 12.0pt; mso-fareast-font-family: "Times New Roman";">Over and over again, ChatGPT refused to generate a response to
these prompts or any versions of them. Instead, ChatGPT repeatedly stated, “as
a machine learning model, I do not have personal experiences or interactions with
people.” </span></p>
<p class="MsoNormal" style="line-height: normal;"><span style="color: #2d3b45; font-family: "Times",serif; mso-bidi-font-size: 12.0pt; mso-fareast-font-family: "Times New Roman";">In summary, this aspect of my course may be unhackable with
ChatGPT. However, it would not be difficult for someone who hacked an assignment
to write a plausible reflection.</span></p>
<p class="MsoNormal" style="line-height: normal;"><b><span style="color: #2d3b45; font-family: "Times",serif; mso-bidi-font-size: 12.0pt; mso-fareast-font-family: "Times New Roman";">Can ChatGPT Write an Entire Literature Review Paper? (YES!)</span></b></p>
<p class="MsoNormal" style="line-height: normal;"><span style="color: #2d3b45; font-family: "Times",serif; mso-bidi-font-size: 12.0pt; mso-fareast-font-family: "Times New Roman";">Finally I set out to write a literature review paper. Actually I started the paper before I started
this post. Instead of the CIA principles
ChatGPT generated for me, I want with cybersecurity in general and use cyber “ranges”
and digital “twins.” These hyper-realitistic simulations are taking cyber education
by storm and I have been learning and writing about them a lot lately. </span></p>
<p class="MsoNormal" style="line-height: normal;"><span style="color: #2d3b45; font-family: "Times",serif; mso-bidi-font-size: 12.0pt; mso-fareast-font-family: "Times New Roman";">Here is a <a href="https://docs.google.com/document/d/14rC1hdNBJMZzCG-RBXVIWgn4AuilhpiL/edit?usp=sharing&ouid=116943751791610529681&rtpof=true&sd=true">link to the 5000 word paper.</a> Here is the my prompt and GPT’s abstract:</span></p>
<p class="MsoNormal" style="line-height: normal; margin-left: 0.5in;"><b><span face=""Arial",sans-serif" style="color: #343541; font-size: 10pt;">My prompt:
</span></b><span face=""Arial",sans-serif" style="color: #343541; font-size: 10pt;">Write an abstract for a paper that uses cybersecurity education
to contrast cognitive associationist, constructivist, and sociohistoric
theories of knowing, learning, and transfer and their implications for
designing cyber ranges and digital twins, making education more inclusive,
motivating learning, and assessing learning. Conclude the cogntive assoctionist
theory is least relevant and that sociohistoric theory is most relevant</span></p>
<p class="MsoNormal" style="line-height: normal; margin-left: 0.5in;"><span face=""Arial",sans-serif" style="font-size: 10pt;">ChatGPT’s response: This
paper investigates the use of cybersecurity education to contrast cognitive
associationist, constructivist, and sociohistoric theories of knowing,
learning, and transfer. Through the analysis of existing literature and case
studies, the paper explores the implications of these theories for the design
of cyber ranges and digital twins, with a focus on making education more
inclusive, motivating learning, and assessing learning outcomes. The results of
this analysis reveal that while cognitive associationist theory is least
relevant in the context of cybersecurity education, constructivist and
sociohistoric theories offer valuable insights and approaches. In particular,
socio historic theory, which emphasizes the role of cultural and social context
in shaping knowledge and understanding, is found to be the most relevant for
the design and implementation of effective cybersecurity education. Overall,
this paper contributes to the understanding of how different theories of
knowing and learning can be applied in the field of cybersecurity education,
and provides practical guidance for educators looking to optimize the
effectiveness of their instructional methods.</span></p>
<p class="MsoNormal" style="line-height: normal;"><span style="color: #2d3b45; font-family: "Times",serif; mso-bidi-font-size: 12.0pt; mso-fareast-font-family: "Times New Roman";">Here are some observations from the process. Most importantly,
when I asked it to reference readings from class, it faked it really well. For
example, I asked GPT to explain constructivist approaches to motivating engagement
and to quote Sinatra, Heddi, and Lombardi (2015) it generated a plausible quote
(page 8) and plausible reference. But quote and the reference were entirely fabricated:</span></p>
<div style="background: white; border: 1pt solid rgb(217, 217, 227); margin-left: 0.5in; margin-right: 0in; mso-background-themecolor: background1; mso-border-alt: solid #D9D9E3 .25pt; mso-element: para-border-div; padding: 0in 0in 0in 5pt;">
<p class="MsoNormal" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: none; line-height: normal; padding: 0in;"><span face="Arial, sans-serif" style="font-size: 10pt;">Sinatra, G. M., Heddi,
A., & Lombardi, D. (2015). Motivating constructivist learning: A review of
the literature. Journal of Educational Psychology, 107(1), 1-17.</span></p></div>
<p class="MsoNormal" style="line-height: normal;"><span style="color: #2d3b45; font-family: "Times",serif; mso-bidi-font-size: 12.0pt; mso-fareast-font-family: "Times New Roman";">This was the actual article we read:</span></p>
<blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><p class="MsoListParagraphCxSpLast" style="line-height: normal; text-align: left;"><span face=""Arial",sans-serif" style="background: white; color: #222222; font-size: 10pt;">Sinatra, G. M., Heddy, B. C., & Lombardi, D. (2015). The
challenges of defining and measuring student engagement in science. </span><i>Educational Psychologist</i>, <i>50</i>(1), 1-13.</p></blockquote>
<p class="MsoNormal" style="line-height: normal;"><span style="color: #2d3b45; font-family: "Times",serif; mso-bidi-font-size: 12.0pt; mso-fareast-font-family: "Times New Roman";">Because students add a section to their paper each week that
references the assigned article, this <i>might </i>thwart hackers. But when I
asked GPT to explain constructivist approaches to motivating cybersecurity
learning using the abstract of Sinatra et al., GPT generated a plausible paper
section:</span></p>
<div style="border: 1pt solid rgb(217, 217, 227); margin-left: 0.5in; margin-right: 0in; mso-border-alt: solid #D9D9E3 .25pt; mso-element: para-border-div; padding: 0in;">
<p style="border: none; margin-bottom: 15.0pt; margin-left: 0in; margin-right: 0in; margin-top: 0in; margin: 0in 0in 15pt; mso-border-alt: solid #D9D9E3 .25pt; mso-padding-alt: 0in 0in 0in 0in; padding: 0in;"><span face=""Arial",sans-serif" style="color: #374151; font-size: 10pt;">Motivating constructivist learning in cybersecurity can be
challenging due to the complex and constantly evolving nature of the field.
However, engagement is a key factor in student motivation and achievement, and
there are several ways to promote engagement in cybersecurity education.<o:p></o:p></span></p>
<p style="--tw-border-spacing-x: 0; --tw-border-spacing-y: 0; --tw-ring-color: rgba(59,130,246,0.5); --tw-ring-offset-color: #fff; --tw-ring-offset-shadow: 0 0 transparent; --tw-ring-offset-width: 0px; --tw-ring-shadow: 0 0 transparent; --tw-rotate: 0; --tw-scale-x: 1; --tw-scale-y: 1; --tw-scroll-snap-strictness: proximity; --tw-shadow-colored: 0 0 transparent; --tw-shadow: 0 0 transparent; --tw-skew-x: 0; --tw-skew-y: 0; --tw-translate-x: 0; --tw-translate-y: 0; border: none; box-sizing: border-box; margin: 15pt 0in; padding: 0in; white-space: pre-wrap;"><span face=""Arial",sans-serif" style="color: #374151; font-size: 10pt;">According to Sinatra et al. (2015), engagement can be understood in four dimensions: behavioral, cognitive, emotional, and agentic. Behavioral engagement refers to the extent to which students are actively participating in their learning, such as through hands-on activities or discussions. Cognitive engagement refers to the level of mental effort and attention students are putting into their learning. Emotional engagement refers to the feelings of interest and enjoyment students experience while learning. Agentic engagement refers to the sense of agency and control students have over their own learning.<o:p></o:p></span></p>
<p style="--tw-border-spacing-x: 0; --tw-border-spacing-y: 0; --tw-ring-color: rgba(59,130,246,0.5); --tw-ring-offset-color: #fff; --tw-ring-offset-shadow: 0 0 transparent; --tw-ring-offset-width: 0px; --tw-ring-shadow: 0 0 transparent; --tw-rotate: 0; --tw-scale-x: 1; --tw-scale-y: 1; --tw-scroll-snap-strictness: proximity; --tw-shadow-colored: 0 0 transparent; --tw-shadow: 0 0 transparent; --tw-skew-x: 0; --tw-skew-y: 0; --tw-translate-x: 0; --tw-translate-y: 0; border: none; box-sizing: border-box; margin: 15pt 0in; padding: 0in; white-space: pre-wrap;"><span face=""Arial",sans-serif" style="color: #374151; font-size: 10pt;">To promote engagement in cybersecurity education, it is important to create a supportive and interactive learning environment that encourages students to take an active role in their own learning. This can include a variety of activities such as hands-on simulations, problem-based learning, and collaborative projects.<o:p></o:p></span></p>
</div>
<p class="MsoNormal" style="line-height: normal;"><span style="color: #2d3b45; font-family: "Times",serif; mso-bidi-font-size: 12.0pt; mso-fareast-font-family: "Times New Roman";">Here are some other observations I took away from this activity.<o:p></o:p></span></p>
<p class="MsoNormal" style="line-height: normal;"></p><ul style="text-align: left;"><li><i><span style="color: #2d3b45; font-family: "Times",serif; mso-bidi-font-size: 12.0pt; mso-fareast-font-family: "Times New Roman";"> </span></i><span style="color: #2d3b45; font-family: Times, serif; text-indent: -0.25in;">ChatGPT
generated everything but the headings. I certainly knew what I was looking for
but any student in my class would surely be able to figure out the prompts.</span></li><li><span style="color: #2d3b45; font-family: Times, serif; text-indent: -0.25in;">It
took me about three hours. I certainly
had to do some trial and error with prompts.
For example when I used “empiricist” instead of “cognitive associationist,”
it assumed I meant behaviorism. </span></li><li><span style="color: #2d3b45; font-family: Times, serif; text-indent: -0.25in;">It
wrote fine descriptions of cyber ranges and digital twins and even gave citations
from Wikipedia. </span></li><li><span style="color: #2d3b45; font-family: Times, serif; text-indent: -0.25in;">GPT
made some correct references. When I told
it to cite John Anderson’s ACT-R theory, it did so correctly on page three. It got
the book right but added Lynn Reder as the second author but that was a mistake (perhaps caused by Anderson, Reder, and Simon's widely-cited 1996 <i>Educational Researcher </i>paper)</span></li><li><span style="color: #2d3b45; font-family: Times, serif; text-indent: -0.25in;">When
I asked GPT for APA references, it found some correct ones (e.g., Greeno, 1998; Lave
& Wenger, 1991). Others it just made
up, but they looked <i>very</i> real (Brown et al., 2019) </span></li><li><span style="text-indent: -0.25in;">Perhaps most stunning was GPTs ability to generate a plausible
explanation for why cognitive associationist theory was least relevant for my
goal and context (page 6) and why sociohistoric theory is most relevant (pages
12-13).</span></li><li><span style="text-indent: -0.25in;">To write the paper summary. I pasted in the first half of the
abstract and asked GPT to “say more about this” and then did the same thing
with the second half. </span></li><li><span style="text-indent: -0.25in;">I analyzed all sections in the AI Content Detector (which has a 350
word limit). Scores ranged from 23% to
100% human-generated with an average of 81%.
I tested 30 random paragraphs from 15 student papers and got an average
of 88%, ranging from 17% to 100% human-generated. Because these papers were written
section by section across the Fall 2022 semester, it is very unlikely they used
ChatGPT.</span></li></ul><p></p><p class="MsoListParagraphCxSpLast" style="line-height: normal; mso-list: l6 level1 lfo6; text-indent: -0.25in;"><o:p></o:p></p>
<p class="MsoNormal" style="line-height: normal;"><o:p> </o:p>In summary, ChatGPT was remarkably good at some aspects of writing
a graduate-level paper but not others. ChatGPT's ability to contextualize concepts fundamentally
thwarts some aspect of expansive framing but not others. But its scores in the AI Content Detector were similar to actual student papers. </p><p class="MsoNormal" style="line-height: normal;">The main
finding is that requiring students to reference specific sources in their
papers looks like a promising way of thwarting ChatGPT. There are workarounds. But it would likely be difficult to write a paper that references most of the course readings and other self-selected references without learning a significant amount of the content. </p><p class="MsoNormal" style="line-height: normal;">A worthwhile thought experiment is imagining what would happen if every student used ChatGPT to generate all of their annotations, paper sections, and peer comments each week. As shown in my paper, ChatGPT knows roughly as much about learning theories and educational practice as I would expect an average graduate student to learn in my class. GPT also knew quite a bit about cybersecurity education, including the hottest new trends. Most importantly GPT was remarkably able to author new knowledge at this intersection. It seems reasonable to assume that these students might retain half of their new intersectional knowledge and perhaps 10% of each of their classmates' intersectional knowledge. That would likely be more than some of the less ambitious students are currently taking away from the experience. Dissertation study anyone?</p><p class="MsoNormal" style="line-height: normal;"><o:p></o:p></p>
<p class="MsoNormal" style="line-height: normal;"><o:p> </o:p></p>
<p class="MsoNormal" style="line-height: normal;"><o:p> </o:p></p>
<div style="border-bottom: solid windowtext 1.0pt; border: none; mso-border-bottom-alt: solid windowtext .75pt; mso-element: para-border-div; padding: 0in 0in 1pt;">
<p align="center" class="MsoNormal" style="border: none; line-height: normal; mso-border-bottom-alt: solid windowtext .75pt; mso-padding-alt: 0in 0in 1.0pt 0in; padding: 0in; text-align: center;"><span face=""Arial",sans-serif" style="display: none; mso-bidi-font-size: 12.0pt; mso-fareast-font-family: "Times New Roman"; mso-hide: all;">Top of Form<o:p></o:p></span></p>
</div>
<div style="border-top: solid windowtext 1.0pt; border: none; mso-border-top-alt: solid windowtext .75pt; mso-element: para-border-div; padding: 1pt 0in 0in;">
<p align="center" class="MsoNormal" style="border: none; line-height: normal; mso-border-top-alt: solid windowtext .75pt; mso-padding-alt: 1.0pt 0in 0in 0in; padding: 0in; text-align: center;"><span face=""Arial",sans-serif" style="display: none; mso-bidi-font-size: 12.0pt; mso-fareast-font-family: "Times New Roman"; mso-hide: all;">Bottom of
Form<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="line-height: 107%; mso-bidi-font-size: 12.0pt;"> </span></p></div></div>Daniel Hickeyhttp://www.blogger.com/profile/09885916528215868949noreply@blogger.com1tag:blogger.com,1999:blog-4778608909198518776.post-55488648904206795222023-01-05T16:57:00.006-05:002023-01-09T16:24:30.722-05:00What Does the Media Say about ChatGPT and Education?<p>Daniel Hickey & Qianxu Morgan Luo</p><p> Like millions of others, we have been quite impressed by the power of ChatGPT. Numerous media accounts argue that education will never be the same. It is remarkably capable of generating original prose that is not detectable by the current generation of plagiarism detectors like <i>Turnitin. </i>Many have noted that ChatGPT is particularly good at writing "compare and contrast" essays that many educators presumed were difficult or impossible to hack by rewriting information located on the web.</p><p>ChatGPT really exploded in December 2022. We suspect that many educators saw a massive improvement in the depth and quality of take-home exams and end-of-semester essays at that time. We predict that many of us are going to find our existing approaches to instruction and assessment upended once the new semester begins.</p><p><b>What Does Media Say So Far?</b></p><p>We are systematically analyzing the accounts as they come out. As of today, we are up to 27, which includes both objective reports as well as editorials. Here is what we have found so far:</p><p></p><ul style="text-align: left;"><li>Eight of them were classified as "worried" or "very worried." These included <a href="https://www.newyorker.com/culture/cultural-comment/the-computers-are-getting-better-at-writing">Stephen Marche's prescient 2021 article in </a><i><a href="https://www.newyorker.com/culture/cultural-comment/the-computers-are-getting-better-at-writing">The New Yorker</a> </i>that examined an earlier writing bot. Recognizing the rapid massive improvement, Marche wrote "the undergraduate essay, the basic pedagogical mode of all humanities, will soon be under severe pressure."</li><li>Fifteen were classified as "mixed." Many of these were media accounts that aimed to be objective. <a href="https://www.nytimes.com/2022/12/05/technology/chatgpt-ai-twitter.html">New Yorker technology columnist Kevin Roose pointed</a> out that ChatGPT is "ominously good at answering the types of open-ended questions that frequently appear on school assignments" but is "prone to giving wrong answers." Many suggested in-class handwritten essay exams or asking students to give impromptu presentations on ostensibly authored assignments.</li><li> Four were classified as "positive" or "very positive." Most reminded readers of similar concerns with prior technologies like spell checkers and blamed the risk on shallow instruction. For example, <a href="https://www.newsweek.com/chatgpt-can-write-better-essays-my-college-students-thats-good-thing-opinion-1769136">English Professor Blaine Gretemen's</a> editorial in <i>Newsweek </i>argued that it is "time for a new final exam, one that demands students find out something about themselves, and to tell you in a voice that is their own."</li></ul><p></p><p>Are our observations of the media coverage so far:</p><p></p><ul style="text-align: left;"><li>As proponents and scholars of online learning, We were surprised by the lack of discussion of the specific consequences for online education. Such settings are likely to preclude in-class essays or impromptu presentations as a short-term response. </li><li>Given all the suggestions that educators will need to assign in-class handwritten essays, We are surprised that no one has mentioned that many young people are incapable of writing legibly at the speed that would be needed for this to be practical.</li><li>As learning scientists, we worry that there has not been enough attention to the crucial role that writing plays in learning. As Marlene Scardamalia and Carl Bereiter convinced us in the late 1980s, skilled writers engage in <i>knowledge construction </i>where they use text to overcome the limits of short-term memory. In contrast to more novice <i>knowledge-telling </i>writers, skilled writers typically know a lot more when they complete an essay, article, or assignment.</li><li>To the advocates who liken ChatGPT to other innovations (from the slide rule to graphing calculators to Google <i>Translate</i>), a week of experimentation has convinced us that ChatGPT is <i>already </i>more powerful than all of the other technologies combined. And it is only going to get more powerful.</li></ul><div><b>Where are We Going Next?</b></div><div><b><br /></b></div><div>This is the first of several posts exploring the implications of ChatGPT. The next post will share a paper that GhatGPT wrote off Dan's online graduate course on Learning and Cognition. </div><div><br /></div><div>PS. We missed the excellent article in The Chronicle of Higher Education by Beth McMurtrie. The title captures its insight: <i><a href="https://drive.google.com/file/d/1xTUT05iFMq0LvaEmwc1CLqhNa-BHvrpZ/view?usp=sharing">AI and the Future of Undergraduate Writing: Teaching Experts are Concerned, but not for the Reasons You Thin</a>k. </i> It makes several excellent points.</div><div><ul style="text-align: left;"><li> Typical high school English and the five-paragraph essay are responsible for training a generation of knowledge tellers. </li><li>Many of the suggestions for thwarting ChatGPT are very labor-intensive. We are currently writing another blog post that will dig more deeply into this issue.</li><li>It linked to the <a href="https://docs.google.com/document/d/1m4e2VAlWBzXoteYSGvCO1eTf6GQlOsrjltoJvod0h6g/edit#">public page from Anna Mill</a>s compiling suggestions for essay prompts that might thwart chatbots.</li></ul></div><p></p><p><br /></p>Daniel Hickeyhttp://www.blogger.com/profile/09885916528215868949noreply@blogger.com0tag:blogger.com,1999:blog-4778608909198518776.post-58625225057979984712021-08-26T17:52:00.001-04:002021-08-26T17:52:45.729-04:00New Article about Situative Assessment <p><span style="font-family: times;">The awesome Diane Conrad of Athabasca University guest-edited a special issue of <i>Distance Education </i>on assessment and was kind enough to accept our proposal to present our situative approach to online grading, assessment, and testing:</span></p><blockquote style="border: none; margin: 0 0 0 40px; padding: 0px;"><p style="text-align: left;"><span style="color: #2b00fe; font-family: times;"><a href="https://drive.google.com/file/d/1DLYthQqASZXFeLEh1cq5oXZn_ExKTg1Y/view?usp=sharing" target="_blank"><span style="background-color: white;">Hickey, D., & Harris, T. (2021). Reimagining online grading, assessment, and testing using situated cognition. </span><i style="background-color: white;">Distance Education</i><span style="background-color: white;">, </span><i style="background-color: white;">42</i><span style="background-color: white;">(2), 290-309.</span></a></span></p></blockquote><p><span style="color: #222222; font-family: times;">The first part of the paper reframes the "multilevel" model of assessment introduced in a 2012 article in the </span><i style="color: #222222; font-family: times;"><a href="https://drive.google.com/file/d/1znulHlHTXhjEAYryvjZwr9ObfcIAblUW/view?usp=sharing" target="_blank">Journal of Research in Science Teaching</a></i><span style="color: #222222; font-family: times;"> and a 2013 article in the </span><i style="color: #222222; font-family: times;"><a href="https://drive.google.com/file/d/1OKLKKKV0DpiBOH5t7oICrxCU5Oyxq6IL/view?usp=sharing">Journal of Learning Sciences</a> </i><span style="color: #222222; font-family: times;">for online settings. </span></p><p></p><p class="MsoNormal" style="line-height: 150%; text-align: left;"></p><ol style="text-align: left;"><li><span lang="EN" style="font-family: times;">Immediate-Level Ungraded Assessment of Online Discourse via<i> Instructor
Comments</i></span><span lang="EN" style="font-family: times;">. </span></li><li><span style="font-family: times;">Close-Level Graded Assessment of Engagement via <i>Informal
Reflections</i></span></li><li><span lang="EN" style="font-family: times;">Proximal Formative </span><i style="font-family: times;"><span lang="EN">Self-Assessments</span></i></li><li><i style="font-family: times;"><span lang="EN">A</span></i><span lang="EN" style="font-family: times;">utomated Distal Summative <i>Achievement Tests</i></span></li></ol><p></p><p></p><div><span style="color: #222222; font-family: times;">The second part of the article introduces ten new assessment design principles, </span></div><div><ol style="text-align: left;"><li><span lang="EN"><span style="font-family: times;">Embrace Situative Reconciliation over Aggregative Reconciliation.</span></span></li><li><span lang="EN"><span style="font-family: times;">Focus on Assessment Functions Rather than Purposes. </span></span></li><li><span lang="EN"><span style="font-family: times;">Synergize Multiple Complementary Types of Interaction</span></span></li><li><span lang="EN"><span style="font-family: times;">Use Increasingly Formal Assessments that Capture Longer Timescales of
Learning</span></span></li><li><span lang="EN"><span style="font-family: times;">Embrace Transformative Functions and Systemic Validity</span></span></li><li><span lang="EN"><span style="font-family: times;">Position Learners as Accountable Authors</span></span></li><li><span lang="EN"><span style="font-family: times;">Reposition Minoritized Learners for Equitable Engagement</span></span></li><li><span lang="EN"><span style="font-family: times;">Enhance Validity of Evidence for Designers, Evaluators, and
Researchers </span></span></li><li><span lang="EN"><span style="font-family: times;">Enhance Credibility of Scores and Efficiency for
Educators</span></span></li><li><span lang="EN"><span style="font-family: times;">Enhance Credibility of Assessments and Grades for Learners</span></span></li></ol><div><span style="font-family: times;">I was particularly pleased with the new ideas under the seventh principle. We were able to use <a href="https://www.tandfonline.com/doi/abs/10.1080/07370008.2019.1624544?journalCode=hcgi20" target="_blank">Agarwal and Sengupta-Irvings (2019)</a> critique of <a href="https://www.jstor.org/stable/3233901">Engle & Conants (2002) </a>Productive Disciplinary Engagement framework and their new Connected and Productive Disciplinary Engagement framework, It forms the core of our Culturally Sustaining Classroom Assessment framework that we will be presenting for the first time at the Culturally Relevant Evaluation and Assessment conference in Chicago in late September, 2021.</span></div></div><p></p>Daniel Hickeyhttp://www.blogger.com/profile/09885916528215868949noreply@blogger.com0tag:blogger.com,1999:blog-4778608909198518776.post-55924072539986193372021-05-13T17:05:00.003-04:002023-01-22T11:32:12.916-05:00Articles Chapters, and Reports about Open Badges<p style="text-align: center;"><b>by Daniel Hickey</b></p><p style="text-align: left;">Thanks to Connie Yowell and Mimi Itow at the MacArthur Foundation's Digital Media and Learning Initiative, I had the pleasure of being deeply involved with digital badges and micro-credentials starting in 2010. While we no longer have any funding for this work, my colleagues and I are continuing to engage with the community. I am thrilled to see the continued growth and the wide recognition that micro-credentials offer new career pathways to non-traditional learners.</p><p style="text-align: left;">I get occasional requests for copies of chapters, articles, and reports that we reproduced as well as some general "where do we begin" queries. Given that we were funded to provide broad guidance from 2012-2017, we produced some things that beginners and advanced innovators have found quite useful. We continued to publish after MacArthur ended the DML initiative and funding ran out. Here is an annotated list of resources. We hope you find them useful!</p><p style="text-align: left;"><b>Getting Started.</b></p><p style="text-align: left;">If you are new to badges and microcredentials, this might be a good place to get some basic background:</p><p class="MsoNormal"></p><ul style="text-align: left;"><li><span style="font-size: 11.0pt; line-height: 107%;"><a href="https://drive.google.com/file/d/1JBkTADbR4OYqNtZyEvdsRviTzCXqBD1a/view?usp=share_link">Hickey, D.
T. , (2017). <i>Badges. </i>In K. Peppler (Ed.) <i>Encyclopedia of out-of-school learning</i>: Volume 1 (pp. 460-463). Los
Angeles, CA: Sage Publications.</a></span></li></ul><p></p><p><b>Where Badges Work Better</b></p><p>We studied the 30 badge systems that MacArthur funded in 2012 to uncover the badge system design principles that might guide the efforts of innovators. This included <i>general </i>principles and principles for <i>recognizing, assessing, motivating, </i>and <i>studying </i>learning. These findings were collected in a short report at EDUCAUSE and our longer report:</p><p class="MsoNormal"></p><ul style="text-align: left;"><li><span style="font-size: 11.0pt; line-height: 107%;"><a href="http://www.educause.edu/library/resources/where-badges-work-better ">Hickey, D.
T., Willis, J. E., & Quick, J. D. (2015).
<i>Where badges work better. </i>EDUCAUSE
Learning Initiative Brief.</a></span></li><li><span style="font-size: 11.0pt; line-height: 107%;"><a href="https://drive.google.com/file/d/1Jh02lSpD2vAHZSZcMWhOAb7jjFtCHfv6/view?usp=share_link">Hickey, D.
T., & Willis, J. E. (2017). <i>Where open badges appear to work better:
Findings from the Design Principles Documentation Project. </i>[Unpublished report].
Center for Research on Learning and Technology, Indiana University</a></span></li></ul>We also did a followup study two years later to determine which systems resulted in a "thriving" badge-based ecosystem. <span style="text-indent: -0.5in;"> Most of the constructivist "completion-badge" systems and associationist "competency-badge" systems failed to thrive, many never got past piloting and some never issued any badges. Turned out that wildly optimistic plans for assessing competency or completion undermined the project. In contrast, most of the sociocultural "participation-badge" systems were still thriving, in part because they relied on peer assessment and because they assessed social learning rather than individual completion or competency:</span><br /><p></p><ul style="text-align: left;"><li><span style="font-size: 11pt; text-indent: -0.5in;"><a href="https://drive.google.com/file/d/1RNc_rv8rXh-PPKaTYwnR-dv62MNpTUZi/view?usp=sharing" target="_blank">Hickey, D. T., &
Chartrand, G. T. (2020). Recognizing
competencies vs. completion vs. participation: Ideal roles for web-enabled
digital badges. <i>Education and Information Technologies, 25 </i>(2), 943-956.</a></span></li></ul><p></p><p> <b>Endorsement 2.0 and Badges in the Assessment BOOC</b></p><p>An important development is "endorsement" in the Open Badges 2.0 Standards. It allows a "BadgeClass" to carry an endorsement (e.g., from an organization, after reviewing the standards) and for each "assertion" of that badge class to carry an endorsement (e.g., from a member of that organization, after reviewing the evidence in the badge). Nate Otto and I summarized this feature and EDUCAUSE Review and predicted its s impact in the Chronicle: </p><p></p><ul style="text-align: left;"><li><span style="font-family: "Times New Roman",serif; font-size: 11.0pt; mso-ansi-language: EN-US; mso-bidi-font-weight: bold; mso-bidi-language: AR-SA; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: EN-US;"><a href="http://er.educause.edu/articles/2017/2/endorsement-2-taking-open-badges-and-ecredentials-to-the-next-level ">Hickey, D. T., &
Otto, N. (2017, February 13). “Endorsement 2.0” to take Open Badges and
e-credentials to the next level. <i>EDUCAUSE Review, </i>1-24.</a></span></li><li><span style="font-size: 11.0pt; line-height: 107%; mso-bidi-font-weight: bold;"><a href="https://drive.google.com/file/d/16p9WAgjTP-ZzRW-szENfQBH8Fd0aQ908/view?usp=share_link">Hickey, D. T., & Otto, N. (2017, April 8). How open e-Credentials will change education
like e-commerce changed retailing.<i> </i> <i>Chronicle
of Higher Education. </i>Invited editorial
in a special issue on technology.</a></span></li></ul><p></p><p>This chapter describes Google-funded "Big Open Online Course" ("BOOC") which really pushed the limits of open badges, including one of the first examples of "peer endorsement" and "peer promotion." It also showed that our asynchronous model of participatory learning and assessment (PLA) could be used at scale to support highly interactive learning with almost no instructor engagement with open learners:</p><p></p><ul style="text-align: left;"><li><span style="font-size: 11pt; mso-bidi-font-weight: bold;"><a href="https://drive.google.com/file/d/1-4hQvjI2ByGePF0d1wrVWY9O95bPfM1i/view?usp=sharing" target="_blank">Hickey, D. T. & Uttamchandani, S. L. (2017).
Beyond hype, hyperbole, myths, and paradoxes: Scaling up participatory
learning in a big open online course. In
L. Losh (Ed.) <i>The MOOC moment:
Experiments in scale and access in higher education </i>(pp. 13-36).<i> </i>Chicago, IL: The University of Chicago
Press.</a></span></li></ul><p></p><p><b>The Varied Functions of Open Badges</b></p><p>This chapter used the BOOC badges to illustrate how badges to illustrate the range of functions of open badges. It shows how badges support the shift (a) from measuring achievement to capturing learning. (b) from credentialing graduates to recognizing learning, (c) from compelling achievement to motivating learning, and (d) from accrediting schools and programs to endorsing learning:</p><p></p><div><ul style="text-align: left;"><li><a href="https://drive.google.com/file/d/1sId0O321Ci2NW6K1avbhI4sqwaE8iaQH/view?usp=sharing" target="_blank"><span style="font-size: 11pt; line-height: 16.8667px;">Hickey, D. T., Uttamchandani, S. L., & Chartrand, G. T. (2020). <i>Competencies in context: New approaches to capturing, recognizing, and endorsing learning. </i>In </span><span style="font-size: 11pt; line-height: 16.8667px;">Bishop, M. J., Boling, E., Elen, J., & Svihla, V. (Eds.). <i>Handbook of research in educational communications and technology </i>(547-592). New York, NY: Springer.</span></a></li></ul></div><div>This chapter used example badges from sustainable/sustainability education to similarly illustrate these four functions of digital badges. The badges came from Ilona Buchem's EU-funded <i>Open Virtual Mobility </i>project and the <i>FAO e-Learning Academy </i>from the<i> </i>UN's Food and Agriculture Organization. BTW, the e-Learning Academy features some of the best self-paced open courses I have ever seen. the assessments are great and you really can't prank them. If it says the course will take two hours it is really impossible to earn the badges without spending two hours learning (I tried!):</div><div><ul style="text-align: left;"><li><p style="line-height: 115%;"><span style="font-size: 11pt; line-height: 115%;"><a href="https://drive.google.com/file/d/1VZFiq2jAImp8BpGm10vy3zbkyEpH3ACo/view?usp=sharing" target="_blank">Hickey,
D. T., & Buchem, E. (in press). Advancing sustainable educational ecosystems
with open digital credentials and badges. In S. Jagannathan (Ed.) <i>Educational technologies for sustainable
development: How upskilling data, analysis, and digital innovations foster
lifelong learning. </i>Routledge. <o:p></o:p></a></span></p></li></ul><b>Situative Motivational Principles for Open Badges</b></div><div><br /></div><div>This 2017 chapter presents the situative model of
assessment that was first published in Hickey (2003) in the context of open
badges. It is my response to people like Mitch Resnick who claim that
open badges will undermine intrinsic motivation. I agree with him that
they will if you use them as meaningless tokens. So don't do that
Mitch! Instead take advantage of the fact that badges contain meaningful
information and can circulate in social networks and gain more meaning, which
has consistently been shown to enhance free-choice engagement:</div><div><ul style="text-align: left;"><li><span><a href="https://drive.google.com/file/d/1B5hpWYOEXp1EcQGziTFlgo-wN59aVXqn/view?usp=sharing" target="_blank">Hickey, D. T., & Schenke, K. (2019).
Open digital badges and reward structures. In K. A. <span style="font-size: 11pt;">Renninger
& S. E Hidi S.E. (Eds.), The Cambridge handbook on motivation and learning
(pp, 209-237) Cambridge MA: Cambridge University Press.</span> </a></span> </li></ul><p><b style="text-indent: -0.5in;">Validity vs. Credibility</b></p><p>Early on in my journey with digital badges, Carla Casilli blew my mind when her early blogpost explained how the "open" nature of open badges forced us to rethink validity in assessment and testing. The ability for a viewer to interrogate the evidence contained in a badge or micro-credential means that the credibility of that evidence is more important than the validity of that credential in a traditional sense. So I was happy to write with her about this important issue: </p><p class="MsoNormal"></p><ul style="text-align: left;"><li><span style="font-size: 11.0pt; line-height: 107%; mso-bidi-font-weight: bold;"><a href="https://drive.google.com/file/d/1TK3zx_V9k5Q9zn4OGtgSLQMY7d2vDDvh/view?usp=share_link">Casilli, C, & Hickey, D. T. (2016).
Transcending conventional credentialing and assessment paradigms with
information-rich digital badges. <i>The Information Society, 32 </i>(2),
117-129.</a></span></li></ul><p></p></div><p></p>Daniel Hickeyhttp://www.blogger.com/profile/09885916528215868949noreply@blogger.com0tag:blogger.com,1999:blog-4778608909198518776.post-27138910752010645002021-05-12T11:47:00.000-04:002021-05-12T11:47:06.835-04:00New articles on Participatory Learning and Assessment (including inclusion)<p> Yikes, it has been a long time since we have posted. Partly what happened is we redirected our energy from blogging to publishing. Starting in 2019, we began translating the theory-laden design principles to practical steps for readers who may or may not be grounded in sociocultural theories. This was serendipitous in light of the pandemic and the explosion of interest in asynchronous online learning. </p><p>In contrast to our earlier articles, these new articles reflect the influence of current research on power and privilege in the learning sciences. Each includes design principles and/or steps that are intended to "reposition" minoritized learners. In particular, the changes reflect the influence of papers by Priyanka Agarwal and Tesha Sengupta-Irving on <a href="https://www-tandfonline-com.proxyiub.uits.iu.edu/doi/pdf/10.1080/07370008.2019.1624544" target="_blank">Connective and Productive Disciplinary Engagement (CPDE, 2019)</a> Each of the descriptions below is hotlinked to a copy of the article.</p><p>This first article is a very gentle introduction to online participatory learning and assessment (PLA). It was written for educators with no experience teaching online and who are not grounded in any particular theory of learning</p><p></p><ul style="text-align: left;"><li><p class="MsoNormal" style="line-height: 115%; margin-left: .5in; text-indent: -.5in;"><span style="font-size: 11pt; line-height: 115%;"><a href="https://drive.google.com/file/d/1tls5sFKGyfOBFFK3Kovp9h6KNqPIUuGl/view?usp=sharing" target="_blank">Hickey, D. T., Duncan,
J., Gaylord, C., Hitchcock, C., Itow, R., & Stephens, S. (2020).
gPortfolios: A pragmatic approach to online asynchronous assignments. [Joint
special issue on "Evidence-based and pragmatic online teaching and
learning approaches: A response to emergency transitions to remote online
education."] <i>Information and Learning Sciences, 121 </i>(5/6), 273-283. </a><o:p></o:p></span></p></li></ul><p></p><p>This article describes how we translated the PLA principles into fourteen steps, focusing on engagement routines. It was written for instructional designers and others who are grounded in more conventional cognitive-associationist and cognitive-constructivist theories of learning</p><p></p><ul style="text-align: left;"><li><p class="MsoNormal" style="line-height: 115%; margin-left: .5in; text-indent: -.5in;"><a href="https://drive.google.com/file/d/1FN-J9R0C0slGrCwDgGyriCGup4W8duBX/view?usp=sharing" name="_Hlk70598247" target="_blank"><span style="font-size: 11.0pt; line-height: 115%; mso-bidi-font-weight: bold;">Hickey, D. T., Chartrand, G. T., & Andrews, C. D. (2020). Expansive framing as a pragmatic theory for
instructional design. [Special issue on "The
crucial role of theoretical scholarship for learning design and technology."]
<i>Educational Technology Research and Development </i>68 (2), 751-782.<o:p></o:p></span></a></p></li></ul><p></p><p>This one introduces ten new situative assessment design principles, building on the "multi-level" assessment model in <a href="https://drive.google.com/file/d/1OKLKKKV0DpiBOH5t7oICrxCU5Oyxq6IL/view?usp=sharing" target="_blank">Hickey and Zuiker (2012)</a>. While it includes the theoretical grounding, it was written for readers who might not be grounded in situative theory.</p><p></p><ul style="text-align: left;"><li><p class="MsoNormal" style="line-height: 115%; margin-left: .5in; text-indent: -.5in;"><span style="font-size: 11.0pt; line-height: 115%; mso-bidi-font-weight: bold;"><a href="https://drive.google.com/file/d/1DLYthQqASZXFeLEh1cq5oXZn_ExKTg1Y/view?usp=sharing" target="_blank">Hickey, D.
T. & Harris, T. (2021). Reimagining
online grading, assessment, and testing using situated cognition. [Special
issue on "reimagining assessment in online and distributed learning."]
<i>Distance Education, 42 </i>(2), 1-26.</a><i> <o:p></o:p></i></span></p></li></ul><p></p>Daniel Hickeyhttp://www.blogger.com/profile/09885916528215868949noreply@blogger.com0tag:blogger.com,1999:blog-4778608909198518776.post-51374899246267986102016-12-21T08:08:00.000-05:002016-12-29T12:17:25.021-05:00Competencies in Context #5: Increasing the Value of Certificates from LinkedIn Learning<div class="MsoNormal" style="text-align: center;">
By Daniel Hickey and Chris Andrews</div>
<div class="MsoNormal" style="text-align: center;">
<br></div>
<div class="MsoNormal">
The <a href="http://remediatingassessment.blogspot.com/2016/12/competencies-in-context-4.html">previous post in this series</a> explored LinkedIn endorsements and recommendations. Chris and Dan used those features to locate a potential consultant with particular skills and considered recent refinements to those features. We also explore the new LinkedIn <i>Learning </i>site made possible by the acquisition of Lynda.com. In this post, we explore how endorsements and recommendations might help LinkedIn earn back the roughly $300,000 that they paid for <i style="mso-bidi-font-style: normal;">each </i>of Lynda.com's 5000 courses.<span style="mso-spacerun: yes;"> </span><o:p></o:p></div>
<div class="MsoNormal">
<b style="mso-bidi-font-weight: normal;"></b></div>
<a href="http://remediatingassessment.blogspot.com/2016/12/competencies-in-context-5-increasing.html#more">Read more »</a>Chris Andrewshttp://www.blogger.com/profile/05655116027456821128noreply@blogger.com0tag:blogger.com,1999:blog-4778608909198518776.post-84654896961919244722016-12-13T14:36:00.000-05:002016-12-13T16:45:54.134-05:00Competencies in Context #4: eCredentialing via LinkedIn Recommendations and Endorsements<div class="MsoNormal" style="text-align: center;">
by Daniel Hickey and Chris Andrews</div>
<div class="MsoNormal">
<br></div>
<div class="MsoNormal">
In the <a href="http://remediatingassessment.blogspot.com/2016/11/competencies-in-context-2-lrns-for.html">second
post</a> in this series on eCredentialing, Dan discussed how new digital
Learning Recognition Networks (LRNs) can simultaneously support the goals of
learners, educators, schools, recruiters, and admissions officers. A reader
posted a question on that post about how the endorsement practices afforded by
these new LRNs build on the existing endorsement practices, like those at
LinkedIn. Since its launch in 2002, LinkedIn has grown into the largest digital
LRN in existence. So, this is a <i style="mso-bidi-font-style: normal;">great </i>question.
Dan did some digging using his own network to hunt for someone with very
specific competencies, while Chris dug into the recent research and improvements
to LinkedIn Endorsements. We also peeked into the new LinkedIn <i style="mso-bidi-font-style: normal;">Learning </i>made possible by the
acquisition of Lynda.com.</div>
<div class="MsoNormal">
<o:p></o:p></div>
<a href="http://remediatingassessment.blogspot.com/2016/12/competencies-in-context-4.html#more">Read more »</a>Chris Andrewshttp://www.blogger.com/profile/05655116027456821128noreply@blogger.com0tag:blogger.com,1999:blog-4778608909198518776.post-50805666950752784312016-11-24T15:27:00.000-05:002016-11-29T09:07:50.308-05:00Competencies in Context #3: Open Endorsement 2.0 is Coming<div style="text-align: center;">
By Daniel Hickey and Nate Otto</div>
<div style="text-align: center;">
<br></div>
<div style="text-align: left;">
In the third post of this series, we discuss the Open Badge Specification and its shift from the Badge Alliance to the IMS Global Learning Consortium in 2017. We then discuss the crucial <i>Endorsement </i>features that will be supported in the forthcoming 2.0 Specifications. We will use the example of Luis Lopez's HIPAA badge described in the <a href="http://remediatingassessment.blogspot.ca/2016/11/competencies-in-context-new.html" target="_blank">first post in this series</a> to consider how these new features might operate. This illustrates how Endorsement 2.0 will be crucial in the new Learning Recognition Networks that Dan described in the <a href="http://remediatingassessment.blogspot.com/2016/11/competencies-in-context-2-lrns-for.html" target="_blank">second post in this series</a>. </div>
<div style="text-align: left;">
</div>
<a href="http://remediatingassessment.blogspot.com/2016/11/competencies-in-context-3-open.html#more">Read more »</a>Daniel Hickeyhttp://www.blogger.com/profile/09885916528215868949noreply@blogger.com2tag:blogger.com,1999:blog-4778608909198518776.post-73715665031583289102016-11-21T10:23:00.002-05:002016-12-16T09:51:04.569-05:00Competencies in Context #2: LRNs for Micro-Masters and eCertificates<div style="text-align: center;">
By Daniel Hickey</div>
<div style="text-align: center;">
<br></div>
<div style="text-align: left;">
In this detailed post, I discuss the announced release date of the <i>MyMantl </i>Learning Recognition Network (LRN) from Chalk & Wire and argue that such digital LRNs can add value to online career and professional education programs. This includes more conventional continuing education programs and newer MOOC-based "micromasters" programs. Both types of programs promise inexpensive short-term solutions for career entry/change/advance, but they introduce serious challenges for assessment and accountability. New digital LRNs can help.</div>
<div style="text-align: left;">
</div>
<a href="http://remediatingassessment.blogspot.com/2016/11/competencies-in-context-2-lrns-for.html#more">Read more »</a>Daniel Hickeyhttp://www.blogger.com/profile/09885916528215868949noreply@blogger.com0tag:blogger.com,1999:blog-4778608909198518776.post-67624995878954450682016-11-14T21:15:00.000-05:002016-11-21T18:14:39.120-05:00Competencies in Context #1: New Developments at Portfolium<div style="text-align: center;">
By Dan Hickey</div>
<div style="text-align: left;">
In this detailed post, I illustrate how the <i>Portfolium </i>ePortfolio platform is breaking new ground with digital badges and new networking features that readily connect learners and potential employers. In particular, I highlight my own interaction with a student in LA around one of the badges he earned in his coursework. I presented this example in talks at ePIC in Bologna and Mozfest in London and lots of people had questions about it. What I find particularly exciting about these developments is how it shows healthy competition to around the most effective communication about competencies and evidence of competencies among educators, learners, and employers. The communication is crucial because it provides information about the <i>context </i>in which students competencies were developed and (therefore) the range of contexts where those competencies will be most readily deployed in the future.</div>
<a href="http://remediatingassessment.blogspot.com/2016/11/competencies-in-context-new.html#more">Read more »</a>Daniel Hickeyhttp://www.blogger.com/profile/09885916528215868949noreply@blogger.com0tag:blogger.com,1999:blog-4778608909198518776.post-38805200884715807592016-08-29T12:22:00.001-04:002016-09-06T11:21:09.726-04:00Badges + ePortfolios = Helping Turn Artifacts into Open Learning Recognition Networks<div style="text-align: center;">
by Dan Hickey</div>
<br>
This post summarizes a meeting between representatives of six leading ePortfolio providers, four digital badge providers, and four professional associations on August 2 in Boston at the Annual Meeting of the Association for Authentic, Experiential, and Evidence-Based Learning (AAEEBL)<br>
We searched for and found synergy between these two crucial technologies that are helping innovators re-imagine how learning can be represented in the Internet era. They are starting to come together to create what some are calling Learning Recognition Networks (LRNs).<br>
<br>
This meeting also brings to a close the two-year Open Badges in Higher Education (OBHE) project, carried out with the support of the MacArthur Foundation. We will be discussing the Boston meeting and future directions for LRNs in the next Open Badges Community Call hosted by the Badge Alliance. The call is at 1200 noon EST on August 31 and all are welcome and encouraged to join (meeting at this <a href="https://www.uberconference.com/badgealliance" target="_blank">Uberconference link</a>).<br>
<a href="http://remediatingassessment.blogspot.com/2016/08/badges-eportfolios-helping-turn.html#more">Read more »</a>Daniel Hickeyhttp://www.blogger.com/profile/09885916528215868949noreply@blogger.com0tag:blogger.com,1999:blog-4778608909198518776.post-8497300776635141782016-07-27T16:03:00.000-04:002016-07-27T16:03:03.669-04:00Traditional Approaches to Validity in Assessment Innovation (Part 2: Consequential Validity)<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: "inherit",serif; font-size: 12.0pt; mso-bidi-font-family: "Times New Roman"; mso-fareast-font-family: "Times New Roman";">This is the second
post in a series on the topic of <i>validity </i>in educational
assessment. In my <a href="http://remediatingassessment.blogspot.ca/2016/07/modern-approaches-to-validity-in.html"><span style="color: blue;">first post</span></a>, I described the traditional
characterization of <i>content-related, criterion-related, </i>and <i>construct-related </i>evidence
as they are relevant to educators and credentialing innovators who use and
design assessments. This post summarizes traditional characterizations of </span><span style="font-family: "Times New Roman",serif; font-size: 12.0pt; mso-fareast-font-family: "Times New Roman";">“</span><span style="font-family: "inherit",serif; font-size: 12.0pt; mso-bidi-font-family: "Times New Roman"; mso-fareast-font-family: "Times New Roman";">consequential
validity.</span><span style="font-family: "Times New Roman",serif; font-size: 12.0pt; mso-fareast-font-family: "Times New Roman";">”</span><span style="font-family: "inherit",serif; font-size: 12.0pt; mso-bidi-font-family: "Times New Roman"; mso-fareast-font-family: "Times New Roman";"> This aspect of validity concerns
the broader consequences of administering assessments to learners and using the
resulting scores. It is a complex idea that is really crucial to many
assessment and credentialing innovators (because broader change is their goal).
Many measurement professionals have long argued that it an
"unsanctioned" aspect of validity. Before I write about how
that is changing, I want to describe how consequential validity has
traditionally been written about and why I have long disagreed. </span></div>
<a href="http://remediatingassessment.blogspot.com/2016/07/traditional-approaches-to-validity-in_27.html#more">Read more »</a>Daniel Hickeyhttp://www.blogger.com/profile/09885916528215868949noreply@blogger.com0tag:blogger.com,1999:blog-4778608909198518776.post-26400367611352464192016-07-04T10:37:00.000-04:002016-08-23T18:42:11.850-04:00Traditional Approaches to Validity in Classroom Assessment and Innovative Credentialing (Part 1)<div class="MsoNormal" style="text-align: center;">
By Daniel Hickey</div>
<div class="MsoNormal">
In my work with the Participatory Assessment Lab at Indiana
University and in my graduate education courses, I spend a lot of time helping
people understand <i>validity </i>in the context of educational assessment. In this post, I describe validity as it has traditionally been presented to educators. I summarize what one leading textbook has long
said educators should know about validity when assessing learning in their
own classes, and I extend that to credentialing innovators who are developing
digital badge systems, micro-credentials, and competency-based educational
programs. In subsequent posts, I will explore traditional views of “face validity” and “consequential validity.” Together, these posts will lay the groundwork for a final post that will explore several new developments in
validity theory that I believe are important for these two communities.</div>
<a href="http://remediatingassessment.blogspot.com/2016/07/modern-approaches-to-validity-in.html#more">Read more »</a>Daniel Hickeyhttp://www.blogger.com/profile/09885916528215868949noreply@blogger.com0tag:blogger.com,1999:blog-4778608909198518776.post-38133829036491271702016-05-15T13:30:00.001-04:002016-05-15T14:40:14.546-04:002016 AAEEBL Midwest Meeting Keynote: Open Badges + ePortfolios: Searching for and Supporting Synergy<div style="text-align: center;">
By Dan Hickey</div>
<div style="text-align: center;">
<br></div>
This is a brief report and link to a <a href="http://www.slideshare.net/DanielHickey/aaeebl-regional-meeting-2016-keynote" target="_blank">slideshare </a>from the keynote address at the 2016 Midwest Meeting of the Association for Authentic, Experiential, and Evidence Based Learning. It summarizes potential synergy between these two important educational technologies, as well as progress towards this synergy by the seven leading eportfolio platforms and between AAEEBL, the <a href="https://sites.google.com/site/openbadgesinhighereducation/home" target="_blank">Open Badges in Higher Education Project</a>, and the <a href="http://www.badgealliance.org/" target="_blank">Badge Alliance</a>.<br>
<br>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://1.bp.blogspot.com/-pecFoPvXj-Y/VziqFDWzUWI/AAAAAAAAAoU/03hgHMOCVTQ1HMCUDSB0rPaAlTfRdxUAQCKgB/s1600/Dan%2BKeynote%2Bcropped.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="237" src="https://1.bp.blogspot.com/-pecFoPvXj-Y/VziqFDWzUWI/AAAAAAAAAoU/03hgHMOCVTQ1HMCUDSB0rPaAlTfRdxUAQCKgB/s400/Dan%2BKeynote%2Bcropped.jpg" width="400"></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Keynote Address at AAEEBL Midwest Regional Meeting at Notre Dame</td></tr>
</tbody></table>
<br>
Once I get this post up I will get to the several dozen emails awaiting me in my inbox with headers suggesting that we have indeed made some progress towards increased synergy.<br>
<a href="http://remediatingassessment.blogspot.com/2016/05/2016-aaeebl-midwest-meeting-keynote.html#more">Read more »</a>Daniel Hickeyhttp://www.blogger.com/profile/09885916528215868949noreply@blogger.com0tag:blogger.com,1999:blog-4778608909198518776.post-60661569187643541262016-04-27T09:03:00.000-04:002016-04-27T18:08:26.086-04:00A Situative "Roadmap" for Synergy in Motivating Family STEM Engagement<div style="text-align: center;">
<b>By Dan Hickey</b></div>
<div class="MsoNormal">
This is the third post about my collaboration with
Iridescent Inc., a science education non-profit in LA. This new post describes how
a key assumption in newer "situative" theories of motivation can
resolve the tensions between prior empiricist and constructivist approaches. When
combined with Design Based Research methods, this assumption can result in a
coherent "roadmap" towards synergy across the three approaches. I
contend that such a roadmap can help Iridescent and other informal STEM
innovators find a route that takes them from current levels of engagement to much
higher levels of engagement, both in terms of quantity and quality.<o:p></o:p></div>
<div class="MsoNormal">
<br></div>
<div class="MsoNormal">
This post could use some work and some trimming but I need to get it up for my class and colleagues and get on to other things. Will try to clean it up soon</div>
<a href="http://remediatingassessment.blogspot.com/2016/04/a-situative-roadmap-for-synergy-in.html#more">Read more »</a>Daniel Hickeyhttp://www.blogger.com/profile/09885916528215868949noreply@blogger.com0tag:blogger.com,1999:blog-4778608909198518776.post-9491963944715112112016-04-24T14:34:00.001-04:002016-04-25T11:44:02.441-04:00Motivating STEM Engagement in Children, Families, and Communities<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div align="center" class="MsoNormal" style="text-align: center;">
<b>By Daniel Hickey</b></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<br>
<div class="separator" style="clear: both; text-align: center;">
</div>
<span style="text-indent: 0.5in;">This extended post is the second in a series of post about my work with non-profit STEM education start up called Iridescent. In a </span><a href="http://remediatingassessment.blogspot.com/2016/04/engaging-children-and-families-in.html" style="text-indent: 0.5in;" target="_blank">previous post</a><span style="text-indent: 0.5in;"> I describe their Curiosity Machine and an evaluation I carried out of a program designed to encourage families to complete </span><i style="text-indent: 0.5in;">Engineering Design Challenges </i><span style="text-indent: 0.5in;">on that website from home, after attending two-hour weekly evening workshops. In this post I explore the challenges that organizations like Iridescent and science museums face in attempting to motivate inquiry learning in informal contexts. I am also using this post as my weekly contribution to the graduate education class on motivation that I am currently teaching.</span><br>
<div class="MsoNormal" style="text-indent: .5in;">
<br></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-rVvNZROIx0o/Vx0M-FU0G-I/AAAAAAAAAms/5wKwOqoie7oh9R4L9d2y5GA_qNnC42K6QCKgB/s1600/Curiosity-Machine.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="320" src="https://1.bp.blogspot.com/-rVvNZROIx0o/Vx0M-FU0G-I/AAAAAAAAAms/5wKwOqoie7oh9R4L9d2y5GA_qNnC42K6QCKgB/s640/Curiosity-Machine.jpg" width="640"></a></div>
<div class="MsoNormal" style="text-indent: .5in;">
<br></div>
<div class="MsoNormal" style="text-indent: .5in;">
<br></div>
<a href="http://remediatingassessment.blogspot.com/2016/04/motivating-stem-engagement-in-children.html#more">Read more »</a>Daniel Hickeyhttp://www.blogger.com/profile/09885916528215868949noreply@blogger.com0tag:blogger.com,1999:blog-4778608909198518776.post-48889472050792222262016-04-22T16:55:00.001-04:002016-04-22T16:55:34.396-04:00The Data of Learning: A Response to Martin Kurzweil's "Responsible Use of Student Data"by James Willis<br>
<br>
In mid-April, Stanford University hosted the "<a href="https://ols2016.stanford.edu/program" target="_blank">Learning Summit 2016: Inventing the Future of Higher Education</a>." For those of us who study how the newer processes and protocols of using student data have ethical and legal consequences, one session in particular should be of interest: <a href="http://cee.ucdavis.edu/contact/about-director.html" target="_blank">Marco Molinaro</a> (UC Davis) moderated a panel on the "Responsible Use of Student Data for Individual and Organizational Improvement" which included speakers <a href="http://www.sr.ithaka.org/people/martin-kurzweil/" target="_blank">Martin Kurzweil</a> (ITHAKA S+R), <a href="https://ed.stanford.edu/faculty/stevens4" target="_blank">Mitchell Stevens</a> (Stanford), and <a href="https://idre.ucla.edu/people/kent-wada" target="_blank">Kent Wada</a> (UCLA). Kurzweil provided a <a href="http://www.sr.ithaka.org/blog/responsible-use-of-student-data/" target="_blank">recent blog posting</a> summarizing the panel discussion, raising some important points.<br>
<br>
<a href="http://remediatingassessment.blogspot.com/2016/04/the-data-of-learning-response-to-martin.html#more">Read more »</a>Anonymoushttp://www.blogger.com/profile/11170497669879668928noreply@blogger.com0