By Daniel Hickey and Qianxu (Morgan) Luo
Pedagogical Prompt Engineering: Analysis #2, Ask Students for “Local” Information”
This lengthy post explores another common recommendation
for educators trying to keep their students from using generative AI to thwart
their learning. We began exploring this issue in previous posts using ChatGPT &
Google’s Bard and using Microsoft’s New Bing. Those posts showed
how students might use what some are calling “prompt engineering” to get around
the recommendation to ask students for “specific” information (e.g., from scholarly
articles) that is not part of a platform’s large language model (LLM).
In this
post, we explore another recommendation that educators ask students about
“local” information outside of the LLM. Our near-term goal is what Victor Lee of Stanford suggested we call “pedagogical prompt engineering.” This
means helping students learn how to refine prompts to actually support their
learning. Our ultimate goal is to use current theories and methods from the
learning sciences to maximize the value and minimize the harm of these powerful
new tools.
Recommendations for Using Local Information
In our initial review of the media accounts and
educator blogs we encountered quite a few recommendations to ask students to
use local information. In an article entitled How
to Prevent ChatGPT Cheating, one of Erik Ofgang’s five suggestions was “incorporate
authentic student experience and student connections into questions.” For
example…
education
college students might be asked to respond to a reading by talking about what
they’ve seen in their own education or in their work in the field (2/6/23)
Similarly, the englishwritingteacher.com asked
Do
teachers need to ask students to weave some highly local information—the
spelling bee yesterday at XYZ School, the performance of substitute teacher
Mrs. Poggi last week—into their writing so that AI has no way to access that
local information into its output, and so students are forced to write for
themselves? (March 20, 2023).
Ashiedu Jude was more directive on LinkedIn:
Classroom activity data can't be
accessed by ChatGPT, no student can prompt the AI to generate local
information, which becomes the data pool for teachers' assessment of students.
There'll be no need for lengthy text-based only term papers, essays, or
presentations without major references to local data or information not easily
found online. Your students have all access to ChatGPT, yet they'll be forced
to rely on original ideas from localized information and activities (1/23/23).
We acknowledge that these recommendations are
alongside many others in these posts and elsewhere. But here, we focus on the recommendation to require local information so we can contrast “problematic prompt
engineering” across the three widely available platforms and speculate about
each for pedagogical prompt engineering. As many students have figured out (or
learned on Reddit), they can bypass this recommendation by engineering prompts
to summarize local events that they observed or otherwise learned about.
Comparing “Shallow” Prompts in Three Platforms
First, we asked all three platforms a shallow question drawing on what Ofgang characterized as “lower
order thinking skills, like being able to recall information, summarize
information, explain information.” But we asked
about relatively sophisticated knowledge, in this case about learning theory:
Our initial prompt: Describe the difference between cognitive-associationist,
socio-constructivist, and sociocultural approaches to education
Elaborations on this question are a mainstay of
teacher education, educational psychology, and learning sciences courses. This
includes Dan’s Learning and Cognition course we explored in an earlier
post. We first compared how each platform
answered this straightforward question.
ChatGPT’s Response: The cognitive-associationist, socio-constructivist, and
sociocultural approaches to education are three different perspectives that have
different assumptions about how learning occurs and how best to facilitate it.
1. Cognitive-Associationist Approach: This approach to education is
rooted in behaviorism and cognitive psychology. It emphasizes the role of
reinforcement and punishment in learning and focuses on observable behaviors.
It suggests that learning is a process of acquiring new associations between
stimuli and responses, and that repetition and practice are important for
learning. In this approach, the teacher plays a central role in transmitting
knowledge to the learner.
2. Socio-Constructivist Approach: This approach to education is
based on the work of Jean Piaget and Lev Vygotsky. It emphasizes the importance
of social interaction and collaboration in learning. It suggests that learning
is a process of constructing new knowledge through interaction with the
environment and other people. In this approach, the teacher serves as a
facilitator of learning, helping students to construct their own understanding
of the world.
3. Sociocultural Approach: This approach to education emphasizes
the cultural and social context in which learning takes place. It suggests that
learning is a process of acquiring cultural practices and ways of thinking, and
that language and social interaction play a central role in this process. In
this approach, the teacher is seen as a mediator of learning, helping students
to connect with the cultural practices and ways of thinking of the community.
Overall, the
cognitive-associationist approach emphasizes the role of the individual
learner, the socio-constructivist approach emphasizes the role of social
interaction, and the sociocultural approach emphasizes the role of cultural
context in learning.
Bard’s Response: Cognitive-associationist, socio-constructivist, and
sociocultural approaches to education are three different ways of thinking
about how people learn. Cognitive-associationist approaches focus on how people
learn by making associations between different ideas. Constructivist approaches
focus on how people learn by constructing their own knowledge through
experience. Sociocultural approaches focus on how people learn in social
settings, through interactions with other people.
Here are some key
differences between cognitive-associationist, socio-constructivist, and
sociocultural approaches to education:
·
Goals: Cognitive-associationist
approaches aim to help students learn facts and information. Constructivist
approaches aim to help students learn complex concepts and to think critically.
Sociocultural approaches aim to help students learn both facts and information
and complex concepts, and to think critically.
·
Focus: Cognitive-associationist
approaches focus on the individual learner. Constructivist approaches focus on
the learner's interaction with the environment. Sociocultural approaches focus
on the learner's interaction with other people.
·
Methodologies: Cognitive-associationist approaches often use traditional
methods of instruction, such as lectures and textbooks. Constructivist approaches
often use active learning methods, such as group work and problem solving.
Sociocultural approaches often use methods that combine elements of both
cognitive-associationist and constructivist approaches.
Each approach has
its own strengths and weaknesses. Cognitive-associationist approaches can be
effective for teaching facts and information, but they can be less effective
for teaching complex concepts or for teaching people to think critically.
Constructivist approaches can be effective for teaching complex concepts and
for teaching people to think critically, but they can be less effective for
teaching facts and information. Sociocultural approaches can be effective for
teaching both facts and information and for teaching complex concepts, and for
teaching people to think critically, but they can be less efficient than other
approaches.
The best approach to
education is likely to vary depending on the subject matter, the students'
learning styles, and the teacher's preferences.
New
Bing’s response: Cognitive-associationist, socio-constructivist, and
sociocultural approaches to education are all different theories of learning.
Cognitive-associationist theory is based on the idea that learning is a process
of forming associations between stimuli and responses. In contrast,
socio-constructivist theory emphasizes the importance of social interaction in
learning. It suggests that learning is a process of constructing knowledge
through social interaction with others. Sociocultural theory is similar to
socio-constructivist theory but places more emphasis on the role of culture in
learning. It suggests that learning is a process of acquiring cultural
tools and practices through social interaction with others123.
I hope this helps! Let me know if you have any other questions.
1.
gsi.berkely.edu 2. verywellmind.com 3. europeanpublisher.com
The most obvious difference is that Bing clearly searched the web for the
information and provided hyperlinks to the sources. Bing also provided a much shorter version in
the initial response. Asking Bing to
“add more detailed” broke the response out into three (similar) paragraphs and added a one-sentence example to each (e.g., “An example of this approach is
rote memorization”). We also see that Bard structured the response differently
and (arguably) more creatively by contrasting the three theories in terms of
goals, focus, and methodologies.
All three responses show that each platform generated a fairly-sophisticated
response. In fact, an undergraduate might request a response “like a typical
undergraduate” or select one of the simpler drafts from the three offered by
Bing.
Engineering the Prompt with Local Information
For our local information prompt, we imagined that an instructor asked students
to apply these theories “to a classroom they just visited.” We saw versions of this as a “revised prompt” proposed by
several university teaching and learning centers. So we imagined that students
visited a classroom where the teacher was delivering a traditional
direct-instruction phonics lesson:
Our engineered prompt: Imagine that a second-grade teacher just delivered a
direct-instruction phonics lesson. Explain what happened using
cognitive-associationist, socio-constructivist, and sociocultural theories of
learning.
Following are the responses to this engineered
prompt from each platform.
1. ChatGPT’s
response: Cognitive-Associationist Theory: According to the
cognitive-associationist theory of learning, the second-grade teacher's
direct-instruction phonics lesson would focus on teaching students to associate
specific letter sounds with corresponding phonemes. The teacher would use a
variety of techniques, such as repetition, drill and practice, and positive
reinforcement to help students learn and reinforce these associations. The
cognitive-associationist theory emphasizes the role of the individual learner
and focuses on observable behaviors.
2. Socio-Constructivist Theory: From the socio-constructivist
perspective, the second-grade teacher's direct-instruction phonics lesson would
be an opportunity for students to construct new knowledge through social
interaction and collaboration with their peers. The teacher might encourage
students to work together in small groups, engage in discussions about sounds
and words, and participate in hands-on activities that allow them to experiment
with phonics concepts. The socio-constructivist theory emphasizes the
importance of social interaction and the role of the teacher as a facilitator
of learning.
3. Sociocultural Theory: The sociocultural theory of learning
emphasizes the role of cultural context and language in learning. From this
perspective, the second-grade teacher's direct-instruction phonics lesson would
be situated within a broader cultural and linguistic context. The teacher might
draw on examples from diverse cultures and encourage students to think about
the ways in which different languages and dialects represent sounds and words.
The teacher might also emphasize the importance of literacy in society and how
it connects to broader cultural practices. The sociocultural theory emphasizes
the role of the teacher as a mediator of learning, helping students to connect
with the cultural practices and ways of thinking of the community.
Bard’s response: Sure. Here is how a direct-instruction phonics lesson can be explained
using cognitive-associationist, socio-constructivist, and sociocultural
theories of learning:
Cognitive-associationist theory:
In a cognitive-associationist theory of learning,
students learn by making associations between different ideas. In a direct-instruction
phonics lesson, students learn the association between the letter sounds and
the corresponding sounds in words. They also learn the association between the
different letters and their corresponding sounds.
Socio-constructivist theory:
In a socio-constructivist theory of learning,
students learn by constructing their own knowledge through experience. In a
direct-instruction phonics lesson, students construct their own knowledge of
the letter sounds and the corresponding sounds in words by actively
participating in the lesson. They also construct their own knowledge of the
different letters and their corresponding sounds by actively listening to the
teacher and by practicing the sounds.
Sociocultural theory:
In a sociocultural theory of learning, students
learn in social settings, through interactions with other people. In a
direct-instruction phonics lesson, students learn in a social setting, through
interactions with the teacher and with other students. They also learn through
interactions with the materials in the lesson, such as the letters and the
sounds.
Each of these theories of learning can provide
insights into how students learn in a direct-instruction phonics lesson.
Cognitive-associationist theory can help us understand how students learn the
association between the letter sounds and the corresponding sounds in words.
Socio-constructivist theory can help us understand how students construct their
own knowledge of the letter sounds and the corresponding sounds in words.
Sociocultural theory can help us understand how students learn in a social
setting, through interactions with the teacher and with other students.
New
Bing’s response:” Sure! Here’s how each theory might explain what
happened in the lesson:
· Cognitive-associationist
theory: In this theory, the teacher would have used
repetition and reinforcement to help students form associations between sounds
and letters. For example, the teacher might have had students repeat a sound
several times while pointing to the corresponding letter.
· Socio-constructivist
theory: In this theory, the teacher would have encouraged
students to construct their own knowledge through experiences and reflection.
For example, the teacher might have had students work in pairs to come up with
words that contain a particular sound.
· Sociocultural theory: In this theory, the teacher would have emphasized the
importance of cultural context and social interaction in learning. For example,
the teacher might have had students work in small groups to practice reading
words that contain a particular sound.
I hope this helps! Let me know if you have any other questions.
Notably, ChatGPT and Bing mostly described different approaches, rather than using the different theories to
explain the practice. This conflation of theories of learning and practices of teaching is quite common with undergraduate and master’s level
teacher education students. In contrast Bard indeed attempted to explain this practice
from each of the the three perspectives. Notably, it did so without acknowledging
that practice is most consistent with cognitive associationist theories (and
less consistent with socio-constructivist and situative/sociocultural theories).
We tried this test with other practices (problem-based learning and
collaborative science investigation) and got similar responses.
These examples showed that all
three platforms “learned” at the intersection of learning theory and educational
practice. But they learned in different ways. Significantly, these differences
are are remarkably akin to the sorts of differences in the ways that real
students learn. We contend that this ability to learn (beyond what was
presumably contained in their LLMs or the web resources accessed) is crucial for
thwarting and supporting student learning using generative
AI
So What Are Educators to Do?
First, this example demonstrates
the value of perhaps the most common recommendation for educators: Test your assessments and assignments with
generative AI. This post further reminds
educators that many students will engineer such “knowledge-rich” prompts, and
some will do so in ways that thwart learning. We further remind educators that
careful research (as summarized in James Lang’s Cheating
Lessons) has shown that many
students will cheat if they assume (a) their classmates are doing so and (b) they
will not be caught. Recent survey evidence showing cheating
with ChatGPT and limitations
of current AI detection systems suggests that many educators (particularly those
online) have reason to be concerned. We acknowledge that our position is at
odds with more trusting observers. But we worry that most efforts to support
pedagogical prompt engineering will fail if they don’t first consider problematic
prompt engineering.
Some teacher educators will have
already recognized the pedagogical potential
of the three different responses
above. Many teacher educators know that this tendency to conflate theory and
practice is deep-seated. Mere explanation is insufficient for addressing this conflation. Fortunately, exploring the differences between the three responses looks like a very promising assignment. The relative potential of such an assignment raises deeper questions about learning theory that quickly gets beyond the scope of this
post. In short, this has to do with the perceived value of student inquiry and the relative value of different forms of inquiry. Constructivist perspective
might worry letting generative AI generate these examples (rather than students) thwarts the learning that occurs when generating the example. Situative
perspectives might worry that using AI to frame learning in this fashion would hinder
the transfer-in of prior knowledge and the ownership and authorship of
ideas that follow. On the other hand, such student inquiry can be messy and inefficient and is particularly
challenging online.
Our thinking here is generally
consistent with a recent Cult of
Pedagogy podcast entitled “How to Use ChatGPT
as an Example Machine.” The deeper (and ultimately empirical) question is whether having students engage in inquiry
around AI-generated examples is as effective as investigating (a) student-generated
examples or (b) educators’ or designers’ expert-generated examples. A pragmatic
answer must take into account the different opportunity costs of all three
options.
Transitioning to Pedagogical
Prompt Engineering
The example above sets up our
transition. Contrasting the three examples above sets up students to compare each
platform’s ability to critically analyze the relationship between learning theory
and educational practice. Thus, we asked each platform the same question:
Our pedagogically engineered prompt: Is direct phonics instruction consistent with
socio-constructivist learning theory?
Here are the initial responses from
the three different platforms
ChatGPT’s response: Direct phonics instruction and socio-constructivist learning
theory have different approaches to teaching and learning.
Direct phonics
instruction emphasizes the explicit teaching of phonics rules and skills to
help students decode words and develop reading fluency. It is a
teacher-centered approach that focuses on the transmission of knowledge from
the teacher to the student.
Socio-constructivist
learning theory, on the other hand, emphasizes the importance of social
interaction and the construction of knowledge through collaborative learning
activities. It is a student-centered approach that focuses on the active
engagement of students in the learning process.
While direct phonics
instruction and socio-constructivist learning theory have different approaches,
they are not necessarily mutually exclusive. It is possible to incorporate
elements of both approaches into teaching phonics. For example, teachers can
use direct instruction to teach phonics rules and skills, and then provide opportunities
for students to practice and apply these skills in collaborative learning
activities.
In summary, while
direct phonics instruction is not consistent with the socio-constructivist
learning theory in its entirety, it can be adapted to fit within the framework
of a socio-constructivist approach to learning.
Bard’s
response: Direct phonics instruction is a systematic,
teacher-led approach to teaching students how to sound out words and read. It
is based on the idea that students learn best when they are directly taught the
skills they need to be successful. Socio-constructivist learning theory is a
theory of learning that emphasizes the importance of students actively
constructing their own knowledge through interaction with others.
Direct phonics instruction can be consistent with
socio-constructivist learning theory when it is used in a way that allows
students to actively participate in their own learning. For example, students
can be given opportunities to work together to decode words, or to create their
own phonics games. When students are given these opportunities, they are able
to actively construct their own knowledge about phonics and reading.
However, direct phonics instruction can also be used
in a way that is not consistent with socio-constructivist learning theory. For
example, if students are simply given a list of phonics rules to memorize, or
if they are drilled on phonics skills without any opportunity to apply them,
then this type of direct phonics instruction would not be consistent with
socio-constructivist learning theory.
In conclusion, direct phonics instruction can be
consistent with socio-constructivist learning theory when it is used in a way
that allows students to actively participate in their own learning. However, it
can also be used in a way that is not consistent with socio-constructivist
learning theory.
New Bing’s response: Direct phonics
instruction is a behaviorist approach to reading that teaches readers to
associate letters with their sounds. On the other hand, social constructivist
theory emphasizes the child's environment as they learn to read1. In
constructivism, students are actively involved in discovering knowledge on
their own2. The teacher guides students in discovering knowledge on
their own. In summary, direct phonics instruction is not consistent with
socio-constructivist learning theory1, 2.
Learn more: 1. theclassroom.com 2. brighthubeducation.com
We see here that the engineered
prompt helped all three platforms distinguish educational practice and
learning theory. We contend that asking students to explore and describe these
differences is a promising way to help them learn the underlying issues. If this
exercise was indeed grounded in actual classroom practice that the students had
observed, students could readily extend this investigation using that experience.
For example, these responses might prompt students to consider whether they had
indeed observed teachers using some of these socio-constructivist practices to
supplement direct instruction in phonics.
Where to Go From Here?
Many readers likely recognize that our
example enters directly into the controversies associated with the “science
of reading” and the corresponding insistence on direct phonics instruction over other methods. Personally, we think it is anti-scientific to
claim that anything else is “non-scientific” and not “evidence-based.” Such
simplistic proclamations obscure the real issues and fuel corrosive denigration
of educators and schools of education. But the debate rages, sometimes even within
particular teacher education programs. Given that the majority of professors of
reading still ascribe
to a “balanced” approach, new elementary teachers must be prepared to engage
productively in this important discussion.
Building on the example above, one possibility
is having students explore what the different platforms have to offer. We found that asking each to “describe the controversies
associated with ‘the science of reading’” generated three somewhat different
responses:
·
ChatGPT
juxtaposed phonics instruction with whole-language instruction and then
considered the implication of each for drawing on students’ prior and using
social media.
·
Bard broke down the controversies associated with phonics instruction, phonemic
awareness, fluency, and comprehension
·
Bing cited
an
editorial from the Washington Post to assert that “the science of reading
is more about reducing ignorance than finding ultimate truths applicable to
every child” and “in the reading wars, scientific certainty is often used rhetorically
to deny reasonable differences and cut off healthy debate, turning science into
scientism.”
We contend that contrasting,
discussing, and synthesizing these three responses would itself be a worthwhile
exercise for teacher-education students. But it would also support further pedagogical
prompt engineering. For example, asking
each platform “what evidence supports balanced reading instruction” (again)
generated very different responses:
·
ChatGPT
summarized balanced reading outcomes beyond standardized reading scores (e.g., enjoyment
of reading and oral language skills, etc).
·
Bard stated
that the 2000 National Reading Panel (often cited by proponents of stricter
methods) also called for comprehension instruction and that a more recent
report by the US DOE’s National Center for Educational Evaluation and Regional
Assistance summarized the evididence supporting balanced reading.
·
Bing summarized
linked resources detailing evidence in support of balanced instruction along
with resources articulating the problems in defining one approach as being uniquely
“evidence-based.”
Asking a similar question about
direct phonics instruction provided generally similar results. But reflecting
the influence and widespread attention to the science of reading, all three
were somewhat more assertive in their support of direct instruction
over other methods.
At this point, it would be great to
hear from colleagues who teach reading. But from our perspective, asking students
to compare these responses across and within platforms would be a powerful and
interesting way of learning about these complex issues. But even in this
regard, some caution is warranted. We asked each platform to “compare opinions
of ChatGPT, Google’s Bard, and Microsofts Bing regarding the science of reading:
·
ChatGPT
described the differences between ChatGPT’s reliance on its LLM from Bing’s
reliance on search (and their general implications) without actually answering
the question. ChatGPT refused to even recognize Bard. Instead ChatGPT hallucinated “Google’s BARD (Basic Reading and Decoding)” a “computer-based reading program.”
·
Bing apologized
for being unable to compare ChatGPT and Bard “as they are not designed to have
opinions. Bing offered to provide it’s own information “but it does not have an
opinion on it as well.” (Deleting “opinion” from the prompt resulted in the
exact same response.)
Somewhat to our surprise, Google’s
Bard generated a serious effort to answer our question
Bard’s
response: ChatGPT, Google's Bard, and Microsoft's Bing are all
large language models (LLMs) that have been trained on a massive amount of text
data. As a result, they all have a deep understanding of the English language
and can generate text, translate languages, write different kinds of creative
content, and answer your questions in an informative way.
However, when it comes to the science of reading, these LLMs have different opinions. ChatGPT is a strong supporter of the science of reading, and believes that it is essential for teaching children to read. Google's Bard is more cautious in its endorsement of the science of reading, and believes that it is important to consider other factors, such as a child's individual learning style, when teaching reading. Microsoft's Bing is the most skeptical of the three LLMs, and believes that the science of reading is not as well-established as some people claim.
Despite their different opinions, all three LLMs agree that reading is an important skill that all children should learn. They also agree that there is no one-size-fits-all approach to teaching reading, and that the best approach will vary depending on the child's individual needs.
Here is a more detailed comparison of the opinions of ChatGPT, Google's Bard, and Microsoft's Bing regarding the science of reading:
ChatGPT is a strong supporter of the science of reading. It believes that the science of reading is essential for teaching children to read. ChatGPT argues that the science of reading provides a clear and concise framework for teaching reading, and that it is based on decades of research. ChatGPT also argues that the science of reading is effective in teaching children to read.
Google's Bard is more cautious in its endorsement of the science of reading. It believes that the science of reading is important, but that it is not the only factor that should be considered when teaching reading. Google's Bard argues that the science of reading is based on averages, and that individual children may learn to read in different ways. Google's Bard also argues that the science of reading is not as well-established as some people claim.
Microsoft's Bing is the most skeptical of the three LLMs. It believes that the science of reading is not as well-established as some people claim. Microsoft's Bing argues that the science of reading is based on a small number of studies, and that these studies have been criticized for their methodology. Microsoft's Bing also argues that the science of reading is not always effective in teaching children to read.
Despite their different opinions, all three LLMs agree that reading is an important skill that all children should learn. They also agree that there is no one-size-fits-all approach to teaching reading, and that the best approach will vary depending on the child's individual needs.
While the accuracy of these claims
might need to be more fully verified, Bard’s response was quite compelling. Unless
an instructor had tested the prompt or seen such a response from multiple
students, this might be an entirely plausible student response. That is a central
point of our larger series!
Finally, we wonder how generative AI might help prepare future teachers navigate the contentious political backdrop which further obscures the science behind the rhetoric of the science of reading. Consider, for example, the widely-cited claims that ChatGPT is "woke" and the inevitable counterclaims. This is significant because conservative legislators and journalists are among the strongest proponents of the science of reading, while teachers' unions are among the strongest skeptics. Asking students to evaluate the stance of each platform in light of this reality seems like a promising assignment for introducing students to the complex crucial issue/
Posts
