AI.2 Can Large Language Models Inform Cognitive Science?
We live during an artificial intelligence (AI) revolution driven by the large language model (LLM). LLMs, a kind of deep belief network, perform complicated tasks because they contain many intermediate processing layers (LeCun et al., 2015). LLMs also include features for processing language (Dong et al., 2023). LLMs learn to predict which words should come next. They achieve incredible accomplishments with such learning: “Give them a human language description or several examples of what one wants them to do, and they can perform tasks for which they were never trained” (Manning, 2022, p. 132).
LLMs can generate long, detailed, meaningful responses to short prompts. LLMs can accomplish many complex tasks, including editing scientific manuscripts, writing or checking programming code, or brainstorming ideas (Mitchell & Krakauer, 2023; Stokel-Walker & Van Noorden, 2023). OpenAI reports its most recent LLM, GPT-4, took a simulated bar exam; GPT-4 placed in the top 10% of test takers. “What is clear is that these models use language in a way that is remarkably human” (Piantadosi, 2023, p. 4, his italics).
LLMs raise many questions in the popular press and in scholarly journals (Mitchell & Krakauer, 2023). Do LLMs understand language? Are LLMs intelligent? Are LLMs sentient or conscious? A recent New York Times headline exclaims “Microsoft Says New A.I. Shows Signs of Human Reasoning.” Scientific journals fear scientists ask LLMs to write research papers, and worry their reviewers ask LLMs to evaluate submitted papers.
A different question interests me: ‘Can LLMs inform cognitive science?’ Below, I argue LLMs can inform cognitive science – but only if researchers study an LLM’s inner workings to discover how the LLM generates responses.
LLMs excite interest because they generate detailed responses to short prompts. LLMs seem to handle language with human-like skill. However, researchers disagree about the relationship between LLM language processing and human language processing.
Cognitive scientists have studied human language for decades. Cognitive science’s most famous theory about language, generative grammar (Chomsky, 1965, 1966, 1995), proposes human cognition uses specialized rules to manipulate complex sentence representations. Generative grammar represents sentences as tree-like forms called phrase markers. A phrase marker encodes a sentence’s word order, the parts of speech to which its words belong, and the sentence’s hierarchical structure. Rules, called transformations, convert one phrase marker into a different phrase marker. For instance, a transformation can convert a phrase marker representing a statement into a phrase marker representing a question. For Chomsky, language – and all cognition – requires the rule-governed manipulation of symbols.
Cognitive scientists who believe human language is the rule-governed manipulation of symbols do not believe LLMs can contribute to cognitive science (Chomsky et al., 2023; Veres, 2022). Chomsky et al. point out “We know from the science of linguistics and the philosophy of knowledge that [LLMs] differ profoundly from how humans’ reason and use language. These differences place significant limitations on what these programs can do, encoding them with ineradicable defects.”
Other researchers argue LLMs provide alternatives to generative grammar (Contreras Kallens et al., 2023). UC Berkeley psychologist Steven Piantadosi agrees with Chomsky: LLMs do not use grammatical rules (Piantadosi, 2023). However, Piantadosi argues LLMs refute Chomskyan linguistics because of what LLMs accomplish without rules. “The success of large language models is a failure for generative theories because it goes against virtually all of the principles these theories have espoused. In fact, none of the principles and innate biases that Chomsky and those who work in his tradition have long claimed necessary needed to be built into these models” (Piantadosi, 2023, pp. 14-15, his italics).
My own research compares theories based on rules and symbols to theories based on artificial neural networks. I recognize a historical precedent for Piantadosi’s position, a precedent relevant to determining whether LLMs can inform cognitive science. In the mid-1980s, cognitive science experienced its connectionist revolution. Cognitive scientists discovered new artificial neural networks, called multilayer perceptrons, powerful enough to model cognitive phenomena. Multilayer perceptrons contained intermediate processors called hidden units which gave them power: with enough hidden units a multilayer perceptron can learn any mapping between stimuli and responses (Lippmann, 1989).
Multilayer perceptrons caused the connectionist revolution because network proponents attacked theories which appealed to rule-governed symbol manipulation. For example, one network converted present-tense verbs into their past-tense form (Rumelhart & McClelland, 1986). Rumelhart and McClelland argued the network works without using grammatical rules: “We suggest that lawful behavior and judgements may be produced by a mechanism in which there is no explicit representation of the rule” (p. 217).
My own research focuses on problems with arguments like Rumelhart and McClelland’s (1986). Connectionist revolutionaries assumed networks abandoned symbols and rules, but never provided evidence to support their assumption, or to show how their networks replaced symbols and rules. Instead, by assuming networks differed from traditional theories, when they trained networks to perform ‘symbolic tasks’ (like converting verb tenses) they claimed, ‘gee whiz we have a non-symbolic model of the task’. I call such work gee whiz connectionism (Dawson, 2009). I distance myself from gee whiz connectionism when I train networks but then analyze them to discover how my networks actually work.
When I look inside my trained networks, I discover symbol-like properties. For example, I trained one network to solve different logic problems. Inside the network I found logical rules like those taught to philosophy students (Berkeley et al., 1995). In another study, I trained networks to classify mushrooms as being edible or poisonous. Inside the network I discovered a traditional symbol/rule system called a production system (Dawson et al., 1997). My results reveal surprising similarities between network models and symbolic models, blurring the distinctions between the two (Dawson, 1998, 2004, 2013, 2018).
My students and I do not always find structures which replicate current symbolic theories. Instead, we often find new structures, symbolic in nature, but which differ from current proposals about symbols and rules.
For example, I train artificial neural networks to make musical judgements. Inside my networks I find structures strongly related to traditional music theory (Dawson, 2009, 2018; Dawson et al., 2020; Perez et al., 2023). However, my network structures depart from traditional music theory in surprising ways.
Traditional music theory treats Western music as consisting of twelve different pitch-classes (C, C#, B, and so on). In contrast, my musical networks treat Western music as consisting of only six different pitch-classes. My networks treat pitch-classes which measure six semitones apart in traditional theory (such as C and F#) as being identical. In short, when I look inside my networks, I find alien – but formal – music theory.
My research makes me believe LLMs will only inform cognitive science when researchers stop simply assuming LLMs differ from rule and symbol models, and instead start studying the similarities and differences between LLMs and cognitive theories.
Why must we look inside LLMs to inform cognitive science? Cognitive scientists know methods completely different from human cognition can produce human-level performance. Consider Joseph Weizenbaum’s program ELIZA (Weizenbaum, 1966). ELIZA conversed with humans, but Weizenbaum did not build language understanding into his program. “ELIZA shows, if nothing else, how easy it is to create and maintain the illusion of understanding, hence perhaps of judgment deserving credibility. A certain danger exists there” (Weizenbaum, 1966, pp.42-43). Examples like ELIZA show why cognitive scientists should compare processes, not performance.
I suspect LLMs processes differ dramatically from human cognition. LLMs use representations unrelated to any proposed by cognitive scientists. LLMs do not use complete sentences or individual words. Instead, they break sentences into tokens, components smaller than individual words. A token is represented as a hundreds-dimensional number vector; LLMs, assign similar vectors to related tokens. If LLM representations differ from human cognition, then LLMs do not refute Chomsky’s approach. Instead, they refute using Chomsky’s approach to explain LLMs!
LLM advocates recognize their models possess hidden procedures which manipulate language, but also realize the difficulties faced when searching for such procedures. “It becomes clear that it could be hard to determine what is going on, even though the theory is definitely in there” (Piantidosi, 2023, p. 8, his italics). To inform cognitive science, to defend claims like ‘LLMs refute Chomsky’, researchers must do the hard work to discover what methods LLMs use.
LLMs’ size and complexity make the work hard. OpenAI’s ChatGPT learns by adjusting roughly 175 billion different parameters. Another LLM, BERT, contains twelve different large layers of intermediate processors. Mitchell and Krakauer (2023, p. 1) note “the inner workings of these networks are largely opaque; even the researchers building them have limited intuitions about systems of such scale.” Piantadosi (2023, p. 8, his italics) concurs: “In fact, we don’t deeply understand how the representations these models create work.”
Fortunately, researchers can create new techniques to understand a LLM’s internal structure. Consider Christopher Manning’s work (Manning, 2022; Manning et al., 2020). Manning probes an LLM’s structure to determine whether the network represents structures found in generative grammar.
For example, Manning et al. (2020) examined LLM components called attention heads. An attention head determines the importance of one word in a sentence to other words in the sentence, or to words in the output being generated by the LLM. Related words receive higher attention. Manning et al. found attention strength captured linguistic properties. Higher attention linked objects to appropriate verbs, linked prepositions to appropriate objects, linked noun premodifiers to appropriate nouns, and so on.
Manning et al. (2020) also used a structural probe method to detect phrase marker trees represented by an LLM’s processors. They measured the distance between different vectors represented in the network. Items whose vectors are close together in the LLM’s space are also close together in a sentence’s phrase marker. Manning et al. reconstructed phrase markers from network properties. In short, “these models learn and represent the syntactic structure of a sentence” (Manning, 2022, p. 131).
I hope similar research is on the horizon. As researchers explore LLM representations, and study how LLMs use representations to generate responses, we move closer to comparing LLMs to human cognition.
