AI.4 Refining Topic Cards For The Essay
Kill your topics before they become needless paragraphs.
The next three steps in developing the scaffold are organizing and evaluating topics (Section 2.7); enhancing existing topic cards (Section 2.8); and converting subtopics into paragraph topics (Section 2.9). My next step was to carry these operations out. However, I felt when I generated topics I was already ‘thinking in paragraphs’, so I emphasized organizing and evaluating my topics. I believed most of my topic cards already expressed paragraph topics, so I had little need to enhance topics (i.e., dividing topics into subtopics) or to convert subtopics into paragraph topics. Nevertheless, the Table AI-1 topics were not organized in proper order; I needed to evaluate my topics and I had to remove many.
I conducted topic organization and evaluation in three phases. The first occurred on May 28, 2023. I took the Table AI-1 topic cards and arranged them in an order which made sense to me. I then placed the cards in a deck and worked through the deck from top to bottom. I looked at the top card on the deck and decided whether I needed to include the topic in the essay. (Remember, Table AI-1 sent me a clear signal: prune!) If I felt the topic card was necessary, I wrote the topic (with some possible editing) on a blank index card; I was creating a new stack of index cards. If I felt the topic card was unnecessary, then I moved to the next card in my Table AI-1 deck.
At the end of the first pass of processing, I had generated a smaller deck of topic cards – my new set included only 22 topics. I laid the new cards out on my coffee table and evaluated their order. Happy with the narrative, I placed them into a deck and put the cards away.
I conducted my second pass the next morning. I took my 22 topic cards and read them in order. As I read the cards, I treated each as expressing a paragraph topic, and I began to think about what the paragraph might say. I found myself satisfied with the narrative until I reached Card 18. At the time, Card 18 was followed by the card numbered 23 in Table AI-2. I had difficulty linking the two paragraph topics. So, I added a new card – the version of Card 19 which is crossed out in Table AI-2. I felt the new topic fixed the narrative. I then numbered all the topic cards because I felt they were ready to be used as cues for writing topic sentences.
When I moved to writing topic sentences later the same afternoon, I discovered narrative problems when I hit the newly added Card 19. I realized I needed to provide more information to make my narrative clearer. Such a discovery indicates I should have expended more effort on the two steps I mostly ignored, refining topics and expanding topics into paragraph topics. Clearly, I had not thought in paragraphs while generating all of my topics.
Discovering my problem, I retreated from writing topic sentences, and returned to developing topic cards. I replaced Card 19 with a new topic, and I then added three additional topic cards to fix my narrative. These topic cards were created during the afternoon of May 29. With the new topic cards, I was able to finish developing topic sentences for my essay’s paragraphs.
The second phase of scaffolding accomplished two goals. First, I reduced the number of topics to permit me to write a shorter essay. By evaluating my topics, I eliminated about half of the topics which appear in Table AI-1. Second, I organized the remaining topics into a solid narrative order. With topics (i.e., paragraph topics) in the correct order, the topics can be converted into sentences.
| Table AI-2. Topic cards after organizing and evaluating the Table AI-1 topics. The first pass created most of the cards provided in the table. The second pass added one new card (the crossed-out version of Card 19) but did not include the four cards (19-21) written in bold font. The cards were numbered during the second pass. The third pass removed the old Card 19 and added the four topic cards in highlighted rows. |
|---|
| 1. What is a large language model (LLM)? |
| 2. LLMs can do amazing things – code, chat, see examples from Mitchell & K., Stokel-Walker |
| 3. LLM abilities lead to big questions: Do they understand language? Are they intelligent? Hotly debated – Mitchell & K |
| 3B. NY Times headline May 16 2023: ‘Microsoft Says New A.I. Shows Signs of Human Reasoning’ |
| 4. For me, more productive question: How might LLMs inform cognitive science? |
| 5. My answer: Can only inform cognitive science if we look past performance and examine internal structure |
| 6. Roadmap here? Decide later |
| 7. LLMs are exciting because of their facility with language |
| 8. In cognitive science, language is explained by appealing to symbols and rules – Chomsky NY Times essay? |
| 9. Chomsky argues LLMs can’t inform cognitive science because LLMs don’t use rules. Veres agrees? |
| 10. Piantadosi agrees LLMs don’t use rules like Chomsky – but then claims this means LLMs refute Chomsky |
| 11. Piantadosi position has historical precedent |
| 12. 1980s PDP nets cause connectionist revolution due to new powerful ANNs |
| 13. Connectionist revolution attacked theories based on symbols and rules – see ‘What is cognitive psychology?’ |
| 14. My concern: connectionist revolutionaries assumed no rules, but didn’t look inside to support the claim – gee whiz connectionism |
| 15. When I looked inside, I saw lots of formal structure (logic network, mushroom network examples) |
| 16. But – network formal structure can be novel and informative (musical network examples) |
| 17. Moral: to inform cognitive science, don’t gee whiz, look inside |
| 18. Consequence: LLMs may indeed inform cognitive science, but only after you figure out how they work |
| 19. Need to look inside because LLMs use very different representations (May 28 added, May 29 deleted) |
| 19. Why do we need to look inside? Different methods produce same behavior; we need to look at methods |
| 20. We know methods are different, because LLMs use novel representations. |
| 21. We need to focus on methods, too, because performance is too tempting – mistake to speculate on child language learning |
| 22. LLM proponents know theory is there – need to figure it out and see if the theory applies to humans |
| 23. Problem: LLMs are big – go to ChatGPT manual, Mitchell & Krakauer quote |
| 24. Researchers are developing interesting new techniques to explore innards of LLMs – manning emerging hierarchies paper |
| 25. More work like Manning’s required |
| 26. Do LLMs refute Chomsky? Don’t know now – look inside to find out! |
