Class Business
For Thurday (May 27) — Class on Thursday will be over Zoom due to a scheduled internet outage in South Hall)
- Readings:
- [Finish this article if you haven’t done so: Minh Hua and Rita Raley, “Playing With Unicorns: AI Dungeon and Citizen NLP” (2020)]
- Emily M. Bender et al., “On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?” (2021)
- [Optional: If you are interested in the controversy and background behind this article, see Tim Simonite, “What Really Happened When Google Ousted Timnit Gebru” (2021)]
- Alan Liu, “What Is Good Writing in the Age of ChatGPT?” (Commencement Address for the UCSB English Dept., 2023).
Due Thursday: Large Language Models & Text-to-Image Large Models Exercise
Word Embedding (continued)
Neural Networks & Large Language Models
Epigraphs to Frame the Discussion
Nicholas Thompson & Geoffrey Hinton (An AI Pioneer Explains the Evolution of Neural Networks,” 2019)
Nicholas Thompson: Explain what neural networks are. Explain the original insight.
Geoffrey Hinton: You have relatively simple processing elements that are very loosely models of neurons. They have connections coming in, each connection has a weight on it, and that weight can be changed through learning. And what a neuron does is take the activities on the connections times the weights, adds them all up, and then decides whether to send an output. If it gets a big enough sum, it sends an output. If the sum is negative, it doesn’t send anything. That’s about it. And all you have to do is just wire up a gazillion of those with a gazillion squared weights, and just figure out how to change the weights, and it’ll do anything. It’s just a question of how you change the weights.
NT: When did you come to understand that this was an approximate representation of how the brain works?
GH: Oh, it was always designed as that. It was designed to be like how the brain works.
Emily M. Bender, et al., “On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?” (2021)
Contrary to how it may seem when we observe its output, an LM is a system for haphazardly stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning: a stochastic parrot. (¶ 6.1)
Albert Einstein (responding in 1926 to quantum mechanics)
The theory produces a good deal but hardly brings us closer to the secret of the Old One. I am at all events convinced that He [God] does not play dice.
Grant Sanderson’s videos on neural networks and large language models on his 3Blue1Brown YouTube channel
From Word Embedding to Sentence Embedding
- Word Embedding (e.g., Tensorflow Projector)
- Sentence Embedding (e.g., Cohere “playground”
example of results)
- Phillip Schmitt, “I Am Sitting In A High-Dimensional Room” (2020)
AI Large Language Models (LLMs)
Feed Forward
- ml4a (Machine Learning for Artists), “Looking Inside Neural Nets” (n. d.)
- Adam W. Harley, “An Interactive Node-Link Visualization of Convolutional Neural Networks” (2015)
Application to word and sentence embeddings:
Backpropagation & Gradient Descent
Transformers (“Attention” after BERT)
“Fine-tuning” Large Language Models
-
SuperAnnotate, “Fine-Tuning Large Language Models (LLMs) in 2024″
(example)
Text-to-Image Large Models (LLMs)
-
Jay Alammar, “The Illustrated Stable Diffusion” (2022)
(diffusion)
|
|
|
- Nomic explorable map of KREA AI’s Stable Diffusion Search Engline
What Kind of Knowledge and Learning is This Compared to the Way Humans Know and Learn?
How Good is Such Knowledge (Beyond “Bot or Not?”)
Minh Hua and Rita Raley, “Playing With Unicorns: AI Dungeon and Citizen NLP” (2020)
If a complete mode of understanding is as-yet unachievable, then evaluation is the next best thing…. (link)
In this endeavor, the General Language Understanding Evaluation benchmark (GLUE), a widely-adopted collection of nine datasets designed to assess a language model’s skills on elementary language operations, remains the standard for the evaluation of GPT-2 and similar transfer learning models…. Especially striking, and central to our analysis, are two points: a model’s performance on GLUE is binary (it either succeeds in the task or it does not)…. But if the training corpus is not univocal — if there is no single voice or style, which is to say no single benchmark — because of its massive size, it is as yet unclear how best to score the model. (link)
Our research questions, then, are these: by what means, with what critical toolbox or with which metrics, can AID [AI Dungeon], as a paradigmatic computational artifact, be qualitatively assessed, and which communities of evaluators ought to be involved in the process? (link)
AID, as an experiment with GPT-2, provides a model for how humanists might more meaningfully and synergistically contribute to the project of qualitative assessment going forward…. (link)
Our presupposition … is that it is not by itself sufficient to bring to bear on the textual output of a machine learning system the apparatus of critical judgment as it has been honed over centuries in relation to language art as a putatively human practice. What is striking even now is the extent to which humanistic evaluation in the domain of language generation is situated as a Turing decision: bot or not. (link)
- Turing Test
