SymSys245:
Interaction Analysis
The Symbolic System Program
Stanford University
Professor Jeff Shrager (jshrager@stanford.edu)
SUMMARY
In this course, we investigate how cognitive processes such as perception, learning, reasoning, and problem solving, inform the design and analysis of complex, interactive systems. We both study these important cognitive processes, and analyze the ways in which they operate in important real-world contexts. Major topics include: cognitive architectures, analysis of skilled performance, complex learning and discovery, adaptive and intelligent interfaces, user assistance systems, and special topics according to students' interests. We also try to consider issues related to overlooked populations, such as the elderly, people with special cognitive needs (e.g., dyslexia), and children.
The class takes place in an interactive discussion format, and participation in these discussions is crucial to the learning process. In addition to weekly readings, and small in- and out-of-class projects, students will produce a larger project examining interactive cognition in a real world setting of their own choosing. Although there are no formal prereqs., exposure to cognitive psychology and/or AI will be helpful.
Prep for Week 3: Problem Solving (iGR) [This is prep for week 3 isntead of week 2 because this year there's no class in week 2!]
___ Read: Course Details Document
___ Read:
→ Bibby and Payne reading link [pdf]
→ Pitot/Static system and symptoms of blockage [link] (esp. read the section on blocked static ports).
→ Aircraft accident analysis of Aeroperu 603. [pdf]
Reading notes: First do a "quick read" of Bibby and Payne to get the idea of the device they are studying and the three sorts of device knowledge that they are investigating (diagram, procedures, table). Next study the online explanations of aircraft instruments, and the "pitot-static" system, esp. the section on system malfunctions from blocked static ports. Try to get your own mental model of how they work. Finally, read the analysis of the crash of Aeroperu flight 603, from Aircraft Accident Analysis. Understand what happened as best you can, but really focus on what the pilots were saying; Try to imagine it as a play going on in the cockpit. (There are parts of this that are in "pilot speak", that you won't understand unless you are a pilot. That's okay; I promise that you'll get the idea!)
Some aircraft vocabulary: ATC = Air Traffic Control (i.e., Ground control). Transponder = A radio that automatically sends some information (esp. altitude) to ATC from the airplane. Stall = A plane has to fly above its "stall speed" to maintain lift and stay in the air. If it goes below this speed, it will simply fall out of the sky. Stick Shaker = A gizmo that physically shakes the pilot's steering wheel in order to indicate that the plane is flying too slowly, and is about to stall. (This is an interesting device because it physically simulates the feel of what would actually happen on a smaller airplane when you approach stall speed!) Radial = A compass point (the 360 radial is north, 090 radial is east, etc. If you are, for example, "thirty miles on the two seven zero (270) radial" you are thirty miles due west.) ILS = Instrument Landing System; A radio system, located at the airport, that provides precision altitude and distance information so pilots can land in bad weather. Vectors = Instructions from ATC telling lost pilots how to turn to get someplace. GPWS = Ground Proximity Warning System; A down-looking radar that yells "Too low; Terrain!", and "Pull up!" if you get too close to the ground. (Artificial) Horizon = Gyroscopic instrument that shows the plane's roll and pitch with respect to the ground. (Note that this does not depend upon the pitot/static system! It's just a gyroscope.) Flight director = A sort of combined electronic Horizon and autopilot that tells you how to fly a particular course.
Optional (but worth watching if you want to get a better sense of what this sort of accident entails): There are several interesting (in a horror movie sort of way) videos that depict various aspects of the Aeroperu accident, for example: [this one of the whole CVR] and [this one recreating the last few minutes of the flight].
___ Observational Activity:
Read Notes for the Standing Observational Activity.
Email me (jshrager@stanford.edu) 1-2 paragraphs on each of two observations . (That is, 2-4 paragraphs all together.)
(I strongly suggest just including the observations in the body of the email message, rather than attaching a separate document. If there are pictures, you can attach them.)
__________________________________________________________________________________________________________________________________________________________________________________________
Prep for Week 3: NO CLASS
****** THERE IS NO CLASS EITHER MON OR WED OF WEEK 2 ******
HOWEVER, I STRONGLY SUGGEST DOING THE READING (first part of the long paper)
___ Read:
→ PART ONE OF: Pirolli, P. (1999) Cognitive Architectures ... [PDF] ONLY up to and including page 466
This is a relatively long and complex paper, and there are some mathematical parts. We're going to be reading it in TWO PARTS. This week read ONLY up to and including page 466! Read for the high level concepts, NOT the mathematical detail. (The mathematical detail is probably wrong anyhow.)
Things to think about: Why are some tasks harder than other, very similar tasks? Why is poker harder than solitaire? (Is it?) Why is flying a 747 harder than flying a flight simulator? (Is it?) Why is navigation in a foreign country harder than navigation in an unfamiliar city in a country that you're used to? (Is it?) Choose a few examples of your own and explain what aspects of cognitive architectures help you to figure this out (if any)? Are there missing elements of CA's that would help you to answer this?
___Try out The Mystery Machine and take a look at these notes on it.
__________________________________________________________________________________________________________________________________________________________________________________________
Prep for Week 4: Cognitive Architecture and Cognitive Operators
___ Read:
→ Cognition for Readers With Short Attention Spans.
→ PART TWO OF: (i.e., pg. 466 to the end) Pirolli, P. (1999). Cognitive Architectures and Cognitive Engineering Models in Human-Computer Interaction. The handbook of applied cognition. John T. Wiley, Sussex, England.[PDF]
___ Watch: Art Benjamin's TED talk (at least from 10:30 through the end, but the whole thing is fun).
___ Observational Activity:
In the next set look especially for the concepts we talked about in the problem solving discussions in the previous week, esp. goals and impasses, subgoals, resources (operators, methods, attention, reasoning, information gathering, and more general resources like time, etc.), and especially interpretations. Also attend to whether the setting is more like a "hot" vs. "cold" cognitive one, or maybe it varies between these (which is quite common!), and what causes it to this way, and/or change.
As needed, review the notes for the standing observational activity. Email me (jshrager@stanford.edu) ~1 paragraph on each of two observations - that is, 2 paragraphs all together. (I suggest just including the observations in the body of the email message, rather than attaching a separate document. If there are pictures, you can attach them.)
___Because I know you didn't do this last week, please try out The Mystery Machine and take a look at these notes on it.
__________________________________________________________________________________________________________________________________________________________________________________________
Prep for Week 5: Skill Acquisition
___ Read:
→ Agre, PE and Shrager, J (1990). Routine evolution as the microgenetic basis of skill acquisition. In Proceedings Cognitive Science Conference- Boston, pages 694-701. [PDF]
Things to think about: In what way, and to what extent is the Agre & Shrager (A&S) theory a challenge to the theories of skill acquisition discussed in, for example, Pirolli's paper? How does the A&S story of "microgenetic" change fit with your own observations of how you learn things? Can you think of an example of something that you've learned recently by pure practice? How about something you learned through small changes in methods? What sorts of domains does each sort of learning apply to?
___ Mystery Machine experiment -- Part 1:
___Once again, please get familiar with The Mystery Machine and take a look at these notes on it.
In this part of the experiment all you're going to do is to train a friend to give verbal protocols in long division. Next week you'll get them to figure out the ? key in the Mystery Machine.
Here are the details. We're only doing PART 1 this week!
___ Skim: PROTOCOL ANALYSIS: methods for eliciting and analyzing valid verbal reports on thinking.
Here are some helpful notes on think-aloud protocols
___ Conduct: PART 1 of the Mystery Machine and verbal protocols.
___ There is nothing to turn in on this activity this week!
___ Observational Activity:
This week pay special attention to the cognitive operators that we talked about in class this past week, esp. use of various memories, attention, how you keep track of where you are in complex activities, and the use of the "conscious spotlight" that you need to use to reason about things.
As usual, Email me (jshrager@stanford.edu) ~1 paragraph on each of two observations - that is, 2 paragraphs all together.
__________________________________________________________________________________________________________________________________________________________________________________________
Prep for Week 6 : Complex Learning: Mystery Machine Experiment
Last week you should have trained a friend to give verbal protocols in long division. This week you'll get them to figure out the ? key in the Mystery Machine. Here are the details. Between last week and this week you should have done PART 1 (practice), and this week you'll do PARTS 2-3 (experiment) AND 4 (analysis). (If you didn't do part one last week, you'll have to do all parts this week.)
___ Reporting: Be sure to check out this example transcript with comments. You don't have to stick to this exact format. And you don't have to transcribe really boring experiments. Focus on the places before and after critical events where the subject realized something important.
___Reading:
Klahr, D., & Dunbar, K. (2000) A paradigm for investigating scientific discovery in the psychology lab. [pdf]
Turner and Fauconnier (1995) Conceptual Integration and Formal Expression. IMPORTANT READING NOTES: Mostly you can skim this paper to get the basic idea (mainly summarized in figure 1 on pg. 18). However, please read in somewhat more detail pages 1-9 (first and second sections) and then 17-23 (esp. the figure on pg. 18).
Things to think about: You're going to wonder what these readings (and listenings) have to do with one another. They all deal with aspects of complex learning, but they come at it from very different perspectives. Klahr is concerned with the structure of experiments and the relationship between experiments and hypotheses. Conceptual Blending is concerned with how novel theories (or knowledge) are created by combining ("blending") different domains of background knowledge to get creative new (mental) models. And the podcast on meaning and interpretation is concerned mostly with how we understand language. Can you see how these three complement one another? Conceptual Blending and Interpretation are supposed to be fundamental cognitive processes that apply all over the place. Think of some commonplace activities, other than exploratory learning, that involve a lot of the same sort of interpretation or re-interpretation processes. Hint: We've talked about a lot of these in class. Double hint: You're doing it RIGHT NOW! ... Triple hint: Actually, you're doing it all the time! (Except maybe when you're asleep!)
___________________________________________________________________________________________________________________________________________________________________________________
Prep for Week 7: NO CLASS
****** THERE IS NO CLASS EITHER MON OR WED OF WEEK 7 ******
HOWEVER, YOU NEED TO INTERACT WITH JEFF ON YOUR FINAL TOPIC (SEE BELOW)
___ Choose a final topic setting:
During this break week you should decide what setting you want to study in more detail for your final project. You might be interested in sports, or art, or video games, or literature, or science -- whatever. Send me and email BEFORE NEXT WEEK (i.e., sometime in this coming week) with the topic area that you are thinking about so that I can approve it.
This page explains the project requirements in detail, and here are some of the projects from years past.
Here's a made-up example:
Many people lose their eyesight or hearing, or become otherwise disabled, late in life. I'm interested in how people modify well-learned strategies when they are faced with novel resource limitations such as these. My idea of how to study this is to study people learning how to cook eggs blindfolded. I'll get a friend who knows how to make scrambled eggs, blindfold them, and get them to try it again several times. I'll video this, and then post it on youtube and it'll go viral as everyone laughs at my friend making a mess and burning themselves over and over ... no, but seriously, I'll video tape this under think aloud instructions, and then analyze for how they modify their strategies from one run to the next.
The goal is to get the topic approved before the next class (week 8) so that we can talk about methods.
__________________________________________________________________________________________________________________________________________________________________________________________
Prep for Week 8: Methods Survey & Project Discussion
This week we're going to get practical. We'll start by reviewing several different interaction analysis methods, and then talk briefly about each of your project ideas, and discuss how various methods might apply to your topic.
___Reading:
Nielson, J.: How to conduct a heuristic evaluation [link]
Jeffries et al. (1991). User Interface Evaluation in the Real World: A Comparison of Four Techniques [link] (Ignore the statistical methods; Just focus on the conclusions, esp. Table 6)
___Skimming:
J Nielson: Ten usability heuristics [link]
AskTOG: First principles (for interface design) [link]
Things to think about: User testing and usability analysis are nearly always painted in terms of computer interfaces, but in this class we've been looking at situations beyond computers. The same ideas that underlie the various testing analyses and recommendations apply to many of the real-world settings that you've observed in the standing observational practice. Which of these techniques do you think would be best suited to studying the design of an educational toy for preschool children? How might you go about user testing a new cockpit interface for a new airliner? How about a new traffic pattern for a busy highway interchange? To get really crazy, think about how congress might go about "user testing" a proposed new law!
__________________________________________________________________________________________________________________________________________________________________________________________
Prep for Week 9: Interaction Analysis in the Age of Intelligent Machines
___Reading:
Pezzulo, et al. (Nov. 2023) Generating meaning: active inference and the scope and limits of passive AI. Trends in Cognitive Sciences.
(Although I recommend reading this whole paper, if you're pressed for time, or just lazy, at least read the 4 boxes.)
___GAI observation: This week we're going to be talking about interacting with AIs. Please come to class with an example of where you (or someone you observed) engaged a generative LLM (aka. GAI, for example ChatGPT, Dall-E, etc.) in an extended, goal-directed, interaction. What do I mean by "extended" and "goal directed"? Let's take "goal-directed" first: By this I mean an interaction with the GAI that has a specific purpose, for example trying to get it to program something, create a logo for a club, write a poem, or whatever. By "extended" I mean that the interaction went on for several turns where you (or the person you observed) needed to go back-and-forth with the GAI to get it to create a satisfactory version of whatever was the goal. We're going to talk about this in class, so as with previous observations, bring them to class (at least in your head). You do not have to email me these nor (obviously) do one online and one offline -- these are obviously online! :-)
___ Preliminary Project Run-Through: Try out your project method as proposed in your project plan. No need to turn anything in on this (beyond the plan described above). But come armed with at least a sense of whether it's going to get you some good data.
__________________________________________________________________________________________________________________________________________________________________________________________
Prep for Week 10: Cognitive Variability, Culture, & Details of Project Reporting
Final Notes on the Final Project
Projects are Due by midnight on SUNDAY March 24th
___Carefully review project requirements, esp. "grading criteria" and "reporting".
___Review Glossary of main concepts notes for the course
Quick review of core concepts (see the glossary and/or Cognition for Readers With Short Attention Spans for more detail on these): Problem solving: iGR: Interpretation, goals (and subgoals-- the goal tree), impasses, impasse-driven subgoaling/learning; Resources, including operators, strategies. Regarding strategies vs. methods: Sometimes you;'ll see the term "methods". For our purposes, methods can be thought of just as a name referring to strategies+operators. But one can also thingk of strategies as specific to the setting where methods are more general. For example, instructionless Learning (the "8 E's" from the Mystery Machine experiment) is a general method, so is "try random things" (although that's usually a pretty poor method). And operators are usually thought of as single actions (although this gets pretty vague when you get down to it.) Anyway, for our purposes, strategies and methods can all be thought of as more-or-less different names for just sequences of operators. Note that most operators are found through affordances, and can include action, attention, information gathering, and even inaction (like "wait for something to happen" or "wait for a period of time"). Every resource brings with it constraints, which are the limits of resource utilization or access. The most important resources, esp. In terms of constraints, are time and attention. Mental models (like of the Mystery Machine or Theory of Mind) are kinds of knowledge, and reasoning is the use if knowledge in service of goals. Thus reasoning is a kind of operator., but involves "cognitive" (internal) operators, strategies, and methods. All of this executes on your cognitive machinery, whose primary hardware is memories of various sorts, including short term, long term, procedural (skills and sequences of actions), attentional and attentional spotlights. When studying learning, the most important changes will be strategy changes (although there are likely to be memory changes effects as well), like we saw in the early part of the power law/seeker game, but there is, of course, also knowledge/mental-model learning.