Several years ago I heard Alan Kay give a talk at the MIT Media Lab about how computers and the internet are transforming human communication, succeeding speech, writing, and the printing press before it as the primary means of pushing ideas around. By way of simile, he continued, if the computer is like writing, and the internet is like the printing press, we are living at a time after the printing press, and before the Enlightenment — when ubiquitous access to books in Europe (by at least the wealthy elite) made for a startling period of rapid intellectual advancement. His talk has always stirred strong feelings in me because it made me want to know what new ways the computer would be used for communication.
Then the other day, I realized that our hands might be very bad inputs for the computer and by Kay’s simile, human communication. Here’s what I mean. In broad strokes, a computer to you and me is a machine that changes makes pictures when we move a mouse or tap our keyboard or more recently just tap directly on the screen. So, in a very real way they are machines which convert manual dexterity into pictures. As an alternative, consider a machine that turns our voices into pictures.
How would a computer like this work? Well, besides a microphone to record your vocalizations, it would need a way to locate where on the screen you are looking — a technology called gaze tracking which works fairly well at this point using a special box that sits underneath your screen and looks up at your face and eyes. Using gaze tracking and a microphone you could actually simulate much of what the mouse does using your voice. The gaze tracker would track your eye position on the computer screen and then when you speak or sing or vocalize, the interpetation is that you spoke to the screen at that location where you were looking.
For example, to surf the internet, you would turn on your computer (perhaps with you voice!) and look at the google application icon and say “open” and then you would look at the search box and say “search for plant nuseries”. Upon seeing the results you might scan with the results of the search with your eyes and fixate on the result a third from the bottom, saying “open”. So your gaze serves to locate where you are speaking and also helps disambiguate what you want to do. But to me, the speech recognition part is not the interesting part, and it’s still not really using voice to make pictures… not really. To me it’s using voice to do some poor imitation of writing.
A more substantial departure from manual input is to use your voice in all of its wonderful tonal complexity (it’s timbre and pitch and volume) to literally make pictures. For example, you could paint with your voice by looking at a particular part of the screen and singing and gradually moving your eyes across the screen, modulating your voice simultaneously. The timbre of your voice would modulate the brush tip, the pitch could modulate the paint color, and the volume could modulate the brush size.
You see, one amazing thing about your voice is that it can transmit information as sound waves beyond your body. This mirrors in some sense the amazing thing about your eyes, namely that they can receive input from beyond your body in the form of light waves. Your hands are not as good at transmitting or receiving information past your body, although you can learn to make and play instruments which accomplish the same task that in some sense your voice does “out of the box”. So there is this very natural affinity between our ability to make sounds with our voices and transmit them as waves and our desire to see information as pictures. The translation between these can be done with the computer.
More about this later. I hope to do some real experiments and report back here.
I Just got back to NYC after a fantastic week of visiting my family in Palm Springs. My niece, Penelope, turned one and nearly her whole living gene pool on her mother’s side flew out from the east coast. I also got to spend some time with my other awesome niece, Catalina, and my nephew, Desmond, and I learned, that Catalina is still very much in love with math.
On the plane ride out there I started to work on a game to help me practice counting. I know, I know, you’ll say, I should know how to do that, and I do, in most situations I encounter. But I know I can get a lot better, and one area I can definitely improve on is “subitization” which is a form of counting which is instantaneous and almost subliminal. To understand the difference between merely counting and subitizing, consider, the following scenario. You are playing with your niece on the floor, and she picks up a pair of wooden blocks from her basket and drops them in front of you. You smile at her and the two blocks. How cute! They are have carved faces of animals on them. Now in her exuberance to show you her other prodigious gifts, she starts to get up off the ground and puts her weight on the basket, flipping its contents over onto the ground. Now there are lots of blocks. How cute! But how many are there? If you’re like me, or my eldest niece, you would have to count the blocks to know how many there are, it’s not as obvious as when there were just two. Most humans can subitize small quantities, like two, three, or even four without difficulty, but for higher numbers like six, seven, or even eight we are generally quite bad at it and we have to count. When you count you move your eyes over all the objects at least once and keep a running total. When you subitize you “count” in a single glance because the arrangement of objects is as obviously that number of things.
The interesting thing is some people can do much better than almost everyone else. Mathematical savants for example, often subitize quantities of 10 or even 14 with ease. I recently finished a wonderful book, an autobiography actually, called “Born on a Blue Day” by Daniel Tammet, in which Mr. Tammet describes his prodigious abilities to count. He writes that he would spend hours as a child making patterns out of blocks in his room and merely counting how many of this or that shape were present. He now can multiply large numbers together in his head instantly and holds the world’s record (or at least did) for most digits of Pi recited — more than 25,000. Are his mathematical abilities due to his spending so much time counting as a child? Nobody knows, but Mr. Tammet definitely sees a connection.
While I don’t think I will ever have Mr. Tammet’s love for numbers, I am fascinated by them and I do want to count them better. There is some research that says you can improve your subitization, so I thought it would be interesting just to make a sort of practice game for myself that gets harder as I get better. I tried hard to keep it barebones with a simple interface and am very pleased with the results. Try it out and let me know what you think.
After playing with my game for an hour or so, I really did start to see some improvement in my abilities, and I stopped having to count as much, and started just seeing the “sevenness” in the arrangement of dots on the screen. I was subitizing — albeit with occasional errors. I think it might actually be much more important for people learning math to spend a lot of time just seeing collections of things and learning to recognize their number by sight — subitizing! — so that when they think of 7 and 5 they see a collection of seven things and a collection of 5 things combining, not the numeral “7” and the numeral “5” with a cross-hair between them which isn’t nearly as expressive.
This is another experiment with my gear system. This time you can make your own setup. It’s still very much a work in progress, but I’m learning how to do collision detection which has been fun. The really hard part has been getting it fast enough to run in a browser and to get the physics to look believable. Funny stuff still happens when you put one gear on top of another one, but they move in beautiful ways already.
I had a wonderful opportunity this afternoon to participate in another “education salon” with my friends Sarah, Spencer, Ashish, and Will (in Absentia). Every couple of weeks we get together to discuss our shared passion of improving education. We have been focusing on sharing compelling research, noteworthy schools, and our own notions of what school should be about.
This week we discussed the question of “What everyone should know by the time they graduate high school”. I took a stab at detailing a list of skills which I think everyone should have by the time they graduate. The list deliberately excludes all the interesting content like History, Art, Science, Math and Literature and just focuses on the underlying capabilities I think everyone should have.
Students will be adept at manipulating themselves
- Identify successful habits which increase your productivity
- Learn effective strategies to memorize information
- Become competent at following your own rules
- Refine your intuition through analysis of personal failure
- Develop consistent motivation by setting achievable goals
- Read and write confidently in essay and story form
- Film and record audio to make compelling arguments
- Use Acting/Drama to understand human motivation
- Use drawing and programming to “speak in pictures”
- Know when you have made a strong argument
- Detect and identify your assumptions
- Use logic to reason about the unknown
- Recognize logical fallacies easily
- Wield words exactly when necessary
- Learn how your ego affects your own actions towards others
- Understand group dynamics and how it affects emotions and reason
- Learn successful methods of collaboration
- Learn to be persuasive one-on-one and in a group
- Learn effective strategies to manage other people’s time effectively
- Develop an appreciation and tolerance of other people’s ideas
Healthy Living Skills
- Have sufficient knowledge of physiology, nutrition, and chemistry to make informed choices about diet and exercise
- Understand how to cook a variety of foods and meals
- Develop a practice of regular exercise
One immediate problem with this list is that it does not specify at all how to teach the skills. This was intentional insofar as I am still developing my ideas on that. What do you all think?
I once had a job for the LEGO company to design a curriculum to teach elementary school mathematics using LEGO. One of the units I came up with used gears of different sizes to introduce the concept of ratios. As I am back in the job of trying to come up with interesting math games for my PhD research, I started talking with my advisor about gears and got interested in them again. What mathematical concepts come out naturally from gears?
Certainly ratios are an intrinsic part of gears. We take advantage of gear ratios when we ride our bikes, by shifting to a lower gear to give ourselves more turns of our crankshaft for every turn of the rear wheel.
But I think what is really interesting about gears from a mathematical perspective is a little different from what is physically useful about them. And in my humble opinion what is really cool about them is that in order to make gears of different sizes mesh together you need to make sure that the teeth are the same size. And this means that smaller gears will have correspondingly fewer teeth and bigger gears will have more teeth. And since bigger gears have more teeth, they make a revolution more slowly than a smaller gear and vice versa.
Now you could ask, and I’m not saying you would, but you could ask, how long does it take two gears to meet back up where they started? It definitely does happen, right, because if it never happened then every tooth on each gear would always be meshing with a different tooth on the other gear, but that would mean that there are infinitely many teeth on the other gear, so that can’t be right! So it definitely does happen that some pair of teeth on each gear meet up again and again, but when is the first time? In the spirit of “show don’t tell” I made a little demo of it using several gears of different sizes. Take a look and count how many teeth mesh before the two little red circles match up! The green gear has 10 teeth and the smaller gear next to it has 5 teeth which is exactly half as many.
Three weeks ago, I went to see a free screening of the incredible film “Home” and was treated to a Q&A with the director Yann Arthus-Bertrand afterwards. Many people in the audience were moved by the film. They praised it for its extraordinary beauty, and also its message: Humans are not living sustainably on this earth. At least a few people asked Yann Arthus-Bertrand what they should or could do, to which he very thoughtfully replied, “I don’t know. I’m a filmmaker, so I made this film. Now it’s up to all of us to figure it out.” And I really appreciated his position that doing art can raise consciousness. So, as my little homage to “Home”, I made a kit called “sustainable poetry” which invites people to experiment with different poems about sustainability. Click the screenshot below to try it out (or click here.)
This experiment is collaborative, so changes you make will be stored on the server and shown in realtime to other users, or saved for the next person who visits. A couple of points: I’m not saving any history of changes at the moment, but I probably should. Also, let me know what words I’m missing, and I’ll add them. And a final disclaimer, this is a work in progress, so suggestions are greatly appreciated!
If you did not see IBM’s new “robot” beat the two best human players in a televised Jeopardy (Ken Jennings and Brad Rutter) two weeks ago, I highly recommend watching it
Whole watching Watson win I was reminded of the other super know-it-all robot called Google, but whereas Google acts a bit like a trained dog who responds to “fetch”, Watson feels so much more human because he fetches and then speaks aloud the best response. So I got to thinking about how well Watson would work as a tutor for humans. There are all sorts of interesting challenges here. Among these are:
First, domain novices usually do not possess domain-specific vocabulary which experts use. So Watson would have to interpret the novice questions and form a best guess response. This is straightforward if the question has a known, factual answer, but could easily be more difficult if the student presented a “what-if” scenario.
Second, tutors often have to do more than merely verbalize a response to a question. For example, they may have to draw pictures, tables, or diagrams. This is complicated because Watson not only has to figure out the question’s intent but may actually have to recall parts of relevant pictures it has mined from the web and then use pieces to make a new drawing.
Third, good human tutors form rich models of the tutee’s knowledge and understanding and constantly ask the tutee questions to update the model. If you’ve ever been lucky enough to be a tutor or get tutored, you know this this is crucial because it allows the tutor to interactively eliminate misapprehensions by constructing relevant examples. This is a very creative process, perhaps it should rightfully be called an art, and to get good at it, Watson would require lots of data from past tutoring sessions with other students. But if enough children use Watson, this might be possible.
All these challenges seem hard to me, and I am reminded of a quotation attributed to Jon Von Neumann who invented the basic architecture of modern computers over 70 years ago. He said, “You insist that there is something that a machine can’t do. If you will tell me precisely what it is that a machine cannot do, then I can always make a machine which will do just that.”
This is the first in a series of short posts about potential topics for research. Thanks to my advisor for suggesting this.
Facebook game(s) for learning math.
I am thinking of a visually interesting math practice game, with lots of activities that illuminate everything from basic arithmetic to geometry to algebra, but crucially, also add a social component. Anecdotally, from research I did last year to develop an arithmetic practice game, kids really like to compete with each other even about math and they can also learn from each other while playing (did I really need to do research to figure that out?) But Facebook offers lots of ways to make even solo games more social. Two of these are, high score lists amongst friends, and posting achievements to a user’s wall (for those wondering, Achievements are in-game awards or messages such as “William Ion put another 65 minutes into mastering fractions today.”) Some argue that achievements may be harmful to games, but isn’t the whole idea of facebook a big game of achievements? And if you’re a kid like William “Billion” Ion who is struggling with say, adding fractions, perhaps an uncle, or your grandmother will offer to play with you in a two-player mode. Parallel to all these features on the front end, we can also collect data about the users play to make improvements. For example, given two possible modules to teach fractions, which is bringing more users back for repeat plays? Which is leading to more improvement in actual learning? Can we recommend games to people based on their similarity to other players who have followed similar learning trajectories?
So I think Facebook is a really interesting place to start looking at social learning games. Are there any out there already?
I am so excited to finally have my own blog and to have found such a lovely theme. The idea for the name of the blog, “Subject to Change”, stems from my short attention span and the need to explore.
Why a blog? Two reasons I can think of.
First, I like to write and I need to write to make my oral conversation more concise and eloquent. Why waste words in oral conversation merely because I am struggling to find the right words? Better to hone my skills in recalling the right words in a written form where I am only wasting my own time to find “le mot juste” so that I sound more direct and laconic in person and don’t waste your time.
Second, I am a doctoral candidate in Computer Science at NYU and as such, am being trained as a scientist and a researcher, and scientists need to write to share their ideas with peers who can change their minds. Though this is often done in a peer-reviewed journal, many of the problems that I am interested in are more vernacular and immediate and I can’t think of a freer pulpit than a blog. But that is subject to change.