Reading language

“The parrot, which was continually watched by the big black cat, who sometimes sat on the mat until displaced by the bigger brown Doberman and sometimes on the garden wall in the hope of catching the sparrow, flew through the window.” (sentence 1)

Why is this difficult to read?  It is grammatically well-formed and has commas in all the right places.  However, the large subordinate clause (if I have got my grammatical terms right) means that by the time you have go to “flew through the window” you have ‘forgotten’ the parrot who it was all about.

In contrast, a computer parser has no such trouble. Take the fragment:

if ( y > 10 ) {
    x1 = y+1;
    x2 = y+2;
    . . .   // 97 more lines of code
    x100 = y+100
} else { . . .

The compiler does not ‘forget’ about the ‘if’, because it is part of a relatively simple syntactic structure:

if ( condition ) { statement_block } else { statement_block }

In the English sentence about the parrot, like the program fragment, even the ‘depth’ of the parsing stack is not deep, yet still there is confusion.
Years ago when I was a student in Cambridge a friend who was studying Politics came to me with a passage in Nozick’s “Anarchy State and Utopia“.  I read the paragraph in question several times.  Each time it made sense, until I came to the last few words, which seemed to be a total non sequitur.  Only after many readings did I realise that the paragraph was a single sentence (all 10 lines of it!), and that the sentence was of precisely the same form as the parrot one, except with nested sub-clauses.   The words that appeared to come from nowhere were in fact the equivalent of “flew through the window” … indeed imagine if the description of the Doberman had extended to another seven lines!

Clearly our ability to hold on to relatively simple hierarchical parsing structures is quite limited, and is definitely smaller than the 7+/- 2 for working memory¹.  However, there is also another mechanism at work.

Consider the following alternative sentences:

“The policeman, which was continually watched by the big black cat, who sometimes sat on the mat until displaced by the bigger brown Doberman and sometimes on the garden wall in the hope of catching the sparrow, flew through the window.” (sentence 2)

“The parrot, which was continually watched by the big black cat, who sometimes sat on the mat until displaced by the bigger brown Doberman and sometimes on the garden wall in the hope of catching a snail, flew through the window.” (sentence 3)

It is hard to read these ‘fresh’ as if you hadn’t read the initial sentences, but notice that the first is far harder than the second.  This is because we have an additional sequential associative mechanism at work as well as syntactic parsing.

Consider instead the much simpler  ’sentence’:

“bone dog eat”

We have little problem making sense of this even though the structure is not grammatical.  The verb ‘eat’ is ‘looking for’ something that eats to act as its subject, and something that can be eaten to be its object.  As there is only one eatable and one eater in the ’sentence’, the parsing (or perhaps connecting) happens purely by association.

If there are several ‘matches’ for the words then the word combinations become ambiguous:

“policeman demonstrator hit”

Some languages, such as English, largely deal with this ambiguity by use of syntactic rules (subject verb object), others, such as Latin, by tagging the words with their role … and I don’t know my Latin enough to give a proper example, but I think something like, in pigeon Latin: “policemanus demonstratori hit” vs “policemani demonstratorus hit”!  I believe Latin has quite strong syntactic rules as well, but there are certainly languages where order is less significant, you can think of them as Lego block languages; you throw the words into a bag, shake them up, and they stick together where the words fit together.

When we are reading a sentence both forms of parsing are happening at once.  There is a level of grammatical parsing, but also when a word, like ‘parrot’ in sentence 1, is encountered and does not get ‘bound’ it is ‘waiting’ for a verb to connect to.  Later in the sentence,  when we encounter ‘flew’ it wants something that can fly to be its subject, if ‘parrot’ is still sufficiently in mind then all is well and we get “The parrot flew through the window”.  However, the last thing we read was ’sparrow’ a thing that can fly.  The sequential associative parsing says “sparrow flew” and even if  the syntactic parsing ‘wins’, the conflict still makes the sentence hard to grasp.

Sentence 2 was chosen to make the binding of the grammatical subject ‘policeman’ unusual to bind to ‘flew’ and so more confusing, whereas sentence 3 is less confusing as there is no obvious flying candidate except the parrot.

Poets make good use of these multiple mechanisms as it allows them to bend grammar and yet have (sometimes) comprehensible sentences.  For my students it is not usually poetry I am after in their PhD theses, but understanding the role of sequential parsing can help us to construct more comprehensible English.

Reading action

It was only quite recently I realised there was a very close parallel to work we had been doing in the TIM project on tasks² – that is action rather than language.

There are various algorithms used in intelligent user interface research to help predict the next action based on the previous one.  Mostly these make use of some sort of Markov model or similar sliding window where the next command is the one that has occurred most often after the preceding N commands.  However, there have been various proposals for performing more hierarchical pattern formation effectively building task structures as found in hierarchical task analysis.

A fundamental block to both approaches is that human activity is not always focused on one task at a time, but we do a little of one, then a little of another, just as I might break off from writing to make a cup of tea.

In the TIM project we have been using a personal ontology for various purposes including helping to propose values for actions such as web forms.  If there is a ‘name’ field, then the ontology includes names of friends so these can be proposed as potential options to the user.

Because of the ontology, we are also able to detect that the telephone number used in one web form is the telephone number of the person in the mail that has just been received.  This semantic binding through the ontology means that we can effectively see that individual actions are or are not related depending on whether they have semantic relationships through the ontology, that is we can thread together the actions that belong a single task sequence and pull them out of the chronological sequence that involves a mix of task and activities.

There are many differences between this and the sequential associative parsing, not least that the basic actions we are considering are typically parameterised (the completion of a web form) while words are singular, but there are also connections.  Previously, the relationship between task analysis and traces of actions has been compared with that between grammar and sentences, basically treating task analysis as a way to ‘parse’ activity³. This parallels the conventional parsing approach.  However, what I only belatedly realised was the use of semantic relationship between actions we are adopting in the TIM project also parallels the use of semantic connections between words in our task inference.

Reading the world

The above is about the way a computer can make sense of human action.   However, as humans we need to make sense of the things that happen around us in the world not least other people.

When we sense the world we are confronted with myriad stimuli, as I write the sound of an ambulance passing outside, the insistent hiss of the kettle on the stove, all clamour against the words I type.  Against this visual, aural and tactile cacophony, we have to unravel the threads of meaning.  If my leg aches I do not ascribe it to the cup of tea I am drinking, or the ambulance that recently passed, but instead to the run round the Circo Massimo earlier this morning.  However, equally I do not ascribe it to running on the beach a weak ago.  Temporal and semantic proximity both play their part in our reading of the signs of day-to-day life.  It is not surprising that these same tools are at play as we listen or read language.

We also perceive patterns of actions in the world, most significantly in the actions of other creatures.  In a keynote at Tamodia in Pisa last year, I discussed the production of human action and the rich intertwining of sequential planned or proceduralised activity with more stimulus-driven reactions.  Both could be explicit or tacit.  The computer aid is trying to ‘read’ these patterns of intentional (and reactive) activity.

It is of course likely that our ability to produce and understand language builds on our ability to act within and to comprehend the world.  However, the key difference is that language is about intentional communication.  As well as expecting parts to link together in fragments, we also expect the whole to make sense.  In a detective novel each incident may seem disparate, but we are sure that in the end they will come together in the dénouement.  This is a large-scale application of Grice’s cooperation principle.  For us interpreting the world and each other’s actions, or for the computer interpreting human action, no such overarching message is expected with the exception of life as a whole: God, or death, depending on your beliefs, as the ultimate dénouement.

1. George A. Miller. The Magical Number Seven, Plus or Minus Two: Some Limits on Our Capacity for Processing Information. The Psychological Review, 1956, vol. 63, pp. 81-97 [back]
2. The best account of this task inference at present is probably my keynote Tasks = data + action + context: automated task assistance through data-oriented analysis at Engineering Interactive Systems 2008 last September in Pisa. [back]
3. Personally, I first used this ‘task analysis as grammar’ approach in teaching task analysis and it appeared in the task model slides for the third edition if the HCI book (but not in the book itself, maybe fourth edition).  However, it has been elaborated in work with Stavros Asimakopoulos and Robert Fyldes “Grammatically interpreted task analysis for supply chain forecasting“. [back]

Often the most innovative programming/scripting environments, Smalltalk, Hypercard, Mathematica, humble spreadsheets, even (for those with very long memories) Filetab, have broken these assumptions, as have whole classes of ‘non-standard’ declarative languages.  More recently Yahoo! Pipes and Scratch have re-introduced more graphical and lego-block style programming to end-users (albeit in the case of Pipes slightly techie ones).

Yahoo! Pipes (from Wikipedia article)	Scratch programming using blocks

What would programming be like if it were more incremental, more focused on live data, less focused on the language and more on the development environment?

Two things have particularly brought this to mind.

First was the bootcamp team I organised at the Winter School on Interactive Technologies in Bangalore¹.  At the bootcamp we were considering “content development through the keyhole”, inspired by a working group at the Mobile Design Dialog conference last April in Cambridge.  The core issue was how one could enable near-end-use development in emerging markets where the dominant, or only, available computation is the mobile phone.  The bootcamp designs focused on more media content development, but one the things we briefly discussed was full code development on a mobile screen (not so impossible, after all home computers used to be 40×25 chars!), and where literate programming might offer some solutions, not for its original aim of producing code readable by others², but instead to allow very succinct code that is readable by the author.

if ( << input invalid >> )
    << error handling code >>
else
    << update data >>

(example of simple literate programming)

The second is that I was doing a series of spreadsheets to produce some Fitts’ Law related modelling.  I could have written the code in Java and run it to produce outputs, but the spreadsheets were more immediate, allowed me to get the answers I needed when I needed them, and didn’t separate the code from the outputs (there were few inputs just a number of variable parameters).  However, complex spreadsheets get unmanageable quickly, notably because the only way to abstract is to drop into the level of complex spreadsheet formulae (not the most readable code!) or VB scripting.  But when I have made spreadsheets that embody calculations, why can’t I ‘abstract’ them rather than writing fresh code?

I have entitled this blog ‘part 1′ as there is more to discuss  than I can manage in one entry!  However, I will return, and focus on each of the above in turn, but in particular questioning some of those assumptions embodied in current programming languages:

(a) code comes before data

(b) you need all the code in place before you can run it

(c) abstraction is about black boxes

(d) the programming language and environment are separate

In my PPIG keynote last September I noted how programming as an activity has changed, become more dynamic, more incremental, but probably also less disciplined.  Through discussions with friends, I am also aware of some of the architectural and efficiency problems of web programming due to the opacity of code, and long standing worries about the dominace of limited models of objects³

So what would programming be like if it supported these practices, but in ways that used the power of the computer itself to help address some of the problems that arise when these practices address issues of substantial complexity?

And can we allow end-users to more easily move from move seemlessly from filling in a spreadsheet, to more complex scripting?

1. The winter school was part of the UK-India Network on Interactive Technologies for the End-User.  See also my blog “From Anzere in the Alps to the Taj Bangelore in two weeks“ [back]
2. such as Knuth’s “TeX: the program” book consisting of the full source code for TeX presented using Knuth’s original literate programming system WEB. [back]
3. I have often referred to object-oriented programming as ‘western individualism embodied in code’. [back]

The congress itself was in Portuguese except my talk, and I  was only able to spend one day there as I needed to get back to pack, and so, by the time I met press and talked to different people, the day flew by  … in time for an evening of typical Portuguese culture of different kinds.

First dinner of roast suckling pig – prepared in the town near Aveiro where this is the traditional dish.  Those who know me know that despite all appearances to the contrary: sandals, long hair, beard; I am NOT a vegetarian   The meat itself reminded me of the rich flavour of belly pork at Sunday dinners when I was a small child; although much more delicate and without the tooth breaking thickness of the older meat.

After traditional cuisine I was given a taste of traditional student life.  This period was a weekend when first year freshmen students all over Portugal have parties … for several nights in a row.  The student organisers I was with had been up to 6am the previous night as well as organising the conference during the day.  And this night, after having eaten suckling pig at 10pm, drove me down to a location in the dockland of Aveiro, far form any homes that would be disturbed by the noise, to a tiny village of food stalls, a huge music tent … and bars run by every student club in the university.

Some of the students wore traditional academic dress of Portugal – a thick black felt cloak hanging nearly to the floor and hats – different for each University.  A group of visiting students from of Braga wore three cornered hats and looked every bit like a troupe of Dick Turpins.  The cloaks are torn around the bottom, where family and friends would tear a gash in the edge … social networking before Facebook.  The middle of the back of the cloak is reserved for the girlfriend or boyfriend to tear … but if a relationship ended you had to sew up the tear ready for the next one!

I talked with a group of students from Évora who explained their tradition of peer tutelage (I have forgotten the name of the practice).  Two older students take a new student under their wing and teach him or her the practices of the University.  The young student did not have his cloak yet as only after six weeks did he become a true freshman and entitled to the cloak.  In the mean time they would carry him home of the parties proved too much and also put him right if he did things wrong … I missed the details of this, but I’m sure this included eggs (!?).  The young student was enjoying the process and the older pair clearly took their responsibilities very seriously.  At the end of the evening they asked me to tear their cloaks, involving using my teeth in order to start the tear.  I was very honoured to be asked.  The idea of biting cloth that had been scraping ground is not something I would normally relish (!), but given the evening consisted largely of groups of students, who had been at the afternoon lecture, pressing shots of various drinks upon me, by 4am, a little dirt didn’t seem to matter so much

And this Friday, no shots of liqueur in chocolate cups, or suckling pig and Portuguese wine, but instead packing a Luton van ready to move up to Tiree for my sabbatical.  Kiel was an absolute star, coming first thing in the morning, lugging filing cabinets and freezers … and boxes some way beyond the current 25kg one-man lift limit.  I recall routinely carrying hundredweight sacks wen I was younger and Kiel spent his youth lugging rolls of cloth around a textile factory, so for both of us 25kg seemed a little wimpish … however, there is a vast difference between a 20kg box and a 30 kg one … so the health and safety people probably have it right … and we did try to keep them all below 25kg, but very hard with boxes of books … and there are many of them a ton weight of books in fact, not to mention another half ton of bookshelving.

So this morning finds me half way up the M6, tomorrow morning we’ll be on the ferry, and the next morning in  Tiree, ready for a year of hermit-like writing and working … not to mention the odd walk on the near empty two-mile beach outside the door … but that will be another story.