PPD10

The PPD10 workshop on Coupled Display Visual Interfaces¹ followed on from a previous event, PPD08 at AVI 2008 and also a workshop on “Designing And Evaluating Mobile Phone-Based Interaction With Public Displays” at CHI2008.  The linking of public and private displays is something I’ve been interested in for some years and it was exciting to see some of the kinds of scenarios discussed at Lancaster as potential futures some years ago now being implemented over a range of technologies.  Many of the key issues and problems proposed then are still to be resolved and new ones arising, but certainly it seems the technology is ‘coming of age’.  As well as much work filling in the space of interactions, there were also papers that pushed some of the existing dimensions/classifications, in particular, Rasmus Gude’s paper on “Digital Hospitality” stretched the public/private dimension by considering the appropriation of technology in the home by house guests.  The full proceedings are available at the PPD10 website.

AVI 2010

AVI is always a joy, and AVI 2010 no exception; a biennial, single-track conference with high-quality papers (20% accept rate this year), and always in lovely places in Italy with good food and good company!  I first went to AVI in 1996 when it was in Gubbio to give a keynote “Closing the Loop: modelling action, perception and information“, and have gone every time since — I always say that Stefano Levialdi is a bit like a drug pusher, the first experience for free and ever after you are hooked! The high spot this year was undoubtedly Hitomi Tsujita‘s “Complete fashion coordinator”², a system for using social networking to help choose clothes to wear — partly just fun with a wonderful video, but also a very thoughtful mix of physical and digital technology.

images from Complete Fashion Coordinator

The keynotes were all great, Daniel Keim gave a really lucid state of the art in Visual Analytics (more later) and Patrick Lynch a fresh view of visual understanding based on many years experience and highlighting particularly on some of the more immediate ‘gut’ reactions we have to interfaces.  Daniel Wigdor gave an almost blow-by-blow account of work at Microsoft on developing interaction methods for next-generation touch-based user interfaces.  His paper is a great methodological exemplar for researchers combining very practical considerations, more principled design space analysis and targeted experimentation.

Looking more at the detail of Daniel’s work at Microsoft, it is interesting that he has a harder job than Apple’s interaction developers.  While Apple can design the hardware and interaction together, MS as system providers need to deal with very diverse hardware, leading to a ‘least common denominator’ approach at the level of quite basic touch interactions.  For walk-up-and use systems such as Microsoft Surface in bar tables, this means that users have a consistent experience across devices.  However, I did wonder whether this approach which is basically the presentation/lexical level of Seeheim was best, or whether it would be better to settle at some higher-level primitives more at the Seeheim dialog level, thinking particularly of the way the iPhone turns pull down menus form web pages into spinning selectors.  For devices that people own it maybe that these more device specific variants of common logical interactions allow a richer user experience.

The complete AVI 2010 proceedings (in colour or B&W) can be found at the conference website.

The very last session of AVI was a panel I chaired on “Visual Analytics: people at the heart of data” with Daniel Keim, Margit Pohl, Bob Spence and Enrico Bertini (in the order they sat at the table!).  The panel was prompted largely because the EU VisMaster Coordinated Action is producing a roadmap document looking at future challenges for visual analytics research in Europe and elsewhere.  I had been worried that it could be a bit dead at 5pm on the last day of the conference, but it was a lively discussion … and Bob served well as the enthusiastic but also slightly sceptical outsider to VisMaster!

As I write this, there is still time (just, literally weeks!) for final input into the VisMaster roadmap and if you would like a draft I’ll be happy to send you a PDF and even happier if you give some feedback

Search Computing

I was invited to go to this one-day workshop and had the joy to travel up on the train from Rome with Stu Card and his daughter Gwyneth.

The search computing workshop was organised by the SeCo project. This is a large single-site project (around 25 people for 5 years) funded as one of the EU’s ‘IDEAS Advanced Grants’ supporting ‘investigation-driven frontier research’.  Really good to see the EU funding work at the bleeding edge as so many national and European projects end up being ‘safe’.

The term search computing was entirely new to me, although instantly brought several concepts to mind.  In fact the principle focus of SeCo is the bringing together of information in deep web resources including combining result rankings; in database terms a form of distributed join over heterogeneous data sources.

The work had many personal connections including work on concept classification using ODP data dating back to aQtive days as well as onCue itself and Snip!t.  It also has similarities with linked data in the semantic web word, however with crucial differences.  SeCo’s service approach uses meta-descriptions of the services to add semantics, whereas linked data in principle includes a degree of semantics in the RDF data.  Also the ‘join’ on services is on values and so uses a degree of run-time identity matching (Stu Card’s example was how to know that LA=’Los Angeles’), whereas linked data relies on URIs so (again in principle) matching has already been done during data preparation.  My feeling is that the linking of the two paradigms would be very powerful, and even for certain kinds of raw data, such as tables, external semantics seems sensible.

One of the real opportunities for both is to harness user interaction with data as an extra source of semantics.  For example, for the identity matching issue, if a user is linking two data sources and notices that ‘LA’ and ‘Los Angeles’ are not identified, this can be added as part of the interaction to serve the user’s own purposes at that time, but by so doing adding a special case that can be used for the benefit of future users.

While SeCo is predominantly focused on the search federation, the broader issue of using search as part of algorithmics is also fascinating.  Traditional algorithmics assumes that knowledge is basically in code or rules and is applied to data.  In contrast we are seeing the rise of web algorithmics where knowledge is garnered from vast volumes of data.  For example, Gianluca Demartini at the workshop mentioned that his group had used the Google suggest API to extend keywords and I’ve seen the same trick used previously³.  To some extent this is like classic techniques of information retrieval, but whereas IR is principally focused on a closed document set, here the document set is being used to establish knowledge that can be used elsewhere.  In work I’ve been involved with, both the concept classification and folksonomy mining with Alessio apply this same broad principle.

The slides from the workshop are appearing (but not all there yet!) at the workshop web page on the SeCo site.

1. yes I know this doesn’t give ‘PPD’ this stands for “public and private displays” [back]
2. Hitomi Tsujita, Koji Tsukada, Keisuke Kambara, Itiro Siio, Complete Fashion Coordinator: A support system for capturing and selecting daily clothes with social network, Proceedings of the Working Conference on Advanced Visual Interfaces (AVI2010), pp.127–132. [back]
3. The Yahoo! Related Suggestions API offers a similar service. [back]

The USP for Dabble DB amongst other online data sharing apps, is that it appears to really be a complete database solution online … and its USB amongst conventional databses is the way they seem to have really thought about real use.  This focus on real use by ordinary users includes dynamically altering the structure of the data as you gradually understand it more.  The model they have is that you start with plain table data from a spreadsheet or other document and gradually add structure as opposed to the “first analyse and then enter” model of traditional DBs.

As I read Enrico’s blog I remembered that he had mailed me about the ‘magic/replace‘ feature ages ago.  This lets you tidy up  data during import (but apparently not data already imported … wonder why?), using a ‘by example’ approach and is a really nice example of all that ‘programming by example‘ and related work that was so hot 15 years ago eventually finding its way into real products.

The downside to Dabble DB is that editing is via forms only … it is often so much easier to enter data in a spreadsheet view, the API is quite limited, and while they have a ‘Dabble DB Commons‘ for public data (rather like Swivel), there is no directory or other way to see what people have put up

I was particularly hoping the API was better as it would have been nice to link it into my web version of Query-by-Browsing. or even integrate with the Query-through-Drilldown approach for constructing complex table joins that Damon Oram implemented more recently.

In general, while the DB and (many) UI features are strong it is not really looking outwards to creating shared linked data (in the broadest sense of the term, not just pure SemWeb world linked data), … so still room there for the absolute killer shared data app!

One of the critical things about the design was that while earlier designs used steel, his design needed to be made out of plastic.  Steel is an obvious material for a lock: strong unyielding; however the plastic lock worked because the lock and the bin around it were designed to yield, to give a little, and is so doing to absorb the shock if kicked by a drunken passer-by.

This is a sort of Judo principle of design: rather than trying to be the strongest or toughest, instead by  yielding in the right way using the strength of your opponent.

This reminded me of trees that bend in the wind and stand the toughest storms (the wind howling down the chimney maybe helps the image), whereas those that are stiffer may break.  Also old wooden pit-props that would moan and screech when they grew weak and gave slightly under the strain of rock; whereas the stronger steel replacements would stand firm and unbending until the day they catastrophically broke.

Years ago I also read about a programme to strengthen bridges as lorries got heavier.  The old arch bridges had an infill of loose rubble, so the engineers simply replaced this with concrete.  In a short time the bridges began to fall down.  When analysed more deeply  the reason become clear.  When an area of the loose infill looses strength, it gives a little, so the strain on it is relieved and the areas around take the strain instead.  However, the concrete is unyielding and instead the weakest point takes more and more strain until eventually cracks form and the bridge collapses.  Twisted ropes work on the same principle.  Although now an old book, “The New Science of Strong Materials” opened my eyes to the wonderful way many natural materials, such as bone, make use of the relative strengths, and weaknesses, of their constituents, and how this is emulated in many composite materials such as glass fibre or carbon fibre.

In contrast both software and bureaucratic procedures are more like chains – if any link breaks the whole thing fails.

Steve’s lock design shows that it is possible to use the principle of strength in weakness when using modern materials, not only in organic elements like wood, or traditional bridge design.  For software also, one of the things I often try to teach is to design for failure – to make sure things work when they go wrong.  In particular, for intelligent user interfaces the idea of appropriate intelligence – making sure that when intelligent algorithms get things wrong, the user experience does not suffer.  It is easy to want to design the cleverest algotithms, the most complex systems – to design for everything, to make it all perfect. While it is of course right to seek the best, often it is the knowledge that what we produce will not be ‘perfect’ that in fact enables us to make it better.

Sony have robot dogs, Phillips robot cats (albeit stuck sitting in one place) but Ugobe have little robot dinosaurs called Pleo. In the videos they do move like little baby creatures and the lady in the shopping mall coos over one as she strokes it.

Central to Pleo seems to be:

1. embodiment – they feel through 40 sensors and move in their environment
2. emotion – they have a relatively complex model of basic drives rather like Cynthia Breazwal describes in her book “Designing Sociable Robots“.

This seems to pay off in people’s reactions, both on Pleo’s own videos (well they would!), but also in owner’s plogs (sic) … one owner says:

“she acts just like a cat concerning keyboards.. just crawl on the darn thing while I’m typing! I know Penny,. you’re so cute it doesn’t matter what you do. But you should have a little sensor strip in your butt to spank when you’re bad¹ or to pat gently to urge you to go explore. Go to sleep my little love” ArcticLotus

Pleos making friends :-/

For researchers there is an open architecture so it should be possible to play oops experiment with them The API doesn’t seem to be published yet, so wait until you get your cheque books out!

1. This could get us into the territory of agent abuse! [back]

In this case the issue is the distributed nature of cognition within the brain and the inadequacy of central executive models. In support of this, Clark (p.39) cites Mitchel Resnick at length and I’ll reproduce the quote:

“people tend to look for the cause, the reason, the driving force, the deciding factor. When people observe patterns and structures in the world (for example, the flocking patterns of birds or foraging patterns of ants), they often assume centralized causes where none exist. And when people try to create patterns or structure in the world (for example, new organizations or new machines), they often impose centralized control where none is needed.” (Resnick 1994, p.124)¹

The take home message is that we tend to think in terms of centralised causes, but the world is not like that. Therefore:

(i) the way we normally think is wrong

(ii) in particular we should expect non-centralised understanding of cognition

However, if our normal ways of thinking are so bad, why is it that we have survived as a species so long? The very fact that we have this tendency to think and design in terms of centralised causes, even when it is a poor model of the world, suggests some advantage to this way of thinking.

Of course, this may simply be an accident of our neural architecture … in which case it would be important for (ii), or may be adapted for a hunter gatherer life, but not 21st Century living – but again would be interesting for (ii). However, the fact that we are still here means it is certainly not too unsuccessful.

Whatever the reason, the fact that we think in these terms is itself an empirical data point for understanding human cognition. We have brains that tend to seek centralised solutions – what are the neural and cognitive mechanisms that drive this and what are the environmental reasons that make it work.²

There are two factors at work here, one is about the way we see the world and the other about the way we plan and act on it.

At the level of perception, one of the Gestalt laws is that things that move together belong together. Even if bushes hide most of a predator from view, the several disconnected tiny moving fragments still form one large animal you need to avoid.

Yes this is a gross simplification of reality. Look at a rock – it is an ‘it’, a single thing – but in fact it is not, it is simply the decentralised activities of millions of millions of millions of millions of atoms interacting, largely locally, with one another. It is not so far unlike Resnick’s flock of birds. However, their general coherence of motion and substance makes it sensible to regard it as one thing. As a scientist understanding the decentralised emergent phenomenon is interesting, but as a gardener wanting to move the rock it is an intellectual luxury and the (incorrect) centralised view of a the rock makes sense.

In real world problems, sometimes decentralised solutions work, other times they don’t.

I was once driving in Rome on a Saturday night (happily with a native Roman to guide me). It was after 11pm so they turned off all the traffic lights (as Italians ignore them anyway) and we came to a massive crossroads with completely full three lane roads in all directions. The space was filled with a criss-cross of apparently grid-locked cars and I thought we would be stuck there until a policeman came, but my navigator told me to simply drive. Every time the slightest gap opened, be it only a few inches, I would edge forward. Eventually, but after a relatively short time, we found ourselves at the other side. Thinking afterwards I realised that always some car was able to get out and I fact the ‘greedy’, decentralised algorithm worked perfectly.

Driving is different in north-west Scotland where there are long stretches of narrow single-track roads with passing places. When you spot a vehicle coming you watch out for a passing place and whichever of you gets to the one first waits there. If you notice too late and meet, then the person closest to a passing place may need to reverse. This is another local, slightly more polite, but semi-greedy algorithm, with each person making independent choices and trying to proceed, but taking into account immediately close road users. However, when single track roads get too full, this can fail. In situations, like the passing goods-train puzzles, where lines of vehicles have to pass with only single passing place, then often long lines of vehicles have to backup, go forward again, reverse again, in apparently disorganised ways – and ways in which each single driver cannot understand from local conditions alone. People have to get out their cars and start to coordinate their efforts.

Note that the decentralised strategies work remarkably well, and when they do require less effort than coordination. However, the reason that we do more than that, and think in ways that have an (at least behavioural) appearance of centralised control is because for certain problems this is needed – not least in complex social and technological situations.

With a HCI hat on, when we come to designing for people, we get the best solutions not when we ignore one aspect or another, but when we recognise the relative strengths of the two and how they can work together.

I recall when I was a child (yes over 30 years ago), seeing a television report about Benetton’s new CAD systems. The problem was cutting out rolls of cloth to make pieces for clothing. Traditionally an experienced cutter would arrange the pieces for a single garment as tightly as possible (to avoid waste), whilst ensuring proper orientations. These were then cut using a special form of guillotine. The new system of cutting from the roll allowed them to take the pieces for several garments and organise them over a long run of cloth for cutting. Doing several garments at once offered savings in terms of less wastage, but was a more challenging arrangement tasks … hence computer aid. The computer would take pieces initially arranged on (virtual) fabric and ‘jiggle’ them until they fitted closer with les waste. However, an experienced cutter would oversee this process and make large scale changes, “what if we tried this large piece over here?” The computer’s activity would have been serial, but could have been parallelised as it involved effectively lots of small local decisions. However, the human made strategic decisions, that themselves made use of the human’s internal associative pattern recognition, but from the point of view of the large system were effectively more centralised. Here a combination of centralised and decentralised thinking/computation together addressed a problem neither could solve on their own.

Finally, it is interesting to reflect on the ability demonstrated in both Clark and Resnick’s writing. They look in at our modes of thinking, see that they are often over-simplistic in terms of assuming central control when there is none, and then consider how to address this. This highly reasoned and reflective process does not arise naturally from decentralised thought that would simply go on using the same old ways of thinking, but is the product of exactly the more ‘rational’ linear, centralised thinking that they seek to expose as outmoded.

1. Mitchel Resnik (1994). Turtles Termites and Traffic Jams: Explorations in Massively Parallel Microworlds. MIT Press. [back]
2. This is similar to the argument in my previous post on the power of sequential thinking, where I pondered the complexity of establishing sequence within an underlying parallel and distributed neural superstructure … but also discussed the advantages it brings. Sequentiality and central control are of course closely linked. [back]

This reminded me of results of brain scans (sadly, I can’t recall the source), which showed that the areas in the brain where you store concepts like ‘chair’ are different from those where you store the sound of the word – and also I’m sure the spelling of it also.

This makes sense of the “tip of the tongue” phenomenon, you know that there is a word for something, but can’t find the exact word. Even more remarkable is that of you know words in different languages you can know this separately for each language.

So, musing on this, there seem to be very good reasons why, even within our own mind, we hold multiple representations for the “same” thing, such as chair, which are connected, but loosely coupled.

In an artificial “brain” like COG, the computational units are physically separate. In our brains things are much less discrete, but we know do have well defined locality for certain functions (e.g. Broca’s area for speech). Also, while there is debate about the extent to which we know what we are doing (or perhaps more important know what we are about to do) still it is clear that at least low-level functions operate semi-autonomously, but for instant reaction (pain withdrawal) and also for controlled actions (play a guitar chord).

In particular, there are particular brain lesions that mean that the patient cannot choose what to say, yet still vaguely grammatical but entirely meaningless utterances are constantly made – rather like James Joyce In between are cases where “nearly right” words come out, perhaps table, or cushion instead of chair. So for speech it seems a “higher level” part of the brain decides what we want to say and makes gentle suggestion for what this should be, but a different part does the final stringing together of words and it is this part that ‘knows’ the rules of grammar, the way words connect into each other and what the words sound like (although the grammar and aural elements may themselves be generated separately). This is rather like the intention to walk and the fine movement of muscles needed to move each leg.

Obviously the representations needed for saying ‘chair’ in a sentence are about the way it fits into grammatical structure, agreement with verbs, the sound of the word, and eventually (maybe at another level again) the way the lips need to be formed and air expelled. In contrast for choosing what to say, it is the semantics of chairs, the fact that you sit on them, they have legs, etc. that are important. The “planning to speak” bit needs to know there is a suitable word, what that word means and whether it will fit with other words, but does not need to know the details of how to say it. Similarly the “planning to move” bit needs to know roughly that legs can move in the desired way, but not the details of movement. Planning needs a model of action (speech or movement) and the model needs to be close enough to reality for it to work most of the time, but without all the details.

So the representations at a higher level need to share or independently represent enough of the lower level functions to be able to make appropriate suggestions for lower level action, but each will also represent different things. In addition, there needs to be some linkage between the two representations. Suppose you form the intention to say something like “The chair has four legs”. (Note “something like” because as you form the intention to say the thing the exact words will probably not be there.) In order for this intention to lead to the words “The chair has four legs”, something has to link the planning ‘chair’, with the saying ‘chair’.

This is not unlike human communication; we need both shared vocabulary and a level of shared meaning: so that when I say to a garage mechanic “the clutch is not working” it is the same thing we are referring to. However, we can also each have additional meanings, annotations etc.: the mechanic will know how the clutch works better than I do.

However, whereas human verbal communication has to be pushed through a discrete medium of signs, it seems more likely that there is a level of direct (but diffuse) connection between the ‘concepts’ used in our brains at different levels of activity.

1. Andy Clark. Being There. MIT Press. 1997. ISBN 0-262-53156-9. book@MIT [back]