Of course if ou are a touch typist things may be different.

I was especially struck by the image of the keyboard on the iPad, which on the ‘F’ and ‘J’ keys has an image of the tiny tactile bar that allows easy finger positioning by touch alone.  However, this is of course a flat screen and I am feeling I am in some sort of surreal world.

PPD10

The PPD10 workshop on Coupled Display Visual Interfaces¹ followed on from a previous event, PPD08 at AVI 2008 and also a workshop on “Designing And Evaluating Mobile Phone-Based Interaction With Public Displays” at CHI2008.  The linking of public and private displays is something I’ve been interested in for some years and it was exciting to see some of the kinds of scenarios discussed at Lancaster as potential futures some years ago now being implemented over a range of technologies.  Many of the key issues and problems proposed then are still to be resolved and new ones arising, but certainly it seems the technology is ‘coming of age’.  As well as much work filling in the space of interactions, there were also papers that pushed some of the existing dimensions/classifications, in particular, Rasmus Gude’s paper on “Digital Hospitality” stretched the public/private dimension by considering the appropriation of technology in the home by house guests.  The full proceedings are available at the PPD10 website.

AVI 2010

AVI is always a joy, and AVI 2010 no exception; a biennial, single-track conference with high-quality papers (20% accept rate this year), and always in lovely places in Italy with good food and good company!  I first went to AVI in 1996 when it was in Gubbio to give a keynote “Closing the Loop: modelling action, perception and information“, and have gone every time since — I always say that Stefano Levialdi is a bit like a drug pusher, the first experience for free and ever after you are hooked! The high spot this year was undoubtedly Hitomi Tsujita‘s “Complete fashion coordinator”², a system for using social networking to help choose clothes to wear — partly just fun with a wonderful video, but also a very thoughtful mix of physical and digital technology.

images from Complete Fashion Coordinator

The keynotes were all great, Daniel Keim gave a really lucid state of the art in Visual Analytics (more later) and Patrick Lynch a fresh view of visual understanding based on many years experience and highlighting particularly on some of the more immediate ‘gut’ reactions we have to interfaces.  Daniel Wigdor gave an almost blow-by-blow account of work at Microsoft on developing interaction methods for next-generation touch-based user interfaces.  His paper is a great methodological exemplar for researchers combining very practical considerations, more principled design space analysis and targeted experimentation.

Looking more at the detail of Daniel’s work at Microsoft, it is interesting that he has a harder job than Apple’s interaction developers.  While Apple can design the hardware and interaction together, MS as system providers need to deal with very diverse hardware, leading to a ‘least common denominator’ approach at the level of quite basic touch interactions.  For walk-up-and use systems such as Microsoft Surface in bar tables, this means that users have a consistent experience across devices.  However, I did wonder whether this approach which is basically the presentation/lexical level of Seeheim was best, or whether it would be better to settle at some higher-level primitives more at the Seeheim dialog level, thinking particularly of the way the iPhone turns pull down menus form web pages into spinning selectors.  For devices that people own it maybe that these more device specific variants of common logical interactions allow a richer user experience.

The complete AVI 2010 proceedings (in colour or B&W) can be found at the conference website.

The very last session of AVI was a panel I chaired on “Visual Analytics: people at the heart of data” with Daniel Keim, Margit Pohl, Bob Spence and Enrico Bertini (in the order they sat at the table!).  The panel was prompted largely because the EU VisMaster Coordinated Action is producing a roadmap document looking at future challenges for visual analytics research in Europe and elsewhere.  I had been worried that it could be a bit dead at 5pm on the last day of the conference, but it was a lively discussion … and Bob served well as the enthusiastic but also slightly sceptical outsider to VisMaster!

As I write this, there is still time (just, literally weeks!) for final input into the VisMaster roadmap and if you would like a draft I’ll be happy to send you a PDF and even happier if you give some feedback

Search Computing

I was invited to go to this one-day workshop and had the joy to travel up on the train from Rome with Stu Card and his daughter Gwyneth.

The search computing workshop was organised by the SeCo project. This is a large single-site project (around 25 people for 5 years) funded as one of the EU’s ‘IDEAS Advanced Grants’ supporting ‘investigation-driven frontier research’.  Really good to see the EU funding work at the bleeding edge as so many national and European projects end up being ‘safe’.

The term search computing was entirely new to me, although instantly brought several concepts to mind.  In fact the principle focus of SeCo is the bringing together of information in deep web resources including combining result rankings; in database terms a form of distributed join over heterogeneous data sources.

The work had many personal connections including work on concept classification using ODP data dating back to aQtive days as well as onCue itself and Snip!t.  It also has similarities with linked data in the semantic web word, however with crucial differences.  SeCo’s service approach uses meta-descriptions of the services to add semantics, whereas linked data in principle includes a degree of semantics in the RDF data.  Also the ‘join’ on services is on values and so uses a degree of run-time identity matching (Stu Card’s example was how to know that LA=’Los Angeles’), whereas linked data relies on URIs so (again in principle) matching has already been done during data preparation.  My feeling is that the linking of the two paradigms would be very powerful, and even for certain kinds of raw data, such as tables, external semantics seems sensible.

One of the real opportunities for both is to harness user interaction with data as an extra source of semantics.  For example, for the identity matching issue, if a user is linking two data sources and notices that ‘LA’ and ‘Los Angeles’ are not identified, this can be added as part of the interaction to serve the user’s own purposes at that time, but by so doing adding a special case that can be used for the benefit of future users.

While SeCo is predominantly focused on the search federation, the broader issue of using search as part of algorithmics is also fascinating.  Traditional algorithmics assumes that knowledge is basically in code or rules and is applied to data.  In contrast we are seeing the rise of web algorithmics where knowledge is garnered from vast volumes of data.  For example, Gianluca Demartini at the workshop mentioned that his group had used the Google suggest API to extend keywords and I’ve seen the same trick used previously³.  To some extent this is like classic techniques of information retrieval, but whereas IR is principally focused on a closed document set, here the document set is being used to establish knowledge that can be used elsewhere.  In work I’ve been involved with, both the concept classification and folksonomy mining with Alessio apply this same broad principle.

The slides from the workshop are appearing (but not all there yet!) at the workshop web page on the SeCo site.

1. yes I know this doesn’t give ‘PPD’ this stands for “public and private displays” [back]
2. Hitomi Tsujita, Koji Tsukada, Keisuke Kambara, Itiro Siio, Complete Fashion Coordinator: A support system for capturing and selecting daily clothes with social network, Proceedings of the Working Conference on Advanced Visual Interfaces (AVI2010), pp.127–132. [back]
3. The Yahoo! Related Suggestions API offers a similar service. [back]

I too have worried about the demise of interest not just in computers, but in science and technology in general.  Also, the way Apple exercise almost draconian control over the platform is well documented (even rejecting an eBook application for fear it could be used to read the Karma Sutra!).

However, is the problem the closedness of the platform?  On the iPhone and other smartphones, it is the apps that catch imagination and these are ‘open’ in the sense that it is possible to programme your own.  Sure Apple charge for the privilege (why – the income surely can’t be major!), but it is free in education.  So what matters, app development, is open … but boy is it hard to get started on the iPhone and many platforms.

It is not the coding itself, but the hoops you need to go through to get anything running, with multiple levels of ritual incantations.  First you need to create a Certificate signing request to get Development certificate and a Provisioning profile based on your Device ID … sorry did I lose you, surely not you haven’t even written a line of code yet, for that you really need to understand the nib file … ooops I’ve lost the web page where I read how to do that, wait while I search the Apple Developer site …

Whatever happened to:

10 print "hello world"

This is not just the iPhone, try building your first Facebook app, … or if you are into open standards X Windows!

Nigel Davies said his 7 year old is just starting to code using Scratch. I recall Harold Thimbleby‘s son, now an award winning Mac developer similarly starting  using Hypercard.

If we would like a generation of children enthused by Facebook and the iPhone, to become the next generation of computer scientists, then we need to give them tools to get started as painless and fun as these.

Devices are clearly everywhere, but the connections between them seem infrequent and poor.

Why is ubiquitous technology still so … well un-ubiquitous?

Why bother

One reason for lack of connectivity, is that for each production and purchase decision there is little need for it.  It is possible to wire up your whole home with sensors as Andy Stanford-Clark‘s twittering house demonstrates.  But for most people why put sensors in your home unless they are for something?  Why bother to buy things unless they give you value now? Home security, central hearing control, warning you when your fuel is running low — these are reasons for sensors, but in general we tend to buy things that do things for us, not things that might possibly plug together to do something interesting. This is maybe even more true of the new generation of digital natives, brought up with single function toys, and even Lego in  pre-determined models rather than generic kits.

Of course, we do buy things with sensors, the electric kettle has a temperature sensor, the TV knows what channel is tuned — but these are single purpose.  Even SCART connections for TV/DVD/satellite boxes are pretty limited unless the devices expect to be chained.  For the manufacturer, why bother to make stand-alone devices connectible; it will cost more in terms of a wireless chip, not to mention complex interfaces to manage privacy.  Even software rarely takes this step to make itself easy to interconnect — despite the mash-culture of Web2.0.  The barcode reader app on the iPhone is a rare exception; here the cost was relatively low and purely in  software (up front cost, but no cost per ‘device’), and yet even this is rare to see.  Once there are hardware costs even generic devices such as phones only include sensors for which they can see an immediate use: for ecxample, despite the presence of GPS and compass in the iPhone, the new iPad, with sightly different expected uses, has neither.  For physical devices, connectivity may come when individual devices require external interfaces, maybe to manage complex installations or monitoring via mini-webservers for less cost than dedicated displays, just as with most home routers.  Then the hardware cost of connectivity has a purpose for the dedicated device, and the additional cost of genericity is software only.

What’s there

It is interesting to look back at Mark Weiser‘s original vision of ubiquitous computing.  It is less connected than Adam’s ‘everywhere’, but equally rooted in research prototypes and potential, and yet has certainly come to pass as computation is certainly ‘ubiquitous’ in many ways.  Like others, I have always been struck by the disconnect between the opening words of Weiser’s The Computer for the 21st Century “The most profound technologies are those that disappear” and the rest of the paper, which focused on displays the visible face of computation.  But, with the exception of miniature MP3 players,  it is precisely these visible displays that are part of our lives.

Weiser classified displays into tabs, pads and boards, with associated scales (inch, foot, yard), and indeed the size of displays is crucial to their uses¹.  However, it is perhaps better to look at Wesier’s definitons of where and how these are used: tabs are ‘on things’ microwaves, some TV remotes, and old mobile phones (the kind that were primarily for telephoning people!), pads are held and carried and while decidely smaller than an foot, the iPhone and similar smartphones are clearly in this category, and finally boards are in the environment, stuck on walls (digital photo frames) and furniture (Microsoft surface).  All can be seen in many homes and offices, although the latter have perhaps had least impact so far,  laregly because of costs of production (foreseen by Wesier), but are clearly coming on stream.

The crucial thing is that all of these displays do something for us.  They are ubiquitous because they are not invisible.

How to use them

Adam’s book describes the profoundly different user experience of ‘everywhere’ interactions (pp:27-39), rather than being task-driven by the user, in situations such as such as automatic room lights “the system precedes the user“, “Whether or not you walk into the room in pursuance of a particular aim or goal, the system’s reaction to your arrival is probably tangential to that goal.“.  This what I have previosly called ‘incidental interaction‘, interactions not driven by the user’s immediate intentions, but by the system’s actions for the user.

Somewhat strangely the discussion in ‘Everywhere’ then turns to Bellotti et al’s “Making sense of sensing systems: five questions for designers and researchers“, which summarises some of the key issues of pervasive systems dating back to the 1984 Phone Slave project², such as knowing how to address a system that has no keyboard and knowing whether it is attending.  These are really important issues, but are about intentional use, getting the system, albeit embedded in the environment, to do something that the user has decided. At a more device level,  Abricht Schimdt‘s concept of implicit interaction³ is also in this area, making the interaction to achieve some purpose as natural and invisible as possible using sensors such as accelerometers. This is now common in devices such as the iPhone, but is still essentially about intentional interaction — again getting the device to do something.

In contrast, we seem to still have little understanding of the user experience of incidental interactions, where the system spontaneously does things for or to us, and they are still rare in the home and office, a few specific devices such as automatic lights, doors and central heating controllers, with little sign of widespread connectivity.  The main exceptions seem to be at a very large scale, Amazon book recommendations or traffic management systems.

The latter is perhaps an interesting comment on ubiquity as, for the ordinary user, the display is simply a red light.

1. see recent papers with Corina Sas “Mobile Personal Devices meet Situated Public Displays: Synergies and Opportunities” and with Lucia Terrenghi and Aaron Quigley “A taxonomy for and analysis of multi-person-display ecosystems“, which extend this size classification to larger and smaller displays. [back]
2. See “Phone Slave: A Graphical Telecommunications Interface” and YouTube video, although it was the longer envusonment video, that raised the user interaction design questions. [back]
3. Schmidt, A. Implicit Human Computer Interaction Through Context. Personal Technologies, 4(2&3), Springer-Verlag, 191–199, 2000 [back]

Their developer documentation has an example iPhone app, but not an example web page.  However, I made a web page using their interface in about 5 minutes (see PHP source of barcode reader).

If you have an iPhone you can try it out and scan a bar code now, although you need to install pic2shop first (but it is free).

By allowing third party apps to use their software they encourage downloads of their app, which will bring them revenue through product purchases.  Through free giving they bring themselves benefit; a good open access story.

The docs describing the Cocoa MVC design pattern in more detail in fact do a detailed comparison with the Smalltalk MVC, but do not refer to Seeheim or PAC, I guess because they are less well known now-a-days.  Only a few weeks ago when discussing architecture with my students, I described Seeheim as being more a conceptual architecture and not used in actual implementations now.  I will have to update my lectures – Seeheim lives!

1. Shocked to find no real web documentation for Seeheim, not even on Wikipedia; looks like CS memory is short.  However, it is described in chapter 8 of the HCI book and in the chapter 8 slides [back]