grue(x) = if ( now() < t ) green(x)
          else blue(x)

The problem is that an emerald apparently changes state from grue to not grue at time t, without any work being done.  Searle’s wall is just an extrapolation of this so that you can interpret the state of the wall at a time to be something arbitrarily complex, but without it ever changing at all.

This issue of the fundamental nature of computation has long seemed to me the ‘black hole’ at the heart of our discipline (I’ve alluded to this before in “What is Computing?“).  Arguably we don’t understand information much either, but at least we can measure it – we have a unit, the bit; but with computation we cannot even measure except without reference to specific implementation architecture whether Turing machine or Intel Core.  Common sense (or at least programmer’s common sense) tells us that any given computational device has only so much computational ‘power’ and that any problem has a minimum amount of computational effort needed to solve it, but we find it hard to quantify precisely.  However,  by Searle’s argument we can do arbitrary amounts of computation with a brick wall.

For me, a defining moment came about 10 years ago, I recall I was in Loughbrough for an examiner’s meeting and clearly looking through MSc scripts had lost it’s thrill as I was daydreaming about computation (as one does).  I was thinking about the relationship between computation and representation and in particular the fast (I think fastest) way to do multiplication of very large numbers, the Schönhage–Strassen algorithm.

If you’ve not come across this, the algorithm hinges on the fact that multiplication is a form of convolution (sum of a[i] * b[n-i]) and a Fourier transform converts convolution into pointwise multiplication  (simply a[i] * b[i]). The algorithm looks something like:

1. represent numbers, a and b, in base B (for suitable B)
2. perform FFT in a and b to give af and bf
3. perform pointwise multiplication on af and bf to give cf
4. perform inverse FFT on cf to give cfi
5. tidy up cfi a but doing carries etc. to give c
6. c is the answer (a*b) in base B

In this the heart of the computation is the pointwise multiplication at step 3, this is what ‘makes it’ multiplication.  However, this is a particularly extreme case where the change of representation (steps 2 and 4) makes the computation easier. What had been a quadratic O(N²) convolution is now a linear O(N) number of pointwise multiplications (strictly O(n) where n = N/log(B) ). This change of representation is in fact so extreme, that now the ‘real work’ of the algorithm in step 3 takes significantly less time (O(n) multiplications) compared to the change in representation at steps 2 and 4 (FFT is O( n log(n) ) multiplications).

Forgetting the mathematics this means the majority of the computational time in doing this multiplication is taken up by the change of representation.

In fact, if the data had been presented for multiplication already in FFT form and result expected in FFT representation, then the computational ‘cost’ of multiplication would have been linear … or to be even more extreme if instead of ‘representing’ two numbers as a and b we instead ‘represent’ them as a*b and a/b, then multiplication is free.  In general, computation lies as much in the complexity of putting something into a representation as it is in the manipulation of it once it is represented.  Computation is change of representation.

In a letter to CACM in 1966 Knuth said¹:

When a scientist conducts an experiment in which he is measuring the value of some quantity, we have four things present, each of which is often called “information”: (a) The true value of the quantity being measured; (b) the approximation to this true value that is actually obtained by the measuring device; (c) the representation of the value (b) in some formal language; and (d) the concepts learned by the scientist from his study of the measurements. It would seem that the word “data” would be most appropriately applied to (c), and the word “information” when used in a technical sense should be further qualified by stating what kind of information is meant.

In these terms problems are about information, whereas algorithms are operating on data … but the ‘cost’ of computation has to also include the cost of turning information into data and back again.

Back to Searle’s wall and the Goodman’s emerald.  The emerald ‘changes’ state from grue to not grue with no cost or work, but in order to ask the question “is this emerald grue?” the answer will involve computation (if (now()<t) …).  Similarly if we have rules like this, but so complicated that Searle’s wall ‘implements’ a word processor, that is fine, but in order to work out what is on the word processor ‘screen’ based on the observation of the (unchanging) wall, the computation involved in making that observation would be equivalent to running the word processor.

At a theoretical computation level this reminds us that when we look at the computation in a Turing machine, vs. an Intel processor or lambda calculus, we need to consider the costs of change of representations between them.  And at a practical level, we all know that 90% of the complexity of any program is in the I/O.

1. Donald Knuth, “Algorithm and Program; Information and Data”, Letters to the editor. Commun. ACM 9, 9, Sep. 1966, 653-654. DOI= http://doi.acm.org/10.1145/365813.858374 [back]

Tags are of course becoming ubiquitous in the digital world: Flickr photos, del.icio.us bookmarks; at the digital/physical boundary: RFID and barcodes; and in the physical world: supermarket price stickers, luggage labels and images of Paddington Bear or wartime evacuees each with a brown paper label round their necks. Indeed we started off the day being given just such brown paper tags to design labels for ourselves.

As well as being labels so we know each other, they were also used as digital identifiers using a mobile-phone-based image-recognition system, which has been used in a number of projects by the project team at Edinburgh (see some student projects here). We could photograph each others tags with our own phones, MMS the picture to a special phone number, then a few moments later an SMS message would arrive with the other person’s profile.

Being focused on a single topic and even single word ‘tag’ soon everything begins to be seen through the lens of “tagging”, so that when we left the building and saw a traffic warden at work outside the building, instantly the thought came “tagging the car”!

The workshop covered loads of ground and included the design and then construction of a real application – part of the project’s methodology of research through design. However, two things that I want to write about. The first is the way the workshop made me think about the ontology or maybe semiology of tags and tagging, and the second is a particular tag (or maybe label, notice?) … on a toilet door … yes the good old British scatological obsession.

String and The Semiology of Tags

We filled sheets full of key terms with arrows and lines between them, but for me one of the key concepts was the string that ties a paper tag to whatever is being tagged. Of course the string is itself simply an emblem: for a price sticker it is the glue and for a Graffiti artist it is the act of painting. However, the crucial thing about the string (or sticky, or whatever), is that it joins tag to tagged.

Because the string joins the tag to the tagged it emphasises the ancillary role of the tag. The tag s not part of the tagged thing, but a add-on, an annotation, or in a Semantic Web world, meta-data.

The tag itself is usually in some way conveying information – in classic semiotics it is a sign. In the case of iconic or textual tags (for example, a price tag) a sign to be directly interpreted by the observer, in the case of digital tags (such as a barcode, or RFID tag) an identifier that is associated with further information, or sometimes both, as was the case with the application that recognised our hand crafted name tags.

The act of sticking or tying a tag onto a physical object, or digitally tagging a photo in Flickr, at one level binds the tag to the tagged object. However, the binding of tag to tagged is not just an association between them, otherwise every knot tying would be a tagging. The crucial thing is that by tying the tag to the tagged an implicit relationship is created between the tagged and thing or concept signified by the tag. In the case of a tied-on tag, the string is the physical representation of the relationship.

The thing signified by the tag and the relationship signified by the string may be of many kinds:

In the case of a tag on a cow’s ear or a brand on a slave the thing signified is a person and the relationship is ownership.

• In the case of a price label, the thing denoted is the money and the relationship cost.
• For tagged images in Flickr, the signified thing is some sort of concept or category and the relationship establishes that thing is (or is believed to be) of the signified category.
• For the evacuee children or destination labels on luggage, the signified is place and the relationship “is going to”.

In some case it may be better to think of the thing signified as a predicate or property and the relationship imputed by the string (or sticking) as ‘satisfies the predicate’. The security stickers on luggage or the frank on a letter does this saying “this luggage has been checked” or “this letter has been sorted”.

Graffiti tags are particularly interesting as the message behind a tag on a bridge or wall is not so much a statement about the wall itself, but says “the tagger was brave (or foolish) enough to climb to this dangerous position, or risk arrest by the police to do this”. Here the tagged thing itself is a sign that represent the danger, illegality and risk associated with the place and the subject of the tag becomes the tagger herself.

There was and is far more to write about this … but as this is already becoming an essay rather than a blog, I will move on. My fascination with the string is perhaps because it is embodying relationship and so picks up on themes of externalisation allowing higher-order thinking …

A Note on the Toilet Door: Performative Labelling

While with a head full of tags and also thinking signage, I noticed that the toilet downstairs from the meeting room had a paper sign on. Presumably there had once been a metal sign, similar to (but different from) that on the women’s toilet opposite, but this had, I guess, fallen off and lost and hence the stuck-on paper sign.

[[image of sign to come]]

This made me ponder on the nature of the sign. For me it was more like a signpost, telling me where to go, or maybe a sort of warning “what lies behind this door is a men’s toilet”.

If there had been no sign, the nature of the room would have been immediately obvious, as you opened the door a line of urinals faced you. The room is a men’s toilet because that is what it is and the sign on the door is merely descriptive, telling you what is true anyway.

[ image to come]]

However, in InfoLab21 where I work, the ladies and gents toilets are identical, the only distinguishing feature is the sign on the door. I was reminded of Searle/Austin’s speech act theory and perfornative utterances, such as the minister at the wedding “I now pronounce you man and wife”. It is the speaking of the words that makes them true. In the case of the InfoLab21 toilets, the sign on the door is a performative tag/label – the toilet is a men’s toilet because the sign says so.

Ownership tags, or a dog peeing on a lamppost are similar … which I guess is an appropriately scatological point to stop.

I felt a little like a teenager complete with T-shirt and iPod dropped into Jurassic Park arguments that I thought had been put to bed in the 1980s suddenly resurfacing – how do you explain this white thing that makes sounds from its earphones to a caveman wearing skins?

However, I also felt a certain sympathy as I often wonder about computer science as a whole; indeed it has its own arguments in the 1960s and ’70s as to whether it was a ‘discipline’ as opposed to just an application domain for maths or electronics, or just a tool for business. Maybe one of the clinchers was the theoretical foundations of computing in the work of Church and Turing … but strangely enough at Lancaster the closest to this, the course on algorithmic complexity, is taught by a HCI person!

One of my worries in computing is that these theoretical foundations are still weak, there is black hole in the theoretical centre of computer science¹. However, these theoretical issues were certainly not what was bothering my colleague. To answer his challenge and my own worries about the discipline we really need to know – what is computing?

Fundamentals: algorithmics and systemics

One way to ‘carve up’ computing is in terms of algorithmics and systemics².

Algorithmics – the way computational devices behave individually and largely internally to achieve particular input/output behaviours.

Systemics – the way computational devices fit together in systems with their environments, in order to achieve external purposes.

When a computer scientist thinks of the ‘core’ of computing, it is the algorithmic side that springs most readily to mind and this is certainly the principle focus of more theoretical computer science. However, when Peter Denning suggests seven categories for fundamental principles of computing: computation, communication, coordination, recollection, automation, evaluation and design; it is notable that only two, computation and recollection, fall relatively soundly under algorithmics³.

Indeed, the topics and sub-disciplines of computing address different aspects of both algorithmics and systemics. Some topics are almost entirely about algorithmics: programming languages (theory and practice), compilation, data structures and algorithms, and algorithmic complexity. Other topics, notably software engineering and human-computer interaction are focused most strongly on systemics. Most topics incorporate aspects of both: for example most courses on databases consider both the context in which databases work and the way in which they connect to code (systemics), but also the data structures and algorithms used to achieve this behaviour (algorithmics); and in my own HCI courses I cover both the more external human design issues and algorithmic and architectural issues of interactive systems⁴.

So, while issues of systemics underlie the majority of modern computer science, it is still algorithmics alone that comprises our caricature of the subject. Systemics is like the Cinderella of the discipline who does all the work but gets none of the limelight.

Taking systemics seriously

Apart from being the grist for bar table discussions of computing does this matter?

It certainly seems to influence what we regard as basics of computing for first year computing students.

When I was in previous departments that had both ‘computer science’ and business computing students, the former were taught Java and the later Visual Basic. VB was regarded as a somewhat Noddy language in particular as it was poor in data structures and so less suitable for fundamentals of real programming such as arrays and iteration (interestingly both fine in VB!). What interested me was that students doing VB started out with event-based programming as the ‘easy’ way into programming with VB is to construct a GUI using visual tools and then add behaviours. In contrast, for those doing Java this would have been an ‘advanced’ topic they did when they eventually got on to GUI programming, or maybe never.

Event-based programming is core when you think systemics – that the computer program you are writing is part of a larger whole, but seems peripheral to a more narrow algorithmic focus. Of course, it is not just graphical user interfaces where event-based programming is core, bur also any sort of distributed, networked or embedded programming and not least web computing both at the server-end with transactional code and now at the client-end also with AJAX. Absolutely central to so many aspects of computing but often an ‘optional extra’ in our teaching

At a theoretical level Wegner has been vocal in advocating that an adequate account of theoretical computing has to take into account interaction with the environment.⁵ Personally, I don’t buy all his arguments, but the crucial message is clearly true: computational devices do not simply take an input and produce (eventually) an output, bur are in fact processes involving their own state and interacting with their environment: input and output are in fact sequences not singular occurrences⁶.

Systemics also forces you to take seriously embodied computation both in the theoretical sense that I have considered myself⁷ and also in the practical sense that computational devices are physically in the world as well as interacting with it and so we need to take this into account. For example, issues of power consumption were central to the One Laptop per Child project as they are in ubiquitous and mobile computing.

With the exceptions of space and time complexity algorithmics focuses us on functional properties whereas systemics connects more closely to non-functional properties. Interestingly all the problems I cited in my recent blog ‘”I just wanted to print a file” are about systemics

1. This demands a discussion of its own, but the basic problem is that while Church and Turing gave us understanding of disembodied computation, we still do not have clear understanding of generic computation when embodied in devices in general only particular architectures. [back]
2. Perhaps I am still being too narrow in this distinction. Arguably there ought to be a heading pragmatics: how we understand and manage the processes external to the computational system, those that give rise to it (e.g. requirements), and those that sustain it (e.g. maintenance, testing, documentation). However, for the time being I’ll lump these under the systemic label. [back]
3. The principles are listed at Denning, P. J. 2007. Computing is a natural science. Comm. ACM 50, 7 (Jul. 2007), 13-18. DOI= http://doi.acm.org/10.1145/1272516.1272529. They are part of Denning’s programme to develop “Great Principles of Computing“. [back]
4. In fact, Lancaster Computing prides itself on its systems orientation and this is reflected in areas of expertise that are strongly focused on systemics: networking, distributed systems, software engineering, HCI, embedded systems. [back]
5. Wegner, P. 1997. Why interaction is more powerful than algorithms. Comm. ACM 40, 5 (May. 1997), 80-91. DOI= http://doi.acm.org/10.1145/253769.253801, see also Peter Wegner’s home page and Dina Goldin’s papers on Interactive Computation [back]
6. In Wegner and Goldin’s work IO are strictly sequences, not time series – they assume synchronised stepwise processing. As an aside it is interesting to think more widely about time in computing. Time is taken into account in various areas notably algorithmic complexity and (naturally) real-time computing. However, the central notion of computability in Church and Turing ignores time so long as it is finite. [back]
7. See “matterealities and the physical embodiment of code” and my notes on embodied computation. [back]

Sony have robot dogs, Phillips robot cats (albeit stuck sitting in one place) but Ugobe have little robot dinosaurs called Pleo. In the videos they do move like little baby creatures and the lady in the shopping mall coos over one as she strokes it.

Central to Pleo seems to be:

1. embodiment – they feel through 40 sensors and move in their environment
2. emotion – they have a relatively complex model of basic drives rather like Cynthia Breazwal describes in her book “Designing Sociable Robots“.

This seems to pay off in people’s reactions, both on Pleo’s own videos (well they would!), but also in owner’s plogs (sic) … one owner says:

“she acts just like a cat concerning keyboards.. just crawl on the darn thing while I’m typing! I know Penny,. you’re so cute it doesn’t matter what you do. But you should have a little sensor strip in your butt to spank when you’re bad¹ or to pat gently to urge you to go explore. Go to sleep my little love” ArcticLotus

Pleos making friends :-/

For researchers there is an open architecture so it should be possible to play oops experiment with them The API doesn’t seem to be published yet, so wait until you get your cheque books out!

1. This could get us into the territory of agent abuse! [back]

The people there included someone who studied people coding about DNA, someone interested in text, anthropologosts, artists and an ex-AI man. We talked about embodied computation¹, the human body as part of computation, the physical nature of code, the role of the social and physical environment in computation … and briefly over lunch I even strayed onto the modeling of regret … but actually a little off topic.

physicality – Played a little with sticks and stones while talking about properties of physical objects: locality of effect, simplicity of state, proportionality and continuity of effect².

physical interaction – Also talked about the DEPtH project and previous work with Masitah on natural interaction. Based on the piccie I may have acted out driving when talking about natural inverse actions

ubiquity of computation – I asked the question I often do “How many computers do you have in your house” … one person admitted to over 10 … and she meant real computers³. However, as soon as you count the computer in the TV and HiFi, the washing machine and microwave, central heating and sewing machine the count gets bigger and bigger. Then there is the number you carry with you: mobile phone, camera, USB memory stick, car keys (security codes), chips on credit cards.

However at the Firefly demo later in the morning they got to see what may be the greatest concentration of computers in the UK … and all on a Christmas Tree. Behind each tiny light (over 1000 of them) is a tiny computer, each as powerful as the first PC I owned allowing them to act together as a single three dimensional display.

embodiment of computation – Real computation always happens in the physical world: electrons zipping across circuit boards and transistors routing signals in silicon. For computation to happen the code (the instruction of what needs to happen) and the data (what it needs to happen with and to) need to be physically together.

The Turing Machine, Alan Turing’s thought experiment, is a lovely example of this. Traditionally the tape in the Turing machine is thought of as being dragged across a read-write head on the little machine itself.

However … if you were really to build one … the tape would get harder and harder to move as you used longer and longer tapes. In fact it makes much more sense to think of the little machine as moving over the tape … the Turing machine is really a touring machine (ouch!). Whichever way it goes, the machine that knows what to do and the tape that it must do it to are brought physically together⁴.

This is also of crucial importance in real computers and one of the major limits on fast computers is the length of the copper tracks on circuit boards – the data must come to the processor, and the longer the track the longer it takes … 10 cm of PCB is a long distance for an electron in a hurry.
brain as a computer - We talked about the way each age reinvents humanity in terms of its own technology: Pygmalion in stone, clockwork figures, pneumatic theories of the nervous system, steam robots, electricity in Shelley’s Frankenstein and now seeing all life through the lens of computation.

This withstanding … I did sort of mention the weird fact (or is it a factoid) that the human brain has similar memory capacity to the web⁵ … this is always a good point to start discussion

While on the topic I did just sort of mention the socio-organisational Church-Turing hyphothesis … but that is another story

more … I recall counting the number of pairs of people and the number of seat orderings to see quadratic (n squared) and exponential effects, the importance of interpretation, why computers are more than and less than numbers, the Java Virtual Machine, and more, more, more, … it was very full hour

1. I just found notes I’d made for web page in embodied computation 5 years ago … so have put the notes online [back]
2. see preface to Physicality 2006 proceedings [back]
3. I just found an online survey on How many computers in your house [back]
4. Yep I know that Universal Turing machine has the code on the tape, but there the ‘instructions’ to be executed are basically temporarily encoded into the UTM’s state while it zips off to the data part of the tape. [back]
5. A. Dix (2005). the brain and the web – a quick backup in case of accidents. Interfaces, 65, pp. 6-7. Winter 2005.
   http://www.hcibook.com/alan/papers/brain-and-web-2005/ [back]