Position Paper for CHI 96 Basic Research Symposium (April 13-14, 1996, Vancouver, BC)

1. Abstract

Experience is presented of a five year study of over three thousand students who used an editor. The paper reviews some of the evidence for and against the proposition that long term user adaptation does occur. It seems that over a period of years very subtle changes in user behaviour do occur. Some methodological and research points are raised.

Keywords: editors, long term field studies, methodology, exploration, temporal aspects of usability, adaptable behaviour.

2. Motivation

In the past, stone masons gradually became master craftsmen, or at least some of them did. In contemporary times engineers can take years to develop and exploit new methods and techniques [4]. In the information age people are becoming life-long computer users. The productivity of employees and the capacity for individuals to grow personally and intellectually are known to be of great importance. In fact there is relatively little prior work on long-term performance and devolpment by computer users. This paper addresses some of these issues from the stand point of a recent 5 year study of over 3000 users.

3. Methodology

We have successfully used a methodology based on command monitoring to track students of computer science during their editor use. Everyone who invoked the editor entered the study unless they selected options to inhibit data generation.

In each of the 5 years hundreds of students would start computer science courses and hence join the study. Some would last less than a semester, many others only taking subjects for a year or two. Naturally some would major in computer science and even proceed to a fourth year.

Although the scale of the low level data has given some unique insights, methodological questions remain. Amongst the most important are:

• The gathering of task details. There are trade-offs between volume of data and quality of data. In our case we know what was invoked but have only very limited knowledge of the detailed tasks users were trying to undertake at particular times. This limits the availability of techniques for extracting meaningful results from the data.
• Survival rates. By their nature, many of the users who start in a monitoring experiment will drop out along the way. In our case, over three years it seems reasonable to expect between 75% and 90% drop outs. It would have been possible to improve this figure, but the loss from first to third year in a modular degree structure with wide choice will always be substantial.
• The value of early research. Because survival rates can be so low, the effective cost per participant of a research exercise, say a questionnaire, at the end of the study may be ten times the unit cost at the start. The granting of research funds also tends to favour early activity rather than later plans.

4. Evidence that long term adaptation occurs

One rather small cohort comprises 63 students who started university in 1991 and chose computer science and to use the editor for three continuous years. This group's data has evidence of long term changes in command invocation and preferences, but on a tiny scale compared with the major settling in that occurs during the first few weeks of use [2]. We are sceptical that all of these changes are task driven, but have yet to prove or quantify this.

Amongst the indicators of gradual, but fairly constant change in behaviour are:

• Exploration. A new operational definition of exploration has been devised. An exploration is deemed to occur the first time a command is invoked regardless of intention or outcome. This is a sensitive and early measure of initial exploratory activity, but it cannot distinguish between continuing exploratory interest and subsequent focused use. There is evidence that users are reluctant to undertake much spontaneous exploratory activity, especially when it is not directly relevant to the task at hand. Had this occurred, exploration patterns might be expected to have been highly irregular. Instead there are widespread indications of a slow but fairly steady level of exploration by users after an initial major burst [1]. Furthermore, it appears that the highest explorers were systematically trying out a range of keyboard commands over a long period. This does not seem to be either spontaneous or task-driven.
• Changes in file write method. In the editor, sam, there are two menu options to write out a file. The paper at CHI'96 [3] shows an apparent pattern in the "spontaneous" changes between methods by individuals. It is highly likely that there are more examples of this type, but they are not easy to find. Only long term studies can give us the underlying models of these clear changes in user behaviour.
• Individual Command Rankings. It is clear that the popularity of various commands does vary over time for single users [5] and also over the whole group. Although the rate of change is often painfully slow, there is clear movement when taken over three years [2]. For example, the cohort was tending to divide into those who did and did not use default window sizes. Furthermore the powerful, but hard to use, undo command was gradually explored and adopted by some.
• Regular expressions. Although the overall adoption of regular expressions within editing commands was disappointingly low, there is a cluster of new activity in the second academic year of use. The correspondence with the taught material in terms of dates is not convincing, but there is no doubt that users first tended to practice these operators in year two.

5. Evidence against adaptation

It is commonly believed that users go through a learning period of a few weeks and then settle into a pretty fixed pattern. Our data reveal that at one level of detail this is certainly so. Relative command frequencies become broadly set at an early stage.

The students in this study used sam on an entirely discretionary basis except in the initial weeks. Some senior students in 1991 were also given access, but no instruction at all. The general level of drop outs from sam is high, and very few of these seniors gave it more than a few minutes of attention. Of course, they may have had tasks and goals inconsistent with trying a new editor, but the poor level of commitment to sam, even to give it a go, does suggest very inflexible behaviour.

The low level of regular expression work suggests a conservative attitude too. Some users may have been adventurous, but the majority were not. Of course the nature of the editing tasks may not have been particularly appropriate for regular expressions, but the apparent lack of curiosity is a different matter.

6. Global view

The study has given the strong intuitive impression that some adaptivity is present in users, and that it may be intrinsically driven rather than due to changes in task or whatever. These latter factors are important, but they may not be the driving force in the long term.

There may be an important element of chance in what people adopt when alternatives are available. Perhaps we should think in terms of propensities to use commands, rather than binary decisions.

7. Research questions

Three important questions that need addressing are:

• To what extent can evidence of command use be taken as evidence of learning the command?
• What makes for adaptive behaviour? Is it even desirable?
• How can adaptivity be tracked and measured?

8. Acknowledgements

The Basser Department of Computer Science, University of Sydney, and especially Judy Kay; School of Cognitive and Computer Sciences, University of Sussex.

The author is on study leave from the Department of Computer Science, The University of Western Australia until July 1996.

References

1. R Cook, J Kay, G Ryan, and R C Thomas. A toolkit for appraising the long-term usability of a text editor. Software Quality Journal, 4(2):131-154, 1995.

2. J Kay and R C Thomas. Studying long-term system use. Communications of the ACM, Special Issue on End-user Training and Learning, 38(7):61-69, July 1995.
3. Judy Kay and Richard C Thomas. Visualisation of entrenched user prefer ences. In Proceedings of CHI '96 Conference. ACM Press, April 1996. [
4. G B Thomas, R C Thomas, and J G Littler. The cognitive role of an engineer in a diagnostic task. In Proceedings of UK IT 90 Conference, pages 259-265. IEE, London, 19-22 March 1990.
5. R C Thomas. Long term variation in user actions. In Papers from a work shop on Temporal Aspects of Usability, number G95-1 in GIST Technical Reports, Department of Computer Science, University of Glasgow, Glasgow, G12 8QQ, UK, July 1995.