A scenario: I have just been writing a reference for a student and have a folder for the references open on my desktop. I select “Save As …” from the Word menu and get a file selection dialogue, but I have to navigate through my hard disk to find the folder even though I can see it right in front of me (and I have over 11000 folders, so it does get annoying).

The solution to this is easy, some sort of virtual folder at the top level of the file tree labelled “Open Folders …” that contains a list of the curently open folder windows in the finder.  Indeed for years I instinctively clicked on the ‘Desktop’ folder expecting this to contain the open windows, but of course this just refers to the various aliases and files permamently on the desktop background, not the open windows I can see in front of me.

In fact as Mac OSX is built on top of UNIX there is an easy very UNIX-ish fix (or maybe hack), the Finder could simply maintain an actual folder (probably on the desktop) called “Finder Folders” and add aliases to folders as you navigate.  Although less in the spirit of Windows, this would certainly be possible there too and of course any of the LINUX based systems.  … so OS developers out there “fix it”, it is easy.

So why is it that this is a persistent and annoying problem and has an easy fix, and yet is still there in every system I have used after 30 years of windowing systems?

First, it is annoying and persistent, but does not stop you getting things done, it is about efficiency but not a ‘bug’ … and system designers love to say, “but it can do X”, and then send flying fingers over the keyboard to show you just how.  So it gets overshadowed by bigger issues and never appears in bug lists – and even though it has annoyed me for years, no, I have never sent a bug report to Apple either.

Second it is only a problem when you have sufficient files.  This means it is unlikely to be encountered during normal user testing.  There are a class of problems like this and ‘expert slips’¹, that require very long term use before they become apparent.  Rigorous user testing is not sufficient to produse usable systems. To be fair many people have a relatively small number of files and folders (often just one enormous “My Documents” folder!), but at a time when PCs ship with hundreds of giga-bytes of disk it does seem slighty odd that so much software fails either in terms of user interface (as in this case) or in terms of functionality (Spotlight is seriously challenged by my disk) when you actually use the space!

Finally, and I think the real reason, is in the implementation architecture.  For all sorts of good software engineering reasons, the functional separation between applications is very strong.  Typically the only way they ‘talk’ is through cut-and-paste or drag-and-drop, with occasional scripting for real experts. In most windowing environments the ‘application’ that lets you navigate files (Finder on the Mac, File Explorer in Windows) is just another application like all the rest.  From a system point of view, the file selection dialogue is part of the lower level toolkit and has no link to the particular application called ‘Finder’.  However, to me as a user, the Finder is special; it appears to me (and I am sure most) as ‘the computer’ and certainly part of the ‘desktop’.  Implementation architecture has a major interface effect.

But even if the Finder is ‘just another application’, the same holds for all applications.  As a user I see them all and if I have selected a font in one application why is it not easier to select the same font in another?  In the semantic web world there is an increasing move towards open data / linked data / web of data², all about moving data out of application silos.  However, this usually refers to persistent data more like the file system of the PC … which actually is shared, at least physically, between applications; what is also needed is that some of the ephemeral state of interaction is also shared on a moment-to-moment basis.

Maybe this will emerge anyway with increasing numbers of micro-applications such as widgets … although if anything they often sit in silos as much as larger applications, just smaller silos.  In fact, I think the opposite is true, micro-applications and desktop mash-ups require us to understand better and develop just these ways to allow applications to ‘open up’, so that they can see what the user sees.

1. see “Causing Trouble with Buttons” for how Steve Brewster and I once forced infrequent expert slips to happen often enough to be user testable [back]
2. For example the Web of Data Practitioners Days I blogged about a couple of months back and the core vision of Talis Platform that I’m on the advisory board of. [back]

% ls -l tmp
total 4
-rw------- 1 user group 1363 Feb 4 2003 getchap-times
% du -k tmp
1914 tmp

Last night I was trying to move copy a directory structure between machines. It failed, for some reason, I guess permissions somewhere, so I tried to clean out some directories with large numbers of ‘temporary’ files that hadn’t got deleted. There were a couple of directories like this and it is not trivial selectively deleting files from very full directories as something like ‘rm search*.out' fails if there are too many files matching ‘search*.out‘¹. I know I could have written a quick C program, but life is short and so after many partial match commands of the form ‘rm search23*.out‘ I started to get there.

Then I turned to a second directory, which seemed to be biggy. However it turned out to have just a dozen files in it. I deleted all but one and assumed it was enormous, but ‘ls -l‘ got:

% ls -l tmp
total 4
-rw------- 1 user group 1363 Feb 4 2003 getchap-times

Just 1363 bytes? I thought I must have been mistaken when I’d thought the directory was full, so checked du again:

% du -k tmp
1914 tmp

“Ah!, I thought, “hidden files”, so a quick ‘ls -l‘

% ls -la tmp
total 3832
drwxrwxrwx 2 user group 1942016 Dec 14 06:37 .
drwxr-xr-x 5 user group 1536 Apr 22 2006 ..
-rw------- 1 user group 1363 Feb 4 2003 getchap-times

No hidden files … but boy look at ‘.’. The directory itself is the biggy. The directory contains one file of 1.3K, but is itself 1.9 meg big!

Several years ago (i.e. since when there will have been reboots, file checks, etc.) the directory had loads of small files in it that I deleted (using the same laborious method I had done in the other directory last night). In order to have so many files the directory itself will have to have stored an entry for each – leading to its whopping 1.9 meg size (there were a LOT of little files, UNIX is good at being scalable). … but when the files are deleted, UNIX does not reclaim this space.

I guess whoever coded the ‘unlink‘ system call never thought about the possibility that directories would get so big and so didn’t try to reclaim the space and there is no ‘garbage collection’ on the file system.

This was on SunOS 5.8 – is this the same on other versions of UNIX – LINUX?

So next time you are wondering where all your space has gone … maybe it is in those empty directories. :-/

1. the ‘search*.out‘ is of course expanded by the shell into a very very big command line! [back]