Temporal Mapping in Arts and Humanities Data: Where and When's Waldo?

The most popular current Web 2.0 representation is geography: putting everything on a map. It's a powerful thing to do: when we can SEE how close registered sex offenders are to schools and day cares, we have certain reactions about where our psychic sense of "too near" or "too far" meets the legal/phyiscal interpretation of "appropriate distance." A little bit of information, as has been said many times, can be a dangerous thing. This particular offender/schools mash up does not provide a brushing interface that, say, relates re-offender statistics based on various distances from schools to help confirm whether our sense of dread is well-founded our not.

It is with this caveat in mind that, our group has been thinking about how adding not just mapping but temporal mapping might be for a project we have called musicSpace to integrate a variety of musicology sources for easy exploration. More recently in a project called continuum we'd been looking at how to map rich data sets like classical music onto timelines so that the visualization doesn't implode. That is, if there's lots of stuff going on at the same time in a time line, all the info looks like a big blob, or if you zoom out, you lose the surrounding context. Our challenge was to solve the "too much info=blob; too little=not enough information" dilemma. Inspired by that work, we'd like to take what we learned there and think map thoughts.

Mapping Time

What we are calling Temporal Mapping is not unknown but it's not common. to be clear, temporal mapping has one meaning in discussions of disease tracking for instance that doesn't involve visualizations; spatio-temporal mapping has another meaning in computing. The kind of temporal mapping we're considering is more akin to an example from the Land Cover Institute which on a map over a relatively stable geography shows how population density has grown and spread over 200 years. Other work shows how the geography of a place itself (such as a river valley) changes over time.

'Istanbul was Constantinople now its Istanbul" - They Might Be Giants

Our sense of temporal mapping it turns out is more complex than these example because it turns out we are looking at a variety or parameters that change: in terms of locations, borders change; names change and even the geography can change. One way to reflect this change is to use maps that can present borders/locations that are accurate for a given period - this assumes that various places recognize the same borders/place names. Consider the mapping of Taiwan as a political representation issue. To use the music examples, if a composer created something in the 1700s, the borders of the domains were different and the place names may be too, so we need to have maps with borders and place names that are accurate for that time. As we discuss below, there are other issues that come into play when, to coin a phrase, wanting to co-map points that cross times, and thus cross representations of locations.

Even if we don't want to co-map, but restrict ourselves to single maps, there are some challenges: if we know where a piece was composed we map that; if we know where a composer was born we map that, if we know where a composer first performed a piece we map that.

There are a few data subtleties there: do all objects in a classical music repository now need Lat/Long data associated with them, as well as a date? Even there the temporal bit is not so obvious: there are kinds of dates and kinds of locations: how tease these out so they are clear in the UI? so it's clear a person is choosing to see performance dates/locations rather than composition dates/locations. What happens if a work was known to have been taken out and put away over a range of places and times? How is that stored in an object in order to be represented?

If we put aside that question of the back end data representations and UI finesse for the moment, let's assume whatever it is we want to map in music we can map, the more glaring, basic challenges are how both borders and place names have changed not just over the centuries but even within decades. Maps that only map against geography lat/long have it somewhat easier than mapping against historically/politically accurate representations.

And as always, the question of how to represent the information is non-obvious. For instance, how handle multiple names or boundaries for a place? Only show the appropriate name for the specific time? Show all versions to provide context not only of place but between times? These kinds of decisions become critical when crossing domain representations. For example, what happens when looking for a location in europe that produced the most major compositions of the Romantic Era relative to location(s) in Europe of most significant performances in early 20thC. The borders and place names from the 1700s and indeed even between 1914 and 1920 change several times.

So Temporal Mapping is?
Perhaps a fast way to begin to think about temporal mapping in arts and humanities data that involves people, places and times is to be able to accurately reflect these places as they were interpreted both in their times, and in ours, and to be able view these comparisons from any variety of perspectives - comparatively, relatively.

Animation and Insight
A potential benefit of developing temporal mapping approaches for arts/humanities data is in meaning that is communicated through animation: if we can step through the various places by time of where Beethoven worked - see who else was in the neighborhoods at various points, and correlate that with specific works, and perhaps specific historical events and their key locations, can we begin, almost at a glance, to get a new appreciation of a domain space? Do seeing these patterns animated over time and space and politics and whatever else let us ask new kinds of questions - questions that would have been potentially intractable to ask before?

These are early days for our investigations, but from early scenarios domain experts have given us, the ability to step through time, and to see events of interest comparatively across time and space, is a thing devoutly to be wished. These representational desires are driving our current UI research efforts.

Why Graphs Suck for Exploring RDF (ie The Semantic Web)

The most popular mechanism for visualizing RDF - the underlying language to represent the Semantic Web - is a Great Big Graph. Take a look at any model you wish: rdf-gravity is a Big Fat Graph; frodo is a UML-like connected flow chart thing; there's RDF Graphs with GSS; and there's a suite of redeployed classic Node style visualizers that have been modeled and applied anew on rdf (pdf). And that's just a light sampling.

What is this obsession with using graphs to represent RDF?

At first i thought it was because Java, a tool frequently used to create these visualizations, comes default with a Touchgraph component, and its bouncy connected parts have a certain "gee whiz-ness" to them - at least the first time one sees them.

Now it seems, the use of graphs has become a (bad) habit, an overused trope, representing what David Karger calls the "pathetic fallacy" of using graphs to represent the Semantic Web ( a quiet exchange in a paper review that we're now teasing out into a Position you're reading here...).

So what's wrong with Great Big Graphs? After all, RDF is a graph. Ya well, as Karger's comments continued, so's the Web (ps), but we don't see people exploring the Web via it's bowtie shaped nodes (pdf), do we? Indeed, Kager takes this assertion further to state that "everything" can be represented by a graph, and yet we do not use graphs to represent "everything." Why not?

Great Big Graphs are known to address two things that we rarely want to do on the Web: show the Shape and Density of some collection of things so that we can say things like "Oh that's really big" or "there's a lot of activity going on down there in that part of the graph, but not much up here." The classic issue with a graph, however is that to get the overview, detail gets lost. Conversely, as we zoom in, the context gets lost. This loss of context is particularly irritating in touch graphs where zooming in on a component apparently breaks it off from the rest of its graph. This focus vs context issue of any graphical representation of information where the goal is to someone show the whole thing and yet also provide detail is a classic problem in Human Computer Interaction, and the subject of considerable research {refs to follow, but in the interim, take a look at the topic "focus + context" in the ACM digital library's search box}.

At the heart of the problem is that at scale (when there is considerable data to represent), or as soon as what a system is trying to model as a complete entity takes up more than one page/screen, the detail/overview compromise kicks in. Whether the graphs are UML diagrams, flow charts, clustering graphs, maps of geographical areas or maps of streets or networks, scale forces a compromise between focus and context, but if the interest IS in exposing an entire domain at once graphs and the techniques for balancing focus+context can be extremely effective (consider the micro map of either a 3D gaming environment that shows where one is in a world from an overview perspective relative to one's first person perspective; or the similar technique used in a Photoshop document where when zoomed in to work on a small section of image a map tool shows where one is working relative to the rest of the image. This map tile outline can also be used to navigate and reposition one's work area in the image). A question emerges here, however: if graphs are typically deployed to show the WHOLE of whatever is graphed, why is that an appropriate model for providing access to the Semantic Web? And if it isn't (a) why are we using it or (b) what alternatives are there for considering how to wrap up RDF data for effective use?

A fundamental question from the HCI perspective related to the above would be: what question/task/need is a given graph or other visualization answering? or perhaps, to put the question another way, what visualizations/representations/interactions would best support the specified tasks? And to push that question one further, what is particular about the semantic web such that new types of interaction designs may be required to support the types of tasks that are semantic web specific? Indeed this last question is the subject of a workshop on the Semantic Web and User Interaction. The challenge becomes: what, if anything, is special about what the Semantic Web enables such that existing UI paradigms don't suffice?

We've suggested that Great Big Graphs (GBG) are not appropriate as a de facto way of presenting the Semantic Web because the tasks it supports are limited. This limitation is not in itself a bad thing, but that we'd suggest that there is not a strong match between what a GBG provides and the kinds of information support people who use the Web have come to expect (try doing email or buying a book with a GBG).

Part of the problem, one may assert, is that SW data is delivered with more in common with a database and its schema than a Web Page - but even that argument doesn't wash, since most commercial web sites are delivered with a database back end now - and they look like web pages. So, the question, to repurpose Freud somewhat may be not why do graphs suck for the Semantic Web, but "what does a (SW) user want?"

Another way of putting the question of what do we as SW users want may be: "what are we trying to do?" Ben Shneiderman, HCI Guru at the U of Maryland, and his student Bill Kules, have more recently been framing the question as "what do you want to know?" effectively, Shneiderman has said forget trying to show everything since we can never see everything at once anyway, and focus on the kinds of things that are of interest to the explorer. Much of shneiderman's work, from spotfire to the more current hierarchical clustering, has indeed focused on enabling researchers to focus on the kinds of questions of interest to them - such as being able to look at the results of a variety of functions when applied to sets of datas - thus being able to see for instance in what conditions are their outliers.

The advantage of keeping the question as "what do we want to do" rather than "what do we want to know" may more explicitly capture one particular attribute of the Semantic Web which it has in common with many Web 2.0 applications: the desire to DO something on the Web with the data itself. To tag it; to edit it; to share it; to push it into new and or other representations.

These attributes of edit/tag/share are possible with Web2 aps, which break one part of pre Web 2 models, where the web is interactively read only. However, the specific affordances and constraints, to use Don Norman's terms, of the Semantic Web may take us beyond even these relatively new ways of interacting with information on the Web.

Difference at Source leading Difference at Interface
One of the interesting features of the semantic web that is not harnessed by simple Great Big Graphs of RDF is the fact that it is increasingly possible to break the paradigm of the page (called for in You've Got Hyperext) and actually enable people to choose a variety of representations for the information out there, depending again on what they want to do with it. Likewise, the immediate possibilities of how one set of data might be repurposed with another set of data automatically is also a remarkable and still largely untapped affordance of the Semantic web.

This capacity is enabled by that same RDF that wraps up and makes communicatable the semantics of the data in relation to itself and to other data. Just as the schema of a database makes visualizations like Spotfire possible, the RDF of the semantic web will make richer mechanisms for engaging with data possible.

We see some of this page-breaking, cross-web, context sensitive flexible repurposing of data in Semantic Web Applications like Haystack, piggy bank, AKTive Futures and /facet (pronounced "slash facet"), and from Semantic Web/Web 2.0 hybrid applications like mSpace and mSpace mobile

AKTive Futures, for instance, uses a cartesian graph as one facet of its interface presentation. The core interaction of the UI is to select countries for one axis and ranges of years for the other to look at trends in oil production in those places and times. By clicking on a spot on a line on the graph, the stories that are associated with those confluences are presented in a secondary window. In this case, the use of a particular kind of graph is appropriate for the task the designers of the application wish to support. Date and output data from MULTIPLE resources, (not just one database), are, as numeric data, represented in a numerically relevant fashion - not as static tables but on a graph where, in Shneiderman's parlance, the person using the service is not presented with all data for all time, but is enabled to select the ranges of interest and focus on them with an appropriate format.

For this site, data is coming from all over the Web and converted where it doesn't already exist in SW format into SW format (ie rdf most usually) so that it can be rendered appropriately for this kind of explorable user interface (UI). Indeed, the graph is used to help find trends of interest (not unlike Spotfire) and to use those relations of interest as the way to find the richly associated information to tease out what may have caused that particular moment.

Pre-existing Sites with a Purpose - Predefined Semantic Web UI aps
The above sites are examples of what happens when a site with A Purpose already exists. Haystack's exemplar is the "universal information client" which integrates calendar information with other associated tasks like hotel and flight booking along with finding relevant and related email to support tasks in process. In this case, Haystack is showing the way of using the Semantic Web to do old tasks better, using familiar UI paradigms in new contexts to make it easier to do related tasks that typically draw down on information from a variety of applications: checking email for when a conference is in order to get the dates into the calendar and check out flights for those times.

mSpace's is making classical music discoverable for people who know nothing about classical music. This discoverability is enabled by adding Preview Cues, or the ability to check out not just a piece, but the sound of an area of music, like sonatas or baroque, quickly and easily. This feature in itself is not driven by the semantic web, but it is powerfully supported by it. For instance, there are other affordances that the interface provides that go beyond online music explorers and into what makes the Semantic Web interesting: the browser automatically associates information from different sources about the music in the explorer with the music - choosing "period: baroque" yields a description of that content. This ap is another case of taking familiar and largely effective models for music library exploration and play back, and enhancing them to enable either improvement of previously doable but difficult or cumbersome tasks.

These sites suck in and make shapable information related to sepecific predefined domains. They use specific graphs to present the data in the domain (calendars, maps, timelines) but these are supplemented with or are supplemental to serving other activities, based on interaction models designed specifically to support certain kinds of information exploration and discovery tasks that are well-enabled by the Semantic Web.

For instance, with mSpace, new dimensions can be added to the domain as they become known; musicological data may be supplemented with technical recording data or historical data. The UI makes it possible (to use spreadsheet language) to pivot from one domain to another on a related term - so one moves from beethoven in the context of music to beethoven in the context of history. Sure yes one can do these pivots with databases and spreadsheets. Indeed, George Roberston's Polyarchy work called "Visual Pivot" (pdf) in fact has shown exactly such pivoting in very interesting ways from one database table to another. One may suggest, however, that the Semantic Web has the potential to break from database scale to greater, messier, heterogeneous Web scale.

Dynamic, Free Form Semantic Web UI aps
One of the challenges of the Semantic Web however is to enable us to just get at that rich data via our own dynamic contexts. For instance, suppose there's an interest in finding Jazz music that may be of interest and there's no pre-made mSpace Jazz explorer? or more intrigued yet, someone is interested in not only exploring the sounds of jazz but of seeing what is happening historically both politically and in architecture at the same time as different trends in music are occurring in order to explore the question what was influencing what when?

The above kind of questions means that a person may wish to be able to start exploring from a particular seed or set of seeds from which to start building and exploring relations (though even how to express these seeds may be challenging - another matter for interaction research innovation (ever know what you want but not the terms to express it so that you can find it on google?)). The above mix query means that samples of music need to be available so someone can audition the songs (we do not assume the Questor is a jazz expert) to see what's of interest; engage historical political period data from different regions; enable this data to be contextualized not only by location but by time, and readily explorable by time and by location visualizations. What's the ideal representation for this information as it is assembled? It is NOT a Great Big Graph (alone or primarily).

Web Founder and Semantic Web co-Founder Tim Berners-Lee has been developing an idea called the Tabulator (which i can never seem to find working), Conceptually, one starts with a specific known source of semantic web data, and then rather than in a graph, one selects cells in a tabular representation of the rdf, which expand into fresh tables, etc (go see the site for an image of this - maybe you can even get the demo to work). The data collected in these expansions can then be re-visioned into either a map, a calendar or a timeline (note the term "or"). There's considerable potential here - currently the source of the data is very geeky and not that non-geek friendly - data is expressed in rdf-ease triples like "colorPicture is mentioned in TAGmobile road trip BOS-> Amerst:photo" Qu'est-ce que se?

The tabulator also seems currently to be informed by the old-school Web-as-Read-Only, where as the impetus of Web 2 (and the semantic web) is towards read/write/re-write - a very much more Ted Nelson-ish hypertext vision ( a good thing) than pre Web2 vision.

Mix and Match on the Fly
So, some of the challenges for Semantic Web UI services besides de-geeking things like Tabulator will be to support data in formats so that the application has information that is relevant to what display options may be appropriate for it (dates, map coordinates, contacts). It's not clear what the solution is: micro formats is one approach; fresnel, defined as "a generic ontology for describing how to render RDF in a human-friendly manner" - where the style sheet for a data chunk effectively travels with that data offers another. It will be interesting to see how these approaches work across heterogeneous data sources and distinct contexts. It will also mean being able to add new data/links/tags(?).

That latter observation of the context in which the data is discovered leads back to the earlier observation that UI's for semantic web data, like all other human-usable systems, need to respect and support what the human wants to do with that data. Being able to establish context for multiple intersecting data domains and data types may be as critical as being able to take advantage of a pre-asserted format for a particular data chunk.

The bottom line is that Great Big Graphs have their place, but overall, it's a pretty limited place. Great Big Graphs are generally also pretty easy. The algorithms for pumping data into many graphs are well known. As Karger says, it's a pathetic fallacy to assert that because the data model is a graph the data should therefore be displayed as a graph. It's also, let's face it, a cop out in usability terms, unless all one wants to see is how big is the data set, where are the dense bits etc. The harder question is "how might this data be used? how will we support those heterogeneous requirements - and do so dynamically, elegantly"

People at the coal face of RDF and Ontology development mayn't see it as their mission to consider that more human-oriented approach to representing information spaces for human usable, human-useful exploration. But why not? The result may well be the generation of a generic Semantic Web browser - a tool that would enable people both to explore and contribute to the rich associations possible in the ((increasingly Social and) Semantic) Web.

[update Aug 17 '06 : the version of this blog entry David and i submitted to SWUI06 is available (in html) at http://eprints.ecs.soton.ac.uk/12911/]

Etymotic Research ER-6i in ear canal phones review: super sound to enhance your ipod

Two things can immediately improve the sounds coming from your ipod:

• - higher bit rates for the mp3's or aac's (ripping at 160 rather than 128 AAC for example)
• - better head phones.

Indeed the better the headphones that you have, the more you'll hear the artefacts of lower bit rate encodings. So, suppose you have some nicely encoded, or indeed entirely lossless tracks on your ipod (try putting your fave album or set of tracks on your ipod imported direct from the CD rather than encoded. Play them up against encoded mp3's or aac's and see what you hear) and you'd like to hear as much of what's there as possible: what headphone style do you choose?

You can choose traditional cans (regular over the ear headphones) - Aside: Next time you're thinking about sony or seinheisser, give Ultrasone HIFI or DJ phones a try - hit "test reports" to read reviews - they're designed to preserve your hearing, too). These types of headphones are dandy for certain kinds of situations - long listening sessions; editing - but for being on the go, they're bulky to say the least, and not great if you want to fall asleep with your tunes.

Another style is the popular earbud phone that comes with most digital audio players like ipods. These sit levered in the ear. In ear phones, like apple's in ear ipod headphones, point more directly into the ear canal, and thus cut out more of the noise from the outside, improving perceptions of things like bass in particular.

Etymotic ER-6i

The next level up (or in) is full into-the-ear-canal phones. When you see musicians on stage at live events and see the lines leading up around their ears, you're seeing custom made in-ear monitors (iem's) where the earpieces fit into the ear canals . The custom fit isolates external sounds and directs sound from the monitors into the ears.

And this is how good headphones can save your hearing: by having the 'phones cut off outside noise by over 20db, you can turn the volume DOWN on the listening device. Add to the isolation effect great transducers to translate the impulses coming from the source (like an ipod) into great sound, and you have a super combination.

The etymotic 6i's, as reviewed by the formerly named "ipod lounge" represent an affordable combination (approx 140 USD/75GBP) of great (hearing-saving) isolation and super sound, designed specifically with digital audio players like ipods in mind. What does that mean?

Etymotics makes a number of great in-ear sound-isolating phones: the 4 as flagship, the 6 as more affordable but still cool cousin and the 6i for ipods. The main thing about the 6i over the 6 is that it provides some much appreciated bass boost, as shown in the chart comparing the 6 and the 6i. This boost is subtle - you won't experience it as the bass pumped up, but as an overall richer sound, where you can better pick out the distinct bass and drums textures from a track.

What sets these headphones apart from other "in ear" phones, however, is the kind of audio detail available across the spectrum, not just the bass. You may hear instruments and touches that you hadn't heard before. In vocals, i've found that i'm hearing much more of the expression of a phrase or a note than with standard earbud or apple's in ear phones or sony's in ear phones (any model) or many traditional cans.

Moving in the world with 6i's on: Isolation of in-ear phones like these means that you do not hear things outside yourself - if you're running or just walking down the street, you may want to think about not putting these in so they're fully seated - that will let you use these more like ear buds where you can still hear the traffic.

Getting used to Canal Phones. Some people do not like the idea of sticking something deep into their ears. If you're one of these folks, these headphones mayn't be for you. One way to check your tolerance is to try out a set of disposable ear plugs: how do you feel about putting these into your ears? If you're ok with that, you can be ok with the ety 6i's

The 6i's come with one set of flanged ear pieces and one set of "memory foam" ear pieces (same stuff used for ear plugs). Each fit slightly differently. Other sizes of foam or flanges are available and etymotic will send you a sample pair different size if you think you need it. You can then get sets of the sizes you want. The degree of isolation is also different: 22db with the flanges; up to 30db with the memory foam. I've found i like the feel of the flange better than the foam, and for the environments i work in, including planes, the 22db is more than adequate.

Fit is critical to the sound experience of these headphones. Some people have said they've made their own modification to the 6i's flanged ear pieces cutting off the top flange so the larger two are left (scroll down this review on ilounge to see comments by people of their pre and post mod experience of the phones - all post mods are "wow these ARE awesome." - perception of the bass is particularly noted.)

Isolation vs Noise Canceling/Noise Reduction. Another attribute of the 6i's or indeed any good isolating phone is that it is better at reducing plane/train noise than "noise cancelation" headphones like the popular bose model - which also require batteries.

Now, a lot of people are happy with their £275 Quiet Comfort bose phones even though their web site does not say how many db's of sound are cancelled, whereas Sony's 99USD fontopia earbud noise cancelling headphones make clear they offer only 10dbs (70% of noise); etymotic's foam ear plugs are 30dbs; their 6i flanges are 22db.

For £80, with etymotic ER-6i's, you'll get both a better sound floor and better sound isolation in a lighter package than either Sony or Bose and one that doesn't require batteries. If you don't believe that, try this simple test: go somewhere where you can try a pair of the bose quiet comforts. Try them on, turned on. Then, remove headphones and stick in a pair of memory foam disposable type ear plugs (you don't squish in these: you roll them first to kinda compress them, then insert into the ear and hold for 5 secs: they expand to fit the ear canal shape). Which produces better noise reduction? If it's the ear plugs, imagine great sound coming through those plugs...

Aside: If you want to spend £275 on in-ear/isolation headphones, though, you can, and you can do much better in terms of both sound and isolation with that cash. See the Next Level section further down for how-to's. For about half that price, you can get some awesome professional level audio headsets, too!

Any Negatives? Hate the case. Ok, hate is a strong negative, but really. Etymotics provides a wee neoprene pouch for the phones with a mesh divider. This means stuffing the cables and earpieces into the thing. It's a mess and timely. ( See update bottom: the case has been changed and is a much better zip case)

An alternative device for storing 6i's is the earPod - a generic earphone case. Not quite as cools as sony's winding cases that come with their own headphones, but more durable, and will definitely fit the etymotic 6is as shown in this review at ipoditude.

Overall. The 6i's are a very good set of headphones. Whether you're listening to classical or house, the degree of detail in sound is impressive. The better the encoding the better the sound, of course, but even with lower bitrate encodings, these phones improve the experience.

The isolation is a boon for being able to turn the sound down and still hear an impressive and enjoyable level of detail. The isolation is also great for cutting out travel noise of the train/plane.

They take a bit to get used to in terms of learning how to put them in and get them out; and as other reviewers have noted, the difference between great and crap sound experience is all in the fit. Taking the time to get that fit right, though, pays off.

If you are looking for
- an improvement in audio experience on your ipod
- an improvement in your travel-with-audio commute experience
- a way to ensure the ongoing protection of your hearing

then these phones are a great option.

The Next Level: "Pro" IEMs

At the start of this article, i claimed these 6i's are an "affordable" package worth the price. Well, at around 150USD, "affordable" may be eye popping for folks used to thinking in terms of 20 bucks for earbuds. But that price is only the beginning.

• Etymotics flag ship single driver EP-4's are $330 (all prices here USD)
• Shure's top end "in ear monitors" dual driver EC5's are $499

• - Sensaphonics Prosonic 2x-s's used from Dave Matthews to Black Sabbath to coldplay are $750 (fitting not included)
• if you're in the UK , try ACS for £499 which includes fitting and fantastic after sales support
• - Ultimate Ears UE-10Pro's are 900 ( £749 from UK distributor + £50 for fitting ).

• Etymotic has dropped the price of 6i's by half: they're now available for 75 USD. They've also improved the case so that it is a top zip closure - which is much much better.(see image)
• Their after sales customer support is second to none.