|
We conducted a formative evaluation of a new prototype, in order to develop a language in which to describe as yet poorly understood phenomena. Our specific objective was to characterise how annotators approached the task we gave them with an unfamiliar tool, paying particular attention to how the affordances of the user interface (that is, the visual cues it provided for interaction) shaped tagging behaviour, summarised quantitatively against various measures, and explained through qualitative coding of the data.
We recruited 13 annotators (referred to as a1-a13) who used ClaimSpotter to annotate a 2 page research paper which they had preferably authored, or were at least very familiar with to avoid any comprehension problems. Ten users were PhD students, two were research fellows and the last was a professor. None had used ClaimSpotter before. Four of them (a1-a4: 1 student, 2 RFs and the professor) were considered 'experts' with the tag linking scheme, being members of the project team. The remaining nine (a5-a13) were considered 'beginners'. Each session was limited to one hour. Screen interactions were recorded with a capture tool, and all comments and discussions recorded, resulting in high quality audio-visual data as digital movies.
The following movies are extracted from the study and are organized to highlight some key moments of a user's 'dialogue' with ClaimSpotter:
Note: each extract is a 1 to 8 Mb. QuickTime file. You may need the EnSharpen video codec from TechSmith to view them.
|
