e-Dance project

It’s always a nice feeling when a book you’ve been working on for months finally lands on your doormat, and so what better way to kick off the week than to proudly hold up Knowledge Cartography: Software Tools and Mapping Techniques!

Co-edited with KMi Research Fellow Ale Okada, and NESTA Fellow Tony Sherborne, Creative Director at the Centre for Science Education at Sheffield Hallam U.

We summarise the book’s orientation in the preface:

While “sense” can be expressed in many ways (non-verbally in gesture, facial expression and dance, and in prose, speech, statistics, film…), knowledge cartography as construed here places particular emphasis on digital representations of connected ideas, specifically designed to:

I. Clarify the intellectual moves and commitments at different levels.
(e.g. Which concepts are seen as more abstract? What relationships are legitimate? What are the key issues? What evidence is being appealed to?)

II. Incorporate further contributions from others, whether in agreement or not.
The map is not closed, but rather, has affordances designed to make it easy for others to extend and restructure it.

III. Provoke, mediate, capture and improve constructive discourse.

e-Dance’s mission is to explore how an approach such as this — which emphasises clarity and rigour of thought, making relationships explicit between ideas — meshes with the modes of cognition and creativity that we find in choreography. Tomorrow at DRHA’08 in Cambridge, we’re hosting a panel session reflecting on the changing possibilities of the digital interface that performers, choreographers and audience may now encounter with dance. Thinking specifically about mapping memetic, intellectual worlds, we might muse the following:

• The concept of “memetic media” is quite an interesting one, pointing to the fact that e-Dance media fragments can have ideas behind them that are being crafted by the choreographer within a knowledge mapping tool
• In terms of the knowledge mapping interface, it’s possible that performers might not see this at all, if it is primarily a tool for the choreographer to reflect on discussions in rehearsal, to maintain her multimedia research archive, and to craft performance and/or research narratives (e.g. paths of ideas)
• Another scenario is that the knowledge mapping interface is exposed to performers, as a useful visualization for discussion, eg. of how ideas develop in the unfolding of a piece, or to show spatial configurations
• Yet another scenario is that the tools generate visualizations of memetic media fragments that are good enough to use as part of the performance, eg. dancers trigger, and/or respond to visualizations
• It is possible that visualizations of the connections between ideas, themes, moods and media fragments might literally provide an orienting map for audiences to follow the connections between activity in different locations and times.

http://www.allhands.org.uk/2008/programme/index.cfm

Just back from the UK e-Science All Hands Conference 2008 held at the University of Edinburgh.  Our presentation was part of the Workshop led by Lorna Hughes, CeRCH http://kcl.ac.uk/iss/cerchfocusing on the use of e-Science in the Arts and Humanities – “e-Science in  the Arts and Humanities: Early Experiments and Systematic Investigations”.  The morning session, of which we were part, concentrated on arts/e-Science projects that, broadly speaking, fell into the ‘beyond text’ category.  In other words those projects that were using e-Science tools as a means of both doing and presenting research in forms that were not primarily concerned with linguistic modes of communication.  Sally-Jane Norman gave a great presentation in this session on her AMUC project (Associated Motion capture User Categories)   http://culturelab.ncl.ac.uk/amuc/ . I was particularly interested in her thinking about the use of mo-cap for social interaction analysis.   She also clearly articulated the significance of tacit knowledge in embodied practices and the need to recognise and assign value to this methodologically.

Our presentation was based on our paper that has been shortlisted for the peer review process for publication in the Royal Society’s Philosophical Transactions A. Here is a pre-print version of the article:

Bailey, H., Bachler, M., Buckingham Shum, S., Le Blanc, A., Popat, S., Rowley, A. and Turner, M. (2009). Dancing on the Grid: Using e-Science Tools to Extend Choreographic Research. Philosophical Transactions of the Royal Society A, 13 July 2009, Vol. 367, No. 1898, pp. 2793-2806. [PDF]

SIGGRAPPH (ACM Special Interest Group in Graphics, http://www.siggraph.org/) has been a long running and very large conference aimed mostly at the technical aspects of Computer Graphics: Games industry, Hollywood, animation productions are all beneficiaries of the outcomes from these research topics. The University of Manchester runs a Professional Chapter of the organisation and was at the meeting (LA August 2008) to promote activities (http://manchester.siggraph.org/).

This year there was an arts orientated keynote Catherine Owens (Artist/Director CO-DIRECTOR, “U2 3D”) Giving Technology Emotion: From the Artist’s Mind to “U2 3D”. She discussed the aspects of creating a 3D movie from U2’s tour of South America. Catherine has been working with U2 for many years, and the move to 3D was adventurous – using multiple 3D camera setups from famous sources. Time and development were key – the post editing took 18 months and new 3D blends were specially developed to allow different 3D scenes to blend together; as well as the use of a multiple sets of expensive 3D cameras.

A quote was 3D should be “easy on the mind”. So there were very few “obvious” 3D effects and more story telling: although quiet a few 3D overlay and blending effects; compare with transparency and overlay requirements.  Link to the official site: http://www.u23dmovie.com/

“Irish artist/director Catherine Owens creates installations that evolve through painting, sculpture, photography, sound, and video. She is well known for her collaboration with the Irish band U2 on their last four world tours. She co-directed “U2 3D,” a documentary of the band’s live performance in South America on their 2006 Vertigo tour. “U2 3D” is the first live-action feature-length 3D digital theatrical release…”

Throughout the meeting projected 3D presentation was a focus, and the screening of U2 3D being a key point. This also demonstrated that mix between arts and science is also blending. This work is an example of a much larger and (obviously) commercial exploitation of similar e-Dance type outcomes and interesting to compare.

      

The fifth research intensive took place between 4 – 8 August in Bedford.  This week-long laboratory included the following project participants: Helen Bailey, Michelle Bachler, Anje Le Blanc and  Andrew Rowley.  We were joined by Video Artist: Catherine Watling, and four Dance Artists: Catherine Bennett, River Carmalt, Amalia Garcia and James Hewison.  There were several aims for this lab: 1. To explore, in a distributed performance context, the software developments that have been made to Memetic and the AG environment for the manipulation of video/audio streams. 2. To integrate other technologies/applications  into this environment. 3. To initiate developments for Compendium. 4. Explore the compositional opportunities that the new developments offer. 

In terms of set-up this has been the most ambitious research intensive of the project.  We were working with four dancers distributed across two networked theatre spaces.  For ease we were working in the theatre auditorium and theatre studio at the University.  Both of which are housed within the same building.  This provided the opportunity for emergency dashes between the two locations if the technology failed.  We worked with three cameras in each space for live material and then pre-recorded material in both locations.  Video Artist Catherine Watling worked with us during the week to generate pre-recorded, edited, video material that could then be integrated into the live context.

1. Software developments in a distributed environment

The significant developments from my perspective here was firstly the the ability of the software to remember the layout of projected windows from one ‘scene’ to another.  This allowed for a much smoother working process and for the development of more complex compositional relationships between images .  The second significant development was the transparency of overlaid windows which allows for the creation of composite live imagery across multiple locations.

What’s really interesting is that the software development is at the stage where we are just beginning to think about user interface design.  During the week we looked at various applications to assess the usefulness of the interface or elements of the design for e-Dance.  We focused on Adobe Premiere, Isadora, Arkaos, Wizywyg and Powerpoint.  These all provided useful insights into where we might take the design. 

2. Integration of other technologies

We had limited opportunity to integrate other technology and applications during the week.  I think this is better left until the software under development is more robust and we have a clear understanding of its functionality.  We did however integrate the Acecad Digi Memo Pads into the live context as graphic tablets.  I was first introduced to these by  those involved in the JISC funded VRE2 VERA Project running concurrently with e-Dance.  This provided an interesting set of possibilities both in terms of operator interface and also the inclusion of the technology within the performance space to be used directly by the performers.

3. Begin Compendium development

The OU software development contribution to the project began in earnest with this intensive.  Michelle was present throughout the week, which gave her the opportunity to really immerse herself in the environment and gain some first-hand experience of the choreographic process and the kinds of working practices that have been adopted by the project team so far.

Throughout the week Michelle created Compendium maps for each day’s activity.  It became clear that the interface would currently militate against the choreographic process we are involved in.  So having someone dedicated to the documentation of the process was very useful.  It also gave Michelle first-hand experience of the problems.  The choreographic process is; studio-based, it is dialogic in terms of the construction of material between choreographer and dancers, it involves the articulation of ideas that are at the edges of verbal communication and judgements are tacitly made and understood.  Michelle’s immediate response to this context was to begin to explore voice-to-text software as a means of mitigating some of these issues.

The maps that were generated during the week are really interesting in that they have already captured thinking, dialogue and decision-making within the creative process that would previously have been lost.   The maps also immediately raise the question about authorship and perspective.  The maps from the intensive had a single author, they were not collaboratively constructed, they represent a single perspective on a collaborative process.  So over the next few months it will be interesting to explore the role/function of collaboration in terms of mapping the process – whether what we should aim for is  a poly-vocal single map that takes account of multiple perspectives or an interconnected series of individual authored maps will need to be considered.

4. Compositional developments

Probably influenced by the recent trip to ISEA (the week before!), the creative or thematic focus for the laboratory was concerned with spatio-temporal structure again but specifically location.  I began with a series of key terms that clustered around this central idea – they were; dislocate, relocate, situate, resituate, trace, map.  A series of generative tasks were developed that would result in either live/projected material or pre-recorded video material.  This material was then organised or rather formed the basis of the following sections of work-in-progress material:

Situate/Resituate Duet (Created in collaboration with and performed by River Carmalt and Amalia Garcia) (approx. 5 minutes)

                                                                

The duet was constructed around the idea of ‘situating’ either part of the body or the whole body, then resituating that material either spatially, or onto another body part or onto the other dancer’s body.  We used an overhead camera to stream this material and then project it live in real-time to scale on the wall of the performance space.  This performed an ambigious mapping of the space. 

                                                         

The Situate/Resituate Duet developed through the integration of the Digi Memo Pads into the performance context.  James and Catherine were seated at a down-stage table where they could be seen drawing.  The drawing was projected as a semi-transparent window over the video stream.  This allowed them to directly ‘annotate’ or graphically interact with the virtual performers. 

    

Auto(bio/geo) graphy (Created in collaboration with and performed by Catherine Bennett, River Carmalt, Amalia Garcia, James Hewison.  Filmed and edited by Catherine Watling) (4 x approx. 2-minute video works)

In this task we integrated Googlemaps into the performance context as both content and scenography.  We used a laptop connected to Googlemap on stage and simultaneously projected  te website onto the back wall of the performance space.   The large projection of the satellite image of the earth resituated the performer into a extra-terrestrial position. 

              

Each performer was asked to navigate Googlemap to track their own movements in the chronological order of their lives.  They could be as selective as they wished whilst maintaining chronological order.  This generated a narrativised mapping of space/time.  This task resulted in a series of edited 2-minute films of each performer mapping their chronological movements around the globe.  Although we didn’t have the time within the week to utilise these films vis-a-vis live performance, we have some very clear ideas for their subesquent integration at the next intensive. 

               

Dislocate/Relocate: Composite Bodies (Created in collaboration with and performed by Catherine Bennett, River Carmalt, Amalia Garcia, James Hewison) (approx. 4-minutes)

This distributed quartet was constructed across two locations.  The ideas of dislocation and relocation through a fragmentation of the four performers’ bodies and then reconstituting two composite virtual bodies from fragments of the four live performers. 

          

We began with the idea of attempting to ‘reconstruct’ a coherent singluar body from the dislocated/relocated bodily fragments.  Then we went on to explore the radical juxtaposition created by each fragmentary body part moving in isolation.

                 

                           

“From Here I Can See…” (A Distributed Monologue) (Created in collaboration with and performed by Catherine Bennett, River Carmalt, James Hewison) (approx. 5 -minutes)

This distributed trio  was initially focused on the construction of a list-based monologue in which the sentence “From here I can see…” was completed in a list form that functioned through  a series of associative relationships.

 In one location Catherine delivered a verbal monologue, another dancer performed a micro-gestural solo with his feet within the same location as Catherine.  A third non-co-located dancer  in the other space performed a floor based solo. 

Trace Duet (Created in collaboration with and performed by Catherine Bennett and James Hewison)

In this duet we focused on bringing togther the new software capability to produce layered transparency of windows and the use of the Acecad Digi Memo Pad as graphics tablet.  We also worked with  a combination of handheld and fixed camera positions.

 

Sita and I went to Singapore at the end of July to present the project paper titled “e-Dance: Relocating choreographic practice as a new modality for performance and documentation”.  The conference took place at venues throughout Singapore including the Singapore Management University, National University of Technology, Nang Yang University and Lasalle College of the Arts.

  

We both had a really interesting time.  There were some great presentations.  The key-notes by Ken Mogi on Creativity and the Brain and Lev Manovich on Visualising Culture were both particularly relevant to the project. Ken Mogi constructed a view of creativity from the perspective of Evolutionary Psychology.  He drew parallels between memory and creativity in terms of brain function.  He talked about the ‘feeling of knowing’ as referred to by Psychologists and Neuroscientists. This seems particularly relevant in relation to Simon’s previous post. 

There was a juried exhibition as part of the symposium and various shows that included a range of installation art work, the majority of which focused on interactivity. 

  

The most compelling performance experience was “True” a Japanese dance-theatre work created collaboratively by an interdisciplinary team of artists and technologists (some of which were members of Dumb Type).  The integration of technology into the live work was so subtlely managed, it was a visual, visceral, auditory treat.

 

Martin and I, together with Dance Artist James Hewison, who has contributed to several of the research intensives, presented a paper at EVA in London in July.  The presentation titled “Choreographic Morphologies”, outlines the key findings from the Morphologies practice-led research.  The conference proceedings have been published by British Society of Computing.

Last week, I attended the fifth eDance intensive in Bedford.  Upon arriving, we discovered that we were to put on a public performance on Friday (unfortunatly after I had to leave).  This increased the pressure somewhat, although our confidence in the recording software had recently improved after it had been successfully used to record the Research Methods Festival “What Is” sessions in Oxford over 4 days without incident.  To add further to this task, we were also to set up two separate spaces for this performance, with three cameras in each and audio conferencing between the spaces, and we hadn’t brought any echo-cancelling microphones and only powerful laptop!  We immediatly got on with the task of setting up the two spaces. 

One of the aspects of eDance that is different from Access Grid is the requirement of high-quality video at high frame-rates.  Access Grid has historically used H.261 video, which is transmitted at a resolution of 352 x 288, and tends to reach about 20 frames per second.  For eDance (and the related CREW project), we decided to use miniDV camcorders for the cameras using the firewire interface.  This allows 25 fps video at 720 x 576, 4 times the quality of regular Access Grid video.  Thankfully, AG is easily extensible, which allow us to make this change without much effort.  The main issue is however with the laptop processing load.  We initially thought that our tool was not very good in terms of CPU loading, and assumed that this was because we are using Java.  However, we tried to use Vic, the video tool that AG uses, and found that with the same resolution and frame-rate, this also tends to result in high CPU usage.  The main advantage that vic has over our tool is that it allows you to control the bandwidth and frame-rate of the transmitted video.  Fortunatly, the piece that Helen was composing did not require us to use all of our video streams simultanously, so we managed to get away without too much overloading (although the fans on the laptops were running full steam ahead)!

The first of the successful uses of the videos between the two spaces was a piece where the video from the top half of one person was displayed over the video of the bottom half of another.  The two dancers then had to coordinate their movements so that they appeared to be one person.  This produced some wonderful images, as well as some good comedy moments!

Our next issue came with a misunderstanding about what our software could do.  We had bragged about how we could now have transparent windows.  We also tested to see if the video underneath would also show through the one on top (rather than drawing a pink area as had happened before).  This lead Helen to start planning a piece that could use this functionality (it looked like it could be really cool – overlaying people onto a model of a village)!  Unfortunalty, the way the technology was working was to capture the scene behind the transparent video, and then statically mix this in.  This meant that the image behind the transparent window could not move.  I set to fixing this problem straight away (I had already thought about how this might work previously), but by the time I had made it work, Helen had moved away to something else.

 During the course of the week, there was a small saga related to the ordering of some drawing pads to arrive by next-day delivery on the Monday.  When these did arrive on the Thursday, Helen immediatly devised a new idea involving overlaying the drawings done on the devices over the video.  As I now had the transparent video working, we immediatly got this working by having the transparent video window on top of the drawings.  Michelle also worked out how to give this a nice black background with a light grey pen colour, so this looked good behind the video windows.

The second successful use of the software was then the drawing on the overlayed transparent video windows.  The dancers moved around the space, and then when the drawing pad operator said “Stop”, they froze while the operator drew around their shapes (and also sometimes the shape of their shadow).  This resulted some quite complex images!

 We also realised on the Thursday that we had still not got the audio working between the spaces.  As we did not have echo-cancelling equipment, our efforts using the audio in the software tended to result in loud noises being played into the spaces!  We decided that we should therefore use some software that had echo-cancelling built in, such as Skype or MSN Messenger.  Skype turned out not to work too well as it had not been tested on the Bedford campus.  Although this was resolved before the end of the week, we decided to stick with MSN Messenger as it seemed to work fine.  This meant that the audio was not recorded, but did mean that it didn’t echo!

Another limitation of the software was revealed when we were asked to play back some recorded media.  During the course of the week, we were joined by a video specialist who had recorded the performers using Google Maps (which looked very impressive) and also whilst out walking in a forest (which gave us some much-needed time to get the transparencies working, and to look at other software – see later).  We then had to ask her to write this back to DV tape to be imported into our software, as we couldn’t yet import avi or mpeg files.  We also realised that we would have to launch a separate instance of the software to play back each of the clips quickly, otherwise the audience would have to wait while we loaded each clip up.  With the CPU load high from receiving video, we had to add an option for these extra instances not to receive the transmitted videos, otherwise the computer would quickly become overloaded.  Anja did this and this worked well.

Finally, one important outcome from the intensive was us all finally working out how the interface might look from the video perspective.  We worked out that we had over-complicated the interface in terms of representing video timelines, and that it might be best if the interface looked a little bit like Microsoft PowerPoint (or other presentation editing software).  This would obviously have major differences, such as being able to edit what was seen on the live screen without exiting what would be Presentation Mode in PowerPoint.  We were also shown the user interfaces to other tools used during and before performances, such as Arkaos for “VJing” (like DJing but with video).  I have since developed a mockup (in PowerPoint funnily enough), for this interface, which takes some features from each of these tools, and adds some of our own.

In the e-Dance research proposal we opened as follows:

A key problem in the academic field of dance is how to capture and document the incremental development of ideas and their material manifestation in the creative process within practice-led research. In improvisational, embodied investigation, the mode of engagement is generative, pre-verbal, intuitive, experiential and fluid. This militates against types of cognitive engagement necessary for analysis, critique and reflection. The problem is most acute in the context of dance: however it is pertinent to all arts-based disciplines. This project is predicated on dialogic processes between dance and e-Science and the fluidity of concepts as they transverse the two domains, making use of recent advances in the visualisation and representation of spatio-temporal structures and discourse.

Drilling down into the theme of pre-linguistic annotation/capture of choreographic design process, we wrote:

The visual nature of the interface employed by these e-Science documentation tools enables exploration of possibilities for pre-linguistic, multi-layered, non-linear representations of process, more aligned with the characteristics of the creative process itself. This reiteration of process will be considered not just as a static archival document of the process, but as a dynamic source of material that can be redeployed, for example, as a site for forensic, archaeological investigation; a score for further developmental commentaries generated through practice; or pre-recorded audio/visual content for re-use in hybrid distributed performance.

As the Open U. software development effort comes on stream in August, and with our Research Intensive in Bedford only a week or so away, we are now thinking about the kinds of user interface that choreographers want to scaffold, capture, provoke, reflect back their creative process. The role of the visual, and perhaps non-verbal human-computer interaction, are to the fore, as we take seriously the pre-linguistic nature of the choreographic design process. We have learnt from Helen and Sita (our choreographers) and from our partial immersion in the choreographic process, that they will ‘just know’ when they see something ‘right’ unfolding before them on the dance floor: a moment that they want to capture without getting verbal, to be revisited and replayed later for reflection and further rehearsal/research.

With these considerations in mind, it seemed providential when in my email I received an article by Guy Claxton:

Claxton, G. (2006). Thinking at the Edge. Cambridge Journal of Education, Vol. 36, No. 3, pp. 351–362

Claxton, whose work on “building learning power” is influential in the UK education system, has a specific interest in how we teach school pupils what it feels like to learn (not always pleasant — “what to do when you don’t know what to do”) — so is of interest to me with other hats in at the Open U. and as a school governor. But this article tries to put language and concepts on how we conceive, and nurture, creativity in general. As I wrote about elsewhere, his interest is in how we work with, not fear or fight, that creative period when things remain inchoate, ill-formed and unsettling:

…there is a softer, slower kind of groping for a way of articulating something that is currently, tantalizingly, beyond our linguistic grasp… It is these gentle, generic, but often unsung, aspects of the creative process that I want to describe.

Intruigingly he uses the particular example of choreographers, as he writes about:

…the intuitive feeling of rightness (or wrongness) that guides the process. This sense of rightness—the same immediate, unjustifiable feeling of ‘Yes, that’s it’ that guides the process of focusing—seems to be essential to the kind of creativity I am exploring. A choreographer may not know what it is she is looking for until she tries out one more move and gets the ‘Yes, that’s it’ response. As I write, I too do not know quite what I want to say, nor precisely where the discussion is heading, until a sentence hits the screen and I get the ‘Yes, that’s it’ feeling. Even research scientists must learn to recognize and trust that intuition.

Inspired by the 1960s work of Eugene Gendlin at the University of Chicago on the pre-conceptual period of creativity and cognitive change, and continued by his associates [Focusing Institute], Claxton writes:

Gendlin’s point is this: that the kind of personal problem-solving that happens in counseling and psychotherapy usually requires slow, delicate, hazy knowing, rather than quick, conceptual, conscious knowing. The main problem is often not one of ‘deciding what to do’, but of coming to understand more precisely what it is that makes things appear problematic to us in the particular way they do. And that kind of knowing requires us to be able to go ‘back to basics’; to start with the complicated, murky ‘felt sense’ of the predicament, and then slow down the process of epistemic evolution so we can (a) see more clearly what kinds of assumptions we might, unconsciously, have been stirring in to the conceptual mix; and (b) allow the emerging conceptualization to be driven, and continually refined, by the felt sense itself, rather than forcing it to fit the pre-existing templates, categories, habits and phrases that we happen to have installed in the skill-scape and the word-scape. Focusing enables people to regain some flexibility, with regard to their ‘ways of knowing’. In particular, it helps them relearn the kind of patient, open-minded attentiveness that allows them to conceptualize their predicament in a way that is fresher, more differentiated, and more accurate, in the sense of being closer to the complex reality of their own embodied knowing.

So, all this is quite intruiging. Embodied, pre-verbal cognition looks to be an interesting seam to mine in the coming months as we seek to build language that bridges cognitive interface design and dance, and concretely, to adapt our visual hypermedia tools (Compendium and Cohere) to support the construction of resources for choreographic, practice-based research.

I’m going to be doing an on-line interview with Torsten Reimer on the www.arts-humanities.net website.  It will start some time towards the end of May.

I did an interview with Rachel Gibson today.  She has been commissioned by the Arts Council of England to carry out a survey of digital technology and new media in dance.  The aim of the survey is to create an overview of digital capacity in the dance sector.  It was really interesting to discuss where the e-Dance Project is at the moment and where the research is going to take us in the next few months and into next year.  It was particularly interesting to think about this project and its relationship to the professional dance/arts sector.  There is a real challenge for us to engage with that community and dissemnate what we are doing effectively.