Sem2 Week1-3


After having a good understanding of TouchDesigner and after doing demos for the Siggraph Asia, I am able to embark on the AR Tangible Tabletop Music project.
The AR Tangible Tabletop Music (ARTTM) will cover technical aspects of building such a system and a few engineering analysis covering design and method. Awesome! =)

Architecture of ARTTM

- A set of Markers will be identified and tracked by the ARtoolKit through the webcam. The coordinates (position and rotation) of the Markers will then be send to TouchDesigner through Shared Memory.
The set of Markers act as INPUTS to the ARTTM. As I am dealing with the theme of music, I have also connected the Wiimote (fantastic tool as it sends out roll and pitch readings) and the WiiDrum to act as INPUTS to TouchDesigner.
After getting readings from the INPUTS (Markers or Wii instruments), the readings will be processed in TouchDesigner and will display Augmented Reality objects on top of the Markers and the music/sound will be processed in Pure Data.

Elements of ARTTM
1. Webcam
2. Dynamic Play Area
    2.1 Initialization (User Interface)
    2.2 Design Constrains
3. Set of Markers
    3.1 Volume Control - Markers 1-7
    3.2 Status of Markers
    3.3 Store Data - Marker 8
    3.4 Delete Stored Data - Marker 9
4. Wiimote / WiiDrums (optional Inputs)

1. Webcam
The ARTTM is aimed at the masses and as such, they would probably own a normal webcam with resolution of only up to 640x480. As such, for this project, I would be working in the 640x480 resolution.

2. Dynamic Play Area

The idea of a tangible tabletop music is for people to compose and play music anytime and anywhere given an empty space area. As such, there is a need for a dynamic play area. In week 2 - week 3, I was able to design a dynamic play area with an intuitive Initialization User Interface. The user of the ARTTM will visualize his play area (of a rectangle/square shape) but ensure that the play area is within range and limited by his webcam. He can then proceed to define this play area using the Initialization User Interface to mark the corners of his play area and finally set the number of  cubes for the x and y -axis.(One can think of it as a rectangular/square area with grids).

The pseudo code and alogrithm for realizing this dynamic play area:
  • Get coordinates of top right corner.
  • Get coordinates of top left corner.
  • Get coordinates of bottom right corner.
  • Store these coordinates.
  • Set custom no of cubes along the x-axis with respect to the stored coordinates.
  • Set custom no of cubes along the y-axis with respect to the stored coordinates.


I then translated this algorithm into the TouchDesigner language which consists of operators and scripts and packed all the operators & script to a pop-up user interface.

2.1 Initialization (User Interface)

- When a user press F1, he will invoke a pop-up of the Initialization User Interface.

- This is the Initialization User Interface which was polished further in week 4 which added a safety lock feature. Notice:
- Initially how the user will set up his play area.
- The user will not be able to press the buttons if the Hiro Marker goes out of range.
- The sliders to set the no of cubes along the x and y-axis to create the play area grid.
- Once the Initialization User Interface is locked, the buttons and sliders are not responsive anymore.

Currently, with a 640x480 resolution and the set of markers (see 3. Set of Markers), the best fit that the webcam can capture is area of 8x8 cube-grid.

2.2 Design Constrains
There are a particular design constrains to the dynamic play area that one can consider.
- Consider a 4x4 cube-grid of area 16cm2.
- Consider the y-axis as pitch and the x-axis as tone of a music score.
- For simplicity, take a normal at the center of the marker and assuming that the camera tracks the normal of the marker.
- In a perfect-fitting play area, we can get 4 pitch and 4 tones but we are wasting a lot of space. (in red)
- Now, consider a 4x8 cube-grid of the same area 16cm2.
- We can now get a more tones (8 tones), however, still within a row only 4 cubes can be fitted perfectly adjacent to each other.
- As such we have to sacrifice by increasing or decreasing a pitch to get all 8 tones or we can simply set the cube-grid to be 2x8 to get all 8 tones of the same pitch.
- The dynamic play area  becomes a give-and-take situation when considering the set-up of the cube-grid.

3. Set of Markers

       1             2              3            4             5              6             7              8            9
I went on to edit a bit of the AR-toolkit code and retrained a few markers for the ARTTM.
For convenience, I are going to mark these markers as Markers 1-9.
Currently, I am using these set of Markers that is of 4.15cm by 4.15cm mounted on a 4.5cm3 styrofoam cube (as seen in the youtube videos). The purpose of using styrofoam cubes is so that users can have a good feel and touch of the markers.
For now, Markers 1-7 will act as playable markers while Markers 8 & 9 are special markers with special functions which will be explained further on later.

3.1 Volume Control - Markers 1-7
One of the design idea for the ARTTM is for each individual playable Markers to be able to control the volume by turning either Clockwise or Counter-Clockwise.
- Clockwise = increase in volume
- Counter-Clockwise = decrease in volume

Initially, I had a few problems doing this as i did not know how to approach this problem. However, after a few paper scribbles and design testing, I came up with a monstrous excessive design that had lots of operators.
I was not too happy with the design and after having a good cold bottle of coke and a few tips from Mr Rodney, I sat down and started writing a better algorithm.

The pseudo code and alogrithm for realizing the volume control:
  • Map the whole rotation (360 degrees) to a number 0-100. 0 being at 0 degrees and 100 at 359 degrees.
  • If there is a rotation movement of the marker, read 2 samples, which are the previous number and the next number.
  • Get the gradient of the 2 samples.
  • A negative gradient would imply a -1 in count while a positive gradient would imply a +1 in count.
  • Limit the count to 0-100 for the volume.
  • Even when the marker is taken out of the play area and introduced again later, the volume is kept in memory.
Again, I then translated this algorithm into the TouchDesigner language.
Eventually, I got a more tighter design that uses less operators.
After playing around with the volume control, there was a small design flaw which i overlooked. Each time I introduced a new Marker into the play area, the volume starts at 0.
However, this design flaw was easily corrected and now, if a new marker is introduced, it will start at a volume of 50 (which actually implies 50%). The min and max of the volume still remains at 0 and 100.

- This is the volume control. Notice:
- Clockwise movement increases the volume.
- Counter-Clockwise movement decreases the volume.
- Volume was capped at 100 even after more CW rotations.

3.2 Status of Markers
After defining the play area using the Initialization Interface, we need to know the coordinates of the Markers on the x and y axis.
At this current stage, I have used coordinates 0,0 to indicate if a marker is not in used in the play area.

The pseudo code and alogrithm for realizing the Marker Status:
  • Use the stored coordinates during the initialization.
  • Track the marker and do a math operation with respect to the stored coordinates and the set cube-grid.
  • Do both x and y axis.
- I design the x and y axis tracking according to the algorithm and but the different axis in different containers as with the volume control.

I then created a pop-up Marker Status in which the user can invoke by pressing F2 as shown below.
- As observed, only Markers 1 -2 are active in the play area.

- This is the Markers Status which was polished further in week 4.
- The play area was set to 7x8 cube-grid.

3.3 Store Data - Marker 8
Marker 8, upon placing it into the play area will used to store the coordinates of the other markers. And thus, the other Markers can then be moved around again.

- Look at the status of Markers 1 -7
- Upon introduction of Marker 8 into the play area, the coordinates are stored.
- Markers 1 -7 can now be placed elsewhere as well as Marker 8 and the previous stored coordinates still exist..

3.4 Delete Stored Data - Marker 9
Marker 9 acts as a tool to delete the stored coordinates in Marker 8.
I created a fire geometry on top Marker 9 to indicate its function. (Fire burns objects).

The pseudo code and alogrithm for realizing the Delete Stored Data Function:
  • Find distance between Marker 8 and Marker 9.
  • If distance between the two Markers are small, delete the stored data in Marker 8.
While the algorithm seems pretty easy to be implemented, however, it is a bit more tricky in TouchDesigner.

- Stored Data is erased upon close proximity with Marker 9.
- Once the Stored Data is erased, Marker 8 is utilized again.

4. Wiimote / WiiDrums (optional Inputs)
Since I was dealing with the theme of  AR, music and tangible devices, I believe that the Wiimote would be a great tangible device as it gives roll and pitch readings and on the remote itself, it has 11 buttons which translates to having a lot of options to "play" around with. The task ahead was to connect the Wiimote to communicate with TouchDesigner. After doing my research, I found out that the best way of doing this was to use GlovePie (think GlovePie as a compiler and it has its own language). Thus after getting my hands dirty brewing my own script in GlovePie, I was able to get the Wiimote to communicate in TouchDesigner! Awesome!
And since I was already on the topic of getting the Wiimote communicating with TouchDesigner, I just carried on scripting and eventually I was even able to communicate both the WiiDrums and the Wiimote with TouchDesigner. And suddenly,  ARTTM just got exponentially interesting!!

- This is Wiimote communicating with TouchDesigner. Left Panel shows me playing around with the Wiimote and Right Panel shows a container in TouchDesigner which is outputting the Wiimote signals.

- screenshot when the  WiiDrums is communicating with TouchDesigner.

- screenshot when both WiiDrums and Wiimote are communicating with TouchDesigner.