HomePage

This is the Wiki home of Timothy Twillman, Josh Berry and Amanda Bernsohn's Intro to Physical Computing midterm at ITP



Observations

We did two sets of observations for this project. Initially we were looking at various ways that player approached and played the Marimba. As the project developed, we transitioned more to a model of straight rhythm training. We investigated ways in which musicians keep time while playing. Many choose to practice along with a metronome - this is true not just for drums, but for piano, guitar etc...

Here are a few videos where musicians are using the metronome technique:
http://www.youtube.com/v/idNca-91LyY&rel=1
http://youtube.com/watch?v=Z8FbMylPUTc
http://youtube.com/watch?v=RKtQNkoPqdk



Marimba
user starts standing very straight. head bent only slightly to look at
the instrument, which is just under waist height. facing long side of instrument. arms are held close to the body; one hand is holding two strikers, the other just one. User looks back and forth along the instrument at notes he is about to play. Sometimes lifts arm with two strikers up, elbow out, to reach the angle he wants. sometimes bends just a little at the waist for reach. Feet move slightly, to move back and forth along instrument to reach different notes. stance is generally fairly closed, though balance is kept well. attention split between arms, and looking for next note. this is a performance so it's certainly restrained; notes are played slowly, often with intervals long as .75 seconds. Intstrument is restricted to basic (though pretty) sounds. face looks serious the whole time.

user starts with back slightly bent, head tilted down facing long side of instrument ; two strikers in each hand. shifts body from foot to foot as he reaches for notes a little farther than very easily reachabl
e with arms close to body. soon he starts bouncing as he plays, reaching a little farther back and forth. feet are a little over shoulder width apart, and widen as his bouncing & moving increase. back is always held close to straight, but with a very very slight stoop. attention focused on the instrument, back and forth between hands as he plays. Instrument is about 10-12 feet long, two pads deep from perspective of user, thoug he appears to only play one layer. Notes played at a modest pace. Face serious.

User very bent over instrument, facing long side, with body moving up and down head quite bent. Playing of notes is very fast. Feet are pretty close together, and move apparently more with pace & body bouncing than to reach notes (though at one time movements do serve to move to different note area). Body is flat towards instrument most of the time but at some points turns to different angle. instrument appears to be about 8-10 feet long, only one level is clear but two may be on the instrument; user only plays one though. User is smiling and looks to be having fun.

(beginning users) Users stand straight, feet close together but not touching, neck slightly bent to look at instrument.. one striker in each hand, hands move together, always same distance apart & strike at the same time. Some swaying as different notes are reached. Most have fairly slow pace. One user does a solo while facing short side of instrument and slightly bending over & reaching for notes. Users look bored. focus appears to be on next note; focus always on area strikers are operating in. Faces serious.

User begins facing long side of instrument, back very straight, neck bent, but soon bends over slightly and then more so his head is over the instrument. Two strikers in each hand (hand closed) one betwen thumb and index finger and the other between the pinky and ring finger. strikers in each hand remain same distance apart throughout observation, though angle is changes by reaching elbow out to play different combinations of notes. Instrument has two layers (one towards user the other slightly higher, farther from the user). Occasionally user will switch between the layers. Some walking up and down instrument to play different sections of musical piece. back straightens sometimes, bends a little again. during slower parts of piece, back tends to be straighter. neck always bent to look at instrument. attention appears focused on what each hand is doing. face very serious, though gives some indication of pride at one point (slight smile, though still looks serious). top of body tilts back & forth quite a lot during fast passages.

User starts with back very straight, neck slightly bent to look down at instrument. Two strikers in each hand, between thumb/forefinger and between pinky/third finger; the strikers are about 10 inches apart at striking end. At beginning user scans back & forth along instrument before playing. focus is generally in area immediately being played; obvious switches of focus only when a far note needs to be reached and user stretches out arm and has to look for note to play. body rocks back & forth a little to reach notes. arms generally close to body, but sometimes reach out quite far. user squats slightly when reaching notes that require a distant reach. strikers generally kept about 3-4 inches above surface of instrument when not striking, though for some notes are brought to a little over a foot above the instrument.

User starts with one striker in each hand, held in fist, body facing long side of instrument, held at about a 45 degree angle with left foot forward, feet slightly over shoulder width apart. back is straight, neck bent at 30 degree angle. Arms are kept close to the body; body pivots back and forth and user walks briskly back & forth along instrument as he plays (normally left hand plays farther row of notes, right the closer but sometimes both play the near row of notes). Pace of music is very quick Users focus normally on area of instrument being played, but looks up at one point. Face very serious; obviously focusing hard.

User begins with two strikers in each hand; one held from center of palm, under thumb (which is held parallel to top of instrument) and the other between index & second finger. One set of strikers appears to be about a foot apart, the other about 6 inches. Back is very straight, neck bent. Neck bobs slightly during playing, though user doesn't move back & forth. Arms close to body. Face looks mildly amused (very mildly) but very focused. Focus on area being played, likely split between hands. Pace is moderately fast.

User starts with feet shoulder's width apart, back straight, neck slightly bent. As he plays, neck becomes more & less bent, and stoops appreciably at times. head bobs back and forth. arms generally close to body, but sometimes elbows go out to sides. some rocking sideways back & fort, a little walking left & right along instrument, always facing the instrument. instrument is about 8-9 feet long. two strikers in each hand, can't make out which fingers they're between. user looks very serious & focused, though happy on completion. Motion comes mostly from shoulders & not as much from trunk, solid balance kept during playing. Pace moderately quick. gaze on area being played, though user seems to be focused inside & movement more through body awareness than sight. user sometimes looks up. instrument is 2 layers deep though mostly the one closest to the user is played.

User begins facing long side of instrument, one striker in each hand, back straight, neck slightly bent. through playing arms remain close to body; body moves left & right rather than arms being reached out. face serious/very focused. movement mostly from elbows. walks left & right along instrument to play different notes. gaze on area being played, but eyes seem to indicate awareness of what's being played mostly coming from body awareness not sight. User then, for second piece, puts two strikers in each hand; one between thumb/forefinger and one beween pinky & third finger — other ends of strikers are about 6" apart though shift to about 8 or 10 inches apart at some points; not clear if this is intentional (likely). arms at this point a little more flared. strikers stay mostly within about 4" of instrument but sometimes, for louder notes, come up to about 8 inches from surface of instrument. playing is always on row of notes closest to the user.

User begins with two strikers in each hand, one btw thumb & forfinger one btw pinky & third finger. stooped a bit over instrument; neck mostly straight from body. arms reaching, body rocks back and forth quite a bit, stepping left and right. gaze back and forth along instrument; movement like a dance, though doesn't always appear as balanced as most other users. gaze somewhat back & forth between hands. user uses both layers of notes pretty much equally.

user begins slightly slumped (back straight to shoulders; head bent down at shoulders rather than neck). two strikers in each hand, standard grip (btw thumb/forefinger and pinky/ring finger). arms mostly held close to body, most movement from elbows, a little from shoulders. gaze restricted to small area around area being played. strikers in left hand come up to about 6" above instrument; right hand about 8". Hand movement is about 2-3 inches up and down. Pace moderately fast. User seems very focused.

(PVC marimbas) Users using two paddles; they appear to be flip flop soles on sticks. Held sticking out from fist. bodies mostly straight, though slightly bent at the waist, heads bent over quite a bit. most movement leaning left & right from legs/waist (though feet move only a few inches). Faces serious/focuesd. Paddle height different among users; some lift them up to head level, some just chest level. Some rotation left & right to reach different notes. much of sound comes from slap of paddles as well as sounding from air in vertical PVC pipes.

User has straight back, neck bent forward at shoulder, top of head lifted. standard 2-striker hold. most motion from elbows, a little from shoulders. hands move very little; about an inch up and down.

User has very stiff posture; head slightly bent at shoulders. two strikers in each hand; ends vary between about 2" and about 8" apart as notes played. feet slightly under shoulder width apart, small steps occationally, feet brought together then moved back apart. back kept straight entire time. most movement from shoulder; hands move up and down about 4-6" depending on note loudness; strikers move at peak from about 2" to about 14" from surface of instrument. occasionally head dips down when a distant note is reached, or when angle of elbow must be quickly shifted. gaze is back & forth along instrument, around where next note will be played.

User stands, back slightly bent, against long side of instrument. head dipped a lot at shoulders (looks uncomfortable), 2-striker grip with one held in palm with fingers wrapped around it, under thumb; the other between index & second finger. looks at where notes being played; when readhing far, looks at area he will be playing. legs close together. plays both levels of notes (front & back). at some points focus appears to narrow greatly on exact notes being played. head turns left & right; body shifts from foot to foot, angle between strikers changes a lot during playing. when playing faster parts, dashes back & forth. most movement from elbows, some shoulders (flaring out elbows) to change sideways angle of strikers relative to instrument.

User starts with back very straight, neck bent just a tiny bit, feet shoulder width apart. two-striker standard hold. bends slightly at waist to reach second layer of notes, shifts back & forth on feet, steps about a foot left & right. sometimes legs open up to about 2 x shoulder width. sometimes one foot in front of other when reaching second row. pace is moderately fast. hands move up and down about 4"; striker ends generally move up and down with peak from about 2" above instrument surface to about a foot above. some head bobbing, but back remains quite straight. some reaching from shoulders but most motion from elbows.

User holding 2 strikers per hand in under-thumb/btw index and second finger grip. back straight, feet slightly apart. user rocks left & right, and occasionally steps about 7-8" left & right to play different areas of instrument. stays on first row of notes. left hand distant striker ends about 8" apart, right hand change from about 4" apart to 8" apart. pace relatively slow.

User has straight back (though slight bend at hips), neck almost straight. elbows bent at between 80&110 degrees (though sometimes bend to about 45). motion from mixture of shoulders & elbows; during some times, shoulders give up to about 75% of movement, others it's all elbows.body bouncing up and down to beat. arms mostly held close to body, though elbows flare here & there. feet about shoulder width apart, some lifting of front & back of foot at times. some stepping left & right, about 5-6". one striker per hand. sometimes shrugs a bit, almost as if trying to give music extra oomph from it. very focused, seemingly on body movements more than visual. serious face.

neck leans over from shoulders, slight bend at waist. three strikers in each hand; mixture of two common grips (one below thumb, one btw first & second finger, one btw third finger & pinky). head bounces with music. also some kind of percussion on leg; playing by alternately lifting feet. gaze shifts left & right along instrument. independently controlling striker held under thumb from other two (moving it left & right to change note span) though the other two seemed fixed in relation to each other.

---

Additional Observations:

The action that we Josh, Tymm and I have chosen is striking an instrument with a mallet to produce sound. We also wish to incorporate a learning environment (deciding between melody and rhythm training). The medium being changed is the sound itself.

The action is taken on the instrument itself, which will be roughly modeled on a marimba or other idiophone. It will be a table-top instrument with a smooth surface for striking. A striker, or mallet, will be held in each hand.

The goal of the activity in this case will be two-fold. The goals will be to produce music and to learn as well. We are still in the process of figuring out this aspect of the project. The ultimate goal (and ideally this will take place after learning) will be to produce music that is pleasing to the user and perhaps his/her audience.

The person stands in one place initially, with feet roughly shoulder width apart.. their hands are actively engaged with the instrument and holding a mallet in each hand. The head is cocked down at a 45 degree angle (roughly) and concentration is focused on both hands. As the playing begins and continues, the person engaged in the activity tends to shift his or her weight and increase movement in the feet.

Standing for long periods of time could be boring or uncomfortable for the feet. Additionally, focusing increased attention on the instrument, with one's head down, can cause cramping.

We are experimenting with placing sensors in both the strikers as well as the instrument being played. The angle of the mallets will alter the sound that is produced. The actions taken in playing this instrument mirror the characteristics of sound in that force produces increased volume and changes the sound (pitch?)

Often, players of these instruments will use more than one striker while they are playing to hit more than one note at a time.

user is using instrument for 1 minute
standing straight
eyes always down
pivoting at the waist for increased reach
occasionally takes one step to the right or the left
holding mallets so pinky finger is approximately two inches from the base
thumb is securing mallet so it doesn't fall

user is bent slightly forward from the waist
looking directly down at instrument from above

feet either shoulder width apart or in a “v” formation heel-to-heel

action is generally more from the wrist - pivot at the wrist with elbows tucked in at sides
sometimes, but less frequently, action is initiated at the elbow.

if playing with two mallets in one hand: mallets are held in a closed fist position, with one extending between the pointer and middle finger and the other extending between the ring and pinky fingers. the ends of the mallets are joined within the closed fist in the v formation.

user changes the angle at which the arm approaches the instrument to vary the distance between the two keys played.

---

Project Prototyping

The Mallet

Percussion board:


Accelerometer:


The whole controller:


After discussions with Amanda and Josh on how we were going to do the project, I worked on the first prototype of the hardware for the midterm. We wanted the project to be wireless, and felt that using an accelerometer would give more data on the motion of a mallet than most sensor types.

We decided on a glove- or wrist-based transmitter; we wouldn’t be able to get Arduino hardware & support small enough to fit in a mallet. But since mallet movement had to be free, we decided the mallet should be attached via flexible wire.

I put the accelerometer (ADXL320 that Analog Devices was kind enough to sample to me some time ago) at the top of the mallet so it would be most sensitive; depending on the results of our tests we have the option to move it elsewhere on the body of the mallet. I made the boards very small so they wouldn’t be in the way. The etching process with these went very smoothly with no bad transfers (I use Press n Peel Blue, which has generally given me good results, though occasionally a transfer will just completely fail and often there are lines which must be filled in with Sharpie markers). This was also my first use of Liquid Tin, which smells a little like canned tuna but does a really nice job of tinning boards. I should probably check out the MSDS on this stuff though, something tells me it’s a bit of a nasty chemical.

I decided to use Xbee radios since I’m familiar with them from previous personal projects, and they are easy to set up and transparent to use, and the Arduino Development Kit base USB interface allows an easy way to get data in to a computer. Due to size constraints, I elected to use an Arduino Stamp as a controller/processor. An added bonus to the choice of hardware is the ability to upload firmware to the Arduino Stamp transparently, though it is still necessary to reset the board manually — though this requires setting the full chain of Arduino-Xbee1-Xbee2-PC? to 19200 baud.

I decided to use one of the AA battery boost regulators as sold at the NYU Computer Store. Unfortunately they only carried the 5v version, however I was able to modify the board to work at about 3v based on a note at http://ace.uci.edu/index.php/wiki/Step-Up_DC-DC_Converter/ — I didn’t have a 182K resistor lying around, but I also only needed somewhere in the range of 2.8 to 3.4v — I found that by shifting the position of the current (~98K resistor) at RF2 and adding a 100K 0603 resistor in series, I was able to get just over 3.0v out from the regulator. I was using a regular soldering station, so it was a bit tricky to heat both ends of the original resistor to move it without damaging it.

I also added female header access to all of the digital and analog IO pins of the Arduino Stamp to allow access for additional prototyping. The main board has really simple traces; no problems going single sided. Etching on this one went pretty well too, though I had to do about 3 minor Sharpie cleanups. I also made a point to put a copper fill on the board, more for a little FeCl? conservation than because I thought I needed a good ground plane. I did make a mistake with an off-by-one error on hooking up the pins; it came from using a Basic Stamp component in Eagle, which doesn’t have analog in pins. Trace cutting & some jumper wire took care of the problem.

I added some foam rubber to the back so the board wouldn’t scratch skin. It’s hot glued on, though it makes it a little hard to make modifications; hopefully no more will be necessary.

So far the Arduino code just reads values from the accelerometer (which, by the way, seems to be decently calibrated out of the box), does some smoothing and sends them. I don’t feel this is optimal when we start getting the percussion processing going, as I believe changes in acceleration may occur faster than the serial communication can push bits. I think it will be necessary to do some front-end processing and detect & send messages on impacts (V = Integ(A) when A = 0?). Note: even with smoothing, output is very jumpy.



The Drum Pad

Here are some pics from one of our midterm prototyping sessions. Initially, we began our project as one which would be built upon the model of a marimba. We were playing with the ideas of both having an instrument and a learning tool (almost like a Simon) for memorizing melodies. Gradually, and after prototyping both ideas (we did not create one model, but rather opted for two) we decided to proceed with the learning tool as a method for learning basic rhythm. We had worked to develop both a drum-pad like device and a system contained withing a mallet, or striker. Tymm worked on the mallet and produced an instrument that produces ethereal beautiful sounds, but we are opting to go ahead with the learning idea and going with the drum pad idea. We all liked the idea of a tool that would encourage interaction and give feedback - though we are still in the prototyping stage, ideally we will go forward and create an instrument through this framework as well .. one that allows the user to improvise and be spontaneous. At this point we are focusing on the interface and the functionality..so far it's just really quarter notes, to be developed into eight and then sixteenth.

We started to develop the pad using piezos and experimented with a number of materials as housing, settling eventually on neoprene and part of an anti-static mat (the initial circuit and design was adapted from one at http://wtcs.ca/wiki/index.php/MIDI_Drum_Pads_Conclusion). The combination of the materials, housed in an empty candy tin, allowed for enough vibration for the piezo and gave good readings. Initially we had cut cork as a playing surface, but it was way too absorbent and killed our readings from the Arduino - they were totally uneven. Though we hope to develop a slightly more sophisticated final prototype for presentation of the project, we will be user testing with this design.


Our piezo hook up:


Readings were steady (reading everything above 50 in serial) when striking on the table:


Cork was way too absorbent:



Using neoprene proved a good alternative (tried it initially using my laptop bag). We bought a number of cheap mousepads and cut them to size. The readings from that point on were useful and steady.

We also adapted the design with ideas pulled from http://dazz.genmaygamers.com/DrumConstructCompress.pdf (adding score lines to the coffee can lid and attaching the piezo pickup via hot glue so it would flex with tiny up & down movements of the metal). A hole was also punched in the center of the lid with a centerpunch, to create a focus point for forces applied to the top, and
a rectifying diode removed from the input stage of the arduino to eliminate the extra voltage drop (now a 5.1v zener diode to ground in parallel with a 680K bleeder resistor is all the conditioning done for input).

Interface Prototype

At this point the software is tracking good and bad hits.

---

User Testing Results


Testing our prototype with users was helpful; we received constructive comments that will help us to improve user experience going forward. Below is some of the feedback that we received. Potential changes that we may decide to make are indicated in curly brackets.

Users used the drum system in one of two ways: a, to follow along with the rhythm training as prescribed and b, to improvise their own sounds using the sticks and drumpad.

There was one user who needed multiple explanations before he understood the goal of the program - suggesting that the interface needs clarification.

Within the group that was following along with the training, the users were fairly evenly split into three groups:
Those who looked at the screen and followed the green dots.
Those who turned their heads away from the screen and only used the sound to guide their hits.
Those who used both the screen and the sound.

Our assumption was that the user would use both interfaces in improving his/her rhythm. It came as somewhat of a surprise that there were users who chose only to listen, and not use the balls on the screen as a guide.

{Allow users to hide visual aspect of the program in favor of a sound-only format - eliminating potential distractions}
{Use external speakers from PC to amplify queue sound}

Additionally, there was some discussion around whether the queue dots should continue to scroll across the screen, or whether there should be a stationary dot in the center of the screen. Those who were in favor of maintaining the scrolling dots suggested that it was reminiscent of a music signature, and the linear method of reading music.
{Replace scrolling dots with one larger circle in the center which could, perhaps, pulse}
{Replace scrolling dots with mode in which pseudo sheet music is provided in their place}

There was a lot of discussion around the screen visuals

Good hits could be opaque, v. bad hits being slightly translucent. Opacity would have a relationship with how far from perfect the hits are. I.E. the farther the hit is from perfect, the less visible it becomes.
Bad hits should be smaller than good hits
Having the visual of the duration of helpful
One user suggested that visuals be incorporated into each hit. Wanted “more to happen on screen”
If hit is bad ball disappears off screen (similar to changing alpha)


The functionality of the software:
Bad hits should produce a bad sound. Users who suggested this also suggested a free-form mode in which this wouldn't be the case (in which one could just play with drum sounds triggered on strike)
Wanted to know how they were doing (averages)
Wanted tool to get more complex/challenging as user progressed
Wanted ability to record own rhythms and hear them played back


Hardware:
Make striking surface bigger
Additional drums
{Add bass and hi-hat}
Create delay from the piezo .. tweak system so only one beat can be hit at a time rather than the piezo vibration creating a bad hit on screen.
Eliminate drag
Set up so two hands can be used more easily (identify particular Left and Right keys on keyboard to encourage two handed use)

Most users who tested the prototype commented that it was 'cool', 'fun' or 'engaging'. The general consensus was that the users wanted more to happen as a result of the bad or good hits than just a color change on screen. For those who were not using the visuals (about 1/3) there would be no way for them to know that their hits were, in fact, misses.

There was also a general sense of wanting more options with regard to sound and difficulty level.

---