The Screen as a Control Surface

Week 2 begins: the phone becomes the instrument. Today — what makes an instrument playable, the craft of mapping gesture to sound, and by evening your first instrument is running on your own phone, from a URL.

10:00 — Weekend Ideas

First, an announcement: the show has its name — chosen from the A2 proposals. From today, the last Friday has a title, a poster wall, and an audience to build for.

Then: three instrument ideas each, on paper, two minutes per person. No feedback yet except clarifying questions — the vocabulary for critique arrives at 10:30, and concept lock isn't until tomorrow evening. Pin the sketches to the wall; they stay up all week.

10:30 — Playability, and the Craft of Mapping

An instrument is not an app. An app helps you complete a task; an instrument gives you a voice. The difference is made almost entirely by mapping — the wiring between gesture and sound. You have been mapping since Day 1: map(mouseX, 0, width, 100, 1000). Now it becomes design:

Five properties worth designing for — use them as a checklist against every idea on the wall:

1. Immediacy. Sound within ~50 ms of the gesture, or the instrument feels dead.
2. Legibility. A watcher can roughly see what causes what.
3. Repeatability. Same gesture, same result — else it's a toy, not an instrument.
4. Failure-tolerance. Wrong notes impossible (scale-locked), or gracefully wrong. A stranger plays it in 30 seconds — that's the A3 bar.
5. Depth. Something left to get better at. Easy to start ≠ quickly exhausted.

Tension to hold: legibility and depth pull against each other. The theremin is maximally legible and brutally deep. A step sequencer is shallow and instantly satisfying. Both are honorable; know which you're building.

11:45 — Instrument Studies

Four browser/phone instruments, played on the room's phones, critiqued with the new vocabulary — name each one's mappings before judging it:

• raga.fm generative raga engine Generative mapping: the player chooses constraints, the engine performs. Where does "playing" begin and end?
• Bloom Eno & Chilvers (2008) Tap = plant a note that loops and decays. One gesture, one-to-many consequences over minutes.
• Ocarina Ge Wang / Smule (2008) Blowing into the mic as breath control — the sensor nobody remembers the phone has. Tomorrow's subject.
• Borderlands Chris Carlson (2012) Granular sampling under multitouch — ten fingers, many-to-many mapping at its richest.

14:00 — Touch in Code

On phones, p5.js gives you touchStarted / touchMoved / touchEnded and a touches[] array (multitouch!). Two rules of the road: return false from touch handlers (stops the page scrolling/zooming), and start audio inside the first touch (browsers unlock sound only after a user gesture).

Build 1 · The pocket theremin

let osc, filt;

function setup() {
  createCanvas(windowWidth, windowHeight);   // fill the phone's screen
  osc  = new p5.Oscillator('sawtooth');
  filt = new p5.LowPass();
  osc.disconnect();                          // unwire the default route...
  osc.connect(filt);                         // ...and go through the filter
}

function touchStarted() {
  userStartAudio();                          // the unlock gesture
  osc.start();
  osc.amp(0.4, 0.05);
  return false;
}

function touchMoved() {
  osc.freq( map(mouseX, 0, width, 80, 880) );        // x → pitch
  filt.freq( map(mouseY, height, 0, 200, 4000) );    // y → brightness
  return false;
}

function touchEnded() {
  if (touches.length === 0) osc.amp(0, 0.2);         // silence on release
  return false;
}

function draw() {                            // the instrument's face
  background(20);
  stroke(255); strokeWeight(3);
  line(mouseX, 0, mouseX, height);
  line(0, mouseY, width, mouseY);
}

A finished, styled version with the same mapping is at starters/pocket-theremin — open it on your phone right now to feel where you're headed. Then build your own anyway: the point is owning every line.

Build 2 · The multitouch pad

A Tone sketch — new sketch, script line swapped per Day 3's rule (the theremin above was a p5.sound sketch; one audio library per sketch).

let synth;
let notes = ["C4", "D4", "E4", "G4", "A4", "C5"];   // pentatonic: no wrong notes

function setup() {
  createCanvas(windowWidth, windowHeight);
  synth = new Tone.PolySynth(Tone.Synth).toDestination();  // many voices
}

function touchStarted() {
  Tone.start();
  for (let t of touches) {                    // EVERY current finger
    let zone = floor( map(t.x, 0, width, 0, notes.length) );
    zone = constrain(zone, 0, notes.length - 1);
    synth.triggerAttackRelease(notes[zone], "8n");
  }
  return false;
}

function draw() {
  background(20);
  stroke(80);
  for (let i = 1; i < notes.length; i++)      // zone boundaries
    line(i * width / notes.length, 0, i * width / notes.length, height);
  fill(200, 60, 30); noStroke();
  for (let t of touches) circle(t.x, t.y, 60); // show every finger
}

Extensions, roughly in order of ambition: map t.y to octave or loudness · sustain while held (triggerAttack / triggerRelease per zone) · two-finger chords by design · swap the PolySynth for Friday's guitar sampler.

15:30 — The Deployment Ritual

The loop that keeps Week 2 alive. Practice it until it's boring:

1. In the p5 editor: save, then File → Share → Fullscreen link. (Editor links are HTTPS — which phones require for audio and sensors. This is why the editor is our deploy platform.)
2. Open the QR tool, paste the link, show your screen.
3. Scan with the phone's camera. Play.

Save → QR → phone: under thirty seconds. Any change, re-save, re-scan (or just reload on the phone — the URL stays the same). Everyone deploys the theremin and the pad to their own phone before leaving today; that is the exit ticket.