Sensors and Ensembles

The phone's hidden organs — accelerometer, gyroscope, microphone — and what happens when six instruments play in one room. The day ends with the week's big decision: concept lock at 17:00.

10:00 — User-Study Debrief

Around the room: what did your home tester try first? Where did they get stuck? Two patterns to listen for — they recur in every study ever run: people try the gesture the surface suggests (not the one you designed), and they stop exploring the moment something sounds "wrong." Both are design inputs for the checklist from yesterday, especially failure-tolerance.

10:30 — Tilt and Shake

Every phone carries an accelerometer (linear force, including gravity) and a gyroscope (rotation). p5.js hands them to you as rotationX / rotationY (tilt, in degrees) and accelerationX / Y / Z. One catch: iOS requires a permission dialog, and the request must come from inside a real touch. This block — already wired into the instrument starter — is the incantation:

function touchStarted() {
  // iOS 13+: motion sensors need explicit permission, requested in a gesture
  if (typeof DeviceMotionEvent !== "undefined" &&
      typeof DeviceMotionEvent.requestPermission === "function") {
    DeviceMotionEvent.requestPermission();
    DeviceOrientationEvent.requestPermission();
  }
  userStartAudio();
  return false;
}

Build 1 · Tilt-to-bend

let osc, playing = false;

function setup() {
  createCanvas(windowWidth, windowHeight);
  osc = new p5.Oscillator('triangle');
}

function touchStarted() {
  /* permission block from above goes here */
  userStartAudio();
  if (!playing) { osc.start(); osc.amp(0.3, 0.1); playing = true; }
  return false;
}

function draw() {
  background(20);
  // rotationY: roughly -90 (tilt left) … +90 (tilt right)
  let f = map(rotationY, -60, 60, 150, 900);
  osc.freq(f, 0.05);                       // tilt = continuous bend, like meend
  fill(230); textSize(24); textAlign(CENTER);
  text(nf(f, 3, 0) + " Hz", width / 2, height / 2);
}

Hold the phone flat and rock it — the pitch pours. This is the gesture no keyboard has: continuous, embodied, analog. Day 4's meend lives in this sensor.

Build 2 · Shake-to-strike

A Tone sketch (script line swapped); the tilt and mic builds on this page are p5.sound sketches. One audio library per sketch.

let drum;

function setup() {
  createCanvas(windowWidth, windowHeight);
  drum = new Tone.MembraneSynth().toDestination();
  setShakeThreshold(35);                   // how hard is a "shake"
}

function deviceShaken() {
  drum.triggerAttackRelease("G1", "8n");   // the phone is now a damru
}

Or read accelerationX/Y/Z yourself and make intensity matter — a many-to-one mapping: total shake energy → drum pitch or drive.

12:00 — The Microphone as a Sensor

The mic isn't only for recording — it is a continuous input device. Level gives you breath control (Ocarina's whole trick); spectrum (FFT — the analyser from Day 1, pointed at the room) lets sound control sound:

let mic, fft, osc;

function setup() {
  createCanvas(windowWidth, windowHeight);
  mic = new p5.AudioIn();
  fft = new p5.FFT();
  fft.setInput(mic);
  osc = new p5.Oscillator('sine');
}

function touchStarted() {
  userStartAudio();
  mic.start();                      // triggers the browser's mic permission
  osc.start(); osc.amp(0);
  return false;
}

function draw() {
  background(20);
  let level = mic.getLevel();       // 0 … ~0.3 — loudness at the mic
  osc.amp( constrain(level * 4, 0, 0.5), 0.05 );   // blow to sound
  let spectrum = fft.analyze();     // 1024 bands, like Day 1's demo
  stroke(200, 60, 30);
  for (let i = 0; i < spectrum.length; i += 8)
    line(i / 8 * 8, height, i / 8 * 8, height - spectrum[i]);
}

Ideas this opens: hum into the phone and have it harmonize · a sound-level "hot potato" game · an instrument only playable by whistling · one phone listening to another (hold that thought until 14:00).

Optional spice, for a team that wants it: hand-pose tracking in the browser (ml5.js handPose) — wave at the camera, play. It costs a webcam-angle setup and some jitter-taming; budget a full build day if you choose it.

14:00 — Ensembles: Making Phones Play Together

Six phones is a sound system with no cables and no mixer — but "together" must be designed. Four strategies, cheap to expensive:

Demo: six phones, one beat

Everyone opens the same metronome sketch; we start on a spoken count of four. Listen to the drift build over one minute — that phasing is exactly Steve Reich's discovery, and the room must decide: is it a bug, or the piece?

15:30 — Concept Work → 17:00 Concept Lock

Solo, pairs, or one band — decide today. By 17:00, each project hands in a one-page instrument spec:

Scope truth, from every project that has ever shipped: the demo that works is the one with one idea done well. The spec exists so that on Thursday, when time is short, you cut features instead of quality.