The Stage – A Sequencer

Video Demo

Screenshots

Sketches

(Unfortunately I did not have this idea in my three initial sketches, but I’ve attached the sprite sheet below, to illustrate the design process:

Initial Research & Motivations

Screenshot from Hidden Folks

I have long been fascinated with the game Hidden Folks, a hidden object game where you’re to locate an item in a scene made of up thousands of animated objects. I love how people inside the game are animated, full of actions (walking around, sitting and jumping, or tinkering with stuff), despite the simple line-drawing art form. Staring at these people move in fantastical ways in their miniature world is quite a fun experience, so I naturally asked the question, “how can I make music out of this visual form?”

Then I think about how to make the story plausible. Or, citing Ge, how to design in a way that people will say, “it’s gotta be this way”? It dawns on me each figure must be an independent player, who can be placed anywhere on the stage and fully in control of an entire instrument over all steps, instead of lining up with other figures to serve as a single, fixed note in a score sheet. Only in this way can the audio-visual correpondence truly feel natural — that this conveys the feeling of a lively scene where a group of free individuals randomly jams music together, not following any premade, mechanical coordination.

Spatial audio techniques also become a core motivation. How does the coordinates of players — now that their position is unconstrained – map to some kind of sound property? I decided to cook up a third type of object, besides player and instruments – modifiers, who’s sole purpose is to modify the nearest instrument. I settled on loud speakers that increase the gain as a function of distance, and it turned out to have a great audio visual correspondence!

Finally as a last thought, I added a pitch system for the pitched instruments, according to the y coordinate of the player, while leaving the x coordinate alone so that visually it still looks we have enough freedom to cavsually move things around. This concept of “loose spatial mapping” is at the core of my design motivation – let’s make a scene that makes use of spatial info without hurting the casualness of choosing any object layout as we like.

System Design

• Timing is driven by Chuck: chuck counts how many beats have elapsed and sends current beat number to unity, and unity updates animation accordingly on the edge.

• Modeling of the sequencer is established in Unity, then synced with ChuCk. Chuck model is designed orthogonally so that ChucK can immediately update the sound in the very next beat.

• As mentioned below, a queue system is invented to send arrays to ChucK by sending one (index, value) pair at a time from the queue head.

• Within unity, here’s how the modeling works:

  • There are three types of objects: the player, the instrument, and the modifier.

  • What is synced to chuck: List of Instrument Descriptions:

    • instrument type
    • beat (inherited from player, 0 means inactive)
    • gain (affected by modifier)
    • y-coord percentage (chuck will then calculate pitch)

  • The instrument is the source of sound: it is activated whenever there’s a player close enough to it.

  • Each player has a “beat” attribute, that is an integer correspond to 8-bit on/off of our 8-step sequencer.

• Within unity, here’s how to update the model according to changes in the scene:

  • Each frame, coupling is updated. Whenever a new (instrument, player) coupling occurs, the instrument’s beat is set to the player’s beat, and the player is marked as active.
  • Whenver a player is playing the upcoming beat, an animation is launched.
  • Each modifier will find the nearest instrument, and update its gain.
  • Finally, instrument y-coord is updated with that of its active player.

Source Code

Download the unity package here.

How to Install

Import the packages in latest version of unity and hit play! You might want to check out Chunity tutorials online to make sure you have Chuck-related dependencies installed first.

How to Use it

List of controls

• Digit key to add instrument (1 - 7)
• N to add a player
• M to add an audience. An audience will clap to the nearest player if they are close enough.
• A/D to adjust global gain.
• -/= to adjust global rate.
• S followed by [0-9] to save current stage to the corresponding slot.
• L followed by [0-9] to load a certain stage.
• G to toggle pitch Grid.
• Drag to black hole to remove elements!! (tribute to Kids)
• When hovering on player,
  • C to clear its beat pattern.
  • Spacebar to add the current beat to the player’s pattern.

Last Thoughts

It’d be nice to have global arrays in Chunity! I implemented a queue system to update arrays to Chuck by sending one (index, value) pair from the queue per frame, but that felt a bit awkward cuz I had to constantly think about doing diffs to minimize wasteful syncs, that complicates the code. If we have global arrays, the entire game state can be synced in one go, and that’ll ease up the spaghetti quite a bit!

Appendix

(Part 3 of 3) "Playable Prototype"
given the short timeframe, please aim for a semi-polished playable prototype
Musical Statement + In-Class Presentation
using your sequencer, create a short musical statement
create a short but polished video of a "mini-performance" of your sequencer
be prepared to present it in class
create a webpage for your audiovisual sequencer, which must contain:
one or more screenshot(s)
your sequencer "mini-performance" video
your initial design sketches + research, motivate and articulate how you made your design decisions
system design: how is your software architected? we recommend a class diagram.
(optional) any code for download
Note
As always, have fun with it! Make it satisfying.
you are welcome to work together, but you must do your own work
comment your code!

Deliverables
a website containing:

1) design sketches (scan them in if necessary)
2) a polished video of your step sequencer in action, and your musical statement
3) screenshots of your step sequencer in action
4) (optional) source code/executable to the project
5) instructions on using your step sequencer
6) instructions on building the project (for example, anyone in the class should be able to download)
7) readme section describing:
any comments you want to share (difficulties and what you enjoyed) on using Chunity
8) upload the website URL / submit all necessary files (code, images, videos, etc) to canvas!