Making Noise with J

Written by Pete Corey on Jul 2, 2018.

I’ve always been fascinated by live-coded music. Frameworks like Chuck, Supercollider, Overtone, Extempore, and Sonic PI, along with popular performers and musicians like Sam Aaron and Andrew Sorensen have never ceased to amaze and inspire me.

That said, whenever I’ve tried to use one of those tools or frameworks to create my own music, I’ve always quickly given up. Maybe it’s because I’m just lazy and learning new things is hard, but I’ve always told myself that it’s because the tools I was using just didn’t fit with how I felt programming music should be. Syntactically, ergonomically, and conceptually, the tools just didn’t jive.

And then I stumbled across J.

J and the entire family of APL languages have a beautiful terseness and closeness to the data being operated on. They’re also fundamentally designed to operate on arrays, a data structure ripe for musical interpretation. I’ve convinced myself that if I can learn J, I’ll be able to build the live coding environment of my dreams!

That’s a big goal, but I’m taking baby steps to get there. Today, I’ll show you how I managed to make noise with J.

Making Noise Without J

My plan for making noise with J doesn’t actually involve my J software producing any noise directly. Instead, it’ll act as a controller that instructs other software on my machine to make noise on its behalf.

The software making the noise will be SimpleSynth, which is a small, easy to use MIDI synthesizer. If you’re following along, feel free to use any other MIDI synth you’d like, or a full audio workstation like Ableton or even GarageBand.

SimpleSynth.

When we fire up SimpleSynth, it’ll ask which MIDI source it should use. MIDI is a protocol that lets us pass around musical information, like when and how loud certain notes should be played, between different devices. SimpleSynth is asking which stream of notes it should listen to and play.

Setting up our J virtual device in MIDI Studio.

I used MacOS’ built-in MIDI Studio to create a virtual MIDI channel called “J”, with a MIDI port called “Bus 1.” After making sure the virtual device was online, I selected it in SimpleSynth.

Selecting our J virtual device in SimpleSynth.

The last piece of the puzzle is finding some way of programmatically sending MIDI messages through my “J Bus 1” to be played by SimpleSynth. Geert Bevin’s SendMIDI command line tool did just the trick.

Once installed, we can use SendMIDI to send MIDI notes to SimpleSynth from our command line:

sendmidi dev "J Bus 1" on 60 100

Turning on note 60, with a velocity of 100 effectively plays a middle C at full volume.

Now we’re making music!

Talking to SendMIDI with J

The next challenge lies in getting J to execute sendmidi commands.

After much searching and head scratching, I learned that J exposes a wide range of miscellaneous functionality under the “foreigns” (!:) verb. Calling 2!:1 y lets you spawn a new process, running whatever command you pass in through y.

Let’s try invoking our spawn verb with our sendmidi command:

   2!:1 'sendmidi dev "J Bus 1" on 60 100'
|interface error
|       2!:1'sendmidi dev "J Bus 1" on 60 100'

After even more searching and head scratching, I realized that I needed to use the fully-qualified sendmidi path when making the call:

   2!:1 '/usr/local/bin/sendmidi dev "J Bus 1" on 60 100'

I hear sound! Success!

Making Music with J

While this is great, it’s not much better just running our sendmidi command directly from the command line. What would make things even better is if we could build ourselves a play verb that plays any notes passed to it.

For example, if I were to run:

   play 60 64 67

I’d expect J to construct and execute our sendmidi command, which should play a C major chord:

sendmidi dev "J Bus 1" on 60 100 on 64 100 on 67 100

After a few brain-expanding weekends of playing around in J, I came up with this version of the play verb:

   on =: ('on ',5|.' 100 ',":)"0
   play =: [:2!:1'/usr/local/bin/sendmidi dev "J Bus 1" ',[:,/on

The on verb turns an integer note into an “on string” of the format, 'on <note> 100 ', and the play verb spawns the result of appending '/usr/local/bin/sendmidi ...' to append mapped over on applied to y.

Put simply, it constructs our sendmidi command and executes it.

We can play a C major chord:

   play 60 64 67

Or any other chord we want:

   play 60 63 54 70 73

Final Thoughts

Please keep in mind that I’m very new to J, and even newer to tacit programming. If you see anything that can be improved, clarified, or corrected, please let me know.

I still feel very clunky and slow when it comes to using J. Building this two line program took hours of my time. That said, I feel like there is potential here. As I grow more used to the tacit paradigm and play with other ways of interacting to DAWs and other audio producers, I feel like J might turn into my ideal music creation environment.

Time will tell.

Limiting Peers with DynamicSupervisor Options

Written by Pete Corey on Jun 18, 2018.

Last month I posted an article about using Elixir’s DynamicSupervisor behavior to recursively connect our Elixir-based node to peers throughout Bitcoin’s peer-to-peer network.

The last part of that article talked about how we could limit the exponential growth of our set of connected peers by setting a hard cap on the number of processes supervised by our dynamic Node.Supervisor process.

We went through the rigmarole of building this child process cap ourselves, but it was pointed out to me that we could have used DynamicSupervisor’s built in :max_children option to accomplish the same thing!

Our Hand-Rolled Solution

When we implemented our own restriction on the number of peers we allow our node to connect to, we did it within the BitcoinNetwork.connect_to_node/2 function:


def connect_to_node(ip, port) do
  if count_peers() < Application.get_env(:bitcoin_network, :max_peers) do
    DynamicSupervisor.start_child(BitcoinNetwork.Node.Supervisor, %{
      id: BitcoinNetwork.Node,
      start: {BitcoinNetwork.Node, :start_link, [{ip, port}]},
      restart: :transient
    })
  else
    {:error, :max_peers}
  end
end

The count_peers/0 helper function simply calls out to DynamicSupervisor.count_children/1 to count the number of processes being supervised by our dynamic Node.Supervisor:


BitcoinNetwork.Node.Supervisor
|> DynamicSupervisor.count_children()
|> Map.get(:active)

If the number of active peers is less than our specified number of :max_peers, we allow the connection. Otherwise, we return an :error tuple.

Elixir’s Solution

If we read through the DynamicSupervisor documentation, we’ll find that we can pass a :max_children option to DynamicSupervisor.start_link/2. Digging through Elixir’s source, we can see that, when present, the :max_children option does literally exactly what we did in our hand-rolled solution:


if dynamic < max_children do
  handle_start_child(child, %{state | dynamic: dynamic + 1})
else
  {:reply, {:error, :max_children}, state}
end

If dynamic, the number of processes currently being supervised by the supervisor, is less than the specified max_children, add the child. Otherwise, return an :error tuple.

Refactoring

Refactoring our original solution to make use of the :max_children option largely consists of removing our original solution. We’ll start by gutting the guard in our BitcoinNetwork.connect_to_node/2 function:


def connect_to_node(ip, port) do
  DynamicSupervisor.start_child(BitcoinNetwork.Node.Supervisor, %{
    id: BitcoinNetwork.Node,
    start: {BitcoinNetwork.Node, :start_link, [{ip, port}]},
    restart: :transient
  })
end

This means we can also remove our count_peers/0 helper function.

Now we simply need to add the :max_children option to our dynamic supervisor when it starts up:


{:ok, pid} =
  Supervisor.start_link(
    [
      {DynamicSupervisor,
        name: BitcoinNetwork.Node.Supervisor,
        strategy: :one_for_one,
        max_children: Application.get_env(:bitcoin_network, :max_peers)}
    ],
    strategy: :one_for_one
  )

That’s all there is to it!

Our limited set of peers.

Spinning up our Bitcoin node with a low value for :max_peers shows that our Node.Supervisor is honoring our limit.

Final Thoughts

My final thoughts are that I should really spend more time reading through the Elixir and Erlang documentation. There’s quite a few gems hidden in plain sight that would do me quite a bit of good to know about.

I’d also like to thank the Redditor who pointed the :max_children option out to me. Thanks, ParticularHabit!

Generating Test Fixtures with Wireshark

Written by Pete Corey on Jun 11, 2018.

My in-progress Elixir-based Bitcoin node is woefully lacking on the test front. This is especially problematic considering how finicky the Bitcoin protocol parsing and serialization process can be.

But how can we test this functionality without going through the mind-numbing process of manually constructing each packet under test and asserting that it parses and serializes as expected?

Thankfully, Wireshark’s support of the Bitcoin protocol turns this into a simple task. Let’s dig into how we can use Wireshark to generate binary fixtures for each of our Bitcoin packets under test, and explore how we can test against them using Elixir.

Generating Our Fixtures

Wireshark supports the Bitcoin protocol out of the box. That makes the process of generating test fixtures incredibly simple. To create a binary fixture for a given Bitcoin packet, we just need to follow these three steps:

Step one: Fire up Wireshark, start capturing on your network interface, and set bitcoin as your display filter:

Filtering for bitcoin packets.

Step two: Start bitcoind, and watch the packets roll in:

Bitcoin packets on the wire.

Step three: Notice that Wireshark teases out the Bitcoin-specific portion of every matching TCP packet it receives. Each packet can be exported by right clicking on the “Bitcoin protocol” breakdown, and choosing “Export Packet Bytes.”

High level packet information.

The bytes we’re exporting represent the entire packet, as it comes in over the wire.

Parsing Our Fixtures

Now that we’ve saved a handful of packets we’d like to test against, we can start the process of incorporating them into our test suite.

Let’s assume that we’ve saved all of our exported packets into a test/fixtures folder within our project. Let’s also assume that we want to start by testing our “version” packet (the most interesting packet we’re able to parse, so far).

Let’s make a new VersionTest test module and lay down some boilerplate:


defmodule BitcoinNetwork.Protocol.VersionTest do
  use ExUnit.Case

  alias BitcoinNetwork.Protocol
  alias BitcoinNetwork.Protocol.{Message, Version}
end

Next, we’ll add our test:


test "parses a version payload" do
end

The first thing we’ll need to do is load the data from our exported version packet binary:


assert {:ok, packet} = File.read("test/fixtures/version.bin")

We use Elixir’s File.read/1 to read the contents of our version.bin file, and assert that we’ll receive an :ok tuple containing the binary contents of our file in our new packet assignment.

Next, we’ll parse the binary, just like we do within our Node with a call to Message.parse/1:


assert {:ok, message, <<>>} = Message.parse(packet)

Once again, we assert that we’ll receive an :ok tuple with our resulting message. Because the data we exported from Wireshark relates specifically to our version packet, we expect the list of remaining, unparsed binary data to be empty (<<>>).

Now that we’ve parsed the message, we can compare the resulting Version struct found in message.parsed_payload with a pre-defined, expected version struct and assert that they’re equal:


assert message.parsed_payload == version

But where does version come from? How can we know the contents of our version.bin packet without manually parsing it ourselves, byte by byte?

Interpreting Our Fixtures

Once again, Wireshark comes to the rescue. In addition to letting us export our Bitcoin packets as raw binaries, Wireshark also lets us inspect the parsed contents of each of our Bitcoin packets.

If we go back to our version packet in our Wireshark capture file, we can open up the “Bitcoin protocol” section and see a complete breakdown of not only the high level message metadata, but also the specific information sent along in the version message:

Filtering for bitcoin packets.

We can use this information to construct our pre-defined version struct at the top of our test:


version = %Version{
  version: 70015,
  services: 13,
  timestamp: 1_528_146_756,
  recv_ip: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 255, 255, 160, 16, 233, 215>>,
  recv_port: 18333,
  recv_services: 9,
  from_ip: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>,
  from_port: 0,
  from_services: 13,
  nonce: 15_116_783_876_185_394_608,
  user_agent: "/Satoshi:0.14.2/",
  start_height: 1_322_730
}

And with that, we have a solid test of our version parsing functionality.

Testing Serialization

We can test the serialization of our version packet much like we tested the parsing functionality.

Let’s start off by adding a new test to our VersionTest module:


test "serializes a version struct" do
end

Once again, we’ll start off by using File.read/1 to load our binary fixture, and using Message.parse/1 to parse the resulting binary:


assert {:ok, packet} = File.read("test/fixtures/version.bin")
assert {:ok, message, <<>>} = Message.parse(packet)

Rather than comparing the message.parsed_payload to some pre-defined Version struct, we’ll instead serialize it with a call to Protocol.serialize/1 and compare the newly serialized version against the message’s payload binary:


assert Protocol.serialize(message.parsed_payload) == message.payload

And that’s it!

If our version serialization code is working correctly, it should return a binary identical to the version portion of the packet exported from Wireshark.

Final Thoughts

I’d like to give a huge shout out to Lucid Simple’s article on “Binary Fixtures with Wireshark”. It was a huge inspiration for me and a very well written article. I highly recommend you check it out if you’d like a more in-depth exploration of using Wireshark-generated binary fixtures.

For what it’s worth, this kind of testing has already resulted in a positive return on investment. Shortly after implementing these tests, I noticed that my version struct was incorrectly serializing messages, resulting in some strange behavior I’d been noticing with my node. Using the tests as a guide, I was able to quickly fix my implementation.

Three cheers for testing!