Algorithmically Fingering Guitar Chords with Elixir

Written by Pete Corey on Aug 13, 2018.

Last time we wrote about using Elixir to generate all possible voicings of a given guitar chord to find the voicing with the best voice leading between another chord.

While this was great, there were several issues. We were conflating the idea of “musical distance” and “physical distance” when calculating optimal voice leading, and we weren’t taking the playability of the progressions we were generating into account.

To address both of these issues, we need to know not only which voicings are possible for a given chord, but also how each of those voicings can be played. We need to generate all possible fingerings for a given guitar chord voicing.

This sounds like a fantastic excuse to flex our Elixir muscles!

Calculating Fingerings

We’ll start our journey into calculating all possible fingerings for a given guitar chord by creating a new Elixir module, Chord.Fingering, and a new fingerings/1 function:

defmodule Chord.Fingering do
  def fingerings(chord)

Our high level plan of attack for computing possible fingerings is fairly straight forward. Given that each chord is a six-element array of frets being played, like [nil, 5, 7, 7, 6, nil], we want to:

  1. Attach all possible fingerings that can be played on each fret.
  2. Choose each possible finger in turn, sieve out all subsequent impossible fingers, and recursively repeat to get all possible fingerings.
  3. Perform any necessary cleanup.

Our final fingerings/1 function makes these steps fairly explicit:

def fingerings(chord),
    |> attach_possible_fingers()
    |> choose_and_sieve()
    |> cleanup()

Possible Fingers? Sieves?

Before we dive deeper into our solution, we should take a detour and talk about how we’re computing fingerings.

Our solution takes inspiration from the “Sieve of Eratosthenes”, which is a clever technique for calculating prime numbers. The basic idea of a “sieve” is that a choice made now can be used to filter out future unwanted results.

To bring it back to our situation, imagine we’re trying to play a D minor chord on the fifth fret:

Our D minor chord.

If we were to start fingering this chord by placing our second finger on the low D note, we know that we couldn’t use our first finger on any of the other notes in the chord. Our first finger would have to wrap over or sneak under our second finger to reach those notes, and that’s essentially impossible:

We can't use our first finger anywhere!

So by choosing to use our second finger on the fifth string and fret, we can sieve out the possibility of using our first finger on any of the remaining notes.

If we think about it, we can also sieve out the possibility of re-using our second finger. A finger can’t be re-used unless it’s forming a bar or a double-stop on an adjacent fret.

Our remaining set of possible fingers for the remaining notes are fingers three and four.

By recursively picking another of our possible fingers on another string and applying our sieving rules, we can come up with our entire set of possible fingers.

Choosing and Sieving

The meat of our algorithm lives in the choose_and_sieve/2 function, which takes an initial chord, complete with “possible fingers”, and a fingerings argument that defaults to an empty list:

defp choose_and_sieve(chord, fingerings \\ [])

The fingerings argument will be used to hold each finger choice for our chord, as we choose them.

Our choose_and_sieve/1 function expects each element of chord to be a two-element tuple, where the first element is the fret being played, and the second element is the set of possible fingers that could be chosen to play that fret.

Our attach_possible_fingers/1 helper function transforms our initial chord into that expected structure:

defp attach_possible_fingers(chord),
  do:, &{&1, 1..4})

Our implementation of choose_and_sieve/2 is recursive, so we should start our implementation by defining our base case. The base case for choose_and_sieve/2 is triggered when chord is empty. At that point, we’ve handled every note in the chord, and need to return our fully constructed fingering:

defp choose_and_sieve([], fingerings),
    |> Enum.reverse()
    |> List.to_tuple()

As we’ll soon see, chosen fingers are appended onto fingerings in reverse order, so we reverse/1 our list to reorient our strings. Lastly we turn our fingerings list into a tuple so that we can safely flatten/1 our resulting list of fingerings without losing our groupings.

Once flattened, our cleanup/1 function maps over this final list and converts each tuple back into an array:

defp cleanup(fingerings),
  do:, &Tuple.to_list/1)

Moving on from our base case, it’s time to start thinking of other simple to handle situations.

If the next element in our chord list is an unplayed string (nil), we add it to our fingerings list and designate it to be played with no finger (nil), and recursively call choose_and_sieve/2 on our remaining chord:

defp choose_and_sieve([{nil, _possible_fingers} | chord], fingerings),
  do: choose_and_sieve(chord, [{nil, nil} | fingerings])

Similarly, if the next element of our chord is an open string, we’re recursively call chose_and_sieve/2, passing in our remaining chord, and our set of fingers appended with the open string played with no finger (nil):

defp choose_and_sieve([{0, _possible_fingers} | chord], fingerings),
  do: choose_and_sieve(chord, [{0, nil} | fingerings])

In the case of actually needing to finger a note, the situation becomes more complicated. In that case, the next element of our chord is a fret and some set of possible_fingers.

We’ll map over each of the possible_fingers, appending each finger and fret to our list of fingerings, sieving out any now-impossible possible_fingerings from the remaining notes in our chord, and then recursively calling our choose_and_sieve/2 function with our newly sieved chord and new_fingerings:

defp choose_and_sieve([{fret, possible_fingers} | chord], fingerings),
    |> finger ->
      new_fingerings = [{fret, finger} | fingerings]

      |> sieve_chord(new_fingerings)
      |> choose_and_sieve(new_fingerings)
    |> List.flatten()

The sieve_chord/2 helper function maps over each of the notes in what’s left of our chord, and updates the possible_fingers tuple element to sieve any fingerings that are now deemed impossible to play after placing our most recent finger:

defp sieve_chord(chord, fingerings),
    |> {fret, possible_fingers} ->
      {fret, sieve_fingers(possible_fingers, fret, fingerings)}

The sieve_fingers/3 helper function is where we make real decisions about the behavior of our fingering algorithm. The sieve_fingers/3 function itself is fairly straight forward. It simply rejects and possible_fingers that are considered “bad” by our bad_finger?/3 helper function:

defp sieve_fingers(possible_fingers, fret, fingerings),
  do: Enum.reject(possible_fingers, &bad_finger?(fret, &1, fingerings))

The bad_finger?/3 function runs each finger/fret combinations through four rules used by our algorithm to determine if a finger choice is “impossible”, and should be culled from our possible_fingers set:

defp bad_finger?(fret, finger, fingerings),
      fret_above_finger_below?(fret, finger, fingerings),
      fret_below_finger_above?(fret, finger, fingerings),
      same_finger?(fret, finger, fingerings),
      impossible_bar?(fret, finger, fingerings)

If any of those rules are violated, the finger is rejected.

The first two rules check if a possible finger would need to stretch over or under an already placed finger, respectively:

defp fret_above_finger_below?(fret, finger, [{new_fret, new_finger} | _]),
  do: fret > new_fret && finger < new_finger

defp fret_below_finger_above?(fret, finger, [{new_fret, new_finger} | _]),
  do: fret < new_fret && finger > new_finger

The third rule verifies that no finger can be used twice, unless when performing a bar or double-stop over adjacent frets:

defp same_finger?(fret, finger, [{new_fret, new_finger} | _]),
  do: finger == new_finger && fret != new_fret

Finally, we need to prevent “impossible bars”, or bars that would mute notes played on lower frets:

defp impossible_bar?(_fret, finger, fingerings = [{new_fret, _} | _]),
    |> Enum.filter(fn {fret, _finger} -> fret > new_fret end)
    |> {_fret, finger} -> finger end)
    |> Enum.member?(finger)

The Results

Now that we’ve implemented our fingering algorithm, let’s try a few examples.

We’ll start by calculating the possible fingerings for the D minor chord we’ve been using as an example. Fingering suggestions are listed below each string:

[nil, 5, 7, 7, 6, nil]
|> Chord.Fingering.fingerings()
|> Enum.join("\n\n")
|> IO.puts

Fingerings for our D minor chord.

Awesome! The first suggested bar can be difficult to play, but with some practice doing Ted Greene-style double-stops, it’s manageable. The second and third suggestions are what I would normally reach for.

Another interesting example is an open G major shape:

[3, 2, 0, 0, 3, 3]
|> Chord.Fingering.fingerings()
|> Enum.join("\n\n")
|> IO.puts

Fingerings for our G chord.

The first few fingering suggestions make sense, but as we get closer to the end of the list, some of the suggestions are increasingly difficult to play. I don’t think I’ll ever be able to play this fingering:

An "impossible" to play fingering.

As a human, I can explain to you why this is difficult to play, but I haven’t been able to come up with a general rule to add to our rule set that would prevent these kinds of fingerings from being suggested. At this point, I’d rather have the algorithm present potentially impossible fingerings, than have it over-aggressively prune possible fingerings from the result set.

What’s Next?

In my previous article on “Voice Leading with Elixir”, I mentioned that I was conflating the ideas of “musical distance” and “physical distance”. In terms of voice leading, all I really care about is optimizing a chord progression for musical distance. But as a guitar player, I also want to consider “physical distance”.

If a set of chords all have the same “musical distance” from a given starting chord, I want to choose the chord that has the lowest “physical distance”. By “physical distance”, I mean literally fret distance, but also how difficult it is to transition from one chord to another. Do I just need to slide one finger? That’s easy! Do I need to lift and replace three fingers while sliding the fourth? That’s not so easy…

We can’t calculate the “physical distance” between chords unless we know the fingerings for the chords in question. Now that we know the potential fingerings for a given chord, we can compute a (modified) levenshtein distance between the fingerings of two chords!

Why is that cool?

Once that’s done, we’ll be able to take a starting chord (optionally with a starting fingering), and find the best voicing of the landing chord in terms of voice leading and ease of playability!

Be sure to check out the entire project on Github, and stay tuned for more.

Voice Leading with Elixir

Written by Pete Corey on Jul 30, 2018.

I play quite a bit of guitar in my free time. Once of the things I’ve been practicing lately is improving my voice leading between chords.

Voice leading refers to how the individual notes, or voices, within a chord move when you transition to another chord. You often want as little movement as possible to keep the transition from sounding jarring (unless you’re going for jarring).

So for example, if I play a G7 way up the neck, I probably wouldn’t want to follow it with a Cmaj7 played towards the nut. Instead, I’d like to find another voicing of Cmaj7 that’s both physically and musically closer to our G7 chord.

Knowing how to voice lead between chords usually requires a vast knowledge of the fretboard, a huge chord vocabulary, and lots of practice. But who needs all then when you have a computer and the Elixir programming language?

Let’s use Elixir to chug through all of the possible Cmaj7 chords and find those with the best voice leading from our G7!

Rendering Chords

Before we start talking about recruiting our computer to help us find the best voice leading between chords, we should take a detour and talk about guitar chords and how we’ll work with them.

When you break it down to the basic, a “guitar” is just six strings attached to a piece of wood. A “chord” is just a set of notes played simultaneously across any number of those strings. Different notes can be played on each string by pressing on any “fret” along the neck.

Given those definitions, the simplest ways to represent a chord using Elixir data structures probably be as a six element list (or tuple).

Here’s our G7 chord represented as an array:

[nil, 10, 12, 10, 12, nil]

From the thickest string to the thinnest, we’re not playing anything on the first string (nil). We’re playing a G on the next string (10), a D on the next string (12), an F on the next string (10), a B on the next string (12), and nothing on the thinnest string (nil).

To make our lives easier, we should come up with some way of displaying these chords in a more guitarist-friendly manner. One common option for displaying guitar chords is with chord charts:

To kick things off, let’s write a Chord.Renderer module with a to_string/2 function that takes a chord and returns a unicode-based chart for the provided chord:

defmodule Chord.Renderer do
  def to_string(chord, chord_name) do

The first thing we’ll need to do is find out the “reach” of our chord. What’s the lowest fret used in the chord and the highest?

{min, max} =
  |> Enum.reject(&(&1 == nil))
  |> Enum.min_max()

We can use Elixir’s Enum.reject/2 to filter out unplayed strings and then use Enum.min_max/1 to easily find both the lowest and highest fret used in the chord.

Next we’ll iterate over every set of frets within the range of the chord and render each row using a row_to_string/4 helper function:

0..max(max - min, 3)
|>, min, chord, chord_name))

Most fret charts render some minimum number of rows, even if the chord only takes up one fret of vertical space. We’ll iterate between 0 and either max - min, or 3, depending on which value is larger. This means we’ll always render at least four rows of frets for each diagram.

We’ll also want to intersperse the horizontal fret lines below each row of fingered notes on each row of frets:

|> Enum.intersperse([:bright, :black, "\n   ├┼┼┼┼┤\n"])

We’re using Elixir’s ANSI color codes to color our fretboard lines a dark grey color, and building our final string as an IO list, rather than a single concatenated string.

Because we’re using ANSI color codes, we need to format and convert our resulting nested list structure into a string before returning it from our to_string/2 function:

|> IO.ANSI.format()
|> IO.chardata_to_string()

Our row_to_string/3 helper function is fairly straight forward. It simply renders a left gutter, the row of frets with any potential fingerings, and a right gutter:

defp row_to_string(offset, base, chord, chord_name),
  do: [
    left_gutter(offset, base + offset),, &fret_to_string(&1, base + offset)),
    right_gutter(offset, chord_name)

The left_gutter/2 helper function renders the lowest fret used in the chord on the first line of the chart:

defp left_gutter(0, fret),
    do: [:bright, :yellow, String.pad_leading("#{fret}", 2, " ") <> " "]

Otherwise, we render a spacer:

defp left_gutter(_, _),
  do: "   "

Similarly, the right_gutter/2 helper function either renders an optional chord_name on the first line of the chord chart:

defp right_gutter(0, chord_name),
  do: [:yellow, " #{chord_name}"]

Or an empty string:

defp right_gutter(_, _),
  do: ""

That’s all there is to it!

Now we can render chords by passing them into Chord.Renderer.to_string/2:

Chord.Renderer.to_string([nil, 10, 12, 10, 12, nil], "G7")
|> IO.puts
10 │●│●││ G7

And in its fully colored glory:

Our G7 chord, as rendered by our new module.

Chord Distance

We can roughly approximate how “good” the voice leading is between two chords by counting the number of frets each finger has to move when changing chords. We can call this the “distance” between the two chords. In the simplest terms, chords with good voice leading have minimal distance between each other.

If we can write a function that computes this distance between chords, we might be able to generate all possible Cmaj7 voicings, and find the voicing that leads best from our G7!

Let’s say that each fret moved on a single string equals one unit of “distance”, and adding or removing a note to or from a string also counts as a single unit of distance.

Using that heuristic, let’s write a new Chord module and a distance/2 function that calculates the distance between two chords.

If both chords are equal, there is zero distance between them:

def distance(chord, chord),
  do: 0

Otherwise, the distance between two chords is the sum of the distance between their individual fretted notes on each string:

def distance([fret_a | rest_a], [fret_b | rest_b]),
  do: distance(fret_a, fret_b) + distance(rest_a, rest_b)

If a the first chord doesn’t have a note fretted on a string, and the next chord does, we’ll add one unit of distance:

def distance(nil, fret),
  do: 1

And visa versa:

def distance(fret, nil),
  do: 1

Otherwise, if both strings have fretted notes, the distance moved on that string is the number of frets between the two chords on that string:

def distance(fret_a, fret_b),
  do: abs(fret_a - fret_b)

We can manually calculate the distance between our G7 chord ([nil, 10, 12, 10, 12, nil]), and a few different Cmaj7 voicings we may know:

Chord.distance([nil, 10, 12, 10, 12, nil], [nil, 3, 5, 4, 5, nil])   # 27
Chord.distance([nil, 10, 12, 10, 12, nil], [8, 10, 9, 9, nil, nil])  # 6

So according to our heuristic, the second voicing of Cmaj7 has much better voice leading between our G7 than the first voicing of Cmaj7.

This is great, but we’re still limited by our knowledge of the fretboard. What if we only know two voicings of a Cmaj7 chord. Is this the best we can do?

Absolutely not!

Brute Forced Voicings

The last piece of this puzzle is to write a function that will generate all possible voicings of a given chord across the neck of the guitar. If we have all of the possible voicings of our Cmaj7, for example, we can easily find the voicing that has the best voice leading from our G7 chord!

Let’s start by creating a new voicings/1 function in our Chord module:

def voicings(notes) do

The voicings/1 function accepts an array of numbers representing the notes we want in our chord. For example, if we wanted all of the voicings of our Cmaj7 chord, we’d call vocings/1 with a C (0), an E (4), a G (7), and a B (11). These numbers correspond to the lowest set of MIDI notes, ranging from 0 to 11.

The first thing we want to do is calculate all of the possible “note sets” that will be spread across our guitar strings:

|> all_note_sets()

If a chord has fewer notes than the number of strings we want to play, some number of those notes will have to be repeated. To illustrate, imagine we want to play our four note Cmaj7 using all six strings of the guitar. We’ll obviously have four strings playing C, E, G, and B, but what will the other two strings play?

The all_note_sets/1 helper functions calculates this list of all possible note sets using some hand-waving combinatorics, and a few unfortunate list comprehensions:

def all_note_sets(notes) do
  for length <- 6..length(notes) do
    for base <- Combination.combine(notes, min(length, length(notes))) do
      for extension <- Combination.combine(notes, length - length(notes)) do
        base ++ extension
  |> Enum.reduce(&Kernel.++/2)
  |> Enum.reduce(&Kernel.++/2)

Next, our voicings/1 function needs to take each of these possible note sets and build all possible chords using that set of notes:


The build_chords/1 helper works by recursively building up all possible chords made of all possible notes in the provided note sets.

def build_chords(note_set, chord \\ [nil, nil, nil, nil, nil, nil], chords \\ [])

It starts by looking at the first note in the provided note set and finds all occurrences of that note across all of the strings of our guitar using the all_notes/1 helper:

|> all_notes

Next, it filters out notes on strings already used in the current chord under construction:

|> Enum.filter(fn {string, fret} ->, string) == nil end)

Finally, it takes each note, inserts it into the current chord, and checks the “stretch” of the chord. If the chord spans more than five frets, we deem it impossible to play and filter it out (which is obviously an over-simplification, especially at higher frets). Otherwise, we recursively call build_chords/3, passing in the newly updated current chord and the remaining set of notes in our note set:

|> {string, fret} ->
  new_chord = List.replace_at(chord, string, fret)

  {min, max} =
    |> Enum.reject(&(&1 == nil))
    |> Enum.min_max(fn -> {0, 0} end)

  if max - min <= 5 do
    build_chords(rest, new_chord, chords)

The all_notes/1 helper function works by accepting the abstract value of the note we’re looking for (C is 0), the optional MIDI notes of the tuning of each string, and the optional number of frets up the neck we want to look for notes:

def all_notes(target_note, strings \\ [40, 45, 50, 55, 59, 64], frets \\ 12) do

It then constructs a two dimensional list of MIDI notes up the neck and across the fretboard:

fretboard =
  for fret <- 0..frets,
    do:, &(&1 + fret))

Once we’ve built up our fretboard, we’ll filter out all of the notes that aren’t the specific note we’re looking for. We loop over every row of frets, and every string:

|> Enum.with_index()
|> {row, index} ->
  |> Enum.with_index()
  |> {note, string} ->

For each note we encounter, we check if rem(note, 12) equals our target_note. If it does, we replace the current note value with a string/index tuple that can be used when building our guitar chord:

if rem(note, 12) == target_note do
  {string, index}

Otherwise, we replace the current note with nil.

Next, we flatten our multidimensional fretboard representation and filter out all of the nil values, leaving us with just the set of notes we’re looking for, and where they can be found on the fretboard.


Let’s try it out by listing the first three voicings of a Cmaj7 chord our new voicings/1 helper finds:

Chord.voicings([0, 4, 7, 11])
|> Enum.take(3)

 0 ││││●│   0 ││││●│   1 ││││●│ 
   ├┼┼┼┼┤     ├┼┼┼┼┤     ├┼┼┼┼┤
   ││││││     ││││││     │●●│││
   ├┼┼┼┼┤     ├┼┼┼┼┤     ├┼┼┼┼┤
   ││●│││     ││●│││     ●││││●
   ├┼┼┼┼┤     ├┼┼┼┼┤     ├┼┼┼┼┤
   ●●│││●     ●●│││●     │││●││
   ├┼┼┼┼┤     ├┼┼┼┼┤
   │││●││     │││●││


Putting it all Together

Now that our voicings/1 helper is finished, we can put all of the pieces together.

Let’s start by calculating all of the possible voicings of our Cmaj7 chord:

[0, 4, 7, 11]
|> Chord.voicings()

Next, let’s map over each voicing and build a tuple who’s first element is the distance from our G7 chord, and who’s second element is the generated voicing itself:

|>{Chord.distance(&1, [nil, 10, 12, 10, 12, nil]), &1})

Now let’s sort that list. Because the distance from our G7 chord is the first element in each tuple, we’re effectively sorting by distance:

|> Enum.sort()

Now the “best” options for Cmaj7 voicings should be at the top of our list. Let’s take the first three:

|> Enum.take(3)

We’ll map each voicing through our chord chart renderer:

|> {distance, chord} -> Chord.to_string(chord, "Cmaj7") end)

Finally, let’s join each of our three charts together with newlines and print the result:

|> Enum.join("\n\n")
|> IO.puts()

Our generated Cmaj7 voicings.

Each of the voicings recommended by our software sound fairly nice. Much nicer than the first voicing we were using way down the neck. The third voicing definitely has an interesting flavor, and is something I never would have reached for without the help of this software, but I’m glad to know it’s there.

Final Thoughts and Future Work

I have many, many final thoughts about this project. If you can’t tell, I’m incredibly excited about this kind of thing.

I’m currently working on improving the “distance” heuristic, which raises many interesting questions about what exactly voice leading is, and who it’s for. Should I optimize for the player, or the listener? Thanks to how the guitar works, chords on wildly different sections of the neck may be very close musically, but my algorithm will filter these chords out as being “too far.” In many ways, I’m conflating “voice leading” between chords with “playability” between chords. Is this what I want?

I’m also working on optimizing the voice leading over entire chord progressions. As you might guess, this is a much more expansive problem.

A generated chord progression.

Lastly, if you’re interested in this kind of thing, I highly recommend you check out Ted Greene’s guitar work. Ted is, in my opinion, one of the true masters of the guitar, and put some serious work into perfecting his voice leading skills.

Check out the Ted Greene archive, the archive’s Youtube page, and definitely check out two of Ted’s books: Chord Chemistry, and Modern Chord Progressions.

I’ve uploaded this entire project to Github, if you’re curious the see the source in its entirety. Check it out!

Obviously, this kind of thing is just a tool, and the chord transitions and progressions generated by the Chord module are just suggestions and starting places, not fully-fleshed out music. That being said, these tools have already given me many new ideas and shown me many interesting chords I never would have reached for without having them shown to me.

Building a Better Receive Loop

Written by Pete Corey on Jul 23, 2018.

I’ve been putting quite a bit of time this past week into overhauling and refactoring my in-progress Elixir-based Bitcoin node.

As a part of that overhaul, I turned my attention to how we’re receiving packets from connected peers. The way we’ve been handling incoming packets is overly complicated and can be greatly simplified by taking advantage of the Bitcoin protocol’s packet structure.

Let’s go over our old solution and dig into how it can be improved.

The Original Receive Loop

Our Bitcoin node uses Erlang’s :gen_tcp module to manage peer to peer communications. Originally, we were using :gen_tcp in “active mode”, which means that incoming packets are delivered to our node’s Elixir process in the form of :tcp messages:

def handle_info({:tcp, _port, data}, state) do

Because TCP is a streaming protocol, no guarantees can be made about the contents of these messages. A single message may contain a complete Bitcoin packet, a partial packet, multiple packets, or any combination of the above. To handle this ambiguity, the Bitcoin protocol deliminates each packet with a sequence of “magic bytes”. Once we reach this magic sequence, we know that everything we’ve received up until that point constitutes a single packet.

My previous receive loop worked by maintaining a backlog of all incoming bytes up until the most recently received sequence of magic bytes. Every time a new message was received, it would append those incoming bytes to the backlog and chunk that binary into a sequence of packets, which could then be handled individually:

{messages, rest} = chunk( <> data)

case handle_messages(messages, state) do
  {:error, reason, state} -> {:disconnect, reason, %{state | rest: rest}}
  state -> {:noreply, %{state | rest: rest}}

This solution works, but there are quite a few moving pieces. Not only do we have to maintain a backlog of all recently received bytes, we also have to build out the functionality to split that stream of bytes into individual packets:

defp chunk(binary, messages \\ []) do
  case Message.parse(binary) do
    {:ok, message, rest} ->
      chunk(rest, messages ++ [message])

    nil ->
      {messages, binary}

Thankfully, there’s a better way.

Taking Advantage of Payload Length

Every message sent through the Bitcoin protocol follows a specific format.

The first four bytes of every packet are reserved for the network’s magic bytes. Next, twelve bytes are reserved for the name of the command being sent across the network. The next four bytes hold the length of the payload being sent, followed by a four byte partial checksum of that payload.

These twenty four bytes can be found at the head of every message sent across the Bitcoin peer-to-peer network, followed by the variable length binary payload representing the meat and potatoes of the command being carried out. Relying on this structure can greatly simplify our receive loop.

By using :gen_tcp in “passive mode” (setting active: false), incoming TCP packets won’t be delivered to our current process as messages. Instead, we can ask for packets using a blocking call to :gen_tcp.recv/2. When requesting packets, we can even specify the number of bytes we want to receive from the incoming TCP stream.

Instead of receiving partial messages of unknown size, we can ask :gen_tcp for the next 24 bytes in the stream:

{:ok, message} <- :gen_tcp.recv(socket, 24)

Next, we can parse the received message bytes and request the payload’s size in bytes from our socket:

{:ok, %{size: size}} <- Message.parse(message),
{:ok, payload} <- :gen_tcp.recv(socket, size)

And now we can parse and handle our payload, knowing that it’s guaranteed to be a single, complete Bitcoin command sent across the peer-to-peer network.

Final Thoughts

There’s more than goes into the solution that I outlined above. For example, if we’re receiving a command like "verack", which has a zero byte payload, asking for zero bytes from :gen_tcp.recv/2 will actually return all of the available bytes it has in its TCP stream.

Complications included, I still think this new solution is superior to our old solution of maintaining and continually chunking an ongoing stream of bytes pulled off the network.

If you’re eager to see the full details of the new receive loop, check it out on Github!

I’d also like to thank Karl Seguin for inspiring me to improve our Bitcoin node using this technique. He posted a message on the Elixir Slack group about prefixing TCP messages with their length to easily determine how many bytes to receive:

I’d length prefix every message with 4 bytes and do two recvs, {:ok, <<length::big-32>>} = recv(socket, 4, TIMEOUT) {:ok, message} = recv(socket, length, TIMEOUT)

This one line comment opened my mind to the realization that the Bitcoin protocol was already doing this, and that I was overcomplicating the process of receiving messages.

Thanks Karl!