Friendly Robot Overlord

A few new bits for the spring synth.

• Event plugins (so keyboard input is optional)
• MIDI support (an "Event plugin" which uses Jack)
• Lua 5.1 (no longer an external dependency too, so it is easier to compile)
• Visualisation plugins (so the OpenGL stuff is optional)
• Cleaned up a whole lot of build stuff to make it easier to build

I compiled it for an N900, and it works. Hopefully with all the changes I made before, it will nearly compile out of the box (just have to disable some of the plugins). If you attempt this, then bear in mind that you will want to simplify the models you are using.

Get spring synth version 0.2 (C code, 253 KB, GPL3) and be frustrated that it still doesn't have any cool examples.

It has been a while. I've been working on something quite fun, it is a physical modelling synth. It is a bit bigger than the usual hackish programs I put up here, and ties lots of different things together. I wouldn't say it is ready for primetime yet, but it is at a stage where it is fun to use.

So it is an audio synth with a 3d wireframe view of the model it is synthesizing. So this particular model sounds like this:

Download (42KB)

To drive it, you create a lua script which builds up the model, saying where to put points, how heavy they are and how the points join together. The joins are all springs, with a fairly simple model. The lua script can also define events, at the moment I bind these to keypresses, so you might have something like this:

function event.key_a(s)
    apply_force_set(s, "hit_me", 0.0, 0.0, -250.0);
end

Which applies a force of 250 units, in some particular direction (either down or up I suspect) to all of the points that have been labeled "hit_me" whenever the "a" key is pressed.

As a heads up, don't use models/scripts from people you don't trust, they are programs in their own right and can do bad stuff like deleting files.

Any springs that are labeled as "air" are used for getting sound out of the model. Since I don't model actual air, I can't model a real microphone. The raw signal is the total displacement of all the springs labeled "air". This is then filtered and limited a bit to allow you to make pretty much anything and get useful sound out of it without having to tweak numbers. However, if you have a loud thing and a soft thing, then the soft thing probably won't be audible. This layer is all a bit of a hack.

So, there was the OpenGL wireframe for output, but you might also want to be able to hear what is going on. I've been a bit busy here too, and made 3 dynamically loadable output plugins. Firstly, it can output a wave file using libsndfile. Then it can also output over PulseAudio. Then it can also output over Jack. Having written these three plugins, I must say that I do like these APIs. They were quite pleasant for what I wanted to do.

The source code ( is all C (except for the demo lua file) and GPL3. It should compile cleanly, but given that is has a few dependencies you might need to poke it a bit. There is a readme which tells you what packages you need, and how to drive it.

This is a bit of experimental code for working with sounds in the PDM domain. The simple explaination of what that is would probably be these pictures:

Input signal:

Converted to PDM (the three lines are independent, I wrapped it to avoid horizontal scrolling), oversampling 32 times:

So as the waveform goes up, the PDM version gets darker, and when the waveform goes down, the PDM gets lighter. This is not quite the same as PWM, as we don't have a specific time window which we try to encode, and so we generally have more transitions from on to off in a time period.

People generally seem to pass the PDM signal to an amplifier, which then spits out a bigger version of the signal, which is then low pass filtered to get back to something resembling the original. This is a simple way to build a D/A converter, and they can be quite cheap, efficient and I believe also sound good.

I thought it would be interesting to do some processing on the signal while it is still in the PDM domain. So, I built a quick test application for taking a wave-file, putting it into PDM domain, doing something to it, and then returning it to PCM and spitting out another wave file.

Things which are trivial in PCM domain (like adjusting the volume) are a bit more involved for a PDM signal. This is (a simplified form of) the code I used:

for (int i = 0; i < get_length(this_channel); i++) {
  this_bit = get_bit(i);

  if (this_bit) error += VOLUME_FACTOR;
  else          error -= VOLUME_FACTOR;

  if (error > 0)   this_value = 0;
  else (error < 0) this_value = 1;

  if (this_value) error += 1;
  else            error -= 1;

  set_bit(this_channel, i, this_value);
}

Whereas for PCM, you would write something like:

for (int i = 0; i < get_length(this_channel); i++) {
  this_value = get_value(i);
  set_value(this_channel, i, this_value * VOLUME_FACTOR);
}

However, this also means that it isn't as obvious when we are going to clip. In PCM, you know you have clipped if the value ends up out of the range you are working in. But here, we are always working with 0 and 1, and there is no way we can get a -1 or a 2. If you drive it too hard, the output will still clip, but it just won't be as obvious that it will clip until you do the conversion back to PCM.

So what the included code does is:

1. Split the input signal into high frequencies (> 2KHz) and low frequencies.
2. Put the low frequency signal into PDM domain.
3. Drive the low frequency signal using code like the code above, where VOLUME_FACTOR is big enough to make it clip frequently.
4. Mix together the high frequencies, some clean lows and some distorted lows.

And it can transform something like this into something like this. Even though it destroys the bass drum (in a cool way), the filtering makes it one of the subtler things I've put up here :)

The code isn't written for speed. On the other hand, you should look at some of the hacks used to get the filters right. I use a stack of 1st order IIR filters to do the low pass filter. Then for the high pass, I delay it by a magic number and subtract this from the original. From what I understand, this shouldn't work very well thanks to the phase response of the IIR... but in practice it seems to work (just don't change the frequency without retuning the cutoff).

The C, GPL3 code.

I was recently given the task of cleaning up some audio tracks that were recorded on a setup where there were word clock synchronisation issues between two of the boxes in the recording chain. The glitches looked a bit like this:

And they were very frequent (about 1 a second, but sometimes more or less, sometimes two within a few samples of each other).

I wrote a program that would fix them automatically, it was written fairly quickly, and fairly experimentally (I didn't know exactly which approach would be best for fixing them, so I tried several, and hacked lots of methods to see if they could be improved). As a result, the code isn't particularly nice to follow, but it is farily short, and gives a nice starting point if you have a similar problem.

The basic approach was to look at windows of samples, and calculate some statistics about them (e.g. mean and variance) then calculate those same statistics, but with the very middle sample ignored. If the statistics for these two windows were too different, then the middle sample is declared to be a possible glitch and given some sort of score for how wrong the stats were.

Then, all the possible glitches are filtered somehow, to get rid of some of the common cases for false positives. For me, this was somewhere in the middle still, there were still some glitches that were missed, and lots of false positives in noisy sounds (like 'esses'). I did experiment with an 'ess' detection thing, but the repairing was so subtle, I turned it off in order to fix as many glitches as possible, since they were very noticable.

Repairing was done by just taking the mean of the adjacent samples. This was fine, but I'm guessing if the glitches were longer than 1 sample, then this would sound bad.

Get the GPL3 C Code for it. Should compile without anything fancier than gcc and make.

---

This is another non-linear audio filter. The idea is that you take the original signal (which is a line of zero thickness in a time-amplitude scale) and trace over it with a thick elliptical brush. This gives an area which follows the shape of the line, but some of the little kinks will be taken out, and the peaks will be rounded. We then have to convert this area into a line, this is done by just taking a point in time and finding the average of the minimum and maximum amplitudes for that time. There are other ways of doing this step which will give different results, but I used this one because it is simple.

In my mind, I was expecting it to be a sort of low-pass filter, as the high frequency stuff would be blurred out by the "brush". What actually happened was that it has more of a comb-filter like effect. If you run it on pink noise you get a frequency response curve that looks like this:

(without filtering, this wouldn't have all the bumps in it).

Since the filter isn't linear, you can't generalise and say it will have a comb filter effect for all input signals, but for the stuff I've tried it on, it generally gives a comb filter like effect. On more aggressive settings, certain notes will sound much louder than other notes. I've chosen two examples that I'd think are usable, and while they are quite obvious, they aren't so aggressive as to add the really noticable loud notes:

Original signal, Paint Filtered 1, Paint Filtered 2.

Get the source (C, GPL3, includes makefile, and is a little bit dumb in one spot where instead of using a circular buffer I shift every value in a buffer across a spot).

Another quick post (quick in terms of me writing this bit - not much proof reading or going out of my way to make things clear), this time with some toys to play with.

Compressor

The first is a cool compressor, which is way more general than anything I've seen. (This is talking about dynamic range compression, not compression in the file size domain like mp3).

For those not in the know, a compressor will try to increase the volume of the quieter signals, and keep the loud signals loud (hence it compresses the dynamic range). There are a couple of uses for it, one is destroying the sound of a song (see commercial radio), another is for making something sound better. This compressor should be general enough to do both.

Basically, the compressor will track the loudness of the signal passing through it, and once it goes beyond a certain threshold, it reduce the gain of the signal by some ratio. The exact time that the gain is reduced is controlled by the attack and release (in order to preserve transients, the gain reduction won't kick in straight away, and once the signal drops below the threshold it should also slowly return to the original signal).

This compressor has a very general method for tracking loudness. It takes the power mean of the samples inside a window (this gives us two parameters - the size of the window, and the type of power mean (a power mean is the generalisation of most means, so you can do an arithmetic mean, rms, or anywhere in between (or outside))). Power mean is then low pass filtered, which prevents it from jumping around lots (another parameter).

So then for every loudness, we have a response curve which maps an input value to an output value. e.g. we could have a curve that halved all the inputs, which would then halve the amplitude of the output, or we could have something that followed a square root curve, or any other curve. Since there are lots of different loudness levels, we can just specify a couple of these response curves, and assign them to loudness levels, the compressor will then interpolate between these curves to get the actual curve for a particular loudness level.

Frequency Splitter

The next toy is a frequency splitter. This takes a wave file and splits it into a bunch of new wave files each with different band pass filters applied to them. The filters aren't particularly steep edged, so you will get plenty of overlap between the output files in the frequency space. For some things this is bad, for others I can imagine it would be good. I made this as a step to make the compressor a bit more general (so it could be a multi-band compressor). Converting the compressor to being multi-band using this should be trivial, the only reason I didn't do it was because I couldn't decide what the interface would look like (that is, the code interface, not the user interface).

It just uses a whole bunch of low pass filters, and then subtracts the outputs from adjacent filters to get particular bands. Writing a simple low pass filter is super easy (output = a * input + (1 - a) * last_input) and different values of a will give you different frequency responses.

64-bit and Naming Conventions

So in the time between writing the audio library I use and now, I have changed my home coding convention (I did it at the time to ensure I wouldn't confuse work code with home code, now I don't know what the new naming convention will be as I start tomorrow). Also in that time, I moved from being a person who used x86 only to a person who uses x86, Ix86-64 and armel architectures. Some of the code in the libary was assuming 32 bits, now it should work in 32-bit, and does work in 64-bit (testing it is boring).

I also can't decide what type to use for referring to a chunk of memory that I know I'm going to access by byte. I previously used a mixture of char* and unsigned char, now I've changed to void which forces me to be really explicit in places. I'm not sure if I'll stick with this, but that is how it is now.

Download

This should compile in amkel, but there is also a make file. Although I don't like writing them, there is the big advantage that make is everywhere, so for the moment I'm going to write them... until I find/make a better solution.

compress and fsplit (C, GPL3)

This is a fairly simple program for doing frequency tracking. Frequency tracking is where you take a sound and try to follow the dominant pitch of that sound. For lots of sounds this dominant pitch doesn't really exist (e.g. a cymbal sound, or multiple sounds playing at once) so I chose to measure the frequency, confidence and whether or not a dominant pitch exists.

There are heaps of ways of doing this, I think at the moment my pet favourite is to use a filter bank. A filter bank is just a whole bunch of bandpass filters covering bands over a bit of the frequency spectrum. So one might cover 50Hz to 60Hz, the next from 60Hz to 75Hz etc. When I first started doing stuff in the frequency domain, I really wanted to make sure that there was no overlap between such bands. After doing lots of stuff in this area, I think I'm of the opinion that it isn't really possible, and even if it was it wouldn't be that good. So, while each filter tries to cover a specific area, they do overlap somewhat.

The magic numbers I've been using mean that I have 200 filters covering the spectrum from 50Hz to 2KHz (spaced in a geometric series, so they are closer together down the bottom). The filter design is pretty simple, I have 201 low pass filters, which work by making each sample a running average of the previous samples (i.e. f[t] = f[t-1]*alpha + f[t]*(1-alpha)) where alpha is chosen to put the cutoff frequency in the right spot. I then apply the filter to a section of sound (512 samples long) and subtract the results to get bandpass filters. So I might subtract a 200Hz lowpass from a 210Hz lowpass to give a 200-210Hz bandpass filter. I then just measure the RMS energy in each band.

Rather than just finding the maximum energy band and declaring it the winner, I keep something resembling a probability distribution over all the bands. It starts out with uniform probability, and then each probability is multiplied by the amount of energy present in that band, and a constant is added (so that we don't get stuck at 0). After this we find the total of all the numbers (which in general won't add up to 1 now) and remember this number as the confidence. We then normalise the numbers, and the largest one is considered the winner.

If the winner is at the high end of the spectrum, then we say it is unpitched. It seems right in practice.

To test it, I made a program which will take a wave file and add a sine wave over the top of the pitch it detected. I don't have many sounds where it is just a single instrument, so I don't have anything particularly good to test it on. But, it seems to go ok, just not when there are multiple instruments.

You can download the pitch tracker here (C code, GPL3, builds in amkel). And you can get some wav files to test it on from youiseek.com (CC Attribution Noncommercial Share Alike) (a local band... not particularly good for testing this on, but given that my wave file loader is so picky I just went for the first thing that worked).

To build it, just compile together all the .c files with -std=c99 (or amkel pt.c). Run with the first argument being the wav file to run on.

You might also notice that I've changed my coding style. Previously I named variable and function identifiers with camelCase, now I use under_scores (I know it is one word). It makes it ugly when I interface with old code (the sound library), but I think changing/avoiding the sound library will be less work than changing/avoiding gtk. Also, it makes it very easy for me to tell which things I develop at home, and which things I do at work (since I use the other convention at work).

Most audio filters are linear, which means that the maths behind them is pretty well understood (not by me, but by smart people), and their behaviour can be predicted, and duplicated easily. There are even devices you can buy that will let you plug an expensive filter into it, give it a moment, and then it will have a perfect duplicate of that effect (see convolution reverbs (and a reverb is just a type of filter)).

But, not all filters are linear, and in my opinion, the interesting ones aren't (like the sorting filter). Mainly because you get a bit more surprised when you hear them applied to different sounds.

So todays filter is a smoothing filter. We start out with a sound wave, and we try to approximate the wave by just picking a few key points on it, and fitting a curve through those points. I've chosen to do it the super inefficient way:

1. Approximate the whole sound wave with 2 points, one at the start and one at the end.
2. Find the place where the approximated wave is the most different to the actual wave.
3. Refine the approximation by adding a new point at this spot with the most error.
4. Go to 2, until we are close enough to the original wave.

So, for this to work, we need to have the entire sound available (you can't do this on a live sound - though you could do it on short windows of sound), and a fair bit of computer time (depending on how long the sound is). The actual effect it produces is probably best explained by a picture:

(top is original, middle is quite filtered, bottom is very much filtered)

And of course, the actual clips (as FLAC):

Original Filtered Aggressively Filtered

I interpolate using a cosin-ish thing, based on the two points either side of the point we are interpolating. Just because it is simple, and it isn't linear.

I'm playing around with a simple program to synthesize sounds, and I've managed to make a sound which looks just like a perfect sine wave, but sounds like a sine wave with some dirt (maybe a bit like a harmonica). I call it stealth distortion. This is what the wave looks like:

This is what it sounds like (FLAC. ~100KB)

I put an envelope around it, so you get half a second of fade in (no distortion), one second of distorted sound and half a second of fade out (no distortion). I'm pretty confused by the whole thing, because I didn't intend for it to sound like this at all.

You might think that I've snuck in some odd harmonics that are subtle so you can't see them. Well, I'd like to think so too - but here is the spectrum of the distorted bit (narrowband filter of 8192 samples, using a Hanning window):

It is late, so I'm assuming tomorrow I will realise that I'm an idiot.

This is a kind of follow up to the previous thing about sorting filters. As well as doing the effect to images (which I did only because I had an easily accessible image library) you can apply it to sounds (which I have now done, since I have an easily accessible sound library). Actually, it makes a bit more sense when you apply it to a sound, and it is easier to see what is going on.

For those who didn't understand what I was doing, it will hopefully make more sense here: you take the waveform of a sound (which is just a collection of numbers) and put the numbers into groups of a fixed size (say groups of 10). In each of these groups, you sort the numbers. This will give you a wave that (typically) has a discontinuity every 10 numbers at the boundary between groups. So we then repeat the process 9 more times, with the groups slightly offset each time, now each of these new waveforms has a discontinuity every 10 numbers, but all the discontinuities are at different spots, and when we mix together all of these waveforms, we get something that is smooth (not true in general, but true in this case). This is what I call a sorting filter. So what does a sorting filter actually do to a sound?

Basically, it will will harshen high frequency sounds. If you have a sine wave that is of a high enough frequency, the sorting filter will turn it into a triangular wave, while a lower frequency sine wave gets through without any trouble. Since the filter is non-linear(*), this doesn't mean that the filtered high+low frequency sound will be a high frequency triangular wave and a low frequency sine wave.

I also modified the filter so that when you sort, you can sort either ascending or descending depending on whether the first sample in this window is above or below the last sample. This gives us 2 variables we can tweak with the filter - the width (how wide the window is) and the sorting policy (always ascend, always descend, same as first/last, opposite to first/last).

So I'm not quite sure how to demonstrate the workings of it. I had four 10 second long clips of audio which I've used as test dummies. I tried to pick different sounding things, you get 3 points for every song you know. The current best score is 12 (by me). I then filtered them, 8 times - 2 different window sizes (10 and 50) and 4 different sorting choices (always ascend, always descend, follow trend between first and last, disobey trend between first and last).

So this recording is just a few of the demonstrations put together in this order (no S or K samples in here, the S ones sound pretty bad, there are some ok K ones)

T(original) T(ascend10) T(oppose10) M(original) M(ascend10) M(follow10) M(oppose10) M(follow50) M(oppose50)

Sample Reel

Things to listen for:

1. Listen to the hi-hat (or lack of) in the T samples.
2. Note how the kick drum in the M samples stays big.
3. Both of the ascend10 filters pretty much destroy the hi-hat.
4. The snare drum in M(follow50)

You will probably hear these and think a lot of them sound rubbish, and there is no possible use for them. You wouldn't take one of these filters and apply it to an entire track, but I can definitely imagine mixing in a little bit of one of these to a track to add something different.

If you want to hear the rest of the sounds, or play with the source (it compiles cleanly with amkel and is GPL2) then get the whole sorting filter package

(*) A linear process is one where if you apply it to two signals separately and then mix the results together you get the same result as if you mixed the two signals together and then applied the process. (for the maths people, f is linear if f(a) + f(b) = f(a + b)) (**)

(**) This isn't quite the textbook definition of linear process. They also include the requirement that if you apply it to a signal which has been multiplied by a constant, then the result is the same as if you applied it to the original signal and then multiplied the result by that constant (or f(kx) = kf(x)). But to me, the number of interesting things where the first condition is true and this condition isn't is so small that I don't really bother caring about this one.