The SID GUTS Review – Getting retro on my ass

ALM003-clipped-680h-400w

The SID GUTS is a fun little module from ALM that brings a SID chip into Eurorack. If your knowledge of ancient integrated circuits is less than stellar I’ll have you know that SID (Sound Interface Device) is a classic sound generator chip that was used in the Commodore 64 home computer.

The SID GUTS module provides CV-control of an actual SID chip. While it doesn’t utilise all the capabilities of the SID chip;
for example the SID has three voices and the SID GUTS only uses one; the SID GUTS is a convenient way of bringing some lo-fi chip-tune sounds into your modular. The module comes without a SID chip, it has a socket in which you mount a SID that you presumably already own. Real SID chips can be hard to find, and draw a lot of power, so an alternative option is using the SwinSID which is a modern SID clone. The SwinSID can be ordered with the SID GUTS and it consumes a lot less power than a real SID which is why I picked a SwinSID for my SID GUTS. The only downside of the SwinSID that I can see is that it doesn’t allow the filter to process external signals. This is no big deal for me because I have a bunch of other filters in my rack already and the SID filter isn’t that impressive to begin with.

So what are the capabilities of the SID GUTS?

Oscillator

As mentioned the SID GUTS provides a single SID voice. The oscillator is a wave-table oscillator with pulse, sawtooth, triangle and noise waveforms. The waveform can be selected manually or it can be cycled from a CV input which can create some pretty glitchy noises. The pitch is controlled by a standard V/Octave CV input. Without modulation or arpeggiation the only waveform that clearly says “C64″ is the noise — it is wonderfully steppy and immediately brings to mind a million explosion sound effects from old C64 games.

Noise :

Sawtooth:

Triangle:

Pulse:

Wave-type modulation through CV:

Filter

The filter is an analogue resonant multi-mode filter which is switchable between a 12dB/octave low-pass, a 12db/octave high-pass and a 6db/octave band-pass mode. The high-pass and low-pass modes can also be combined into a band-reject (or notch) mode. The resonance and the cutoff frequency can both be voltage-controlled. As previously mentioned the filter can be used on external signals but only if you have a real SID chip in the module. The filter resonance is quite subtle and, I’m told, varies a lot between different models of SID chip. With the SwinSID the resonance is audible when cranked up to max but I would not go so far as to say that it is clearly audible. At least to not my amateur ears. If I A/B a filter sweep with the resonance at 0 and then at maximum, I could hopefully tell which is which. On the other hand, if I was presented with a single filter sweep and had to tell if the resonance was on or off I’m not sure that I could deliver an accurate verdict.

Low-pass filter sweep with resonance:

PWM

The width of the pulse wave can be changed with the PWM knob or modulated through a CV input. This sounds gorgeous and is, for me, the best bit of the SID GUTS.

Pulse Width Modulation:

Modulation Oscillator

The SID GUTS uses one of the remaining two oscillators in the SID as a modulation source. The modulation oscillator can be used for oscillator sync — the main oscillator waveform is reset at the frequency of the modulation oscillator — or for ring modulation; where both oscillators are set to use the triangle waveform and the result is the sum and difference of the carrier and modulator waveforms.

The SID triangle wave being ring-modulated by another triangle wave.

Oscillator sync:

Modulation type cycled through CV:

 

In use

Modulating the oscillator frequency or the pulse width with a CV signal works fine for slow modulation rates. Since the SID only updates its inputs at 50Hz, modulation rates close to or above this rate doesn’t work as expected and you can forget about FM or audio-rate PWM.

I use the Expert Sleepers ES-3 module as my computer-modular interface and it requires a calibration step every time it is started. The calibration consists of the ES-3 sending a range of control voltages to the V/Octave input of the oscillator being calibrated and then detecting the actual output frequencies that are generated by those voltages. This process does not work well with the SID GUTS. In fact, I could not get it to work at all. I suspect that the waveforms generated by the SID are too noisy for the ES-3 calibration plugin to be able to extract a solid frequency from them.

Looking at the waveforms with an oscilloscope it is apparent that there is a constant signal with an amplitude of 2.5V and a frequency of around 1.25kHz superimposed on the wave. This tripped up the auto set function on my oscilloscope and I would guess that this is what confuses the ES-3 calibration as well.

This is the triangle wave from the SID.
If you look closer you see the overlaid signal.
This is the same closeup of the triangle wave from a TipTop Audio Z3000 oscillator. Clinical to some, clean to some.
This is the same closeup of the triangle wave from a TipTop Audio Z3000 oscillator. Clinical to some, clean to some.

The SID GUTS excels at chip-tune-like bleeps, bloops and chirps. Playing rapid arpeggios to simulate chords with a single voice still gets you that Rob Hubbard feel which I presume is the reason you would want a SID-chip in your modular.

Arpeggio:

I tried to play the theremin part of the Doctor Who Theme but was bested. A 2-octave portamento glide sounded truly horrible, and not in a good way.

Building it

I really didn’t have room for the SID GUTS in my modular budget so I bought it as a kit because that allowed me pretend that it was a crafts project instead. ALM sells (or sold) complete kits with PCBs, the panel, all the knobs, buttons and other components, a very good build instruction and, optionally, a SwinSID. The software is open source and available for download so if you want to brew your own firmware that is possible too.

This is the SID GUTS kit prior to being massacred by an amateur.
This is the SID GUTS kit prior to being massacred by an amateur.

Building the kit was great fun. It pays to have some experience soldering because debugging a badly soldered pin can be seriously tricky. I’m by no means an experienced solderer and although I have dabbled, in hindsight I should have done a couple of practice runs to brush up before starting on the real build. Some of the solder joints, particularly the early ones, ended up a little anaemic but the only really bad one was a pin that I missed completely. Happily that was fairly easy to see once I gave the board a careful inspection.

SID PCB
This is where the SID goes. You did remember to get one, didn’t you?
The pots and jacks must be aligned with the panel before soldering them onto the PCB.

I did make one other mistake. The SID GUTS consists of two boards that are joined together, sandwich style, using four single-row pin-headers. Following the instructions, I soldered the male and female parts of the pin-headers onto their respective boards separately. However, I ended up soldering the female parts at a slight angle. This made joining the two boards a difficult and nerve-wracking experience — I had to use so much force bending the pin-headers to get the pins to fit that I was afraid I’d break the boards in half. Naturally the previously mentioned unsoldered pin was at this point A) undiscovered and B) unreachable so I had to separate the boards and rejoin them an extra time to fix that. In hindsight I should have joined the male and female parts of the pin headers together first, then put the boards together, and then soldered the pins to the board. Lesson learned.

Almost done. Mating the two PCBs should be no trouble at all really...
Almost done. Mating the two PCBs should be no trouble at all really…

Conclusion

So what do I think of the SID GUTS? It was fun to build. It looks cool and it certainly sounds retro. Is is useful? For sound effects, definitely. I don’t find myself using it much for melodies due to tuning issues but I would not definitely blame that on the SID GUTS — it could just as well be down to pilot incompetence. When it comes down to it, can you really resist having a SID in your modular? If you can then you’re probably from a different generation than me.

A Brief Guide to Wireless Security

Illustrations by Jonas Ekman.

WiFi security is a mess. The subject has an acronym ratio that would make a NASA space shuttle manual proud and it seems like most areas of it overlap in some non-obvious fashion. I find that understanding why something strange is strange helps, but the “why” is never clearly explained in the various standards and design guidelines that constitute the pillars of the WiFi security morass. Yes, I prefer my metaphors mixed and non-sensical.

So the purpose of this text is to try to bring some clarity into the alphabet soup — without delving too deeply into the details — while also illuminating some of the reasons things ended up the way they are. There are a couple of good texts that explains the nitty-gritty but if you just want to know how the hell TKIP relates to RC4 or what the difference is between WPA2 and RSN: read on.

Houston, we have a problem.
Houston, we have a problem.

A concept that is important to keep straight is the difference between authentication and confidentiality. Usually people think “encryption” and are satisfied with that, because an encrypted communications channel of any quality provides both authentication and confidentiality. But for this discussion it is important to remember that the two concepts are different. Authentication means that you know with whom you are communicating. Confidentiality means that only the intended recipient can understand your communication. Clearly, confidentiality without authentication is a non-starter. After all, if you don’t know with whom you are communicating, you don’t know who is reading your messages. Authentication without confidentiality however, is conceivably useful. For instance, if you don’t care about keeping your messages secret, but want to prohibit your neighbour from freeloading on your WiFi network, then you might implement authentication without confidentiality.

I’ll be using the words “access point” and “station” a lot. An access point (or AP) is the WiFi “base station” — it might be a wireless router or an Apple TimeCapsule or a dedicated access point. It is the thing that provides WiFi access (hence the name) to a wired network. A station is the thing that connects to the access point — a PC or a tablet or a phone for example. If you want to be strict about it an access point is also a station but outside of the actual standard “station” usually means “a WiFi-device that is not an access point”. That is also what I will mean by “station”.

The need for WiFi security

Lets begin at the bottom: the ethernet layer. In a normal ethernet network, data is transmitted in the clear over physical wires. And people seem to be fine with that. Anyone with physical access can eavesdrop on all the traffic, but on the other hand anyone with physical access can cart off your whole computer. In a WiFi network, traffic is ending up all over the place. You can easily eavesdrop on WiFi traffic from a nearby location. You can also easily send fake traffic into the WiFi network. The designers of WiFi thought that this was a problem (because the only way to live with the dullness of standards work is vigorous porn surfing), so they added some security measures to the standard: something called Wired Equivalent Privacy, or WEP.

WEP

WEP is pretty simple to understand. It deals only with the ethernet layer. It provides two authentication methods called Open System (which is actually no authentication at all, so that makes sense…) and Shared Key. It also provides confidentiality by encryption using the RC4 cipher. Note the difference between confidentiality and authentication — it is perfectly fine to use Shared Key authentication but no encryption. If your WiFi adapter is steam powered perhaps traffic encryption is prohibitively slow but since the authentication procedure happens only once you could probably live with that being a bit sluggish.

I should mention here that Shared Key authentication has a flaw that makes it easier to crack a WEP encryption key. For this reason you should never use Shared Key authentication. If you must use WEP, use Open System authentication with traffic encryption (and also be aware that WEP as a whole is easily cracked these days).

Both the Shared Key authentication system and the RC4 traffic encryption needs some sort of key to work. The WiFi standard, IEEE 802.11, specifies what the keys should look like but not (yeah, I’m laying down the law here) how you get them. This is important. How the keys end up in the encryption engine is “out of scope of the standard” which is standardese for “if you ask us about this we’ll set fire to your car”. What this means in practice is that it is up the designer of the WiFi adapter to decide on a way to enter encryption keys into the system. It might be via a keyboard (sensible), via punch card (impractical) or via Aldis lamp (silly).

WEP keys can be 40 or 104 bits long and honest people calls that WEP-40 and WEP-104 while people who can’t count calls it WEP-64 and WEP-128 (because that sounds more secure).

WEP, if Swedish was a cipher. And let's face it, Swedish is a cipher.
WEP, if Swedish was a cipher. And let’s face it, Swedish is a cipher.

It is worth mentioning that WEP keys are strings of binary data, not “passwords” in the usual sense (like your mothers maiden name for instance). The security professional, being a bit of a bastard, calls what you normally think of as a password a “pass phrase”. People normally find it easier to remember some kind of word than a string of hexadecimal digits so it would be nice to have some way to turn a pass phrase into a WEP key and there are indeed many ways that this can be accomplished. Unfortunately, translating a pass phrase to a WEP key is “out of scope of the standard”. Which means that any access point might do it their own way, resulting in incompatibilities and chaos. In fact, what most manufacturers do (if they support pass phrases at all) is allowing fixed-length pass phrases where the ASCII representation of the pass phrase is used directly as the key.
This is pretty simple to implement but it significantly weakens WEP because it effectively restricts the number of actual bits in the key which makes it easier to crack.

As you may have gathered by now, WEP is a bit crap. The standards guys soon started getting angry notes from their bosses about the porn surfing again. Being engineers, they immediately set to work trying to solve the problem properly by introducing lots of acronyms.

In all fairness it should be stated that WEP never was intended to be a super-secure solution. As the name implies it was intended to merely provide “wired-equivalent” security. Something some people, red faced and with veins pumping at the temples, will loudly denounce as “no security at all!”

802.11i – RSN

The 802.11i standards team meant to fix the security problems present in WEP. They defined something called a Robust Security Network, or RSN, by which they mean a WiFi network using the new security measures described in the 802.11i standard, as opposed to pre-RSN security measures (meaning WEP). In order for us to have another acronym a RSN is established through a RSN Association, or RSNA.

802.11i defines two new types of confidentiality, essentially types of encryption although calling them types of encryption can lead to some confusion as we will see.

The first type is called Temporal Key Integrity Protocol, or TKIP for short. TKIP is intended to be a transitional encryption type (translation : TKIP is also a bit crap, but less crap than WEP). TKIP uses the same type of encryption algorithm, RC4, as WEP which is what’s transitional about it — it is possible to implement TKIP on hardware designed for the original 802.11 standard, hardware that normally has RC4 encryption implemented in… er… hardware.

Why not call TKIP RC4 you ask? In fact, why not call WEP RC4 as well and get rid of two acronyms at once? Well, as the P in TKIP implies, TKIP is really a protocol. It specifies what the keys should look like. RC4 is a stream cipher and the data to be encrypted is packet based so the protocol also specifies such things as how the data to be encrypted should be “chunked” before being encrypted, how to pad any chunks that are too small, and so on. These details are different for WEP and TKIP, thus the different names. Another way of looking at it is that if the hardware supports RC4 encryption, TKIP and WEP can be implemented in software on top of it.

The second type of confidentiality is the splendidly named Counter Mode with Cipher-Block Chaining Message Authentication Protocol, or CCMP (if you don’t think the acronym makes sense, CCMP is actually short for CTR with CBC-MAC Protocol; “My acronym is too long guys. This calls for a new acronym!”). In the same way as TKIP is a protocol using the RC4 encryption algorithm, so CCMP is a protocol using the AES encryption algorithm.

CCMP — now that is a confidential type of confidentiality if you ever saw one. Rest assured, your porn is safe here! At least for the moment. At least as far as I know.

One important aspect of both TKIP and CCMP is that they use so-called Temporal Keys. A temporal key is only used for a short time and is then discarded. Limiting the amount of data being encrypted by the same key makes the key harder to break. Two parties that are to communicate using TKIP or CCMP needs to do a little dance called a “four-way handshake” during which they negotiate a set of temporal keys that are used only for this session. The four-way handshake derives the temporal keys from the “normal” key. This “normal” key is called a master key and is normally equivalent to a pass phrase (with some extra processing to make a binary blob out of it). There are two types of master keys. Pairwise Master Keys are used for unicast data and Group Master Keys are used for multicast data. Avoiding acronyms is just not serious security so these keys are called PMKs and GMKs for short. For the sake of this discussion the pairwise and group master keys behave in the same way so I’ll just say “master key” and you can substitute it with “PMK or GMK” if you like. How the master key ends up in the system is… you guessed it, “out of scope of the standard”, but it is commonly entered by the user (or the users children).

EAPOL

There is a slight problem: the four way handshake happens before encryption can start because the encryption needs the key that is the result of the four way handshake. This means that the four way handshake must happen in the clear. This is such a massively complicated concept (“First unencrypted? THEN encrypted??? My God, man! I need to go lie down for a bit.”), that we apparently needed another standard for it. 802.1x (not a typo, 802.1x is part of the 802.1 family of standards, not the 802.11 family, and is applicable to other types of networks as well) essentially describes, in a very complicated way, a model where you communicate using two ports. One encrypted port and one unencrypted port. These ports are opened and closed magically.

I’ll summarise the important points about 802.1x here: A 802.1x port has nothing to do with ports in a switch, ports in a firewall, ports in a storm, disgusting beverages or the left side of a boat. To quote Monthy Python and the Holy Grail : “It’s only a model.”. 802.1x defines a packet type, called EAPOL (for EAP Over LAN (or Extensible Authentication Protocol Over Local Area Network (are we LISP yet?))), which is used to negotiate authentication parameters in the clear. The four-way handshake is performed using EAPOL packets. Once a temporal key has been generated then traffic starts being encrypted. When the 802.1x standard says “sending to the unauthenticated port” it means sending data without encryption and when is says “sending to the authenticated port” it means sending data with encryption. The unauthenticated port only allows EAPOL packets, all other packets are discarded.

Establishing a RSN assumes that that both parties (the access point and the station) have copies of the same master key, which has been put there through occult means. The station connects to the access point using the Open System authentication method (in other words: the connection is unencrypted and unauthenticated). When the connection has been made a program called a “supplicant” on the station performs the four-way handshake with a program called an “authenticator” on the access point. As we’ve already seen this handshake is performed by exchanging EAPOL packets. When the handshake has been completed then the station and access point each have an agreed-upon temporal key which they then start to use. If you sniff the WiFi traffic when a RSN connection is set up you can see the EAPOL packets in the clear and then encrypted packets after the handshake has been completed.

A 802.1x four-way handshake, juvenile style.
A 802.1x four-way handshake, juvenile style.

One consequence of the use of temporal keys is that if you want to decrypt a TKIP or CCMP connection using a packet sniffer like Wireshark, it is not enough to have the pass phrase. The sniffer has to have access to all the packets in the four-way handshake because that information is necessary to generate the temporal key from the pass phrase.

WPA/WPA2

Now, you’d be excused for grumbling “Why are you going on and on about this RSN thing that no-one has ever heard of? I want to know about WPA.” There’s actually a funny story behind that…

Ok, the story is not that funny. What happened was that the IEEE group that developed the RSN-specification (called, as mentioned, 802.11i) took a long time to agree. The 802.11i specification spent years and years in “draft” status which meant that you could read it and comment on it but it was not approved as a finished standard. In stepped the WiFi Alliance: a group of super-villains^H^H^HWiFi Manufacturers that banded together to try to make WiFi easier to use. The WiFi Alliance actually owns the trademark on the word “WiFi”; if not for them you would have called it “802.11b and 802.11g and 802.11n and 802.11ac” instead, so be grateful.

The WiFi Alliance recognised the threat to porn surfing posed by the weakness of WEP and the delays of 802.11i, so they took action. By picking some of the most useful bits of the 802.11i draft they could set their own intermediate standard that manufacturers could start to use while waiting for the 802.11i working group to extract their digits from their posterior apertures. The result was named Wireless Protected Access, WPA for short. WPA is pretty similar to RSN but lacks some features such as pre-authentication. It uses the same four-way handshake, the same crypto (although CCMP is optional in WPA) and the same key types as RSN.

Access point capabilities are announced in something called “Information Elements”. RSN-capabilities are announced in a special RSN information element but WPA could not use this since it would conflict with 802.11i if that ever settled. Therefore WPA had to use a special information element that had been set aside for vendor-specific extensions: the so called Vendor Specific Information Element. The vendor is identified in the vendor-specific information element by including a OUI prefix and for some reason WPA uses the OUI prefix belonging to Microsoft. Don your tin-foil hat here.

The WiFi Alliance has also specified WPA2 which is based on the final 802.11i standard. WPA2 includes all parts of 802.11i that are mandatory and is fully compatible with 802.11i. So you might variously see security settings being named “802.11i”, “RSN”, “WPA2″ or as combinations thereof — they all describe the same thing. If you want to be strict about it, 802.11i _defines_ RSN and WPA2 uses 802.11i to define a limited but compatible RSN, so the proper name for this type of security would really be “RSN”. Since WPA isn’t compatible with the RSN specified by 802.11i (because it uses a different information element) it cannot really be called a RSN (in my humble opinion). Device manufacturers might be forgiven for presenting the options as “WEP, WPA or WPA2″ instead of “WEP, WPA or RSN”.

WPA-Personal and WPA-Enterprise

You may remember me claiming that the way the master key is entered into the system is out-of-scope of the standard (how could you possibly forget?). So how would you actually manage the master key in the real world? The simplest way, which is probably the most common, is as a so-called Pre-Shared Key. The Pre-Shared Key is a pass-phrase that you enter into your access point on the one hand and into the devices you want to connect to the access point on the other hand. It turns out that you sometimes need a more sophisticated solution though. For example, say that you manage an company with a few hundred employees. Every person who needs to use your WiFi network needs to know the pass phrase. This is already pretty bad because every employee can decrypt everyone else’s traffic when they know the pass phrase. Even worse, if you have to strap a minion to an ejection seat after they’ve sold company secrets to a competitor you need to change the pass phrase everywhere to keep him or her out.

Wouldn’t it be pretty convenient to have a key management system where you could assign pass phrases to individual users in a centralised fashion and then use those pass phrases to generate unique master keys for users as they try to connect to the WiFi network? Is that a “Hells Yes!”? I thought so.

The master key is managed by the key master. Or a RADIUS server.
The master key is managed by the key master. Or a RADIUS server.

In order to support a centralised key management system the access point must allow the station to somehow communicate with a third party for the key exchange : the station needs to verify the user pass phrase with the key management system. This is where the layering abstraction gets a bit muddled. It would be nice and clean if the 802.11i standard could just leave the whole key-management issue to some upper layer but unfortunately it can’t — it must provide a means for conveying the authentication traffic between the station and whatever key management system exist in the network. This traffic doesn’t really belong in the “ethernet” layer as it could span many different network technologies. If this makes you feel as if 802.11i is trespassing on OSI layers that it should leave well enough alone, I understand you. On the other hand, if the 802.11i working group had tried to fix it some other way they would have had to create a bunch of extra architecture-astronaut standards and there would probably have been XML in there somewhere. Let us count our blessings and live with the warts.

As fortune would have it, supporting authentication in a network is basically what our old friend 802.1x was designed for so 802.11i uses that. In 802.1x the password management system is called an authentication server (AS). The authentication server works together with the supplicant (on the station) and the authenticator (on the access point).

The role of the authenticator is different in the pre-shared key case and in the 802.1x case. With a pre-shared key the authenticator is the party that negotiates the temporal key generation with the supplicant. With 802.1x the authenticator only acts as an intermediary: it allows the supplicant to communicate with the authentication server using EAPOL while blocking all other traffic to prevent the station (which is still unauthenticated, remember) from accessing the network in any other way.

Using 802.1x for key management is a great idea and also a cause of terminological befuddlement. You see, the 802.11i standard calls the pre-shared key system “PSK” (for Pre-Shared Key), which is fine. But then it calls the password management system “802.1x” which is technically correct but confusing because the pre-shared key system also uses 802.1x for the four-way handshake.

In the interest of keeping things simple (an admirable goal) the WiFi Alliance left out the key-management parts of 802.11i from the “basic” WPA specification; what they called “WPA-Personal”,
sometimes also called “WPA-PSK”. Instead they defined an enhanced version of WPA called “WPA-Enterprise”, sometimes called “WPA-802.1x”, which includes the key-management support.

The most common key management system is called RADIUS. RADIUS is short for Remote Authentication Dial In User Service which tells you that RADIUS is not the most hip acronym on the block. Dial In was exciting back when porn was distributed on magnetic tape. In 802.11i RADIUS is not the only option for key management, other systems are allowed. However, WPA-Enterprise requires RADIUS and it is so common that when you speak of “RSN with 802.1x” (as you do) then RADIUS is often assumed.

The Brass Tacks

The brass… the… what? What? “Brass tacks”? Get it together English, that makes no sense at all.

What did we learn today? We learned that a RSNA using TKIP (RC4) or CCMP (AES) via PSK or a 802.1x AS is established through a WPA or RSN IE from STA to AP so the PTK/GTK can be derived from the PMK using EAPOL in IEEE 802.11i. FYI: that SOB is a real PITA.

Also, I lied when I titled this post “A Brief Guide…”

If you made it this far you deserve a cookie.
If you made it this far you deserve a cookie.

The Case for Simpler Tools

George R. R. Martin famously writes his books using WordStar 4.0 on a DOS-machine. He has a “real” computer on which he reads email and surfs the web and does all the other things required of modern man. His writing computer has no network and can only do one thing. It doesn’t try to spell check for him. It doesn’t pop up alerts that an email has arrived, or that Stephen King wants to chat. A focused tool allows him to focus. I think there’s a lesson here for all of us: when you focus you end up killing all your protagonists. No, that’s not it. When you focus you start thinking that the phrase “her nether lips” has no cheese in it? Not that either. Is it that when you focus you can write endless amounts of stuff? Yes! Exactly! Focus increases productivity.

Focus level one million.
Focus level one million.

I’m going to go ahead and postulate that focused tools help you be more productive, based on no evidence whatsoever. I feel this to be true and I find myself gravitating towards tools that are designed to do one thing well and that otherwise gets out of my way as much as possible. I’m not trying to describe an objective reality: when I say that IDEs are evil I don’t mean that you are a bad person if you like IDEs. I’m just saying that I hate you and that your lineage is questionable.

Computer programs used to be very focused. This was because the computers of old were very limited so their programs had to be simple.  As computers and their operating systems got more powerful and more and more tasks were computerised, it became possible to try to do more than one thing at a time. And we did. I’m inclined to think that it is non-controversial to say that perhaps we overdid it.

I think I can detect a slight resurgence of the focused tools though. Smartphones and tablets have brought some of the old focus back. An “app” commonly occupies the whole screen and doesn’t trick you into trying to do five different things at once. And some people — like old George R.R. — has started to recognise that multitasking isn’t helping us do what we want to do (unless “what” is stressing out and dying at 45). Multitasking, by definition, removes focus. Less focus, less actual work gets done. As I’m sure is proven by a paper somewhere.

A rant for your entertainment

Most IDEs, or Integrated Development Environments (in case you’re reading this for the jokes), illustrates the lack of focus that I mean. I don’t know about you but when I code I want to see code. As much of it as I possibly can. I have a seriously defective short-term memory so I’m constantly jumping around the code, checking what a function is doing; what a parameter is supposed to be; what that struct field name is called. Sometimes I need to see some trace output and two or three source files at the same time. To me, screen real estate is precious.

An IDE commonly splits the screen into a series of panes or windows. There’s usually a project window on the left side where you can see files and folders. Why the heck would I want to see files and folders while I’m programming? If I’m looking for a file then a window for that could pop up, but it’s not like I’m constantly checking the file names while I’m writing code. On the bottom there may be some sort of message or status window. The status is that I’m programming! Leave me alone, message window! On the right side you might find a debugger window or a symbol browser or something like that. A GUI can be really great when debugging but that is only like 90% of the time so please get it out of my sight!

In the middle, fighting for relevance, is (if you are lucky) a poor source code editor in which you do your actual work. In the end your sweet 24″ monitor is reduced to a 17″ editor window and that is just no way to live. I realise that you can probably apply XML voodoo somewhere and get most IDEs to look like EMACS and that is great. My point is that viewing all the available tools all the time may look cool in a screenshot but is just distracting in practice.

A totally representative screenshot of an IDE.
A totally representative screenshot of an IDE.

I won’t go on and on about applications that I think are overly complicated or badly designed. I will instead write a little about those that get it right. Tools that I think exhibits an admirable level of focus and are a joy to use because of it. Things that help me focus. I apologise if this makes me sound like a management consultant peddling a trivial slogan, “simplicity and focus”, as some kind of road to success. I would never sell out like that unless someone offered me actual money.

EMACS

My preferred editor is EMACS. I recognise that I would absolutely hate EMACS if I had been introduced to it today. Back when I learned EMACS I was young and dynamic, curious, hungry for learning, and smelling slightly of peach. The inscrutability of EMACS made me want to learn it because knowing it would be impressive, and then (in the immortal words of Eddie Izzard), “people would shag me”. Now I have become a grumpy older guy. Some would say “mature” but that always makes me think of a particularly stinky cheese which is actually disturbingly accurate… These days I just want to get on with things. Write the code, fix the bug, have some coffee and get back home in time to mow the lawn and watch a bunch of identical TV shows about home styling. But I digress.

The reason I bring up EMACS is not because Meta-Control-Cokebottle-Hashtag-Detonate is the natural way to type the letter ‘n’, it is because EMACS illustrates the way good tools gets out of the way. EMACS is pretty hard to learn, but that does not mean that it is hard to use. Easy to use is often implicitly taken to mean easy to use for the beginner. This is not at all the same thing. Great tools are not designed to be easy to learn, they are designed to be easy to use. Easy to use, easy to learn, powerful. Pick any two.

Once you know the basics of it, EMACS allows you to edit text without EMACS getting in your way. It doesn’t hide some things in obscure menus, it hides everything. There’s no menu bar to steal pixels from your precious text. Well, actually, there is but fortunately you can turn that off with some LISP voodoo (clearly superior to XML voodoo). There’s no toolbar with stupid icons. Ok, it has that too but it can thankfully be disabled with some LISP magic. There no annoying status area. Eh, ok there is a status area and I’m not sure that you can turn that off. But it is really small! As I have convincingly demonstrated, EMACS is clearly superior to any IDE.

When all the voodoo has been applied EMACS really doesn’t get in your way.

Distraction-free EMACS. You can almost smell the awesome.
Distraction-free EMACS. You can almost smell the awesome.

iA Writer

Another editor that I really like is the one I’m using to write this very text: a program called iA Writer. iA Writer is designed explicitly to embody the idea of focus and simplicity. In its full screen mode iA Writer displays only the text that you write. No menus, no widgets, just black text on a white background. Options are severely limited: no font selection, no layout, no pictures. Just text. It’s not that iA Writer has no features, but the features have been very carefully picked. I find that this really helps. iA Writer is my WordStar 4.0.

I’m writing this on a bus on my way to work. That means no Internet. The combination of iA Writer and no Internet really makes a huge difference to my productivity. Being on a bus with nowhere else to go also helps.

If you don't get writers block by staring at this full-screen actual screenshot of iA Writer then you never will.
If you don’t get writers block by staring at this full-screen actual screenshot of iA Writer then you never will.

Xmonad

I recently ended a 13 year long love affair with the Sawfish window manager when I left it for Xmonad. Xmonad is a tiling window manager which means that it will put the windows on the screen according to a predefined layout, with no gaps. This means that there is no desktop, which I think is great. The desktop metaphor was useful when people didn’t know what a GUI was but it doesn’t provide any benefit to me. Why would I want a bunch of files lying around behind everything else?

The tiling aspect of Xmonad sort of forces you to be more organised. Since you cannot have windows that overlap you necessarily always see every open application. Instead if having a stack of windows ten deep, you need to put things on separate desktops. While this can at times feel restrictive, and doesn’t work well for some applications that love to open gaggles of small windows (GIMP, I’m looking at you!), it is remarkably liberating once you get used to it.

My Xmonad desktop, not at all posing for the shot.
My Xmonad desktop, not at all posing for the shot.

And the rest…

I’ll add two examples of simplicity and focus from the wonderful world of web.

This is the web page of Berkshire Hathaway. It is easily one of the 10 biggest publicly held companies in the world. Berkshire Hathaway owns, among many things, large parts of the Coca Cola Company, Wells Fargo and American Express. A single share of Berkshire Hathaway is worth $190.000 at the time of writing. Look at that web page again. It takes a certain kind of self confidence to maintain a web page that simple. And yet, nothing more is really needed.

Programming in the twenty-first century” by James Hague is my favourite programming blog. The site is, apparently, generated by 269 lines of Perl, including the HTML templates. Both the site and its implementation embodies simplicity and focus. Although I haven’t seen the Perl code I’m assuming that it is simple and pretty, based on mr Hagues blog posts. But I guess that it could also be Perl code.

Finally, I’ve written a little bit about analogue modular synthesisers. While a modular synth is by no means simple, I think the resurgence and current popularity (ok, “popular” is a rather relative concept) of analogue modulars at least in part reflects a wish to remove options. These days, when you make music in a computer your power is almost unlimited. Major music-making applications are huge and very complex and they encourage you to dive down and micro-edit the music. With a modular synth you remove a lot of the control. While you can nudge an analogue synth into a direction of your choice, you cannot force it to do your bidding exactly. You need to let go of some control and let the machine work on its own a little bit. This can be very freeing and creative. To be fair, a modular synth can also be an example of the exact opposite: an example of a tool that is so complex (and engrossing) that it consumes all your time and prevents you from spending any of it on the reason you bought  the damned thing the first place — the music.

Obligatory parting words of wisdom

Like anything in life, simplicity is best in moderation. A word processor in which you could only write the word “goat” would, perhaps, be too simple. A car without seat belts would be simpler, but not better. But in some cases, simpler really is better. The UNIX philosophy, if you will. The value of focus is readily evident in physical objects. Most people would probably agree that, all else being equal, a simple object is better than a complex one. Compare the Apple TV remote with the remote of any cheap TV. The TV remote isn’t complex because those who designed it thought that complex was better, it is complex because they couldn’t afford — or simply didn’t know how — to make it simple. Well judged simplicity is a valuable skill. Treasure it.

A Bug’s Life : Security Level Cray

Illustrations by Jonas Ekman.

When the nineties were getting old and people could value a pet shop at 100 million dollars as long as it had a web page, I was missing out on the hype by working as a sysadmin at a supercomputing site. This was before supercomputing got boring; when a super was a really odd machine running a demented OS instead of just being a thousand of what you have on your desktop. One of my babies was a Cray J90, more specifically a J932SE.

The J90 was, admittedly, not one of the awesomest Crays. It did not have an integrated couch (Cray 1, Cray XMP, Cray YMP).

Cray 1, when you want your super in the lobby.
Cray 1, when you want your super in the lobby.

You couldn’t serve beer on it (Cray 2).

The Cray 2 gets my vote as the coolest looking computer ever. It is a coolant aquarium!
The Cray 2 gets my vote as the greatest looking computer ever. It is a coolant aquarium!

It did not have a waterfall (Cray T90).

T916
A Cray T90 with coolant waterfall behind it.

However, it was a 32-CPU shared-memory vector supercomputer with a gigantic (at the time) 8GB of RAM. It was the size of 2 largish fridges, ran UNICOS — the Cray flavour of UNIX — and could multiply 64-bit floating point numbers like nobody’s business.

A Cray J90 wishing it had more awesome looks.
A Cray J90 wishing it had more awesome looks.

Seymour Cray was famous for his disdain for virtual memory and thus UNICOS did not have any. If you had 8GB of RAM then that was all you could use. When you wanted to start a process you’d better hope that the system could find a free, continuous chunk of RAM of the appropriate size. If your process ran out of memory it would be swapped out to disk in its entirety. Needless to say, swapping was death for performance.

A Cray machine was a reasonably expensive piece of equipment (some may even have called it unreasonably expensive) and most, very likely all, were bought to service many users at the same time. One of the system software packages Cray sold was called Multi-Level Security, MLS for short. MLS allowed you to virtually partition the machine in a fairly bullet-proof fashion. You could buy a 32-CPU system and let some users see 30 CPUs and half the memory while other users only saw 2 CPUs. If you were running in a partition you couldn’t access or even see processes running in another partition. In fact, you could be a system administrator and still have access to just a part of the physical machine. The effect was similar to virtualisation but without the performance penalty. MLS was a fabulous idea because it allowed, say, the NSA and the North Korean embassy to pool their resources and buy a Cray together and their users wouldn’t see each other. What could possibly go wrong?

MLS was a pig and a half and any sane sysadmin would run from it screaming. Some poor souls were forced to use it because it sounded like a great deal to management. We were lucky because our machine was open and we did not run MLS.

A typical sysadmin reaction.
A typical sysadmin reaction.

One carefree day I came into the office and found an envelope in my mail box with a DAT-tape in it. The label on the tape said UNICOS 10.0. UNICOS 10 had been out for a while and we could not put it off any longer, it was time to upgrade the OS on our machine from 9.0 to the latest and greatest.

An OS upgrade on a million dollar multi-user system is a pretty major undertaking. The machine ran batch jobs for dozens of users. Jobs could be queued for weeks waiting for a time slot to run in, and a single job could execute for days. Just turning the machine off would get you murdered by a PhD student (bad) or lectured at by some mechanics professor (worse). The OS upgrade had to be announced weeks in advance and planned like a Prussian wedding. The plan was written, the upgrade announced, the sacrificial goat was prepared and magical runes were appropriately carved on the computer room doors to ward off evil upgrade gremlins… the process for OS upgrades can be highly site-dependent.

The day arrived. We stopped the batch queues, rebooted the machine into single-user mode and went to work. The upgrade went well and after a few hours of testing we could restart the batch queue and let the users back on. Mission accomplished.

A week or so later I was doing some routine work (the technical term is “dicking around”) on the Cray and I discovered something strange: there was a file in the root directory that was owned by a normal user. The user was one of my co-workers and he did not have root access to the Cray. The file should have been unpossible. My immediate worry was that we had been hacked. A supercomputing site is a juicy target for hackers and a super, with its huge I/O capability and usually great network connectivity, can make an awesome denial-of-service booster.

The co-worker was trusted so I asked him about the file. He had been testing a script with a path bug in it so he could explain where the file came from. He could not explain how it had managed to be written in a read-only directory. This pretty much discounted the theory that the file was part of a hacking/cracking/fracking operation. Looking around some more I quickly found other files that were owned by users in places where those users didn’t have write access. After some testing the horrible truth dawned: any user could write anywhere. While academic institutions have a reputation for openness even Richard Stallman might agree that this was a step too far.

We had a common login system for all our machines. It was based on an implementation of Kerberos called Heimdal that had been developed locally and luckily for me I shared an office with one of the main authors of Heimdal. He was the kind of wizard every computing site needs. If you complained vaguely about some proprietary protocol not working he would sit quietly for 30 minutes (not acknowledging that he had even heard you) and then suddenly announce that he had reverse-engineered the protocol and mailed you the source code for a replacement server that he had coded up from scratch. Once, when confronted with what appeared to be a corrupt file name in a directory his first action was to hexdump the directory inode. While he did not have an actual, physical, beard, his virtual Unix beard was mighty to behold.

Everyone needs a UNIX beard.
Everyone needs a UNIX beard.

I was a bit out of my depth so I recruited my wizardly office mate to the cause. The best way to do that was to not ask for help directly but to just complain generally until he either got tired of listening to it or got curious. After applying beard to the problem for a while he managed to figure out what was going on.

The Heimdal kxd program was responsible for accepting logins via the network, authenticating them and spawning a user login session on the machine. When doing this it had to change the owner of the spawned process to the UID of the user logging in, which it did with the setuid() system call. This used to work just fine. But now the setuid() call changed the UID and left the process with root file access permissions.

As a part of the planning for the OS upgrade I had pored over the release notes for UNICOS 10 time and again to make sure that I hadn’t missed something that would cause us problems. One of the changes that Cray had made was that MLS, the Multi-Level Security that everyone loved so much, had been included as a default component in the OS. It used to be a separate product but now everyone got it. The release notes promised that if you disabled MLS in the kernel it would work exactly as a non-MLS system did in UNICOS 9. What the release note should have said was that if you disabled MLS it would work exactly as a non-MLS system did except for the part where we changed the way access permissions are inherited by the setuid() system call. A consensus was developing around the view that MLS was the spawn of satan.

Are you sure you didn't order a MLRS?
Are you sure you didn’t order a MLRS?

The virtually bearded one went quiet for 15 minutes or so and then announced that he had fixed it. The fix can still be found in the Heimdal source code today:

 if (setgid (passwd->pw_gid) ||
     initgroups(passwd->pw_name, passwd->pw_gid) ||
 #ifdef HAVE_GETUDBNAM /* XXX this happens on crays */
     setjob(passwd->pw_uid, 0) == -1 ||
 #endif
         setuid(passwd->pw_uid)) {
         syslog(LOG_ERR, "setting uid/groups: %m");
         fatal (kc, sock, "cannot set uid");
 }

I would frankly have been a lot less restrained in the comment…

Cleaning up the mess took a while, we had to check all file systems for integrity. Happily we had dumped all system disks to tape after the upgrade but prior to letting any users on so we had a known good set to compare against.

Lessons learned? If you change the semantics of setuid() then maybe, just maybe, drop a line about it in the release notes. Also, a paranoid mind set is a great asset to a system administrator (although it can definitely go off the rails without some healthy pushback from management) and preparation is key to any successful system upgrade.

While this MLS thing was a bit of a fiasco I have to say that UNICOS was incredibly stable. It just ran for years and years and the only crashes that I can remember were caused by hardware failures (typically hard drives). Oh, and that one time when I shorted out the entire machine with a carelessly applied screwdriver…

A Bug’s Life : The One Sample Solution

At one point in my life I found myself working for a company that made synthesisers of the musical instrument kind. The guts of the synths consisted of a bunch of DSPs that did all the audio processing, some memory, and a general purpose CPU that handled the rest: scanning the keyboard, scanning the knobs and buttons on the control panel and providing MIDI and USB connectivity.

One of my jobs was to clean out a list of bugs that users had reported on a released product. Most of these bugs were minor and sometimes obscure but the company prided itself on high quality so they would strive to fix all reported bugs, no matter how minor. Unless the bug report was that “the product should cost 50% less, be all-analogue, and be able to transport me to my place of work”. Some people have strong feelings of entitlement.

One of the bugs on my list was that the delay glitched if you turned it off and on again quickly. A delay is a type of audio effect that simulates an echo: if you send a sound into it then the sound will be repeated some number of times while fading out. Early delays were implemented using magnetic tape. If the recording head was placed some distance ahead of the playback head then a sound that was recorded into the delay unit would play back as an echo a short while later as the tape passed by the playback head. In this particular case the delay was digital and consisted of a circular buffer of audio samples that the DSP would continuously feed data into (mixed together with feedback from audio that was already in the buffer). The buffer would then be played back by being mixed with the main audio output signal.

A tape-based delay effect unit.
A tape-based delay effect unit.

The delay had an on/off button on the front panel of the synth and when you switched it off the DSP would set the input volume to the delay buffer to 0 so that it would fill up with silent samples. However, since this happened at the normal audio rate it would take a few seconds before the whole buffer was filled with zeros. A user had discovered that if you played something with the delay enabled, then switched it off and then quickly switched it on again then parts of the delay buffer would still contain sound that you would hear, sometimes with nasty transients. The solution was to program the DSP to quickly zero the delay buffer using a DMA transfer whenever the delay was turned off.

This may sound trivial but the code running on the DSP was hand-coded assembly optimised to within an inch of its life. The DSP had one job: to present a pair of completely processed 24-bit samples — a stereo signal — to the Digital-Analogue converter inputs at the sample rate, which was 44100 times per second. If a sample wasn’t ready in time then digital noise would result at the output. This was frowned upon because it would sound horrible and if it happened while the instrument was being played on stage, through a serious sound system, you might as well poke out the eardrums of your audience using an ice pick. This made the rules of the game for the DSP pretty simple: if it could execute one instruction per cycle and was running at F cycles per second then that meant that it could spend N=F/(2*44100) instructions per sample. In fact it had to spend exactly N instructions or the timing would drift off. Any unused cycles had to be filled in with “No-op” instructions that do nothing but waste time. In this case N was a couple of hundred cycles. This meant that the DSP code was a couple of hundred instructions long, which was good because there was not so much of it, but bad because there were very few unused cycles left in which to set up the DMA.

This type of DSP is built to do one thing: to perform arithmetic on 24-bit fixed point numbers. It is a multiply-add engine. Multiplying and offsetting 24-bit fixed point numbers is easy and everything else is a pain in the upholstery. Instructions are often very limited as to which registers they can operate on and the data you want to operate on is therefore, as per Murphy’s law, always in the wrong type of register.

After scrounging up some spare cycles I managed to set up a DMA that zeroed the delay buffer whenever it was turned off. Apparently. I tested it: no problem. I told my boss and he came in and listened. Now, my boss had worked with theses synths for years and years and he immediately heard what I didn’t: a diminutive “click” sound that was so weak that I couldn’t hear it at all. “Probably you didn’t turn off the input to the delay buffer so a couple of samples gets in there while the DMA is running.” I verified it but no, the input was properly turned off.

Everyone needs Wilson's Common Sense Ear Drums.
Everyone needs Wilson’s Common Sense Ear Drums.

Now that I knew what to listen for I could hear the click if I put on headphones and turned the volume up. Headphones are always scary when programming audio applications because if you screwed up the code somewhere you might very well suddenly get random data at the DACs which means very loud digital noise in the headphones which means broken eardrums and soiled pants in the programmer. In contrast, a single sample of random data at 44.1kHz sounds a little bit like a hair being displaced close to your ear. In the beginning I had to stop breathing and sit absolutely still to hear the click noise. Moving the head just a little would make mechanical noise as the headphones resettled, noise that would drown out the click. After a while though, my brain became better at picking out the clicks, after a day or so I could hear it through loudspeakers. Unfortunately I soon started to hear single-sample clicks everywhere…

"Wait, wait! I think I can hear it!"
“Wait, wait! I think I can hear it!”
The good thing about this bug was that it was fairly repeatable. I like to think that no repeatable bug is fundamentally hard. Unless the system is very opaque you just have to keep digging and eventually you’ll get there. It may take a long time, but you’ll get there.

So what was going on? Was the delay buffer size off by one? No. Was the playback pointer getting clobbered? No. Did the DMA do anything at all? Yes, by filling the buffer with a non-zero value and then running the DMA I could see that the buffer was zeroed afterwards. Was it some sort of timing glitch that caused a bad value to appear at the DACs? Not that I could tell. Blaming the compiler is always an option but in this case the code was hand-written assembly so that alternative turned out to be less satisfying. A compiler can’t defend itself but your co-worker can…

brainstorm-blame-workplace-ecard-someecards

The debugging environment was pretty basic. The only tool available was a home-grown debugger that could display something like 8 words of memory together with the register contents and not much else. Looking through a megabyte or so of delay buffer data through an 8 word window might sound like fun but I guess I’m easily bored…

One thing that became apparent after a while was that the click sound would appear even when the delay buffer was empty to begin with. This indicated that the bad data was put there instead of being left behind. At this point I started to throw things out of the code. I find this to be a pretty good way forward if a bug checks your early advances. Remove stuff until the bug goes away, then you can start to add stuff back again. After a while I had just the delay code left. I wrote a routine that filled the delay buffer with a known value, ran the DMA, and then stepped through the whole buffer, verifying the contents word-by-word. And bingo, it found a bad sample.

bingo2

Seeing it made it real. As long as I only heard it, the defect could theoretically have been introduced somewhere later in the processing or in the output chain. But now I could see the offending sample through my tiny window, sitting there in the middle of a sea of zeros. What is more, the bad sample did not contain the known value that the delay buffer was initialised with. At this point suspicions were raised against the DMA itself. A quick look through the DSP manual revealed an errata list on the DMA block several pages long. This DSP had a horrendously complex DMA engine that could do all sorts of things. For example, it could copy a buffer to a destination using only odd or even destination addresses — in other words it could copy a flat buffer into to a single channel of a stereo pair. It seemed like half of these modes didn’t really work.

None of the issues listed in the errata list fit what I was seeing but I still eyed the DMA engine suspiciously. I therefore tried to zero the delay buffer using a different DMA mode and it worked! Ah, hardware: can’t love it, can’t kill it. Hardware designers on the other hand…

bad_dog

So the bug was put to rest and I moved on to the next issue on my list. Did we learn something? When you can’t blame the compiler, blaming hardware remains an option. In the end I came away pretty impressed by the dedication to quality and stability that this small company displayed. The original bug report was on a glitch that many companies wouldn’t bother to correct at all. The initial fix took maybe half a day to implement and took the amount of unwanted data in the delay from 1-2 seconds (between 90000 and 180000 samples) down to just a single, barely audible, sample in what was already a rare corner case. Fixing it completely took about a week. In other words, it took four hours to fix 99.99999% of the bug and 36 hours to fix the rest of it. But the message was pretty clear : fix it and fix it right.

And to all the semiconductor companies out there: don’t let your intern design the DMA engine.

The Philosophy of Bug-fixing

I spend way to much time fixing bugs. I don’t say that because my code is excessively buggy (although that is also true), I say that because quite a few of the bugs I fix are probably not worth fixing. Not really. Let’s imagine…

Your customer has submitted, along with vague threats of ninja assassins being dispatched to your company headquarters, a bug report about a crash in your software. He cannot really supply any useful information on what led up to the crash but the error message tells you exactly where in the code the crash occurred. Firing up your favourite editor you take a look. The crash happens because an unallocated resource is freed.

A bug :

mystificate(wizard)
 ...

 fire(wizard)

The crash is easily prevented, just check if the resource is allocated before freeing it. The thing is, that resource should be allocated at this point in the code. You know that it usually is or you would have seen this crash before. The crash, while a bug in itself, is clearly also a symptom of another bug. What do you do and why?

A bug fix :

mystificate(wizard)
 ...

 if (hired(wizard))
    fire(wizard)

Any diligent programmer would at least spend a little time considering the implications of this discovery. If you can easily figure out how the resource came to be unallocated then perhaps you can fix that too. And you should. You may also realise that the unallocated resource is a fatal error which absolutely, positively, must be fixed or the plague breaks out. Put on your mining helmet and start digging. Perhaps your product is a pacemaker. Fix the bug.

Most of the time though, the situation is not as clear cut. You just don’t know what the implications are. The bug could be a corner case with no other consequences than the crash that you just fixed. Or it could be a symptom of some intricate failure mode that may cause other bad things to happen randomly. I would guess that most competent programmers would feel decidedly uncomfortable papering over the problem. Some may even find it hard to sleep at night, knowing that the bug is out there, plotting against you. They lie awake worrying about when and how the bug will crop up next. So you start to investigate.

Initial state.
Initial state.

You rule out the most likely causes. You try to repeat the problem. You try to make it happen more often. If you’re an embedded programmer you verify five times that RX and TX pins haven’t been swapped somewhere. You instrument the code. You meditate over reams of trace printouts. You start to suspect that this is a compiler bug, but that is almost never really the case. You read the code (desperation has taken hold). You poke around in a debugger. You remove more and more code, trying to reduce the problem. You start to become certain that this is a compiler bug (it never is). You dream about race conditions. You stop showering. You start to have conversations with an interrupt handler like it was a person. You dig into the assembly. You now know that it is a compiler bug (it isn’t). You mutter angrily to yourself on the subway. A colleague rewrites a completely unrelated part of the code and the symptom inexplicably goes away. Your cat shuns you. You start to question everything; how certain are you that a byte is 8 bits on this machine?

Intermediate stage.
Intermediate stage.

After aeons of hell a co-worker drops by. She looks over your shoulder and asks “isn’t that resource freed right there?” And sure enough — in the code that you’ve been staring at for weeks, a mere 4 lines above the bad free, the resource is freed the first time. To say that the error had been staring you in the face the whole time would be understating it. You kill yourself. The cycle starts again.

The final state.
The final state.

The terrible truth:

mystificate(wizard)
   fire(wizard)

   hire(izzard)
       if (flag)
           occupy(india)

   if (hired(wizard))
       fire(wizard)

Or maybe the bug turned out to be something really gnarly that could explain dozens of different bug reports.

Should you have fixed that bug? The thing about bugs is that they are unknown. Until you know what is going on you can’t tell wether the bug will cause catastrophic errors or is relatively harmless, right? The bug is what Donald Rumsfeld would call a “known unknown,” something that you know that you don’t know. Not tracking down the bug means living with uncertainty. Fixing bugs increases the quality of the software so fixing any bug (correctly) makes the software better. You can easily convince yourself that fixing the bug was the right thing to do, even if it turned out to be relatively harmless. But you’re probably wrong.

sheldon

The “known unknown” argument cuts both ways. Even if the bug turns out to be serious you didn’t know that beforehand. Putting in serious effort before you know what the payoff would be may not be the wisest way to allocate your time. And the question isn’t really “did the fix make the software better” but “was the time spent on the fix the best way to make the software better”.

Living with uncertainty can be difficult but lets face it, your code probably has hundreds or thousands of unknown bugs. Does it actually make a difference that you know that this one is out there?

Am I saying that it was wrong to start looking for the bug? Definitely not. If you suspect that something is amiss, you should absolutely strive to understand what is going on. The difficulty lies in knowing when to stop. There’s always a sense that you are this close to finding the problem. If you would just spend 10 more minutes you would figure it out, or stumble upon some clue that would really help. Now you’ve spent half a day on the bug, might as well spend the whole day since you’ve just managed to fit the problem inside your head. You’ve spent a day on it, might as well put in another hour just to see where this latest lead goes. You’ve spent three days and if you don’t fix it now you’ve just wasted three days. You’ve spent two weeks and dropping the whole thing without resolution at this point would cause irreparable psychological damage. The more you dig the deeper the hole becomes.

So what do you do to prevent this downward spiral? Talk to a co-worker. I know it might seem uncomfortable and unsanitary and possibly even unpossible, but it can be an invaluable strategy even if your colleagues are all muppets. Take ten minutes a day to talk about where you are and how it is going. Just describing the problem often makes you realise what the solution is. “So the flag is unset when frobnitz() is called and… and… never mind, I am an idiot” is a common conversation pattern (although “never mind, you are an idiot” is idiomatic in some places). Sometimes this even works with inanimate objects like managers. This is sometimes called “Rubber duck debugging“: carefully explain your problem to your rubber duck and the fix will become apparent. Don’t own a rubber duck? What is wrong with you??

Even if describing the problem doesn’t help, a sentient co-worker has a tendency to ask if you have checked 10 different things that are so obvious that how can you even ask, and do you think I’m completely incompetent and, by the way, no, I meant to get around to it. “Did I recompile the code after changing it? No, do you have to?” If your colleague is very polite you may have to insult him a bit before he asks the obvious questions. Obvious questions can be hard on the ego but very useful because the odds are in favour of the obvious things. For some reason the human mind seems to prefer the mystical and complex (“it’s a compiler bug”) to the simple and likely (“I forgot to recompile”).

The final reason it is good to talk to a co-worker is that you get some distance from the problem. Just discussing loosely where you are and what you know and how much time you’ve spent can really help when you start to suspect that you should strap your bug-fixing project to an ejection seat.

Conclusions:
  • Don’t get lost fixing a hard bug, keep an eye on the big picture. This is a lot easier to write than to do.
  • Learn to live with uncertainty. If this is impossible: embrace misery.
  • Co-workers can serve a useful purpose other than being on the opposite side of indentation wars.

Eurorack?

Eurorack is a type of modular synthesiser. In a non-modular synthesiser the various functional blocks that generate and modify sound (such as oscillators and filters) are preconfigured — they are connected in a predetermined way. In a modular synth these blocks are connected manually using patch cables. This makes them harder to use but more flexible. It also makes them expandable; if you need another oscillator you just buy another oscillator module. There are several different modular synth formats. Eurorack is popular because it is open, the modules are physically small and there are many hundreds of modules to choose from, from a large number of manufacturers.

A fixed architecture synth. The oscillators, filters and envelopes are pre-connected.
A fixed architecture synth. The oscillators, filters, amplifiers and envelopes are pre-connected.

 

A modular synth. The oscillators, filters, envelopes, amplifiers and other gadgets are manually connected with patch chords. Each sound can have it's own synth architecture.
A modular synth. The oscillators, filters, envelopes, amplifiers and other gadgets are manually connected with patch chords. Each sound can have its own synth architecture.

 

Modular synth fans have a tendency to go slightly overboard.
Another modular synth. Modular synth fans have a tendency to go slightly overboard.

 

 

A Self-centered Introduction

My anorak.
My anorak.

In this blog I’ll talk a little bit about my journey into Eurorack-hood. What modules I’m looking at, what I want to do with the system and perhaps some reviews of the components once I get a working system installed. Hopefully it can function as a kind of beginners guide to Eurorack modular synths. But why would a person choose to expose himself to a sordid world of holy wars, rabid gangs and addiction? Is it even a choice, or is it an inevitability caused by the social structures of my upbringing? The answer must surely be found in a brief narcissistic soliloquy…

My youth was profitably wasted playing computer games, reading about computer games and collecting huge stashes of pirated computer games (on cassette tapes, no less). At about the same time I graduated from my trusty C64 to an Atari ST I started to get interested in synthesisers. The ST was the musicians computer of that age because it had built-in MIDI. The Atari magazines would sometimes include reviews of sequencer softwares like Cubase and sometimes even actual hardware synths. I listened to a lot of Front 242 and played with an interesting tracker-type program called Quartet and dreamt of having a synth that could make cool sounds. Then I bought a Roland D5. I kept dreaming of having a synth that could make cool sounds.

roland-d5
The slightly less than classic Roland D5.

Well, perhaps that’s a bit unfair. The Roland D5 wasn’t that bad. The keys were good and the “LA-synthesis” was a kind of poor-mans sample playback that made some decent impressions of real-life instruments. But the D5 had no effects (I should have gotten a D10), no sequencer (I should have gotten a D20) and was easier to program than a Yamaha DX7 (I should not have gotten a DX7). It was programmed by stepping cryptic 8-bit values using a plus and minus key while peering confusedly at the lavish 16 by 2 character display and wondering wether “TVF Freq KF” would be better at 1/2 or 5/8 and why. On the plus side it did have a Cowbell patch.

cowbell
You always need more cowbell.

Basically, I couldn’t make the D5 sound cool and Front 242 had an Emulator II and there was no way in hell I could ever afford that so I went back to computer games.

After a while though, I read about the Nord Lead virtual analog synth and thought that that seemed like a much better way of learning synthesis. It had knobs for everything. It was Swedish. It was red. Jean-Michel Jarre had 5 of them. I drooled for months and then splurged on a slightly used Nord Lead 2 and (wiser this time) a Zoom 1204 multi-effect unit.

jarre
Jean-Michel Jarre receives the very first Nord Lead 2.

This turned out to be a lot more fun. I could actually learn the basics of subtractive synthesis on the Nord Lead 2 and I could even put some reverb on it. Ok, I drenched everything in reverb. I’m a real sucker for reverb.

I could totally not play though. I never really got the hang of the sequencer programs: I played so poorly that they never could quantize my creations properly and when I tried to program using the grid they would never keep an accurate beat.

On the theory that more gear would cure my inability to play I bought a Yamaha A4000 rack-mounted sampler which had both knobs and a pretty cool Grand Piano sample if you could stand waiting for it to load. I really wanted to use the A4000 more than I did but every time inspiration struck it would be 30 minutes of swearing over Windows sound drivers, 10 minutes of staring at the A4000 display, 10 minutes of playing around with the sound and 240 minutes of Baldur’s Gate.

The Yammy A4000 in a restful pose.
The Yammy A4000 in a restful pose.

Then came Reason. I loved Reason because I didn’t have to muck about with all that other stuff. It just worked. I would play around with the Subtractor and it was a bit like the Nord Lead (which isn’t all that surprising if you know the history). I took the leap and switched from Windows to Mac, which is perhaps less dramatic for a Unix-nut (they both suck, but at least the Mac sucks less). But it was when I got Reason 6.5 that the penny dropped. Thor. Thor was complicated and scary and could say “I Am Thor” like a deaf person. It turned out that a guy named Gordon Reid had written a series of Thor tutorials that were published on the Propellerhead web site and once I started going through those the truth emerged : Thor is awesome. One thing in particular makes it awesome (for me) and that is the modulation matrix. Having a bunch of switchable oscillators and filters is all well and good but it’s when you want to control the LFO rate with an envelope, and you can, that the excrement gets tangible.

So now I’m a happy Reason-user. My Nord Lead 2 and A4000 gathers dust, the D5 was donated to the needy. I’ve even managed to put together some “music” that I will only inflict on my closest friends and most hated enemies. So why the heck do I want to ruin my life with a modular synth?

First and, let’s be honest, foremost, modular synths are cool. Like a bowtie or a Fez. Who wouldn’t want a wall full of knobs, patch cords and blinking lights, and where can those people get help?

A modular synth is _this_ cool.
Modular synths are this cool.

Secondly: I’m renovating an old room so that I can finally have the space I need to keep all my gear connected and organised. It has to have a 19-inch rack for my rack-mounted gear. Tallying up the height requirements of my vast equipment stash I came up with 3U. 2U for the A4000 and 1U for the 1204 effect unit. I will never use them again and they have to be rack mounted so they won’t figure that out. But a rack with 3U used is just pathetic, I need more rack-mounted gear!

Thirdly: While I have a fair grasp of the basics of subtractive synthesis and have a non-strict “no presets” policy for my music, there is lots more to learn. If I need a flute-sound or an analogue bass drum I can dial that up from scratch but once you need the more advanced stuff like ring-modulation, oscillator sync or (god forbid) FM, I can play with it, but I can’t use it with a goal in mind. Now it turns out that Gordon Reid (remember him?), that titan among men, has written a hugely ambitious series of articles about synth programming called “Synth Secrets“. Synth Secrets ran in 63 (!) installments in the great magazine “Sound On Sound” and is available online. It is also the final motivation for my modular future : I’m going to build a Eurorack system (because it is cool) and use it to build the patches in all 63 articles of Synth Secrets. After that I’ll come out the other end poorer, older and slightly harder of hearing. Or, if my 45 minute epic “856Hz Sinewave and Timpani” becomes the hit it will deserve to be, a millionaire. I’m not saying I will blog about all 63 lessons, or necessarily any of them. I will primarily write about designing and installing the actual system and, perhaps, review some of the modules once they are making noise properly.

So the goal here is not to build a 1000-module behemoth (at least not yet), but a pretty basic system. I’m going to control it from Reason, so no sequencers, and I won’t be digging into any of the esoteric math modules or any of the other crazy stuff.

Exclaimer!

Beware, for I am not an electrical engineer. While I have some hazy mental model of voltages and currents, and have managed to solder simple components without setting myself on fire, I really have no clue what I’m talking about. So please, for your own safety, judge all information on these pages as if I’m actively trying to kill you. Another way of looking at it is to read every sentence as if it ended with “, apparently.” Corrections are most welcome!

You should perhaps also know that I’m far from an audiophile or an analogue nut. I don’t think the terms “analogue” or “digital” really tells you anything about the sound and I don’t expect my modular system to sound better than Thor or the Subtractor in Reason. It might, and I’ll be interested to hear what my ears think, but I don’t expect it to. The attraction of the modular for me is the immediacy of knobs and the flexibility of patch cords.

In closing: Welcome and lets geek out!

Some Holistic Considerations

When it comes to designing a system I found it very helpful to use a “modular planner” website. There are a couple of different ones out there but I’ve found www.modulargrid.net to be the best by far. It has a huge database of modules that you can drag onto a rack so you can try out different layouts. It tallies the current draw and cost of your picks so it gives you helpful ballpark figures for these important metrics. As an added bonus it allows you to show off your creation to others.

The Eurorack format specification was defined by Doepfer and it can be found on their web page:
Technical Details :
http://www.doepfer.de/a100_man/a100t_e.htm
Mechanical Details :
http://www.doepfer.de/a100_man/a100m_e.htm

We’ve established that I’m building a fairly traditional subtractive analogue synth: an East Coast style modular. Modular synthesizers were pioneered in the 1960s by Bob Moog on the American east coast and Don Buchla on the American west coast.
Actually it seems that most countries had at least one crazy bearded guy who built a modular synth before everyone else in the world, but no-one cares about those guys. Anyway, an East Coast (Moog) instrument is usually played by means of a keyboard captained by a serious man in a suit.

All inhibitions are lost at a lively east coast performance.
All inhibitions are lost at a lively east coast performance.

A West Coast (Buchla) instrument is usually played by injecting psychedelic drugs in the audience and waving various extremities in the general vicinity of the synth.

A west coast player jams on the Buchla Breath Controller.
A west coast player jams on the Buchla Breath Controller.

As a further illustration of the difference I will now list the modules that I planned to get at one point for my system, as well as the modules for a possible West Coast type system that I picked out on modulargrid.net

East Coast :
Doepfer A-190-2 MIDI/CV Interface
Doepfer A-110 VCO
Doepfer A-118 Noise Generator
Doepfer A-182-1 Switched Multiple
Doepfer A-140 ADSR
Doepfer A-132-3 Dual DVCA
Doepfer A-138 Mixer
Doepfer A-114 Ring Modulator
Doepfer A-148 Dual Sample&Hold
TipTop Audio Z3000 Oscillator
TipTop Audio Z4000 ADSR

An East Coast type modular.
An East Coast type modular.

West Coast :
Harvestman Piston Honda
4ms Euro Noise Swash
Make Noise Maths
Make Noise dual Prismatic Oscillator
Make Noise Pressure Points
Malekko Heavy Industry Assmaster
WMD Micro Hadron Collider
Harvestman Zorlon Cannon
Synthetic Sound Labs Modulation Orgy
Bananalogue Serge 3P

A West Coast modular.
A West Coast modular.

See what I mean? Did I just pick the modules with the weirdest looks and strangest names on modulargrid.net? Maybe I did. But I’m not that far off. And now I’ve managed to make a system I haven’t even bought yet seem a little boring…

Ok, ok. The main difference between the two styles lies in how sound is generated and shaped. The East coast style is to generate sounds using simple waveforms and shape them using filters and envelopes. The West coast style is to use advanced modulation techniques to generate harmonically complex sounds directly. And then drugs.

Anyway, I looked around for “starter” systems a bit but found surprisingly little. TipTop Audio has a “Happy Ending” kit which seems to be just an enclosure with a power-supply. Pittsburgh Modular has the “Foundation” system which looks nice and Doepfer has a “MiniSystem” which looks similar to the Foundation but is quite a bit cheaper. I chose to go with Doepfer as my “default” — if there is a Doepfer module that solves my problem, I’ll go with that unless I have a compelling reason not to.

Why Doepfer? They pioneered the Eurorack format. They are “competitively priced” (because “cheap” sounds cheap). They’ve been around for ages and have loads of modules. When I emailed them to ask if their 6U-enclosure could be split into two 3U halves I got a reply from Doctor Dieter Doepfer himself (it was “no,” in case you were wondering (but more politely put)). And yes, yes, I know that Americans did it first, but let’s face facts : a modular analogue synthesiser is a very German instrument.

An italian masquerading as a german weakens my point.
An italian masquerading as a german weakens my point.

The Enclosure

The Doepfer MiniSystem looked cool but I want to rack-mount my system so the cheap wooden box (the A-100LC6) it comes in would have to be replaced. I’ve basically settled on a A-100G6 which is a 6U (so it has two rows for modules) metal enclosure with rack-ears. It comes with a 1200mA power supply (which seems to be plenty) and a common bus for the two rows. It looks sturdy with decent room for expansion, should the fancy take me.

A lonely A-100G6 awaiting tennants.
A lonely A-100G6 awaiting tennants.

So the way the Eurorack system works is that you have a “bus” which supplies power (+12V and -12V, +5V and ground), CV and Gate signals. Not all modules use all the signals. Many modules don’t need the +5V pin and the standard Doepfer power supply doesn’t even supply power to that pin without an extra adapter. None of the modules I’m considering needs 5V so I’ll do without the adapter. One (and only one!) module on the bus can use the internal CV and Gate signals as outputs. In my case it will be a MIDI-CV interface module. Other modules may use the internal CV and Gate signals as inputs and in this case several can do that at the same time. All other connections have to be made using patch cables in the front of the system.

Each module is connected to the bus using a ribbon cable and apparently these cables are often incorrectly built. If the cable is incorrect or connected the wrong way you may end up letting the blue smoke out of the module you’re connecting (that’s bad). I’ll be sure to inspect each cable carefully before using it.