DIY FPGA-based HDMI ambient lighting

Ambient lighting is a technique that creates light effects around the television that correspond to the video content. It has been pioneered by Philips under the brand Ambilight. In this project we will create a basic FPGA-based ambient lighting system that reads the video signal over HDMI. This means we are not limited to computer output. We can use it together with DVD players, video game consoles, etc.

The complete source code for the entire project is available on GitHub.

Zero-confirmation bitcoin transactions

Bitcoin transactions and confirmations

Bitcoin transactions are incorporated into the bitcoin blockchain, that acts as a shared ledger. Once the transaction has been part of the blockchain for a long enough period of time, it is accepted as truth by all the nodes in the network and becomes an unalterable part of the bitcoin permanent record. This makes it irreversible and resistant to both network events and deliberate tampering in the form of double-spending attacks.

A block that is added to the blockchain that contains the transaction is called a confirmation. So a newly broadcast transaction is said to have zero confirmations. When it's included in a block, it has one confirmation. Another block is mined on top of the block containing the transaction, we have two confirmations. Generally, a transaction is not considered definitive unless it has six confirmations. On average this takes about an hour.

In this post we will develop a crude system for handling zero-confirmation transactions without incurring into any appreciable risk.

Retro Pinball: Reverse Engineering the Bally AS-2518-51 Sound Module Assembly

One of my friends has a beautiful 1980 Nitro Ground Shaker pinball machine. It's in great condition, but there is absolutely no sound output. As I was trying to figure out a way to troubleshoot the sound module, I found myself falling deeper and deeper into the 8-bit pinball rabbit hole. In this post I will try and explain the inner workings of the amazing Bally AS-2518-51 Sound Module Assembly.

Overview

The 2518-51 Sound Module is based on the famous Motorola 6800 series microprocessor (U3 in the picture and in the schematic). Sound synthesis is performed by a General Instrument AY-3-8910 (U1). There's also a 6820 Peripheral Interface Adapter (PIA) (U2) to interface the AY38910 to the MPU, a ROM (U4) that contains game-specific code and data and a RAM chip (U10). The microprocessor is clocked by a 3.58 MHz crystal. Due to the built-in clock divider the 6800 operates at 895 kHz.

The Bally AS-2518-51 Sound Module Assembly as found in Nitro Ground Shaker

The pinball system interfaces with the sound module using a 15 pin connector (J1) that provides both power and input. The only output of the sound module happens through the speaker connector (J2).

Complex CPU Addressing modes, for free!

One of the most powerful features of a central processing units is the ability to access memory using complex indexing modes. These modes are essential to implement compound data structures such as arrays, structures and classes in an efficient manner. Our FPGA soft-processor core will need to do all these things too.
 
The instruction set architecture (ISA) that we have developed in the previous post support only basic register indirection, like this:

MOV [R1], R2
MOV R1,[R2]

where the [R1] denotes the memory address pointed to by the register R1.

However, as we shall see, we can support rather complex indexing modes essentially for free!
 
Recall the layout of our execution unit:

XENTRAL: A Simple FPGA CPU

XENTRAL is a simple Harvard Architecture CPU. It targets the Spartan6 FPGA present on the Digilent's Nexys3 board, but it's all written using portable behavioral VHDL¹. You should be able to use your favorite FPGA, synthesizer and simulator.

All project files are available on github under the Creative Commons Attribution Non-Commercial (CC-by-NC) license.

Architecture Overview

XENTRAL is a Harvard Architecture processor. This means that code and data are handled separately.

The picture above is a very high-level schematic representation of the XENTRAL architecture. Instructions are fetched from ROM³. Based on the instruction, the control unit manages the execution unit. The execution unit performs the actual execution of the instruction, by e.g. adding the contents of two CPU registers, writing a value in RAM or performing I/O.

XENTRAL is a Harvard Architecture processor. This means that code and data are handled separately. This effectively precludes the use of self-modifying code. It also prevents the processor from loading its own program. The program must be programmed into the instruction memory. This is a common approach for embedded devices and microcontrollers, but it does not play nice with operating systems and boot loaders. As the project grows, we will re-visit this limitation.

Overview of the Execution Unit

Schematic representation of the execution unit

LightDHT - A lightweight python implementation of the Bittorrent distributed hashtable

I have always been fascinated by distributed data structures, but my efforts have always been dampened by the fact that they are really awkward to play with. In order to get anything that resembles real-life usage, you need to be running hundreds of nodes. Needless to say, wrangling hundreds of processes gets rather cumbersome. Fortunately, there are already thousands of people out there that are participating in one of the largest distributed data structures in operation today: the bittorrent distributed hashtable.

In order to facilitate experimentation with this wonderful system, I have written a simple python implementation of a DHT node. It is mostly (with a few exceptions -- see below) compliant with BEP0005. It should be able to act as a full member of the DHT, processing requests without disrupting the network.

Interested in playing around? Get the code on github!

From the documentation:

The aim of LightDHT is to provide a simple, flexible implementation of the Bittorrent DHT for use in research applications. If you want to trade files, you have come to the wrong place. LightDHT does not implement the actual file transfer parts of the bittorrent protocol. It only takes part in the DHT. 

The main philosophy of LightDHT comes down to two considerations: ease of use over performance and adaptability over scalability. 

In order to keep LightDHT easy to use, all DHT RPC calls are performed synchronously. This means that when you call a DHT method, your program will block until you have an answer to your request. That answer will be the return value of the function. This has the advantage that it keeps the logical program flow intact, and makes it more comfortable to use. 

In order to maintain O(log N) scaling across the network, BEP0005 (the standard governing the DHT) mandates that implementations use a bucket-based approach to the routing table. This enables the node to fulfill all requests in constant time and (more or less) constant memory. In LightDHT, we throw that recommendation to the wind. 

Since the main focus of LightDHT is reseach, we are going to keep around all the data we can. This means that we keep around every single node we know about. Since in practice the number of nodes is limited and the request rates are rather low, we do not bother keeping the routing table organized in a tree structure for quick lookups. Instead we keep it in a dictionary and sort on-demand. The performance penalty is well worth the reduced complexity of the implementation, and the flexibility of having all nodes in an easy to use data structure.

Festive and Nerdy

You might have seen those big outdoor displays where every pixel is composed out of a little three-color LED light. In this post I'll tell you how I turned a string of those pixel lights into Christmas tree lights!

Adafruit industries, one of my favorite suppliers, recently started selling strings of a particularly advanced model of these pixel lights. Have a look at the product image, and tell me if it doesn't already look like a string a Christmas tree lights..

"12mm Diffused Digital RGB LED Pixels"

As you might have seen in my previous post, these pixels are rather easy to control from a micro-controller. However, stuffing a breadboard in your tree is not exactly festive. Thankfully, you can get these handy little tins that are just about the right size for a wall-wart plug, a teensy 2.0 with female headers (courtesy of Pieter Floris) and a big mess of wires.

A wall-wart plug, a Teensy2.0 and a big mess of wires.

Note that, thanks to the female-pin Teensy, everything is wired up using jumper wires. That's right - this is a no-solder project. Because the underside of the Teensy is not isolated, and because I have a number of wires in there that might come loose, I have isolated the interior of the tin using electrical tape. This is most likely not compliant with any code enacted after 1953 or so, so any imitation is purely at your own risk.

The only remaining question is how to fasten the plugs and wires onto the enclosure. Ideally you would use specially molded plastic caps and bits, but the odds of finding them in the right size and shape at your local hardware store are quite slim. Instead, I liberally applied black sugru around the holes I had cut into the tin, and then pressed the wires and the plug into it. The sugru just molds around the objects. After a quick touch-up I left it to cure overnight, and the result was really really good.

Now we just pop closed the tin and voila, a presentable programmable Christmas tree lights driver!

The controller and a single string of LED pixels, running a test pattern.

And here's the tree, all festive and nerdy:

Festive and Nerdy

It's the season! (preview)

I while ago my eye dropped on this string of individually addressable RGB LEDs, and I realized they are practically ready-made Christmas lights. A quick order, a long wait, and an evening with my Teensy++ later, I have a sneak preview of our new Christmas tree decoration!

A WS2801 LED string is practically already a Christmas tree decoration!

Since this creation will actually be used "in production" I need to spend some time making it "living-room acceptable". I can tell you this will involve some patience and a lot of Sugru!

 

PCB as a Capacitive Soil Moisture Sensor

Electronics enthusiasts have been designing ways of automating plant care for decades, with mixed results. A traditional weak point of these installations is the resisitive moisture sensor, that is inaccurate and prone to degradation. In this post we will design and fabricate an inexpensive capacitive soil moisture sensor out of a printed circuit board that exhibits none of the weaknesses of its resistive brethren.

Selectively Unchecking Iterators: the key to High Performance C++ in Visual Studio

In Visual Studio 2005, Microsoft introduced checked iterators for STL containers. This means that every single operation that uses these iterators is range-checked to prevent memory accesses outside the container's allocated storage. If an range violation occurs, the program terminates with an exception at the location of the violation, instead of trashing potentially unrelated memory as C++ has been happily doing for decades.

This safety net comes with a price: every single iterator change undergoes scrutiny by the runtime to determine whether it is safe. For most programs this is a paltry price to pay for the peace of mind of safe container access. However, for performance sensitive code that include tight loops that iterate over STL containers, the penalties can be massive.

The folks at Microsoft realized this, and implemented a way to disable this functionality. If you define _SECURE_SCL to 0, the checking goes away. You are back at the level of performance (and unsafeness) of  C++ that you know and love. Unfortunately, there is a little snag. You can't exchange STL containers between modules that are compiled with _SECURE_SCL set to 0 and modules that are not. It may look like it's working, but it will blow up in your face when you least expect it. Also note that std::string is an STL container, and it features prominently in most modern C++ interfaces. The reality of the matter is this: if you are defining _SECURE_SCL to 0 you cannot use any C++ code that you did not compile yourself. No libraries, no DLLs, no nothing. For any non-trivial program, that's a pretty tall order.

Also, by setting _SECURE_SCL to 0 you are disabling iterator checking for the entire application. In any given program there are maybe 100s of places where STL iterators are used, but only a handful (one function, two iterators in my current code) where these checks hinder performance to any real degree. Disabling iterator checking globally, if it is even possible given the limitations discussed previously, is rarely desirable.

It turns out that, as of VS2010, there is a way to selectively disable iterator checking. It is a fairly obscure piece of Visual Studio trivia, and I will share it with you so that it is preserved for posterity:

std::vector<int> v; 
for (int j=0;j<1024*1024*5;++j) {  
    v.push_back(j);}  
}  
std::vector<int>::iterator i = v.begin();  
std::vector<int>::iterator::_Unchecked_type ui;  
int tot = 0;  
std::vector<int>::iterator::_Unchecked_type uiend = _Unchecked(v.end());  
for (ui=_Unchecked(v.begin());ui!=uiend;++ui) {  
    tot += *ui;      
}  

Note that we can uncheck the end iterator outside the loop, as the size of the vector does not change during the iteration. I encourage you to try it with checked and unchecked iterators to get a feel for the difference in performance. Be aware however that the inner loop here is very thin: performance gains will be inflated since the inner loop goes from doing mainly range checking to doing mainly nothing!

Readers are encouraged to contribute a set of macros that abstract iterator unchecking in a convenient way.

Attacks on Certificates are a Good Sign

Recently, there has been a bit of a commotion regarding hackers gaining access to valid SSL and code signing certificates. First there was the stuxnet worm, that was signed with actual certificates stolen from Realtek and JMicron. Then there was comodo-gate. The most recent, and most serious incident happened at dutch company diginotar, and resulted in an (as of yet) unknown number of certificates being issued for unknown sites to unknown parties.

As reporters start to catch on to the magnitude of the diginotar case, and begin to understand the trust implications of a compromised root Certificate Authority, the conversation takes a turn toward gloom and despair. In this post I will offer an alternative reading that strongly suggest that hackers attacking certificates and certificate authorities is actually a huge step forward for security and privacy on the internet.

Ephemeria, the latest just-in-time procedural content experiment

The last few months I've been dabbling in procedurally generated content. My latest project is called Ephemeria, a just-in-time procedurally generated city.

I keep track of a 3 block radius around the player. The good thing is that the whole generation thing is repeatable. If you were to more 3000 miles in a random direction and come back, all the buildings would be the same. We can then keep track of any player- or script-induced alterations and replay them every time a block is re-generated.

I need to figure out what I want to use it for before digging too deep into improvements. For example, for a helicopter type of simulation I need to flesh out the roofs. For a racing sim, I need to get to work on the ground-level geometry. Either way, better building geometry is essential. 

It's here!

It's here! The PCB that I designed has arrived! Actually, it has been for a while now, I just have not had the time to write about it. It's an amazing feeling to open a package from the other side of the world to uncover something that you created yourself out of bit and bytes.

USB prototyping board part 3: Would you like some PCB with that?

Here it is, my very first PCB design!

The USB connector attaches on the top-right side. As you have have read on this blog, I wanted to make a PCB edge connector but the costs for the thicker board needed were an order of magnitude larger than the price of the connectors.

The top of the board contains the USB logic. A largish prototype area -- with rails for power and ground, adorns the bottom part of the board.

After the jump: getting started with PCB design, Eagle tips and some miscellaneous ranting.

USB prototyping board part 2: Breadboarding it

You can draw schematics all day long, but the only way to make sure the circuit works is to get busy on the bench. You can see the result in the picture below. I affectionately call it "the world messiest USB keyboard". 

Let's run through the main parts of the circuit:

To Edge Connect or Not to Edge Connect

PCB edge connectors are a cheap and easy way to connect USB devices to a host by using the PCB itself as a USB-A plug. It looks like this:

You really want the PCB to be thick enough so that it lodges tightly into the USB-A socket. A thickness of 2mm is recommended. My manufacturer quoted a surcharge of $40 for 10 PCBs to lay out my design on a 2mm board. An USB-B socket costs 30 cents.

The USB prototype board will not sport a PCB edge connector.

USB prototyping board part 1: Design and Schematic

The USB connector has become the lingua franca of electronics. Whether you need to receive data, send data or just get quick access to DC power, the little USB plugs are now the preferred way to interface with gadgets large and small. Unfortunately, getting USB to work with hobbyist electronics is still a bit of a black art. In order to facilitate experimentation with USB-connected devices, in a series of blog posts we are going to develop a basic, cheap, low part count USB prototyping board.

This project will also double as my first-ever PCB design

Design Outline

There are AVRs that include a USB controller on the microcontroller die, but they are only available in hobbyist-hostile TFQP packages. Instead we are going to base the design on a small 8-bit AVR micro-controller using the VUSB firmware-only USB stack. We want to keep the part cost down, as well as the PCB real estate footprint so that we have more room left over for prototyping. To that end we select an ATTiny micro-controller in DIP8 format. Our micro-controller needs to have at least 4kB of flash memory. The VUSB stack needs 2kB, and we need enough flash left over for the application we are going to prototype. Our micro-controller will be the ATTiny45.

Our main challenge is micro-controller pin allocation: of the eight in the package, two are used up by power and ground, and another one is used up by reset to enable in-system programming. Another two are used by the differential USB transmission lines D+ and D-. Since we'd rather not sacrifice two of our three remaining pins to connect an external clock source (crystal), we will need to use the internal clock at 16.5 MHz. This means that we need to run the micro-controller at 5V. 

According to the VUSB documentation, the reference project for running VUSB ousign the microcontroller's internal oscilaltor is EasyLogger. We will be using this design as a starting point.

Schematic

As I mentioned before, the schematic is derived from the Easylogger.

Because we are running the micro-controller at 5V, we need to adjust the pull up resistor (R3) upward from 1K5 as mandated by the USB standard so that the proper voltages are seen on the bus on the host side. We also need two zener diodes to limit the voltage on the USB bus. Tip: if you can find LEDs with a forward drop between 3.3V and 3.6V (usually blue), you can use them instead of the zeners to get instant activity indicators. Remember that in this case you need to forward-bias them by connecting the cathode to ground.

We have added two additional connectors. The connector labelled ISP is a standard 6-pin in-system programming connector. This will allow us to program the micro-controller without having to remove it from the circuit. The pull-up resistors on the NRES (R4) and MOSI (R3) pins have been dimensioned not to interfere with ISP. If ISP is not required, this connector can be left unpopulated.

The connector labelled PROTO is a female six-pin connector that enables us to quickly wire any micro-controller pin (with the exception of the ones used for USB communication) to a breadboard using jumper wires. If this functionality is not desired, the header can be left unpopulated. The pads can be connected the prototyping area using components of the circuit being prototyped, wiring or zero-ohm resistors. Note that adding strong pull-up or pull-down resistors to the pins used for ISP can interfere with programming.