DIY FPGA-based HDMI ambient lighting

Ambient lighting is a technique that creates light effects around the television that correspond to the video content. It has been pioneered by Philips under the brand Ambilight. In this project we will create a basic FPGA-based ambient lighting system that reads the video signal over HDMI. This means we are not limited to computer output. We can use it together with DVD players, video game consoles, etc.

The complete source code for the entire project is available on GitHub.

Design outline

Ideally we would like to snag the signal off the HDMI cable without disturbing it. However, the signals are really really fast current mode logic differential pairs, and signal integrity issues make any kind of passive tap a non-starter. We are stuck with doing the next best thing: decoding and recoding the signal. This has the advantage that we gain the ability to modify the signal on the way through the system to add debug information etc.

Thanks to our "man in the middle" position between the player and the display we can see all the pixel values, aggregate them in properly sized and positioned boxes and use this signal to drive a strand of LEDs over their SPI interface. These LEDs will be arranged in a ring around the back of the display, resulting in an ambient lighting effect.

Due to the limitations of the FPGA boards currently available on a hobby budget, we will not be able to process video signals at resolutions higher than 720p. That's unfortunate, but I expect this limitation to go away as series 7 FPGAs drop in price.

Meet the components

At the heart of the system is the FPGA. We are going to save ourselves a lot of trouble, and use a board that comes with an FPGA suitable for HDMI deserialisation/serialisation and two HDMI connectors already affixed. We are going to use the wonderful Scarab miniSpartan6+.

For the leds we are going to use a strand of 25 digitally addressable LEDs based on the WS2801 controller. There are denser, brighter, cheaper addressable LED strands out there now, but I have these lying around from another project, so that's what we will use.

Finally we need a way to do logic level conversions. The FPGA operates at 3.3V, but the WS2801 expects to see 5V as a high logic level. Therefore we will need to use a level shifter circuit. We are going to use a really simple, cheap MOSFET-based level shifter by adafruit. We could do our own level shifting, but this unit is so simple and cheap, there's no reason not to use it.

HDMI Decoding/Re-encoding

The HDMI specification is huge, sprawling and complex and contains lots of odd video modes and assorted features. For this project, we shall limit ourselves to the 24 bits per pixel RGB modes, as these are both the most common modes and the sanest ones to support. We will also ignore HDCP encryption, EDID (which means we need to configure the proper modes on the HDMI source), CEC, and all other sorts of frills.

Electrically, the video signal is sent over four shielded differential pairs. Three are used for the color channels, and one is used for the pixel clock. Note that the pixel clock beats once for every pixel, not once every bit of every pixel. Also, the HDMI specification allows the channels to have significant delays with respect to each other. This means that we will need to perform clock recovery to receive the individual bits, and we will need to delay the individual channels to get them to line up.
The actual bits are encoded using a special form of 8b/10b encoding. Decoding the channel data yields the pixel values and the control signals. For re-encoding, this procedure needs to be reversed.

Xilinx has released a whole bunch of documentation on implementing video interfaces using the Spartan 6 family of FPGAs. The most relevant one is the application note Implementing a TMDS Video Interface in the Spartan-6 FPGA. As we can see the procedure is fairly involved, but not overly complicated, and reference designs for both the HDMI receiver and transmitter are provided along with the application note, that target  Digilent's Atlys board. The Spartan 6 FPGA series contains a bunch of hard IP that assists greatly in implementing HDMI receivers and transmitters.

The app note includes the following diagram of an HDMI receiver:

Practically this means the following:

1. We need to send all four input signals (three data/color channels R,G,B and the clock signal CLK) to differential input buffers, configured to TDMS operation.
2. CLK is fed into a phase-locked loop (PLL) and used to derive CLKx10 and CLKx2 signals. 
3. CLKx10 has the same frequency as the bitstream, so it's used to clock three input de-serializer blocks (ISERDES2) and associated delay circuitry (IODELAY2). These blocks capture five bits at a time and output them every beat of the CLKx2 clock.
4. Two five-bit blocks are glued back together in the 5:10 Gear Box, yielding a potentially correct 10 bit symbol.
5. Looking for known values in the data stream, we sync up the symbols for each channel. If we don't see the known values, we shift the window by one bit and try again. Eventually every channel should end up in sync, and we should be receiving valid symbols from every channels.
6. Then we bring the channels in sync with each other, so that the three symbols of each channel are asserted on each beat of the pixel clock.
7. Finally the symbols are fed through the TMDS decoder to yield the actual RGB pixel values and control signals.

Re-encoding and transmission follows a similar process but in reverse. The pixel values are passed through a TMDS encoder yielding 10 bit symbols. The symbols are hacked up in two halves of five bits each and are shoved onto the differential pairs using an output serialization block (OSERDES2) connected to a differential output buffer configured for TMDS operation.

Standing on the shoulders of ... hamsters?

Normally we would try and port the reference designs to the miniSpartan6+. However, it turns out we can steal borrow most of the HDMI handling from Mike Fields of Hamsterworks. His project MiniSpartan6+ DVID Logo performs exactly the kind of decoding/re-encoding we need. It's solid, legible code that happens to target the specific hardware we are using. Therefore we are going to use it as a basis for this project. Mike, if you are reading this, you're the man. Seriously.

Averaging the HDMI signal

The decoded video stream is presented as an old fashioned VGA signal, and consists of the following signals:

1. The pixel clock. We can sample the pixel color values on the rising edge of this clock.
2. Three 8-bit values for the red, green and blue channels of the current pixel
3. HSync and VSynch, signals that alert us to the start of a line or screen respectively
4. Blank, that signals the blanking period. Originally this signal was used in CRT monitors to turn off the electron beam while it returned to the left edge of the screen after a line.

Using the three control signals and a few counters we can determine the (x,y) position of the current pixel. The value of x is increased for every pixel clock. The value of x is set to zero and the value of y is increased at the start of the blanking period (or at hsync). At vsync, both x and y are set to zero.

Using red, green, blue, x and y, we can do on the fly averaging. By comparing with x and y, we can determine what sampling box (connected to an LED) each individual pixel should contribute to, if any. Then we sum the color values into an accumulation register per LED. To obtain the average value, we need to divide the value in the register by the number of pixels. We are going to make sure that the sampling boxes contain a number of pixels that is a power of two. Then we can just wire the MSBs of the registers to the output to get the average values without performing expensive divisions.

The process for a single sampling box is illustrated in the figure below.

We are going to be using sampling boxes of 128x128 pixels. This means that our accumulation registers will need to be able to hold the sum of  2**7 x 2**7 8-bit values for every (red, green, blue) channel, and will therefore have to be 22 bit wide. In total, we will require 1650 bits of accumulators. However, the accumulator only contains the proper value at the end of the frame. Therefore we need another 600 bits to act as a "double buffer" to keep the values of the MSBs stable as we continuously drive the LEDs over their SPI interface. In total we require 2250 bits of state, which is a puny amount, even for a small FPGA like the LX9.

Driving the LEDs

Our LED string uses a SPI-like system to set the color values for the individual pixels. It's made to run at 5V and it uses the WS2801 controller. We would like to drive it directly from the digital output pins of the FPGA, but as soon as we open the datasheet of the WS2801 controller, we notice we've hit a pretty serious snag:

In order for the inputs to register as logic level high, they need to be at 0.8Vdd. Since the LEDs are powered at 5V, this means that we need to get above 4V for the controller to register a high signal. The FPGA is a 3.3V device. This means we will be needing to perform level shifting.

We are going to use a really simple, cheap MOSFET-based level shifter by adafruit that I have lying around. It's designed to work with I2C, SPI and serial and supports bidirectional operation. The circuit follows the suggested schematic from the NXP I2C level shifting app note, but with four channels instead of two:

By passing the signals through this device we can convert between 3.3V and 5V logic levels. We must ensure every line is not driven from both ends at the same time, but in this application that is not a concern because the WS2801s in our design have no user-accessible outputs (they are wired to the LEDs).

Now that the electrical part is taken care of, we still need to write the actual values to the LEDs. For the sake of simplicity, we just continuously send the values in the SPI buffer to the LEDs, instead of trying to sync with the HDMI source frame rate. This may lead to some tearing artifact during rapid transitions, but in practice this is not a problem.

From the data sheet, the timing diagram for the WS2801 is as follows:

A new frame starts when the clock is low for more than 0.5 ms. Then we can output the RGB values R-first, MSB-first without pausing between the LEDs. Theoretically the WS2801 can operate at up to 25MHz, but since we're dealing with lots of wires and breadboard prototyping techniques, we are going to settle for a much more modest clock speed.

We generate a 25 kHz clock by dividing the on-board 50 MHz MEMS oscillator of the miniSpartan6+ board. We output all 600 bits of the SPI buffer in order on the falling edge of the clock, so that the WS2801 can sample them on the following rising edge. When we are done, we bring the clock line low and wait for the equivalent of 40 bits to reset the data frame. This means that we perform about 40 updates of the LEDs every second, This slower than the data is pouring in from the HDMI averager module, but it does not suffer from visible lag.

If we blindy output the averages to the LEDs, the colors will be very washed out. We need to perform gamma correction in order to match the colors of the LEDs to the colors on screen. Experimentally, we get the best results using a gamma value of 2.5. We create the VHDL code for the gamma lookup table using a tiny python script:

for i in range(256):
    g = int(math.pow(float(i) / 255.0, 2.5) * 254.0 + 1.5)
    print 'X"%02X",' % (g),

Notice how we map the values to the interval 1-255, instead of 0-255. We do this because at intensity 0 the individual "subpixel" LEDs actually turn off completely, and this results in significant color artifacts.

Putting everything together

Now can assemble everything and wire all the components together.

In the picture above we can see the LEDs affixed to a piece of cardboard arranged in the proper pattern. The level shifter is placed on a solder-less breadboard that also contains the 5V power supply. The FPGA board is powered from the 5V power supply and drives the leds through the level shifter. The cardboard support can be attached to the back of the television using M4 screws threaded through the VESA mount, or just by using more masking tape.

Now we can hook up an HDMI source (such as my trusty old WD TV Live HD) and enjoy the show!