## Wednesday, April 22, 2015

### DIY FPGA-based HDMI ambient lighting

Ambient lighting is a technique that creates light effects around the television that correspond to the video content. It has been pioneered by Philips under the brand Ambilight. In this project we will create a basic FPGA-based ambient lighting system that reads the video signal over HDMI. This means we are not limited to computer output. We can use it together with DVD players, video game consoles, etc.

The complete source code for the entire project is available on GitHub.

### Design outline

Ideally we would like to snag the signal off the HDMI cable without disturbing it. However, the signals are really really fast current mode logic differential pairs, and signal integrity issues make any kind of passive tap a non-starter. We are stuck with doing the next best thing: decoding and recoding the signal. This has the advantage that we gain the ability to modify the signal on the way through the system to add debug information etc.

Thanks to our "man in the middle" position between the player and the display we can see all the pixel values, aggregate them in properly sized and positioned boxes and use this signal to drive a strand of LEDs over their SPI interface. These LEDs will be arranged in a ring around the back of the display, resulting in an ambient lighting effect.

Due to the limitations of the FPGA boards currently available on a hobby budget, we will not be able to process video signals at resolutions higher than 720p. That's unfortunate, but I expect this limitation to go away as series 7 FPGAs drop in price.

### Meet the components

At the heart of the system is the FPGA. We are going to save ourselves a lot of trouble, and use a board that comes with an FPGA suitable for HDMI deserialisation/serialisation and two HDMI connectors already affixed. We are going to use the wonderful Scarab miniSpartan6+.

For the leds we are going to use a strand of 25 digitally addressable LEDs based on the WS2801 controller. There are denser, brighter, cheaper addressable LED strands out there now, but I have these lying around from another project, so that's what we will use.

Finally we need a way to do logic level conversions. The FPGA operates at 3.3V, but the WS2801 expects to see 5V as a high logic level. Therefore we will need to use a level shifter circuit. We are going to use a really simple, cheap MOSFET-based level shifter by adafruit. We could do our own level shifting, but this unit is so simple and cheap, there's no reason not to use it.

### HDMI Decoding/Re-encoding

The HDMI specification is huge, sprawling and complex and contains lots of odd video modes and assorted features. For this project, we shall limit ourselves to the 24 bits per pixel RGB modes, as these are both the most common modes and the sanest ones to support. We will also ignore HDCP encryption, EDID (which means we need to configure the proper modes on the HDMI source), CEC, and all other sorts of frills.

Electrically, the video signal is sent over four shielded differential pairs. Three are used for the color channels, and one is used for the pixel clock. Note that the pixel clock beats once for every pixel, not once every bit of every pixel. Also, the HDMI specification allows the channels to have significant delays with respect to each other. This means that we will need to perform clock recovery to receive the individual bits, and we will need to delay the individual channels to get them to line up.
The actual bits are encoded using a special form of 8b/10b encoding. Decoding the channel data yields the pixel values and the control signals. For re-encoding, this procedure needs to be reversed.

Xilinx has released a whole bunch of documentation on implementing video interfaces using the Spartan 6 family of FPGAs. The most relevant one is the application note Implementing a TMDS Video Interface in the Spartan-6 FPGA. As we can see the procedure is fairly involved, but not overly complicated, and reference designs for both the HDMI receiver and transmitter are provided along with the application note, that target  Digilent's Atlys board. The Spartan 6 FPGA series contains a bunch of hard IP that assists greatly in implementing HDMI receivers and transmitters.

The app note includes the following diagram of an HDMI receiver:

Practically this means the following:

1. We need to send all four input signals (three data/color channels R,G,B and the clock signal CLK) to differential input buffers, configured to TDMS operation.
2. CLK is fed into a phase-locked loop (PLL) and used to derive CLKx10 and CLKx2 signals.
3. CLKx10 has the same frequency as the bitstream, so it's used to clock three input de-serializer blocks (ISERDES2) and associated delay circuitry (IODELAY2). These blocks capture five bits at a time and output them every beat of the CLKx2 clock.
4. Two five-bit blocks are glued back together in the 5:10 Gear Box, yielding a potentially correct 10 bit symbol.
5. Looking for known values in the data stream, we sync up the symbols for each channel. If we don't see the known values, we shift the window by one bit and try again. Eventually every channel should end up in sync, and we should be receiving valid symbols from every channels.
6. Then we bring the channels in sync with each other, so that the three symbols of each channel are asserted on each beat of the pixel clock.
7. Finally the symbols are fed through the TMDS decoder to yield the actual RGB pixel values and control signals.

Re-encoding and transmission follows a similar process but in reverse. The pixel values are passed through a TMDS encoder yielding 10 bit symbols. The symbols are hacked up in two halves of five bits each and are shoved onto the differential pairs using an output serialization block (OSERDES2) connected to a differential output buffer configured for TMDS operation.

### Standing on the shoulders of ... hamsters?

Normally we would try and port the reference designs to the miniSpartan6+. However, it turns out we can steal borrow most of the HDMI handling from Mike Fields of Hamsterworks. His project MiniSpartan6+ DVID Logo performs exactly the kind of decoding/re-encoding we need. It's solid, legible code that happens to target the specific hardware we are using. Therefore we are going to use it as a basis for this project. Mike, if you are reading this, you're the man. Seriously.

### Averaging the HDMI signal

The decoded video stream is presented as an old fashioned VGA signal, and consists of the following signals:
1. The pixel clock. We can sample the pixel color values on the rising edge of this clock.
2. Three 8-bit values for the red, green and blue channels of the current pixel
3. HSync and VSynch, signals that alert us to the start of a line or screen respectively
4. Blank, that signals the blanking period. Originally this signal was used in CRT monitors to turn off the electron beam while it returned to the left edge of the screen after a line.
Using the three control signals and a few counters we can determine the (x,y) position of the current pixel. The value of x is increased for every pixel clock. The value of x is set to zero and the value of y is increased at the start of the blanking period (or at hsync). At vsync, both x and y are set to zero.

Using red, green, blue, x and y, we can do on the fly averaging. By comparing with x and y, we can determine what sampling box (connected to an LED) each individual pixel should contribute to, if any. Then we sum the color values into an accumulation register per LED. To obtain the average value, we need to divide the value in the register by the number of pixels. We are going to make sure that the sampling boxes contain a number of pixels that is a power of two. Then we can just wire the MSBs of the registers to the output to get the average values without performing expensive divisions.
The process for a single sampling box is illustrated in the figure below.

We are going to be using sampling boxes of 128x128 pixels. This means that our accumulation registers will need to be able to hold the sum of  2**7 x 2**7 8-bit values for every (red, green, blue) channel, and will therefore have to be 22 bit wide. In total, we will require 1650 bits of accumulators. However, the accumulator only contains the proper value at the end of the frame. Therefore we need another 600 bits to act as a "double buffer" to keep the values of the MSBs stable as we continuously drive the LEDs over their SPI interface. In total we require 2250 bits of state, which is a puny amount, even for a small FPGA like the LX9.

### Driving the LEDs

Our LED string uses a SPI-like system to set the color values for the individual pixels. It's made to run at 5V and it uses the WS2801 controller. We would like to drive it directly from the digital output pins of the FPGA, but as soon as we open the datasheet of the WS2801 controller, we notice we've hit a pretty serious snag:

In order for the inputs to register as logic level high, they need to be at 0.8Vdd. Since the LEDs are powered at 5V, this means that we need to get above 4V for the controller to register a high signal. The FPGA is a 3.3V device. This means we will be needing to perform level shifting.

We are going to use a really simple, cheap MOSFET-based level shifter by adafruit that I have lying around. It's designed to work with I2C, SPI and serial and supports bidirectional operation. The circuit follows the suggested schematic from the NXP I2C level shifting app note, but with four channels instead of two:

By passing the signals through this device we can convert between 3.3V and 5V logic levels. We must ensure every line is not driven from both ends at the same time, but in this application that is not a concern because the WS2801s in our design have no user-accessible outputs (they are wired to the LEDs).

Now that the electrical part is taken care of, we still need to write the actual values to the LEDs. For the sake of simplicity, we just continuously send the values in the SPI buffer to the LEDs, instead of trying to sync with the HDMI source frame rate. This may lead to some tearing artifact during rapid transitions, but in practice this is not a problem.

From the data sheet, the timing diagram for the WS2801 is as follows:

A new frame starts when the clock is low for more than 0.5 ms. Then we can output the RGB values R-first, MSB-first without pausing between the LEDs. Theoretically the WS2801 can operate at up to 25MHz, but since we're dealing with lots of wires and breadboard prototyping techniques, we are going to settle for a much more modest clock speed.

We generate a 25 kHz clock by dividing the on-board 50 MHz MEMS oscillator of the miniSpartan6+ board. We output all 600 bits of the SPI buffer in order on the falling edge of the clock, so that the WS2801 can sample them on the following rising edge. When we are done, we bring the clock line low and wait for the equivalent of 40 bits to reset the data frame. This means that we perform about 40 updates of the LEDs every second, This slower than the data is pouring in from the HDMI averager module, but it does not suffer from visible lag.

If we blindy output the averages to the LEDs, the colors will be very washed out. We need to perform gamma correction in order to match the colors of the LEDs to the colors on screen. Experimentally, we get the best results using a gamma value of 2.5. We create the VHDL code for the gamma lookup table using a tiny python script:

for i in range(256):
g = int(math.pow(float(i) / 255.0, 2.5) * 254.0 + 1.5)
print 'X"%02X",' % (g),

Notice how we map the values to the interval 1-255, instead of 0-255. We do this because at intensity 0 the individual "subpixel" LEDs actually turn off completely, and this results in significant color artifacts.

### Putting everything together

Now can assemble everything and wire all the components together.

In the picture above we can see the LEDs affixed to a piece of cardboard arranged in the proper pattern. The level shifter is placed on a solder-less breadboard that also contains the 5V power supply. The FPGA board is powered from the 5V power supply and drives the leds through the level shifter. The cardboard support can be attached to the back of the television using M4 screws threaded through the VESA mount, or just by using more masking tape.

Now we can hook up an HDMI source (such as my trusty old WD TV Live HD) and enjoy the show!

1. This is awesome! Can we have a look at the back of the TV?

2. what can I say, 吊吊吊！

3. If you throw this up on Kickstarter, I'll pay $50~75 for a production version with 1080p and HDCP support. The unintrusive HDMI-based implementation is much better than the alternative: https://www.kickstarter.com/projects/woodenshark/lightpack-ambient-backlight-for-your-displays 1. I consider that price-point unlikely at the moment, for a number of reasons. However, the entire thing is open source, so I dare anyone to prove me wrong ;) 2.$75 in small quantities is doable if you were to not allow reprogramability you could load the program on an asic and then don't need most of the other components/ less ram so boards would be ~\$20 to make. A resizable bracket for the LEDS would be tough to get in the price so would need to leave that to the purchaser to figure out.

3. I'm not sure how you would do an ASIC for small quantities without spending like, hundreds of dollars for each device and sinking six figures into non-recurring engineering costs and software/tools licensing.

4. I wonder if you could create this in an ASIC using www.viadesigner.com. They were supposed to make creating ASIC's cheaper and easier for small quantity projects.

5. FPGAs are already ideal for small production ASIC applications, that's precisely what they're designed for. The Spartan6 LX9 used in this project is under 20 bucks each in single quantities and it has remaining capacity.

4. Take a look: https://www.indiegogo.com/projects/ambivision-standalone-ambient-lighting-device/x/8427436

That's a hardware solution but thanks to external coverter it could copes with different resolutions.

1. Designing, manufacturing and selling a complete device (as opposed to a kit) is something that takes enormous skill and perseverance. I wish them good luck.

As far as I can tell though, it's more in the line of this project than the one described above: http://hackaday.com/2013/08/05/no-computer-ambilight-clone-uses-a-computer/

5. Hi DrX, Can you please post up a higher res pic / wiring schematic of the wiring between the miniSpartan6+ and the levelshifter/source. I am attempting to run mine of USB as my tv outputs a 5v usb but I am having difficulties and believe I may have wired things up wrong.

1. I'll draw up a schematic later, as I don't have access to the prototype right now. However, this is a brief summary of the power situation.

Depending on the number of LEDs and the color settings, the LED string can draw upwards of 2A at 5V, or 10W. In order to meet this requirement the whole thing is powered from a wall-wart power supply. The 5V from the supply powers directly:

1) The LED string's 5V rail
2) The high side of the level shifter
3) The FPGA board, through the 5V pin near the micro-SD card (when not connected to USB! Disconnect this before programming!)

The 3V3 rail of the FPGA then feeds the low side of the level shifter.
Obviously, the grounds are all connected together.

Now, looking at your description, I'm guessing your problem is that you are drawing too much current from your TV. Standard USB can only supply up to 500mA or so, which is only about 2.5W.

6. Hi, very cool project! I have a question : you say that Spartan 6 isn't enough for processing video signals at resolutions higher than 720p, is it isn't enough for convert tmds to rgb or for the on the fly averaging or for both?
I just want to implement you project, but I want to use tfp401 hdmi reciever ic which do the the tmds-rgb conversation and I trying to find out will it help to handle 1080 video stream

1. Yeah, that should totally work. The tfp401 outputs RGB at 24 bits/pixel + control signals, so you're left with routing 26 traces. The good news is that the pixel clock is only 160MHz or so, so definitely within range of FPGA IO without any special tricks. Personally I was considering using the breakout from Adafruit (https://learn.adafruit.com/adafruit-tfp401-hdmi-slash-dvi-decoder-to-40-pin-ttl-display/overview) with a matching cable. Spinning your own board is obviously also an option, but hand-soldering .5mm pitch parts can be a bit hairy.

2. Thnx for reply! I already have my own board, it was the easiest part, the hardest for me is code, I very newbie in all this vhdl staff, so can you give me some advice what changes do I need to do in main and averager parts of code to handle 1080p stream?

3. You should be able to feed averager.vhd directly from the signals of the TFP401 in 1pixel/clock mode. You have the 8bits/channel/pixel signals on QE/QO, you have HSYNC and VSYNC. You can derive BLANK from inverting DE. Just wiring these signals directly to averager.vhd and getting rid of the current input pipeline (dvid_in.vhd and friends) should give you what you need.

If you can't meet 160MHz timings, you can fall back to 2/pixel/clock mode. Then you need twice as many signals, and the logic becomes slightly more involved, but not overly so.

4. I see SDRAM ic on FPGA board, Is it used in you design of this project? I want to make my own board and want to understand will just only FPGA ic be enough or I need this external sdram?

5. No, the SDRAM is not used at all. The only pins mapped in the constraints file are the HDMI ports, the clock signal and the LEDs.

7. I've been playing with one of these LED strings and some FPGAs. The level shifter is not necessary, communication is unidirectional so the LED string is not going to put 5V on a pin and damage the FPGA, and 3.3V is within the range considered as a logic high by 5V TTL. When using very long strings level shifting might be needed but for 25 LEDs it's perfectly reliable just connected directly.

1. Are you sure you have WS2801 LEDs? As you can see in the datasheet above, the logic level high for WS2801 is 0.8 VDD or 4V. I tried to get them to work with 3.3V signals, but they just wouldn't budge.

Another strategy might be to run the LEDs at lower voltage, so that 0.8VDD is lower than 3.3V. We can run the WS2801 chip all the way down to 3.3V, so that's no problem. I can't say whether we sacrifice much brightness that way, since I don't have a properly rated 3.3V power supply.

1. Hey there! The problem is that the Spartan6 at speed grade -3 (which is the one I'm using here) cannot process signals of the bandwidth (speed) required for 1080p while remaining in-spec. However, I have heard from reliable sources that with careful manual placement of the logic you can get it to actually work at 1080p. I haven't tried yet.

I suppose you could try and rescale it with some sort of hardware downscaler like this one: http://atlona.com/product/at-hd550/ I have no idea whether that would be cost-effective, but it certainly would work. Keep in mind that the current design expects a 720p signal. It will not lock onto anything else without tweaking the receiver code.

The WS2801 and the WS2812 are not compatible. If you want to use the WS2812, you will need to recode the LED driver to output a different control signal. While the WS2801 uses more or less straight-up SPI, the WS2812 uses pulses of varying width to signal the bits. You will also need to make sure the level shifting circuit you are using can meet the timing requirements, as you don't have the luxury of slowing down the signal until it works.

9. There is now a commercial product called DreamScreen using this concept and currently have a campaign on Kickstarter http://zerocharactersleft.blogspot.se/2015/04/diy-fpga-based-hdmi-ambient-lighting.html