LightDHT - A lightweight python implementation of the Bittorrent distributed hashtable

I have always been fascinated by distributed data structures, but my efforts have always been dampened by the fact that they are really awkward to play with. In order to get anything that resembles real-life usage, you need to be running hundreds of nodes. Needless to say, wrangling hundreds of processes gets rather cumbersome. Fortunately, there are already thousands of people out there that are participating in one of the largest distributed data structures in operation today: the bittorrent distributed hashtable.

In order to facilitate experimentation with this wonderful system, I have written a simple python implementation of a DHT node. It is mostly (with a few exceptions -- see below) compliant with BEP0005. It should be able to act as a full member of the DHT, processing requests without disrupting the network.

Interested in playing around? Get the code on github!

From the documentation:

The aim of LightDHT is to provide a simple, flexible implementation of the Bittorrent DHT for use in research applications. If you want to trade files, you have come to the wrong place. LightDHT does not implement the actual file transfer parts of the bittorrent protocol. It only takes part in the DHT. 

The main philosophy of LightDHT comes down to two considerations: ease of use over performance and adaptability over scalability. 

In order to keep LightDHT easy to use, all DHT RPC calls are performed synchronously. This means that when you call a DHT method, your program will block until you have an answer to your request. That answer will be the return value of the function. This has the advantage that it keeps the logical program flow intact, and makes it more comfortable to use. 

In order to maintain O(log N) scaling across the network, BEP0005 (the standard governing the DHT) mandates that implementations use a bucket-based approach to the routing table. This enables the node to fulfill all requests in constant time and (more or less) constant memory. In LightDHT, we throw that recommendation to the wind. 

Since the main focus of LightDHT is reseach, we are going to keep around all the data we can. This means that we keep around every single node we know about. Since in practice the number of nodes is limited and the request rates are rather low, we do not bother keeping the routing table organized in a tree structure for quick lookups. Instead we keep it in a dictionary and sort on-demand. The performance penalty is well worth the reduced complexity of the implementation, and the flexibility of having all nodes in an easy to use data structure.

Festive and Nerdy

You might have seen those big outdoor displays where every pixel is composed out of a little three-color LED light. In this post I'll tell you how I turned a string of those pixel lights into Christmas tree lights!

Adafruit industries, one of my favorite suppliers, recently started selling strings of a particularly advanced model of these pixel lights. Have a look at the product image, and tell me if it doesn't already look like a string a Christmas tree lights..

"12mm Diffused Digital RGB LED Pixels"

As you might have seen in my previous post, these pixels are rather easy to control from a micro-controller. However, stuffing a breadboard in your tree is not exactly festive. Thankfully, you can get these handy little tins that are just about the right size for a wall-wart plug, a teensy 2.0 with female headers (courtesy of Pieter Floris) and a big mess of wires.

A wall-wart plug, a Teensy2.0 and a big mess of wires.

Note that, thanks to the female-pin Teensy, everything is wired up using jumper wires. That's right - this is a no-solder project. Because the underside of the Teensy is not isolated, and because I have a number of wires in there that might come loose, I have isolated the interior of the tin using electrical tape. This is most likely not compliant with any code enacted after 1953 or so, so any imitation is purely at your own risk.

The only remaining question is how to fasten the plugs and wires onto the enclosure. Ideally you would use specially molded plastic caps and bits, but the odds of finding them in the right size and shape at your local hardware store are quite slim. Instead, I liberally applied black sugru around the holes I had cut into the tin, and then pressed the wires and the plug into it. The sugru just molds around the objects. After a quick touch-up I left it to cure overnight, and the result was really really good.

Now we just pop closed the tin and voila, a presentable programmable Christmas tree lights driver!

The controller and a single string of LED pixels, running a test pattern.

And here's the tree, all festive and nerdy:

Festive and Nerdy

It's the season! (preview)

I while ago my eye dropped on this string of individually addressable RGB LEDs, and I realized they are practically ready-made Christmas lights. A quick order, a long wait, and an evening with my Teensy++ later, I have a sneak preview of our new Christmas tree decoration!

A WS2801 LED string is practically already a Christmas tree decoration!

Since this creation will actually be used "in production" I need to spend some time making it "living-room acceptable". I can tell you this will involve some patience and a lot of Sugru!

 

PCB as a Capacitive Soil Moisture Sensor

Electronics enthusiasts have been designing ways of automating plant care for decades, with mixed results. A traditional weak point of these installations is the resisitive moisture sensor, that is inaccurate and prone to degradation. In this post we will design and fabricate an inexpensive capacitive soil moisture sensor out of a printed circuit board that exhibits none of the weaknesses of its resistive brethren.

Selectively Unchecking Iterators: the key to High Performance C++ in Visual Studio

In Visual Studio 2005, Microsoft introduced checked iterators for STL containers. This means that every single operation that uses these iterators is range-checked to prevent memory accesses outside the container's allocated storage. If an range violation occurs, the program terminates with an exception at the location of the violation, instead of trashing potentially unrelated memory as C++ has been happily doing for decades.

This safety net comes with a price: every single iterator change undergoes scrutiny by the runtime to determine whether it is safe. For most programs this is a paltry price to pay for the peace of mind of safe container access. However, for performance sensitive code that include tight loops that iterate over STL containers, the penalties can be massive.

The folks at Microsoft realized this, and implemented a way to disable this functionality. If you define _SECURE_SCL to 0, the checking goes away. You are back at the level of performance (and unsafeness) of  C++ that you know and love. Unfortunately, there is a little snag. You can't exchange STL containers between modules that are compiled with _SECURE_SCL set to 0 and modules that are not. It may look like it's working, but it will blow up in your face when you least expect it. Also note that std::string is an STL container, and it features prominently in most modern C++ interfaces. The reality of the matter is this: if you are defining _SECURE_SCL to 0 you cannot use any C++ code that you did not compile yourself. No libraries, no DLLs, no nothing. For any non-trivial program, that's a pretty tall order.

Also, by setting _SECURE_SCL to 0 you are disabling iterator checking for the entire application. In any given program there are maybe 100s of places where STL iterators are used, but only a handful (one function, two iterators in my current code) where these checks hinder performance to any real degree. Disabling iterator checking globally, if it is even possible given the limitations discussed previously, is rarely desirable.

It turns out that, as of VS2010, there is a way to selectively disable iterator checking. It is a fairly obscure piece of Visual Studio trivia, and I will share it with you so that it is preserved for posterity:

std::vector<int> v; 
for (int j=0;j<1024*1024*5;++j) {  
    v.push_back(j);}  
}  
std::vector<int>::iterator i = v.begin();  
std::vector<int>::iterator::_Unchecked_type ui;  
int tot = 0;  
std::vector<int>::iterator::_Unchecked_type uiend = _Unchecked(v.end());  
for (ui=_Unchecked(v.begin());ui!=uiend;++ui) {  
    tot += *ui;      
}  

Note that we can uncheck the end iterator outside the loop, as the size of the vector does not change during the iteration. I encourage you to try it with checked and unchecked iterators to get a feel for the difference in performance. Be aware however that the inner loop here is very thin: performance gains will be inflated since the inner loop goes from doing mainly range checking to doing mainly nothing!

Readers are encouraged to contribute a set of macros that abstract iterator unchecking in a convenient way.

Attacks on Certificates are a Good Sign

Recently, there has been a bit of a commotion regarding hackers gaining access to valid SSL and code signing certificates. First there was the stuxnet worm, that was signed with actual certificates stolen from Realtek and JMicron. Then there was comodo-gate. The most recent, and most serious incident happened at dutch company diginotar, and resulted in an (as of yet) unknown number of certificates being issued for unknown sites to unknown parties.

As reporters start to catch on to the magnitude of the diginotar case, and begin to understand the trust implications of a compromised root Certificate Authority, the conversation takes a turn toward gloom and despair. In this post I will offer an alternative reading that strongly suggest that hackers attacking certificates and certificate authorities is actually a huge step forward for security and privacy on the internet.

Ephemeria, the latest just-in-time procedural content experiment

The last few months I've been dabbling in procedurally generated content. My latest project is called Ephemeria, a just-in-time procedurally generated city.

I keep track of a 3 block radius around the player. The good thing is that the whole generation thing is repeatable. If you were to more 3000 miles in a random direction and come back, all the buildings would be the same. We can then keep track of any player- or script-induced alterations and replay them every time a block is re-generated.

I need to figure out what I want to use it for before digging too deep into improvements. For example, for a helicopter type of simulation I need to flesh out the roofs. For a racing sim, I need to get to work on the ground-level geometry. Either way, better building geometry is essential.