IDRing

A major feature of the Romana Project is topology-aware IPAM, and an integral part of it is the ability to assign consecutive IP addresses in a block (and reuse freed up addresses, starting with the minimal).

Since IPv4 addresses are essentially 32-bit uint, the problem is basically that of maintaining a sequence of uints, while allowing reuse.

To that end, a data structure called IDRing was developed. I’ll describe it here. It is not yet factored out into a separate project, but as Romana is under Apache 2.0 license, it can still be reused.

  • The IDRing structure is constructed with a NewIDRing() method, that provides the lower and upper bound (inclusive) of the interval from which to give out IDs. For example, specifying 5 and 10 will allow one to generate IDs 5,6,7,8,9,10 and then return errors because the interval is exhausted. 

    Optionally, a locker can be provided to ensure that all operations on the structure are synchronized. If nil is specified, then synchronization is the responsibility of the user.

  • To get a new ID, call GetID() method. The new ID is guaranteed to be the smallest available.
  • When an ID is no longer needed (in our use case — when an IP is deallocated), call ReclaimID().
  • A useful method is Invert(): it returns an IDRing whose available IDs are the ones that are allocated in the current one, and whose allocated ones are the available ones in the current one. In other words, a reverse of an IDRing with min 1, max 10 and taken IDs from 4 to 8 inclusively is an IDRing with the following
    available ranges: [1,3], [9,10].
  • You can see examples of the usage in the test code and in actual IP allocation logic.
  • Persisting it is as easy as using the locker correctly and just encoding the structure to/decoding it from JSON.

A post-mortem of a project: Wildboard

Once upon a time, I was working at Snaplogic, and at that time its office was in downtown San Mateo. Pretty much across the street from a great coffee shop, Kaffeehaus.

If you’ve been there, or even if you just looked at the website, you’d realize that the owner put quite an effort into it being a Viennese-style coffee house, with all the interior design decisions that go with it.

Now, a local coffee shop is often a place where people expect to post some local notices and ads (“lost dog”, “handyman available”, “local church choir concert”, etc). And here’s a conundrum. A simple cork bulletin board with a bunch of papers pinned to it just did not seem to fit the overall mood/interior/decor of the cafe:

Yet the cafe does want to serve local community and become an institution.

This being Silicon Valley, Val, the Kaffeehaus owner, had a vision — what about a virtual board, as a touch-screen.

The name was quickly chosen to be Wildboard — because it is, well, a bulletin board and in honor of the boar’s head that is prominently featured on the wall:

.

A multi-touch-based virtual bulletin board sounded interesting. Most touch-screen kiosks I’ve seen so far — in hotels and malls, for instance, or things like ImageSurge — only allow tap, not true multi-touch. (To be honest, multi-touch may or may not be useful — but see below and see also P.S. — but it is a very nice “pizzazz”).

And we — that is, myself and Vio — got to work. And in short order we had:

  • Fully multi-touch (with rotation, zoom, etc) web UI — as a Windows 8 CSS/Javascript app (source).
  • Wildboard “board server” — a Python app running on the same computer as the UI. It is responsible for polling the web server (below) and serving information to the UI (source).
  • Wildboard web server — a PHP app based on an existing web classified application(source). This allows users to submit ads (or they can do it via a mobile app, as below). It is also modified to automatically create QR codes based on user-provided information (map, contact, calendar, etc) and adds them to an ad.
  • Wildboard mobile app — PhoneGap/Cordova based app for both Android and iPhone (source)
    This app allows one to:

    • Post an ad
    • Scan an ad’s QR code
    • And, finally, for the “Wow!” effect during the demo, one can drag an ad from the screen into the phone. Here it is, in action:

  • Wildboard orchestrator — a Node.js app (source) designed to coordinate interactions between the mobile app and the board. It is the one that is determines which mobile app is near which board and orchestrates the fancy “drag” operation shown above.
  • For more information, check out spec and the writeup.

Charismatic Val somehow managed to get a big touch screen from Elo Touch. Here’s how it fit in the decor:

A network of such bulletin boards, allowing hyper-local advertising, seems like a good idea. Monetization can be done in a number of ways:

  • Charging for additional QR codes — e.g., map, contact, schedule.
  • Custom ad design (including interactive and advanced multimedia features — sound, animation, video).
  • A CPA (cost-per-acquisition) model, while tracking interaction via an app — per saved contact, per scheduled appointment, per phone call.
  • Premium section.

But… alas… This is as far as we got.

P.S. One notable exception is a touch-screen showing suggestions in Whole Foods in Redwood City.

OpenDSP – stay tuned

Every marketer, it seems, wants to participate in real-time bidding (RTB). But what is it that they really want?

They want an ability to price (price, not target!) a particular impression in real-time. Based on the secret-sauce business logic and data science. Fair enough.

But that secret sauce, part of their core competence, is just the tip of the iceberg — and the submerged part is all that is required to keep that tip above water. To wit:

  • Designing, developing, testing and maintaining actual code for the UI for targeting, the bidder for bidding, reporting, and data management
  • Scaling and deploying such code in some infrastructure (own data center,
    clouds like AWS, GCE, Azure), etc.
  • Integrating with all exchanges of interest, including the following steps:
    • Code: passing functional tests (understanding the exchange’s requirements for parsing request and sending response)
    • Infrastructure: ensuring the response is being sent to the exchange within the double-digit-millisecond limit
    • Scaling: As above, but under real load (hundreds of thousands of queries per second)
    • Business: Paperwork to ensure seat on the exchange, including credit agreements when necessary
  • Operations: Ongoing monitoring of the operations, including technical (increased latency) and business (low fill level, high disapproval level) concerns (whether these concerns are triggered by clients, exchange partners or,
    ideally, pro-actively addressed internally.

None of which is their core competence. We propose to address the underwater part. It’ll be exciting.

Enter OpenDSP. We got something cool coming up here. Stay tuned.

Watch this space, and by this space I mean this blog, this GitHub account.

Continuous integration with code coverage in Python

Update

This code has been integrated into the main tree.

I decided it would be good to have a coverage report of our Python code,
with nice visualization like Clover.

So I took Ron Smith’s PyAntTasks, and added py-cover task to them. This will run coverage for every test, and a cumulative one. In other words, you can see what code a particular test exercises, and what code all the tests in your tree exercise.

This also modifies py-test task to include packagedtests attribute – see below.

The newly added py-cover task runs Ned Batchelder’s coverage.py (download it separately), and is specified as follows in your build.xml:

 

    
      


     
     


Here, is a FileSet specifying tests to run, is a FileSet specifying source code to cover.
The attributes are:

  • reportsDir – where the coverage reports go
  • packagedtests – this idiosyncrasy is prompted by our tree setup. If this attribute is true it means that the test files reside in Python packages, false otherwise. (In our case, they do not; they are in the tree but are not packages. Note that the original py-test task assumed they are in packages, I have changed this too).
  • coverage – path to coverage.py on your system (which you downloaded separately, right?)

P.S. I know about the colorize.py thingie, but I rolled my own (uglier, of course) for this one.

XmlRpc for AS 3.0 HOWTO

Since people ask (in emails, at the short-lived MT version of this blog and on flexcoders@yahoo), here is how I use it.

DISCLAIMER: as I’ve mentioned, I hacked this from the original when I was just starting with Flex 2, so I gutted some code before I knew what the hell I was doing,
but in this state it works for me… 

I use Flex Builder. I have the contents of the zip file right in my Flex Builder project. (NOTE: This was before I read that flex should not be a part of a package name per Adobe’s license; change it, willya?)

Then I choose have a simple facade like so:

package com.qbf.flex.ct {
  // Insert correct imports here, of course...

  public class Utils {
    public static function serverCall(method:String, 
                                          params:Array,    
                                          handler:Function): void {
        SERVER_CALL.call(method, 
                         params, 
                         handler, 
                         DisplayObject(Application.application));
    }

    public static const SERVER_CALL:XmlRpcService = 
   new XmlRpcService("http://localhost:8081/", 
                        Application.application.url.indexOf("http://") == 0? null :
   null :
   // This is explained further below... 
   QbfFakeResponse.SINGLETON); 
}

And from everywhere else in the code I use Utils.serverCall() as
follows:

var params:Array = [{value : resdefDataToServer, type: "struct"}, 
uri, server];
Utils.serverCall("doStuff", params,
        // This callback will be executed when the server 
        // call returns
 function(result:Object):void {
           process(result);
        });

Notice that you only need to specify types for parameters if it’s not a string. If all your parameters are string, you can just pass an array of their values to the serverCall() –a little sugar…

Now about this QbfFakeResponse.SINGLETON business (it’s not necessary, just use null in its place if you don’t need it). The definition ofSERVER_CALL checks whether the application URL starts with http:// — in my case, it means that it’s running “for real”. If it is running in Flex Builder’s debugger, the URL will start with file://,
and so I will use “fake” responses to simulate server calls (this is sometimes useful for debugging, so that I don’t have to run a server). This fake response class is an implementation of com.qbf.flex.util.xmlrpc.FakeResponse interface. All you need to do is implement getResponse() method and return whatever you need by checking the method name and parameters provided.

Dbdb – a JPDA-based single-stack debugger for mixed-language programming

Dbdb project is officially up for adoption, because I have no plans for working on it (I am sick of it).

Dbdb is a proof-of-concept of a JPDA-based single-stack debugger for mixed-language programming, done as an Eclipse plugin (but doesn’t have to be). It is based on Java 6 (“Mustang”). The proof-of-concept is allowing a developer to debug Java code that calls a PL/SQL stored procedure. The debugging session in Java proceeds normally, nothing to write home about. When a Statement.execute() (or similar) statement is executed, however, the debugger connects to the Oracle’s VM and shows a combined call stack, from Java down into PL/SQL. (See screenshot). The idea, of course, that it can be done with other combinations, but Java-into-Oracle-stored-proc is a very common scenario.

P.S. This is a rehash of an older post. I am trying to see what Blogger is like vs. LJ (for instance, LJ breaks javablogs feeds).