Sorta Live

JSON support in native code.

2008-11-17T00:02:00.000-08:00

I recently needed to look at JSON support libraries for a project I'm working on. It was required that it be a native C or C++ implementation. Some Boost libraries were available. This is a brief report on what I've found so far.

json-c v0.7 (internally its marked as 0.3) was the first one I looked at, mainly because I had used before in an Objective-C environment.

Building it on OSX was easy, creating a libjson.dylib that I could link to. The programming interface is basic C, creating 'json_object's, and adding them to other 'json_object's and 'json_object_array's. There is an incremental tokenizer/parser, as well as the ability to convert a JSON object into a standard JSON string. Its simple, seems lightweight, and I know it works since I've used it before.

TINYJSON-1.3.0 is the next one I examined. This one is heavily dependent upon the Boost C++ Template Libraries, also pulling in the Boost Spirit parser, that I'm unfamiliar with. This was my first attempt at using Boost on OSX, so I first had to pull it down, build and install it, not a big deal. The TinyJSON distribution included a basic set of Boost TEST_CASE's which took me awhile to figure out how ot configure in XCode. Next hurdle is the programming model - its all based on C++ templates, which is useful but sometimes hard to fathom.

Main problem with it though is that its only defined for reading JSON data, not writing it. I need writing.

JSONCPP is the last one I've tried so far. This one looks promising, but I've had problems just getting it to build - it uses something called 'scons' to manage its build process. After following all the instructions, I had no luck. I tried pulling the project tree into XCode and building it manually, and was able to make that work at least. Object structure looks promising, although it has the usual c++ verbosity. Worse though, are no examples and I find that all of the test code is written in Python, and doens't work. i've probably not built something correctly, I just don't know what yet.

A couple of other libaries that I haven't had time to examine closely yet: Jost and Jaula, the former is C++, the latter strictly C. A closer look at Jost is required, as I see it is also Boost-based and it seems relatively simple.

After this little exercise, I'll probably just use the libjson support - its simple and it works. If/when I get more time, I'll look into the C++ versions again.

I was surprised at the dearth of C++ -based versions, I suppose that this is because libjson support is good enough.

OSCON 2008 summary (belated)

2008-08-31T15:27:00.000-07:00

I attended OSCON 2008 in Portland, Oregon in July.

It's a good opportunity to see what the Open Source community is up to, and what interesting
technologies capture the attention. There were a lot of presentations, many of them concurrent, fortunately many of them have also made it online.

I took a few pictures while I was there, you can find them at6 my flickr oscon set.
(Also, they had a photo contest from prior years - and one of my balloon photos from 2007 won first place!)

General themes I found most interesting, and which had a lot of other interest:

large data sets and their implementations (Hadoop, Bigdata)
scaling and performance (Facebook, Flickr, and others)
propagating data using Jabber's XMPP protocol (aka XMPP Pub Sub)
OAuth (open authentication protocol for services)
Microblogging (identi.ca)
dynamic languages (Groovy, JRuby,Ruby,Erlang)
web frameworks (Django, RoR, etc.)
virtualization (I attended only a couple of these, lots of interest though)

Of the sessions I attended, here are a few highlights.

****
Open source applications making inroads into mainstream usage:
Django, Alfresco, Zimbra.

****
The XMPP Pub Sub idea got a lot of interest. It was inspired by the massive crawling
that FriendFeed was performing on Flickr, looking for images updated. This idea will enable a listener to subscribe to content changes, reusing the Jabber XMPP IM protocol (containing an Atom payload.)

****
Groovy vs JRuby discussed when you might prefer one language over the other.
Summary was that:

Groovy had a better fit with Java, good (and improving) performance and is best for tight and/or heavy Java code integration. Also there is a Groovy compiler that will generate .JAR files, useful for "stealth" integration.
JRuby code integration with Java good but still needs work. Best for general scripting and light code integration. But you have both the JDK and Ruby libraries to work with.

Note also that Sun has dedicated resources for JRuby (and Jython)

****
Facebook developer's mentioned they had >400 memcached hosts, using multi-retrieval code
that they've written (and shared.)

Also some discussion on using TCP vs UDP, and about high-latency problems caused by East/West coast server traffic.

****

Web Frameworks

There were several discussions centered on frameworks. One new term was SOFEA. Idea consists mainly of moving more functionality into the client, rather then relying upon server-side framework to perform everything. (Basically this is what's happening with the RIA model, such as Dojo, Laszlo, and Flex enable.)

Some quick summary judgements on different frameworks:

OpenLaszlo: pummeled by Adobe
JavaFX: no one using it (yet at least)
Rails: ActiveScaffold adds REST and Ajax, Google Trends shows peak in 2005.
Grails: scaffolding not very good yet, but has better performance than Rails
Flex: Flex + Rails an interesting platform, doesn't yet support all HTML well.

Performance notes:
LinkedIn has a Rails-based Facebook application that supports 1M requests/month - named "Bumper Sticker".

On a comparison scale, in turns of ms/iteration, we see that:

Java, C++ : very low, less than 1;
JRuby: 100
Groovy: 215
Python: 225
PHP: 600

In a session I missed, but saw notes from later, there were some significant performance improvements in Ruby performance too. They've introduced a compiler that generates code for the LLVM machine, and it's much faster than the C-based "Matz" interpreter. Name of this project is Rubinius.

****

Mozilla developers discussed ways to implement static analysis of C++ programs.
They have developed a plug-in for GCC that let's them run JavaScript in the compiler, giving
them access to the program graphs directly. This is "Dehydra" project.

They're already using this technique to refactor some of their existing code - and they're
actually converting it into JavaScript - more on this in a minute. Now that they have access to the program graph, they're looking for other things they can do to the code too: security analysis, bug analysis, standards enforcement, etc.

About the JavaScript conversion: they're using Trace Compilation in the interpreter that takes
care of performance bottlenecks. Doesn't fix one-time execution though. Still, they claim that
much of their code is easily translated into JS, and its just as fast as the C++ and safer. Target language is JS 2.0 which allows for class/struct within traditional JS objects, making
them safe for C access.

****

The Open Microblogging discussions were interesting. 'identi.ca' is an open source service using Twitter api, that's federated (supports multiple servers). Project code is 'laconi.ca'. Currently uses HTTP between servers, but they know this won't scale. They intend to use the XMPP Pub Sub idea. There's a lot of interest in this technology, and some good ideas.

****

Mark Shuttleworth from Canonical (Ubuntu's business org) gave an interesting talk about their development practices. Its a mix of lean and agile techniques. They really try to amplify learning and not specialization. "Decide late, deliver early" was mentioned. But then later he talked about how knowledge and expertise was more important than colocation, so obviously some level of specialization occurs.

Some of the other practices mentioned:

cadence/cycle
track bugs, features, ideas
branch/merge, keeps cadence trunk pristine, merging important.
code review, but they stay away from voting, too divisive
automated tests: unit, integration, utilization, full app, profile usage.
pre-commit testing, with trunk locked to everyone but a robot that runs test before commit.

****
There was a good discussion on Python & C++ integration using the SWIG modules.
SWIG is now based on a full compiler that reads C/C++ declarations and generates C extensions that allow Python access. Python code can even extend and override C++ classes.
One might have to cleanup the headers a little, but often it works just fine. While its not the best generated code,it works and cuts out most of the work that would otherwise need to be done by hand.

As an example, they put wxPython on it. 6M LOC, 90-95% generated by SWIG, enabling one person to do most of the maintenance on it. (I experimented with SWIG 2-3 years ago and wasn't encouraged, looks like its improved!)

****
A different discussion from an engineer at Meebo, describing how they implemented a hiring process, and the trials & tribulations they went through trying to "staff up". One unusual practice they instituted was a "simulation" where did a 4-hour exercise reflecting "everyday" tasks. (Can't have candidates do real work though.)

****
There was the big announcement that Microsoft was becoming a Platinum member of the Apache Software Foundation, as well as assurances that they are continuing to evaluate and license "open source" technology. PFIF/Samba agreement, PHP support in Win2008 (ADODB), Ruby libs were all mentioned.

****

photographic technology mashup

2007-11-17T22:38:00.001-08:00

I recently had a chance to experiment with making digital negatives and using them for contact printing with platinum/palladium emulsions.

Above is a quick scan of one of the images, to give you some idea of what one looks like - although it does not do justice to the final print.

This was something I've been wanting to experiment with since I first heard about using imagesetters for this purpose, but just never got to it.

Recently, Ron Reeder taught a workshop on the subject at Photographic Center Northwest, so I couldn't resist any longer. Given my background in digital printing and photography in general, it was an easy way to build on both of these experiences and create something new. The workshop itself was two days long - the first day covering the theory and calibration process, the second day generating negatives and printing them.

The technology is elegantly simple: you take the image you want to print, apply transfer functions to it to map its density range into the range of the print emulsion, then generate a full-sized negative onto translucent film. (Its the determination of these transfer functions, the calibration process, that adds to the complexity. But being able to do this step on a computer makes this task far easier than working with film - as photographers who used this technique in the 19th century did! I'm certainly not complaining!)

This negative is then used to make contact prints in the wet lab. For this workshop, we were using hand-applied emulsion that's sensitive to UV light - approximately 5 minute exposures, then "developed" and "cleared". This process is using chemicals I'm not used to handling in the darkroom, but which are not particularly noxious or otherwise dangerous to handle. The result after washing is an archival print, with all the smooth gradations that only a contact print makes easy, perhaps slightly soft but not in a bad way.

This same technique can be applied to other historical and modern printing processes, cyanotypes and albumen prints are something I'd like to try, and even modern silver gelatin prints can be created, with some variations. I'll probably experiment with silver a bit first, because I have to build UV light box in order to work with many of the other techniques.

I can work with silver now - I still have a functional wet lab setup (which admittedly does not get used much these days.)

A good starting point for this technology, as well as some wonderful images created using it, are available at Ron Reeder's, site.

catching up on this week

2007-10-25T23:17:00.000-07:00

this week started out bad - i got completely derailed from my integration work into investigating customer issues with a new release. I must have burned 2.5 days nailing this stupid thing down. the usual problems getting things setup and reproduced - and of course the debug build didn't show the problem. eventually i found it - a measly little off-by-one error where one of my predecessors compensated for the null byte of a C string twice. In just the right circumstances, the source string would exactly fill the allocated storage, and attempts to copy out that extra byte caused an access fault. first i had to decode the algorithm though - turns out its some hashing variation from Sedgewick - it completely avoids compacting in return for quick response. I just wonder how quick it is now compared to the more standard implementations such as STL. And of course this is legacy code with no test cases.

iPhone Tech Talk on Tuesday was moderately interesting. Much of the technical material now available online, at least in video format, but it was good to have some discussion around it. I thought the UI portion was particularly good - advice for general UI design, with a slant toward mobile devices.
Very customer-centric and conceptual - sometimes develoers et al have a hard time with drilling down into "features" a bit early.

Also, we had a good trails-planning session last week, and should have some awesome work planned for Grand Forest East this weekend. Come help! see http://www.bitrails.org/ for details.

Oh, also last week, SeaJUG hosted Daniel Shore talking about Agile Development implementation, quite a good discusson on both the technical and structural/social aspects of implementing any agile methodology.

mid-October

2007-10-14T21:57:00.000-07:00

things are kind of busy as usual, have the following projects going on recently:

on the technical side:

Learning about ASP.NET 2.0 Http pipeline, and how to insert modules into it. Pretty slick, and a lot easier than writing ISAPI filters in C. I've been working on an authentication layer for a legacy application using this stuff, and there's a minimum of hacks required.

Trying to learn enough DreamWeaver to maintain an existing school web site, and enough CSS to apply it to a revamped design.
So far so good, thanks in part to DW providing basic CSS templates I can hit the ground running.
Its all pretty static content, all client-side, but there are monthly meeting updates that need to get posted, so it changes enough I want to make it easier. Already I've gotten rid of the tables, and all the little pixel-GIF hacks (and replaced them with CSS pixel hacks I guess!)

Have to read another chapter of the Erlang book this week - Mnesia database is next.

on the local side:

a good walkthrough on Grand Forest East yesterday. Trying to get a feel for future maintenance work and possible reroutes.
Nothing firm yet - current goal though is to "overlay" a new trail network on the old one, providing something easier to maintain and more suitable for all users. None of it is easy, but over the years both volunteers and staff have put a lot of time into maintaining the old trail network, that started out as logging roads of course. Now if we could just get some of the many equestrians who use those trails to help out...

attended the North Madison open house. sheesh, you'd think it would not be a big deal to widen a shoulder on the uphill side, and install a pedestrian path on the other. There seem to be a couple of arguments against it,including "preserving the rural ambience" and 'keeping it safer by keeping it narrow". My thought is that there are enough walkers, joggers, and cyclists that keeping this arterial as a no-shoulder road is foolish, not safer. Its only preserving the "rural ambience" for people driving, those of us to walk or cycle along it just put up with it. Frankly, I think much of the energy is being driven by people who, over the years, have incorporated part of the existing right-of-way into an extension of their own property, and now don't want to give it back. Everyone seems afraid to say it, but given the folks who attended this meeting and were the most loudly opposed, it seems pretty obvious. In any case, its time to realize that if we want to build a safe route for someone other than auto drivers, roads have to change to accomodate multiple users safely.

enough for now.

trails update

2007-09-24T08:12:00.001-07:00

on Saturday, 9/22, we had our first trails work party under the new 'regime'. no last-minute phone calls to remind people. no signage put out several days before to drum up local support (and guide people). no phone calls to the Park District staff. Oh, and competition from local soccer leagues.

so guess what happened? Three people showed up. Parks staff forgot. Combined with myself and Ed (the other crew leader) that makes five of us, working with whatever tools we brought, which wasn't a lot!

end result: we got a good amount of work done - finishing up the sides & surface of the trail from the main junction down to the Fort Ward fence. Maybe its better that so few showed up, we got to work in a couple of clusters, and move quickly.
also we had a chance to complain about staff once more forgetting about our regularly-scheduled work party. Even after they signed up on the mailing list.

For those of you who need a reminder or want to know where the next one is located: its always 4th Sat of the month - signup for email on http://www.bitrails.org.

when is Blowfish valid?

2007-09-11T23:40:00.001-07:00

apart from the usual weird technical problems this week (GUID changes, encrypting ASP.NET credentials, Cocoa/WebKit/Java interactions on Panther) I've got to figure out what's wrong with a C-based implementation of Blowfish, and how to make it right AND make it match the results of the "official" Java implementation. It seems to work correctly with a "known vectors' test case, and round-tripping works, but the results don't match the Java version. Block-cipher, sensitive to the endian-ness of the machine, character representation, all probably contribute to obfuscating the problem. i need to start with a reference implementation I think.

trails all over

2007-09-11T22:47:00.000-07:00

two trails-related mtgs this week, the first of the Trails Advisory Committee (TAC) for Fall 2007, the second an open house sponsored by the Non-Motorized Transportation Committee (NMTC). Both mtgs lightly attended (we are having awfully nice weather after all.)

The NMTC showed maps of potential trail corridors, for addition to the City GIS so that they show up during future development projects, before they get a house built smack in the middle. A lot of trails have disappeared over the years, but there still is a loosely-connected mesh. The NMTC is working to identify what's there, and help determine where best to add new trails to offer better connectivity throughout. (sounds like a network doens't it! next we'll be talking about the graph topology...)

As for the TAC mtg, we are at some sort of inflection point I think. Apparently when the old District dissolved and transferred to the new District, they passed along the entire reporting structure of the citizen advisory committees over to the staff, who
seem to have a decidely different take on the role of volunteers advising them.

What happens next depends upon both staff and TAC members. Since the new staff have taken over, we've had several points of contention - and each time they've revolved around the construction of a trail that had not been reviewed. Often this was at the urging of someone "higher-up" and the resulting trail violated both the existing trail standards (such as they are) as well as the model of low-maintenance trails we've been attempting to develop.

The most recent occurrence of this is the weird little trail that was punched through at the old Lovgreen Road gravel pit, supposedly to improve non-motorized access. I don't think we're going to see many cyclists using it for commuting - its way too steep, has a 90 degree turn halfway down the slope and a 180 degree turn at the bottom. I'll give them credit - they built one helluva culvert - we'll see whether its enough to deal with the runoff on that slope.

Much of the TAC work has been to go in afterwards and fix things, and its getting a little tiring. Sometimes they're more than a volunteer group can manage effectively using manual labor. its one thing to fix up historical trails, half of which were old logging roads or jeep roads anyway, but to find new ones built w/o regard to topography or aesthetics is very frustrating.

As a TAC member, and as a citizen, I've asked informally for a trail-building protocol, and was flatly denied by staff during this meeting, although staff then did promise to try and work with us. Exactly which trail is so important to build "right now" that it needs to be built w/o design review and/or community input, I'm not quite sure. Okay, i'm being cynical.
Maybe this will work out with an "informal" protocol, but given the past incidents I have my doubts.
If not, I guess the next step is to petition the Board or the City Council to establish a formal design review process - and that probably really will slow things down!

In the meantime, next work project continues at Fort Ward Hill trail, and supposedly a design process is going to start in Grand Forest East this winter.

local political & blog scene heating up!

2007-08-18T16:04:00.000-07:00

I guess its just the coming election - but we're suddenly seeing a lot of blog activity locally. I count at least 4 actively covering Bainbridge Island and its diverse politics. Attack ads in the local newspaper, stealth telephone campaigns, mailers and well-funded but "cheap-looking" hand-painted signs seem to be the tools of the local property-rights/anti-environmental/anti-govt camp (seems to be a vendetta - they don't propose their own candidate); while so far doorbelling and support letters are used by the candidates themselves. Gee, which do you think will be most effective in a campaign? I hope not the attack ads, although it certainly worked in national elections.

I hope that after the election some of these people remain actively involved. There is (and probably always has been) a lot of crap that goes on, some of it intentional, some of it simply naive. There are real issues to sort out, implications understood, decisions made.

Here's an example that bugs me at the moment: a planned boardwalk & trail linking a neighborhood to Winslow and the Ferry just disappeared off the table.
provides a picture and info.
What happened to this? I've worked with members of the NMTP and the TAC for a couple of years now, helping to site and scope this potential link. Its been discussed at several TAC meetings. The City has known about it, and supposedly budgeted for it. City wetlands experts were brought in. Permit issues were discussed and resolved. The whole project was a no-brainer: just a simple matter of finding the money for the boardwalk. Not cheap of course.

In a recent TAC meeting, when asked about this project, Park District staff claimed to have never heard of it, and questioned the need for it anyway. Case closed. Nothing to see here. WTF? First off, the City was supposed to be funding it as it was a non-motorized link. And its been discussed several times in open public meetings. So, what's happening here? At this point I don't think people really forgot, althought its certainly possible. Perhaps their workload just exceeded their capacity/funding and so they decided to reprioritize projects themselves? Unfortunately with an active volunteer community and neighborhood groups, that's neither fair nor just. But I guess what it comes down to is that we don't really know. One month this project was one the list of work to be done, awhile later its gone. Who made that decision and why?

struggling with authentication, nicely

2007-08-09T00:38:00.000-07:00

I've been trying for a way to merge the two worlds of XHR and ASP.NET authentication. Talk about a foot in each camp! I keep getting close, and then it slips away from me again. The power to rebiuld the HTTP processing pipeline would seem to provide a lot of power, yet the otherwise-closed source of FormsAuthenticationModule keeps getting in the way. The insertion of an HttpModule allowing me to intercept its attempts at redirect seemed like the answer, but so far it blindly proceeds anyway.
its late, hopefully I'm just screwing up something.

anonymous local blogging

2007-08-08T23:06:00.001-07:00

Today i ran actoss another blog started in my community (biword.blogspot.com). It had a first posting questioning the thoroughness and bias of the local print media - they've been going up and down for years and have never been known for their depth of coverage. Great, they'd like to provide detailed, in-depth coverage of stories ignored by local print media. Nothing wrong with that idea.

So then I want to the Profile section, since they're local I figured I might know the authors - when I found the blog creators had decided to remain anonymous. I can see why commentors on a story might want to remain anonymous, but I find the credibility of someone who wants to replace local media a little suspect when I don't know who they are.

I guess we'll see. There are certainly reasons why an author of some pithy piece of local news would want to remain anonymous.
We all have stories we'll only tell over a beer. Is an anonymous blog the place for them? I guess I'll have to create a pseudonym if I want to share any of them now!

Obligatory first post

2007-08-05T23:59:00.000-07:00

I keep putting off "the Blog", although I did start one on .Mac for awhile, but couldn't modify it to suit and didn't keep it going. I keep meaning to setup WordPress, or something like it, on one of the hosting services, but more important things keep coming up. so, here we are.
Flickr, Twitter, and now Blogspot. I'm getting sucked in. you're here too.
Welcome to my world, already in progress.