while we're talking about fairs

Soon after the Jefferson County Fair is the Kitsap County Fair. This one is much larger, and has a midway with all the carnie trappings. It's got the 4-H component for the kids, but it just seems a bit seamier. One thing to note - I never saw so many toy guns.

my favorite thing at this fair was the 'Walk on Water' ride: someone got creative and rented out bubbles you can walk around in!

a few more photos here: Kitsap Fair 2010 Pix

Jefferson County Fair

Now for something non-technical: my family and I visited the Jefferson County Fair again this year. This fair is small, and low-key, really oriented towards the local community rather than attempting to be some big event for tourists. I like it this way, and I encourage them to keep it going.

But even so, it's got a certain tension between the main threads of activity:First, there's the 4-H theme: cat shows, animal husbandry, model rocketry, bee-keeping, etc. These are right in line with the the adult-themed activities: arts, textiles, baking, canning, rock-collecting and of course the equestrian events that cross-over generations.

In keeping on the agrarian theme, it also has the horse-pulls: where draft horse teams compete to pull a loaded sled a fixed distance. This year's winning team pulled 7,500 lbs I think. Most of these teams are regulars: I recognize both the horses and their drivers going on several years now.

But then there's the motorist events: where else could you find a demolition derby where the competitors are high school students? What better way to blow off steam? This is gender-neutral, last year's winner was female and her younger sister was driving this year (she didn't win, although another girl placed third.) And this is followed the next day by the adult version - the "mud drags" wherein people race their vehicles through pits filled with mud? The unlucky ones get towed out, sometimes with a broken axle... Totally different crowd.

If you're interested in more pictures, the latest set is on my Flickr stream at 2010 Jefferson County Fair and you'll find prior year's sets in my Fairs and Festivals collection.

OSCON 2010 report

OSCON 2010 Summary

There were several overall themes to the sessions I attended at OSCON this year, summarized below. The conference had a good feel to it, although I found myself routinely conflicted over which session to attend (and didn't always make the right choice.) Too many interesting topics is better than too few though.

Scala Summit and sessions

(a recap of my earlier post on this)

There was a day spent talking about the Scala language, environment and experience using it, there were also a few sessions that covered it. Overall they were quite interesting and renewed my interest in exploring Scala again. (But with caveats based upon my earlier experience that I've heard echoed repeatedly: Scala has complex subtleties, perhaps akin to the metaprogramming models expressed in C++! It is interesting, and appears to be a good fit for some projects, but its not simple.)

For me the big takeaways were:

• use the Akka concurrency framework to address standard library problems with actors. Supports event-based and thread-based models, very lightweight and fast. (http://akkasource.org)
• use the Simple Build Tool (sbt), it's the best way to build and manage Scala projects. It uses Ivy for dependency management, builds Maven-compatible hierarchy. Much easier than integrating with existing Eclipse and Maven projects.
• Android: fairly complicated toolchain, but Scala is usable for Android development;
• Don't bring Ruby on Rails conventions and patterns into Scala, it's just not the same;
• Expect problems hiring (and training the reluctant) developers;

Big Data and the Processing of it (aka NoSQL - Not Only SQL)

• Good talk about patterns of database scalability (both RDBMS and NoSQL). One comment that stuck with me was that sharding is not always a good option - sometimes its better to divide operations and duplicate the data;
• MongoDB was the subject of much discussion (and repeated Twitter jokes) - mainly because its implemented using virtual memory and people don't trust it, so that redundancy is required. Its an object store, with none of the other overhead of an RDBMS. No Joins and no complex transactions == horizontal scalability. SourceForge is using it now.
• Pig seems to hold a lot of promise: its a high-level data flow language that compiles down into map-reduce jobs. (aside: when the speaker showed his slide of all the Java code a few lines of Pig replaced - a baby in the audience started crying. Twitter erupted.) Estimate that it requires 5% of the code required by Java, similar reduction in developer time, and its within 25% of the execution time of a hand-coded map-reduce job now.
• Scribe, the logging system used internally at FaceBook, handles 130TB per day. Talked a little about the architecture variations and how popular it was internally;
• Mahout - an overview of the open-source machine learning library, talked a bit about scaling it on Hadoop and some of the different algorithms it currently supports.

Client-side Development

• Phonegap promises a cross-platform mobile development kit. Its for writing web applications that have access to the native device, getting there using a native wrapper that hosts a web browser that your web application runs in. The web application uses phonegap.js to access the native environment. Still UI issues between platforms, debugging is a pain. Speaker also favored qUnit, XUI instead of jQuery, and Lawnchair for client-side persistence.
• Android talks were sprinkled throughout, but the big event was an extra workshop that evening that required a separate signup. The 300+ developers that attended were rewarded with in-depth tutorials on Android UI design, asynchronous programming technique, and a Nexus One device to test it all out on.
• JavaScript, many sessions and a recurring theme throughout the conference. At one point, I heard the phrase "JavaScript is the language of the web, deal with it!"

Server-side Topics

• Tomcat security covered many of the vulnerabilities in TC6, which were fixed in TC7. XSS attacks are growing, and one has to keep adding filters for them. TC7 allows regular expression filters to help with this.
• Django was covered in a few sessions, I learned a little more about its deployment, which is pretty typical when you try to scale it up.
• Spring 3 Framework: I missed some of this talk, but had a chance to talk with the presenter afterwards, and review the slides. The most interesting parts were their growing support for Dojo, continued support for Flex, and their new Roo tool - allowing one to quickly create and scaffold a Spring-compatible server, much as does Rails. No more copy and prune! At least for development sites.
• Chef is an intriguing system for deploying applications and systems software to various systems in a large network. Looks a lot nicer than the one commercial offering I've had experience with. This might be a good way to spin up development servers quickly as well. Supports a variety of systems.

Miscellaneous Topics

• Mirah language was introduced by Charles Nutter. Basically its a Ruby-like language with a twist: it compiles directly into Java bytecode or source, and does not require a runtime library. Basically its (mostly) Ruby with static typing.
• Concurrency topics continued, with a presentation by Tim Bray capturing much of the essence. Basically its the functional vs procedural programming, event-based vs thread-based arguments. Functional and event-based is gaining interest, and racking up impressive results. Node.js in JavaScript and Event Machine in Ruby are two popular examples. Tim's presentation was very judgemental, although he favored an Erlang model at least (message-passing actors.) Perhaps Go will help there. (I am not yet convinced that event-loops are fundamentally better than thread-based models on modern multicore systems.)
• Go language, presented by Rob Pike, both in sessions and keynotes, garnered new interest from me. (Actually, I've pretty much ignored it, thinking the last thing the world needed was another C-based language!) Its simple like C, but has promising concurrency constructs.
• Testing was covered in a couple of sessions. One tool I need to look at is Cucumber. A speaker commented that they though Mock's were useless because they decoupled reality from your tests, then fail and are ignored. Also the need to keep tests fast, otherwise people push them off to the CI servers and they just ignore those messages. Also, think "outside in": if you are too close to the model or implementation you're not really testing functionality.

There were separate tracks on Health Care computing, Cloud and Emerging Languages. Most of these sessions were being recorded, so I skipped them when there was an interesting conflict. I hope to catch up on some of them later.

The keynotes: make sure you watch the videos by Simon Wardley (talking about innovation, process, commodities and and services), and Rob Pike (talking about the reasons why we need a new programming language so as Go.)

using sbt to setup a project

Following up the earlier Scala post, now that I've actually had a chance to try sbt and talk to others about using it.
Following the instruction here: sbt-setup
you don't even need to install Scala - it evaluates dependencies and installs it for you. That's nice and just what you'd expect it to be capable of!

That said, I'm not yet convinced it will work well with existing Maven repositories, as it has its own lib_managed directory too.
Because it's Ivy-based, it also does not work directly with Maven plug-ins.
But you can start with a POM and it will build and run it.
The Scala interpreter allows direct access to the Spring context.
I'm told that if using Scala 2.8.0 you can work correctly with the nested annotations in a Hibernate-JPA project, but I haven't seen this actually happen yet.

In short, I think its probably still not a seamless integration into a legacy project, but it does appear that it could be used to integrate with and provide a new way to build a project with existing code. Further experimentation remains.

Scala Summit @ OSCON 2010

I attended the Scala Summit at OSCON this year. I have worked with Scala off and on in the past year, but have never really used it on any substantial project.
There were several presentations throughout the day, here is a quick rundown on what I saw:

Akka framework, providing support for concurrency, fault-tolerance, and STM. To me this was the most interesting presentation, in part because I think Scala provides great promise for building concurrent applications more safely. This framework borrows concurrency and fault-tolerance ideas from Erlang, using message-passing between lightweight event-based actors, allowing for thousands of such concurrent actors. It claims to be much more performant than the default Scala concurrency libraries.
more here: akka

sbt - the Simple Build Tool, is clearly the tool of choice to manage the Scala/Java build process. It incorporates dependency management using Ivy, allows for build programs to be written in Scala. It looks useful as a general replacement for other build tools as well. simple-build-tool/

Specs - a domain-specific language for creating specifications and validating code against it.
see more here: specs

Lift framework: another presentation on Lift that, I think, suffered from the presenter doing live code instead of working from slides. There was some interesting discussion of the benefits of static-typing in such a web framework (a bit less testing required for example.) The other thing I came away from this with was how powerful the Comet library is, looking forward to diving into this further.

Android/Scala integration - some discussion on toolchain, problems with larger libraries, and the role of type-safety.

Scala in Practice: developers from Twitter and ??? ran a Q&A session. A couple of things that echoed my experience was the difficulty of integrating new Scala development into existing Eclipse & Maven toolchains. Also, remember you're not writing Ruby on Rails, or Java - don't just bring the same idioms. These folks also gave up on the standard concurrency libraries due to performance, rolling their own (Akka wasn't yet available, but they said they would use it now.)
The other thing that really rang with me was that, rather than trying to introduce Scala in a low-risk "test" project, it would be better to write something important. (At least it worked for them!)
There was also some discussion about lack of support for nested annotations and how this affects persistence frameworks - something I need to look into more.

Advanced Scala Language Features - last session of the day, some of which was over my head as a Scala newbie.
The two concepts I took away was the notion of "higher kinds" and continuations.
Kinds are a level above types (Kind systems classify Types, Type systems classify Values.) They allow you to construct and manipulate Types.
Continuations are a feature common in a few other languages, Scala 2.8 has a form that the speaker argued was not really that useful. Remains to be seen

I also had some offline discussions with people about integrating Java and Scala. All thought it was great, but also warned me that the Eclipse 3.5/Scala 2.7 integration was a bit brittle. So at least it wasn't just me having the problems!
All swore by the 'sbt' tool. I hope to get a chance to new Eclipse and Scala 2.8 soon and see if its better to integrate. i'm also looking forward to digging into the Akka framework.

Flex client/server performance

I had a chance recently to do some Flex-based client/server development work with a small team. During the course of this project questions arose about data transfer using various protocols. Originally the team decided to use a SOAP interface, and this started out fine, but during development someone accidentally requested a deeply-nested graph of objects - and we hit a wall where data retrieval started taking many seconds.

Time for some performance testing, comparing three different protocols:
- SOAP marshaled into ActionScript class-based objects
- RESTful/JSON marshaled into dynamic objects (using as3corlib for JSON decoding)
- RESTful/XML marshaled into the ObjectProxy objects that Flex 3 creates by default

Basic results - the above list shows the performance ranking, SOAP being 3X to 7X slower than the RESTful/XML case.
JSON decoding added approx 20% overhead - presumably because it all happens in AS3 code rather than in native code.

The other thing that stood out - increasing the object graph depth caused an exponential increase in processing time, even though there was only a linear increase in the number of objects in the graph.

Another advantage of the RESTful/XML case is that the ObjectProxy objects created are directly bindable by the Flex UI controls, so if you can live with 'duck-typing' no further marshaling is required.

Later, on a different test setup, the speed of AMF3 was tested, and it was a little faster, with the added advantage (to some) that it was marshaling into class-based objects. (It also handles object graphs better because it uses server-side instance folding). But its a binary format with a limited user base.

Conclusions for now - another strike against SOAP in this environment, RESTful/XML ends up being a pretty compromise in performance and usefulness in the UI.