Saturday, June 28, 2008

Itching my programming nerve


Photo: oreillygmt

I've just got back from the first ever commercial Erlang conference. Some 40 talks in two days all related in some way or other to Erlang. It was a chance to meet old friends, make new friends and connect people together in the hope that new synergy effects would arise.

The most exciting thing was the emergence of what I think might be the first killer applications written in Erlang. I might be wrong, but my gut feeling is that what Alexander Reinefeld showed us will be the first killer application in Erlang.

Only a few language nerds are interested in programming languages in their own right. Most people are more interested in what you can do with a programming language than with the underlying language. Thus is was with Ruby. Ruby on rails was the application that drew developers into Ruby. It made them want to learn Ruby so that they could easily build web applications.

Alexander Reinefeld told us about an Erlang implementation of the Wikipedia that not only has a stunningly beautiful architecture, but which outperforms the existing Wikipedia.

I'll talk about the Wikipedia implementation later in this posting, but it was not the only notable talk there were many other great talks.

Claes Vikström (Klacke) gave a great lecture which was a mixture of battle stories, history and what he was doing today.

Klacke is the master of the one-line throw-away remark - "and then I implemented a DBMS ... and a web server" this was the technical stuff, and on the business side
...then we started a company and made a whole lot of money...
At the end of his lecture Klacke said something like:
... so how come we have this great technology and people are just doing boring things and not writing stock exchanges ... there aren't any killer applications ...
Just as an aside Joel Reymont did stir up the Erlang mailing list with his announcement that he wanted to write an open source stock exchange as a publicity stunt, but this ran out in the sand. Perhaps later we can resurrect this idea, it would be a bit of fun.

The Wiki

Now for the fun stuff. Alexander Reinefeld video answered Klackes call for action and for a non-boring application as he described how he had implemented the Wikipedia on a new p2p system now called Scalaris.

Here's my version of his story:
  1. They make a peer to peer system based on the chord algorithm
  2. They added a replication later using the paxos algorithm
  3. They added a transaction layer
  4. The injected the wikipedia
  5. It went faster that the existing wikipedia
Applied to Wikipedia, Scalaris serves 2,500 transactions per second with just 16 CPUs, which is better than the public Wikipedia.
Alexander is a tall bespectacled academic who usually only turns up at academic conferences. He was very worried at the start of the talk when he was introduced as "Professor Reinefeld" I think he thought it would frighten people.

The system they described won an IEEE prize for scalable systems and was also presented at the Google conference on scalability. I asked Alexander why publicity about what they had done was so hard to find.
"I'm academic, we usually publish papers," he said.
He'd also said he'd started a company that "wasn't doing very well" (tip to VCs - check this one out and give the guy some help).

So my take on this is that this is one of the sexiest applications I've seen in many a year. I've been waiting for this to happen for a long while. The work is backed by quadzillion Ph.D's and is really good believe me.

On second thoughts don't believe me but check out the video lecture. You can also download the code.

CouchDb

When Alexander had blown my mind Jan Lehnardt popped up for the next section and blew it even further by presenting CouchDB - am I going mad are we seeing the emergence of two killer applications? This cannot be.

Jan Lehnardt has a presentation technique that is a joy to watch - it reminded me of why I love programming.

Jan communicates on two level simultaneously. His body language oozes enthusiasm - he waves his arms so fast and hops up and down so we think that he was either a helicopter in a previous life or that he wants to get a job as a windmill. Words tumble out of his mouth so fast that his tongue often trips over the end of his sentences and falls flat into the middle of his next sentence conveniently missing out the middle of the last sentence.

Check out the video and you'll see that I mean - You'll see Jan almost taking off as he impersonates a helicopter. For more information see the slides of the talk. You can download CouchDb from the Apache incubator site.

This style of lecturing is amazing. Jan communicates simultaneously on two entirely different level. His enthusiasm is received by the amygdala in the limbic system and his slides go via your eyes to the pre-frontal cortex for analytic processing.

So what was this great stuff the Jan was so enthusiastic about?

CouchDb - is an Erlang application that turns a Key-Value JSON store into a system with a RESTFUL interface that stores arbitrary data structures data in a way that fits nicely in with the Erlang system. When I got home I downloaded CouchDb and took a look, and there is was, nicely packaged with a Mochiweb server, the same server which is used by facebook for their comet web chat system.

Like most good ideas the CouchDB is deceptively simple. Once you've seen it you think - yes that's how it should be, how simple, how beautiful. But designing simple things is not easy it requires many false starts and takes a long time to get right. Hats off to Damien Katz for the initial design and to his collaborators, and thanks Jan for telling us about it.

Synergy...

Now it just so happens that both Jan Lehnardt and Alexander Reinefeld both live in Berlin, both are working with key-value stores (the details, vary) which are programmed in Erlang, both of them are working on what might be the next killer Erlang application and ... they have never met.

I introduced them and then stood back. Wow.

After a moment of shyness they changed to speaking German and Jan started bouncing up and down and speaking even faster than in English - this was getting dangerous - this time Jan did turn into a helicopter and narrowly missed causing an accident as he flew out of the room.

Going home

Klacke and I took the same flight back to Sweden the next day.

"That itched my programming nerve," said Klacke

"Precisely ..."








Tuesday, June 24, 2008

Invasion of Privacy

On 18 June the Swedish Parliament passed a law giving sweeping new powers to the FRA (Swedish Defense Radio Establishment) allowing them to wiretap people in Sweden through phone conversations, email, text messages and more.

All people in Sweden using electronic communication can have their communication monitored despite the fact that they are not suspected of committing any crime.

In my view this is in direct contravention of article 12 of the UN declaration on human rights to which Sweden is a signatory.
Article 12.

No one shall be subjected to arbitrary interference with his privacy, family, home or correspondence, nor to attacks upon his honour and reputation. Everyone has the right to the protection of the law against such interference or attacks.
This shameful law applies to me - if you send me email (I get a lot of emails from readers of this blog and of my books) then you should be aware of the fact that your mails are not private but will be read by FRA employees - so your rights will be violated.

The Swedish government is sensitive to foreign opinion so if you wish to protest about the fact that your privacy would be violated if you mail me then I suggest you send mail to one of the leaders of the four political parties (Maud Olofsson, Fredrik Reinfeldt, Jan Björklund, Goran Hagglund) that voted this law through. They can be contacted through The Prime Minister and Ministers.

I urge you to make your opinions known.

My daughter asked me:

Does this mean they will read my MSN chats?
I said "yes".
She said "that sucks"


I get pretty pissed off when they confiscate my water bottle when I have to travel by air, but I guess I can live with this but reading my email, and the email of my wife and kids is totally unacceptable. I would never read my wife's or kids' email - such behavior is totally unacceptable in a civilized society.

All this is done in the name of saving us from terrorism - after 9/11 western politicians promised that they would not allow acts of terrorism to change our way of life. Well, spying on 9 million people who have not broken the law is a funny way of "not changing our way of life."

This really pisses me off.

/Joe Armstrong

[In other areas Sweden is a pretty decent place to live, with decent human values, but this new legislation is totally unacceptable]

[Note also - all mail to the erlang mailing list will be monitored by FRA - if this upsets you then please mail the people responsible (see above). If this worries enough people we can move the list to a country that respects human rights]