Nerd time issue 16

I use to work at a research lab and while it was cutting edge in some ways, it seemed unaware of things going on in software and web.  I've since left the research lab, and Nerd Time started as a mailing list to tell my friends still at the lab about things that they might be able to use in their own projects. 

————

Hey all,

It's that time again.  I never get feedback about whether you guys read this or not.  But apparently, some of you do, since my last trip back to MD.  This is just a collection of things I found interesting since the last nerd time.  Obviously, there are other trends I'm missing.

If you want off, just lemme know. 

This is rather long since it's been a good 6 months since the last nerd time.  Work has kept me busy, and I don't read as much as I use to.  Since there's no theme, but lots of trends, they're in no particular order this time.  Skim through it and see if there's anything that catches your eye. 

If you have questions about stuff, feel free to ask me (don't reply all!)

First, some stuff I did on the side:
Senate Majroity vs National Debt
I was talking with Ian about graphing public data, and this was what he wanted to know.  This sort of thing should be so much easier.  If you find the process of getting this data to graph, lemme know.  I imagine it goes in line with a lot of the net-centric buzzwording that does on in DoD projects.

http://iamwil.posterous.com/senate-majority-vs-national-debt-getting-at-p

Frock, a chicken flocking simulator

I wanted to get to know the Lua programming language, so I chose this as a project.  I'm getting it to support more chickens still.
http://webjazz.blogspot.com/2009/02/introducing-frock-flocking-chicken.html

And now, the other stuff.

Twitter
So I'm sure most of you by now have heard of twitter.  Considering that Oprah did a show on it, it's crossed over to mainstream.  A lot of you might not think of it as anything to pay attention to.  However, it's one of those things where its value depends on whom you follow.  Beyond the hype, it's mainly a messaging multicast system that has a dead-simple API, so that other people can build things on top of it.  People have made things that twitter, such as plants that tell you when you need to water it, when bridges go up and down, when a meteor almost hits the earth, etc. 

http://www.twitter.com
http://twollars.com/
http://www.botanicalls.com/kits/

http://twitter.com/riverthames

Wolfram Alpha and Google Squared and YQL
Wolfram Alpha and Google Squared had both been announced in the last
month or so.  Both are looking towards being able to query large
amounts of structured data.  However, wolfram curates this data with
experts, and google squared attempts to make structured data from
indexing tables of data on the web.  In addition, Yahoo released YQL,
which is a query you can use to scrap the web and treat it as just
another database.
http://www.wolframalpha.com
http://www.google.com/squared/

http://tables.googlelabs.com/Home?pli=1

http://developer.yahoo.com/yql/

Real-time search
Real-time search seems to be the wrestling ground for the next
generation of search right now.  There are a number of competitors in
this field, including giants and startups.  It's evident with the death
of Michael Jackson that news doesn't just travel through the old
channels anymore.
http://venturebeat.com/2009/06/20/who-rules-real-time-search-a-look-at-9-contenders/

http://search.twitter.com
http://www.scoopler.com

Google Wave
If you haven't heard, google released a new communication tool called google wave.  It's what email would be like if it was reinvented today.  It's basically combining different aspects of our communication tools and merging them all together.  It's best if you watch the video and play with the demo.  If you want to play with wave yourself, you don't have to wait for an invite, but can sign up with a wave server that someone set up themselves.  I recommend watching the video, as it breaks your presumptions of what's possible with HTML5 and the web.

http://wave.google.com/
http://wavety.com/pygo-wave-server/

Git and github
To me, this is really old news, but just in case you're still using SVN, you should checkout Git instead for your source control.  It's ass-kicking good, though it has a slight learning curve.  I won't say too much more about it, but you should really look into it.  It'll expand your mind.

http://git-scm.com/
http://github.com/

Key-value stores
Lately, there's been a flurry of attention on key-value stores.  I've mentioned one of them before, CouchDb.  There are a bunch of others.  Tokyo Cabinet (link #2) is used at Mixx.jp.co, a social network in Japan.  Cassandra (link #3) is used at facebook.  Amazon has SimpleDB and Dynamo.  I've only played with tokyo cabinet and couchdb, so I can't really do a compare and contrast between them all.  But to me, TC, couchdb, and redis seem to be the most interesting.  This marks a shift away from relational dbs as the default data store.  Not that they'll replace relational db, but we're finding there are a different class of constraints for the web not necessarily taken care of by relational dbs.  In addition, they have properties not avail to relational dbs, such as being schema-less, an http server built in, replication, distributed, etc.

http://couchdb.apache.org/
http://tokyocabinet.sourceforge.net/index.html
http://incubator.apache.org/cassandra/

http://code.google.com/p/redis/
http://memcachedb.org/
http://github.com/cliffmoon/dynomite/tree/master

http://opensource.plurk.com/LightCloud/

http://www.allthingsdistributed.com/2007/10/amazons_dynamo.html
http://www.oracle.com/technology/products/berkeley-db/index.html

http://code.google.com/p/thrudb/
http://code.google.com/p/scalaris/

The internet of things
It's something further out, but these first two talks from TED got me thinking about where the web was heading.  I don't think that the semantic web, as we imagine it will come to fruition.  However, having the things we own talk to each other over the internet is not unfathomable.  They'll be able to negotiate with each other to perform a task, or they'll be able to keep a history of what they're doing or how you're interacting with them. 

http://www.ted.com/index.php/talks/kevin_kelly_on_the_next_5_000_days_of_the_web.html
http://www.ted.com/index.php/talks/tim_berners_lee_on_the_next_web.html

http://siftables.com/
http://www.readwriteweb.com/archives/5_companies_building_the_internet_of_things.php

www.makerbot.com
http://siftables.com/

Cheap hardware boards
Hardware is already cheap, but building hardware yourself has still been somewhat of a pain.  I remember having to use Rabbit boards before.  There are better ones now.  I've mentioned arduino before.  Beagle board is a full board that you can run Ubuntu on.  Teensy is a small USB microcontroller.

http://www.arduino.cc/
http://beagleboard.org/
http://www.pjrc.com/teensy/?not_a_duplicate

Quake online
Gaming often is looked on as child's play, when in fact, it's some of the hardest programming around, and often drives innovation and progress in graphical techniques, AI, and hardware.  Carmack, the guy that wrote Quake, wants to put Quake on the browser.  For the longest time, people derided the web, saying it'll never match the performance of desktop apps, and never give the same user experience.  If Carmack can run quake on a native browser, then I believe desktop will lose.  If he's delivering quake as a video stream, then that's another matter altogether.

http://www.gamasutra.com/view/feature/3946/building_quake_live_carmack_speaks.php?print=1

Reverse HTTP
HTTP is by design a pull model, where the client requests resources from a server.  If you wanted to push data to a browser client, you had to rely on a bit of javascript finangling called Comet (cousin to AJAX), where you open an http connection to the client, and leave it open until you want to push stuff to the client.  This certainly puts a load on servers because you have to keep connections open.  Alternatively, you can have the client keep polling the server.  That sucks too.  Reverse HTTP doesn't need to keep the connection open.  It basically takes advantage of the upgrade field in the HTTP header normally used to find a more appropriate protocol, and instead to turn the connection around from the server to the client.  It's still experimental, but it makes a lot more things simple instead of messing with javascript on clients to push data to browsers.

http://www.reversehttp.net/
http://wiki.secondlife.com/wiki/Reverse_HTTP
http://www.ietf.org/internet-drafts/draft-lentczner-rhttp-00.txt

Whiteboarding in real time
Many collaboration tools have come out.  We've discovered that the web is essentially a communications medium.  Anyway, this set of collaboration tools lets you whiteboard, compose text, and revise docs in real time as other people are editing them.  The last link shows you a re-play of paul graham writing one of his essays.  This allows people to see how they edit their text over time, and shows others how other people think as they write.  It'd be useful as an educational tool.

http://www.twiddla.com/

http://www.etherpad.com/
http://www.wikirage.com/
http://www.revizr.com/

http://etherpad.com/ep/pad/slider/13sentences

Facebook's walled garden

Facebook is the AOL of today.  It's basically a walled garden of data, where users live.  There's a bunch of effort to break them open.  Facebook also wants to open itself out as a fast follower to twitter.  I won't say much more here, but there's an ongoing battle about where data gets to go on this front.

http://www.wired.com/techbiz/it/magazine/17-07/ff_facebookwall?currentPage=all

http://googledataapis.blogspot.com/2008/03/3-2-1-contact-api-has-landed.html
http://www.complang.org/dsnp/

Data.gov and the sunlight foundation

Since Obama took office, there's been a big push and initiative to open up the government to its citizens in the name of transparency.  One of the things they're doing is data.gov and opening raw public data up to developers or anyone that wants to use it.  The sunlight foundation is doing the same for legistator and voting data.  I expect that we'll have more apps that will be able to take advantage of this data in the near future, not just to help the people govern their govenment, but also to lead more informed lives.

http://services.sunlightlabs.com/api/
http://www.data.gov/catalog/raw/category/0/agency/0/filter//type//sort//page/1/count/10

DNA engineering
A front that I don't know too much about, but is probably a bigger revolution than the information age and the internet are things that have to do with genetic and bio engineering.  23andme lets you submit cell samples of yourself, and they'll do genetic testing to tell you if you have genetic diseases, among other things.  You can now submit gene sequences and get them built for a modest amount of money–not super expensive, but still out of reach for hobbists)  As the cost goes down, you'll soon see designer pets and bacteria.  The last post is about a guy that theoretically hacks a more potent variant of swine flu.

http://23andme.com
http://mrgene.com/desktopdefault.aspx/tabid-2/

http://www.bunniestudios.com/blog/?p=353

GWT, sproutcore, and Cappucino

Javascript is the most widely used language in the world.  And while it has its merits as a functional language, people are trying to develop frameworks that compile to javascript.  Javascript is not bad when you're using jQuery.  Scriptaculous 2 just got released also. 

http://code.google.com/webtoolkit/
http://www.sproutcore.com/
http://cappuccino.org/

http://jquery.com
http://scripty2.com/

Chat on couchdb, standalone web apps, and taking your data with you
This was curious.  Couchdb is a key-value document orientated database with an http server as its frontend.  They were able to demonstrate that you can fit an entire web app just in the database.  Data is code, and code is data.  Not only that, you can use the database's replication to port your data and sync it where-ever you go.  It's an interesting head turn, even if it's just a demo.

http://damienkatz.net/2009/05/realtime_chat_on_couchdb.html

http://www.jasondavies.com/blog/2009/05/08/couchdb-on-wheels/
http://devlindaley.com/2009/2/14/couchdb-bloom-filters

Mozilla Ubiquity again, but this time hooks into webapps
I've mentioned ubiquity before, which is like a commandline interface for your browser.  I use it myself, but only in limited amounts.  What's interesting about the direction is that they leverage web services to complete tasks it can't complete for itself.  I think that high level languages will eventually adopt the idea of being able to easily hook into web services as a natural part of the language, without extra libraries.

http://www.readwriteweb.com/archives/firefoxs_ubiqity_starts_thinking_for_itself.php

Mozilla Jetpack lets you write Firefox addons with the web
Traditionally, web developers have stayed out of the realm of desktop developers.  This is one of the many indications I have that a lot of programming–especially those with user interfaces or a social aspect–will move towards web programming constructs.  https://jetpack.mozillalabs.com/


Clojure, Scala, Haskell, Erlang

I'm not going to say too much about these programming languages, since I've mentioned them before, but just as a reminder, there's more than Java out there.  These four to me, represent the edge of programming languages that have potential in the future.  With the advent of multicores, it's likely that functional programming will lead the way in giving us adequate programming constructs to deal with multicores.  If you're a programmer, it'd probably serve you well to learn at least one of these in the coming 4 years.

http://clojure.org/
http://www.scala-lang.org/
http://www.haskell.org/
http://erlang.org/

ParrotVM and mod_parrot
Admittedly, I don't know much about Parrot.  But the claims it makes is big.  With the rise and popularity of dynamic programming languages like Python and Ruby, we're struggling for a fast virtual machine.  And to have to build a new virtual machine every time we have a new language is a pain.  ParrotVM is suppose to take care of easing that pain.  If that's the case, it might be easier to make languages catered to our problem domain.

http://www.linux-mag.com/cache/7373/1.html
http://www.smashing.org/talks/mod_parrot.ppt

Augmented reality and zombies
We've moved closer to having augmented reality.  This is a far cry from the geeky headcam helmets and laptop backpacks that dorky MIT profs wore a decade ago.  It still relies on a 2D barcode, and has limited uses, but now with the iPhone3GS out (it has a compass), we might see more augmented reality apps (as well as on android phones)

http://singularityhub.com/2009/06/19/augmented-reality-is-full-of-zombies/
http://www.thinkartificial.org/machine-interfaces/augmented-reality-iphone/

http://news.cnet.com/8301-17938_105-10266380-1.html

Sysadmin tips
Here are some good sysadmin tips.  I'd like to think I know my way around linux, when in fact, I've just started.

http://www.ibm.com/developerworks/linux/library/l-10sysadtips/

http://blog.transmit.net/2008/10/my-best-unix-tricks.html

Probabilistic chips
I don't know anymore than what's written in the article.  So read about it.
http://www.chron.com/disp/story.mpl/metropolitan/6252697.html

Google Moderator
Voting on websites is old hat since about 2005 with the advent of
Digg.com and reddit.com.  I found it curious that google has a
moderator app, to help facilitate the asking of questions.  If you want
your own, you can create a white label voting site at slinkset.
http://moderator.appspot.com/
http://www.slinkset.com

The rest of these are related to software, but not about code.  If you can only watch/read one, I'd recommend the poisonous people one.  That applies to more than open source projects.  In it, SVN core devs talk about how someone came in and told them they were all wrong.  I have a feeling that was Linus Torvalds, as he rails on the SVN guys in his talk.

The Business and Politics of Software
http://blog.cusec.net/2009/01/05/zed-shaw-the-acl-is-dead-cusec-2008/

Pivoting, or knowing when to stop.
http://startuplessonslearned.blogspot.com/2009/06/pivot-dont-jump-to-new-vision.html

How open source projects survive poisonous people.

Linus Torvalds on Git

Build or buy?
http://sitecanary.com/blog/5

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s