Crazy small nuances

I’m not particularly fond of nuances. Ruby is suppose to be easy to use and intuitive to the user. But as it grows, it has some crazy little nuances.

For example, there’s a difference between system and exec. I wouldn’t have known, as I was skimming the docs for something I needed. Were it not for this post by jayfields, I would have had no idea.

And there’s something in Ruby 1.9 called “funcall”, in which it’s definitely not at all intuitive how that’s different from “send”. I hope these are just growing pains, because while Ruby is nice when you’re taking it out for dinner on the first couple of dates, I hope it doesn’t get abusive the more time that you spend with it.

Users know the problem. You know the solution

What’s considered by now to be mantra out there in ‘user-centric web development land’, is “listen to your users“. But in all that hubbub, you’ll also hear, “don’t listen to your users all that much.”

So what’s up with the paradox? Well, like all puzzles, it’s only a puzzle if you think it’s all or nothing. It seems pretty obvious once you think about it, but it bears reiterating:

Listen to your users when it comes to what’s wrong with your stuff. Users are more familiar with problems with your application than you are. They are pained by how much your application sucks, and how it doesn’t help them get on with their lives. So they’ll complain to you, in hopes that you’re listening and you’ll fix it.

But users often times have no idea what solutions are good. They’ll often offer up solutions to their problems with your application when they’re telling you what’s wrong with it. Often times, they won’t even tell you the problem, they’ll just offer solutions.

“You should put tagging in here.”

“Why not put a chat room so we users can talk to each other?”

However, they usually don’t have the overall vision, sufficient scope, and adequate background for improving the product. That’s your job. You have to get beyond what users are saying to figure out exactly what the problem is, and find a solution that fits the overall vision of the product, and perhaps solves other problems all at once also.

A default behavior for failure or nil

I read “The Rails Way“, mainly because Jamis Buck writes there. Three days ago, they had actually posted one thing that I had address just a few days prior–that is, how to prevent users from looking at other users’ data.

I have to admit, I was rather tickled by my solution. It’s nice when you figure things out. But alas, after reading Koz’s post on association proxies, I have to admit, I like his solution better. At least I made it to anti-pattern #3.

So what did he do? He simply used the find in the association, and let things throw an exception otherwise.

def show
@todo_list = current_user.todo_lists.find(params[:id])
rescue ActiveRecord::RecordNotFound => e
flash[:warning] = "Stop playing around with your urls"
redirect_to '/'

It also takes care of the case where you have say, many items that belong to a todo_list. You can load it by using :condition in the association find. The only reason I can think of not to use it is if it happens to be a slow solution. But no use optimizing if you have no measurements and hard numbers.

Koz’s solution let the default behavior of the method take care of things. This is a way of thinking that I need to start using more of.

I was use to nil being a failure state, something that you checked, and if it happened, everything’s gotta stop–like the examples I often saw in C:

status = CallSomeMethod(with, some, parameters);
if status == NULL {
// or throw some exception here if you're using C++ or Java

Often times, these things got cumbersome, because NULL (or even an empty array or hash) was not considered to be a valid input for many functions, and would stop computation by returning error codes and throwing exceptions in C++.

One of the things that I found nice about the code from Ruby gurus was that it was conventional to do (and subsequently, I saw in Perl):

setting = params[:setting] || "default"

Instead of:

setting = params[:setting].nil? ? "default" : params[:setting]


setting ||= "default"

Instead of:

setting = setting.nil? ? "default" : setting

It was because the operator || took nil by default, and had an appropriate behavior for dealing with nil, and that has made all the difference in being able to chain functions together. In addition, having default failure behaviors require you not have to write error checking code all over the place either.

I’m not sure what this is called (if anyone knows, enlighten me), but a completeness and liberal in what you accept for your input and strict in what you output also applies here to methods and functions. I think it makes code more readable, because it is not peppered with common sense error checking code.

Capistrano and Mongrel are easy to use, but deployment is still hard

So I finally got around to deploying with Capistrano. It was like learning how to walk all over again. Before, I had learned to deploy using just lightty behind apache, and that was a pain every time, since there would be steps I’d be forgetting. A good introduction to deploying with capistrano is “It’s time for a grown up server.” I also checked out Mongrel at the same time. It seems pretty neat.

I’m sure that Cappy is easy to use once it’s up and running, but boy, setting it up on a server from scratch has taken all day. This Yariv’s guy has a point when he says there’s so much stuff to install, compared to web development in Erlang–all you need is Yaws and Mnesia. But then again, all this ‘stuff’ is suppose to make our lives easier.

There are plenty of tutorials out there about setting up Capistrano and Mongrel, but deviate from them just a little, you’d better be ready to use your problem solving noggin.

SVN is just another TLA

One of the biggest problems I’ve run into while setting things up is that if you have a password on your svn repository, Cappy’s not going to like it too much. This is an old problem, and the old solution from Jamis exists. However, I noticed that when I used his solution, the options –username and –password was there twice. I’m guessing that you can do the following in your capistrano deploy.rb, and it would work…BUT!:

set :svn_username, ENV['USER'] || "default_username"
set :svn_password, { Capistrano::CLI.password_prompt('SVN Password: ') }
set :repository, "{application}/trunk"

This only works if the machine you’re developing on is the same as the machine that you’re deploying on, per the example in Agile Web Development with Rails (and already the example is outdated). When I did the cold deploy, I got this:

$ cap cold_deploy
* executing task cold_deploy
SVN Password:
...some other stuff clipped...
** [out :: remote_server] Authentication realm:
** [out :: remote_server] Password for 'someapp':
** [out :: remote_server] subversion is asking for a password
** [out :: remote_server] Authentication realm:
** [out :: remote_server] Username:

It seems like Capistrano checks the SVN twice. Once to find the latest version number when it is on the local development machine, and once again to checkout the latest version from the repository on the remote server machine. Since I had no way of controlling what goes on in the script while being executed on the remote machine, the prompts for a username and password for svn just got logged, and Cappy just waits on the local machine.

The only solution I came up with was just to go back to the Old Ways, where you ssh into the remote server machine and do a temp checkout by hand first. And you do the same on your local development machine.

$ svn co
$ rm -rf trunk

You can then just leave the deploy.rb script as:

set :repository, "{application}/trunk"

This way, svn caches your authentication on the remote machine (and local machine), and probably won’t ask for it again until maybe you reboot.

To get the pack of Mongrels running, you’d better get things right

Things were going hunky-dory, and I thought I was in the clear, but then there was a slight detour. I ran into this problem when I tried to start mongrel on the remote server.

$ mongrel_rails start -d
/usr/lib/ruby/gems/1.8/gems/mongrel-1.0.1/lib/mongrel/rails.rb:32: uninitialized constant Mongrel::HttpHandler (NameError)
from /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:27:in `require'
from /usr/lib/ruby/gems/1.8/gems/mongrel-1.0.1/bin/mongrel_rails:10
from /usr/bin/mongrel_rails:18

This one took a little bit of time to figure out, but it was easier than the next two. Gems said that mongrel was successful at the end, when in fact, it didn’t build at all! This was because I didn’t have make on my machine. To get it on Ubuntu, apt-get the build-essential package, and then reinstall the mongrel and mongrel_cluster gems. Odd thing is, I needed make to build rubygems on Ubuntu. Must have got removed when I did apt-get autoremove

However, the real kicker stumped me for a while here:

$ cap cold_deploy
** transaction: commit
* executing task spinner
* executing task start_mongrel_cluster
* executing "sudo mongrel_rails cluster::start -C /var/www/apps/my_app/current/config/mongrel_cluster.yml"
servers: ["remote_server"]
[remote_server] executing command
command finished
command "sudo mongrel_rails cluster::start -C /var/www/apps/my_app/current/config/mongrel_cluster.yml" failed on remote_server

Huh? What’s going on? I was stumped for most of the day. It’s always the little thing that get you, but you learn, and alas you learn. Ubuntu, by default, does not add any subsequent users to sudo. I had created another user on the remote server machine to do the deployment. Therefore, it had no sudo privileges. Run:

sudo usermod -G admin username

And after that, I ran into:

$ cap cold_deploy
servers: ["remote_server"]
[remote_server] executing command
** [out :: remote_server] Starting 2 Mongrel servers...
** [out :: remote_server] !!! Path to log file not valid: log/mongrel.log
** [out :: remote_server] mongrel::start reported an error. Use mongrel_rails mongrel::start -h to get help.
** [out :: remote_server] mongrel_rails start -d -e production -p 8000 -a -P log/ -c /var/www/apps/my_app
** [out :: remote_server] !!! Path to log file not valid: log/mongrel.log
** [out :: remote_server] mongrel::start reported an error. Use mongrel_rails mongrel::start -h to get help.
** [out :: aeolus] mongrel_rails start -d -e production -p 8001 -a -P log/ -c /var/www/apps/my_app
command finished

After reading through some google posts, I realized it was really simple. The configuration file for mongrel had the wrong path to it. It didn’t have the right path. in the mongrel setup, you need to make sure the deploy path as “current” on the end. Remember to add /current/ to wherever you’re deploying your application!

mongrel_rails cluster::configure -e production -p 8000 -a -N 2 -c /deploy/path/my_app/current

So hopefully, if any of you out there has run into the same problems, I’ve saved you a little bit of time. I can’t really say this is a tip, more like a log of my adventures deploying. Remember, if just doing it once will save you lots of time in the future, it’s probably worth it to learn how to do it.

Does link_to_remote() submit using a GET or a POST?

A little while back, I had a post about how to have two submit buttons with form_remote_tag. A question was raised about having a POST as opposed to a GET request. It seemed like a short enough question, but after looking into it for about 15 minutes, I figured it was worth it to type something up.

To recap, when I first was doing web stuff in high school, I never got what the difference between POST and GET were. It seemed like the author of my HTML book then didn’t really get it either. However, it wasn’t until DHH started raving about REST as a web service did I get it. in short, when a user submits a form, GET puts the form contents in the url, and POST puts the form contents in the body of the HTTP request. The implication (apparently related to REST) is that you’d only use GET for server requests that didn’t change content on the server–the request had no side-effects. That way, the same URL would refer to the same resource, time after time. Alternatively, POST is for server requests that change the content of the server. And that’s why we care.

I find that reading documentation helps. So I first looked at the browser source for a link_to_remote() call and it’s pretty clear it uses Form.serialize() to convert the form, and then sends it along using Ajax.Request(). The question is, does it send via POST or GET?

According to the documentation for link_to_remote(), it sends via POST by default.

The method used is by default POST. You can also specify GET or you can simulate PUT or DELETE over POST. All specified with options[:method]


  link_to_remote "Destroy", :url => person_url(:id => person), :method => :delete

I also checked it out with a quick check inside a controller with a breakpoint to see if a link_to_remote() call really sends a post. I put a breakpoint inside a controller method called by link_to_remote:

$ ./script/breakpointer 
Executing break point at ./script/../config/../app/controllers/friend_controller.rb:60 in `test_ajax'
irb(#):002:0> request.get?
=> false
=> true

As for a regular form_for() call, simply use the :method option. I think by default it also uses POST.

Erlang and neural networks, part I

So it’s been a whole week since my interesting post about the OwnershipFilter. I was investigating several things all at once, all the while wrestling with internal motivation (another post, at another time). In any case, I thought I’d blog about it entirely when I had something full and concrete to show you. However, if it takes me that long, I might as well blog about it as I go along–and it might be more fun for you anyway.

Trace it back

It started with an article about how the free lunch is over for software engineers that a friend sent to me about two years ago. It basically stated that developers have been riding on the wave of Moore’s law to save their butts, and it’s not going to last forever. In addition, it’s been known for a while that chip manufacturers are moving towards multi-core processors to increase performance. If developers are going to take advantage of hardware like they have been, they’re going to have to learn how to program concurrent programs.

The problem is, programmers suck at it. It’s well known that concurrent programming, as it stands, is not easy for humans. Even Tim Sweeney, the guy that architected the Unreal Engine (no mere programming mortal), thought it was hard. It was when I started looking beyond threads as a concurrency abstraction that I tripped over a programming language developed specifically for concurrency.

Yet Another Programming Language

A friend of mine, who is a teacher (i.e. not a programmer), recently asked me, “Why learn more than one programming language?” Ahh, little did she know that programming languages inspire what verges on religious debates between programmers. My short answer was, “Each tool is better at one task than another.” I was looking for another language that might do concurrency better.

I had always thought that I should learn more about functional programming. It seemed like an odd beast to me, especially since you don’t change state through side-effects. “How do you get anything done?” It’s kinda like when you first learned that you don’t need GOTO, and subsequently, when you learned that FOR loops suck.

And yet, I never really found a need or a small project I could do with functional programming that might prove to be satisfying. It was only due to the search for better concurrency abstractions that I ran across Erlang, a functional programming language that is used explicitly because it’s good at concurrency. In fact, it’s pretty much the only one out there that touts concurrency as its strength.

It uses the actor model, where processes share no data and just pass messages around. Because there’s nothing shared, there’s no issue of synchronization or deadlocks. While not as sexy-sounding as futures or software transactional memory, the actor model falls nicely along the lines of complex and emergent systems–systems that have locally interacting parts with a global emergent behavior. Hrm…could one of these systems be good for a small side project to do in Erlang?

How Gestalt, Mr. Brain

Artificial neural networks seemed to be the perfect thing actually. A quick, quick diversion into what they are.

A feed-forward artificial neural network is basically a network of perceptrons that can be trained to classify (ie. recognize) patterns. You give the network a pattern as an input, it can tell you the classification of that input as an output.

You can think of a perceptron much like a neuron in your brain, where it has lots of inputs and one output. It’s connected to other perceptrons through these inputs and outputs and there are weights attached to the input connections. If there is a certain type pattern of input, and it passes a threshold, the perceptron ‘fires’ (i.e. outputs a value). This in turn might activate other perceptrons.

Even simpler, a perceptron is modeled as a function that takes a vector x as an input and outputs a number y. All it does is take the dot product of the input vector x with weights vector w, and pass it through a non-linear and continuous thresholding function, usually a sigmoid function. And you connect them up in layers, and you get an artificial neural network, that can learn to recognizeclassify patterns if you train it with examples.

It has to learn patterns by adjusting the weights between perceptrons in the network after each training example, and you tell it how wrong it was in recognizing the pattern. It does this by an algorithm called back propagation. It’s the same page I lifted all these pictures from. I put all their pictures in an animated gif to illustrate (click on it to watch):

In the first part, the example propagates forward to an output. Then it propagates back the error. Lastly, it propagates forward the adjusted weights from the calculated error.

I think the shoe fits, sir

Why would this be a good fit as a subject to play with Erlang? Well, if you’ll notice, each perceptron only takes input from its neighboring perceptrons, and only outputs to its neighbors. This is very much in line with the actor model of concurrency. Each process would be a perceptron, and would act as an autonomous agent that only interacts with other processes it comes into contact with–in this case, only other perceptrons it’s connected to.

In addition, you’ll also notice that in the animation, the perceptron values are calculated neuron by neuron. In a concurrent system, there’s no reason to do this! You can actually do the calculation layer by layer, since the calculations of any individual perceptron only comes from the outputs of the perceptrons in the layers before it. Therefore, all outputs for perceptrons in a layer can be calculated in parallel.

Notice, however, that layers need to be calculated serially. I had originally thought that with the learning process propagating back and forth, maybe it could be pipelined. On closer examination, however, the best one can do is to feed-forward the next input one layer behind the adjusting of weights, to make the learning process go faster.

So I’m going to give it a shot on the side, and document it along the way, since a quick search on google revealed that though people talked about it, no one’s ever really given some snippets on it. Wish me luck!

Erlang and Neural Networks Part I
Erlang and Neural Networks Part II
Erlang and Neural Networks Part III

Stream Copy YouTube, Revver, Etc.

For lots of times in the past, I was always annoyed that I couldn’t save content offline, and always had to view it online–especially when there’s no guarantee that the content will stay there. So, I was a little bemused, but not altogether surprised, that you can save streaming copies of videos using Ruby. The source code is fairly short…maybe 20 lines or so. And while I haven’t tried it out, I’ve actually been able to read it through, and it’s nice to be able to read the works for people far better than me, even if it’s just table scraps.

Well, if you want to try it out, the balloon is here. A balloon is a web page that has ruby embedded in it, so that you can run it in ruby. Just make sure you have ruby installed on your system, and follow the instructions on the balloon.

Keep users from looking at other people’s data with a simple ownership filter

Happy St. Patrick’s day everyone!

Well. Everyone’s pretty familiar with authentication. That’s where you force the users to login, before you’ll show any of the pages. Usually, this is achieved with a before_filter, so that you’re not checking if a user has logged in in the beginning of each action in the controller. This keeps your code mighty DRY. But what about ownership of data? Within any web app, you have data that’s owned by some users, and other data that’s owned by other users. Just because they’re logged in doesn’t mean that they should be able to see other users’ data.

Oh, these tables of mine

A good example are friends in a social network app. Each user has a set of Friend records in the database that belongs to them. (btw, don’t use the User table below in your own app. you never want to store cleartext passwords in the database. Check out authentication tutorials)

create_table :users, :type => "InnoDB" do |t|
t.column :username, :string
t.column :password, :string

create_table :friends, :type => "InnoDB" do |t|
t.column :user_id, :string
t.column :name, :string
t.column :email_address, :string

create_table :posts, :type => "InnoDB" do |t|
t.column :user_id, :string
t.column :timestamp, :datetime
t.column :body, :text

Your first instinct sucks

You only want those records of friends that belong to them to be available to them. So what’s the first thing that you’re inclined to do? Assume that you have the User record as an indication of being logged in the session data.

class FriendController  ["user_id = ?", session[:user].id])

def show
@friend = Friend.find(params[:id])
unless @friend.user_id == session[:user].id
flash[:error] = "That friend does not exist"
redirect_to :action => :list

In list, you want to make sure that the list of friends returned belongs to the user, and that’s why you find all by the user_id. In show, a user can easily change the URL’s id number to reflect another record. You want to check whether that friend actually belongs to them.

While this is all good and well, the problem is, you’d have to do this for every method that you write. We are a lazy kind, so there HAS to be a better way.

Scope it, my brotha from anotha motha

Of course, the Rails geniuses have come up with with_scope() for all ActionControllers.

class FriendController  { :conditions => ["user_id = ?", session[:user].id] }) do
@friends = Friend.find(:all)

def show
Friend.with_scope(:find => { :conditions => ["user_id = ?", session[:user].id] }) do
@friend = Friend.find(params[:id])
if @friend.nil?
flash[:error] = "That friend does not exist"
redirect_to :action => :list

“But hold on, there’s still duplication!” This is only part of the solution. with_scope() is useful if you have multiple database finds within the same block. That way you don’t need to keep putting it in the conditions. So how do you get rid of the duplications? With filters, of course.

Add a sprinkle of filter magic

I personally liken filters to programming ‘common sense’ into the classes. It’s what’s intuitively understood to have to be done before and after every action. Luckily, there’s a filter called around_filter that we can use.

class FriendController  { :conditions => ["user_id = ?", session[:user].id] }) do

def list
@friends = Friend.find(:all)

def show
@friend = Friend.find(params[:id])
if @friend.nil?
flash[:error] = "That friend does not exist"
redirect_to :action => :list

So it should be pretty obvious what happened. The around_filter allows you to do one responsibility, both before and after every action in the controller. When the method yields, it gives control to one of the actions below. So you’re essentially wrapping every action in the with_scope() defined in the filter.

Yay, that’s pretty cool. The code’s been DRY’d. We’re done, right? But you know as well as I, that because there’s more text after this sentence, we can actually take it a step further.

I pull out the method inside, served it and fried

Of course, you have more than one ActiveRecord model that’s owned by the user, and in this case, there’s another one called Post. So instead of repeating ownership_filter method in every ActiveRecord Object, let’s pull it out of FriendController into a class of its own, so that other controllers can use it, using some easy meta programming. Note that you now have a require up top and around filter changed.

require 'ownership_filter'

class FriendController :list

And this is where we extracted the method to…a file named “ownership_filter.rb” You can put this in your app/controller directory.

class OwnershipFilter
def filter(controller)
model_class_name = controller.controller_name.capitalize.to_sym # => :Friend
model_class = Object.const_get(model_class_name) # => Friend class
model_class.with_scope(:find => { :conditions => ["account_id = ?", controller.session[:user].id] }) do

It actually took me a while to find out how to do this. I knew you could call methods dynamically with send(), but how do you dynamically get a class? Good thing for posts on ruby-talk. So basically, you take the controller’s name and you turn it into a symbol :Friend that is used to find the class, using const_get(), that the constant :Friend refers to, namely, the Friend class.

After that, it’s the same as before, you call with_scope() with it. So now, you can use this with any of the controllers that you have in the same ways as you did with the friend controller. I haven’t tried it yet with the finds pushed down to the ActiveRecords, but I think it should work the same way. Tip!

Update: I saw that the method I described above is actually an anti-pattern according to Jamis Buck. I’ve posted subsequently on this topic.

Noiz2sa by Kenta Cho (2002) |

“Noiz2sa is an early study in the art of bullet barrages and playing strategies. It is Cho’s Well Tempered Clavier, in which he challenges several of the aesthetic assumptions of the genre, showing off a talent for composing dizzyingly varied game stages out of a self-imposed set of constraints. “

Here’s a diversion for you. I found Noiz2sa by accident in Ubuntu’s repositories. It’s an abstract 2D space shooter. The review quoted had the funniest–and pretty much the only review for it. The game really is like an exploration of the genre. It’s like he said to himself, “what if we had, lots of bullets, and it was hard for you to get shot?” You have no powerups, and you have to make it through 10 substages. It’s actually a fun and addictive game, not to mention pretty.

Beyond being a diversion, I thought it was noteworthy that he expresses the bullet patterns in BulletML, a markup language for bullet patterns in 2D shooters. This makes it interesting, since you can attach it to a genetic algorithm, and evolve it as the player got better at dodging your previous bullet patterns through the game. It would not be far fetched to write a server that served these BulletML files in front of of the genetic algorithm, so players can download it, and then report how they did.

BulletML actually just describes a particle system. It might be applicable in certain data visualizations–though I don’t know what. What other uses do you think BulletML would have?

Vertical markets for social networking sites

Goodreads is a social networking site for book lovers. That really puzzled and surprised me. I know for a fact that vertical (niche) markets are always better to start off with when you’re starting out. It lets you build a community of like-minded people, and it makes you focus your product, which makes it not only easier to build, but makes you design a tighter product.

But the question to me at first impression was, why would people join a vertical market for something that already exists?

Goodreads is a well designed site. It has a core, limited feature, and it’s clean, and it’s easy to add books that you’ve read and write reviews on them. One can also see what your friends are reading and what they’d recommend.

However, the big three social networking sites, Friendster, MySpace, and Facebook all have a section where one can list and name books that interest you. And Amazon, already has both places to read/write reviews for each book, and a recommendation system that works pretty well. In addition, you can actually buy books there.

So what does Goodreads have to offer? Well, for one, at this early stage, it’s fast and responsive, and the mechanisms for actually adding a book is very easy. In addition, it plays on one of the elements of gaming, which is namely, collecting. And that goes along very well with those that read books…they like to have a full bookshelf (almost like trophies on display). There are many games that focus on the collection aspect. Yugi Oh! and Pokemon are the two that spring to mind. Goodreads is like pokemon…for people that like to read.

In addition, I think the book covers help enforce that idea of a collection, so you can see books in your bookshelf. And not only that, you can segment them according to what you’ve read, are currently reading, didn’t finish, etc. That’s something you can’t do with a normal bookshelf.

I won’t say that it’s inevitable that a niche market in an already saturated global market of social networks is going to succeed. But the way Goodreads did it makes sense. In fact, the same mechanism of collection that I outlined above can readily be applied to a social network of fashion geeks.

I own this shirt, or that blouse. I can see what my friends are wearing the next day. I can ask to trade clothes. I can make outfits for myself from my closet, and recommend them to others that have the same pieces of clothing. I can rate clothes and brands, and I can see what the overall trend for different types of clothing are.

I’d be tempted to make it myself, since I usually have an awful time shopping for clothes–I can never remember what I already have. However, 1) I’m not much of a clothes person myself, so I’ll only probably work on the ‘help me choose my clothes’ part, and 2) there are already plenty of niche market fashion sites out there.

What other vertical markets for social networks do you see that would have potential?