Details of CouchDB Implementation

Media_http3bpblogspot_wthub

Found a detailed piece hidden away in the mailing list archives of how Couchdb works. It’s from way back in 2008, but I think most of it still holds true. It’s a good read, especially because to me, mysql and postgres has always been black boxes.

CouchDb’s architecture seems simple enough to understand as a whole, and while the devil’s in the details, it goes to show that CS 225 was super important

Advertisements

A short script to start work and block sites

Sometimes, I’m able to focus and I can have noise in the background, but at other times, it can be hard. If you find yourself starting to type “faceb…” every time you hit upon a slightly hard problem, then maybe what you need is to block websites that you frequently go to when you’re bored.

There are plenty of services that lets you do this, including Rescuetime, I believe. Anyway, being a techie, you should know that you’re able to do this just by editing your etc/hosts file.

But, I know that I need something where in order to block sites, the barrier is low, and in order to unblock them, the barrier should be high. That way, I can still unblock something when I need it, but really, it’s mostly just an awareness that’s needed. Once you hit upon “This website can’t be found” a couple times, you really just buckle down every time there’s a hard problem, instead of dodging it.

All you need to do is put the startwork script file somewhere in your execution path, and then edit your /etc/hosts files with the websites that you want blocked pointing to your localhost (127.0.0.1). 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
##
# Host Database
#
# localhost is used to configure the loopback interface
# when the system is booting. Do not change this entry.
##
127.0.0.1 localhost
255.255.255.255 broadcasthost
::1 localhost
fe80::1%lo0 localhost

127.0.0.1 www.onemorelevel.com
127.0.0.1 onemorelevel.com
#127.0.0.1 www.hulu.com
#127.0.0.1 news.ycombinator.com
#127.0.0.1 www.facebook.com
#127.0.0.1 www.youtube.com

In order to run it, just type “sudo startwork”. And then in order to unblock a site, “sudo [use your favorite editor] /etc/hosts” and then comment out the sites you do want with “#”. Yay. Now get to work.

If MVP fell out of the sky and wiggled on your face, would you know it?

Minimum viable product. It’s something that most of us working on a product or project know we should do, but hard to actually do in practice.

For me as an engineer, it was particularly hard because you have the ability to build, and you like doing it. Therefore you’re use to building something first, and then showing it to people. That’s because just telling people that you have something without having actually built it just seems like…lying.

Half the problem of an MVP is picking the right scope for your problem. But beyond that, there are other reasons that might derail you from building an MVP.

Often times, we don’t know what actual MVP might look like. Other times, we have fear that gets in the way of letting us release MVP. I’ll tackle the first, and then the latter in a future post.

When you’re thinking about the solution to your problem, think of the simplest thing that could still work. Then cut and have it do less. Does it still solve the problem? If it doesn’t, see if you can shrink the scope of the problem. If does, see if you can shrink the solution. Keep going until you only have one basic unit of good/help that you’re giving to the world.

This basic unit of good is your core value proposition, and it’s also what you’re testing to see if it’s something people will want and find useful, because at this point, you don’t really know. You might know whether you’d find it useful, but you may not know if others will.

 If the core proposition of your product is compelling, your users will forgive all sorts of un-polish. Don’t fall into the trap of thinking that the next feature you add will be what brings the boys to the yard. There will not be just one feature that rockets you into the stratosphere. You’ll just end up building fluff and throw pillows on top of a cornucopia of crap features. I think it’s what they call polishing a turd (or putting lipstick on pigs). 

Think about software for an airport control tower. What’s its main core value proposition? To land planes. If it doesn’t do that, it doesn’t matter that it can land big planes, small planes, on any number of different frequencies, or how many it can do at once. No one will care because it doesn’t land planes.

Another good example is Mixpanel’s reporting. They report all their times in UTC, probably because they just didn’t want to mess around with timezones. You’d think that in building the MVP for web analytics, you’d need to show the stats in the user’s timezone. Apparently not. Mixpanel’s core proposition is real-time tracking of events generated by your visitors.

Delicious’s bookmarking (and by same reasoning, Flickr) is another good example. It’s core value proposition is for me to save web pages and organize it easily for recall later. It’s useful to me even if there’s no other users around. However, the public nature of the bookmarks and tags has the added side-effect of having aggregations. It has to be first useful to a single user, even without other users around. Many new startups ignore this aspect and concentrate on the benefits when there are already a lot of users. Very rarely will you have a scale of users to leverage in the beginning.

Along the lines of building single unit of good, don’t build for anyone other than your active users. When smart people, investors, or potential users/customers weigh in on your product, they’ll have some insights, but they are not your users (yet). In general, you should know the core value your active users get from your product better than them.

Build things for active users, I can’t stress that enough. It’s far too easy to fall into the trap of “If I build this or that feature they said they’d use, they will signup/use it.” Do this enough times, you have a mish-mash of features no one wants.

Sometimes, you don’t actually need to build out what you envision in your long term vision. You can leverage your landing page to set the users’ expectations for what the application is suppose to do in its broad scope, ambition, and intent. As long as you solve the simple version of that core problem for users, they’ll be ok with your un-polish.

If all of your potential users are all saying they want different things, either you haven’t set the expectations correctly when you’re selling your vision or you haven’t nailed down a core value.

But what if you whittled things down and it looks too simple? Too laughable? Too stupid? To that, I say don’t worry if the core value seems small as long as it’s something people want or needs to use. Everything is deeper than it looks.

You can much more easily position and market yourself when more focused. And when it comes to partnering, or being acquired, there’s less chance for conflict. This is all so logical and, yet, there’s a resistance to focusing. I think it comes from a fear of being trivial. Just remember: If you get to be #1 in your category, but your category is too small, then you can broaden your scope—and you can do so with leverage. – Evan Williams
In addition, most things that we take for granted nowadays as being useful started off looking like toys. The car, plane, wikipedia, music radio, TV, facebook, rails, javascript, twitter all started off looking like a joke–toys that can’t be taken seriously. And in their first inception, rightfully so. They had a lot of weaknesses overshadowed by other things that did it better. The horse could outrun the first car. The plane was an easy way to die. Can Wikipedia really be accurate with lots of anonymous edits? With music wireless boxes, concerts sound much better. So on and so forth.

That’s not to say all things that look like jokes will become useful (converse is not true!), but don’t be discouraged because your thing looks like a toy. Things are always deeper and more interesting the closer you look.

In the end, being able to identify your MVP takes clear thinking, understanding of your users, and the courage to be laughed at. 

Using quizzes to train your brain as an L1 cache for recalling people

Today, we launched quizzes at Noteleaf, where Noteleaf generates quizzes about people based on the notes you’ve taken of them. You can either start it from the web, or Noteleaf will email you a quiz every week. You can change the setting to ‘every day’, or if your memory is perfect you can set it to ‘never’.

Quizzes came about from feedback and our own experiences. Usually, we have time to look people up on Noteleaf before we go and talk to them. However, there are times where we’re expected to remember people on the spot, especially when we just run into them. In these cases, we can’t take the time to look up people on our devices–until we have bio-implants with an interface to the web that is.

While we don’t have bio-implants, we realized that we could use our brains as sort of a cache. It just needs a little bit of nudging for recall, and that’s where quizzes come in. It’s a fun way to review the things that you’ve written and use that to strengthen your brain cache.

As basic as it is, it’s actually pretty fun to take the quizzes, especially if you have a lot of notes. Alex Trebec, eat your heart out.

It had meant to be a short quick feature that we pumped out, but it ended up taking more than twice the amount of time. In hindsight, the estimation was right. Quizzes didn’t take that long at all–maybe two or three days. But what we didn’t take into account was all the infrastructure that goes with it. The background processing jobs. The cron script setups. The mixpanel analytics. The copy.

So next time that you’re doing estimation, take those things into account as well. 

 

Running Resque Web

Today, I finally dove into Rack. Whenever I hear about a new technology, I know what it is, but don’t have anything specific in mind, so I never end up playing with it. Until I have to–which is both good and bad. Good in that I never end up wasting time with dead-end technologies. Bad in that I could have solved other problems with this tool had I known about it more in depth.

I wanted to run resque‘s resque-web along-side of the Noteleaf app. It’s basically a simple sinatra app that tells you the current state of the background workers. After reading about it for a while, I found different solutions. , and .

The one on stackoverflow suggested that you write write your own config.ru. Btw, you put config.ru in your Rails root directory. No one seems to mention that. After a lot of reading I came up with the rather vanilla:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
require "config/environment"

# HTTP Auth For Resque Console
AUTH_PASSWORD = "password" || ENV['AUTH']
if AUTH_PASSWORD
  Resque::Server.use Rack::Auth::Basic do |username, password|
    password == AUTH_PASSWORD
  end
end

use Rails::Rack::Static
run Rack::URLMap.new(
  "/" => ActionController::Dispatcher.new,
  "/resque" => Resque::Server.new
)

However, when deploying it to the server, rackup didn’t seem to be able to find my rails gem. So looking around some more, I did find another hint on engineyard, which lead me to find this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
require 'sinatra/base'

class ResqueWeb < Sinatra::Base
  require 'resque/server'
  use Rack::ShowExceptions
  
  # Set the AUTH env variable to your basic auth password to protect Resque.
  AUTH_PASSWORD = ENV['AUTH']
  if AUTH_PASSWORD
    Resque::Server.use Rack::Auth::Basic do |username, password|
      password == AUTH_PASSWORD
    end
  end

  def call(env)
    if env["PATH_INFO"] =~ /^/resque/
      env["PATH_INFO"].sub!(/^/resque/, '')
      env['SCRIPT_NAME'] = '/resque'
      app = Resque::Server.new
      app.call(env)
    else
      super
    end
  end

end

You edit your config/environments.rb file like so, and resque_web.rb can either be required in your environments file or put in the initializers, where it’s loaded before environments is.

In hindsight, that works well, because it’s a small demo of how to insert your own middleware into rails. And it’s not that hard either. Just have to look up the rack calls, and environment variables, so you’ll know what’s available. 

Finally having something concrete for using Rack is great. I had been thinking about a host of other stuff that’s considered the underbelly of a web app that can be better implemented as Rack middleware apps, rather than contaminating your web app itself. Neato.