tl;dr - Go vote for tldr.it on http://railsrumble.com!
(That’s the screen cast I was going to post where the “How it works” section is now on the page, but I didn’t have time. View it on Vimeo, if you’d like.)
So, wow, it’s really been a wild ride for my Rumble app so far. I thought maybe I’d share a little bit about the app’s story today, and then maybe share a bit about how it works in another entry tomorrow-ish.
I never really used RSS until last year. I knew what it was, why people used it, and so on, but I just figured I’d rather (a) read things in a visually attractive environment and (b) give ad revenue to the sites that I really like reading. While I still hold the latter sentiment, I discovered two things. As I expanded my reading list and started reading more and more blogs and news sites everyday, it became really time consuming to go to each site individually, and only about half of the sites I read are actually attractive (the other half are really ugly). I also realized that most of the feeds that are worth my time and effort to support actually put ads in their feeds. So my RSS usage ramped up.
After using RSS for a while, I realized that while I could get to more information faster, a lot of it (a) wasn’t worth my time or (b) said a little that was excellent but it was wrapped in a lot of crap. After suffering with the annoyance for a while, I deleted a lot of feeds from my list, but that wasn’t working for me either since I really did like reading some of the ones whose signal to noise ratio wasn’t quite to my standards. So it dawned on me: there must be a way to dig out what’s good. If you could summarize what’s in the feed, you can see whether reading the full article is worth the effort. I formulated some ideas about how it would work, played with a few names, and drew up some plans.
And then that sat there for months.
But one day when talking to a good friend of mine about news and RSS and such, I shared the idea I had. He thought it was such a good idea, he instantly bought the domain for me and threatened to lock me in his basement until I’d finished building it. Of course, I claimed I’d try to do it soon, got wrapped up in the holidays and changing jobs, and then forgot about it.
Until the Rumble this year.
So the Rails Rumble rolled around this year, and this idea seemed prime for some 48-hour construction action. I lost most of my notes from my previous planning, so I drew up new ones and phased everything out for 48 hours. I made a few choices so that I could definitely fit it into two days:
Some of these decisions were great decisions (Feedzirra being a prime example of that), some of these ended up changing (I did have time for URL’s), and so some of them came back to bite me (more on that later).
So, I built it. I did not spend the whole 48 hours glued to my keyboard, but I spent a significant portion of my weekend working on it. I think most people look at the app and think “gee, that’s really simple” but they don’t see all the code that goes into the content extraction, the preparations for summarization, the background processing, and so on. So the front end code is quite simple, but the backend is pretty complex (I’ll discuss it more in another post).
So I launched the app about 5 minutes before the end of the competition. I’d have launched it sooner, but my dj runners were giving me fits and required some extra fiddling.
The app essentially summarizes text. So, for example, let’s say you’re reading this Guardian story on DADT. It’s fairly long and something I’ve read a lot of articles on already. It’d be great if I could figure out if this says anything new without having to waste time reading a ton of text. If you plug it in, you’ll get this back as the medium summary:
The lifting of a ban on gays serving openly in the US military proved shortlived after a federal appeals court ruled late on Wednesday in favour of granting the Obama administration a temporary delay.
Although President Barack Obama favours an end to the ‘don’t ask, don’t tell’ policy in which gays could serve in the military, as long as their sexual orientation remained secret, his justice department went to the courts on Wednesday seeking a temporary delay to allow the military time to prepare for the end of the gay ban, and, possibly, allow Congress to legislate.
OK, not bad, but nothing new. Skipped!
Now, imagine you can do that with almost any page or RSS feed. With tldr.it, you can. It summarizes the text of articles down to 15%-30% of the original length. Of course, the algorithms aren’t perfect (they were built in 48 hours after all and word frequency isn’t the best summarization algorithm), but that’s the idea that the app hopes to build on.
I posted it on Hacker News (like any good Rumble competitor) and tweeted it. A few people retweeted it, but then I noticed that Robert Scoble (@scobleizer) favorited my tweet about it. “Hm. Maybe people will be interested in it,” I thought. That was quickly followed by the app hitting the front page of Hacker News, and my first real technical problem.
You see, we were given a 512MB Linode instance for use with the Rumble. Serving up normal traffic and traffic from judging wouldn’t even cause that sort of box to break a sweat. Unfortunately, as it started getting tweeted around and featured in different places, Passenger and my background workers chewed through the RAM quite quickly, causing paging and serving things really, really slowly. Once I adjusted those things around, I was finding that it was then hitting the CPU boundary because it was putting so much stress on so few workers. So I decided to deal with the paging until I could get a bigger box setup. I setup a 1024MB instance, copied the app over, and pointed the main domain over at that. Crisis averted for the time being.
Then @nickbilton of the NYT was nice enough to tweet about it, causing another flood of traffic to hit the box. It actually withstood that storm fairly well, but the way it handled it made me hope it didn’t get hit with anything bigger.
Then I got an e-mail from TechCrunch. “Great,” I thought. “That’ll be excellent press, especially when voting opens.” So I answered the questions and asked that they please, please hold the story for another day so I could get my box up to speed and fix my stack.
So, when the TechCrunch traffic came, the box went into total fail mode. At first, the load overwhelmed it. Then it seemed to just give up (no resources were being taxed but Apache wasn’t serving; a restart of the box did nothing to fix this). Unfortunately I was a Florida Creatives/ORUG meetup that night, so I couldn’t do anything about it for a while. Once I got home and realized Apache was being an epic pile of fail, a really nice fellow Rumbler (@vertis) was nice enough to jump in and get nginx going for me in about 20 minutes (would’ve taken me a few hours easily). It started serving requests (fast, might I add), and all was well.
The app hit Techcrunch JP the next morning without even blinking, and it’s served up nicely ever since.
From there, it’s been featured in a lot of places…
The attention has been incredible and humbling. It’s a little hack for me, and it seems a lot of people are interested in it.
So, what are the numbers. Press is great, but if you aren’t getting users, it doesn’t matter, right?
And they’re still growing. The attention and usage has been very exciting, and I hope moving forward it continues.
I don’t know what next steps are exactly. I haven’t quite decided. Intridea has signaled that they would be open to continuing development on it and maintaining it as one of our products. That option is attractive since I think it is a really cool product. I have a lot of good ideas for monetization and further product development (including a battery of immediate fixes I need to get in…) that could easily turn into a solid product roadmap.
Then again, I may list it on Flippa and turn it over to a team that I know can really invest a lot of time and interest in it. I don’t know how much interest I would find there, but it is something I’m thinking about.
So, it’s up to vote now. The only problem is I’m not doing so well! I don’t know if I’m being trolled (I was at 3.4 and after one of the earliest refreshes in the competition, I obviously got a slew of 2/2/2/2 or lower votes for some reason) or (more likely) maybe the app isn’t as good as I thought it was. Either way, I’d really appreciate your votes.
I’ve been extremely humbled by how people have spoken of my app, and I hope people continue to use it. Let me know if you have any ideas or feedback. Next up is a post on the more technical aspects of the application.
Powered by Tumblr; designed by Adam Lloyd.