Disqus - Latest Comments for btaylor

Re: How FriendFeed uses MySQL to store schema-less data - Bret Taylor's blog

Kris Jordan — Tue, 03 Nov 2009 16:26:16 -0000

What level of gz compression did you all settle on as the appropriate size/speed trade-off?

Re: The technology behind Tornado, FriendFeed's web server - Bret Taylor's blog

stanley — Wed, 30 Sep 2009 15:19:42 -0000

Running said demo gives 500 error.

Re: test - David Krisch's blog

davidkrisch — Sat, 19 Sep 2009 23:27:05 -0000

My apologies. I have disabled the application until I have learned more about Disqus configuration.

Re: test - David Krisch's blog

btaylor — Sat, 19 Sep 2009 15:59:58 -0000

Mind updating your disqus ID? I am getting emails for all these comments because they are still associated with the "bretttaylor" disqus ID from my blog template that you copied.

Re: Experimenting with Google App Engine - Bret Taylor's blog

btaylor — Sat, 12 Sep 2009 17:41:10 -0000

The source code for this blog is now checked in at http://github.com/finiteloop/blog/tree/master now that we have open sourced Tornado (http://bret.appspot.com/entry/tornado-web-server).

Re: The technology behind Tornado, FriendFeed's web server - Bret Taylor's blog

btaylor — Sat, 12 Sep 2009 12:07:10 -0000

The asynchronous features are not available in Google AppEngine, but a subset of Tornado's features do work in AppEngine. See http://www.tornadoweb.org/documentation#wsgi-an... and the included "appengine" example for a full-featured App Engine example app.

Re: The technology behind Tornado, FriendFeed's web server - Bret Taylor's blog

roder — Fri, 11 Sep 2009 11:49:16 -0000

I posted a performance test of Tornado and Twisted.web here:

http://www.apparatusproject.org/blog/2009/09/tw...

Re: The technology behind Tornado, FriendFeed's web server - Bret Taylor's blog

Thomas Hervé — Fri, 11 Sep 2009 03:12:44 -0000

> In general, it seems like Twisted is full of demo-quality stuff, but most of the protocols have tons of bugs.

It seems like a gratuitous comment. Most protocols implemented in Twisted are the most achieved implementations available in Python. Coming from a project where we can't see any single unittests, I'm a bit skeptical.

The complexity in Twisted core loop also comes from the support of multiple platforms. I can also see the ideas that were picked from the Twisted implementation in your loop code :) I'd like to see what would Tornado loop looks like if it supported OS X and Windows (in an efficient way, select doesn't really count).

Also, did you consider edge-triggered epoll? It would be a really interesting test that could make a real difference from Twisted.

Re: The technology behind Tornado, FriendFeed's web server - Bret Taylor's blog

zooko — Thu, 10 Sep 2009 21:29:24 -0000

Did you file any bug reports? I don't mean this as an implicit criticism -- I honestly wonder if you reported some bugs that might be relevant to me as a Twisted user or as a Twisted developer.

Re: The technology behind Tornado, FriendFeed's web server - Bret Taylor's blog

btaylor — Thu, 10 Sep 2009 17:42:25 -0000

I can't imagine there is much of a performance difference. The bottom is not that complex in my opinion. See ioloop.py and iostream.py for the details.

Re: The technology behind Tornado, FriendFeed's web server - Bret Taylor's blog

progrium — Thu, 10 Sep 2009 14:56:42 -0000

Yeah, I'm not a fan of the web stuff in Twisted ... however you're doing the same basic thing at the bottom... So how does Tornado compare to Twisted Web performance wise? Roughly the same?

Re: The technology behind Tornado, FriendFeed's web server - Bret Taylor's blog

dlsspy — Thu, 10 Sep 2009 14:55:25 -0000

I suppose it's an application-specific thing. inlineDeferreds is very easy, though. I wish some of the twisted community were able to address your issues early on.

The bigger issue is that there are places where tornado could be used today to do great good if it were built on twisted (things like buildbot really need a web framework).

Of course, I don't mean to sound like I don't appreciate the work you guys did and gave away. I was very excited to see it. I just can't use it for anything right now. :(

Re: The technology behind Tornado, FriendFeed's web server - Bret Taylor's blog

btaylor — Thu, 10 Sep 2009 13:50:36 -0000

When we started, we did use Twisted. In practice, I found Twisted tedious. The deferred abstraction works, but I didn't love it in practice. Likewise, the HTTP/web support in Twisted is very chaotic (see http://twistedmatrix.com/trac/wiki/WebDevelopme... - even they acknowledge this). In general, it seems like Twisted is full of demo-quality stuff, but most of the protocols have tons of bugs.

Given all those factors, it didn't seem to provide a lot of value. Our core I/O loop is actually pretty small and simple, and I think resulted in fewer bugs than would have come up if we had used Twisted.

Re: The technology behind Tornado, FriendFeed's web server - Bret Taylor's blog

btaylor — Thu, 10 Sep 2009 13:41:54 -0000

Disqus was having issues. Testing comments.

Re: How FriendFeed uses MySQL to store schema-less data - Bret Taylor's blog

VictusFate — Sun, 16 Aug 2009 14:23:31 -0000

Rockin!
I'll try and run this by database folks far smarter than myself and see if we can use it.
Thanks again Bret.

Re: How FriendFeed uses MySQL to store schema-less data - Bret Taylor's blog

btaylor — Sun, 16 Aug 2009 14:13:15 -0000

Replicate away - that's why we published it. FYI, the entire site is licensed under Creative Commons (http://creativecommons.org/licenses/by/2.5/), and all the source code under Apache 2. I am not sure which type of license is appropriate for a description of a scheme like this, so if you want an official license, you can consider it licensed under both and choose whichever seems most appropriate.

Re: How FriendFeed uses MySQL to store schema-less data - Bret Taylor's blog

bobdvirident — Tue, 21 Jul 2009 14:14:35 -0000

Bret,

We're working with the folks at Mixi and Tokyo cabinet is intersting. bobd@virident.com if you'd like to connect. We're deliveirng unparralled MySQL performance there with our Virident appliance. Cheers,

Re: How FriendFeed uses MySQL to store schema-less data - Bret Taylor's blog

Timandes — Sun, 28 Jun 2009 22:05:08 -0000

I'm puzzled by this part.....how to get newest records?
For example, i can get them from table `entities` by:
Select `body` From `entities` Order By `added_id` Desc Limit 10
or
Select `body` From `entities` Order By `updated` Desc Limit 10

But how to use `index_user_id` to do the same thing?
Select `entity_id` From `index_user_id` Where `user_id`='asdfasdfasdf' Order By ?????? Desc Limit 10

I'm bad in English....sorry about that.....

Re: How FriendFeed uses MySQL to store schema-less data - Bret Taylor's blog

alexpopescu — Tue, 26 May 2009 18:49:48 -0000

Bret, firstly thanks a lot for your post: it offered me THE argument to move forward with some of my old proposals.

It looks to me that the "cleaner" is a very smart piece in your solution. As far as you explained it, it is an external process that populates/updates the indexes. And that means it has to read sequentially through your main tables to unpack the BLOBs extracting the field values to be placed in the index_tables and I'm wondering how have you implemented this without impacting the performance (in this case sequential access being almost the same issue as adding new indexes). I assume the sharded table is small enough (probably somewhere around 1mil records) in which case adding the index would probably perform decently enough. Or is there some secret sauce in the cleaner that I'm missing?

I really hope you'll find the time to give a bit more details about it. Thanks in advance.

Re: How FriendFeed uses MySQL to store schema-less data - Bret Taylor's blog

btaylor — Mon, 25 May 2009 14:30:59 -0000

Indexes are stored as btrees on disk, so you can iterate over them both backwards and forwards equally efficiently.

Re: How FriendFeed uses MySQL to store schema-less data - Bret Taylor's blog

zennon — Sun, 24 May 2009 05:48:33 -0000

Hi Bret,

Thanks for sharing this. I have a question and I hope that you have a time to answer me.
Is the sort_time value here reversed (f.e future date minus actual time) so the PK index is ordered exactly as the user stream visually, the newest activity on the top?

Thanks in advance for your reply

Re: Why I Didn’t Get to See FriendFeed’s Redesign

sarahintampa — Mon, 06 Apr 2009 13:45:34 -0000

Thanks for the personal response.

Re: Why I Didn’t Get to See FriendFeed’s Redesign

btaylor — Mon, 06 Apr 2009 13:32:20 -0000

So sorry. We were pretty swamped finishing things this weekend, and am extremely sorry we didn't connect. We will make ourselves more accessible in the future. We appreciate your being so candid, and sorry again for the miscommunication on our end.

Re: How FriendFeed uses MySQL to store schema-less data - Bret Taylor's blog

Paul Kinlan — Mon, 30 Mar 2009 07:02:39 -0000

I am definatly keen on seeing the db abstraction open sourced, because your methods for achieving schema-less db is something I really like and probably need for a future project I am planning.

Thanks,
Paul Kinlan

Re: How FriendFeed uses MySQL to store schema-less data - Bret Taylor's blog

coder — Sun, 08 Mar 2009 17:56:20 -0000

Thanks for the info.