Disqus - Latest Comments for jeremyfelt

Re: OPML for Twitter lists (Scripting News)

jeremyfelt — Tue, 03 Nov 2009 19:27:35 -0000

Nicely done! I can't wait to see how smoothly this ends up including other pieces of a distributed 140 character system... and more! Then the fun really happens. :)

Re: On pubsubhubbub (Part 2) – Get with it, PuSH, you’re supposed to be realtime.

jeremyfelt — Wed, 30 Sep 2009 20:02:25 -0000

Hi Brett, thanks for stopping by.

This comment stream aside, the original post was written more as my perception than science. Rereading, it's a pretty unorganized perception. Ahh, late nights. There's more too the rambling, but if you come away with one thing from the above, it's that I don't see the FeedBurner stuff being real time as I thought it would.

I'll be grinding through the data more closely as the week goes on. The initial conclusions are based on a snapshot look at the initial 24 hours or so of use.

I can't answer to the latency yet, but I also can't imagine it being too high. Not a perfect answer, I know, but the server is on Amazon's EC2 and overall latency (network and system) seems low. Almost the only traffic coming in is from rssCloud and PubSubHubBub notifications. From watching Dave's rssCloud log (light pings), the time posted is usually less than .300 seconds.

The feeds that I've noticed the most issues with are from FeedBurner. My guess is that the delay and re-pushes are due to the ping scheduling between publisher->FeedBurner->PuSH. Once publishers start pinging directly to the hub instead of relying on a middle man, I would think that these issues clear up. See previous post directed at publishers.

The feeds that I've noticed the best response with are from Google Reader shared items. Again, perception, but things seem to run pretty smoothly here.

My generated twitter-link feeds seem to be sporadic when done in quick succession. @mmastrac pointed out after I posted last night that this could be a "race" between the feed writing and the hub reading if things are happening quickly enough. I still need to explore that.

You've given me a bunch of stuff to look at. I'll do what I can to start logging and parsing all of it and then provide the results. Hopefully I find a few problems with my code to fix along the way. :)

Re: On pubsubhubbub (Part 2) – Get with it, PuSH, you’re supposed to be realtime.

Brett Slatkin — Wed, 30 Sep 2009 19:21:21 -0000

Hey Jeremy, Thanks for reporting your experiences so far. How long was the sample period for your testing? I wonder what your results would be over the course of a week. It would be great to see some more data on end-to-end latency, retry attempts, duplicate deliveries, bandwidth, etc, especially if it were broken down by feed type.

Otherwise, what is your subscriber's average latency for handling notifications? The reference Hub is defensive about delivering to subscribers that track many feeds and are slow to respond. So, if you're taking over 5 seconds, you may see slowdowns. It's best to process incoming notifications asynchronously if you can.

Re: On pubsubhubbub (Part 2) – Get with it, PuSH, you’re supposed to be realtime.

jeremyfelt — Wed, 30 Sep 2009 18:02:33 -0000

Google did make announcements though.

http://adsenseforfeeds.blogspot.com/2009/07/wha... - Google announces PubSubHubBub support in FeedBurner feeds for AdSense, notifying a "Google-run Hub".

http://googlereader.blogspot.com/2009/08/pubsub... - Google announces PubSubHubBub support for shared items in Reader.

http://buzz.blogger.com/2009/08/blogger-joins-h... - "All blog post feeds now contain a "hub" element, and will ping Google's hub on every post update."

http://googlecode.blogspot.com/2009/08/towards-... - "we have gone a step further and added PubSubHubbub support to Google Alerts."

I'm not trying to push any blame on Brett. Google owns this now and should help any issues along. I've written about some of the issues I'm seeing with Google's PubSubHubBub Hub.

Constructive discussion about the issues I've been seeing is definitely welcome.

Re: On pubsubhubbub (Part 2) – Get with it, PuSH, you’re supposed to be realtime.

Amos Jones — Wed, 30 Sep 2009 16:41:17 -0000

Google did not make any announcements about Brett Slatkin's hub. It's not a Google product.

There are rough edges on all the work that's going on. We should cut everybody some slack including Brett Slatkin.

Re: On pubsubhubbub (Part 2) – Get with it, PuSH, you’re supposed to be realtime.

jeremyfelt — Wed, 30 Sep 2009 14:03:31 -0000

Cool, I'm not a big AppEngine guy. Not trying to argue the architecture. My perception is that it's stable and stays up.

Plugging the hub into many feeds is a great way to test. But, two things.

1) When you're big like Google and you announce the implementation, the perception given to me, the developer, is that you're ready.

2) When I, the developer, do start using it, I'll get a perception on how it's working and I'll share it.

If others have details on how it's working for them, I'd love to share examples. I'm having fun working with both rssCloud and PubSubHubBub and coding > arguing.

Re: On pubsubhubbub (Part 2) – Get with it, PuSH, you’re supposed to be realtime.

Amos Jones — Wed, 30 Sep 2009 13:10:59 -0000

All applications running on App Engine are forced to use multiple IP addresses.

You should not infer that applications running on App Engine have a guarantee of distribution or uptime.

Recent blog posts from the App Engine team indicate that applications run in a single data center at a time. The apps are single homed, not distributed across multiple data centers.

Plugging the hub into many feeds seems like a great way to bootstrap and test the realtime cloud, even if there are some rough edges.

Re: On pubsubhubbub (Part 2) – Get with it, PuSH, you’re supposed to be realtime.

jeremyfelt — Wed, 30 Sep 2009 12:03:07 -0000

Yes, referring to the official PubSubHubBub server that Google employee Bret Slatkin wrote and deployed. The one that Google then Pushed to their Blogger feeds.

Got it, no multiple hubs. But by using multiple IPs from the AppEngine network, a distributed network and uptime is inferred.

My point is - if you're going to push yourself into millions of feeds as a solution, then you are ready to go.

BTW, I'm not arguing against PubSubHubbub here. I want it to work. That's all that I'm getting at.

Re: On pubsubhubbub (Part 2) – Get with it, PuSH, you’re supposed to be realtime.

Amos Jones — Wed, 30 Sep 2009 11:49:51 -0000

The "official" PubSubHubBub server that you are probably referring to is the server written by Bret Slatkin and deployed to Google App Engine. You can find the source for the server at the PubSubHubBub website.

A quick browse of the source will show you that it's a simple app. There's no "network of hubs".

All App Engine applications use multiple IP addresses for connections. The number if IP addresses used by these apps should not be used to infer a guarantee of uptime.

By the way, an application running on App Engine can not subscribe to rssCloud because these applications do not use the same IP address for inbound and outbound connections.

Re: On pubsubhubbub (Part 2) – Get with it, PuSH, you’re supposed to be realtime.

jeremyfelt — Wed, 30 Sep 2009 11:02:13 -0000

The two circumstances are different. There are always rough edges, but....

The rpc.rsscloud.org server you are probably referring to is maintained by Dave with no promise of uptime (correct me if I'm wrong, obviously) as a place to test your implementation. It is possible that the server can be rebooted for changes at any time. No big company is providing a constant connection here.

The WP plugin for rssCloud creates a server on every blog that installs it. Problems have been few and far between with this.

The pubsubhubbub server hosted by Google has been pushed into every blogger feed and implemented heavily in FeedBurner feeds by Google. It pushes updates from multiple IPs, indicating a network of hubs that they are using to guarantee uptime. By doing this, they have told me that they are ready for real time.

Re: A question for DNS gurus out there in InternetLand (Scripting News)

jeremyfelt — Wed, 23 Sep 2009 22:13:03 -0000

Looks like it's still propagating. When I do a "nslookup r2.ly" from my Windows machine, I get my openDNS resolver, which returns 208.69.36.132 when I ping it.

But when I use this (http://network-tools.com/nslook/Default.asp) to do an nslookup, it sees that slicehost has it. Should probably fix itself soon. Assuming that you've gotten word within the last few hours, of course. If it's been a day, then I'd start to wonder.

Re: Hooking the lizard brain up to the cerebral cortex (Scripting News)

jeremyfelt — Mon, 21 Sep 2009 09:37:36 -0000

I'm probably confusing things by not being completely finished with it yet. :)

That subdomain didn't exist until last night. I added "rss" as a TXT record first, and then later as an A record.

I didn't get around to doing the redirect - was having too much fun with other things. Right now it goes to a "tesT" page on the server, but will redirect later to my feed.

Re: Hooking the lizard brain up to the cerebral cortex (Scripting News)

mterenzio — Mon, 21 Sep 2009 09:30:04 -0000

I'm missing something. Where was rss.jeremyfelt.com going before this?

Re: Hooking the lizard brain up to the cerebral cortex (Scripting News)

jeremyfelt — Mon, 21 Sep 2009 09:24:56 -0000

The main purpose is to eliminate confusion, which came up quickly in the previous DNS post. We all plug the address in our browsers to see what happens because seeing the TXT record isn't as easy.

Re: Hooking the lizard brain up to the cerebral cortex (Scripting News)

mterenzio — Mon, 21 Sep 2009 09:15:23 -0000

I had a long complicated DNS question but then realized a simpler one.

What is the advantage of the A record for rss.jeremyfelt.com?

Re: Hooking the lizard brain up to the cerebral cortex (Scripting News)

dave — Sun, 20 Sep 2009 23:52:39 -0000

That's why you have to try this stuff out and then listen. Everyone expected it to work, so then you have to figure it out, if it's at all possible. One of the cool things about this is you can subscribe to http://dave.supercloud.org/ and you don't even need to modify the client. :-)

Re: Hooking the lizard brain up to the cerebral cortex (Scripting News)

jeremyfelt — Sun, 20 Sep 2009 23:48:29 -0000

Awesome! Just tried it and it works on my previously registered supercloud already. Good, good stuff. I'm going to go add an A record for rss.jeremyfelt.com that I setup a TXT record for after your earlier post.

I probably overstate this, but I have so much fun with tech that slides together well.

Re: DNS for RSS feeds (Scripting News)

dave — Sun, 20 Sep 2009 21:56:31 -0000

What I like about it is that it's connecting up the lizard brain -- DNS with the cerebral cortex -- HTTP. They're at opposite ends of the evolutionary spectrum, but they're still a nice way to integrate them.

Re: DNS for RSS feeds (Scripting News)

jeremyfelt — Sun, 20 Sep 2009 21:39:52 -0000

I like that idea, hadn't thought of a combo deal. :)

Anything where the user only needs to know one thing (their address) and the technology (browser/cloud app) figures the rest out seamless is a good way to start.

Re: DNS for RSS feeds (Scripting News)

dave — Sun, 20 Sep 2009 21:34:33 -0000

BTW, it's not at all impossible to have dave.supercloud.org redirect to my feed, even if it's just a TXT record, without doing anything horrible. The DNS would return the same IP address for all *.supercloud.org hosts, where the web server would look up the host's TXT record and redirect to it. Nicely loosely coupled, no hacks, it should work.

Re: DNS for RSS feeds (Scripting News)

rbonini — Sun, 20 Sep 2009 14:28:13 -0000

Great idea. You get to keep your SEO juice as well.

Re: DNS for RSS feeds (Scripting News)

dave — Sun, 20 Sep 2009 14:24:03 -0000

Everyone expects to be able to resolve it in the browser, I understand
that DNS isn't set up to do this, that a domain can't redirect to a
specific URL but people expect it to.

Re: DNS for RSS feeds (Scripting News)

jeremyfelt — Sun, 20 Sep 2009 14:14:58 -0000

It doesn't make a website or redirect to your feed, it only adds a DNS TXT record for the feed specified.

Once you have this, an application that is looking for TXT records will know how to translate connectme.supercloud.org to whatever feed is stored as TXT.

Re: DNS for RSS feeds (Scripting News)

jeremyfelt — Sun, 20 Sep 2009 14:03:49 -0000

The practical applications are pretty clear to me.

You can store your feed in a TXT record under a domain name. If you move from 140char Client1 to 140char Client2, you can change your TXT record from feed1 to feed2 and the aggregators that are paying attention won't be required to notice the change. They'll just deal with it. You can change services without your friends having to follow around manually.

As for handling situations where other TXT records exist under that domain... It's up to the aggregator to know what a feed is/isn't and if it is/isn't cloud enabled. We'll just grab them and check.

Re: Someone give Om an award (Scripting News)

mterenzio — Tue, 08 Sep 2009 19:05:16 -0000

I'm pretty sure NYT uses a custom version of MultiUser Wordpress they inherited from About.com At least that was the case about two years ago. So they'd have to handle it themselves.