This is the last major piece of the rssCloud architecture, the only piece not specified in 2001. #

With Cloud Pipe, it is possible to flow realtime updates from the cloud to desktop and mobile devices, even if they are not "net-accessible," that is, behind a firewall or NAT.#

The technology is modeled after the realtime part of the FriendFeed API. I chose this approach because: 1. It's known to work. 2. It's easy to implement. 3. The server-side scales. 4. Other developers are familiar with it. 5. There aren't many ways to solve this problem. #

This is the second revision of the document, the first was written on Dec 8, 2009.#

This is the first version that specifies the actual protocol used in the demonstration client and server I've put together. #

This should not be considered frozen, and anyone who implements this protocol should be prepared for it to change, perhaps radically. Caveat implementor! #

The previous paragraph is the one that people will pretend they didn't read. So please read it. Thanks. #

The server defines an endpoint, specified by a URL. #

The server is net-accessible. It is not behind a firewall or NAT. #

It can receive rssCloud or PubSubHubBub notifications, perhaps others. At this time, the server I have implemented only supports rssCloud.#

Users are identified by a username and password. #

The protocol is REST. Basic HTTP authentication is used.#

To tell the server what feeds the user is following, the client sends a POST to the endpoint. The body of the POST is an OPML subscription list. #

The feeds may be realtime feeds or require polling. For realtime feeds, the server registers a notification request. #

The server returns immediately, with only a <system> packet as a response. (See Formats, below.)#

The client simulates a "long poll." #

On startup, it enters a loop. #

At the top of the loop it opens an HTTP connection to the cloud server by making a GET request, with a long timeout, say five minutes. #

The server checks to see if there are any updates for the client. If so, it returns them as a sequence of XML elements, one for each update. (See Formats, below.)#

If there aren't any updates, the server goes to sleep for a short period (on my server, three seconds) then checks again. After 180 seconds it times out and returns without any updates, with only a <system> packet.#

The client is expected to call again. By having the server timeout after a few minutes we keep it clear of clients that may have died. It assures us that there is someone on the other side of the connection waiting for a response, or at least there was a few minutes ago.#

The server response is in XML, a single <packets> element. #

If there were N updates, there are N+1 sub-elements; one <fatPing> element for each update and a <system> element, with information about the server. As usual with XML data, the client ignores information it is not interested in.#

Each <fatPing> element has a single attribute called "feed" which is the URL of the feed the update came from. The value of the fatPing element is encoded XML text, the contents of a single RSS <item> or Atom <entry>. #

The server has processed the feed and determined that this is a new element. Further, it's valid XML. It's transmitted as CDATA because it may use elements from namespaces that are not declared by in the <packets> element. It's encoded because it may itself contain CDATA.#

A <system> element has no attributes and (as of this writing) three sub-elements: <serialnum>, <when> and <secs>. serialnum is a number that is incremented for each packet. when is a RFC822-formatted string representation of the time the system packet was generated on the server. secs is the number of seconds the server thread was running when it returned. #

A response that represents a single item from the NY Times home page feed.#

A response that only has a <system> packet.#

The most recent response from my development server to my desktop client.#

An OPML subscription list sent to my server.#

Source listings of my client and server.#

I've set up a server for people to test CloudPipe apps with. #

The endpoint: http://rpc.cloudpipe.org:5337/river2/cloudPipeServer#

To use the service you must have an account on Identi.ca. Your identi.ca username and password identify you. I call them once to verify that the username and password are valid, then I retain a copy of the password and use it to authenticate you from then on. I do not under any circumstances use the identity to post to their service or to read any of the data in your account. #

There's a limit to 15 feeds you can follow on this server. That's just to keep things manageable for testing.#

If you want, you can follow the same feeds I'm testing with. It's a good mix of realtime and static feeds that update fairly frequently. By using the same feeds you really keep the server load down. Here's the OPML text. #

My server only supports rssCloud. If your feed uses another realtime protocol, I'll poll the feed, once an hour. #

If you have any questions post a comment here. Also if you get an implementation running, please let us know by posting a comment! #