Last update: Thu, Feb 9, 2017 at 7:23 AM.
Introducing the "Cloud Pipe"
    • This is the last major piece of the rssCloud architecture, the only piece not specified in 2001.
    • With Cloud Pipe, it is possible to flow realtime updates from the cloud to desktop and mobile devices, even if they are not "net-accessible," that is, behind a firewall or NAT.
    • The technology is modeled after the realtime part of the FriendFeed API. I chose this approach because: 1. It's known to work. 2. It's easy to implement. 3. The server-side scales. 4. Other developers are familiar with it. 5. There aren't many ways to solve this problem.
    • This is the second revision of the document, the first was written on Dec 8, 2009.
    • This is the first version that specifies the actual protocol used in the demonstration client and server I've put together.
    • This should not be considered frozen, and anyone who implements this protocol should be prepared for it to change, perhaps radically. Caveat implementor!
    • The previous paragraph is the one that people will pretend they didn't read. So please read it. Thanks. :-)
    • The server defines an endpoint, specified by a URL.
    • The server is net-accessible. It is not behind a firewall or NAT.
    • It can receive rssCloud or PubSubHubBub notifications, perhaps others. At this time, the server I have implemented only supports rssCloud.
    • Users are identified by a username and password.
    • The protocol is REST. Basic HTTP authentication is used.
    • To tell the server what feeds the user is following, the client sends a POST to the endpoint. The body of the POST is an OPML subscription list.
    • The feeds may be realtime feeds or require polling. For realtime feeds, the server registers a notification request.
    • The server returns immediately, with only a <system> packet as a response. (See Formats, below.)
    • The client simulates a "long poll."
    • On startup, it enters a loop.
    • At the top of the loop it opens an HTTP connection to the cloud server by making a GET request, with a long timeout, say five minutes.
    • The server checks to see if there are any updates for the client. If so, it returns them as a sequence of XML elements, one for each update. (See Formats, below.)
    • If there aren't any updates, the server goes to sleep for a short period (on my server, three seconds) then checks again. After 180 seconds it times out and returns without any updates, with only a <system> packet.
    • The client is expected to call again. By having the server timeout after a few minutes we keep it clear of clients that may have died. It assures us that there is someone on the other side of the connection waiting for a response, or at least there was a few minutes ago.
    • The server response is in XML, a single <packets> element.
    • If there were N updates, there are N+1 sub-elements; one <fatPing> element for each update and a <system> element, with information about the server. As usual with XML data, the client ignores information it is not interested in.
    • Each <fatPing> element has a single attribute called "feed" which is the URL of the feed the update came from. The value of the fatPing element is encoded XML text, the contents of a single RSS <item> or Atom <entry>.
    • The server has processed the feed and determined that this is a new element. Further, it's valid XML. It's transmitted as CDATA because it may use elements from namespaces that are not declared by in the <packets> element. It's encoded because it may itself contain CDATA.
    • A <system> element has no attributes and (as of this writing) three sub-elements: <serialnum>, <when> and <secs>. serialnum is a number that is incremented for each packet. when is a RFC822-formatted string representation of the time the system packet was generated on the server. secs is the number of seconds the server thread was running when it returned.
    • I've set up a server for people to test CloudPipe apps with.
    • The endpoint: http://rpc.cloudpipe.org:5337/river2/cloudPipeServer
    • To use the service you must have an account on Identi.ca. Your identi.ca username and password identify you. I call them once to verify that the username and password are valid, then I retain a copy of the password and use it to authenticate you from then on. I do not under any circumstances use the identity to post to their service or to read any of the data in your account.
    • There's a limit to 15 feeds you can follow on this server. That's just to keep things manageable for testing.
    • If you want, you can follow the same feeds I'm testing with. It's a good mix of realtime and static feeds that update fairly frequently. By using the same feeds you really keep the server load down. Here's the OPML text.
    • My server only supports rssCloud. If your feed uses another realtime protocol, I'll poll the feed, once an hour.
    • If you have any questions post a comment here. Also if you get an implementation running, please let us know by posting a comment! :-)