Your SlideShare is downloading. ×
0
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Ctp
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Ctp

672

Published on

DivConq's Common Transfer Protocol is a protocol that supports remote calls for both small messages and large files over various network layers - including accelerated transfers over UDP.

DivConq's Common Transfer Protocol is a protocol that supports remote calls for both small messages and large files over various network layers - including accelerated transfers over UDP.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
672
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
13
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. DivConq Framework’s CTP
  • 2. A file and message transfer protocol that transcends the network protocol.• Common Transfer Protocol provides a complete spec for messaging commands as well as file transfers (SIGN ON, DIR, GET, PUT)• It is abstracted from network layer, could be implemented in TCP, UDP or through either HTTP or Web Socket• It does not assume full duplex communications• It does not (necessarily) associate network layer client credentials with the file transfer, which can lead to scalability and efficiency issues• It is platform and language neutral • Server could be implemented in Java, C, Python or similarly suitable languages • Client can be implemented in any language with TCP or HTTP capabilities• It is open and free
  • 3. Client Server CTP App Service CTP CTP Session Network Session Connector LayerCTP Session provides the desired abstraction by living just outside of the networkprotocol. A CTP Session represents one authenticated user, it can accept duplexor half-duplex interactions and otherwise doesn’t care about the network layer.The CTP Service processes the commands sent to the session and providessupport for the file systems the server exposes.
  • 4. Client Server CTP App Service CTP CTP Session Network Session Connector LayerThe CTP Session Connector may be JavaScript in a web page, a Python or Javalibrary, or otherwise. It keeps track of session context for all calls to the server.The session connector may be paired with a full-duplex or a half-duplexnetworking library. To the App developer the API calls will look pretty much thesame either way (with a few exceptions for straight HTTP support).
  • 5. The past has many examples of file transfer protocols.• FTP – commands and network completely intertwined• SFTP – the file transfer protocol *is* separate from the network protocol, a good model but • Effectively stuck on TCP/IP because of SSH • Assumes full duplex communications • Associates the communications client credentials with the file transfer, which can lead to scalability issues • Can be challenging to debug/trace, especially if custom extensions are present• HTTP POST/GET – commands and network not separated• Existing accelerated transfer protocols – commands and network not separatedNo existing candidates for a Common Transfer Protocol, but a lot to learn from allof these.
  • 6. DivConq CTP
  • 7. Client 1 Server HTTPS App HTTP CTP Port 443 CTP HTTP CTP Session Session Service Connector UDP CTP CTP Session Session Client 2 WSS App CTP UDP UDP Session Port 2020 Web Socket Connector Port 2022 Client 3Session Connectors paired with networking WebSocketprotocols. A single server can listen for differentnetworking protocols on different ports – but App CTP Sessionagain CTP Session and CTP Service can be Connectormostly ignorant of the network protocol.
  • 8. Client 2 Server CTP TCP Service 15 App 15 CTP Port 2021 TCP Threads Session TCP Connectors 15 CTP Sessions 3 ConnectionsFull duplex communications allow for much more efficient use of a TCPconnection. Traditionally (think FTP or HTTP) we follow a send request thenwait for response (half-duplex) approach to issuing commands. If you justsend a request and assume a response will come back when it is ready, thenyou can keep sending more commands and data blocks.Because of this efficiency, and because we do not tie a user to the networkconnection, we could be servicing 15 “users” (Sessions) on the client on aslittle as 3 TCP connections.
  • 9. Server CTP Service HTTPS 443 WSS 443 UDP 2020 TCP 2021 (HCTP) (WCTP) (ACTP) (SCTP)Though not limited to only these options, the design of CTP has support thefollowing options within its design:• HTTP/HTTPS – henceforth called H(yperText)CTP• WS/WSS – henceforth called W(ebSocket)CTP [using WS Binary mode]• TCP - henceforth called S(tandard)CTP• UDP - henceforth called A(ccelerated)CTP
  • 10. DivConq CTP
  • 11. Many of the following slides present details and examples of a referenceimplementation of CTP. As long as the results are the same over the wire, itdoesn’t really matter how the CTP server or client is implemented. Butunderstanding a reference implementation will help expose the design ideas.
  • 12. CTP ServerConnectors UDP (ACTP) Common Security TCP Common (SCTP) Handler Session Handler WS (WCTP) Web Security HTTP HTTP (HCTP) Handler There are four options for network connectivity, but there are really only three paths through the system. Each message is required to have a session identity by the time it reaches the Session Handler. Typically this means each message contains a session identity, except with HTTP which has the identity in a secure HTTP only cookie.
  • 13. CTP ServerConnectors UDP Common Security Common TCP Handler Session Handler Web Socket Web Security HTTP HTTP Server Handler With Common Security the host name provided on initial network connection becomes the default host name, but individual messages may override that name when authenticating a user. Although encryption of these connections resembles SSH, the client key given at connection is not a user key – it is just a way to confirm that the client app is allowed. User keys are provided during user authentication. A single network connection may support multiple users/sessions.
  • 14. CTP ServerConnectors UDP Common Security Common TCP Handler Session Handler Web Socket Web Security HTTP HTTP Server Handler With Web Security the host name provided on connection may not be overridden by a message. Encryption of these connections uses SSL, the client key (if any) *is* considered to be the user’s key. There can only be one user and one session per connection. An Authenticate message must still be sent, however, before a Session is ready to use.
  • 15. CTP ServerConnectors UDP Common Security Common TCP Handler Session Handler Web Socket Web Security HTTP HTTP Server Handler With Common Handler communications are expected to be full duplex. The server or client may push messages at any time. There is no need to wait for responses. Messages entering Session Handler can be either “send and forget” or “send”, there is no “send and wait”.
  • 16. CTP ServerConnectors UDP Common Security Common TCP Handler Session Handler Web Socket Web Security HTTP HTTP Server Handler With HTTP Handler communications are expected to be half duplex, the server may NOT push messages at all. Caller must wait on responses. Messages entering Session Handler can be “send and forget” or “send and wait”. HTTP Handler sets a SessionId cookie to track the session identity.
  • 17. DivConq CTP
  • 18. Example Authentication Message:To the client and server message structure is JSON- {like (exact details vary depending on network layer). Service: "Sessions",Actually messages are YAML-like in that data types Feature: "Manager", Op: "Start",may be associated with fields, but for practical Body: { Credentials: {purposes messages are JSON-like. UserName: "fredsmith", Password: "temp123" }Messages contain fields. Fields may hold a scalar } }value (string, number, true, false or null) or an objector an array. Just like JSON. Structures may benested.Web page scripts may deal with messages “natively”using JSON. But JSON is also easy enough to workwith in many programming languages.When not dealing with large binary data themessage structure works fine – smallish binary datacan be base64 encoded and treated as text.
  • 19. When dealing with large binary data though, base64 is not the way to go. WithUPD (ACTP), TCP (SCTP) and Web Sockets (WCTP) we can take advantageof the fact that there are data types and build that awareness into the networklayer. The network layer then sees a sort of “binary JSON” or more accuratelya “binary YAML”.Web Sockets and JavaScript have binary capability (in HTML5), which is whyWCTP is listed above. HTTP, however, is still better off in text (XML/JSON)based calls to the server (XHR) – especially for legacy support. ThereforeHTTP will use JSON/XHR to perform all CTP commands, and will open aseparate connection for pure upload or download of binary content.Although this exception for HTTP makes the implementation of CTP morecomplex, it does ensure highest compatibility since any HTTP 1.1 library canaccess this service. It also avoids some of the biggest mistakes of FTPbecause there is no “active mode” data transfer and there is no separate portfor data transfer – use the same port just a different URL. So proxies and firewalls are not as big an issue here as with FTP.
  • 20. Typical Fields:When making a request message (client) there are {some fields that will appear frequently in the Service: "XXX",following examples. A quick overview: Feature: "YYY", Op: "ZZZ", RespondTag: "010A0D0502", Body: {Service – the name of the destination service ...Feature – a part of the service } }Op – operation to perform with FeatureRespondTag – only present if you want a responseto the message, provides a way for the client toassociate a response message with a callbackBody – the payload (or parameters) for thefeature/operation requested
  • 21. DivConq CTP
  • 22. Example Message as seen by Session Handler: ServerRPC { UDP Service: "Sessions", Common Feature: "Manager", Security Op: "Start",RPC RespondTag: "010A0D0502", TCP Common Body: { Handler Domain: "divconq.com", Origin: "rpc:[ipaddress]", Session Web Credentials: { Handler Socket UserName: "nnnn", Password: "nnnn" Web } Security } HTTP HTTP } Server Handler When the Message gets to Session Handler an Origin field has been injected by Common Security. If no Domain was provided by the original sender, then that too is injected (based on the default Host for that network connection).
  • 23. Example Message: Server { Service: "Session", UDP Feature: "Manager", Common Op: "Start", Security RespondTag: "010A0D0502", Body: { TCP Common Domain: "divconq.com", Handler Origin: "rpc:[ipaddress]", Session Handler Credentials: {RPC Web UserName: "nnnn", Socket Password: "nnnn", SslThumb: "010A0D0502" Web } Security } HTTP HTTP } Server Handler When the Message gets to Session Handler the Origin, Domain and SslThumb (optional) fields have been injected by Web Security.
  • 24. Example Message: Server { Service: "Session", UDP Feature: "Manager", Common Op: "Start", Security RespondTag: "SendWait", Body: { TCP Common Domain: "divconq.com", Handler Origin: "rpc:[ipaddress]", Session Credentials: { Web Handler UserName: "nnnn", Socket Password: "nnnn", SslThumb: "010A0D0502" Web } Security }RPC HTTP HTTP } Server HandlerWhen the Message gets to Session Handler theOrigin, Domain and SslThumb (optional) fields havebeen injected by Web Security. [more next slide]
  • 25. Example Message: Server { Service: "Session", UDP Feature: "Manager", Common Op: "Start", Security RespondTag: "SendWait", Body: { TCP Common Domain: "divconq.com", Handler Origin: "rpc:[ipaddress]", Session Credentials: { Web Handler UserName: "nnnn", Socket Password: "nnnn", SslThumb: "010A0D0502" Web } Security }RPC + Web HTTP HTTP } Server HandlerBoth browser requests for web pages as well as XHR, require a Session Context.To support both models the session context is extracted by HTTP handler from asecure HTTP [server only] cookie. Server only is used to protect the cookie fromcross domain attacks and such.
  • 26. DivConq Collab
  • 27. App Server UDP Common Security TCP Common Handler Session Web Handler Socket Web Security ARPC HTTP HTTP Server HandlerInitial RPC/Web calls cause a Session to be created in Session Handler. Calls witha SessionId cookie will be connected to an existing Session.Although the App Server cannot push messages over HTTP, it can queuemessages for an (HTTP originated) Actor of a Session.
  • 28. Example RPC Request Message: App Server { Service: "Session", UDP Feature: "Manager", Common Op: "Start", Security RespondTag: "SendWait", Body: { TCP Common Credentials: { Handler UserName: "nnnn", Session Password: "mmmm" Web Handler } Socket } } Web Security ARPC HTTP HTTP Server HandlerExample of an Initial RPC call. Credentials areoptional, if not present then the session will be aguest session. It takes Trust points to create asession, too many attempts will block the client.
  • 29. Example Response Message: Stored in Session: App Server Context: { { Domain: "divconq.com", UDP Result: 0, // success Origin: "http:[ipaddress]", Body: { UserId: "119", Common UserId: "119", UserName: "awhite", Security UserName: "awhite", FirstName: "Andy", LastName: "White", TCP FirstName: "Andy", Common Email: "andy.white@divconq.com", LastName: "White", Handler AuthToken: "010A0D0502", Email: "andy.white@divconq.com", Session Chronology: "/America/Chicago", Chronology: "/America/Chicago", Locale: "en-US", Web Handler Credentials: { Locale: "en-US", UserName: "nnnn", Socket Session: { Password: "mmmm" Web Id: "nnnn", } Security Key: "010A0D0502" A }RPC HTTP } HTTP Server } Handler } Response Session Cookie: Session: "nnnn_mmmm_010A0D0502"Client gets back User info (JSON) and Session info (cookie). An AuthenticationToken as well as the User info are stored in the Session. Anyone with the sessionId and Key can now act as this authenticated user (see guest next slide)
  • 30. Example Response Message: App Server { UDP Result: 0, // success Body: { Common Chronology: "/America/Chicago", Security Locale: "en-US", TCP Common Session: { Handler Id: "nnnn", Session Actor: "mmmm", Handler Key: "010A0D0502" Web } Socket } Web } Security ARPC HTTP HTTP Response Session Cookie: Server Handler Session: "nnnn_mmmm_010A0D0502“ Stored in Session: Context: { Domain: "divconq.com",If a guest session (no credentials), then all we get Origin: "http:[ipaddress]",back is session info. The default (guest) Chronology: "/America/Chicago", Locale: "en-US"Chronology and Locale are returned. }
  • 31. Example RPC Request Message: App Server { Service: "X", UDP Feature: "Y", Common Op: "Z", Security RespondTag: "SendWait", Body: { TCP Common ... Handler } Session } Web Handler Socket Session Cookie: Web A Session: "nnnn_mmmm_010A0D0502" SecurityRPC HTTP HTTP X Server HandlerMessage requesting service X is processed through the session. The sessionbecomes the proxy for the client. User Context is restored from session cookie by avisit to the session state before passing the message on to service X. Service Xhas context and a destination for response, thanks to the session. When thesession gets the response it then sends it back to the client (with HTTP the client isblocking until the response comes).
  • 32. Example RPC Response Message: App Server { UDP Result: 0, // success Tag: "SendWait", Common Body: { Security ... TCP Common } Handler } Session Web Handler Socket Web Security ARPC HTTP HTTP X Server HandlerNothing special is returned. Result is non-zero if error. Tag for the RespondTagfield sent. Message for any error message. Body for the data of the response.
  • 33. App Server UDP Common Security TCP Common Handler Session Web Handler Socket Web Security A HTTP HTTP ABC Server HandlerSomeone sends a message for session A. A is configured (by HTTP Handler) toqueue incoming messages.
  • 34. Example Request Message: App Server { Service: "Session", UDP Feature: "Mail", Common Op: "Check", Security RespondTag: "SendWait", Body: { TCP Common ... Handler } Session } Web Handler Socket Session Cookie: Web A Session: "nnnn_mmmm_010A0D0502" SecurityRPC HTTP HTTP ABC Server HandlerClient must actively decide to check mail (queue) by issuing a request. The requestmay be long polling. Messages on the queue are popped and returned in a list.
  • 35. Example Response Message: App Server { Body: [ UDP { Common Service: "Desktop", Security Feature: "App", Op: "Open", TCP Common Body: ... Handler } Session ] Web Handler } Socket Web Security ARPC HTTP HTTP ABC Server HandlerEach response has a target Service, Feature, Op, Body. FromHub, RespondTo andRespondTag are allowed. ToHub and Tag are removed as those where valuesassociated with the Session.
  • 36. DivConq CTP
  • 37. App ServerRPC UDP CommonRPC Security TCP Common Session Handler HandlerRPC Web Socket Web A Security HTTP HTTP Server Handler Each request via Common Handler must contain a session context or be requesting a session context.
  • 38. Example RPC Request Message: App Server {RPC Service: "Session", UDP Feature: "Manager", Common Op: "Start", Security RespondTag: "010A0D0502",RPC Body: { TCP Common Session Domain: "divconq.com", Handler Handler Credentials: { UserName: "nnnn",RPC Web Password: "mmmm" Socket } Web A } } Security HTTP HTTP Server Handler Example of an Initial RPC call. Credentials are optional, if not present then the session will be a guest session. It takes Trust points to create a session, too many attempts will (deplete Trust) block the client.
  • 39. Example Response Message: Stored in Session: App Server Context: {RPC { Domain: "divconq.com", UDP Result: 0, // success Origin: "http:[ipaddress]", Tag: "010A0D0502", UserId: "119", Body: { Common UserName: "awhite",RPC Security UserId: "119", FirstName: "Andy", LastName: "White", TCP UserName: "awhite", Common Session Email: "andy.white@divconq.com", FirstName: "Andy", Handler Handler AuthToken: "010A0D0502", LastName: "White", Chronology: "/America/Chicago", Email: "andy.white@divconq.com", Locale: "en-US",RPC Web Credentials: { Chronology: "/America/Chicago", UserName: "nnnn", Socket Locale: "en-US", A Password: "mmmm" Session: { Web } Security Id: "nnnn", } HTTP Key: "010A0D0502" HTTP Server } Handler } } Client gets back User and Session info. Authentication Token and User info are stored in Session. Anyone with the session Id and Key can now act as this authenticated user (see guest next slide)
  • 40. Example Response Message: App ServerRPC { UDP Result: 0, // success Tag: "010A0D0502", Common Body: {RPC Security Chronology: "/America/Chicago", TCP Common Session Locale: "en-US", Handler Handler Session: { Id: "nnnn",RPC Actor: "mmmm", Web Key: "010A0D0502" Socket } Web A } Security } HTTP HTTP Server Handler Stored in Session: Context: { Domain: "divconq.com", Origin: "http:[ipaddress]", Chronology: "/America/Chicago", Locale: "en-US" If a guest session, then all we get back is session } info. The default (guest) Chronology and Locale are returned.
  • 41. Example RPC Request Message: App Server {RPC Service: "X", UDP Feature: "Y", Common Op: "Z", Security RespondTag: "abcd",RPC Session: { TCP Common Session Id: "nnnn", Handler Handler Actor: "mmmm", Key: "010A0D0502"RPC Web }, Socket Body: { Web A ... } Security } HTTP HTTP Server Handler X User Context is restored from session field before passing the message on to service X. Service X sees a RespondTo and RespondTag corresponding to the HubId and Session_Actor. If the incoming request has a RespondTag, it is added to the new RespondTag as in Session_Actor_Tag. The RespondTag is properly restored to Tag on response.
  • 42. Example RPC Response Message: App ServerRPC { UDP Result: 0, // success Tag: "abcd", Common Body: {RPC Security ... TCP Common Session } Handler Handler }RPC Web Socket Web A Security HTTP HTTP X Server Handler Nothing special is returned. Tag is present if a RespondTag was sent. Body for the response payload. Service X does not have to know anything about the client.
  • 43. App ServerRPC UDP CommonRPC Security TCP Common Session Handler HandlerRPC Web Socket Web A Security HTTP HTTP ABC Server Handler Someone sends a message for session A. A is configured (by Common Handler) to directly route incoming messages over the full duplex network layer. Tag is adjusted to remove the session id, but otherwise the message is fairly consistent with what ABC sent.
  • 44. DivConq CTP
  • 45. HTTP Path Request Message: WS Path Request Message: Common Path Request Message:{ { { Service: "Session", Service: "Session", Service: "Session", Feature: "Manager", Feature: "Manager", Feature: "Manager", Op: "Start", Op: "Start", Op: "Start", RespondTag: "abcd", RespondTag: "abcd", RespondTag: "010A0D0502", Body: { Body: { Body: { Credentials: { Credentials: { Domain: "divconq.com", UserName: "nnnn", UserName: "nnnn", Credentials: { Password: "mmmm", Password: "mmmm", UserName: "nnnn", SslThumb: "mmmm" SslThumb: "mmmm" Password: "mmmm", } } SignedKey: "mmmm" } } }} } } }
  • 46. HTTP Path Request Message: WS Path Request Message: Common Path Request Message:{ { { Result: 0, // success Result: 0, // success Result: 0, // success Tag: "010A0D0502", Tag: "010A0D0502", Tag: "010A0D0502", Body: { Body: { Body: { UserId: "119", UserId: "119", UserId: "119", UserName: "awhite", UserName: "awhite", UserName: "awhite", FirstName: "Andy", FirstName: "Andy", FirstName: "Andy", LastName: "White", LastName: "White", LastName: "White", Email: "andy.white@divconq.com", Email: "andy.white@divconq.com", Email: "andy.white@divconq.com", Chronology: "/America/Chicago", Chronology: "/America/Chicago", Chronology: "/America/Chicago", Locale: "en-US", Locale: "en-US", Locale: "en-US", Session: { Session: { Session: { Id: "nnnn", Id: "nnnn", Id: "nnnn", Actor: "mmmm", Actor: "mmmm", Actor: "mmmm", Key: "010A0D0502" Key: "010A0D0502" Key: "010A0D0502" } } } } } }} } }Response Session Cookie:Session: "nnnn_mmmm_010A0D0502"
  • 47. HTTP Path Request Message: WS Path Request Message: Common Path Request Message:{ { { Service: "X", Service: "X", Service: "X", Feature: "Y", Feature: "Y", Feature: "Y", Op: "Z", Op: "Z", Op: "Z", RespondTag: "abcd", RespondTag: "abcd", RespondTag: "abcd", Body: { Session: { Session: { ... Id: "nnnn", Id: "nnnn", } Actor: "mmmm", Actor: "mmmm",} Key: "010A0D0502" Key: "010A0D0502" }, },Session Cookie: Body: { Body: { ... ...Session: "nnnn_mmmm_010A0D0502" } } } }
  • 48. HTTP Path Request Message: WS Path Request Message: Common Path Request Message:{ { { Result: 0, // success Result: 0, // success Result: 0, // success Tag: "abcd", Tag: "abcd", Tag: "abcd", Body: { Body: { Body: { ... ... ... } } }} } }
  • 49. DivConq RPC
  • 50. Since the “real” user context/authentication is stored in the session, and because othervital data may be stored with a session, if a session is lost then the client must start anew one.1) When client makes a call it will fail because the session id cannot be matched (or server is down).2) The client should raise an event “session lost” or “connectivity lost”. Every part of the client app that relies on session data should clear their flags/states/cache. This should not trigger new RPC’s, wait for “session connected”3) Try to reconnect and start session4) Whether 3 works or not, return error to caller because we don’t know if caller relied on session info5) If 3 worked then trigger “session connected” event.
  • 51. DivConq CTP
  • 52. Client App using API App Server App Service Web App RPC A Upload Connector Session DownloadOne Connector represents one user (set of credentials), one or more connectionsand one session on server. Connections pooled by browser, just simpleSendWait or SendForget support. Uploads and Downloads have their own URLs.
  • 53. Client App using API App Server App Service App RPC A Connector SessionEach Connector represents one user (set of credentials), one connection and onesession on server. No connection pool, just simple Send or SendForget support.App Server may issue a “Session Moving” message to inform client to use a newserver (e.g. from a server farm).
  • 54. Client App using API App Server A B C App Sessions App Services RPC Sessions Pool App Server M N O Sessions App ServicesEach client Session represents one user(set of credentials) and one session on App Serverserver. Connections are pooled, message X Y Zfrom client session to server session mayuse any connection to that server. New Sessions App Servicesclient session may start on any connectedserver, but session stays with that server.App Server may issue a “Session Moving”message to inform client to use a newserver (e.g. from a server farm).
  • 55. DivConq CTP
  • 56. This first example (just one server) may appear overly complex until we review thesecond example (three servers). CTP is designed to work well within a distributedapplication environment where some parts may be running on separate servers. Butin this first example all parts are running on one server.
  • 57. App Server RPC A Service Session ZRPC caller sends service message to service Z asking for a download. Hub andSession Id are included as part of the request so that Z knows where to route thedownload to.
  • 58. App Server RPC A Service Session Z PlanIf Z approves the download, then it creates a Plan which will coordinate all tasksrequired for the download(s). Plan holds File Store Access Code, downloadMilestones and all details on sessions participating in download.
  • 59. App Server RPC A Service Session Z Session Plan BPlan starts a session on the File Store (same server in the example) by passingthe FS Access Code. Meanwhile the RPC client told that transfer was approvedand which Plan holds the milestones.
  • 60. App Server RPC A Service Session Z Session Plan BSession A registers with Plan, waits for Milestone 1. Meanwhile Session Bprepares the download and then registers Milestone 1. This means that theSource (B) is ready.
  • 61. App Server RPC A Service Session Z Session Plan BSession A (Destination) calls Session B (Source) requesting the next (starting withfirst) block of the file to transfer. As a block is received by A, it immediately asks forthe next block from B and pushes the current block to the client. A will never havemore than two blocks in holding – if client is slow to accept blocks then B waits untilat least one is accepted by client before proceeding to the next block from B.
  • 62. App Server RPC A Service Session Z Session Plan BB sends “final block” flag with final block of file. A passing that flag on to clientwhen possible (with HTTP download, the response is streamed (chunked) so wejust finish the response to indicate done).
  • 63. App Server A Service Session Z Session Plan BAfter client has file session A sets Milestone 2 with the Plan. Plan knows that thismeans it is done, so it kills session B and itself.
  • 64. DivConq Database
  • 65. DMZ Server DMZ Server RPC A Service Session Z LAN Firewall File Store ServerRPC caller sends service message to service Z asking for a download. Hub andSession Id are included as part of the request so that Z knows where to route thedownload to.
  • 66. DMZ Server DMZ Server RPC A Service Session Z LAN Firewall File Store Server PlanIf Z approves the download, then it creates a Plan which will coordinate all tasksrequired for the download(s). Plan holds File Store Access Code, downloadMilestones and all details on sessions participating in download.
  • 67. DMZ Server DMZ Server RPC A Service Session Z LAN Firewall File Store Server Plan Session BPlan starts a session on the File Store (same server in the example) by passingthe FS Access Code. Meanwhile the RPC client told that transfer was approvedand which Plan holds the milestones.
  • 68. DMZ Server DMZ Server RPC A Service Session Z LAN Firewall File Store Server Plan Session BSession A registers with Plan, waits for Milestone 1. Meanwhile Session Bprepares the download and then registers Milestone 1. This means that theSource (B) is ready.
  • 69. DMZ Server DMZ Server RPC A Service Session Z LAN Firewall File Store Server Plan Session BSession A (Destination) calls Session B (Source) requesting the next (starting withfirst) block of the file to transfer. As a block is received by A, it immediately asks forthe next block from B and pushes the current block to the client. A will never havemore than two blocks in holding – if client is slow to accept blocks then B waits untilat least one is accepted by client before proceeding to the next block from B.
  • 70. DMZ Server DMZ Server RPC A Service Session Z LAN Firewall File Store Server Plan Session BB sends “final block” flag with final block of file. A passing that flag on to clientwhen possible (with HTTP download, the response is streamed (chunked) so wejust finish the response to indicate done).
  • 71. DMZ Server DMZ Server RPC A Service Session Z LAN Firewall File Store Server Plan Session BAfter client has file session A sets Milestone 2 with the Plan. Plan knows that thismeans it is done, so it kills session B and itself.
  • 72. DivConq Database
  • 73. DMZ Server DMZ Server RPC A Service Session Z LAN Firewall File Store ServerClient sends service Z a message asking for an upload. Hub and Session Id areincluded as part of the request so that Z knows where to route the download to.
  • 74. DMZ Server DMZ Server RPC A Service Session Z LAN Firewall File Store Server PlanIf Z approves the upload, then it creates a Plan which will coordinate all tasksrequired for the upload. Plan holds File Store Access Code, upload Milestonesand all details on sessions participating in upload.
  • 75. DMZ Server DMZ Server RPC A Service Session Z LAN Firewall File Store Server Plan Session BPlan starts a session on the File Store (same server in the example) by passingthe FS Access Code. Meanwhile the client is told that transfer was approved andwhich Plan holds the milestones.
  • 76. DMZ Server DMZ Server RPC A Service Session Z LAN Firewall File Store Server Plan Session BSession A registers with Plan, sets Milestone 1. This means that the Source (A) isready. Meanwhile Session B prepares for the upload, registers with Plan andthen waits for Milestone 1.
  • 77. DMZ Server DMZ Server RPC A Service Session Z LAN Firewall File Store Server Plan Session BSession B (Destination) calls Session A (Source) requesting the next (starting withfirst) block of the file to transfer. As a block is received by B, it immediately asks forthe next block from S and writes the current block to the store. B will never havemore than two blocks in holding – if the store is slow to accept blocks then B waitsuntil at least one is written before proceeding to the next block from A.
  • 78. DMZ Server DMZ Server RPC A Service Session Z LAN Firewall File Store Server Plan Session BA sends “final block” flag with final block of file. B closes the file after final write.
  • 79. DMZ Server DMZ Server RPC A Service Session Z LAN Firewall File Store Server Plan Session BAfter client has finished upload file session A sets Milestone 2 with the Plan. Planknows that this means it is done, so it kills session B and itself.
  • 80. DivConq CTP
  • 81. Client App App Server A M X FT App Collab Session File Store RPC A M X Sessions PoolTo speed up a large file upload, use “Parts”. In this case we’ll transfer three parts at thesame time.To transfer parts the requests go to the same server (or same squad/farm) for eachsession. Only one file name is given for upload, each session transfers a different set ofblocks for the file.
  • 82. DivConq CTP
  • 83. Client App App Server A FT App Collab Session File Store RPC A M X Sessions Pool App Server M Collab Session File StoreEach client Session represents onesession on a different server (squad/farm). App ServerOne file name is given. Partial files areverified independently – client knows that X Collaball of part A is intact, all of part M is intact Session File Storeand that all of part X is intact. Also knowsthat the offsets are correct. It all adds upto one complete file even if that file is inmultiple geographies.
  • 84. Common Transfer Protocol can meet its goals.• A complete spec for messaging commands as well as file transfers (SIGN ON, DIR, GET, PUT)• It is abstracted from network layer, could be implemented in TCP, UDP or through either HTTP or Web Socket• It does not assume full duplex communications• It does not (necessarily) associate network layer client credentials with the file transfer, which can lead to scalability and efficiency issues• It is platform and language neutral • Server could be implemented in Java, C, Python or similarly suitable languages • Client can be implemented in any language with TCP or HTTP capabilities• It is open and free

×