Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

PyClientServer: a Python powererd Client-Server Architecture over a compressed JSON/TCP channel


Published on

This presentation aims at introducing the interested reader to the very important topics of (i) Data Compression and (ii) Lightweight Network Communication in SoSE (Systems of Systems Engineering). These very interesting topics in the SOA context, in particular for data-centric application integrations. A working implementation is provided in Python, a very powerful main stream programming language.

Check the project out from GitHub at

Published in: Technology, Education
  • Be the first to comment

PyClientServer: a Python powererd Client-Server Architecture over a compressed JSON/TCP channel

  1. 1. PYClientServer Python powered Client-Server architecture on compressed JSON channel
  2. 2. Outline ❖ Intro ❖ Requirements ❖ Architecture ❖ Implementation ❖ Tests ❖ Conclusion
  3. 3. Introduction - 1 of 2 ❏ A need: seamless integration using a lightweight data exchange format ❏ JSON is emerging as the modern de-facto standard data format for services integration ❏ TCP is a must, it is reliable and can be secure ❏ JSON over TCP is a trustable solution ❏ Why compression? JSON strings are naturally verbose, but less than XML, so ❏ compression can help in reducing the footprint ❏ bandwidth utilisation can be effectively improved ❏ RTT latency benefits from (much more responsiveness)
  4. 4. Introduction - 2 of 2 ❏ And, Python? Well, it is an agile main stream programming language ❏ very well suited for network communication ❏ very well supported ❏ platform-independent (so, portable solutions) ❏ powerful in string/data processing ❏ ... ❏ PYClientServer represents an attempt to implement a working example of the previously outlined needs
  5. 5. Requirements ❏ Let’s go through the needs introduced a. services integration over the network b. seamless lightweight communication c. effective bandwidth utilisation d. easy extensibility e. rapid development f. working code to extend for enhancements ❏ Let’s summarize as follows a. A Server offering a data-centric service b. A Client asking for the service c. A flexible Data Model offering ❏ marshalling/unmarshalling facility ❏ compression/decompression facility
  6. 6. Architecture Client Server TCP Channel Decoder Encoder Process#1 Thread#1 Process#2 Thread#1 File System 1. JSON request 2. File Reading 3. Text compression 4. Server response 5. Decompressed Text 0101010101 { GET, 100 }
  7. 7. Implementation - 1 of 5 ❏ Key implementation aspects concerning the Server a. it takes advantage of Python’s powerful network library: 1 server handler in few LoC b. it receives text-based request and replies providing compressed data ❏ Let’s have a look at the code class TCPRequestHandler(SocketServer.BaseRequestHandler): … try: print "[TCPRequestHandler][handle] Connection accepted... processing" # Reading request (assuming a small amount of bytes) data = self.request.recv(1024).strip() # Unmarshall the request request = Request('', 0) data = request.from_json(data) …
  8. 8. Implementation - 2 of 5 # Read lines from a text file resource = [] resource.append(self.__resourcepath) resource.append(self.__filename) testfile = open(''.join(resource), 'r') # Prepare the response data response = Data(True, [], 0) list = testfile.readlines() response.vector = list response.nrbytes = len(list) # Marshall JSON representation json_str = response.to_json() c_response = self.__compression.compress(json_str) start = time.time() self.request.sendall(c_response) print "[TCPRequestHandler][handle] Bunch of compressed data sent back!" end = time.time() if (self.__isdebug): print "Delivery::Time Elapsed::", str(end - start)
  9. 9. Implementation - 3 of 5 ❏ As already seen in the Server code, system Data Model is very important, because a. it allows to seamlessly marshall/unmarshall the data b. it allows to perform compression/decompression, according to the communication protocol class Request(object): def __init__(self, name, code): = name self.code = code def to_json(self): self.__dict__['name'] = self.__dict__['code'] = self.code return json.dumps(self.__dict__, indent=4) def from_json(self, sting): dict = json.loads(sting) self.__dict__ = dict
  10. 10. Implementation - 4 of 5 class Utility(object): def compress(self, data): string = str(data) return zlib.compress(string, zlib.Z_BEST_COMPRESSION) def decompress(self, data): string = str(zlib.decompress(data)) return string ❏ Till now, Server and Data Model have been briefly discussed, but those are only simple snippets of code ❏ To have a look at the full code of Server and Data Model, check the project out on GitHub.
  11. 11. Implementation - 5 of 5 ❏ Client thread is the most simple entity involved in the scenario sketched on the architecture, in fact, it acts as follows ❏ send a pure JSON request (Data Model aids in doing that) ❏ upon a reception of server data, it performs the decompression as dictated by the defined communication protocol # Request creation data = Request('GET', 100) # Client socket binding sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) sock.connect((self.__serverhost, self.__serverport)) # Sending JSON data over the socket sock.send(data.to_json()) response = self.__receive_data(sock) # Treating compressed data result = self.__compression.decompress(response) data = Data(False, [], 0) data.from_json(result)
  12. 12. Tests ❏ Hereafter some screenshots captured during a session test are proposed Server-side: it is important to consider how the compression reduces drastically the dimensions (form 4.2MB to 1MB) Client-Side: in less than 1 second, the data are received. Afterwards, data are decompressed and the first line of the document is disaplyed
  13. 13. Conclusion - 1 of 2 ❏ Architecting the communication protocol for the service integration as shown in the architectural sketch, brings several advantages a. effective bandwidth utilization: the compression reduces the verbosity b. much more responsiveness: the entire data frame is trasferred in less than a second ❏ Python network programming is really powerful: with few lines of code, the application programmer can define and deploy a working multithreaded service ❏ JSON format is really amazing to work with in Python: as shown, with few lines of code, the library makes the serialization and deserialization a pretty easy job
  14. 14. ❏ Despite the simplified snippets presented, in the previous slides ❏ Client, Server and Data Model have been presented as the three main entities involved by design, but some other interesting features have to be discovered ❏ System Architecture has been depicted and, as seen, it is simple but powerful at the same time: there is the reason to believe that that Architecture might scale very well ❏ If you want to contribute or to have a deeper look at the code, do not hesitate to check the project out from GitHub. Conclusion - 2 of 2