Eudoxus Technology & Architecture - JHUG May 2011 Meeting


Published on

1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Eudoxus Technology & Architecture - JHUG May 2011 Meeting

  1. 1. Eudoxus.grTechnology & Architecture   Fotis Stamatelopoulos (@fstama)     JHUG Meeting @CollabAthens May 28, 2011
  2. 2. What is Eudoxus - Disclaimer • A distributed system that supports and streamlines processes and operations related to the textbooks distribution for the higher education institutions of Greece  • Stakeholder: Hellenic Ministry of Education  • Developed, operated and supported by GRNET   • Users: Students, Secretariats, Publishers, Distribution Points (e.g. Bookstores), related Ministry depts, Courier service, Helpdesk- More than 270.000 individual users • Multiple APIs and integration with other ISs
  3. 3. Presentation Focus • Design and software architecture o by design based on 100% FOSS infrastructure o full JEE stack, rich web clients (GWT/Java) o scalability, performance and bottlenecks  • Building within a very tight schedule and ever changing requirements o agile methodology o human environment o “collateral damage”
  4. 4. Design Goals - Challenges• Tight schedule, fixed milestones - deadlines o Incrementally release modules in production• Not finalized requirements o Be flexible, handle changing requirements• High availability o Redundant architecture o Live application updates – no downtime
  5. 5. Design Goals – Challenges (contd)• Impossible to predict usage load – design for scalability, – elastically use infrastructure to serve growing usage, – prepare for the worst – efficient use of hardware infrastructure• Safeguard the transactional nature of related processes• Synchronize data and connect to other systems
  6. 6. Technology decisions• Web-based user interfaces• Support various APIs • RESTFul APIs (JSON / XML payload) • use SOAP-based web services for integration in specific cases (e.g. courier service)• Java-based core (business logic tier)• Javascript-based / GWT-based (Java again!) rich GUIs
  7. 7. Technology decisions (contd)• Back-end storage: RDBMS vs noSQL decision o noSQL options offer better scalability than typical RDBMs o needed at least a minimal transactional core - eventual consistency is not acceptable o dropped the initial hybrid approach - full RDBMS design• Use caching as much as possible• Authentication / Credentials • Stateless mechanism (killed JAAS) • Support both – Shibboleth-based authentication for students – form based login (other user classes)
  8. 8. High level architecture
  9. 9. Infrastructure Software• Google Web Toolkit (GWT) for the rich GUI clients (browser)• Apache, nginx front-ends• Varnish web cache• JBoss Application Server o stateless EJBs implementing core business logic o JPA ORM (hibernate substrate) o JMS Queues and Message Driven Beans o JMX MBeans for scheduled tasks and maintanance• PostgreSQL RDBMS• Solr (Lucene server) indexing / search server
  10. 10. Technical Architecture search.,
  11. 11. Losing the HTTP Session• No HTTP session => workers dont need to replicate – synchronize• Application state is kept in the client• Authentication generates a short-lived auth token for the specific user & session• Token is kept in a cookie and send back with every request (over SSL), be it a REST or GWT-remoting call – In GSS used the token to encrypt the header• Completely stateless middle tier
  12. 12. The worker pool
  13. 13. The worker pool (current)
  14. 14. The worker• Stateless EJBs implement business functions• POJOs (JPA) for the data model• Mbeans for management tasks (no UI – access via the jmx-console)• Persistent JMS queues (hornetQ) & MDBs for syncing with other systems, serializing reports, handling file indexing (Solr)• REST API implementaiton will move to JAX-RS (Jersey)
  15. 15. The data store• Single RDBMS (postgresql) instance with fallback• “Plan B” alternatives – Cold replicas for reports – Sharding / hot replication (one writer / multiple readers) – Moving parts of the data to NoSQL or even use Solr as a NoSQL data store• Currently we handle the load without sweating
  16. 16. More on caching • Varnish caches specific HTML output and REST responses for URLs that do not change (Eudoxus controls this via http headers) • 2nd level caching - EHCache (per worker) • We will be introducing distributed read-write second level caching (working with Infinispan right now)
  17. 17. Meeting Tight Schedules• Typical setup o VCS (mercurial), wiki for requirements, issue tracker o dedicated release engineer, dedicated DBA o local development -> test server -> production server o FOSS dev tools (Eclipse, ant, etc)• Agile approach• Documentation was the first victim of the pressing schedule :( “updated unit tests” was the second :o• A few statistics on the team o 5 months from scratch to production o ~10 developers o ~10 testers (functional testing) o ~20 helpdesk members for end-user support o Full GRNET NOC support
  18. 18. @fstama