Do you have trouble recruiting and retaining good gateway developers? Common challenges include the allure of the cutting-edge commercial sector, academic pay scales, and the specialized knowledge necessary for development. The Indiana University Science Gateways Research Center is attempting to turn this challenge into an opportunity by teaching Web-based cyberinfrastructure systems using concepts and technologies that will benefit students in their non-academic careers. Such distributed systems require knowledge of both classic and cutting-edge topics, such as microservices and “DevOps” practices. Meanwhile, the instructors can identify and connect with promising students while keeping themselves aware of important trends and technologies. This webinar will provide an overview of the material and concepts that we cover in the class, describe the student project assignments and class organization, and present outcomes and student feedback that we have gotten over the last two semesters. We will also discuss possibilities of making the course material and instruction available to other interested universities. Course information and material is available from http://courses.airavata.org/.

1. Welcome to the SGCI Webinar!
• We will be starting shortly.
• Your audio has been muted, and you are
encouraged to turn off your video during the
presentation.
• Controls for these are near the bottom of
the right-side control panel for BlueJeans.
• You may submit questions at any time
using Chat, and the moderator will share
them with the presenter when appropriate.
• This presentation will be recorded and slides will be
posted.
1

2. Creating a developer pipeline by
teaching gateway technologies
Marlon Pierce, Suresh Marru
Science Gateways Research Center, Indiana University
marpierc@iu.edu, smarru@iu.edu

3. http://sgrc.iu.edu

4. The Pipeline Problem
Finding and retaining people to work on science
gateways is tough.

6. If only there was an untapped reservoir of talent close by....

8. The Pipeline Solution
Recruit students to work with us by teaching a graduate level
class on the underpinnings of gateways.
Embrace change.

9. What Is a Science Gateway?
A Science Gateway is a user-centric environment
for conducting online computational science

10. (Not actual IU students)

11. What if Mark Zuckerberg Read More
Chemistry and Biology Books?

13. Goal 1: Apply basic distributed
computing concepts to Science
Gateways.

14. Goal 2: Apply new architectures,
methodologies, and technologies to
Science Gateways: Microservices, DevOps

15. Browser
Web Interface Server
Application Server
Server SDK
Client SDK
IU: Big Red 2
Resource Plugins
XSEDE:
Stampede
XSEDE: Comet Juelich: Jureca
HTTPS
HTTP or TCP/IP
Facebook
doesn’t look
like this, so
why should
science
gateways?

16. Goal 3: Teach open source software
practices

17. Challenges for Science Gateways
• Providing a rich user experience
• Defining an API for the application server
• Defining the right microservices.
• Implementing the components, wiring them together correctly.
• Supporting multiple gateway tenants
• Fault tolerance for components
• State management
• Continuous delivery
• Security management
• Supporting full scientific exploratory cycle

19. What Do We Teach?
A quick survey of course material

20. Micro-
Services
DevOps
Distributed
Systems

21. Putting It All Together: Gateways and
Microservices

22. Virtualization, Containers, Docker

23. API and Metadata Model Design

24. Messaging Systems: RabbitMQ, Apache Kafka

25. Resource Management and Scheduling: Mesos,
Aurora, and Torque

26. Distributed State Management: Zookeeper,
Consul

27. Security: OAuth2 and OpenIDConnect

28. Putting It All In
Place:
Continuous
Integration and
Deployment

30. Mission Accomplished!
Students learned on
their own.

35. More Information and Next Steps
• http://courses.airavata.org
• Additional material in Canvas
• Details on assignments
• We are interested in taking this to a broader audience

36. Thank you for participating!
• Please offer your feedback with our 30-second evaluation:
http://sciencegateways.org/webinareval
• Join us next month (March 8) for
Building a Modern Research Data Portal with Globus –
Introduction to the Globus Platform
36

37. Science!

38. Spring 2016 Course Overview
• Students organized into teams of 3-4
• 8 biweekly project milestones, each depended on the previous
assignment.
• Basic sequence, Part 1:
• Learn to submit jobs on IU’s Karst Cluster, MOAB scheduler
• Develop core job management application
• Design an API
• Design a Web front end
• Capture job execution metadata
• All code must be in GitHub, must build cleanly on the grader’s laptop

39. Spring 2016 Project Milestones, Part 2
• Integrate with a continuous integration and deployment system
• Travis-CI and Amazon Code-Deploy
• Apache Jenkins on Jetstream
• Break up your monolithic Web applications into microservices
• Combine with CI/CD

40. Spring 2016: What Worked, What Didn’t?
• Small class, we got to know the
students pretty well.
• Students managed to build basic
functioning gateways that used
CI/CD automatically linked to
GitHub accounts.
• Students got bogged down in job
submission issues.
• Students had trouble breaking
up their monolithic Web
applications into microservices.

41. Fall 2016: Begin at the End
• For Fall 2016, we reverse the order.
• Students learned CI/CD and microservices in the first assignments.
• We moved aggressively to problems of load balancing, fault tolerance,
and containerization of services.
• We de-emphasized job submission to traditional clusters
• Used Apache Mesos and Apache Aurora for this.

42. Fall 2016: What Worked, What Didn’t
• All the students were able to
keep up with the assignments.
• Many were using technologies for
the first time.
• Some surprises
• Some teams took it as a challenge
to use different approaches.
• Didn’t have the chance to get to
know students as well as before.
• Larger class, graders
• Monthly assignments were too
large.
• Biweekly worked better.

43. Spring 2017: Advanced Topics
• In the Spring 2016 and Fall 2016 courses, we knew the answers
(mostly).
• Now in Spring 2017 course, we tackle problems we don’t know the
answers to.
• Or, if you prefer, for a given problem,
• Introductory Class: Find an answer that works
• Advanced Class: Find the best answer
• How do you do that?