Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Building an Academic Program Database and API with Contentful and Amazon Web Services

263 views

Published on

How many degree listings does your institution’s website have? How robust is that information? How consistent and on-brand is it? The amount of information related to academic programs is vast and varied. Tuition, scholarships, plans of study, facilities, profiles, media and more. Having clear and consistent academic information would be a differentiator for many schools. A single source-of-truth for academic content might be the holy grail.

This presentation shares how West Virginia University has started to tackle this problem. Their Academic Programs API combines Contentful, a headless CMS, with Amazon Web Services. This has led to a flexible, easy-to-update system for authors, developers and designers.

In this session, you’ll learn how to:

* Work with content owners to show them the importance of centralized content and how to source it
* Define content models and relationships in Contentful
* Use AWS’s Lambda, DynamoDB and API Gateway services to build an API
* Expand your efforts beyond academic information
* Take control of your institution’s content

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Building an Academic Program Database and API with Contentful and Amazon Web Services

  1. 1. BUILDING AN ACADEMIC PROGRAM DATABASE AND API Dave Olsen / @dmolsen West Virginia University
  2. 2. OUR STORY BEGINS IN 2015
  3. 3. GUIDING PRINCIPLE ACADEMICS COST = VALUE
  4. 4. IMPORTANCE OF ACADEMICS of our admits ranked academic reputation as “very important” when selecting a school. 72% of enrolling students rated us as “very good” or “excellent” for academic reputation. 75% of non-enrolling students described WVU as having a good academic reputation. 24% vs.
  5. 5. GUIDING PRINCIPLE ACADEMICS COST = VALUE
  6. 6. PROGRAM PAGE EXAMPLE
  7. 7. ORIGINAL PROGRAM DB
  8. 8. SUMMER OF 2016
  9. 9. LESSONS LEARNED: 3RD PARTY CONTENT TOOL The bad: • Could not build DRY content. No content relationships. • API very limited. No search or preview. The good: • Content modeling was the right direction. 👌 • Having a 3rd party host and maintain an editing interface was awesome. 👍
  10. 10. LESSONS LEARNED: UNIVERSITY CONTENT The bad: • No one could agree on how many programs we had or their names. # • No data to fill in blanks we had (e.g. outcomes) • Program search not robust enough. The good: • So much content related back to programs. We had an opportunity.
  11. 11. SPOILER ALERT • Still no agreement on how many programs we have or their names. # # # • Still no true outcome data.
  12. 12. OUR PLAN OF ACTION • Gather a team • Develop goals and requirements • Evaluate 3rd party and technical options • Source content from domain experts • Build technical solution and design new program pages • Maintain!
  13. 13. Team: The Duo (actual team headshot) The writer and organizer Sarah Gould Developer and ne’er-do-well Dave Olsen
  14. 14. Team: The “Creatives” (actual team headshot) Front end dev #1 Adam Johnson Front end dev #3 Dustin Mazon Front end dev #2 Adam Glenn
  15. 15. Team: Supporting Cast (actual team headshot)
  16. 16. Team: The Champion (actual champion headshot) Project Champion
  17. 17. PRO TIP: SOURCING ACADEMIC CONTENT REQUIRES ACADEMICS It was nearly impossible to source content with communicators and recruiters alone. Get someone from Provost Office or similar on the team.
  18. 18. OUR GOALS • Standardize content. • Highlight our value proposition. • Address full life-cycle of a student. • Optimize for search engines. • Deploy accurate content quickly. • Expand to all three regional campuses.
  19. 19. OPTIMAL FINAL OUTCOME Course catalog SSC data O*NET career data Program pages Major Maps Other campus outlets via API Share what we’ve collected Consume content from others Make connections Curriculum matrix Admissions Registrar CLASS Programs Database
  20. 20. WHAT WASN’T A GOAL? We didn’t want this to be a primary source of truth for academic content for internal stakeholders. That is asking for trouble.
  21. 21. OUR REQUIREMENTS • Build glue, not CRUD. • 3rd party • Robust API system • Deep content relationships (COPE). • Ability to preview changes. • Improved program search.
  22. 22. OUR PLAN OF ACTION • Gather a team • Develop goals and requirements • Evaluate 3rd party and technical options • Source content from domain experts • Build technical solution and design new program pages • Maintain!
  23. 23. ENTER CONTENTFUL
  24. 24. CONTENTFUL: THE GOOD • APIs for images and content preview, delivery and management plus webhook support. • Click-and-edit content modeling. • Rich, pre-built field types. • Can also create UI Extensions to extend their interface. • Media management with generous storage. • Multi-language using “locales.” • Free to try. 😎
  25. 25. CONTENTFUL: BUILDING A CONTENT MODEL
  26. 26. CONTENTFUL: THE BAD • Need to use their SDKs to access API and data. • No easy way to limit how much data was returned when using API. Objects > 7MB. • 50 max fields per content type. • Rate limiting when posting and getting data. • No XML option. • API calls were limited. 💵
  27. 27. COMING SOON: GRAPHQL
  28. 28. MEASURING UP Build glue, not CRUD. 3rd party Robust API system Deep content relationships (COPE). Ability to preview changes. Improved program search.
  29. 29. Hmm, Contentful… •Is great for content organization! 😁 •Has a high bar for content re-use by myself and other campus developers. 😞
  30. 30. ENTER AWS LAMBDA
  31. 31. ENTER AWS DYNAMODB
  32. 32. ENTER AWS CLOUDSEARCH
  33. 33. ENTER AWS API GATEWAY
  34. 34. ENTER AWS SNS
  35. 35. AWS SERVICES: THE GOOD • It’s all glue: • No hardware to manage. “Serverless.” 🎉 • Events/SNS - easily stitch products together. • Testable • Define my own API and become dumb data endpoint. • Click-and-edit configurations. • Could implement Contentful workarounds: • “Reverse GraphQL” and “post-process” rules to optimize content for outlets. Allows for inheritance, key building and richer objects. • No rate limiting and layer of resiliency. • XML and JSON
  36. 36. AWS SERVICES: THE GOOD “Reverse GraphQL” “Post-process”
  37. 37. AWS SERVICES: THE GOOD “Reverse GraphQL” “Post-process”
  38. 38. AWS SERVICES: THE BAD • Actual content is further away from final outlet. Sometimes need to force pull data to update. • Need to update “reverse GraphQL” configurations when content models change. • DynamoDB doesn’t like large objects when scanning. Have had to develop *Thin tables. • Supported programming languages are limited.
  39. 39. PRO TIP: EMBRACE NODE.JS
  40. 40. PRO TIP: BETA/STABLE • Set-up BETA alias in Lambda to point at $LATEST version. • Use versions in Lambda to publish stable code. • Set-up STABLE alias to point at latest stable version. • Add stage variables to the targeted Lambda function in API Gateway. • Extend to DynamoDB for beta and stable datasets.
  41. 41. OUR PLAN OF ACTION • Gather a team • Develop goals and requirements • Evaluate 3rd party and technical options • Source content from domain experts • Build technical solution and design new program pages • Maintain!
  42. 42. • Read every major’s entry in the course catalog. Twice. Take notes. • Design or content first? 🐔 or 🥚? • Use meetings and workshops over worksheets. A traveling circus. 🎪 • Identify owners of certain types of content (e.g. tuition and requirements). Always defer to them and their processes. LESSONS LEARNED: SOURCING CONTENT
  43. 43. • Automate the import of content whenever possible. • Find proxies for missing content. Our career outcomes come from O*NET. • Find experts to review the proxies. Our Career Services Office chose the careers for each major. • Prep for naming complaints and groups wanting to “opt-out.” This won’t happen until you’re done and reality hits. LESSONS LEARNED: SOURCING CONTENT
  44. 44. OUR PLAN OF ACTION • Gather a team • Develop goals and requirements • Evaluate 3rd party and technical options • Source content from domain experts • Build technical solution and design new program pages • Maintain!
  45. 45. CONTENTFUL CONTENT MODELS An early attempt to document content models.
  46. 46. • Admission Requirement Rules • Announcements • Areas of Emphasis • Building Types • Buildings • Campuses • Capstone Projects • Career Abilities • Career Interests • Career Knowledge • Careers • College and Schools • Companies • Cost of Living Rules CONTENTFUL CONTENT MODELS 68Content Models
  47. 47. AWS ARCHITECTURE Poor diagram of the system architecture.
  48. 48. AWS ARCHITECTURE: INPUT Store Data Lambda, Events, DynamoDB and CloudSearch Post-Process Lambda Validate Lambda and SNS Invoke Contentful
  49. 49. AWS ARCHITECTURE: INPUT Store Data Lambda, Events, DynamoDB and CloudSearch Validate Lambda and SNS Invoke Contentful Post-Process Lambda
  50. 50. AWS ARCHITECTURE: INPUT Post-Process Lambda Validate Lambda and SNS Invoke Contentful Store Data Lambda, Events, DynamoDB and CloudSearch
  51. 51. AWS ARCHITECTURE: INPUT Validate Lambda and SNS Invoke Contentful Post-Process Lambda Store Data Lambda, Events, DynamoDB and CloudSearch
  52. 52. AWS ARCHITECTURE: INPUT Store Data Lambda, Events, DynamoDB and CloudSearch Validate Lambda and SNS Invoke Contentful Post-Process Lambda
  53. 53. AWS ARCHITECTURE: OUTPUT Request and Render PHP, JavaScript and Ruby Route, Transform and Deliver API Gateway Validate and Process Lambda Fetch Data DynamoDB and CloudSearch
  54. 54. AWS ARCHITECTURE: OUTPUT Request and Render PHP, JavaScript and Ruby Route, Transform and Deliver API Gateway Validate and Process Lambda Fetch Data DynamoDB and CloudSearch
  55. 55. AWS ARCHITECTURE: OUTPUT Request and Render PHP, JavaScript and Ruby Route, Transform and Deliver API Gateway Validate and Process Lambda Fetch Data DynamoDB and CloudSearch
  56. 56. AWS ARCHITECTURE: OUTPUT Request and Render PHP, JavaScript and Ruby Route, Transform and Deliver API Gateway Validate and Process Lambda Fetch Data DynamoDB and CloudSearch
  57. 57. AWS ARCHITECTURE: OUTPUT Request and Render PHP, JavaScript and Ruby Route, Transform and Deliver API Gateway Validate and Process Lambda Fetch Data DynamoDB and CloudSearch
  58. 58. AWS ARCHITECTURE: OUTPUT Request and Render PHP, JavaScript and Ruby Route, Transform and Deliver API Gateway Validate and Process Lambda Fetch Data DynamoDB and CloudSearch
  59. 59. API ENDPOINTS Scan: Search: By key: Endpoints support query string options like “format,” “sortby” or “groupby.” The {contentType} path variable gets matched to a config by the Lambda function. {contentKey} is used in the DynamoDB look-up. /{contentType}?options /{contentType}/search?options /{contentType}/{contentKey}?options
  60. 60. FINISHED PRODUCT: API CONTENT
  61. 61. FINISHED PRODUCT: PROGRAM PAGES go.wvu.edu/program-page
  62. 62. FINISHED PRODUCT: ENGAGEMENT? Heat map Scroll tracking
  63. 63. FINISHED PRODUCT: MAJOR MAPS go.wvu.edu/major-map
  64. 64. FINISHED PRODUCT: CAREER PATHWAYS
  65. 65. FINISHED PRODUCT: SCHOLARSHIP CALCULATOR
  66. 66. FINISHED PRODUCT: LANDING PAGE FEATURES
  67. 67. OUR PLAN OF ACTION • Gather a team • Develop goals and requirements • Evaluate 3rd party and technical options • Source content from domain experts • Build technical solution and design new program pages • Maintain!
  68. 68. Maintaining the Database • Contact primary sources of central data and get updates. • Three times a year ask recruiters and communicators to update content. • Give them access to an internal website and Word docs with content. • Almost zero feedback. 😢
  69. 69. REVIEW OUR PLAN OF ACTION • Gather a team • Develop goals and requirements • Evaluate 3rd party and technical options • Source content from domain experts • Build technical solution and design new program pages • Maintain!
  70. 70. REVIEW OUR GOALS Standardize content. Highlight our value proposition. Address full life-cycle of a student. Optimize for search engines. Deploy accurate content quickly. Expand to all three regional campuses.
  71. 71. Hero Block Model & Pattern Quicklinks Block Model & Pattern Profile Block Model & Pattern OUR FUTURE: CONTENT MODELS + LAYOUT MODELS
  72. 72. OUR FUTURE: CONTENT MODELS + LAYOUT MODELS Program Title Degree Designation Course Delivery Option Advisement Sheets Areas of Emphasis … more … Name Background Image Description Program Interests … more … Blocks - Student 🎩 Content Model Layout Model Post-Process Magic Design System Pattern/Layout - Student
  73. 73. OUR FUTURE: CONTENT MODELS + LAYOUT MODELS power.wvu.edu wisdom.wvu.edu courage.wvu.edu Static Sites Design system-based static site generator “Reverse GraphQL” / post-process Contentful DynamoDB Build Pipeline
  74. 74. OUR FUTURE: CONTENT MODELS + LAYOUT MODELS
  75. 75. THANK YOU / QUESTIONS? Dave Olsen / david.olsen@mail.wvu.edu

×