How I make a podcast website using serverless technology in 2023
•
0 likes•68 views
In 2022, I launched a Kotlin podcast show, Kotlin Fireside Chat, with my friends in Kotlin User Group. Now, we're taking things to the next level by building a podcast website with a statistical analysis dashboard. We've built a static website using Jamstack and connected it to a serverless API written in Kotlin. In order to aggregate the listener number across platforms, such as YouTube, BiliBili, Ximalaya, Lizhi, Qingting, we've also created a crawler to grab those numbers and display them in a custom dashboard. Everything is made by open source technology and deployed on Google Cloud Platform. In this talk, I'll share my experience and the lessons I learned when using all the technologies in 2023.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to How I make a podcast website using serverless technology in 2023
Similar to How I make a podcast website using serverless technology in 2023 (20)
More from Shengyou Fan
More from Shengyou Fan (20)
Recently uploaded
Recently uploaded (20)
How I make a podcast website using serverless technology in 2023
- 1. How I make a podcast website Photo by Michal Czyz on Unsplash using serverless technology in 2023 Shengyou Fan (范聖佑) HKOSCon 2023 2023/06/10
- 2. Story
- 7. Kotlin Advocacy pt.5 — 18 KUGs in Greater China region
- 11. An evolution process of the show — • Make a logo, pick a theme music and start! • Prepare a microphone for better sound quality • Start being picky with background and lighting • Put more efforts on post editing and design • Working on SPECIAL ROADSIDE episode recently
- 12. What’s next?
- 13. Things we wanna evolve — Website Statistics
- 15. TL;DR - Report —
- 16. Overall structure — • Website - a bunch of static webpages • Data API - provide content to the website • Crawler - grab number from different platforms • Report - give an image of the trend
- 17. Technology selection — • Website - Jamstack approach with Astro + Cloud Storage • Data API - Cloud Functions with Kotlin • Crawler - jsoup with Kotlin + Cloud Functions & Scheduler • Report - Spring Boot with Kotlin + Vaadin Chart
- 19. Build a website using Astro
- 20. Getting started with Astro — $ npm create astro@latest public/ robots.txt favicon.svg src/ ... astro.config.mjs package.json ... tsconfig.json # Initialize an Astro project $ npm run dev # Live preview Astro project
- 21. The src folder — src/ Header.astro components/ layouts/ pages/ scripts/ styles/ MainLayout.astro index.astro Reusable units of code for your HTML page " A special kind of component that wrap some content in a larger page layout " A special kind of component used to create new pages on your site " Store your JS or TS files " Store your CSS or Sass files "
- 22. Create MainLayout — --- export interface Props { title: string; } const {title} = Astro.props; --- <!doctype html> <html> <head> <meta charset="utf-8"> <title>{title}</title> </head> <body> <slot/> </body> </html> pass page-specific values as props # Scripts for this component # HTML content for a layout " " use a slot element to place page contents within a layout
- 23. Use layout on a page — --- import MainLayout from "../layouts/MainLayout.astro"; --- <MainLayout title="⾸⾴"> <main id="content" class="padding-top-bottom"> <div class="container"> </div> </main> </MainLayout> <style> </style> # Import the component # Use MainLayout component # Apply style only to this page
- 24. Make reusable components — --- --- <header id="top" class="navbar navbar-sticky"> // ... </header> --- import Header from "../components/Header.astro"; --- <body> <TopHeader!" <slot !" !#body> # Header component # Any other components " put component in a page " import
- 25. Make a home using Cloud Storage — • Create a Bucket using domain name on Cloud Storage • Verify domain name ownership via Google Search Console • Point the domain name to Cloud Storage • Make Bucket accessible publicly • Specifies the object name for default page • Specifies the error page (404) to serve
- 26. CI/CD using Cloud Build — steps: - id: 'Install dependencies' name: node entrypoint: npm args: ['install'] - id: 'Build' name: node entrypoint: npm args: ['run', 'build'] - id: 'Deploy to Cloud Storage' name: 'gcr.io/google.com/cloudsdktool/cloud-sdk' args: ['gsutil', '-m', 'rsync', '-R', 'dist', 'gs://...'] # Install node modules # Generate static files # Deploy using Cloud SDK
- 27. Build & deploy —
- 28. Build an API using Cloud Function
- 29. <dependencies> <dependency> <groupId>com.google.cloud.functions</groupId> <artifactId>functions-framework-api</artifactId> <version>...</version> <scope>provided</scope> </dependency> </dependencies> <plugins> <plugin> <groupId>com.google.cloud.functions</groupId> <artifactId>function-maven-plugin</artifactId> <version>...</version> <configuration> <functionTarget>...</functionTarget> </configuration> </plugin> </plugins> Getting started with Cloud Functions — # Add Functions Framework # Plugin for extra tasks
- 30. Data Class — @Serializable data class Episode( val id: Int, val slug: String, val title: String, val abstract: String, val content: String, val timeline: String, val cover: String, val permalink: String, val publishedAt: String, )
- 31. class Handler: HttpFunction { @Throws(IOException::class) override fun service(req: HttpRequest, res: HttpResponse) { // ... } } HTTP Function — # Extend from HttpFunction # Implement service()
- 32. class Handler: HttpFunction { @Throws(IOException::class) override fun service(req: HttpRequest, res: HttpResponse) { // ... val data = Json.encodeToString( mapOf("data" to episodes) ) with(res) { setStatusCode(HttpURLConnection.HTTP_OK) setContentType("application/json") writer.write(data) } } } JSON response — # Setup response details # Serialization collection
- 33. CI/CD using Cloud Build — steps: - id: 'Build' name: 'gcr.io/cloud-builders/mvn' args: - 'clean' - 'verify' - id: 'Deploy' name: 'gcr.io/cloud-builders/gcloud' args: - 'functions' - 'deploy' - '...' - '--trigger-http' - '--allow-unauthenticated' - '--region' - '...' - '--gen2' - '--runtime' - 'java17' - '--memory' - '...MB' - '--entry-point' - '...' # Build project using maven # Deploy using gcloud Choose memory size " Specify the entry class name " Choose JVM runtime " Use 2nd Gen Functions " Choose region " Open for HTTP without auth "
- 35. Back to frontend
- 36. Loading data in Astro — --- let response = await fetch("...") let episodes = await response.json(); --- <div class="episodes-listing"> { episodes.map((episode) => ( <Episode slug={`${episode.slug}`} title={`${episode.title}`} cover={`${episode.cover}`} abstract={`${episode.abstract}`}/> )) } </div> # API call to get data # Listing data in a component
- 37. Generate pages in Astro — // [id].astro --- const { id } = Astro.params; let response = await fetch(`.../${id}`) let episode = await response.json(); export async function getStaticPaths() { let data = await fetch("...") let episodes = await data.json(); return episodes.map((episode) => ({ params: { id: episode.id }, props: { episode: episode }, })); } --- <header id="..." class="..."> <h1 class="..." set:html={episode.title}/> </header> # Fetch data for single page # getStaticPaths() should return an array of objects to determine which paths will be pre- rendered # Display the episode data uses dynamic params in pages’ filename "
- 38. Build crawlers using jsoup
- 39. jsoup: a HTML Parser — https://jsoup.org/ // fetch the webpage val url = "..." val doc = Jsoup.connect(url).get() // use selector-syntax to find elements val elements = doc.select("div.info div.category .count") // extract what you need from elements elements.text() elements.html() elements.outerHtml() elements.attr("...")
- 40. Grab text from 喜⻢拉雅 — <!-- webpage html --> <div class="info kn_"> <div class="category kn_"> <span class="count kn_"> <i class="xuicon xuicon-erji1 kn_"></i> 546 </span> </div> </div> // fetch webpage val url = "https://www.ximalaya.com/sound/$soundId" val doc = Jsoup.connect(url).get() val element = doc.select("div.info div.category .count")[1] // grab text element.text().trim().toInt()
- 41. Grab text from YouTube — <!-- webpage html --> <!DOCTYPE html> <html> <head> <div> <meta itemprop="interactionCount" content="128"> </div> </head> </html> // fetch webpage val url = "https://www.youtube.com/watch?v=$videoId" val doc = Jsoup.connect(url).get() val element = doc .select("meta[itemprop=interactionCount]").first() // grab text element?.attr("content")?.toIntOrNull() ?:0
- 42. Trigger it hourly via Cloud Scheduler —
- 43. Display statistic data usingVaadin Chart
- 44. Getting started with Spring Boot + Vaadin —
- 45. Add Vaadin addon — // build.gradle.kts repositories { // ... maven(url = “https://maven.vaadin.com/vaadin-addons") // ... } dependencies { // ... implementation("com.vaadin:vaadin-charts-flow") // ... }
- 46. Create chart — @Route("...") @PageTitle("...") class SingleEpisodeStatisticChartView( @Autowired val service: StatisticsService, ) : VerticalLayout() { init { val chart= Chart(ChartType.COLUMN) // ... add(chart) } } # Prepare data and chart
- 47. Chart configuration — @Route("...") @PageTitle("...") class SingleEpisodeStatisticChartView( @Autowired val service: StatisticsService, ) : VerticalLayout() { init { // ... val conf = chart.configuration with(conf) { title = Title("...") subTitle = Subtitle("...") isExporting = true } // ... } }
- 48. Plot options — @Route("...") @PageTitle("...") class SingleEpisodeStatisticChartView( @Autowired val service: StatisticsService, ) : VerticalLayout() { init { // ... val column = PlotOptionsColumn() column.cursor = Cursor.POINTER column.dataLabels = DataLabels(true) conf.setPlotOptions(column) // ... } }
- 49. Axis configuration — @Route("...") @PageTitle("...") class SingleEpisodeStatisticChartView( @Autowired val service: StatisticsService, ) : VerticalLayout() { init { // ... val x = XAxis() x.type = AxisType.CATEGORY conf.addxAxis(x) val y = YAxis() y.setTitle("!!$") conf.addyAxis(y) // ... } }
- 50. Chart data series — @Route("...") @PageTitle("...") class SingleEpisodeStatisticChartView( @Autowired val service: StatisticsService, ) : VerticalLayout() { init { // ... val episodeStatistic = service .getEpisodeStatisticByPlatform(/* ep. */) val dataSeries = DataSeries() episodeStatistic.forEach { (key, value) !" dataSeries.add(DataSeriesItem(key, value)) } conf.addSeries(dataSeries) // ... } }
- 51. Retrospective
- 52. Technologies — • Astro for website • Function Framework with Kotlin for API • jsoup with Kotlin for crawler • Spring Boot + Vaadin Chart for Report
- 53. GCP services — • Cloud Storage • Cloud Functions • Cloud Scheduler • Cloud Build
- 54. Moving forward — • Paying down technical debt • Support 2 locales on website • Download/Import CSV data from Google, Apple, Firstory • Build cross-platform mobile app
- 56. Let’s talk at COSCUP booth — 29th-30th July See you in Taipei!
- 57. How I make a podcast website Photo by Michal Czyz on Unsplash using serverless technology in 2023 Shengyou Fan (范聖佑) JetBrains Developer Advocate shengyou.fan@jetbrains.com Thank you!