Designing an extensible tooling platform (Dmitry Zeldin, Skyscanner)
•Download as PPTX, PDF•
0 likes•230 views
At Skyscanner, we use a lot of internal tools to simplify and enhance developer experience. However, this abundance raises an issue of its own, as each tool has its own control panel with a separate authentication scheme. In this talk, I will show you how we solved this issue by utilising micro frontends architecture to create an extensible, reusable platform for hosting Internal tools. The application is built around OpenComponents framework created by OpenTable. Dmitry Zeldin is a software engineer at Skyscanner. While working with multiple companies in Russia, Germany and the UK, he gained experience in building scalable, robust and user-friendly web applications. Dmitry is passionate about microservices, GraphQL and Go. Key takeaways: - Overview of Micro Frontends architecture - Deep dive into Open Components by OpenTable - How to design and implement an app that combines multiple micro frontends
Recommended
Recommended
More Related Content
Similar to Designing an extensible tooling platform (Dmitry Zeldin, Skyscanner)
Similar to Designing an extensible tooling platform (Dmitry Zeldin, Skyscanner) (20)
More from London Microservices
More from London Microservices (8)
Recently uploaded
Recently uploaded (20)
Designing an extensible tooling platform (Dmitry Zeldin, Skyscanner)
- 1. Designing an extensible tooling platform
- 3. Designing an extensible tooling platform Who am I? Dmitry Zeldin Software Engineer @ Skyscanner dmitry.zeldin@skyscanner.net @ZelJin
- 4. Designing an extensible tooling platform
- 5. Designing an extensible tooling platform
- 6. Designing an extensible tooling platform
- 7. Designing an extensible tooling platform 🤷
- 8. Designing an extensible tooling platform One tool to rule them all
- 9. Designing an extensible tooling platform
- 10. Designing an extensible tooling platform
- 11. Designing an extensible tooling platform
- 12. Designing an extensible tooling platform Key requirements • Single authentication system • Same look-and-feel for all internal tools • Extensible UI
- 13. Micro Frontends
- 14. Designing an extensible tooling platform
- 15. Designing an extensible tooling platform
- 16. Designing an extensible tooling platform
- 17. Designing an extensible tooling platform Why choose micro frontends? • Complex frontend • Developed by multiple teams • With independent release cycle
- 18. Designing an extensible tooling platform The good • Technology agnostic • Source code isolation
- 19. Designing an extensible tooling platform The bad • Increased complexity • Debugging
- 20. Designing an extensible tooling platform The ugly
- 21. Designing an extensible tooling platform
- 22. Designing an extensible tooling platform
- 23. Implementation
- 24. Designing an extensible tooling platform OpenComponents
- 25. Designing an extensible tooling platform Architecture Registry CLI Client librariesComponents Template system
- 26. Designing an extensible tooling platform ├── hello-world/ ├── package.json ├── template.hbs ├── server.js ├── img/ ├── logo.png ├── src/ ├── components/ ├── index.js Component Structure
- 27. Designing an extensible tooling platform package.json "oc": { "files": { "data": "server.js", "template": { "src": "template.hbs", "type": "handlebars” }, "static": [ "build” ] }, "parameters": { ... "repository": { "type": "string", "description": "Repository which PRs should be displayed", "example": "?repository=tower/tower-ui" } } }
- 28. Designing an extensible tooling platform <link rel="stylesheet" href="{{staticPath}}{{css}}"> <div id="tower-github-pr-root"></div> <script> window['tower-github-pr'] = { props: {{{ props }}}, params: {{{ params }}} }; </script> <script defer src="{{staticPath}}{{js}}” crossorigin="anonymous"></script> Template
- 29. Designing an extensible tooling platform import assets from './../../build/asset-manifest.json'; import { version } from '../../package.json'; // ... export const data = (context, callback) => { const staticPath = `${context.staticPath}build/`; const { autoHydrate, repository, baseStylesheet } = context.params; const templateData = { baseStylesheet, staticPath, css: assets['main.css’], js: assets['main.js’], props: JSON.stringify({ repository }), params: JSON.stringify({autoHydrate, version}), }; callback(null, templateData); }; Server-side code
- 30. Designing an extensible tooling platform
- 31. Designing an extensible tooling platform
- 32. Designing an extensible tooling platform <oc-component href=http://0.0.0.0:3000/tower-github-pr> </oc-component> Client
- 33. Designing an extensible tooling platform <link rel="stylesheet" href="{{staticPath}}{{css}}"> <div id="tower-github-pr-root"></div> <script> window['tower-github-pr'] = { props: {{{ props }}}, params: {{{ params }}} }; </script> <script defer src="{{staticPath}}{{js}}” crossorigin="anonymous"></script> Client
- 34. Designing an extensible tooling platform <link rel="stylesheet" href="http://localhost:3000/tower-github-pr/0.0.7/static/build/static/css/main.af206699.css" /> <div id="tower-github-pr-root"></div> <script> window["tower-github-pr"] = { props: {}, params: { autoHydrate: false, version: "0.0.7" } }; </script> <script defer src="http://localhost:3000/tower-github-pr/0.0.7/static/build/static/js/main.d89a7503.js" crossorigin="anonymous" ></script> Client
- 35. Designing an extensible tooling platform Request Fetch Compile & Populate Serve
- 36. Example 👨💻
- 37. Designing an extensible tooling platform
- 38. Designing an extensible tooling platform
- 39. Designing an extensible tooling platform
- 40. Designing an extensible tooling platform
- 41. Designing an extensible tooling platform
- 42. Designing an extensible tooling platform
- 43. Designing an extensible tooling platform
- 44. Designing an extensible tooling platform
- 45. Designing an extensible tooling platform ❤️
- 46. Designing an extensible tooling platform 🤷
- 47. Pain points • Credentials storage • Performance issues • Development complexity
- 48. Key takeaways • Prefer client-side rendering • Make server-side layer thin • More tutorials, better on-boarding
- 49. Unresolved issues • Role-based authentication • Dependency management
- 50. Thank you!
- 51. Q & A
Editor's Notes
- Good evening everyone! Thank you for coming here today, It’s really nice to see you all! I’m Dmitry and I work at Developer Enablement Tribe in Skyscanner. Today we’re going to talk about building an extensible tooling platform. The talk should take around 30 minutes, and there will be a 10 minute Q&A session in the end. If you have a question later on, do not hesitate to reach me out via email or Twitter.
- The company itself probably doesn’t need a long introduction. Skyscanner is a leading global travel search site offering a comprehensive and free flight search service as well as online comparisons for hotels and car hire. We have around 70m unique monthly visitors on our website
- Our iOS and Android apps have been downloaded over 70m times.
- In Developer Enablement Tribe, we are trying to empower Skyscanner engineers to deliver secure, resilient and efficient services at scale. We offer loads of services and provide robust runtime environment to our developers so they can spend the majority of their time solving customer’s problems rather than re-inventing the wheel. However, this abundance of services comes at a cost – each internal tool usually comes with its own UI
- As an example, here is a common workflow – a developer would like to deploy the project to production. First of all, developer approves a pull request in GitHub, then navigates to Drone – open-source CI tools to check the project build status and to SonarQube to access code quality metrics. Finally, developer might like to check the deployment progress in Slingshot – our in-house deployment system and AWS Console. By the time the build succeeds, our developer has visited five tools with five different UIs. Some of the tools are developed in-house, but most of them are 3rd party
- We are trying to solve this issue by introducing Tower – first point of contact for all internal tools at Skyscanner It’s not going to replace all other internal tool UIs, but it will make it easier to access most commonly used features
- Tower is still in its early development phase. Here is what developers can already do: they can find their project or a 3rd-party service using rich search.
- A brief summary is displayed for each project or service. It includes owner and compliance information, links to documentation and useful tools.
- And here we can see a list of open pull requests for a certain GitHub repository.
- Here are the key requirements for Tower we’ve came up with: First, all internal tools should be available under single authentication system. No further authentication should be required. Each internal tool, especially a 3rd-party one, has its own set of design guidelines that are hard or almost impossible to change. This causes additional friction when you switch from one tool to another. Tower aims to solve this problem by using a single design paradigm that is familiar to all Skyscanner developers. Finally, Tower should be easily extensible, lowering the barrier to add or remove an internal tool.
- These requirements lead us to the idea of using micro frontends architecture for Tower. Let’s look into it.
- In monolithic architecture, there is a single application that combines both frontend and backend. These applications are easy to develop, test and deploy when the application is small enough, but the development process does not scale properly with the increase in app complexity
- Microservices architecture addresses this issue by splitting the backend into multiple loosely coupled components. You all know the benefits and trade-offs of splitting a monolith backend into multiple services. However, until recently teams struggled to decouple front-end monoliths.
- Recent advances in JavaScript, like ES6 modules and modern bundlers like WebPack allowed micro frontends architecture to finally take shape. In this approach, monolithic frontend is broken up by its pages and features, where each feature is owned by a single team. Micro frontend is entirely presentation and does not have any server-side logic. Instead, it requests data from one or many microservices.
- So why have we decided to adopt micro frontend architecture? The team responsible for Tower simply cannot support extensions for all internal tools – there is not enough manpower for such a complex task. Therefore, development is distributed across multiple teams that have their own development cycle. This combination basically begs for source code separation.
- Adopting micro frontends architecture gave us a lot of benefits. First of all, it gives us the freedom to choose the right tools for the job – each micro frontend is completely technologically agnostic from the other one. For example, one tool can be written in vue.js and rendered on the client side, while another one can be implemented in React and rendered on the server side. Source code isolation makes it easier to maintain and test each individual integration. That also means every integration can be deployed independently of any other system.
- The negative impact of microservices approach is the increased complexity of the whole system – and such complexity makes it harder to debug any occurring issues.
- Here comes the ugly part
- In this architecture diagram, one important layer is missing: micro frontends are no longer web apps. Instead, they are UI fragments that have to be combined together on a single web page.
- A so-called “Stitching layer”, or a host web application combines multiple micro frontends on one page to enable seamless user experience. This layer has proven to be the most complex bit in micro frontend adoption, and it’s probably the reason this architecture is not so widely spread at the moment.
- Let’s move on to our implementation of a micro frontend system.
- We decided to use OpenComponents framework developed by OpenTable. This open-source framework has seen its first release in 2014 and is proved to be mature enough for the task.
- OpenComponents has 5 main building blocks: Components are our micro frontends that usually consist of HTML, JavaScript and CSS files. They can also contain server-side logic that is used to compose a model which renders the final view. Template system takes the component with the model from the server-side and generates an HTML document that is injected onto a page. Registry provides a REST API to consume, retrieve and publish components to a library. Client libraries are available for multiple languages and provide functionality to load and render a component from the registry. Finally, OpenComponents comes with a nice CLI tools, allowing developers to create, develop and test components locally. It also helps to publish components to the registry.
- This is a typical structure of a component. It’s a Node.js application, where the template is defined in a Handlebars file, and server-side logic is defined in server.js.
- Application manifest, package.json, contains a dedicated section named “OC”, where component configuration is stored. Here you can customise the location of the server-side logic, choice of template and templating engine and a lot more. Component can be parametrised. For that, all parameters have to be listed in the package.json Parameters can be passed to a component via querystring, when a GET request is sent to the registry to retrieve a component.
- This is an example of a Handlebars template for a single-page application. It takes parameters generated on the server side and returns a valid HTML with links to compiled CSS and JS files, as well as a root container where a component is rendered. In this particular case, a React app is rendered, that’s why we included a script to pass props to the root React component.
- Server-side code can fetch data from various APIs to generate a data object that is passed to the template. In this particular example, we do not call anything and are just passing parameters communicated via GET request. Please note that this code is running in a node environment of the registry, thus it supports only a limited number of dependencies, and the only way to add an npm module is to install it to the registry.
- OC registry also offers another UI where you can see the list of all available components
- For each component, you can check the parameters it takes and even render it in a sandbox environment.
- Let’s move on to the stitching layer. A host application that loads components has an OC client library as a dependency. Then a component can be included on the page as an OC-component tag. For each tag, OC client library requests an open component from the registry.
- OC registry runs server-side logic and generates data necessary to populate a view
- Here you can see a populated view for the React application that renders open GitHub pull requests. Finally, generated HTML is sent back to the host application, and OC client library injects this onto the page.
- Here you can see an overview of the whole process: OC client library requests the parametrised component from the Registry, Registry fetches the component, it compiles and populates its template that is being returned to the host application. OC Registry can be extended with plugins, and one useful plugin is the addition of caching. Each successfully rendered component template is cached, and, next time it’s requested, OC registry returns a cached version.
- Let’s take a look how it’s actually implemented
- Let’s return to Tower and look at the component view once again The UI element that displays pull requests is actually a component that is fetched from the registry Let’s take a look at the requests tab to see actual network calls. The component has loaded and shows that we have exactly one open pull request for this project. Let’s have a look at the development console to see the actual network calls Tower did. It fetched the component from the Registry, then the OC client has injected it onto the page. Then OC component fetches required libraries. To save on the bundle size, commonly used libraries are left outside of the component and loaded separately, I already have these libraries in my browser’s cache, so no additional network calls are made. When component is initialised, it fetches GitHub API to retrieve the number of pull requests. It passes the same token that is used to login within Tower.
- First, OC Client requests a component from the registry. Registry fetches the component, compiles and returns the template
- Here is the output from the registry. OC Client takes html and injects this html onto the page.
- When html is injected, css and javascript dependencies are parsed and loaded.
- When html is injected, css and javascript dependencies are parsed and loaded.
- Once the Component is loaded, it queries GitHub API to get the list of pull requests. This step requires authentication, and GitHub Enterprise uses a slightly different authentication system than the one that is used in Tower. Our Internal authentication is based on JSON Web Token and tied to Active Directory. The component call a microservice that verifies JSON Web Token, identifies the user, generates user GitHub token and passes the request to GitHub API.
- Commonly used dependencies like React are not part of the bundle and are downloaded separately. Unless cache is not disabled in your browser, these dependencies are cached and loaded only once.
- Commonly used dependencies like React are not part of the bundle and are downloaded separately. Unless cache is not disabled in your browser, these dependencies are cached and loaded only once.
- Developers at Skyscanner have a long history developing micro frontends with OpenComponents. This technology is used on the actual skyscanner.net. It allowed us to split the frontend codebase to develop features faster, and has proved to be pretty much robust along the way.
- However, the learning curve for this technology is really steep. A lot of nerve cells have been lost before we achieved the level of comfort and stability we have today.
- The technology is subject to certain limitations. The most impactful one is a severe limitation on the complexity of the server-side component. All components are running in a shared environment, without separation of concerns. As a result, there is only one source of secrets for all components running in one registry, meaning a single component can access secrets reserved for all components running on this registry. Shared environment also makes it too risky to run complex backend tasks or do complex server-side rendering. Furthermore, if a single component crashes the Registry because of an uncaught rendering error, the whole Registry is brought down for a short period of time. Steep learning curve makes it hard to enrol new engineers into developing components.
- To improve the performance and stability of the Registry, we took the following decisions. We decided to limit the usage of server-side rendering to applications where it’s absolutely necessary. We make our server-side layer extremely thin and extract all complex tasks to separate microservices that run outside of the Registry’s environment. We have put a considerable effort into writing tutorials, improving documentation and creating on-boarding process for engineers who’d like to write integrations for Tower. I would like to point out these takeaways are valid for internal development, as some of these features we decided not to use, like server-side rendering, is vital for skyscanner.com website.
- However, there are still some unresolved issues. Some internal tools have different roles assigned for different users. We still have not found a way how we can provide a single, role-based authentication system for all available internal tools. Another problem, which originates from the freedom of technology stack choice and independence of development cycle, would be extra dependencies that increase the overall size of the application and negatively affect loading times. For example, different React components can have multiple React versions in their dependencies. As a result, all React versions have to be downloaded before the whole page would work.
- Thank you very much for your time! I hope you enjoyed the presentation! I believe Microfrontends architecture has a lot of potential, and it’s definitely worth a try. However, a true benefit of micro frontends may not be visible for smaller projects.
- Now, let’s move to the Q & A session.