A couple more slides than the original https://www.slideshare.net/alexmass/swiss-transport-in-real-time-tribulations-in-the-big-data-stack
And a great evening in Bern
Swiss Transport in Real Time: Tribulations in the Big Data StackAlexandre Masselot
A lot of data are available in realtime on Swiss public transportation. Vehicles positions, station board (with delays) etc.
We use these data to illustrate a common pattern and build a proof of concept project. The idea is to address the question: "Is it possible to build a simple scalable infrastructure, to dispatch, transform and visualize 'near real time' massive data and achieve a posteriori analysis?"
We will describe such an infrastructure, focusing on the different bricks:
* streaming events with Kafka and Logstash;
* flow transformation with Akka or Play Streaming;
* storage in Elasticsearch;
* real time visualization with ReactJS and d3.js;
* a posteriori analysis with Python and Jupyter;
* not to forget DevOps with Docker, GCE and AWS.
A conference given at softshake 2016 in Geneva - www.softshake.ch
Swiss Transport in Real Time: Tribulations in the Big Data StackAlexandre Masselot
A lot of data are available in realtime on Swiss public transportation. Vehicles positions, station board (with delays) etc.
We use these data to illustrate a common pattern and build a proof of concept project. The idea is to address the question: "Is it possible to build a simple scalable infrastructure, to dispatch, transform and visualize 'near real time' massive data and achieve a posteriori analysis?"
We will describe such an infrastructure, focusing on the different bricks:
* streaming events with Kafka and Logstash;
* flow transformation with Akka or Play Streaming;
* storage in Elasticsearch;
* real time visualization with ReactJS and d3.js;
* a posteriori analysis with Python and Jupyter;
* not to forget DevOps with Docker, GCE and AWS.
A conference given at softshake 2016 in Geneva - www.softshake.ch
PUNTOG - HEAD SHOP - VAPE SHOP - HEMP SHOP - GROW SHOP
VENDITA E ASSISTENZA ON LINE SU BONG E PIPE AD ACQUA
www.puntog-shop.com
Mobile +39 345 889 3933
Email : info@puntog-shop.com
ORDINI TELEFONICI PER BONG E PIPE AD ACQUA
Spediamo la merce in tutta Italia con Consegna in pacco assolutamente robusto. Riceverai la merce entro tre giorni lavorativi dall’ordine. Chiama Ora!
Badanie i implementacja aspektu QoE (ang. Quality of Experience) w aplikacjac...Mikolaj Leszczuk
Zapewnianie nie tylko wysokiego poziomu tradycyjnej jakości usług (ang. Quality of Service, QoS), ale także jakości doświadczanej (ang. Quality of Experience, QoE) jest wyzwaniem dla dostawców usług internetowych, usług audiowizualnych, nadawców oraz nowych dostawców usług Over-The-Top (OTT). W celu monitorowania i rozwiązywania problemów, a także analizowania i ustanawiania wzorców jakości dla aplikacji treści pracujących w czasie rzeczywistym lub offline często są realizowane obiektywne metryki treści audiowizualnych. Od 2000 roku prace związanie z pojęciem QoE w kontekstach różnych aplikacji nabrały tempa i zyskały szerokie uznanie biznesowe. Roboczą definicję QoE podaje biała księga sieci QUALINET: „QUALINET White Paper on QoE Definitions” z 2012 roku:
„Quality of Experience (QoE) jest stopniem zadowolenia lub irytacji użytkownika aplikacji lub usługi. QoE wynika z realizacji oczekiwań użytkownika względem użyteczności i/lub zadowolenia z aplikacji lub usługi w świetle aktualnych preferencji użytkownika.”
Big Data is a new term used in Business Analytics to identify datasets that we can not manage with current methodologies or data mining software tools due to their large size and complexity. Big Data mining is the capability of extracting useful information from these large datasets or streams of data. New mining techniques are necessary due to the volume, variability, and velocity, of such data.
In this talk, we will focus on advanced techniques in Big Data mining in real time using evolving data stream techniques: using a small amount of time and memory resources, and being able to adapt to changes. We will discuss a social network application of data stream mining to compute user influence probabilities. And finally, we will present the MOA software framework with classification, regression, and frequent pattern methods, and the SAMOA distributed streaming software that runs on top of Storm, Samza and S4.
This was presented for an O'Reilly Media webcast. http://www.oreilly.com/pub/e/3152?cmp=tw-na-webcast-product-webcast_an_introduction_to_apache_accumulo
This webcast will cover the basics of Apache Accumulo architecture and how it works, along with examples of how it is used. We'll also talk about some interesting use cases, such as text indexing, fine-grained multi-level access controls, and storing large-scale graphs. We'll also briefly touch on what sets Accumulo apart from other similar and not-so similar systems and where we think the Accumulo project is headed in a technical direction.
A description of Accumulo from the Apache Accumulo website:
The Apache Accumulo sorted, distributed key/value store is a robust, scalable, high performance data storage and retrieval system. Apache Accumulo is based on Google's BigTable design and is built on top of Apache Hadoop, Zookeeper, and Thrift. Apache Accumulo features a few novel improvements on the BigTable design in the form of cell-based access control and a server-side programming mechanism that can modify key/value pairs at various points in the data management process. Other notable improvements and feature are outlined here. Google published the design of BigTable in 2006. Several other open source projects have implemented aspects of this design including HBase, Hypertable, and Cassandra. Accumulo began its development in 2008 and joined the Apache community in 2011.
Building and deploying Kibana plugins… And should I do it?
Developing Kibana plugins seems an appealing way to enhance the Elastic frontend. Even though tactical problems can be overcome, it might not be the best choice from a strategical point of view
To prepare a clean, quality weld seam, it’s important to have a superior edge, and the correct preparation sequence prevents costly rework. The NorBevel-6 and NorBevel-12 range of machines and accessories are new additions to Norton’s metal fabrication solutions.
Bible Studies for Life - Connecting at Every AgeRonnie Floyd
A dedicated Senior Pastor at Cross Church Northwest Arkansas, Dr. Ronnie Floyd is also serves as the general editor of LifeWay’s Bible Studies for Life (BSL) curriculum series, which is used by more than
150,000 small groups nationwide.
Dear all,
ITS World Congress 2009 is now only few hours away and we hope you are ready. We are!
The CVIS team has set up an impressive set of demonstrations of cooperative systems (v2v and v2i based technologies and applications). We can offer you not less than four types of demonstrations:
1. Public road tour where you will be able to experience 12 CVIS applications
2. Test track demonstrations at the Cooperative Systems Demo Area where CVIS is presenting amongst others joint work with SAFESPOT and COOPERS projects
3. Theatrical play on cooperative systems at the Demonstration Theatre (the play is jointly prepared with SAFESPOT)
4. Simulator demonstrations at the European Commission stand (joint work with SAFESPOT)
In addition when you visit these congress highlights you can also win exclusive prizes.
Interested? Learn more from the CVIS Live! brochure.
See you soon and have a nice trip!
CVIS Consortium
Transport-as-a-Service (TaaS) - How we build next generation plug-and-play IT...Christoffer Vig
The digitalization of everyday life, where the average bus passenger would rather ignore the surroundings and stare blankly into his phone for snapchat updates, creates great opportunities for digitalization of public transport, as well as some challenges. Ruter, Norway’s largest public transport authority, is putting significant effort into improving digital services for customers. However, the current technology stack behind Ruter's operations, is not able to provide the necessary real-time information.
In this talk I will explain how Ruter is participating in the development of new European standards for information technology for public transport (ITxPT), and how Apache Kafka and Kafka streams serve as the core components in building "Transport-as-a-Service".
Presented at Berlin Buzzwords 2018 https://berlinbuzzwords.de/18/session/transport-service-taas-how-we-build-next-generation-plug-and-play-it-systems-public
Christian Leysen, Ahlers on Economy: Get Ready for the Rebound'eyefortransport
Christian Leysen, Executive Chairman of Ahlers speaks at a session on 'The Economy: Get Ready for the Rebound' at at the 7th European 3PL Summit in Brussels, November 25th 2009.
To download all of the slides from the conference for free visit www.3PLsummit.com/eu_2009ppts
PUNTOG - HEAD SHOP - VAPE SHOP - HEMP SHOP - GROW SHOP
VENDITA E ASSISTENZA ON LINE SU BONG E PIPE AD ACQUA
www.puntog-shop.com
Mobile +39 345 889 3933
Email : info@puntog-shop.com
ORDINI TELEFONICI PER BONG E PIPE AD ACQUA
Spediamo la merce in tutta Italia con Consegna in pacco assolutamente robusto. Riceverai la merce entro tre giorni lavorativi dall’ordine. Chiama Ora!
Badanie i implementacja aspektu QoE (ang. Quality of Experience) w aplikacjac...Mikolaj Leszczuk
Zapewnianie nie tylko wysokiego poziomu tradycyjnej jakości usług (ang. Quality of Service, QoS), ale także jakości doświadczanej (ang. Quality of Experience, QoE) jest wyzwaniem dla dostawców usług internetowych, usług audiowizualnych, nadawców oraz nowych dostawców usług Over-The-Top (OTT). W celu monitorowania i rozwiązywania problemów, a także analizowania i ustanawiania wzorców jakości dla aplikacji treści pracujących w czasie rzeczywistym lub offline często są realizowane obiektywne metryki treści audiowizualnych. Od 2000 roku prace związanie z pojęciem QoE w kontekstach różnych aplikacji nabrały tempa i zyskały szerokie uznanie biznesowe. Roboczą definicję QoE podaje biała księga sieci QUALINET: „QUALINET White Paper on QoE Definitions” z 2012 roku:
„Quality of Experience (QoE) jest stopniem zadowolenia lub irytacji użytkownika aplikacji lub usługi. QoE wynika z realizacji oczekiwań użytkownika względem użyteczności i/lub zadowolenia z aplikacji lub usługi w świetle aktualnych preferencji użytkownika.”
Big Data is a new term used in Business Analytics to identify datasets that we can not manage with current methodologies or data mining software tools due to their large size and complexity. Big Data mining is the capability of extracting useful information from these large datasets or streams of data. New mining techniques are necessary due to the volume, variability, and velocity, of such data.
In this talk, we will focus on advanced techniques in Big Data mining in real time using evolving data stream techniques: using a small amount of time and memory resources, and being able to adapt to changes. We will discuss a social network application of data stream mining to compute user influence probabilities. And finally, we will present the MOA software framework with classification, regression, and frequent pattern methods, and the SAMOA distributed streaming software that runs on top of Storm, Samza and S4.
This was presented for an O'Reilly Media webcast. http://www.oreilly.com/pub/e/3152?cmp=tw-na-webcast-product-webcast_an_introduction_to_apache_accumulo
This webcast will cover the basics of Apache Accumulo architecture and how it works, along with examples of how it is used. We'll also talk about some interesting use cases, such as text indexing, fine-grained multi-level access controls, and storing large-scale graphs. We'll also briefly touch on what sets Accumulo apart from other similar and not-so similar systems and where we think the Accumulo project is headed in a technical direction.
A description of Accumulo from the Apache Accumulo website:
The Apache Accumulo sorted, distributed key/value store is a robust, scalable, high performance data storage and retrieval system. Apache Accumulo is based on Google's BigTable design and is built on top of Apache Hadoop, Zookeeper, and Thrift. Apache Accumulo features a few novel improvements on the BigTable design in the form of cell-based access control and a server-side programming mechanism that can modify key/value pairs at various points in the data management process. Other notable improvements and feature are outlined here. Google published the design of BigTable in 2006. Several other open source projects have implemented aspects of this design including HBase, Hypertable, and Cassandra. Accumulo began its development in 2008 and joined the Apache community in 2011.
Building and deploying Kibana plugins… And should I do it?
Developing Kibana plugins seems an appealing way to enhance the Elastic frontend. Even though tactical problems can be overcome, it might not be the best choice from a strategical point of view
To prepare a clean, quality weld seam, it’s important to have a superior edge, and the correct preparation sequence prevents costly rework. The NorBevel-6 and NorBevel-12 range of machines and accessories are new additions to Norton’s metal fabrication solutions.
Bible Studies for Life - Connecting at Every AgeRonnie Floyd
A dedicated Senior Pastor at Cross Church Northwest Arkansas, Dr. Ronnie Floyd is also serves as the general editor of LifeWay’s Bible Studies for Life (BSL) curriculum series, which is used by more than
150,000 small groups nationwide.
Dear all,
ITS World Congress 2009 is now only few hours away and we hope you are ready. We are!
The CVIS team has set up an impressive set of demonstrations of cooperative systems (v2v and v2i based technologies and applications). We can offer you not less than four types of demonstrations:
1. Public road tour where you will be able to experience 12 CVIS applications
2. Test track demonstrations at the Cooperative Systems Demo Area where CVIS is presenting amongst others joint work with SAFESPOT and COOPERS projects
3. Theatrical play on cooperative systems at the Demonstration Theatre (the play is jointly prepared with SAFESPOT)
4. Simulator demonstrations at the European Commission stand (joint work with SAFESPOT)
In addition when you visit these congress highlights you can also win exclusive prizes.
Interested? Learn more from the CVIS Live! brochure.
See you soon and have a nice trip!
CVIS Consortium
Transport-as-a-Service (TaaS) - How we build next generation plug-and-play IT...Christoffer Vig
The digitalization of everyday life, where the average bus passenger would rather ignore the surroundings and stare blankly into his phone for snapchat updates, creates great opportunities for digitalization of public transport, as well as some challenges. Ruter, Norway’s largest public transport authority, is putting significant effort into improving digital services for customers. However, the current technology stack behind Ruter's operations, is not able to provide the necessary real-time information.
In this talk I will explain how Ruter is participating in the development of new European standards for information technology for public transport (ITxPT), and how Apache Kafka and Kafka streams serve as the core components in building "Transport-as-a-Service".
Presented at Berlin Buzzwords 2018 https://berlinbuzzwords.de/18/session/transport-service-taas-how-we-build-next-generation-plug-and-play-it-systems-public
Christian Leysen, Ahlers on Economy: Get Ready for the Rebound'eyefortransport
Christian Leysen, Executive Chairman of Ahlers speaks at a session on 'The Economy: Get Ready for the Rebound' at at the 7th European 3PL Summit in Brussels, November 25th 2009.
To download all of the slides from the conference for free visit www.3PLsummit.com/eu_2009ppts
Similar to Dev Wednesday - Swiss Transport in Real Time: Tribulations in the Big Data Stack (7)
Offshoring software development in Switzerland: You can do itAlexandre Masselot
If offshoring is often driven by short term considerations and economical drivers, there are better reasons to externalize software development. And why not coming to Switzerland?. Over the past two years, we have reversed the offshoring flow, pursuing ambitious IT projects for US companies, from the Swiss mountains.
The Agile Tour conference in Lausanne is the occasion of the lessons learned during adventures and to reflect on how we can improve the development of inhouse digital products.
Top Features to Include in Your Winzo Clone App for Business Growth (4).pptxrickgrimesss22
Discover the essential features to incorporate in your Winzo clone app to boost business growth, enhance user engagement, and drive revenue. Learn how to create a compelling gaming experience that stands out in the competitive market.
top nidhi software solution freedownloadvrstrong314
This presentation emphasizes the importance of data security and legal compliance for Nidhi companies in India. It highlights how online Nidhi software solutions, like Vector Nidhi Software, offer advanced features tailored to these needs. Key aspects include encryption, access controls, and audit trails to ensure data security. The software complies with regulatory guidelines from the MCA and RBI and adheres to Nidhi Rules, 2014. With customizable, user-friendly interfaces and real-time features, these Nidhi software solutions enhance efficiency, support growth, and provide exceptional member services. The presentation concludes with contact information for further inquiries.
Globus Compute wth IRI Workflows - GlobusWorld 2024Globus
As part of the DOE Integrated Research Infrastructure (IRI) program, NERSC at Lawrence Berkeley National Lab and ALCF at Argonne National Lab are working closely with General Atomics on accelerating the computing requirements of the DIII-D experiment. As part of the work the team is investigating ways to speedup the time to solution for many different parts of the DIII-D workflow including how they run jobs on HPC systems. One of these routes is looking at Globus Compute as a way to replace the current method for managing tasks and we describe a brief proof of concept showing how Globus Compute could help to schedule jobs and be a tool to connect compute at different facilities.
First Steps with Globus Compute Multi-User EndpointsGlobus
In this presentation we will share our experiences around getting started with the Globus Compute multi-user endpoint. Working with the Pharmacology group at the University of Auckland, we have previously written an application using Globus Compute that can offload computationally expensive steps in the researcher's workflows, which they wish to manage from their familiar Windows environments, onto the NeSI (New Zealand eScience Infrastructure) cluster. Some of the challenges we have encountered were that each researcher had to set up and manage their own single-user globus compute endpoint and that the workloads had varying resource requirements (CPUs, memory and wall time) between different runs. We hope that the multi-user endpoint will help to address these challenges and share an update on our progress here.
Enhancing Research Orchestration Capabilities at ORNL.pdfGlobus
Cross-facility research orchestration comes with ever-changing constraints regarding the availability and suitability of various compute and data resources. In short, a flexible data and processing fabric is needed to enable the dynamic redirection of data and compute tasks throughout the lifecycle of an experiment. In this talk, we illustrate how we easily leveraged Globus services to instrument the ACE research testbed at the Oak Ridge Leadership Computing Facility with flexible data and task orchestration capabilities.
Code reviews are vital for ensuring good code quality. They serve as one of our last lines of defense against bugs and subpar code reaching production.
Yet, they often turn into annoying tasks riddled with frustration, hostility, unclear feedback and lack of standards. How can we improve this crucial process?
In this session we will cover:
- The Art of Effective Code Reviews
- Streamlining the Review Process
- Elevating Reviews with Automated Tools
By the end of this presentation, you'll have the knowledge on how to organize and improve your code review proces
Experience our free, in-depth three-part Tendenci Platform Corporate Membership Management workshop series! In Session 1 on May 14th, 2024, we began with an Introduction and Setup, mastering the configuration of your Corporate Membership Module settings to establish membership types, applications, and more. Then, on May 16th, 2024, in Session 2, we focused on binding individual members to a Corporate Membership and Corporate Reps, teaching you how to add individual members and assign Corporate Representatives to manage dues, renewals, and associated members. Finally, on May 28th, 2024, in Session 3, we covered questions and concerns, addressing any queries or issues you may have.
For more Tendenci AMS events, check out www.tendenci.com/events
Unleash Unlimited Potential with One-Time Purchase
BoxLang is more than just a language; it's a community. By choosing a Visionary License, you're not just investing in your success, you're actively contributing to the ongoing development and support of BoxLang.
Cyaniclab : Software Development Agency Portfolio.pdfCyanic lab
CyanicLab, an offshore custom software development company based in Sweden,India, Finland, is your go-to partner for startup development and innovative web design solutions. Our expert team specializes in crafting cutting-edge software tailored to meet the unique needs of startups and established enterprises alike. From conceptualization to execution, we offer comprehensive services including web and mobile app development, UI/UX design, and ongoing software maintenance. Ready to elevate your business? Contact CyanicLab today and let us propel your vision to success with our top-notch IT solutions.
Providing Globus Services to Users of JASMIN for Environmental Data AnalysisGlobus
JASMIN is the UK’s high-performance data analysis platform for environmental science, operated by STFC on behalf of the UK Natural Environment Research Council (NERC). In addition to its role in hosting the CEDA Archive (NERC’s long-term repository for climate, atmospheric science & Earth observation data in the UK), JASMIN provides a collaborative platform to a community of around 2,000 scientists in the UK and beyond, providing nearly 400 environmental science projects with working space, compute resources and tools to facilitate their work. High-performance data transfer into and out of JASMIN has always been a key feature, with many scientists bringing model outputs from supercomputers elsewhere in the UK, to analyse against observational or other model data in the CEDA Archive. A growing number of JASMIN users are now realising the benefits of using the Globus service to provide reliable and efficient data movement and other tasks in this and other contexts. Further use cases involve long-distance (intercontinental) transfers to and from JASMIN, and collecting results from a mobile atmospheric radar system, pushing data to JASMIN via a lightweight Globus deployment. We provide details of how Globus fits into our current infrastructure, our experience of the recent migration to GCSv5.4, and of our interest in developing use of the wider ecosystem of Globus services for the benefit of our user community.
Prosigns: Transforming Business with Tailored Technology SolutionsProsigns
Unlocking Business Potential: Tailored Technology Solutions by Prosigns
Discover how Prosigns, a leading technology solutions provider, partners with businesses to drive innovation and success. Our presentation showcases our comprehensive range of services, including custom software development, web and mobile app development, AI & ML solutions, blockchain integration, DevOps services, and Microsoft Dynamics 365 support.
Custom Software Development: Prosigns specializes in creating bespoke software solutions that cater to your unique business needs. Our team of experts works closely with you to understand your requirements and deliver tailor-made software that enhances efficiency and drives growth.
Web and Mobile App Development: From responsive websites to intuitive mobile applications, Prosigns develops cutting-edge solutions that engage users and deliver seamless experiences across devices.
AI & ML Solutions: Harnessing the power of Artificial Intelligence and Machine Learning, Prosigns provides smart solutions that automate processes, provide valuable insights, and drive informed decision-making.
Blockchain Integration: Prosigns offers comprehensive blockchain solutions, including development, integration, and consulting services, enabling businesses to leverage blockchain technology for enhanced security, transparency, and efficiency.
DevOps Services: Prosigns' DevOps services streamline development and operations processes, ensuring faster and more reliable software delivery through automation and continuous integration.
Microsoft Dynamics 365 Support: Prosigns provides comprehensive support and maintenance services for Microsoft Dynamics 365, ensuring your system is always up-to-date, secure, and running smoothly.
Learn how our collaborative approach and dedication to excellence help businesses achieve their goals and stay ahead in today's digital landscape. From concept to deployment, Prosigns is your trusted partner for transforming ideas into reality and unlocking the full potential of your business.
Join us on a journey of innovation and growth. Let's partner for success with Prosigns.
Enterprise Resource Planning System includes various modules that reduce any business's workload. Additionally, it organizes the workflows, which drives towards enhancing productivity. Here are a detailed explanation of the ERP modules. Going through the points will help you understand how the software is changing the work dynamics.
To know more details here: https://blogs.nyggs.com/nyggs/enterprise-resource-planning-erp-system-modules/
Accelerate Enterprise Software Engineering with PlatformlessWSO2
Key takeaways:
Challenges of building platforms and the benefits of platformless.
Key principles of platformless, including API-first, cloud-native middleware, platform engineering, and developer experience.
How Choreo enables the platformless experience.
How key concepts like application architecture, domain-driven design, zero trust, and cell-based architecture are inherently a part of Choreo.
Demo of an end-to-end app built and deployed on Choreo.
Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...Globus
The Earth System Grid Federation (ESGF) is a global network of data servers that archives and distributes the planet’s largest collection of Earth system model output for thousands of climate and environmental scientists worldwide. Many of these petabyte-scale data archives are located in proximity to large high-performance computing (HPC) or cloud computing resources, but the primary workflow for data users consists of transferring data, and applying computations on a different system. As a part of the ESGF 2.0 US project (funded by the United States Department of Energy Office of Science), we developed pre-defined data workflows, which can be run on-demand, capable of applying many data reduction and data analysis to the large ESGF data archives, transferring only the resultant analysis (ex. visualizations, smaller data files). In this talk, we will showcase a few of these workflows, highlighting how Globus Flows can be used for petabyte-scale climate analysis.
Into the Box Keynote Day 2: Unveiling amazing updates and announcements for modern CFML developers! Get ready for exciting releases and updates on Ortus tools and products. Stay tuned for cutting-edge innovations designed to boost your productivity.
SOCRadar Research Team: Latest Activities of IntelBrokerSOCRadar
The European Union Agency for Law Enforcement Cooperation (Europol) has suffered an alleged data breach after a notorious threat actor claimed to have exfiltrated data from its systems. Infamous data leaker IntelBroker posted on the even more infamous BreachForums hacking forum, saying that Europol suffered a data breach this month.
The alleged breach affected Europol agencies CCSE, EC3, Europol Platform for Experts, Law Enforcement Forum, and SIRIUS. Infiltration of these entities can disrupt ongoing investigations and compromise sensitive intelligence shared among international law enforcement agencies.
However, this is neither the first nor the last activity of IntekBroker. We have compiled for you what happened in the last few days. To track such hacker activities on dark web sources like hacker forums, private Telegram channels, and other hidden platforms where cyber threats often originate, you can check SOCRadar’s Dark Web News.
Stay Informed on Threat Actors’ Activity on the Dark Web with SOCRadar!
In software engineering, the right architecture is essential for robust, scalable platforms. Wix has undergone a pivotal shift from event sourcing to a CRUD-based model for its microservices. This talk will chart the course of this pivotal journey.
Event sourcing, which records state changes as immutable events, provided robust auditing and "time travel" debugging for Wix Stores' microservices. Despite its benefits, the complexity it introduced in state management slowed development. Wix responded by adopting a simpler, unified CRUD model. This talk will explore the challenges of event sourcing and the advantages of Wix's new "CRUD on steroids" approach, which streamlines API integration and domain event management while preserving data integrity and system resilience.
Participants will gain valuable insights into Wix's strategies for ensuring atomicity in database updates and event production, as well as caching, materialization, and performance optimization techniques within a distributed system.
Join us to discover how Wix has mastered the art of balancing simplicity and extensibility, and learn how the re-adoption of the modest CRUD has turbocharged their development velocity, resilience, and scalability in a high-growth environment.
Dev Wednesday - Swiss Transport in Real Time: Tribulations in the Big Data Stack
1. Swiss Transport in Real Time:
Tribulations in the Big Data Stack
Alexandre Masselot
Dev. Wednesday
March 2017
@alex_mass
2. Swiss Transport in Real Time:
Tribulations in the Big Data Stack
Alexandre Masselot
Dev. Wednesday
March 2017
@alex_mass
3. AVENUE DU THÉÂTRE, 7 – 1005 LAUSANNE > SUISSE > WWW.OCTO.CH
OCTO Suisse RECRUTE
5 consultants en 2017
rejoins.octo.com
Architecte
Software
Craftsman DataGeek
Coach
Méthodo
Expert
DevOps
Consultant
en Stratégie
4. Is it possible to build
a simple scalable infrastructure, to
dispatch, store, transform
and visualize “near real time” data
and achieve a posteriori analysis?
This is only
a POC!!!
5. Finding a dataset
• social media
• finance
• sport
• energy
• transport
• log analysis
• meteorology
• bioinformatics
• personalized health
• monitoring
• security
• IOT
6. Finding a dataset
• social media
• finance
• sport
• energy
• transport
• log analysis
• meteorology
• bioinformatics
• personalized health
• monitoring
• security
• IOT
11. AAGL Autobus AG Liestal
AAGR Auto AG Rothenburg
AAGS Auto AG Schwyz
AAGU AUTO AG URI
AB Appenzeller Bahnen AG
ABl Autolinee Bleniesi SA
ABF Autobusbetrieb Freienbach
AFA Automobilverkehr Frutigen Adelboden AG
AMSA Autolinea Mendrisiense SA
AOT Autokurse Oberthurgau AG
ARAG Rottal Auto AG
ARBAG Aletsch Riederalp Bahnen AG
ARL Autolinee Regionali Luganesi
AS Autobetrieb Sernftal AG
ASGS Autotransports Sion-Grône-Sierre
ASm Aare Seeland mobil AG
AVG Autoverkehr Grindelwald AG
AVJ Autotransports de la Vallée de Joux
AWA Autobetrieb Weesen-Amden
AZZK Autobus Zürich-Zollikon-Küsnacht
BB Bürgenstock Bahnen
BBA Busbetrieb Aarau AAR bus+bahn
BBBW Bus-Betrieb Binggeli
BDWM BDWM Transport AG
BGU BGU Busbetrieb Grenchen und Umgebung AG
BLAG Busland AG
BLM Bergbahn Lauterbrunnen-Mürren AG
BLS BLS AG
BLT BLT Baselland Transport AG
BLWE Busbetrieb Lichtensteig-Wattwil-Ebnat-Kappel
BOB Berner Oberland-Bahnen AG
BOGG Busbetrieb Olten Gösgen Gäu AG
BOS BUS Ostschweiz AG
BOS-M BOS Management AG
BRB Brienz Rothorn Bahn AG
BRER Busbetrieb Rapperswil-Eschenbach-Rüti
BRSB Braunwald-Standseilbahn AG
BSU Busbetrieb Solothurn und Umgebung AG
BVB Basler Verkehrs-Betriebe
CGN CGN SA
CJ Compagnie des chemins de fer du Jura (C.J.) SA
CROS Crossrail AG
DBSCH DB Schenker Rail Schweiz GmbH
DBZ Dolderbahn Zürich
ETB Emmentalbahn, Huttwil
FART Ferrovie Autolinee Regionali Ticinesi
FB Forchbahn AG
FC FUNICAR Kursbetriebe AG
FLP Ferrovie Luganesi SA
FW Frauenfeld-Wil-Bahn AG
GGB Gornergrat Bahn AG HBSAG Hafenbahn Schweiz AG
JB Jungfraubahn AG
LEB Chemin de fer Lausanne-Echallens-Bercher
LLB AG für Verkehrsbetriebe Leuk-Leukerbad und Umgebung
LSMS Schilthornbahn AG
MBC Transports de la région Morges-Bière-Cossonay SA
MG Ferrovia Monte Generoso SA
MGB Matterhorn Gotthard Bahn
MIB Kraftwerke Oberhasli AG Meiringen-Innertkirchen-Bahn
MOB Chemin de fer Montreux-Oberland Bernois
MVR Transports Montreux-Vevey-Riviera SA
NHB Niederhornbahn
NB Niesenbahn AG
NStCM Chemin de fer Nyon-St. Cergue-Morez
OeBB Oensingen-Balsthal-Bahn
PAG PostAuto Schweiz AG
PB PILATUS-BAHNEN AG
RA RegionAlps SA
RAILG Railgate AG
RB RIGI BAHNEN AG
RBL Regionalbus Lenzburg AG
RBS Regionalverkehr Bern-Solothurn AG
REGO Regiobus Gossau AG
RhB Rhätische Bahn AG
RNCH DB Schenker Rail Schweiz GmbH
RLC railCare
RVBW Regionale Verkehrsbetriebe Baden-Wettingen AG
RVSH SchaffhausenBus, Regionale Verkehrsbetriebe SH AG
SBB SBB AG
SBB-D SBB GmbH
SBC Stadtbus Chur AG
SBF Stadtbus Frauenfeld
SBW Stadtbus Winterthur
SMC Cie de Chemin de Fer+d'Autobus Sierre-Montana-Crans (SMC) SA
SMGN Société des Mouettes Genevoises Navigation SA
SMtS Funiculaire St-Imier - Mont-Soleil SA
SOB Schweizerische Südostbahn AG
SRTAG Swiss Rail Traffic AG
SSIF Società Subalpina di Imprese Ferroviarie S.p.A.
ST Sursee-Triengen-Bahn
STB Sensetalbahn AG
STI Verkehrsbetriebe STI AG
SVB BERNMOBIL Städt. Verkehrsbetriebe Bern
SWAG Seilbahn Weissenstein AG
SZU Sihltal Zürich Uetliberg Bahn SZU AG
THURBO Thurbo AG
TL Transports publics de la région lausannoise SA
TMR TRANSPORTS DE MARTIGNY ET REGIONS SA
TPC Transports Publics du Chablais SA
TPF Transports publics fribourgeois SA
TPG Transports publics genevois
TPL Trasporti Pubblici Luganesi SA
TPN Transports Publics de la Région Nyonnaise SA
TRN Transports Publics Neuchâtelois SA
TRAVYS TRAVYS SA Transports Vallée de Joux-Yverdon-Sainte-Croix
TSD Theytaz Excursions Sion
VB Verkehrsbetriebe Biel
VBD Verkehrsbetrieb der Landschaft Davos
VBG VBG Verkehrsbetriebe Glattal AG
VBH Verkehrsbetriebe Herisau
VBL Verkehrsbetriebe Luzern AG
VBSG Verkehrsbetriebe St.Gallen
VBSH Verkehrsbetriebe Schaffhausen
VBZ Verkehrsbetriebe Zürich
VMCV Transports publics Vevey-Montreux-Chillon-Villeneuve
VSSU Verband Schweizerischer Schifffahrtsunternehmen
VZO Verkehrsbetriebe Zürichsee und Oberland AG
WAB Wengernalpbahn AG
WB Waldenburgerbahn AG
WRS Widmer Rail Services Personal AG
WSB Wynental- und Suhrentalbahn AAR bus+bahn
ZB zb Zentralbahn AG
ZVB Zugerland Verkehrsbetriebe AG
ZVV Zürcher Verkehrsverbund ZVV
AES Ägerisee Schifffahrt AG
BLS BLS AG Schifffahrt Berner Oberland Thuner- und Brienzersee
BPG Basler Personenschifffahrt AG
BSG Bielersee-Schifffahrts-Gesellschaft AG
CGN CGN SA
FHM Zürichsee-Fähre Horgen-Meilen AG
LNM Société de Navigation Lacs de Neuchâtel et Morat SA
NLM Navigazione Lago Maggiore
SBS SBS Schifffahrt AG
SGG Schifffahrts-Genossenschaft Greifensee
SGH Schifffahrtsgesellschaft Hallwilersee AG
SGV Schifffahrtsgesellschaft des Vierwaldstättersees
SGZ Schifffahrtsgesellschaft für den Zugersee AG / Ägerisee
SNL Società Navigazione del Lago di Lugano SA
SW Schiffsbetrieb Walensee AG
URh Schweiz. Schifffahrtsgesellschaft Untersee und Rhein AG
ZSG Zürichsee-Schifffahrtsgesellschaft AG
12.
13. What do we propose?
https://github.com/alexmasselot/swiss-transport-realtime
14.
15. Is it possible to build
a simple scalable infrastructure, to
dispatch, transform and visualize
“near real time” massive data
and achieve a posteriori analysis?
17. Is it possible to build
a simple scalable infrastructure, to
dispatch, transform and visualize
“near real time” massive data
and achieve a posteriori analysis?
21. Is it possible to build
a simple scalable infrastructure, to
dispatch, transform and visualize
“near real time” massive data
and achieve a posteriori analysis?
26. Events are streamed to
“Kafka is used for building real-
time data pipelines and
streaming apps. It is horizontally
scalable, fault-tolerant, wicked
fast, and runs in production in
thousands of companies.”
kafka.apache.org
real time
offline
32. Is it possible to build
a simple scalable infrastructure, to
dispatch, transform and visualize
“near real time” massive data
and achieve a posteriori analysis?
33. Stream transformation
• We have an input flow of events and want to:
• know if a train is stopped into a station;
• know if a train as exited the network;
• expose an aggregated station board.
• We need to:
• digest the input flow;
• process with temporary state persistance;
• be able to expose snapshots.
34. Stream transformation
• Scala is The language for Big Data (functional & OO)
• Akka (actors):
• lightweight entities (one per train, per station);
• easy asynchronous communications;
• the perfect use case.
• Play framework for REST service, configuration etc.
37. : putting everything together
• The “simple” infrastructure is not so light;
• A developper should have everything on his/her
laptop without polluting the machine;
• Docker comes to the rescue:
• lightweight containers,
• pre-existing images,
• docker-compose to describe the infrastructure
• deploy directly to a cloud.
39. Performance: 2 numbers
15% CPU: nodeJS + kafka + akka + play
15x faster ajax queries (vs SBB rest)
to gather 30 times more trains
40. Is it possible to build
a simple scalable infrastructure, to
dispatch, transform and visualize
“near real time” massive data
and achieve a posteriori analysis?
41. A scalable infrastructure
Kafka partitioning and zookeeper
Logstash ? (but naturally recover on failure)
Elasticsearch partitioning
Spark streaming
distributed by essence
& write ahead logs
Akka
aka cluster, supervisors
& failure strategy
Docker
Kubernetes
AWS, GCE, Exoscale, Hidora
43. Is it possible to build
a simple scalable infrastructure, to
dispatch, transform and visualize
“near real time” massive data
and achieve a posteriori analysis?
44.
45.
46. JS for large data set
• Only a rendering library (but fast);
• Use a flux architecture;
• Built by Facebook. Dispatcher
Store
View
Action
Action
47. JavaScript for big data viz
• React can handle viz >100k elements (don’t show
them individually!)
• Beware of performance issue;
• Testing is not an option.
49. Is it possible to build
a simple scalable infrastructure, to
dispatch, transform and visualize
“near real time” massive data
and achieve a posteriori analysis?
50. 4.5 months of data
A. What is the train occupancy during weekdays,
between Lausanne and Geneva?
B. When are the train the most delayed?
C. Where are the train the most delayed?
64. • Web application
• Interactively edit and run pieces of code (analysis
steps)
• Inclined towards Python (although other languages
are available)
• Beware of performance with large dataset (sample
data or use Spark mode)
a data science notebook
68. Swiss transport in real time,
is that only the beginning?
• Bus & trains dispatch their actual positions in real time
• High availability & scalability
• Performance in the browser
• Better long term storage
• More data analysis questions (what’s yours?)
• Don’t forget to have fun!
https://github.com/alexmasselot/swiss-transport-realtime
@alex_mass
This is only
a POC!!!