Streamly is a Haskell library for writing programs in a high level, declarative data flow programming paradigm. It provides a simple API, very close to standard Haskell lists. A program is expressed as a composition of data processing pipes, generally known as streams. Streams can be generated, merged, chained, mapped, zipped, and consumed concurrently – enabling a high level, declarative yet concurrent composition of programs. Programs can be concurrent or non-concurrent without any significant change. Concurrency is auto scaled based on consumption rate. Programmers do not have to be aware of threads, locking or synchronization to write scalable concurrent programs. Streamly provides C like performance, orders of magnitude better compared to existing streaming libraries.
Driving Overall Equipment Effectiveness with AWS IoT SiteWise - SVC213 - Chic...Amazon Web Services
Gathering performance metrics such as Overall Equipment Effectiveness (OEE) from industrial equipment is difficult because data is often locked into proprietary on-premises data stores and typically requires specialized expertise to retrieve and put into a useful format for searching and analysis. In this session, we show you how AWS IoT SiteWise makes it easy to collect and organize data from industrial equipment. We show you how to compute common industrial performance metrics, build applications to analyze industrial equipment data, prevent costly equipment issues, and reduce production inefficiencies.
Chaos Testing with F# and Azure by Rachel Reese at Codemotion DubaiCodemotion Dubai
Some of the biggest growing pains we've experienced with our microservice architecture at Jet is in preparing for system outages. I'll cover the benefits of choosing F# specifically, and functional programming in general, as well as Azure, for a chaos program. I'll follow up with a discussion of why your team needs to implement a chaos testing program, and finish with showing off our methods and code in depth.
Why you care about relational algebra (even though you didn’t know it)Julian Hyde
A talk given by Julian Hyde at Enterprise Data World on in Washington, DC on April 2nd, 2015.
With data in different systems, in different formats, and accessed via different tools, we need a lingua franca for data. Not all tools speak SQL, and data cannot be moved into a single convenient location.
Relational algebra underpins SQL and many other DB languages. It is also perfect for optimizing, caching and mediating.
Apache Calcite (formerly Optiq) is a framework for building and optimizing expressions in relational algebra. We show how to write queries, optimize queries using rewrite rules, and write adapters for back-end systems. We also show to configure Calcite to materialize queries, so your interactive analytics are effectively running against a fast in-memory database.
Swift: Apple's New Programming Language for iOS and OS XSasha Goldshtein
Presentation from Software Architect 2014, covering Swift -- Apple's new programming language for iOS and OS X. The presentation focuses on Swift's language features, which make it so different from mainstream programming languages. Towards the end of the live talk, we also built an iOS app using Swift.
ClickHouse tips and tricks. Webinar slides. By Robert Hodges, Altinity CEOAltinity Ltd
From webinar on December 3, 2019
New users of ClickHouse love the speed but may run into a few surprises when designing applications. Column storage turns classic SQL design precepts on their heads. This talk shares our favorite tricks for building great applications. We'll talk about fact tables and dimensions, materialized views, codecs, arrays, and skip indexes, to name a few of our favorites. We'll show examples of each and also reserve time to handle questions. Join us to take your next step to ClickHouse guruhood!
Speaker Bio:
Robert Hodges is CEO of Altinity, which offers enterprise support for ClickHouse. He has over three decades of experience in data management spanning 20 different DBMS types. ClickHouse is his current favorite. ;)
Michael DeSa will go over some of the advanced topics in Kapacitor such as joins, templated tasks, and debugging your tasks. Prerequisite: Intro To Kapacitor.
At Unite Copenhagen, we presented the latest and greatest from R&D in the traditional form of a roadmap talk. We talked about what’s coming in Unity 2019.3 and 2020.1, and presented our vision for what you should expect in 2020.
Video: https://www.youtube.com/watch?v=hRJUt3voDHQ
What to expect in 2020: Unity roadmap - Unite Copenhagen 2019Unity Technologies
Hear about Unity's roadmap for 2020 and get an overview of Unity's 3-year vision. Learn about upcoming developments for the Data-Oriented Technology Stack (DOTS), graphics, artist tooling, connected games and much more.
Speaker: Will Goldstone- Unity
Watch the session on Youtube: https://youtu.be/hRJUt3voDHQ
InfluxDB IOx Tech Talks: A Rusty Introduction to Apache Arrow and How it App...InfluxData
InfluxDB IOx Tech Talks - December 2020
A Rusty Introduction to Apache Arrow and How it Applies to a Time Series Database
This session will start with a tech talk from an InfluxDB IOx team member. This is your chance to interact directly with Influxers who are available to answer your questions about all things InfluxDB IOx and time series — including Paul Dix, Founder and CTO of InfluxData. This event will last about an hour and there will be time for live Q&A.
Enhancing Research Orchestration Capabilities at ORNL.pdfGlobus
Cross-facility research orchestration comes with ever-changing constraints regarding the availability and suitability of various compute and data resources. In short, a flexible data and processing fabric is needed to enable the dynamic redirection of data and compute tasks throughout the lifecycle of an experiment. In this talk, we illustrate how we easily leveraged Globus services to instrument the ACE research testbed at the Oak Ridge Leadership Computing Facility with flexible data and task orchestration capabilities.
Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...Globus
The Earth System Grid Federation (ESGF) is a global network of data servers that archives and distributes the planet’s largest collection of Earth system model output for thousands of climate and environmental scientists worldwide. Many of these petabyte-scale data archives are located in proximity to large high-performance computing (HPC) or cloud computing resources, but the primary workflow for data users consists of transferring data, and applying computations on a different system. As a part of the ESGF 2.0 US project (funded by the United States Department of Energy Office of Science), we developed pre-defined data workflows, which can be run on-demand, capable of applying many data reduction and data analysis to the large ESGF data archives, transferring only the resultant analysis (ex. visualizations, smaller data files). In this talk, we will showcase a few of these workflows, highlighting how Globus Flows can be used for petabyte-scale climate analysis.
More Related Content
Similar to Streamly: Concurrent Data Flow Programming
Driving Overall Equipment Effectiveness with AWS IoT SiteWise - SVC213 - Chic...Amazon Web Services
Gathering performance metrics such as Overall Equipment Effectiveness (OEE) from industrial equipment is difficult because data is often locked into proprietary on-premises data stores and typically requires specialized expertise to retrieve and put into a useful format for searching and analysis. In this session, we show you how AWS IoT SiteWise makes it easy to collect and organize data from industrial equipment. We show you how to compute common industrial performance metrics, build applications to analyze industrial equipment data, prevent costly equipment issues, and reduce production inefficiencies.
Chaos Testing with F# and Azure by Rachel Reese at Codemotion DubaiCodemotion Dubai
Some of the biggest growing pains we've experienced with our microservice architecture at Jet is in preparing for system outages. I'll cover the benefits of choosing F# specifically, and functional programming in general, as well as Azure, for a chaos program. I'll follow up with a discussion of why your team needs to implement a chaos testing program, and finish with showing off our methods and code in depth.
Why you care about relational algebra (even though you didn’t know it)Julian Hyde
A talk given by Julian Hyde at Enterprise Data World on in Washington, DC on April 2nd, 2015.
With data in different systems, in different formats, and accessed via different tools, we need a lingua franca for data. Not all tools speak SQL, and data cannot be moved into a single convenient location.
Relational algebra underpins SQL and many other DB languages. It is also perfect for optimizing, caching and mediating.
Apache Calcite (formerly Optiq) is a framework for building and optimizing expressions in relational algebra. We show how to write queries, optimize queries using rewrite rules, and write adapters for back-end systems. We also show to configure Calcite to materialize queries, so your interactive analytics are effectively running against a fast in-memory database.
Swift: Apple's New Programming Language for iOS and OS XSasha Goldshtein
Presentation from Software Architect 2014, covering Swift -- Apple's new programming language for iOS and OS X. The presentation focuses on Swift's language features, which make it so different from mainstream programming languages. Towards the end of the live talk, we also built an iOS app using Swift.
ClickHouse tips and tricks. Webinar slides. By Robert Hodges, Altinity CEOAltinity Ltd
From webinar on December 3, 2019
New users of ClickHouse love the speed but may run into a few surprises when designing applications. Column storage turns classic SQL design precepts on their heads. This talk shares our favorite tricks for building great applications. We'll talk about fact tables and dimensions, materialized views, codecs, arrays, and skip indexes, to name a few of our favorites. We'll show examples of each and also reserve time to handle questions. Join us to take your next step to ClickHouse guruhood!
Speaker Bio:
Robert Hodges is CEO of Altinity, which offers enterprise support for ClickHouse. He has over three decades of experience in data management spanning 20 different DBMS types. ClickHouse is his current favorite. ;)
Michael DeSa will go over some of the advanced topics in Kapacitor such as joins, templated tasks, and debugging your tasks. Prerequisite: Intro To Kapacitor.
At Unite Copenhagen, we presented the latest and greatest from R&D in the traditional form of a roadmap talk. We talked about what’s coming in Unity 2019.3 and 2020.1, and presented our vision for what you should expect in 2020.
Video: https://www.youtube.com/watch?v=hRJUt3voDHQ
What to expect in 2020: Unity roadmap - Unite Copenhagen 2019Unity Technologies
Hear about Unity's roadmap for 2020 and get an overview of Unity's 3-year vision. Learn about upcoming developments for the Data-Oriented Technology Stack (DOTS), graphics, artist tooling, connected games and much more.
Speaker: Will Goldstone- Unity
Watch the session on Youtube: https://youtu.be/hRJUt3voDHQ
InfluxDB IOx Tech Talks: A Rusty Introduction to Apache Arrow and How it App...InfluxData
InfluxDB IOx Tech Talks - December 2020
A Rusty Introduction to Apache Arrow and How it Applies to a Time Series Database
This session will start with a tech talk from an InfluxDB IOx team member. This is your chance to interact directly with Influxers who are available to answer your questions about all things InfluxDB IOx and time series — including Paul Dix, Founder and CTO of InfluxData. This event will last about an hour and there will be time for live Q&A.
Enhancing Research Orchestration Capabilities at ORNL.pdfGlobus
Cross-facility research orchestration comes with ever-changing constraints regarding the availability and suitability of various compute and data resources. In short, a flexible data and processing fabric is needed to enable the dynamic redirection of data and compute tasks throughout the lifecycle of an experiment. In this talk, we illustrate how we easily leveraged Globus services to instrument the ACE research testbed at the Oak Ridge Leadership Computing Facility with flexible data and task orchestration capabilities.
Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...Globus
The Earth System Grid Federation (ESGF) is a global network of data servers that archives and distributes the planet’s largest collection of Earth system model output for thousands of climate and environmental scientists worldwide. Many of these petabyte-scale data archives are located in proximity to large high-performance computing (HPC) or cloud computing resources, but the primary workflow for data users consists of transferring data, and applying computations on a different system. As a part of the ESGF 2.0 US project (funded by the United States Department of Energy Office of Science), we developed pre-defined data workflows, which can be run on-demand, capable of applying many data reduction and data analysis to the large ESGF data archives, transferring only the resultant analysis (ex. visualizations, smaller data files). In this talk, we will showcase a few of these workflows, highlighting how Globus Flows can be used for petabyte-scale climate analysis.
Quarkus Hidden and Forbidden ExtensionsMax Andersen
Quarkus has a vast extension ecosystem and is known for its subsonic and subatomic feature set. Some of these features are not as well known, and some extensions are less talked about, but that does not make them less interesting - quite the opposite.
Come join this talk to see some tips and tricks for using Quarkus and some of the lesser known features, extensions and development techniques.
How to Position Your Globus Data Portal for Success Ten Good PracticesGlobus
Science gateways allow science and engineering communities to access shared data, software, computing services, and instruments. Science gateways have gained a lot of traction in the last twenty years, as evidenced by projects such as the Science Gateways Community Institute (SGCI) and the Center of Excellence on Science Gateways (SGX3) in the US, The Australian Research Data Commons (ARDC) and its platforms in Australia, and the projects around Virtual Research Environments in Europe. A few mature frameworks have evolved with their different strengths and foci and have been taken up by a larger community such as the Globus Data Portal, Hubzero, Tapis, and Galaxy. However, even when gateways are built on successful frameworks, they continue to face the challenges of ongoing maintenance costs and how to meet the ever-expanding needs of the community they serve with enhanced features. It is not uncommon that gateways with compelling use cases are nonetheless unable to get past the prototype phase and become a full production service, or if they do, they don't survive more than a couple of years. While there is no guaranteed pathway to success, it seems likely that for any gateway there is a need for a strong community and/or solid funding streams to create and sustain its success. With over twenty years of examples to draw from, this presentation goes into detail for ten factors common to successful and enduring gateways that effectively serve as best practices for any new or developing gateway.
Top Features to Include in Your Winzo Clone App for Business Growth (4).pptxrickgrimesss22
Discover the essential features to incorporate in your Winzo clone app to boost business growth, enhance user engagement, and drive revenue. Learn how to create a compelling gaming experience that stands out in the competitive market.
Navigating the Metaverse: A Journey into Virtual Evolution"Donna Lenk
Join us for an exploration of the Metaverse's evolution, where innovation meets imagination. Discover new dimensions of virtual events, engage with thought-provoking discussions, and witness the transformative power of digital realms."
Understanding Globus Data Transfers with NetSageGlobus
NetSage is an open privacy-aware network measurement, analysis, and visualization service designed to help end-users visualize and reason about large data transfers. NetSage traditionally has used a combination of passive measurements, including SNMP and flow data, as well as active measurements, mainly perfSONAR, to provide longitudinal network performance data visualization. It has been deployed by dozens of networks world wide, and is supported domestically by the Engagement and Performance Operations Center (EPOC), NSF #2328479. We have recently expanded the NetSage data sources to include logs for Globus data transfers, following the same privacy-preserving approach as for Flow data. Using the logs for the Texas Advanced Computing Center (TACC) as an example, this talk will walk through several different example use cases that NetSage can answer, including: Who is using Globus to share data with my institution, and what kind of performance are they able to achieve? How many transfers has Globus supported for us? Which sites are we sharing the most data with, and how is that changing over time? How is my site using Globus to move data internally, and what kind of performance do we see for those transfers? What percentage of data transfers at my institution used Globus, and how did the overall data transfer performance compare to the Globus users?
OpenMetadata Community Meeting - 5th June 2024OpenMetadata
The OpenMetadata Community Meeting was held on June 5th, 2024. In this meeting, we discussed about the data quality capabilities that are integrated with the Incident Manager, providing a complete solution to handle your data observability needs. Watch the end-to-end demo of the data quality features.
* How to run your own data quality framework
* What is the performance impact of running data quality frameworks
* How to run the test cases in your own ETL pipelines
* How the Incident Manager is integrated
* Get notified with alerts when test cases fail
Watch the meeting recording here - https://www.youtube.com/watch?v=UbNOje0kf6E
We describe the deployment and use of Globus Compute for remote computation. This content is aimed at researchers who wish to compute on remote resources using a unified programming interface, as well as system administrators who will deploy and operate Globus Compute services on their research computing infrastructure.
Atelier - Innover avec l’IA Générative et les graphes de connaissancesNeo4j
Atelier - Innover avec l’IA Générative et les graphes de connaissances
Allez au-delà du battage médiatique autour de l’IA et découvrez des techniques pratiques pour utiliser l’IA de manière responsable à travers les données de votre organisation. Explorez comment utiliser les graphes de connaissances pour augmenter la précision, la transparence et la capacité d’explication dans les systèmes d’IA générative. Vous partirez avec une expérience pratique combinant les relations entre les données et les LLM pour apporter du contexte spécifique à votre domaine et améliorer votre raisonnement.
Amenez votre ordinateur portable et nous vous guiderons sur la mise en place de votre propre pile d’IA générative, en vous fournissant des exemples pratiques et codés pour démarrer en quelques minutes.
Introducing Crescat - Event Management Software for Venues, Festivals and Eve...Crescat
Crescat is industry-trusted event management software, built by event professionals for event professionals. Founded in 2017, we have three key products tailored for the live event industry.
Crescat Event for concert promoters and event agencies. Crescat Venue for music venues, conference centers, wedding venues, concert halls and more. And Crescat Festival for festivals, conferences and complex events.
With a wide range of popular features such as event scheduling, shift management, volunteer and crew coordination, artist booking and much more, Crescat is designed for customisation and ease-of-use.
Over 125,000 events have been planned in Crescat and with hundreds of customers of all shapes and sizes, from boutique event agencies through to international concert promoters, Crescat is rigged for success. What's more, we highly value feedback from our users and we are constantly improving our software with updates, new features and improvements.
If you plan events, run a venue or produce festivals and you're looking for ways to make your life easier, then we have a solution for you. Try our software for free or schedule a no-obligation demo with one of our product specialists today at crescat.io
Software Engineering, Software Consulting, Tech Lead, Spring Boot, Spring Cloud, Spring Core, Spring JDBC, Spring Transaction, Spring MVC, OpenShift Cloud Platform, Kafka, REST, SOAP, LLD & HLD.
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I ...Juraj Vysvader
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I didn't get rich from it but it did have 63K downloads (powered possible tens of thousands of websites).
First Steps with Globus Compute Multi-User EndpointsGlobus
In this presentation we will share our experiences around getting started with the Globus Compute multi-user endpoint. Working with the Pharmacology group at the University of Auckland, we have previously written an application using Globus Compute that can offload computationally expensive steps in the researcher's workflows, which they wish to manage from their familiar Windows environments, onto the NeSI (New Zealand eScience Infrastructure) cluster. Some of the challenges we have encountered were that each researcher had to set up and manage their own single-user globus compute endpoint and that the workloads had varying resource requirements (CPUs, memory and wall time) between different runs. We hope that the multi-user endpoint will help to address these challenges and share an update on our progress here.
Listen to the keynote address and hear about the latest developments from Rachana Ananthakrishnan and Ian Foster who review the updates to the Globus Platform and Service, and the relevance of Globus to the scientific community as an automation platform to accelerate scientific discovery.