Hands-On: Managing Slowly Changing Dimensions Using TD WorkflowTreasure Data, Inc.
In this hands-on webinar we'll explore the data warehousing concept of Slowly Changing Dimensions (SCDs) and common use cases for managing SCDs when dealing with customer data. This webinar will demonstrate different methods for tracking SCDs in a data warehouse, and how Treasure Data Workflow can be used to create robust data pipelines to handle these processes.
Webinar: Introducing the MongoDB Connector for BI 2.0 with TableauMongoDB
Pairing your real-time operational data stored in a modern database like MongoDB with first-class business intelligence platforms like Tableau enables new insights to be discovered faster than ever before.
Many leading organizations already use MongoDB in conjunction with Tableau including a top American investment bank and the world’s largest airline. With the Connector for BI 2.0, it’s never been easier to streamline the connection process between these two systems.
In this webinar, we will create a live connection from Tableau Desktop to a MongoDB cluster using the Connector for BI. Once we have Tableau Desktop and MongoDB connected, we will demonstrate the visual power of Tableau to explore the agile data storage of MongoDB.
You’ll walk away knowing:
- How to configure MongoDB with Tableau using the updated connector
- Best practices for working with documents in a BI environment
- How leading companies are using big data visualization strategies to transform their businesses
Webinar: Live Data Visualisation with Tableau and MongoDBMongoDB
MongoDB 3.2 introduces a new way for familiar Business Intelligence (BI) tools to access your real-time operational data – opening it up to data analysts and data scientist, enabling new insights to be discovered faster than ever before. Tableau accesses the JSON document data stored in MongoDB via this new BI connector. We will cover how the BI connector works by creating a relational view definition of a JSON data set that is then used to present a tabular SQL/ODBC interface to Tableau. Then we will set-up a live connection from Tableau Desktop to the MongoDB Connector for BI. Once we have Tableau Desktop and MongoDB connected, we will demonstrate the visual power of Tableau to explore the agile data storage of MongoDB. This webinar will cover:
What is the MongoDB BI Connector?
Setting up a connection from Tableau to the MongoDB BI Connector.
How to perform data discovery Tableau connected to MongoDB live data.
Publishing a Tableau Dashboard for sharing insights.
MongoDB has been conceived for the cloud age. Making sure that MongoDB is compatible and performant around cloud providers is mandatory to achieve complete integration with platforms and systems. Azure is one of biggest IaaS platforms available and very popular amongst developers that work on Microsoft Stack.
Analyze and visualize non-relational data with DocumentDB + Power BISriram Hariharan
The session will show how to do Analyze and visualize non-relational data with DocumentDB + Power BI. We are in the midst of a paradigm shift on how we store and analyze data. Unstructured or flexible schema data represents a large portion of data within an organization. Everyone is obsessed to turn this data into meaningful business information. Unstructured data analytics do not need to be time consuming and complex. Come learn how to analyze and visualize unstructured data in DocumentDB.
Hands-On: Managing Slowly Changing Dimensions Using TD WorkflowTreasure Data, Inc.
In this hands-on webinar we'll explore the data warehousing concept of Slowly Changing Dimensions (SCDs) and common use cases for managing SCDs when dealing with customer data. This webinar will demonstrate different methods for tracking SCDs in a data warehouse, and how Treasure Data Workflow can be used to create robust data pipelines to handle these processes.
Webinar: Introducing the MongoDB Connector for BI 2.0 with TableauMongoDB
Pairing your real-time operational data stored in a modern database like MongoDB with first-class business intelligence platforms like Tableau enables new insights to be discovered faster than ever before.
Many leading organizations already use MongoDB in conjunction with Tableau including a top American investment bank and the world’s largest airline. With the Connector for BI 2.0, it’s never been easier to streamline the connection process between these two systems.
In this webinar, we will create a live connection from Tableau Desktop to a MongoDB cluster using the Connector for BI. Once we have Tableau Desktop and MongoDB connected, we will demonstrate the visual power of Tableau to explore the agile data storage of MongoDB.
You’ll walk away knowing:
- How to configure MongoDB with Tableau using the updated connector
- Best practices for working with documents in a BI environment
- How leading companies are using big data visualization strategies to transform their businesses
Webinar: Live Data Visualisation with Tableau and MongoDBMongoDB
MongoDB 3.2 introduces a new way for familiar Business Intelligence (BI) tools to access your real-time operational data – opening it up to data analysts and data scientist, enabling new insights to be discovered faster than ever before. Tableau accesses the JSON document data stored in MongoDB via this new BI connector. We will cover how the BI connector works by creating a relational view definition of a JSON data set that is then used to present a tabular SQL/ODBC interface to Tableau. Then we will set-up a live connection from Tableau Desktop to the MongoDB Connector for BI. Once we have Tableau Desktop and MongoDB connected, we will demonstrate the visual power of Tableau to explore the agile data storage of MongoDB. This webinar will cover:
What is the MongoDB BI Connector?
Setting up a connection from Tableau to the MongoDB BI Connector.
How to perform data discovery Tableau connected to MongoDB live data.
Publishing a Tableau Dashboard for sharing insights.
MongoDB has been conceived for the cloud age. Making sure that MongoDB is compatible and performant around cloud providers is mandatory to achieve complete integration with platforms and systems. Azure is one of biggest IaaS platforms available and very popular amongst developers that work on Microsoft Stack.
Analyze and visualize non-relational data with DocumentDB + Power BISriram Hariharan
The session will show how to do Analyze and visualize non-relational data with DocumentDB + Power BI. We are in the midst of a paradigm shift on how we store and analyze data. Unstructured or flexible schema data represents a large portion of data within an organization. Everyone is obsessed to turn this data into meaningful business information. Unstructured data analytics do not need to be time consuming and complex. Come learn how to analyze and visualize unstructured data in DocumentDB.
MongoDB .local Munich 2019: MongoDB Atlas Data Lake Technical Deep DiveMongoDB
MongoDB Atlas Data Lake is a new service offered by MongoDB Atlas. Many organizations store long term, archival data in cost-effective storage like S3, GCP, and Azure Blobs. However, many of them do not have robust systems or tools to effectively utilize large amounts of data to inform decision making. MongoDB Atlas Data Lake is a service allowing organizations to analyze their long-term data to discover a wealth of information about their business.
This session will take a deep dive into the features that are currently available in MongoDB Atlas Data Lake and how they are implemented. In addition, we'll discuss future plans and opportunities and offer ample Q&A time with the engineers on the project.
MongoDB Atlas Autoscaling automatically changes both the storage and compute capacity of your MongoDB Atlas cluster, in response to changing traffic patterns. This enables MongoDB Atlas to continuously maximize performance while minimizing cost, with just a press of a button. Plan to attend this session and learn more about how autoscaling works behind the scenes, and the best ways to use it.
Rapid Development and Performance By Transitioning from RDBMSs to MongoDB
Modern day application requirements demand rich & dynamic data structures, fast response times, easy scaling, and low TCO to match the rapidly changing customer & business requirements plus the powerful programming languages used in today's software landscape.
Traditional approaches to solutions development with RDBMSs increasingly expose the gap between the modern development languages and the relational data model, and between scaling up vs. scaling horizontally on commodity hardware. Development time is wasted as the bulk of the work has shifted from adding business features to struggling with the RDBMSs.
MongoDB, the premier NoSQL database, offers a flexible and scalable solution to focus on quickly adding business value again.
In this session, we will provide:
- Overview of MongoDB's capabilities
- Code-level exploration of the MongoDB programming model and APIs and how they transform the way developers interact with a database
- Update of the exciting features in MongoDB 3.0
Modern architectures are moving away from a "one size fits all" approach. We are well aware that we need to use the best tools for the job. Given the large selection of options available today, chances are that you will end up managing data in MongoDB for your operational workload and with Spark for your high speed data processing needs.
Description: When we model documents or data structures there are some key aspects that need to be examined not only for functional and architectural purposes but also to take into consideration the distribution of data nodes, streaming capabilities, aggregation and queryability options and how we can integrate the different data processing software, like Spark, that can benefit from subtle but substantial model changes. A clear example is when embedding or referencing documents and their implications on high speed processing.
Over the course of this talk we will detail the benefits of a good document model for the operational workload. As well as what type of transformations we should incorporate in our document model to adjust for the high speed processing capabilities of Spark.
We will look into the different options that we have to connect these two different systems, how to model according to different workloads, what kind of operators we need to be aware of for top performance and what kind of design and architectures we should put in place to make sure that all of these systems work well together.
Over the course of the talk we will showcase different libraries that enable the integration between spark and MongoDB, such as MongoDB Hadoop Connector, Stratio Connector and MongoDB Spark Native Connector.
By the end of the talk I expect the attendees to have an understanding of:
How they connect their MongoDB clusters with Spark
Which use cases show a net benefit for connecting these two systems
What kind of architecture design should be considered for making the most of Spark + MongoDB
How documents can be modeled for better performance and operational process, while processing these data sets stored in MongoDB.
The talk is suitable for:
Developers that want to understand how to leverage Spark
Architects that want to integrate their existing MongoDB cluster and have real time high speed processing needs
Data scientists that know about Spark, are playing with Spark and want to integrate with MongoDB for their persistency layer
During this presentation, Infusion and MongoDB shared their mainframe optimization experiences and best practices. These have been gained from working with a variety of organizations, including a case study from one of the world’s largest banks. MongoDB and Infusion bring a tested approach that provides a new way of modernizing mainframe applications, while keeping pace with the demand for new digital services.
Improve your SQL workload with observabilityOVHcloud
La majeure partie du SI d'OVH repose sur des bases de données relationnelles (PostgreSQL, MySQL, MariaDB). En termes de volumétrie cela représente 400 bases pesants plus de 20To de données réparties sur 60 clusters dans deux zones géographiques le tout propulsant 3000 applications.
Comment tout voir dans notre parc ? Mieux encore, comment faire pour que tout le monde puisse suivre l'activité de sa base de données ? C'est le challenge que nous nous sommes fixés, un an après nous pouvons partager notre expérience.
Et si l'observability n'était pas juste un buzzword, mais avait un réel impact sur la production ?
MongoDB .local Munich 2019: Managing a Heterogeneous Stack with MongoDB & SQLMongoDB
Data administrators face the challenge of integrating disparate data technologies into a cohesive and performant data platform. This is especially true when using diverse query languages and protocols. This session will focus on how to integrate SQL-aware applications into a MongoDB data platform.
In this hands-on webinar we will cover how to leverage the Treasure Data Javascript SDK library to ensure user stitching of web data into the Treasure Data Customer Data Platform to provide a holistic view of prospects and customers.
We will demo the native SDK, as well as deploying the SDK inside of Adobe DTM and Google Tag Manager.
Doing Joins in MongoDB: Best Practices for Using $lookupMongoDB
Speaker: Austin Zellner, Solutions Architect, MongoDB
Level: 200 (Intermediate)
Track: Data Analytics
$lookup is a pipeline stage in the aggregation framework that performs a left outer join. In this session, you will learn how to leverage $lookup in your applications and best practices for implementing features with $lookup.
What You Will Learn:
- Fundamentals of $lookup and its syntax.
- How to use $lookup stages in your aggregation pipelines.
- Best practices for using $lookup to implement application features.
Agile Software Development is becoming the defacto way of building software these days. More and more enterprises, from large fortune 500 to small shop start-ups, are adopting agile development methodologies. But Agile Software development is more than just a methodology or a practice. It's also a combined set of tools and platforms that today are at our disposal to allows to iterate faster, get-to-market sooner and also fail faster. These set of tools augment our development cycles by a few orders of magnitude and allow developers to be much more productive.
How to get the best of both: MongoDB is great for low latency quick access of recent data; Treasure Data is great for infinitely growing store of historical data. In the latter case, one need not worry about scaling.
Azure DocumentDB for Healthcare IntegrationBizTalk360
In this session,
You will learn what the series is about, and see what we want to accomplish.
For this session you will be learning about Azure DocumentDB, its features and capabilities.
You will learn how to create a DocumentDB database and configure it to support CRUD operations.
You will also learn about the two API’s provided for DocumentDB
You will learn how DocumentDB can be leveraged as a repository for HL7 documents
We will take a look at using DocumentDB with both API and Logic apps
MongoDB .local Munich 2019: MongoDB Atlas Data Lake Technical Deep DiveMongoDB
MongoDB Atlas Data Lake is a new service offered by MongoDB Atlas. Many organizations store long term, archival data in cost-effective storage like S3, GCP, and Azure Blobs. However, many of them do not have robust systems or tools to effectively utilize large amounts of data to inform decision making. MongoDB Atlas Data Lake is a service allowing organizations to analyze their long-term data to discover a wealth of information about their business.
This session will take a deep dive into the features that are currently available in MongoDB Atlas Data Lake and how they are implemented. In addition, we'll discuss future plans and opportunities and offer ample Q&A time with the engineers on the project.
MongoDB Atlas Autoscaling automatically changes both the storage and compute capacity of your MongoDB Atlas cluster, in response to changing traffic patterns. This enables MongoDB Atlas to continuously maximize performance while minimizing cost, with just a press of a button. Plan to attend this session and learn more about how autoscaling works behind the scenes, and the best ways to use it.
Rapid Development and Performance By Transitioning from RDBMSs to MongoDB
Modern day application requirements demand rich & dynamic data structures, fast response times, easy scaling, and low TCO to match the rapidly changing customer & business requirements plus the powerful programming languages used in today's software landscape.
Traditional approaches to solutions development with RDBMSs increasingly expose the gap between the modern development languages and the relational data model, and between scaling up vs. scaling horizontally on commodity hardware. Development time is wasted as the bulk of the work has shifted from adding business features to struggling with the RDBMSs.
MongoDB, the premier NoSQL database, offers a flexible and scalable solution to focus on quickly adding business value again.
In this session, we will provide:
- Overview of MongoDB's capabilities
- Code-level exploration of the MongoDB programming model and APIs and how they transform the way developers interact with a database
- Update of the exciting features in MongoDB 3.0
Modern architectures are moving away from a "one size fits all" approach. We are well aware that we need to use the best tools for the job. Given the large selection of options available today, chances are that you will end up managing data in MongoDB for your operational workload and with Spark for your high speed data processing needs.
Description: When we model documents or data structures there are some key aspects that need to be examined not only for functional and architectural purposes but also to take into consideration the distribution of data nodes, streaming capabilities, aggregation and queryability options and how we can integrate the different data processing software, like Spark, that can benefit from subtle but substantial model changes. A clear example is when embedding or referencing documents and their implications on high speed processing.
Over the course of this talk we will detail the benefits of a good document model for the operational workload. As well as what type of transformations we should incorporate in our document model to adjust for the high speed processing capabilities of Spark.
We will look into the different options that we have to connect these two different systems, how to model according to different workloads, what kind of operators we need to be aware of for top performance and what kind of design and architectures we should put in place to make sure that all of these systems work well together.
Over the course of the talk we will showcase different libraries that enable the integration between spark and MongoDB, such as MongoDB Hadoop Connector, Stratio Connector and MongoDB Spark Native Connector.
By the end of the talk I expect the attendees to have an understanding of:
How they connect their MongoDB clusters with Spark
Which use cases show a net benefit for connecting these two systems
What kind of architecture design should be considered for making the most of Spark + MongoDB
How documents can be modeled for better performance and operational process, while processing these data sets stored in MongoDB.
The talk is suitable for:
Developers that want to understand how to leverage Spark
Architects that want to integrate their existing MongoDB cluster and have real time high speed processing needs
Data scientists that know about Spark, are playing with Spark and want to integrate with MongoDB for their persistency layer
During this presentation, Infusion and MongoDB shared their mainframe optimization experiences and best practices. These have been gained from working with a variety of organizations, including a case study from one of the world’s largest banks. MongoDB and Infusion bring a tested approach that provides a new way of modernizing mainframe applications, while keeping pace with the demand for new digital services.
Improve your SQL workload with observabilityOVHcloud
La majeure partie du SI d'OVH repose sur des bases de données relationnelles (PostgreSQL, MySQL, MariaDB). En termes de volumétrie cela représente 400 bases pesants plus de 20To de données réparties sur 60 clusters dans deux zones géographiques le tout propulsant 3000 applications.
Comment tout voir dans notre parc ? Mieux encore, comment faire pour que tout le monde puisse suivre l'activité de sa base de données ? C'est le challenge que nous nous sommes fixés, un an après nous pouvons partager notre expérience.
Et si l'observability n'était pas juste un buzzword, mais avait un réel impact sur la production ?
MongoDB .local Munich 2019: Managing a Heterogeneous Stack with MongoDB & SQLMongoDB
Data administrators face the challenge of integrating disparate data technologies into a cohesive and performant data platform. This is especially true when using diverse query languages and protocols. This session will focus on how to integrate SQL-aware applications into a MongoDB data platform.
In this hands-on webinar we will cover how to leverage the Treasure Data Javascript SDK library to ensure user stitching of web data into the Treasure Data Customer Data Platform to provide a holistic view of prospects and customers.
We will demo the native SDK, as well as deploying the SDK inside of Adobe DTM and Google Tag Manager.
Doing Joins in MongoDB: Best Practices for Using $lookupMongoDB
Speaker: Austin Zellner, Solutions Architect, MongoDB
Level: 200 (Intermediate)
Track: Data Analytics
$lookup is a pipeline stage in the aggregation framework that performs a left outer join. In this session, you will learn how to leverage $lookup in your applications and best practices for implementing features with $lookup.
What You Will Learn:
- Fundamentals of $lookup and its syntax.
- How to use $lookup stages in your aggregation pipelines.
- Best practices for using $lookup to implement application features.
Agile Software Development is becoming the defacto way of building software these days. More and more enterprises, from large fortune 500 to small shop start-ups, are adopting agile development methodologies. But Agile Software development is more than just a methodology or a practice. It's also a combined set of tools and platforms that today are at our disposal to allows to iterate faster, get-to-market sooner and also fail faster. These set of tools augment our development cycles by a few orders of magnitude and allow developers to be much more productive.
How to get the best of both: MongoDB is great for low latency quick access of recent data; Treasure Data is great for infinitely growing store of historical data. In the latter case, one need not worry about scaling.
Azure DocumentDB for Healthcare IntegrationBizTalk360
In this session,
You will learn what the series is about, and see what we want to accomplish.
For this session you will be learning about Azure DocumentDB, its features and capabilities.
You will learn how to create a DocumentDB database and configure it to support CRUD operations.
You will also learn about the two API’s provided for DocumentDB
You will learn how DocumentDB can be leveraged as a repository for HL7 documents
We will take a look at using DocumentDB with both API and Logic apps
Why Standards-Based Drivers Offer Better API IntegrationNordic APIs
As enterprises grow, so too does the number and variety of data sources they use to drive business. The average company is using at least sixteen SaaS applications and has data in at least that many on-premises data stores and internal apps.
With such disparate data, each tied to a unique API, integrating, managing, and maintaining integrations for all of a company’s data creates a whole new set of challenges. Thankfully, solutions exist that enable enterprises to rely on data to drive business without causing undue strain. In this session, we’ll explore and compare the different options for solving the data integration problem and explain why you should be using standards-based drivers to abstract your API integrations.
Why Standards-Based Drivers Offer Better API IntegrationJerod Johnson
A brief overview of API integration solutions (direct, SDK, middleware, drivers) and an argument in favor of using drivers to solve your integration needs.
PayPal datalake journey | teradata - edge of next | san diego | 2017 october ...Deepak Chandramouli
PayPal Data Lake Journey | 2017-Oct | San Diego | Teradata Edge of Next
Gimel [http://www.gimel.io] is a Big Data Processing Library, open sourced by PayPal.
https://www.youtube.com/watch?v=52PdNno_9cU&t=3s
Gimel empowers analysts, scientists, data engineers alike to access a variety of Big Data / Traditional Data Stores - with just SQL or a single line of code (Unified Data API).
This is possible via the Catalog of Technical properties abstracted from users, along with a rich collection of Data Store Connectors available in Gimel Library.
A Catalog provider can be Hive or User Supplied (runtime) or UDC.
In addition, PayPal recently open sourced UDC [Unified Data Catalog], which can host and serve the Technical Metatada of the Data Stores & Objects. Visit http://www.unifieddatacatalog.io to experience first hand.
Presto: Fast SQL-on-Anything (including Delta Lake, Snowflake, Elasticsearch ...Databricks
Presto, an open source distributed SQL engine, is widely recognized for its low-latency queries, high concurrency, and native ability to query multiple data sources. Proven at scale in a variety of use cases at Airbnb, Comcast, GrubHub, Facebook, FINRA, LinkedIn, Lyft, Netflix, Twitter, and Uber, in the last few years Presto experienced an unprecedented growth in popularity in both on-premises and cloud deployments over Object Stores, HDFS, NoSQL and RDBMS data stores.
Qlik and Confluent Success Stories with Kafka - How Generali and Skechers Kee...HostedbyConfluent
Converting production databases into live data streams for Apache Kafka can be labor intensive and costly. As Kafka architectures grow, complexity also rises as data teams begin to configure clusters for redundancy, partitions for performance, as well as for consumer groups for correlated analytics processing. In this breakout session, you’ll hear data streaming success stories from Generali and Skechers that leverage Qlik Data Integration and Confluent. You’ll discover how Qlik’s data integration platform lets organizations automatically produce real-time transaction streams into Kafka, Confluent Platform, or Confluent Cloud, deliver faster business insights from data, enable streaming analytics, as well as streaming ingestion for modern analytics. Learn how these customer use Qlik and Confluent to: - Turn databases into live data feeds - Simplify and automate the real-time data streaming process - Accelerate data delivery to enable real-time analytics Learn how Skechers and Generali breathe new life into data in the cloud, stay ahead of changing demands, while lowering over-reliance on resources, production time and costs.
Integrate MongoDB & SQL data with a single REST APIEspresso Logic
Webinar slides. Describes how you create a backend application, in the cloud or on premise, that join data from MongoDb and SQL databases with a single RESTful API.
This is a run-through at a 200 level of the Microsoft Azure Big Data Analytics for the Cloud data platform based on the Cortana Intelligence Suite offerings.
Building Operational Data Lake using Spark and SequoiaDB with Yang PengDatabricks
This topic describes the use of Spark and SequoiaDB in the Operational Data Lake of China’s financial industry, including how to use SequoiaDB to provide online high concurrent services and how to use Spark for data processing and machine learning. China has the world’s largest population, and also the world’s second largest economy. Many of the best technologies used in the United States and Europe are difficult to play effectively in China. This topic will show you how Spark and SequoiaDB are able to provide online financial services to billions of population.
In this session you will learn how Qlik’s Data Integration platform (formerly Attunity) reduces time to market and time to insights for modern data architectures through real-time automated pipelines for data warehouse and data lake initiatives. Hear how pipeline automation has impacted large financial services organizations ability to rapidly deliver value and see how to build an automated near real-time pipeline to efficiently load and transform data into a Snowflake data warehouse on AWS in under 10 minutes.
Developing Enterprise Consciousness: Building Modern Open Data PlatformsScyllaDB
ScyllaDB, along side some of the other major distributed real-time technologies gives businesses a unique opportunity to achieve enterprise consciousness - a business platform that delivers data to the people that need when they need it any time, anywhere.
This talk covers how modern tools in the open data platform can help companies synchronize data across their applications using open source tools and technologies and more modern low-code ETL/ReverseETL tools.
Topics:
- Business Platform Challenges
- What Enterprise Consciousness Solves
- How ScyllaDB Empowers Enterprise Consciousness
- What can ScyllaDB do for Big Companies
- What can ScyllaDB do for smaller companies.
Webinar: How to Drive Business Value in Financial Services with MongoDBMongoDB
Huge upheaval in the finance industry has led to a major strain on existing IT infrastructure and systems. New finance industry regulation has meant increased volume, velocity and variability of data. This coupled with cost pressures from the business has led these institutions to seek alternatives. Top tier institutions like MetLife have turned to MongoDB because of the enormous business value it enables.
In this session, hear how MongoDB enabled these successful real world examples:
Single View of a Customer - 3 months and $2M for a single view of a customer across 50 source systems
Reference Data Management - $40M in cost savings from migrating to MongoDB for reference data management
Private cloud - MongoDB as a PaaS across a tier 1 bank for enabling agility for operations, not just the developer
The use cases are specific to financial services but the patterns of usage - agility, scale, global distribution - will be applicable across many industries.
Today, data lakes are widely used and have become extremely affordable as data volumes have grown. However, they are only meant for storage and by themselves provide no direct value. With up to 80% of data stored in the data lake today, how do you unlock the value of the data lake? The value lies in the compute engine that runs on top of a data lake.
Join us for this webinar where Ahana co-founder and Chief Product Officer Dipti Borkar will discuss how to unlock the value of your data lake with the emerging Open Data Lake analytics architecture.
Dipti will cover:
-Open Data Lake analytics - what it is and what use cases it supports
-Why companies are moving to an open data lake analytics approach
-Why the open source data lake query engine Presto is critical to this approach
Similar to CData Data Today: A Developer's Dilemma (20)
Navigating the Metaverse: A Journey into Virtual Evolution"Donna Lenk
Join us for an exploration of the Metaverse's evolution, where innovation meets imagination. Discover new dimensions of virtual events, engage with thought-provoking discussions, and witness the transformative power of digital realms."
Prosigns: Transforming Business with Tailored Technology SolutionsProsigns
Unlocking Business Potential: Tailored Technology Solutions by Prosigns
Discover how Prosigns, a leading technology solutions provider, partners with businesses to drive innovation and success. Our presentation showcases our comprehensive range of services, including custom software development, web and mobile app development, AI & ML solutions, blockchain integration, DevOps services, and Microsoft Dynamics 365 support.
Custom Software Development: Prosigns specializes in creating bespoke software solutions that cater to your unique business needs. Our team of experts works closely with you to understand your requirements and deliver tailor-made software that enhances efficiency and drives growth.
Web and Mobile App Development: From responsive websites to intuitive mobile applications, Prosigns develops cutting-edge solutions that engage users and deliver seamless experiences across devices.
AI & ML Solutions: Harnessing the power of Artificial Intelligence and Machine Learning, Prosigns provides smart solutions that automate processes, provide valuable insights, and drive informed decision-making.
Blockchain Integration: Prosigns offers comprehensive blockchain solutions, including development, integration, and consulting services, enabling businesses to leverage blockchain technology for enhanced security, transparency, and efficiency.
DevOps Services: Prosigns' DevOps services streamline development and operations processes, ensuring faster and more reliable software delivery through automation and continuous integration.
Microsoft Dynamics 365 Support: Prosigns provides comprehensive support and maintenance services for Microsoft Dynamics 365, ensuring your system is always up-to-date, secure, and running smoothly.
Learn how our collaborative approach and dedication to excellence help businesses achieve their goals and stay ahead in today's digital landscape. From concept to deployment, Prosigns is your trusted partner for transforming ideas into reality and unlocking the full potential of your business.
Join us on a journey of innovation and growth. Let's partner for success with Prosigns.
First Steps with Globus Compute Multi-User EndpointsGlobus
In this presentation we will share our experiences around getting started with the Globus Compute multi-user endpoint. Working with the Pharmacology group at the University of Auckland, we have previously written an application using Globus Compute that can offload computationally expensive steps in the researcher's workflows, which they wish to manage from their familiar Windows environments, onto the NeSI (New Zealand eScience Infrastructure) cluster. Some of the challenges we have encountered were that each researcher had to set up and manage their own single-user globus compute endpoint and that the workloads had varying resource requirements (CPUs, memory and wall time) between different runs. We hope that the multi-user endpoint will help to address these challenges and share an update on our progress here.
Field Employee Tracking System| MiTrack App| Best Employee Tracking Solution|...informapgpstrackings
Keep tabs on your field staff effortlessly with Informap Technology Centre LLC. Real-time tracking, task assignment, and smart features for efficient management. Request a live demo today!
For more details, visit us : https://informapuae.com/field-staff-tracking/
AI Pilot Review: The World’s First Virtual Assistant Marketing SuiteGoogle
AI Pilot Review: The World’s First Virtual Assistant Marketing Suite
👉👉 Click Here To Get More Info 👇👇
https://sumonreview.com/ai-pilot-review/
AI Pilot Review: Key Features
✅Deploy AI expert bots in Any Niche With Just A Click
✅With one keyword, generate complete funnels, websites, landing pages, and more.
✅More than 85 AI features are included in the AI pilot.
✅No setup or configuration; use your voice (like Siri) to do whatever you want.
✅You Can Use AI Pilot To Create your version of AI Pilot And Charge People For It…
✅ZERO Manual Work With AI Pilot. Never write, Design, Or Code Again.
✅ZERO Limits On Features Or Usages
✅Use Our AI-powered Traffic To Get Hundreds Of Customers
✅No Complicated Setup: Get Up And Running In 2 Minutes
✅99.99% Up-Time Guaranteed
✅30 Days Money-Back Guarantee
✅ZERO Upfront Cost
See My Other Reviews Article:
(1) TubeTrivia AI Review: https://sumonreview.com/tubetrivia-ai-review
(2) SocioWave Review: https://sumonreview.com/sociowave-review
(3) AI Partner & Profit Review: https://sumonreview.com/ai-partner-profit-review
(4) AI Ebook Suite Review: https://sumonreview.com/ai-ebook-suite-review
top nidhi software solution freedownloadvrstrong314
This presentation emphasizes the importance of data security and legal compliance for Nidhi companies in India. It highlights how online Nidhi software solutions, like Vector Nidhi Software, offer advanced features tailored to these needs. Key aspects include encryption, access controls, and audit trails to ensure data security. The software complies with regulatory guidelines from the MCA and RBI and adheres to Nidhi Rules, 2014. With customizable, user-friendly interfaces and real-time features, these Nidhi software solutions enhance efficiency, support growth, and provide exceptional member services. The presentation concludes with contact information for further inquiries.
Developing Distributed High-performance Computing Capabilities of an Open Sci...Globus
COVID-19 had an unprecedented impact on scientific collaboration. The pandemic and its broad response from the scientific community has forged new relationships among public health practitioners, mathematical modelers, and scientific computing specialists, while revealing critical gaps in exploiting advanced computing systems to support urgent decision making. Informed by our team’s work in applying high-performance computing in support of public health decision makers during the COVID-19 pandemic, we present how Globus technologies are enabling the development of an open science platform for robust epidemic analysis, with the goal of collaborative, secure, distributed, on-demand, and fast time-to-solution analyses to support public health.
TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERRORTier1 app
Even though at surface level ‘java.lang.OutOfMemoryError’ appears as one single error; underlyingly there are 9 types of OutOfMemoryError. Each type of OutOfMemoryError has different causes, diagnosis approaches and solutions. This session equips you with the knowledge, tools, and techniques needed to troubleshoot and conquer OutOfMemoryError in all its forms, ensuring smoother, more efficient Java applications.
Software Engineering, Software Consulting, Tech Lead.
Spring Boot, Spring Cloud, Spring Core, Spring JDBC, Spring Security,
Spring Transaction, Spring MVC,
Log4j, REST/SOAP WEB-SERVICES.
Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...Globus
The Earth System Grid Federation (ESGF) is a global network of data servers that archives and distributes the planet’s largest collection of Earth system model output for thousands of climate and environmental scientists worldwide. Many of these petabyte-scale data archives are located in proximity to large high-performance computing (HPC) or cloud computing resources, but the primary workflow for data users consists of transferring data, and applying computations on a different system. As a part of the ESGF 2.0 US project (funded by the United States Department of Energy Office of Science), we developed pre-defined data workflows, which can be run on-demand, capable of applying many data reduction and data analysis to the large ESGF data archives, transferring only the resultant analysis (ex. visualizations, smaller data files). In this talk, we will showcase a few of these workflows, highlighting how Globus Flows can be used for petabyte-scale climate analysis.
Enhancing Project Management Efficiency_ Leveraging AI Tools like ChatGPT.pdfJay Das
With the advent of artificial intelligence or AI tools, project management processes are undergoing a transformative shift. By using tools like ChatGPT, and Bard organizations can empower their leaders and managers to plan, execute, and monitor projects more effectively.
Paketo Buildpacks : la meilleure façon de construire des images OCI? DevopsDa...Anthony Dahanne
Les Buildpacks existent depuis plus de 10 ans ! D’abord, ils étaient utilisés pour détecter et construire une application avant de la déployer sur certains PaaS. Ensuite, nous avons pu créer des images Docker (OCI) avec leur dernière génération, les Cloud Native Buildpacks (CNCF en incubation). Sont-ils une bonne alternative au Dockerfile ? Que sont les buildpacks Paketo ? Quelles communautés les soutiennent et comment ?
Venez le découvrir lors de cette session ignite
May Marketo Masterclass, London MUG May 22 2024.pdfAdele Miller
Can't make Adobe Summit in Vegas? No sweat because the EMEA Marketo Engage Champions are coming to London to share their Summit sessions, insights and more!
This is a MUG with a twist you don't want to miss.
Into the Box Keynote Day 2: Unveiling amazing updates and announcements for modern CFML developers! Get ready for exciting releases and updates on Ortus tools and products. Stay tuned for cutting-edge innovations designed to boost your productivity.
Innovating Inference - Remote Triggering of Large Language Models on HPC Clus...Globus
Large Language Models (LLMs) are currently the center of attention in the tech world, particularly for their potential to advance research. In this presentation, we'll explore a straightforward and effective method for quickly initiating inference runs on supercomputers using the vLLM tool with Globus Compute, specifically on the Polaris system at ALCF. We'll begin by briefly discussing the popularity and applications of LLMs in various fields. Following this, we will introduce the vLLM tool, and explain how it integrates with Globus Compute to efficiently manage LLM operations on Polaris. Attendees will learn the practical aspects of setting up and remotely triggering LLMs from local machines, focusing on ease of use and efficiency. This talk is ideal for researchers and practitioners looking to leverage the power of LLMs in their work, offering a clear guide to harnessing supercomputing resources for quick and effective LLM inference.
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I ...Juraj Vysvader
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I didn't get rich from it but it did have 63K downloads (powered possible tens of thousands of websites).
Globus Compute wth IRI Workflows - GlobusWorld 2024Globus
As part of the DOE Integrated Research Infrastructure (IRI) program, NERSC at Lawrence Berkeley National Lab and ALCF at Argonne National Lab are working closely with General Atomics on accelerating the computing requirements of the DIII-D experiment. As part of the work the team is investigating ways to speedup the time to solution for many different parts of the DIII-D workflow including how they run jobs on HPC systems. One of these routes is looking at Globus Compute as a way to replace the current method for managing tasks and we describe a brief proof of concept showing how Globus Compute could help to schedule jobs and be a tool to connect compute at different facilities.
2. See the World as a Database
Agenda
• Introduction
• A Sample Company’s Data
• The Problem: Disparate APIs & SDKs
• The Solution: Common Interfaces
• Code Samples
3. See the World as a Database
• Over 2 decades of experience
(since 1994) in building
integration components.
• 80+ Data Sources
• ‘See the World as a Database’
About CData Software
• Started: 1994
• Location: Chapel Hill, NC
• CData Software (formerly
RSSBus), a spin-off of
/n software.
4. See the World as a Database
A Sample Company’s Data
QuickBooks (Accounting & PoS)
Exact Online (Warehousing & Logistics)
HubSpot (CRM)
Microsoft Project (Project/Scheduling)
On-premises & cloud-based MySQL databases
Facebook & Twitter (Social Media data)
5. See the World as a Database
Accessing A Sample Company’s Data
REST APIs
XML APIs
SDKs
Authentication
Basic & OAuth
6. See the World as a Database
The Problem: Disparate APIs & SDKs
REST/SOAP APIs & SDKs model data differently
SDKs are language-dependent
Changes break existing tools
Lack of data-centric functionality
7. See the World as a Database
The Solution: A Common Interface
Built using established standards
Developers learn a single protocol
Data-centric model
Power of SQL (JOINs, filtering, aggregation)
Easily browse data
API/SDK/data model changes are covered
8. See the World as a Database
Digging In: How Do They Work?
Mapping APIs & SDKs to a relational model
Tables and Views correspond with resources
One resource per row
Nested values represented as JSON
Sub-collections are treated as related tables
SQL queries roughly match HTTP verbs
Stored procedures are used for other operations
9. See the World as a Database
Digging In: How Do They Work?
INSERT means more than creating a resource
Major SQL features supported across sources
Complex queries utilize API/SDK calls and memory
Bulk operations are supported, where possible
Advanced queries utilize API/SDK functionality
10. See the World as a Database
When are SDK/APIs better?
Data streaming
Event-driven applications
Language-centric development
11. See the World as a Database
Code Sample: Reading MySQL Data
using System.Data.CData.MySQL;
String connectionString =
"User=myUser;Password=myPassword;Database=myDatabase;Server=myServer;Port=3306;";
using (MySQLConnection conn = new MySQLConnection(connectionString)) {
MySQLCommand cmd = new MySQLCommand("", conn);
MySQLDataReader rdr;
cmd.CommandText = "SELECT * FROM Orders";
rdr = cmd.ExecuteReader();
while (rdr.Read()) {
//process results
}
}
12. See the World as a Database
Code Sample: Updating QuickBooks Data
using System.Data.CData.QuickBooks;
String connectionString = "User=test;Password=test;URL=http://localhost:8166;";
using (QuickBooksConnection conn = new QuickBooksConnection(connectionString)) {
QuickBooksCommand cmd = new QuickBooksCommand("", conn);
cmd.CommandText = "UPDATE Customers SET Name='Hook, Captain' WHERE Id = @myId";
cmd.Parameters.Add(new QuickBooksParameter("myId", "12345678-9876543210"));
cmd.ExecuteNonQuery();
}
13. See the World as a Database
Calling Stored Procedures
Stored procedures represent non-CRUD actions
Called using name and passing appropriate parameters
Results can be read, stored, and processed
14. See the World as a Database
Code Sample: Upload Media to Twitter
using System.Data.CData.Twitter;
String connectionString = "OAuthAccessToken=...;OAuthAccessTokenSecret=...;" +
"OAuthClientId=...;OAuthClientSecret=...;";
using (TwitterConnection conn = new TwitterConnection(connectionString)) {
TwitterCommand cmd = new TwitterCommand("UploadMedia", conn);
cmd.CommandType = CommandType.StoredProcedure;
cmd.Parameters.Add(new TwitterParameter("MediaFilePath#1", "path/to/file1"));
TwitterDataReader rdr = cmd.ExecuteReader();
while (rdr.Read()) {
//process results
}
}
15. See the World as a Database
Desktop/Server apps call the Stored Procedure:
GetOAuthAccessToken
Web apps call the Stored Procedures:
GetOAuthAuthorizationUrl
GetOAuthAccessToken
Optional: allow the Drivers to manage OAuth flow
Performing OAuth Authentication
16. See the World as a Database
Code Sample: Exact Online OAuth Flow
using System.Data.CData.ExactOnline;
string connectionString = "Region=’United States’;OAuthClientId=...;OAuthClientSecret=...;";
string accessToken, refreshToken, expiresIn;
accessToken = refreshToken = expiresIn = string.Empty;
using (ExactOnlineConnection conn = new ExactOnlineConnection(connectionString)){
ExactOnlineCommand cmd = new ExactOnlineCommand("GetOAuthAccessToken", conn);
cmd.CommandType = CommandType.StoredProcedure;
cmd.Parameters.Add(new ExactOnlineParameter("CallBackUrl", "https://localhost:7777"));
ExactOnlineDataReader reader = cmd.ExecuteReader();
if(reader.Read()){
accessToken = (string)reader["OAuthAccessToken"];
refreshToken = (string)reader["OAuthRefreshToken"];
expiresIn = (string)reader["ExpiresIn"];
}
}
connectionString += "OAuthAccessToken=" + accessToken + ";OAuthRefreshToken=" + refreshToken + ";";
18. See the World as a Database
Thank You!
In Conclusion
Learning Disparate APIs/SDKs are Costly
Common Interfaces Provide Easy Access to Data
Questions?
Editor's Notes
What am I here to show you today?
The average company is running 15 to 20 Software-as-a-Service (SaaS) apps and at least as many on-premise apps.
In this presentation, I’m going to show you a sample company’s different data stores, the disparate APIs & SDKs that are typically used to access that data, and then explain how common interfaces can be used to make integrating with that data much easier.
Time permitting, I’ll show some code samples that demonstrate the common interface
Introduce myself, >3 yrs with the company, worked through support -> development -> technical marketing.
Started in 1994, foundation of over 2 decades. Located in Chapel Hill, just a few short hours up the road from here in Atlanta.
We’re a spin-off of a spin-off. Our roots are in /n software, an integration components company. Spun out of that technology was RSSBus, which is a company that focuses on EDI integration. From RSSBus, we saw space for standards-based data integration and moved into that sphere, launching CData Software.
CData Software (www.cdata.com) is a leading provider of data access and connectivity solutions specializing in the development of standard drivers for real-time access to both Cloud and On-Premises Applications, Databases, and APIs.
Our drivers are universally accessible, providing access to data through established data standards and application platforms such as ODBC, JDBC, ADO.NET, Mobile (Android/Xamarin), OData, SQL Server SSIS, Microsoft BizTalk, Microsoft Excel, and more.
Imagine a sample manufacturing company that has an online web store.
They use QuickBooks for their accounting and point of sale, Exact Online for their warehousing/logistics, Hubspot as their CRM, MS Project as their project/scheduling management solution, and also want to be able to work with social media data and aggregate some other data in a MySQL database that is used to help populate some of their web content.
This company wants to create a set of applications, both installed and web-based, to help them manage all of their data, and to build connections between their data sources. As a developer, your time is better spent focusing on your companies core competencies, not on building and maintaining rapidly changing service integrations
REST API = HTTP calls to different endpoints for each kind of entry
Exact Online, Facebook, Twitter, HubSpot, MS Project, QuickBooks Online, QuickBooks POS
XML API = Document import/export, still web calls
Exact Online
SDK = Language specific class library
Facebook, MS Project, MySQL, QuickBooks Desktop
Connectors
MySQL
Basic & OAuth Authentication
All sources have read/write capabilities.
While protocols like OData exist, there is no consensus regarding how data is modelled in an API, or how actions, like sending an email, or “liking” a social media post, should be performed. On top of that, standards are rarely standard. Every source implements connectivity (OAuth etc.) a different way and SDKs written in the same language can have vastly different structures.
One benefit of using an SDK is that they are typically built with the language (Java, .NET, C++, etc) in mind, but this is a problem if you aren’t as well versed in the languages available for the SDKs of your data source or the languages available don’t fit in your application model.
Updates to SDKs and APIs can break your existing tools, due to changes in the data model, the class libraries, etc. If you integrate an SDK, update it, and there has been an interface change, your application will no longer compile.
Some SDKs/APIs support relationships between datasets, but data-centric functionality, like JOINs, filtering, and aggregation are not typically natively supported, which means that you will have to code parts of that logic yourself.
You may have noticed that MySQL had something that Sample Company’s other data sources didn’t: connectors.
Connectors are built using established standards. Provide uniform experience for CRUD operations.
Instead of your development teams being forced to learn 6 or more different protocols, including the varying authentication protocols, they only learn one. With connectors, you get wrappers that provide a common interface for working with data. The CData drivers take SQL queries and translate them to the appropriate request based on the API/protocol/SDK.
While an SDK gives you more tightly bound language integration than a Driver ever will (a Java library will always be more 'Java' than a JDBC driver), a driver offers a more data-centric model for service interactivity.
Using a Driver, developers can more easily work with the actual service data in meaningful ways (joins, filters, aggregates, caching, etc). SDK's require reams of additional code to implement a fraction of data capabilities available through our drivers.
Drivers make data available in the IDEs of your language (Visual Studio, Eclipse, NetBeans, IntelliJ, even Excel).
Drivers are more easily interchangeable and make applications more robust. They provide automatic loose-coupling between code and source API(the Driver interface doesn't change, even if the API/data model does).
How to the CData Data Drivers work?
They map an API/SDK to a relational model, allow you to treat any data source as if it were a database.
Tables and views correspond roughly to resource collections.
Individual singleton resources from a collection generally correspond to rows in a table or view, with attributes mapped to columns.
Nested values in attributes may be represented as JSON strings in a table column (the driver’s SQL dialect supports JSON functions to manipulate these).
Sub-collections may map to a table with a foreign-key relationship to the parent table (SalesOrders and SalesOrderLines)
CRUD operations on resources correspond roughly to SQL statements, with HTTP verb GET aligning with SQL SELECT; POST, PUT and PATCH with INSERT or UPDATE; DELETE with DELETE; and HEAD with a SQL EXISTS predicate.
Stored procedures are used judiciously to expose operations and resources not otherwise easily represented.
How to the CData Data Drivers work?
an INSERT SQL statement can post to a user’s Facebook feed or send a GMail message.
Major SQL features are supported across sources, regardless of native API/SDK support
A complex query that joins data will probably have to make one or more separate GET calls for each table resource collection, combining the data in memory
an INSERT/UPDATE/DELETE of multiple rows may translate into multiple web service calls to PUT/POST/DELETE each new resource
More advanced applications—for example, CRM and ERP applications—may support their own query languages (like Salesforce SOQL), or at least offer API methods for complex operations like joining master and detail data. Wherever such features are available, the SQL query optimizer will generate calls that exploit them.
It’s not always best to use a connector/driver as the common interface to your data.
For example, if you are streaming data, the SQL-like functionality of a driver/provider won’t give you access to the streaming data in a way that is meaningful.
Similarly, if you are building an event-driven application (and not a data-driven one), then again, the SQL-like interface is a limiting factor.
And, of course, if you and your development team primarily work in a single development language, then the native support offered by SDKs/APIs often makes the most sense.
The ADO.NET Providers us similar classes to those found in the System.Data.SQLClient library: SQLConnection, SQLCommand, SQLDataReader, etc…
In this code sample, we are simply reading data from a MySQL database.
Follows the same connection process as SQLClient library, though the CData libraries will open the connection implicitly.
Create connection string with connection properties
Create the connection
Create & define a command (SQL query)
Execute the command
Read the results
The same action using the native MySQL SDK would require the developer to learn the SDK and then implement the SDK within the code. Which is all fine and well until the developer needs to connect to another data source.
In this code sample, we see an example of updating QuickBooks data. To avoid SQL Injection (every remembers little Bobby Tables), we can use parameterized queries, using the Parameters property of our Command object and custom QuickBooksParameter(s).
Because of the common interface, the developer can easily establish a connection to the QuickBooks instance and send the command to update the Customers entity. Without the common interface, the developer would be required to learn another SDK, which is very likely to look quite different from the MySQL SDK the developer would have needed to know for the prior query.
With data connectors, stored procedures are used to represent actions that don’t quite fit into the CRUD model typically seen in SQL queries. For example, import or exporting attachments to emails or SharePoint items, performing OAuth Authentication, Creating/Deleting Entities, etc.
Simply set the *CommandText to the name of the stored procedure and add the appropriate parameters.
Results are managed in the DataReader and can be read, stored, or otherwise processed.
Example of calling a stored procedure to upload a media file to twitter that can be later referenced.
Parameters property serves the exact same function as the Parameters property of the familiar SQLCommand class. The data connectors use custom Parameter objects which extend the SQLParameter class.
Once all of the appropriate parameters are added, the command can be execute and the results read.
Thanks to the common interface, the developer is easily able to call the stored procedure and process the results.
CData Drivers use stored procedures to allow you to manage the OAuth flow with the data source manually.
If you are developing a desktop/server application, you can simply call the GetOAuthAccessToken stored procedure and make use of the light-weight, internal web server to receive the OAuth response. Once the response is received, the results can be stored and used in further connections. It is up to the developer to refresh expired (or expiring) tokens.
For Web apps (not pictured), you will initially need to call the GetOAuthAuthorizationURL stored procedure first, which returns the url that should be used int he GetOAuthAuthorization stored procedure.
Alternatively, you can allow the provider to manage the OAuth flow internally. Simply set the InitiateOAuth connection property to GETANDREFRESH and the provider will perform the initial OAuth flow and then store the OAuth credentials on disk and refresh them silently as needed.
Regardless of how you manage the OAuth flow, the stored procedures will have the same names, regardless of data source.
To perform OAuth, you need to set the necessary connection properties for connecting to the data source PLUS the OAuthClientId and OAuthClientSecret properties.
While these property names are uniform across Providers, the original data source may use different names. Such cases are well-documented.
With the connection configured, the developer can call the stored procedure, passing the appropriate parameter values.
Again, thanks to the common interface, the developer is able to read the results and process them appropriately (in this case, appending the Auth Token to the connection string to create a valid connection for future actions.
The Sample Company’s data represents a fraction of the actual number of data sources that a large, enterprise company would need to develop against. For companies with just the average number of sources (20+), the need for a common interface becomes more and more apparent. With CData drivers you gain access to 80+ Accounting, CRM, ERP, Collaboration, NoSQL, Big Data, Financial, and E-Commerce data sources, representing both on-premises and cloud-based instances.
As businesses expand, data is more and more disparate, meaning that coding against that data becomes costly, both in terms of initial development and continued maintenance.
Standards-based drivers offer a common interface that provides for easy, uniform access to your data.
Are there any questions?