HPCC Systems 6.4.0 and Beyond
•
0 likes•245 views
From the 2017 HPCC Systems Community Day: Gavin Halliday shares the latest features and functionality for what's coming in the next release. Gavin Halliday Enterprise Architect, LexisNexis Gavin is a Lead Enterprise Architect for LexisNexis Risk Solutions and one of the original developers for the HPCC Systems platform, having responsibility for the ECL Compiler. Gavin holds a Physics degree from the University of Oxford and resides in the UK.
Report
Share
Report
Share
Recommended
Kubernetes: A Modern Approach for Scalable Infrastructure
Kubernetes is an open-source system that automates deployment, scaling, and management of containerized applications. It addresses common issues with horizontally scaling applications, such as load balancing and failover. Kubernetes provides features like horizontal scaling, automated rollouts and rollbacks, self-healing, service discovery and load balancing. The presentation demonstrates Kubernetes architecture and how it works, including its use of etcd for distributed key-value store, resource quotas, health checking, monitoring, logging, and networking model. Kubernetes can be set up on VM, bare metal, or run in docker containers and is supported by several cloud providers.
Black friday logs - Scaling Elasticsearch
Video in french at https://www.youtube.com/watch?v=9LNnNh63rBI
Sizing an Elasticsearch cluster has to consider many dimensions. In this presentation we go through the different elements and features you should consider to handle big and varying loads of log data.
Keynote -- Percona Live Europe 2018
This deck was the keynote speech delivered by Kevin Xu (GM of Global Strategy at Operations) and Shen Li (VP of Engineering at PingCAP) on TiDB architecture, tools and migration path, and TiDB Cloud fully-managed offering at Percona Live Europe 2018 in Frankfurt, Germany.
CouchbasetoHadoop_Matt_Michael_Justin v4
This document discusses LinkedIn's use of Kafka, Hadoop, Storm, and Couchbase in their big data pipeline. It provides an overview of each technology and how LinkedIn uses them together. Specifically, it describes how LinkedIn uses Kafka to stream data to Hadoop for analytics and report generation. It also discusses how LinkedIn uses Hadoop to pre-build and warm Couchbase buckets for improved performance. The presentation includes a use case of streaming member profile and activity data through Kafka to both Hadoop and Couchbase clusters.
RedisConf18 - Redis Cluster Provisioning with Kubernetes Service-Catalog Exte...
This document discusses Amadeus' use of Redis clusters with Kubernetes and their plans to offer Redis as a managed service. It introduces the Redis cluster operator which manages Kubernetes resources for Redis clusters. It then explains how the operator can be combined with an Open Service Broker to provision Redis clusters through the Kubernetes Service Catalog, allowing Redis to be offered as a managed service to applications in a simple way. A demo of this Redis as a service setup is planned.
Real-Time Dynamic Data Export Using the Kafka Ecosystem
(Preston Thompson, Braze) Kafka Summit SF 2018
If you collect billions of data points every day and create billions more sending and tracking messages, then you know you need to get your infrastructure right. Our clients use Braze to engage their users over their lifecycle via push notifications, emails, in-app messages and more. Using our Currents product, clients can enable multiple configurable integrations to export this event data in real time to a variety of third-party systems, allowing them to tightly integrate with the rest of their operations and understand the impacts of their engagement strategy.
We use Kafka and the Kafka ecosystem to power this high volume real-time export. As you’d expect in a big data environment, we take data collected from a variety of sources—our SDKs, email partner APIs, our own systems—and produce it to Kafka, with topics for each type of event (about 30 types). Kafka Streams filters and transforms this data according to the configurations set by our clients. Clients can choose which types of events should be sent to which third-party systems. Kafka Connect helps to export the data to third-party systems in real time using custom developed connectors. We run a connector instance for each integration for each customer that consumes from the integration-specific topic. On top of it all, we built a service to manage the pipeline. The service provides configurations to the Streams application and also creates topics for new integrations and uses the Connect REST API to create and manage connectors.
In this talk, I will discuss:
-How we started our journey in designing this large-scale streaming architecture
-Why streaming technologies were necessary to solve our technology and business issues
-The lessons we learned along the way that can help you with your Kafka-based architecture
From logging to monitoring to reactive insights - C Schneider
OSGi Community Event 2017 Presentation by Christian Schneider [Adobe]
In a highly distributed world it is crucial to have the best possible insights into your application.
In the old days we achieved this using plain logging and jmx. This does not scale well to highly distributed processing like in microservices though.
This talk shows how to pimp you OSGi based systems with top notch monitoring and tracing capabilities. Using Apache Karaf Decanter we let karaf instances log into a common elastic search instance with almost zero effort and get insights using kibana. The Apache CXF REST and SOAP integrations allow to log and correlate service requests and replies.
Building on top of this we use the OSGi based reactive components framework and reactor.io streams to provide complex event processing capabilties. A typical use case for this is to describe and monitor service level agreements.
Flink September 2015 Community Update
This document summarizes the September 2015 community update for Apache Flink. Key highlights include Matthias Sax joining as a new committer, the release of version 0.9.1, and discussions starting around releasing version 0.10. Version 0.10 will include improvements to window operators, memory allocation, and new connectors to HDFS, Elasticsearch, and Kafka. The community held various meetups and presentations around the world in September and Flink was recognized as one of the best open source big data tools.
Recommended
Kubernetes: A Modern Approach for Scalable Infrastructure
Kubernetes is an open-source system that automates deployment, scaling, and management of containerized applications. It addresses common issues with horizontally scaling applications, such as load balancing and failover. Kubernetes provides features like horizontal scaling, automated rollouts and rollbacks, self-healing, service discovery and load balancing. The presentation demonstrates Kubernetes architecture and how it works, including its use of etcd for distributed key-value store, resource quotas, health checking, monitoring, logging, and networking model. Kubernetes can be set up on VM, bare metal, or run in docker containers and is supported by several cloud providers.
Black friday logs - Scaling Elasticsearch
Video in french at https://www.youtube.com/watch?v=9LNnNh63rBI
Sizing an Elasticsearch cluster has to consider many dimensions. In this presentation we go through the different elements and features you should consider to handle big and varying loads of log data.
Keynote -- Percona Live Europe 2018
This deck was the keynote speech delivered by Kevin Xu (GM of Global Strategy at Operations) and Shen Li (VP of Engineering at PingCAP) on TiDB architecture, tools and migration path, and TiDB Cloud fully-managed offering at Percona Live Europe 2018 in Frankfurt, Germany.
CouchbasetoHadoop_Matt_Michael_Justin v4
This document discusses LinkedIn's use of Kafka, Hadoop, Storm, and Couchbase in their big data pipeline. It provides an overview of each technology and how LinkedIn uses them together. Specifically, it describes how LinkedIn uses Kafka to stream data to Hadoop for analytics and report generation. It also discusses how LinkedIn uses Hadoop to pre-build and warm Couchbase buckets for improved performance. The presentation includes a use case of streaming member profile and activity data through Kafka to both Hadoop and Couchbase clusters.
RedisConf18 - Redis Cluster Provisioning with Kubernetes Service-Catalog Exte...
This document discusses Amadeus' use of Redis clusters with Kubernetes and their plans to offer Redis as a managed service. It introduces the Redis cluster operator which manages Kubernetes resources for Redis clusters. It then explains how the operator can be combined with an Open Service Broker to provision Redis clusters through the Kubernetes Service Catalog, allowing Redis to be offered as a managed service to applications in a simple way. A demo of this Redis as a service setup is planned.
Real-Time Dynamic Data Export Using the Kafka Ecosystem
(Preston Thompson, Braze) Kafka Summit SF 2018
If you collect billions of data points every day and create billions more sending and tracking messages, then you know you need to get your infrastructure right. Our clients use Braze to engage their users over their lifecycle via push notifications, emails, in-app messages and more. Using our Currents product, clients can enable multiple configurable integrations to export this event data in real time to a variety of third-party systems, allowing them to tightly integrate with the rest of their operations and understand the impacts of their engagement strategy.
We use Kafka and the Kafka ecosystem to power this high volume real-time export. As you’d expect in a big data environment, we take data collected from a variety of sources—our SDKs, email partner APIs, our own systems—and produce it to Kafka, with topics for each type of event (about 30 types). Kafka Streams filters and transforms this data according to the configurations set by our clients. Clients can choose which types of events should be sent to which third-party systems. Kafka Connect helps to export the data to third-party systems in real time using custom developed connectors. We run a connector instance for each integration for each customer that consumes from the integration-specific topic. On top of it all, we built a service to manage the pipeline. The service provides configurations to the Streams application and also creates topics for new integrations and uses the Connect REST API to create and manage connectors.
In this talk, I will discuss:
-How we started our journey in designing this large-scale streaming architecture
-Why streaming technologies were necessary to solve our technology and business issues
-The lessons we learned along the way that can help you with your Kafka-based architecture
From logging to monitoring to reactive insights - C Schneider
OSGi Community Event 2017 Presentation by Christian Schneider [Adobe]
In a highly distributed world it is crucial to have the best possible insights into your application.
In the old days we achieved this using plain logging and jmx. This does not scale well to highly distributed processing like in microservices though.
This talk shows how to pimp you OSGi based systems with top notch monitoring and tracing capabilities. Using Apache Karaf Decanter we let karaf instances log into a common elastic search instance with almost zero effort and get insights using kibana. The Apache CXF REST and SOAP integrations allow to log and correlate service requests and replies.
Building on top of this we use the OSGi based reactive components framework and reactor.io streams to provide complex event processing capabilties. A typical use case for this is to describe and monitor service level agreements.
Flink September 2015 Community Update
This document summarizes the September 2015 community update for Apache Flink. Key highlights include Matthias Sax joining as a new committer, the release of version 0.9.1, and discussions starting around releasing version 0.10. Version 0.10 will include improvements to window operators, memory allocation, and new connectors to HDFS, Elasticsearch, and Kafka. The community held various meetups and presentations around the world in September and Flink was recognized as one of the best open source big data tools.
Kafka and Kafka Streams in the Global Schibsted Data Platform
This document discusses Schibsted's use of Kafka and Kafka Streams in its global data platform. It describes how Schibsted processes streaming data using Kafka Streams to build a new streaming pipeline called Yggdrasil that provides low latency and delivery guarantees. It also discusses how Kafka Connect is used to get data into and out of the platform and how third party analytics tools can connect. The document outlines some challenges experienced in scaling up the platform.
Google Cloud and Data Pipeline Patterns
1. The document discusses various Google Cloud Platform products and patterns for data pipelines, including virtual machines, storage, data warehousing, streaming analytics, machine learning, internet of things, and bioinformatics.
2. Demos and examples are provided of storage, virtual machines, BigQuery, Cloud Spanner, and machine learning on the Google Cloud Platform.
3. The core Google Cloud Platform products discussed for various data and analytics use cases include Cloud Storage, BigQuery, Cloud Dataflow, Compute Engine, Cloud Pub/Sub, and Bigtable.
Kibana + timelion: time series with the elastic stack
The document discusses Kibana and Timelion, which are tools for visualizing and analyzing time series data in the Elastic Stack. It provides an overview of Kibana's evolution and capabilities for creating dashboards. Timelion is introduced as a scripting language that allows users to transform, aggregate, and calculate on time series data from multiple sources to create visualizations. The document demonstrates Timelion's expression language, which includes functions, combinations, filtering, and attributes to process and render time series graphs.
Kibana overview
Kibana is a data visualization tool that is part of the ELK stack (Elasticsearch, Logstash, Kibana) and allows users to search, analyze, and visualize data stored in Elasticsearch. The document discusses Kibana's essential features including Discover to query data, Visualize to create visualizations, and Dashboard to combine them. It also covers additional tools like Dev Tools, X-Pack plugins, and Machine Learning capabilities.
Ten Commandants For Picking NoSQL Database
The slides will walk thru the requirements and will enable to pick appropriate No SQL database as a best match for the requirements i.e. fitness for the purpose
Rootconf 2017 - State of the Open Source monitoring landscape
A talk about various Open Source tools for availability and functional monitoring, time series and metrics as well log and event management.
You Must Construct Additional Pipelines: Pub-Sub on Kafka at Blizzard
(Stephen Parente + Jeff Field, Blizzard) Kafka Summit SF 2018
Blizzard’s global data platform has become a driving force in both business and operational analytics. As more internal customers onboard with the system, there is increasing demand for custom applications to access this data in near real time. In order to avoid many independent teams with varying levels of Kafka expertise all accessing the firehose from our critical production Kafkas, we developed our own pub-sub system on top of Kafka to provide specific datasets to customers on their own cloud deployed Kafka clusters.
Distributed Data Storage & Streaming for Real-time Decisioning Using Kafka, S...
Real-time connectivity of databases and systems is critical in enterprises adopting digital transformation to support super-fast decisioning to drive applications like fraud detection, digital payments, recommendation engines. This talk will focus on the many functions that database streaming serves with Kafka, Spark and Aerospike. We will explore how to eliminate the wall between transaction processing and analytics by synthesizing streaming data with system of record data, to gain key insights in real-time.
Stream Processing @ Lyft
Lyft's streaming platform uses Apache Flink for stream processing and Apache Kafka for messaging. Flink was chosen for its capabilities around state management, exactly-once processing, and flexible APIs. Kafka was chosen for its durability, low latency, and consumer fanout. However, open problems remain around rescaling Kafka while preserving per-key ordering, enabling dynamic stream computations, long-term event storage, and zero downtime deployments. Lyft is working to solve these challenges as it builds out its next generation streaming platform.
Real-Time Vote Platform Benchmark
Benchmark Background:
- Requested by TV Broadcaster for a voting platform
- Choose the best NoSQL DB for the use case
- Push the DB to the max limit
- AWS infrastructure
Goal:
- 2M votes/sec at the best TCO
- 2M Votes = ~7M DB Ops/sec
How a distributed graph analytics platform uses Apache Kafka for data ingesti...
Using Kafka to stream data into TigerGraph, a distributed graph database, is a common pattern in our customers’ data architecture. In the TigerGraph database, Kafka Connect framework was used to build the native S3 data loader. In TigerGraph Cloud, we will be building native integration with many data sources such as Azure Blob Storage and Google Cloud Storage using Kafka as an integrated component for the Cloud Portal.
In this session, we will be discussing both architectures: 1. built-in Kafka Connect framework within TigerGraph database; 2. using Kafka cluster for cloud native integration with other popular data sources. Demo will be provided for both data streaming processes.
"What's New With Globus" Webinar: Spring 2018
In this presentation from June 26, 2018, Globus co-founder Steve Tuecke discussed Globus Connect Server 5.1 with HTTPS file access; plans for new premium storage connectors; upcoming publication services including the new Globus Search and Identifiers services; the new Globus Web App, SSH with Globus Auth, and more.
The journey of Moving from AWS ELK to GCP Data Pipeline
This is a real case from VMfive to shifting ELK architecture from AWS. Currently GCP Data Pipeline provide us more efficiency and stable environment for running our service.
Athens BigData Meetup - Sept 17
Landoop presentation in the Athens Big Data meetup, about streaming technologies on Apache Kafka. Introduction to the Lenses SQL engine and the Lenses platform and our open-source projects.
Presto Summit 2018 - 07 - Lyft
Presto and you are there (Nishant Rayan, Lyft)
Presto Summit 2018 (https://www.starburstdata.com/technical-blog/presto-summit-2018-recap/)
Master thesis
My master thesis presentation: "Avoiding CRUD operations lock-in in NoSQL databases: extension of the CPIM library"
Safer Commutes & Streaming Data | George Padavick, Ohio Department of Transpo...
The Ohio Department of Transportation has adopted Confluent as the event driven enabler of DriveOhio, a modern Intelligent Transportation System. DriveOhio digitally links sensors, cameras, speed monitoring equipment, and smart highway assets in real time, to dynamically adjust the surface road network to maximize the safety and efficiency for travelers. Over the past 24 months the team has increased the number and types of devices within the DriveOhio environment, while also working to see their vendors adopt Kafka to better participate in data sharing.
Bringing Streaming Data To The Masses: Lowering The “Cost Of Admission” For Y...
(Bob Lehmann, Bayer) Kafka Summit SF 2018
You’ve built your streaming data platform. The early adopters are “all in” and have developed producers, consumers and stream processing apps for a number of use cases. A large percentage of the enterprise, however, has expressed interest but hasn’t made the leap. Why?
In 2014, Bayer Crop Science (formerly Monsanto) adopted a cloud first strategy and started a multi-year transition to the cloud. A Kafka-based cross-datacenter DataHub was created to facilitate this migration and to drive the shift to real-time stream processing. The DataHub has seen strong enterprise adoption and supports a myriad of use cases. Data is ingested from a wide variety of sources and the data can move effortlessly between an on premise datacenter, AWS and Google Cloud. The DataHub has evolved continuously over time to meet the current and anticipated needs of our internal customers. The “cost of admission” for the platform has been lowered dramatically over time via our DataHub Portal and technologies such as Kafka Connect, Kubernetes and Presto. Most operations are now self-service, onboarding of new data sources is relatively painless and stream processing via KSQL and other technologies is being incorporated into the core DataHub platform.
In this talk, Bob Lehmann will describe the origins and evolution of the Enterprise DataHub with an emphasis on steps that were taken to drive user adoption. Bob will also talk about integrations between the DataHub and other key data platforms at Bayer, lessons learned and the future direction for streaming data and stream processing at Bayer.
A Walkthrough of InfluxCloud 2.0 by Tim Hall
Tim Hall, VP of Products at InfluxData, will demonstrate how to setup and use the next InfluxCloud 2.0 in this InfluxDays NYC 2019 presentation. He provides a brief history of InfluxCloud followed by an overview of InfluxDB 2.0 and a demo.
Building Streaming Data Pipelines with Google Cloud Dataflow and Confluent Cl...
We will demonstrate how easy it is to use Confluent Cloud as the data source of your Beam pipelines. You will learn how to process the information that comes from Confluent Cloud in real time, make transformations on such information and feed it back to your Kafka topics and other parts of your architecture.
A First Look at HPCC Systems 7.0, Innovation in Action
As part of the 2018 HPCC Systems Summit Community Day event:
The latest version of the platform contains improvements to functionality, usability and interoperability. This talk gives an overview of the changes and explains how you might find them useful.
Gavin Halliday's primary focus is on the code generator, which converts ECL into the queries which run on the platform. Gavin enjoys working on problems together with the development team and the varied nature of the work keeps him engaged. Gavin shares how the platform compares with competitive platforms, including scalability and coding simplicity. He enjoys working on the platform and the elegant solutions the development team is able to implement. Gavin encourages people to give it a try!
HDF Update
Update on HDF, including recent changes to the software, upcoming releases, collaborations, future plans. Will include an overview of the upcoming HDF5 1.8 release, and updates on the netCDF4/HDF5 merge, HDF5 support for indexing, BioHDF, the HDF5-Storage Resource Broker project, the NPOESS BAA, HDF5-OPeNDAP project, HDF-EOS library and website supports and the HDF spin-off THG.
More Related Content
What's hot
Kafka and Kafka Streams in the Global Schibsted Data Platform
This document discusses Schibsted's use of Kafka and Kafka Streams in its global data platform. It describes how Schibsted processes streaming data using Kafka Streams to build a new streaming pipeline called Yggdrasil that provides low latency and delivery guarantees. It also discusses how Kafka Connect is used to get data into and out of the platform and how third party analytics tools can connect. The document outlines some challenges experienced in scaling up the platform.
Google Cloud and Data Pipeline Patterns
1. The document discusses various Google Cloud Platform products and patterns for data pipelines, including virtual machines, storage, data warehousing, streaming analytics, machine learning, internet of things, and bioinformatics.
2. Demos and examples are provided of storage, virtual machines, BigQuery, Cloud Spanner, and machine learning on the Google Cloud Platform.
3. The core Google Cloud Platform products discussed for various data and analytics use cases include Cloud Storage, BigQuery, Cloud Dataflow, Compute Engine, Cloud Pub/Sub, and Bigtable.
Kibana + timelion: time series with the elastic stack
The document discusses Kibana and Timelion, which are tools for visualizing and analyzing time series data in the Elastic Stack. It provides an overview of Kibana's evolution and capabilities for creating dashboards. Timelion is introduced as a scripting language that allows users to transform, aggregate, and calculate on time series data from multiple sources to create visualizations. The document demonstrates Timelion's expression language, which includes functions, combinations, filtering, and attributes to process and render time series graphs.
Kibana overview
Kibana is a data visualization tool that is part of the ELK stack (Elasticsearch, Logstash, Kibana) and allows users to search, analyze, and visualize data stored in Elasticsearch. The document discusses Kibana's essential features including Discover to query data, Visualize to create visualizations, and Dashboard to combine them. It also covers additional tools like Dev Tools, X-Pack plugins, and Machine Learning capabilities.
Ten Commandants For Picking NoSQL Database
The slides will walk thru the requirements and will enable to pick appropriate No SQL database as a best match for the requirements i.e. fitness for the purpose
Rootconf 2017 - State of the Open Source monitoring landscape
A talk about various Open Source tools for availability and functional monitoring, time series and metrics as well log and event management.
You Must Construct Additional Pipelines: Pub-Sub on Kafka at Blizzard
(Stephen Parente + Jeff Field, Blizzard) Kafka Summit SF 2018
Blizzard’s global data platform has become a driving force in both business and operational analytics. As more internal customers onboard with the system, there is increasing demand for custom applications to access this data in near real time. In order to avoid many independent teams with varying levels of Kafka expertise all accessing the firehose from our critical production Kafkas, we developed our own pub-sub system on top of Kafka to provide specific datasets to customers on their own cloud deployed Kafka clusters.
Distributed Data Storage & Streaming for Real-time Decisioning Using Kafka, S...
Real-time connectivity of databases and systems is critical in enterprises adopting digital transformation to support super-fast decisioning to drive applications like fraud detection, digital payments, recommendation engines. This talk will focus on the many functions that database streaming serves with Kafka, Spark and Aerospike. We will explore how to eliminate the wall between transaction processing and analytics by synthesizing streaming data with system of record data, to gain key insights in real-time.
Stream Processing @ Lyft
Lyft's streaming platform uses Apache Flink for stream processing and Apache Kafka for messaging. Flink was chosen for its capabilities around state management, exactly-once processing, and flexible APIs. Kafka was chosen for its durability, low latency, and consumer fanout. However, open problems remain around rescaling Kafka while preserving per-key ordering, enabling dynamic stream computations, long-term event storage, and zero downtime deployments. Lyft is working to solve these challenges as it builds out its next generation streaming platform.
Real-Time Vote Platform Benchmark
Benchmark Background:
- Requested by TV Broadcaster for a voting platform
- Choose the best NoSQL DB for the use case
- Push the DB to the max limit
- AWS infrastructure
Goal:
- 2M votes/sec at the best TCO
- 2M Votes = ~7M DB Ops/sec
How a distributed graph analytics platform uses Apache Kafka for data ingesti...
Using Kafka to stream data into TigerGraph, a distributed graph database, is a common pattern in our customers’ data architecture. In the TigerGraph database, Kafka Connect framework was used to build the native S3 data loader. In TigerGraph Cloud, we will be building native integration with many data sources such as Azure Blob Storage and Google Cloud Storage using Kafka as an integrated component for the Cloud Portal.
In this session, we will be discussing both architectures: 1. built-in Kafka Connect framework within TigerGraph database; 2. using Kafka cluster for cloud native integration with other popular data sources. Demo will be provided for both data streaming processes.
"What's New With Globus" Webinar: Spring 2018
In this presentation from June 26, 2018, Globus co-founder Steve Tuecke discussed Globus Connect Server 5.1 with HTTPS file access; plans for new premium storage connectors; upcoming publication services including the new Globus Search and Identifiers services; the new Globus Web App, SSH with Globus Auth, and more.
The journey of Moving from AWS ELK to GCP Data Pipeline
This is a real case from VMfive to shifting ELK architecture from AWS. Currently GCP Data Pipeline provide us more efficiency and stable environment for running our service.
Athens BigData Meetup - Sept 17
Landoop presentation in the Athens Big Data meetup, about streaming technologies on Apache Kafka. Introduction to the Lenses SQL engine and the Lenses platform and our open-source projects.
Presto Summit 2018 - 07 - Lyft
Presto and you are there (Nishant Rayan, Lyft)
Presto Summit 2018 (https://www.starburstdata.com/technical-blog/presto-summit-2018-recap/)
Master thesis
My master thesis presentation: "Avoiding CRUD operations lock-in in NoSQL databases: extension of the CPIM library"
Safer Commutes & Streaming Data | George Padavick, Ohio Department of Transpo...
The Ohio Department of Transportation has adopted Confluent as the event driven enabler of DriveOhio, a modern Intelligent Transportation System. DriveOhio digitally links sensors, cameras, speed monitoring equipment, and smart highway assets in real time, to dynamically adjust the surface road network to maximize the safety and efficiency for travelers. Over the past 24 months the team has increased the number and types of devices within the DriveOhio environment, while also working to see their vendors adopt Kafka to better participate in data sharing.
Bringing Streaming Data To The Masses: Lowering The “Cost Of Admission” For Y...
(Bob Lehmann, Bayer) Kafka Summit SF 2018
You’ve built your streaming data platform. The early adopters are “all in” and have developed producers, consumers and stream processing apps for a number of use cases. A large percentage of the enterprise, however, has expressed interest but hasn’t made the leap. Why?
In 2014, Bayer Crop Science (formerly Monsanto) adopted a cloud first strategy and started a multi-year transition to the cloud. A Kafka-based cross-datacenter DataHub was created to facilitate this migration and to drive the shift to real-time stream processing. The DataHub has seen strong enterprise adoption and supports a myriad of use cases. Data is ingested from a wide variety of sources and the data can move effortlessly between an on premise datacenter, AWS and Google Cloud. The DataHub has evolved continuously over time to meet the current and anticipated needs of our internal customers. The “cost of admission” for the platform has been lowered dramatically over time via our DataHub Portal and technologies such as Kafka Connect, Kubernetes and Presto. Most operations are now self-service, onboarding of new data sources is relatively painless and stream processing via KSQL and other technologies is being incorporated into the core DataHub platform.
In this talk, Bob Lehmann will describe the origins and evolution of the Enterprise DataHub with an emphasis on steps that were taken to drive user adoption. Bob will also talk about integrations between the DataHub and other key data platforms at Bayer, lessons learned and the future direction for streaming data and stream processing at Bayer.
A Walkthrough of InfluxCloud 2.0 by Tim Hall
Tim Hall, VP of Products at InfluxData, will demonstrate how to setup and use the next InfluxCloud 2.0 in this InfluxDays NYC 2019 presentation. He provides a brief history of InfluxCloud followed by an overview of InfluxDB 2.0 and a demo.
Building Streaming Data Pipelines with Google Cloud Dataflow and Confluent Cl...
We will demonstrate how easy it is to use Confluent Cloud as the data source of your Beam pipelines. You will learn how to process the information that comes from Confluent Cloud in real time, make transformations on such information and feed it back to your Kafka topics and other parts of your architecture.
What's hot (20)
Kafka and Kafka Streams in the Global Schibsted Data Platform
Kafka and Kafka Streams in the Global Schibsted Data Platform
Kibana + timelion: time series with the elastic stack
Kibana + timelion: time series with the elastic stack
Rootconf 2017 - State of the Open Source monitoring landscape
Rootconf 2017 - State of the Open Source monitoring landscape
You Must Construct Additional Pipelines: Pub-Sub on Kafka at Blizzard
You Must Construct Additional Pipelines: Pub-Sub on Kafka at Blizzard
Distributed Data Storage & Streaming for Real-time Decisioning Using Kafka, S...
Distributed Data Storage & Streaming for Real-time Decisioning Using Kafka, S...
How a distributed graph analytics platform uses Apache Kafka for data ingesti...
How a distributed graph analytics platform uses Apache Kafka for data ingesti...
The journey of Moving from AWS ELK to GCP Data Pipeline
The journey of Moving from AWS ELK to GCP Data Pipeline
Safer Commutes & Streaming Data | George Padavick, Ohio Department of Transpo...
Safer Commutes & Streaming Data | George Padavick, Ohio Department of Transpo...
Bringing Streaming Data To The Masses: Lowering The “Cost Of Admission” For Y...
Bringing Streaming Data To The Masses: Lowering The “Cost Of Admission” For Y...
Building Streaming Data Pipelines with Google Cloud Dataflow and Confluent Cl...
Building Streaming Data Pipelines with Google Cloud Dataflow and Confluent Cl...
Similar to HPCC Systems 6.4.0 and Beyond
A First Look at HPCC Systems 7.0, Innovation in Action
As part of the 2018 HPCC Systems Summit Community Day event:
The latest version of the platform contains improvements to functionality, usability and interoperability. This talk gives an overview of the changes and explains how you might find them useful.
Gavin Halliday's primary focus is on the code generator, which converts ECL into the queries which run on the platform. Gavin enjoys working on problems together with the development team and the varied nature of the work keeps him engaged. Gavin shares how the platform compares with competitive platforms, including scalability and coding simplicity. He enjoys working on the platform and the elegant solutions the development team is able to implement. Gavin encourages people to give it a try!
HDF Update
Update on HDF, including recent changes to the software, upcoming releases, collaborations, future plans. Will include an overview of the upcoming HDF5 1.8 release, and updates on the netCDF4/HDF5 merge, HDF5 support for indexing, BioHDF, the HDF5-Storage Resource Broker project, the NPOESS BAA, HDF5-OPeNDAP project, HDF-EOS library and website supports and the HDF spin-off THG.
6.0 is coming
New features coming soon with the Elastic Stack 6.0 (Elasticsearch, Logstash, Kibana, Beats). Presented at the Elasticsearch Meetup San Francisco on Nov 2, 2017.
Revolution R Enterprise 7.4 - Presentation by Bill Jacobs 11Jun15
This document outlines several improvements and updates to ScaleR including new capabilities for DeployR, an upgraded R Engine, improved performance for various models, added support for HDFS caching and updated security features. It also notes changes to packages, platforms, and the separate installation of the R Engine.
Path to 8.0
Come hear a brief overview on the direction the HPCC Systems platform is heading, and get a glimpse into some of the likely highlights included in the next minor and major versions.
Innovation with Connection, The new HPCC Systems Plugins and Modules
As part of the 2018 HPCC Systems Summit Community Day event:
The HPCC Systems platform team continues to expand interoperability with third party systems, which increases the platform feature-set and facilitates custom solutions. James will share an update on the latest connectors available, including the Spark-HPCC, and the upcoming HDFS connector plugin.
James McMullan has a broad range Software Engineering experience from developing low level system drivers for X-Ray fluorescence equipment to mobile video games and web applications. He is a recent addition to the Lexis Nexis team and is part of the HPCC Systems Platform team where he has been working on connectors integrating HPCC Systems with the Spark & Hadoop ecosystems.
HPCC Systems 6.0.0 Highlights
Highlights of Features Coming Soon in HPCC Systems 6.0.0!
Come learn how the upcoming 6.0 release can help you solve Big Data problems faster and more efficient. Topics include:
· How using the new Virtual slave Thor makes using a smart/lookup join faster
· How to add and leave tracing in your code without affecting the graph
· How the HPCC Systems Visualisations Framework provides easy and fast access to visualisations from data included in a workunit or Roxie query
· Plus, hear how our success with GSoC (Google Summer of Code) in 2015 is preparing us for this year
Graph Day 2017 Spring Boot
A slide deck on an example implementation of the technology we used at ten-x to enable business users to find data in a graph.
HDF Update
Mike Folk from the National Center for Supercomputing Applications gave an update on HDF software in 2003. HDF is supported by several government agencies for applications in earth science, simulations, and data-intensive computing. Version 4.2 Release 1 was planned for October 2003 with bug fixes and new features. HDF5 1.6.0 was released in July 2003 with new filters, properties, and performance improvements. Work was also being done on high-level APIs, parallel HDF5, tools, and collaborations with other projects.
Running secured Spark job in Kubernetes compute cluster and integrating with ...
This presentation will provide technical design and development insights to run a secured Spark job in Kubernetes compute cluster that accesses job data from a Kerberized HDFS cluster. Joy will show how to run a long-running machine learning or ETL Spark job in Kubernetes and to access data from HDFS using Kerberos Principal and Delegation token.
The first part of this presentation will unleash the design and best practices to deploy and run Spark in Kubernetes integrated with HDFS that creates on-demand multi-node Spark cluster during job submission, installing/resolving software dependencies (packages), executing/monitoring the workload, and finally disposing the resources at the end of job completion. The second part of this presentation covers the design and development details to setup a Spark+Kubernetes cluster that supports long-running jobs accessing data from secured HDFS storage by creating and renewing Kerberos delegation tokens seamlessly from end-user's Kerberos Principal.
All the techniques covered in this presentation are essential in order to set up a Spark+Kubernetes compute cluster that accesses data securely from distributed storage cluster such as HDFS in a corporate environment. No prior knowledge of any of these technologies is required to attend this presentation.
Speaker
Joy Chakraborty, Data Architect
2015 HPCC Systems Engineering Summit Community Day Wrap-up
This document summarizes the agenda and key topics from the 2015 LexisNexis Risk Solutions HPCC Engineering Summit, which focused on accelerating innovation with HPCC Systems. The event covered applications of HPCC Systems in analytics training, streaming data, customer understanding, and disease tracking. Technology enhancements discussed included integration with Logi Analytics and porting HPCC to OpenPower. Plans for the future include expanded academic collaborations, a Google Summer of Code program, and certification initiatives.
KKBOX WWDC17 Swift and Foundation - Liyao
KKBOX iOS 工程師 Liyao 分享
影片位置 https://youtu.be/nactBecSuYU
相關 Sessions
- [What's New in Swift] (https://developer.apple.com/videos/play/wwdc2017/402/)
- [What's New in Foundation] (https://developer.apple.com/videos/play/wwdc2017/212/)
- [Modernizing Grand Central Dispatch Usage] (https://developer.apple.com/videos/wwdc2017/videos/play/wwdc2017/706/)
- [Efficient Interactions with Frameworks] (https://developer.apple.com/videos/wwdc2017/videos/play/wwdc2017/244/)
Webinar Slides: What's new in Tungsten Replicator 5.2 and Tungsten Clustering...
Continuent Tungsten 5.2 is just around the corner. This is one of our most exciting product releases for some time! In this webinar we’re going to have a look at a host of new features in the new release, including:
- Three new Replication Applier targets (Kafka, Cassandra and Elasticsearch)
- New improvements to our core command line tools trepctl and thl
- New foundations for our filtering services, and
- Improvements to the compatibility between replication and clustering.
This webinar is going be a packed session and we’ll show all the exciting stuff with more in-depth follow-up sessions in the coming weeks. You’ll also learn about more exciting changes coming in the upcoming releases (5.2.1 and 5.3), and our major 6.0 release due out by the end of the year. Come and join us to get the lowdown on everything related to Tungsten Replicator 5.2 and Tungsten Clustering 5.2!
Hybris 6.0.0 to 6.3.0 comparision
Slides which covers the major features and deprecations added to SAP Hybris Ecommerce product between versions 6.0.0 and 6.3.0
Presto @ Facebook: Past, Present and Future
Presto is a distributed SQL query engine optimized for interactive analysis of large datasets across multiple data sources. It aims to improve on Hadoop by allowing data scientists to run queries with low latency. Presto's architecture allows it to distribute queries across a cluster and retrieve data in memory for fast performance. It supports various connectors to data sources like HDFS, Cassandra and Hive. The document outlines Presto's features and performance advantages. It also discusses the open source project's future plans to add more SQL features, improve large joins and aggregations, develop an ODBC driver and potentially introduce a native storage format.
SQL Server 2019 Editions Datasheet 2
SQL Server 2019 provides several editions - Express/web, Standard, and Enterprise. Express/web supports up to 4 cores and 1 GB RAM, Standard up to 24 cores and 128 GB RAM, and Enterprise has unlimited cores and RAM. All editions feature in-memory technologies, encryption, containers, and UTF-8 encoding. Higher editions provide additional capabilities like Big Data Clusters, PolyBase, and more advanced intelligence and query processing. SQL Server 2019 aims to deliver intelligence over all data, choice of languages and platforms, high performance, security, and business insights.
Webinar - Macy’s: Why Your Database Decision Directly Impacts Customer Experi...
According to a recent Harvard Business Review study, there’s only a 43% chance that customers who have a poor experience will stick with you for the next 12 months. Contrast that to the 74% that will remain your customer if they have a great experience. Learn how Macy’s, a leading American department store chain founded in 1858 with over 750 stores in North America, is transforming their customer experience with DataStax Enterprise.
Webinar recording: https://youtu.be/CiUVxh6Ov_E
View current and past DataStax webinars: http://www.datastax.com/resources/webinars
Spark volume requirements 2018
This document discusses storage requirements for running Spark workloads on Kubernetes. It recommends using a distributed file system like HDFS or DBFS for distributed storage and emptyDir or NFS for local temp scratch space. Logs can be stored in emptyDir or pushed to object storage. Features that would improve Spark on Kubernetes include image volumes, flexible PV to PVC mappings, encrypted volumes, and clean deletion for compliance. The document provides an overview of Spark, Kubernetes benefits, and typical Spark deployments.
HUG Ireland Event - HPCC Presentation Slides
This document provides a summary of HPCC Systems, including:
1. A brief history and overview of the architecture with a use case example of calculating insurance policy data within a specified radius.
2. Descriptions of the main components of HPCC Systems - Thor for batch processing, Roxie for real-time queries, and ECL as the data-oriented programming language.
3. Information on how HPCC Systems can be integrated with other systems and technologies through connectors, drivers, and the ability to embed other languages.
Storage Requirements and Options for Running Spark on Kubernetes
In a world of serverless computing users tend to be frugal when it comes to expenditure on compute, storage and other resources. Paying for the same when they aren’t in use becomes a significant factor. Offering Spark as service on cloud presents very unique challenges. Running Spark on Kubernetes presents a lot of challenges especially around storage and persistence. Spark workloads have very unique requirements of Storage for intermediate data, long time persistence, Share file system and requirements become very tight when it same need to be offered as a service for enterprise to mange GDPR and other compliance like ISO 27001 and HIPAA certifications.
This talk covers challenges involved in providing Serverless Spark Clusters share the specific issues one can encounter when running large Kubernetes clusters in production especially covering the scenarios related to persistence.
This talk will help people using Kubernetes or docker runtime in production and help them understand various storage options available and which is more suitable for running Spark workloads on Kubernetes and what more can be done
Similar to HPCC Systems 6.4.0 and Beyond (20)
A First Look at HPCC Systems 7.0, Innovation in Action
A First Look at HPCC Systems 7.0, Innovation in Action
Revolution R Enterprise 7.4 - Presentation by Bill Jacobs 11Jun15
Revolution R Enterprise 7.4 - Presentation by Bill Jacobs 11Jun15
Innovation with Connection, The new HPCC Systems Plugins and Modules
Innovation with Connection, The new HPCC Systems Plugins and Modules
Running secured Spark job in Kubernetes compute cluster and integrating with ...
Running secured Spark job in Kubernetes compute cluster and integrating with ...
2015 HPCC Systems Engineering Summit Community Day Wrap-up
2015 HPCC Systems Engineering Summit Community Day Wrap-up
Webinar Slides: What's new in Tungsten Replicator 5.2 and Tungsten Clustering...
Webinar Slides: What's new in Tungsten Replicator 5.2 and Tungsten Clustering...
Webinar - Macy’s: Why Your Database Decision Directly Impacts Customer Experi...
Webinar - Macy’s: Why Your Database Decision Directly Impacts Customer Experi...
Storage Requirements and Options for Running Spark on Kubernetes
Storage Requirements and Options for Running Spark on Kubernetes
More from HPCC Systems
Natural Language to SQL Query conversion using Machine Learning Techniques on...
Presented at the RV College of Engineering Multidisciplinary Trends in Information Technology, July 21 - 25, 2020.
Improving Efficiency of Machine Learning Algorithms using HPCC Systems
1) The document discusses improving the efficiency of machine learning algorithms using the HPCC Systems platform through parallelization.
2) It describes the HPCC Systems architecture and its advantages for distributed machine learning.
3) A parallel DBSCAN algorithm is implemented on the HPCC platform which shows improved performance over the serial algorithm, with execution times decreasing as more nodes are used.
Towards Trustable AI for Complex Systems
This document provides a summary of a presentation on achieving trustable artificial intelligence (AI) for complex systems. The presentation discusses making data, systems understanding, and AI algorithms more trustable. It suggests deeper data extraction, wider integration of multi-modal data, and augmenting limited data to make data more trustable. A holistic view of systems and balancing simplification and complication can aid understanding. Moving beyond correlation to causation and explaining rather than treating AI as a black box can improve trust in algorithms. The overall goal is to develop explainable and reliable AI that humans will feel confident using to understand and manage complex life science and information technology systems.
Welcome
The document thanks sponsors of different levels for an HPCC summit. It also provides a link to a YouTube video from the summit that is 20 minutes and 19 seconds long and is part of a playlist.
Closing / Adjourn
The document thanks sponsors of different levels for the #HPCCSummit. It also provides a link to a 1 hour and 41 minute long YouTube video from a playlist related to the summit. The video can be viewed at the URL listed and is the tenth video in the playlist.
Community Website: Virtual Ribbon Cutting
The reveal of how HPCC Systems is improving and expanding around the world for building a global community.
Release Cycle Changes
This talk will explain the reasoning behind the release cycle changes, and how overcoming the challenges faced in the previous practice of automated testing has introduced new benefits and wider acceptance from the wider community.
Geohashing with Uber’s H3 Geospatial Index
An introductory look at the ECL H3 Plugin (available since v7.2.0) - a journey from lat/long to ROXIE Service driven visualizations
Advancements in HPCC Systems Machine Learning
This presentation will provide an overview of the latest advancements in Machine Learning modules over the past year, including Clustering, Natural Language Processing, Deep Learning, and the Expanded Model Evaluation Metrics.
Clustering Methods of the HPCC Systems Machine Learning Library
The clustering method is an important part of unsupervised learning. To gain the unsupervised learning capability, two widely applied clustering methods, KMeans and DBSCAN are adopted to the current Machine Learning library. This presentation will introduce the newly developed clustering algorithms and the evaluation methods.
Docker Support
Learn how to package the HPCC Systems Platform in a Docker container and deploy it locally, and build an HPCC Systems Platform AMI followed by an AWS deployment.
Expanding HPCC Systems Deep Neural Network Capabilities
The training process for modern deep neural networks requires big data and large computational power. Though HPCC Systems excels at both of these, HPCC Systems is limited to utilizing the CPU only. It has been shown that GPU acceleration vastly improves Deep Learning training time. In this talk, Robert will explain how HPCC Systems became the first GPU accelerated library while also greatly expanding its deep neural network capabilities.
Leveraging Intra-Node Parallelization in HPCC Systems
This document discusses leveraging intra-node parallelization in HPCC Systems to improve the performance of set similarity joins (SSJ). It describes a naïve approach to computing SSJ that suffers from memory exhaustion and straggling executors. The presented approach replicates and groups independent data using hashing to address these issues while enabling efficient use of multiple CPU cores through multithreading. Experiments show the approach scales to larger datasets and achieves better performance by increasing the number of threads per executor. Lessons learned include that less complex optimizations are more robust in distributed environments.
DataPatterns - Profiling in ECL Watch
DataPatterns is an ECL bundle that provides data profiling and research tools. It has been integrated into the HPCC Systems ECL Standard Library and ECL Watch. The presentation discusses improvements to DataPatterns, how it can be used from ECL code or within ECL Watch to generate profiling reports, and differences between installations from the ECL bundle, Standard Library, or ECL Watch. It also demonstrates using DataDetectors models to examine sample data and previews the new Cloud IDE for browser-based ECL programming.
Leveraging the Spark-HPCC Ecosystem
Join us for an introductory walk-through of using the Spark-HPCC Systems ecosystem to analyze your HPCC Systems data using a collaborative Apache Zeppelin notebook environment.
Work Unit Analysis Tool
The Workunit Analyser examines the entire workunit to produce advice that both novices and experienced ECL developers should find useful. The Workunit Analyser is a post-execution analyser that identifies potential issues and assists users in writing better ECL.
Community Award Ceremony
Let's congratulate the winners of the 2019 HPCC Systems Poster Competition and other award recipients.
Dapper Tool - A Bundle to Make your ECL Neater
Have you ever written a long project for a simple column rename and thought, this should be easier? What about nicely named output statements? Yeah they bother me too. Oh, and DEDUP(SORT(DISTINCT()))? There is a better way! Learn how Dapper can help!
A Success Story of Challenging the Status Quo: Gadget Girls and the Inclusion...
Join NSU University School student and program leader for girls in robotics, Ronnie Shashoua, as she talks about Gadget Girls - a project in collaboration with the NSU Alvin Sherman Library, NSU University School, the South Florida Girl Scouts, and sponsorship from the HPCC Systems Academic Program. Gadget Girls is a program aimed at encouraging girls in fourth and fifth grade to explore their interests in and love for STEM, especially robotics and engineering. Shashoua will discuss the underrepresentation of girls in the Florida Vex Robotics circuit, such as how it demonstrates a larger trend of low numbers of women undertaking STEM educational and career paths and the role it played in inspiring the creation of Gadget Girls.
Beyond the Spectrum – Creating an Environment of Diversity and Empowerment wi...
Hear how the Florida Atlantic University Center for Autism and Related Disabilities has partnered with the HPCC Systems community to provide young people with autism both the technology and professional skills needed to compete in today’s workplace. Mentoring and hands-on coding through ECL workshops have positively impacted students, opening doors to new opportunities for both students and employers.
Using High Dimensional Representation of Words (CBOW) to Find Domain Based Co...
1) The document presents a methodology for generating domain-based common word lists using word embeddings. It trains word vectors on biomedical journal abstracts and identifies words closest to the centroid as common words.
2) Applying the domain-based word list to a random forest classifier on the journal data improved accuracy from 49% to 53%.
3) Topic modeling before and after removing the domain words showed improved separation of topics in the final representation.
More from HPCC Systems (20)
Natural Language to SQL Query conversion using Machine Learning Techniques on...
Natural Language to SQL Query conversion using Machine Learning Techniques on...
Improving Efficiency of Machine Learning Algorithms using HPCC Systems
Improving Efficiency of Machine Learning Algorithms using HPCC Systems
Expanding HPCC Systems Deep Neural Network Capabilities
Expanding HPCC Systems Deep Neural Network Capabilities
Leveraging Intra-Node Parallelization in HPCC Systems
Leveraging Intra-Node Parallelization in HPCC Systems
A Success Story of Challenging the Status Quo: Gadget Girls and the Inclusion...
A Success Story of Challenging the Status Quo: Gadget Girls and the Inclusion...
Beyond the Spectrum – Creating an Environment of Diversity and Empowerment wi...
Beyond the Spectrum – Creating an Environment of Diversity and Empowerment wi...
Using High Dimensional Representation of Words (CBOW) to Find Domain Based Co...
Using High Dimensional Representation of Words (CBOW) to Find Domain Based Co...
Recently uploaded
一比一原版(Unimelb毕业证书)墨尔本大学毕业证如何办理
原版制作【微信:41543339】【(Unimelb毕业证书)墨尔本大学毕业证】【微信:41543339】《成绩单、外壳、雅思、offer、留信学历认证（永久存档/真实可查）》采用学校原版纸张、特殊工艺完全按照原版一比一制作（包括：隐形水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠，文字图案浮雕，激光镭射，紫外荧光，温感，复印防伪）行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备，十五年致力于帮助留学生解决难题，业务范围有加拿大、英国、澳洲、韩国、美国、新加坡，新西兰等学历材料，包您满意。
【我们承诺采用的是学校原版纸张（纸质、底色、纹路），我们拥有全套进口原装设备，特殊工艺都是采用不同进口机器一比一制作，仿真度基本可以达到100%，所有工艺效果都可提前给客户展示，不满意可以根据客户要求进行调整，直到满意为止！】
【业务选择办理准则】
一、工作未确定，回国需先给父母、亲戚朋友看下文凭的情况，办理一份就读学校的毕业证【微信41543339】文凭即可
二、回国进私企、外企、自己做生意的情况，这些单位是不查询毕业证真伪的，而且国内没有渠道去查询国外文凭的真假，也不需要提供真实教育部认证。鉴于此，办理一份毕业证【微信41543339】即可
三、进国企，银行，事业单位，考公务员等等，这些单位是必需要提供真实教育部认证的，办理教育部认证所需资料众多且烦琐，所有材料您都必须提供原件，我们凭借丰富的经验，快捷的绿色通道帮您快速整合材料，让您少走弯路。
留信网认证的作用:
1:该专业认证可证明留学生真实身份
2:同时对留学生所学专业登记给予评定
3:国家专业人才认证中心颁发入库证书
4:这个认证书并且可以归档倒地方
5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息
6:个人职称评审加20分
7:个人信誉贷款加10分
8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才
留信网服务项目：
1、留学生专业人才库服务（留信分析）
2、国（境）学习人员提供就业推荐信服务
3、留学人员区块链存储服务
【关于价格问题（保证一手价格）】
我们所定的价格是非常合理的，而且我们现在做得单子大多数都是代理和回头客户介绍的所以一般现在有新的单子 我给客户的都是第一手的代理价格，因为我想坦诚对待大家 不想跟大家在价格方面浪费时间
对于老客户或者被老客户介绍过来的朋友，我们都会适当给一些优惠。
选择实体注册公司办理，更放心，更安全！我们的承诺：客户在留信官方认证查询网站查询到认证通过结果后付款，不成功不收费！
一比一原版(GWU,GW文凭证书)乔治·华盛顿大学毕业证如何办理
毕业原版【微信:176555708】【(GWU,GW毕业证书)乔治·华盛顿大学毕业证】【微信:176555708】成绩单、外壳、offer、留信学历认证（永久存档真实可查）采用学校原版纸张、特殊工艺完全按照原版一比一制作（包括：隐形水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠，文字图案浮雕，激光镭射，紫外荧光，温感，复印防伪）行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备，十五年致力于帮助留学生解决难题，业务范围有加拿大、英国、澳洲、韩国、美国、新加坡，新西兰等学历材料，包您满意。
【我们承诺采用的是学校原版纸张（纸质、底色、纹路），我们拥有全套进口原装设备，特殊工艺都是采用不同机器制作，仿真度基本可以达到100%，所有工艺效果都可提前给客户展示，不满意可以根据客户要求进行调整，直到满意为止！】
【业务选择办理准则】
一、工作未确定，回国需先给父母、亲戚朋友看下文凭的情况，办理一份就读学校的毕业证【微信176555708】文凭即可
二、回国进私企、外企、自己做生意的情况，这些单位是不查询毕业证真伪的，而且国内没有渠道去查询国外文凭的真假，也不需要提供真实教育部认证。鉴于此，办理一份毕业证【微信176555708】即可
三、进国企，银行，事业单位，考公务员等等，这些单位是必需要提供真实教育部认证的，办理教育部认证所需资料众多且烦琐，所有材料您都必须提供原件，我们凭借丰富的经验，快捷的绿色通道帮您快速整合材料，让您少走弯路。
留信网认证的作用:
1:该专业认证可证明留学生真实身份
2:同时对留学生所学专业登记给予评定
3:国家专业人才认证中心颁发入库证书
4:这个认证书并且可以归档倒地方
5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息
6:个人职称评审加20分
7:个人信誉贷款加10分
8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才
留信网服务项目：
1、留学生专业人才库服务（留信分析）
2、国（境）学习人员提供就业推荐信服务
3、留学人员区块链存储服务
→ 【关于价格问题（保证一手价格）】
我们所定的价格是非常合理的，而且我们现在做得单子大多数都是代理和回头客户介绍的所以一般现在有新的单子 我给客户的都是第一手的代理价格，因为我想坦诚对待大家 不想跟大家在价格方面浪费时间
对于老客户或者被老客户介绍过来的朋友，我们都会适当给一些优惠。
选择实体注册公司办理，更放心，更安全！我们的承诺：客户在留信官方认证查询网站查询到认证通过结果后付款，不成功不收费！
STATATHON: Unleashing the Power of Statistics in a 48-Hour Knowledge Extravag...
"Join us for STATATHON, a dynamic 2-day event dedicated to exploring statistical knowledge and its real-world applications. From theory to practice, participants engage in intensive learning sessions, workshops, and challenges, fostering a deeper understanding of statistical methodologies and their significance in various fields."
原版一比一利兹贝克特大学毕业证(LeedsBeckett毕业证书)如何办理
原版制作【微信:41543339】【利兹贝克特大学毕业证(LeedsBeckett毕业证书)】【微信:41543339】《成绩单、外壳、雅思、offer、真实留信官方学历认证（永久存档/真实可查）》采用学校原版纸张、特殊工艺完全按照原版一比一制作（包括：隐形水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠，文字图案浮雕，激光镭射，紫外荧光，温感，复印防伪）行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备，十五年致力于帮助留学生解决难题，业务范围有加拿大、英国、澳洲、韩国、美国、新加坡，新西兰等学历材料，包您满意。
【我们承诺采用的是学校原版纸张（纸质、底色、纹路）我们拥有全套进口原装设备，特殊工艺都是采用不同机器制作，仿真度基本可以达到100%，所有工艺效果都可提前给客户展示，不满意可以根据客户要求进行调整，直到满意为止！】
【业务选择办理准则】
一、工作未确定，回国需先给父母、亲戚朋友看下文凭的情况，办理一份就读学校的毕业证【微信41543339】文凭即可
二、回国进私企、外企、自己做生意的情况，这些单位是不查询毕业证真伪的，而且国内没有渠道去查询国外文凭的真假，也不需要提供真实教育部认证。鉴于此，办理一份毕业证【微信41543339】即可
三、进国企，银行，事业单位，考公务员等等，这些单位是必需要提供真实教育部认证的，办理教育部认证所需资料众多且烦琐，所有材料您都必须提供原件，我们凭借丰富的经验，快捷的绿色通道帮您快速整合材料，让您少走弯路。
留信网认证的作用:
1:该专业认证可证明留学生真实身份
2:同时对留学生所学专业登记给予评定
3:国家专业人才认证中心颁发入库证书
4:这个认证书并且可以归档倒地方
5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息
6:个人职称评审加20分
7:个人信誉贷款加10分
8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才
留信网服务项目：
1、留学生专业人才库服务（留信分析）
2、国（境）学习人员提供就业推荐信服务
3、留学人员区块链存储服务
【关于价格问题（保证一手价格）】
我们所定的价格是非常合理的，而且我们现在做得单子大多数都是代理和回头客户介绍的所以一般现在有新的单子 我给客户的都是第一手的代理价格，因为我想坦诚对待大家 不想跟大家在价格方面浪费时间
对于老客户或者被老客户介绍过来的朋友，我们都会适当给一些优惠。
选择实体注册公司办理，更放心，更安全！我们的承诺：客户在留信官方认证查询网站查询到认证通过结果后付款，不成功不收费！
The Building Blocks of QuestDB, a Time Series Database
Talk Delivered at Valencia Codes Meetup 2024-06.
Traditionally, databases have treated timestamps just as another data type. However, when performing real-time analytics, timestamps should be first class citizens and we need rich time semantics to get the most out of our data. We also need to deal with ever growing datasets while keeping performant, which is as fun as it sounds.
It is no wonder time-series databases are now more popular than ever before. Join me in this session to learn about the internal architecture and building blocks of QuestDB, an open source time-series database designed for speed. We will also review a history of some of the changes we have gone over the past two years to deal with late and unordered data, non-blocking writes, read-replicas, or faster batch ingestion.
在线办理(英国UCA毕业证书)创意艺术大学毕业证在读证明一模一样
学校原件一模一样【微信：741003700 】《(英国UCA毕业证书)创意艺术大学毕业证》【微信：741003700 】学位证，留信认证（真实可查，永久存档）原件一模一样纸张工艺/offer、雅思、外壳等材料/诚信可靠,可直接看成品样本，帮您解决无法毕业带来的各种难题！外壳，原版制作，诚信可靠，可直接看成品样本。行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备。十五年致力于帮助留学生解决难题，包您满意。
本公司拥有海外各大学样板无数，能完美还原。
1:1完美还原海外各大学毕业材料上的工艺：水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠。文字图案浮雕、激光镭射、紫外荧光、温感、复印防伪等防伪工艺。材料咨询办理、认证咨询办理请加学历顾问Q/微741003700
【主营项目】
一.毕业证【q微741003700】成绩单、使馆认证、教育部认证、雅思托福成绩单、学生卡等！
二.真实使馆公证(即留学回国人员证明,不成功不收费)
三.真实教育部学历学位认证（教育部存档！教育部留服网站永久可查）
四.办理各国各大学文凭(一对一专业服务,可全程监控跟踪进度)
如果您处于以下几种情况：
◇在校期间，因各种原因未能顺利毕业……拿不到官方毕业证【q/微741003700】
◇面对父母的压力，希望尽快拿到；
◇不清楚认证流程以及材料该如何准备；
◇回国时间很长，忘记办理；
◇回国马上就要找工作，办给用人单位看；
◇企事业单位必须要求办理的
◇需要报考公务员、购买免税车、落转户口
◇申请留学生创业基金
留信网认证的作用:
1:该专业认证可证明留学生真实身份
2:同时对留学生所学专业登记给予评定
3:国家专业人才认证中心颁发入库证书
4:这个认证书并且可以归档倒地方
5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息
6:个人职称评审加20分
7:个人信誉贷款加10分
8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才
一比一原版(UMN文凭证书)明尼苏达大学毕业证如何办理
毕业原版【微信:176555708】【(UMN毕业证书)明尼苏达大学毕业证】【微信:176555708】成绩单、外壳、offer、留信学历认证（永久存档真实可查）采用学校原版纸张、特殊工艺完全按照原版一比一制作（包括：隐形水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠，文字图案浮雕，激光镭射，紫外荧光，温感，复印防伪）行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备，十五年致力于帮助留学生解决难题，业务范围有加拿大、英国、澳洲、韩国、美国、新加坡，新西兰等学历材料，包您满意。
【我们承诺采用的是学校原版纸张（纸质、底色、纹路），我们拥有全套进口原装设备，特殊工艺都是采用不同机器制作，仿真度基本可以达到100%，所有工艺效果都可提前给客户展示，不满意可以根据客户要求进行调整，直到满意为止！】
【业务选择办理准则】
一、工作未确定，回国需先给父母、亲戚朋友看下文凭的情况，办理一份就读学校的毕业证【微信176555708】文凭即可
二、回国进私企、外企、自己做生意的情况，这些单位是不查询毕业证真伪的，而且国内没有渠道去查询国外文凭的真假，也不需要提供真实教育部认证。鉴于此，办理一份毕业证【微信176555708】即可
三、进国企，银行，事业单位，考公务员等等，这些单位是必需要提供真实教育部认证的，办理教育部认证所需资料众多且烦琐，所有材料您都必须提供原件，我们凭借丰富的经验，快捷的绿色通道帮您快速整合材料，让您少走弯路。
留信网认证的作用:
1:该专业认证可证明留学生真实身份
2:同时对留学生所学专业登记给予评定
3:国家专业人才认证中心颁发入库证书
4:这个认证书并且可以归档倒地方
5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息
6:个人职称评审加20分
7:个人信誉贷款加10分
8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才
留信网服务项目：
1、留学生专业人才库服务（留信分析）
2、国（境）学习人员提供就业推荐信服务
3、留学人员区块链存储服务
→ 【关于价格问题（保证一手价格）】
我们所定的价格是非常合理的，而且我们现在做得单子大多数都是代理和回头客户介绍的所以一般现在有新的单子 我给客户的都是第一手的代理价格，因为我想坦诚对待大家 不想跟大家在价格方面浪费时间
对于老客户或者被老客户介绍过来的朋友，我们都会适当给一些优惠。
选择实体注册公司办理，更放心，更安全！我们的承诺：客户在留信官方认证查询网站查询到认证通过结果后付款，不成功不收费！
4th Modern Marketing Reckoner by MMA Global India & Group M: 60+ experts on W...
The Modern Marketing Reckoner (MMR) is a comprehensive resource packed with POVs from 60+ industry leaders on how AI is transforming the 4 key pillars of marketing – product, place, price and promotions.
一比一原版(UCSB文凭证书)圣芭芭拉分校毕业证如何办理
毕业原版【微信:176555708】【(UCSB毕业证书)圣芭芭拉分校毕业证】【微信:176555708】成绩单、外壳、offer、留信学历认证（永久存档真实可查）采用学校原版纸张、特殊工艺完全按照原版一比一制作（包括：隐形水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠，文字图案浮雕，激光镭射，紫外荧光，温感，复印防伪）行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备，十五年致力于帮助留学生解决难题，业务范围有加拿大、英国、澳洲、韩国、美国、新加坡，新西兰等学历材料，包您满意。
【我们承诺采用的是学校原版纸张（纸质、底色、纹路），我们拥有全套进口原装设备，特殊工艺都是采用不同机器制作，仿真度基本可以达到100%，所有工艺效果都可提前给客户展示，不满意可以根据客户要求进行调整，直到满意为止！】
【业务选择办理准则】
一、工作未确定，回国需先给父母、亲戚朋友看下文凭的情况，办理一份就读学校的毕业证【微信176555708】文凭即可
二、回国进私企、外企、自己做生意的情况，这些单位是不查询毕业证真伪的，而且国内没有渠道去查询国外文凭的真假，也不需要提供真实教育部认证。鉴于此，办理一份毕业证【微信176555708】即可
三、进国企，银行，事业单位，考公务员等等，这些单位是必需要提供真实教育部认证的，办理教育部认证所需资料众多且烦琐，所有材料您都必须提供原件，我们凭借丰富的经验，快捷的绿色通道帮您快速整合材料，让您少走弯路。
留信网认证的作用:
1:该专业认证可证明留学生真实身份
2:同时对留学生所学专业登记给予评定
3:国家专业人才认证中心颁发入库证书
4:这个认证书并且可以归档倒地方
5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息
6:个人职称评审加20分
7:个人信誉贷款加10分
8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才
留信网服务项目：
1、留学生专业人才库服务（留信分析）
2、国（境）学习人员提供就业推荐信服务
3、留学人员区块链存储服务
→ 【关于价格问题（保证一手价格）】
我们所定的价格是非常合理的，而且我们现在做得单子大多数都是代理和回头客户介绍的所以一般现在有新的单子 我给客户的都是第一手的代理价格，因为我想坦诚对待大家 不想跟大家在价格方面浪费时间
对于老客户或者被老客户介绍过来的朋友，我们都会适当给一些优惠。
选择实体注册公司办理，更放心，更安全！我们的承诺：客户在留信官方认证查询网站查询到认证通过结果后付款，不成功不收费！
Learn SQL from basic queries to Advance queries
Dive into the world of data analysis with our comprehensive guide on mastering SQL! This presentation offers a practical approach to learning SQL, focusing on real-world applications and hands-on practice. Whether you're a beginner or looking to sharpen your skills, this guide provides the tools you need to extract, analyze, and interpret data effectively.
Key Highlights:
Foundations of SQL: Understand the basics of SQL, including data retrieval, filtering, and aggregation.
Advanced Queries: Learn to craft complex queries to uncover deep insights from your data.
Data Trends and Patterns: Discover how to identify and interpret trends and patterns in your datasets.
Practical Examples: Follow step-by-step examples to apply SQL techniques in real-world scenarios.
Actionable Insights: Gain the skills to derive actionable insights that drive informed decision-making.
Join us on this journey to enhance your data analysis capabilities and unlock the full potential of SQL. Perfect for data enthusiasts, analysts, and anyone eager to harness the power of data!
#DataAnalysis #SQL #LearningSQL #DataInsights #DataScience #Analytics
Global Situational Awareness of A.I. and where its headed
You can see the future first in San Francisco.
Over the past year, the talk of the town has shifted from $10 billion compute clusters to $100 billion clusters to trillion-dollar clusters. Every six months another zero is added to the boardroom plans. Behind the scenes, there’s a fierce scramble to secure every power contract still available for the rest of the decade, every voltage transformer that can possibly be procured. American big business is gearing up to pour trillions of dollars into a long-unseen mobilization of American industrial might. By the end of the decade, American electricity production will have grown tens of percent; from the shale fields of Pennsylvania to the solar farms of Nevada, hundreds of millions of GPUs will hum.
The AGI race has begun. We are building machines that can think and reason. By 2025/26, these machines will outpace college graduates. By the end of the decade, they will be smarter than you or I; we will have superintelligence, in the true sense of the word. Along the way, national security forces not seen in half a century will be un-leashed, and before long, The Project will be on. If we’re lucky, we’ll be in an all-out race with the CCP; if we’re unlucky, an all-out war.
Everyone is now talking about AI, but few have the faintest glimmer of what is about to hit them. Nvidia analysts still think 2024 might be close to the peak. Mainstream pundits are stuck on the wilful blindness of “it’s just predicting the next word”. They see only hype and business-as-usual; at most they entertain another internet-scale technological change.
Before long, the world will wake up. But right now, there are perhaps a few hundred people, most of them in San Francisco and the AI labs, that have situational awareness. Through whatever peculiar forces of fate, I have found myself amongst them. A few years ago, these people were derided as crazy—but they trusted the trendlines, which allowed them to correctly predict the AI advances of the past few years. Whether these people are also right about the next few years remains to be seen. But these are very smart people—the smartest people I have ever met—and they are the ones building this technology. Perhaps they will be an odd footnote in history, or perhaps they will go down in history like Szilard and Oppenheimer and Teller. If they are seeing the future even close to correctly, we are in for a wild ride.
Let me tell you what we see.
Predictably Improve Your B2B Tech Company's Performance by Leveraging Data
Harness the power of AI-backed reports, benchmarking and data analysis to predict trends and detect anomalies in your marketing efforts.
Peter Caputa, CEO at Databox, reveals how you can discover the strategies and tools to increase your growth rate (and margins!).
From metrics to track to data habits to pick up, enhance your reporting for powerful insights to improve your B2B tech company's marketing.
- - -
This is the webinar recording from the June 2024 HubSpot User Group (HUG) for B2B Technology USA.
Watch the video recording at https://youtu.be/5vjwGfPN9lw
Sign up for future HUG events at https://events.hubspot.com/b2b-technology-usa/
06-12-2024-BudapestDataForum-BuildingReal-timePipelineswithFLaNK AIM
06-12-2024-BudapestDataForum-BuildingReal-timePipelineswithFLaNK AIM
by
Timothy Spann
Principal Developer Advocate
https://budapestdata.hu/2024/en/
https://budapestml.hu/2024/en/
tim.spann@zilliz.com
https://www.linkedin.com/in/timothyspann/
https://x.com/paasdev
https://github.com/tspannhw
https://www.youtube.com/@flank-stack
milvus
vector database
gen ai
generative ai
deep learning
machine learning
apache nifi
apache pulsar
apache kafka
apache flink
Analysis insight about a Flyball dog competition team's performance
Insight of my analysis about a Flyball dog competition team's last year performance. Find more: https://github.com/rolandnagy-ds/flyball_race_analysis/tree/main
Intelligence supported media monitoring in veterinary medicine
Media monitoring in veterinary medicien
ViewShift: Hassle-free Dynamic Policy Enforcement for Every Data Lake
Dynamic policy enforcement is becoming an increasingly important topic in today’s world where data privacy and compliance is a top priority for companies, individuals, and regulators alike. In these slides, we discuss how LinkedIn implements a powerful dynamic policy enforcement engine, called ViewShift, and integrates it within its data lake. We show the query engine architecture and how catalog implementations can automatically route table resolutions to compliance-enforcing SQL views. Such views have a set of very interesting properties: (1) They are auto-generated from declarative data annotations. (2) They respect user-level consent and preferences (3) They are context-aware, encoding a different set of transformations for different use cases (4) They are portable; while the SQL logic is only implemented in one SQL dialect, it is accessible in all engines.
#SQL #Views #Privacy #Compliance #DataLake
End-to-end pipeline agility - Berlin Buzzwords 2024
We describe how we achieve high change agility in data engineering by eliminating the fear of breaking downstream data pipelines through end-to-end pipeline testing, and by using schema metaprogramming to safely eliminate boilerplate involved in changes that affect whole pipelines.
A quick poll on agility in changing pipelines from end to end indicated a huge span in capabilities. For the question "How long time does it take for all downstream pipelines to be adapted to an upstream change," the median response was 6 months, but some respondents could do it in less than a day. When quantitative data engineering differences between the best and worst are measured, the span is often 100x-1000x, sometimes even more.
A long time ago, we suffered at Spotify from fear of changing pipelines due to not knowing what the impact might be downstream. We made plans for a technical solution to test pipelines end-to-end to mitigate that fear, but the effort failed for cultural reasons. We eventually solved this challenge, but in a different context. In this presentation we will describe how we test full pipelines effectively by manipulating workflow orchestration, which enables us to make changes in pipelines without fear of breaking downstream.
Making schema changes that affect many jobs also involves a lot of toil and boilerplate. Using schema-on-read mitigates some of it, but has drawbacks since it makes it more difficult to detect errors early. We will describe how we have rejected this tradeoff by applying schema metaprogramming, eliminating boilerplate but keeping the protection of static typing, thereby further improving agility to quickly modify data pipelines without fear.
DSSML24_tspann_CodelessGenerativeAIPipelines
Codeless Generative AI Pipelines
(GenAI with Milvus)
https://ml.dssconf.pl/user.html#!/lecture/DSSML24-041a/rate
Discover the potential of real-time streaming in the context of GenAI as we delve into the intricacies of Apache NiFi and its capabilities. Learn how this tool can significantly simplify the data engineering workflow for GenAI applications, allowing you to focus on the creative aspects rather than the technical complexities. I will guide you through practical examples and use cases, showing the impact of automation on prompt building. From data ingestion to transformation and delivery, witness how Apache NiFi streamlines the entire pipeline, ensuring a smooth and hassle-free experience.
Timothy Spann
https://www.youtube.com/@FLaNK-Stack
https://medium.com/@tspann
https://www.datainmotion.dev/
milvus, unstructured data, vector database, zilliz, cloud, vectors, python, deep learning, generative ai, genai, nifi, kafka, flink, streaming, iot, edge
Recently uploaded (20)
A presentation that explain the Power BI Licensing
A presentation that explain the Power BI Licensing
STATATHON: Unleashing the Power of Statistics in a 48-Hour Knowledge Extravag...
STATATHON: Unleashing the Power of Statistics in a 48-Hour Knowledge Extravag...
The Building Blocks of QuestDB, a Time Series Database
The Building Blocks of QuestDB, a Time Series Database
4th Modern Marketing Reckoner by MMA Global India & Group M: 60+ experts on W...
4th Modern Marketing Reckoner by MMA Global India & Group M: 60+ experts on W...
Global Situational Awareness of A.I. and where its headed
Global Situational Awareness of A.I. and where its headed
Predictably Improve Your B2B Tech Company's Performance by Leveraging Data
Predictably Improve Your B2B Tech Company's Performance by Leveraging Data
06-12-2024-BudapestDataForum-BuildingReal-timePipelineswithFLaNK AIM
06-12-2024-BudapestDataForum-BuildingReal-timePipelineswithFLaNK AIM
Analysis insight about a Flyball dog competition team's performance
Analysis insight about a Flyball dog competition team's performance
Intelligence supported media monitoring in veterinary medicine
Intelligence supported media monitoring in veterinary medicine
ViewShift: Hassle-free Dynamic Policy Enforcement for Every Data Lake
ViewShift: Hassle-free Dynamic Policy Enforcement for Every Data Lake
Challenges of Nation Building-1.pptx with more important
Challenges of Nation Building-1.pptx with more important
End-to-end pipeline agility - Berlin Buzzwords 2024
End-to-end pipeline agility - Berlin Buzzwords 2024
HPCC Systems 6.4.0 and Beyond
- 1. October 4, 2017 Gavin Halliday, Enterprise Architect, LexisNexis HPCC Systems 6.4.0 and Beyond
- 2. Productivity
- 3. Productivity 6.4 – ECL Developers • Git • 64 bit windows client • Improved compiler speed • --fastsyntax option • LOOKUP - compile time record resolution • IDE • Usability • Language integration (HIPIE, KEL, ESDL) • Dynamic ESDL • ML library in core and bundles HPCC Systems 6.4.0 and Beyond 3
- 4. Productivity 6.4 - Operations • File history • Cluster migration tool • Package map tools • Copying packages • Parts manipulation • Improved error reporting HPCC Systems 6.4.0 and Beyond 4
- 5. Productivity 7.0 • EclWatch • Dashboards • Better access to query statistics • Session management • Compiler improvements • Ecl indexer • --fastsyntax v2 • New config manager • Language enhancements • MATCHED(LEFT) HPCC Systems 6.4.0 and Beyond 5
- 7. Interoperability 6.4 • New support • Python 3 • Couchbase • AWS SQS (Simple Queue Service) • New capabilities • Nested datasets in R • Persistent data • Compile time embed • WsSQL HPCC Systems 6.4.0 and Beyond 7
- 8. Interoperability 7.0 • Spark • UnicodeLib • R? • Tensorflow? HPCC Systems 6.4.0 and Beyond 8
- 9. Performance
- 10. Performance 6.4 • Roxie • Xml reading • Index reading • EclWatch wildcards • Cassandra • ML Myriad interface for Logistic and Linear regression • New Hardware • Knights Landing (244 threads) • Containers HPCC Systems 6.4.0 and Beyond 10
- 11. Performance 7.0 • More of the same • More XML improvements • Memory manager • Thor • Remote file operations • Filtering and projecting • Keyed joins • R threading HPCC Systems 6.4.0 and Beyond 11
- 12. Security
- 13. Security • Roxie SSL • Encryption in transit • Restrict file access to signed code • Dropzone whitelists HPCC Systems 6.4.0 and Beyond 13
- 14. Reliability
- 15. Reliability • Fixes • Major minor point • RCA (Regress/Point incompatible) • Catching problems • Smoke test • Coverity • Regression suite • Performance suite HPCC Systems 6.4.0 and Beyond 15
- 16. Reliability HPCC Systems 6.4.0 and Beyond 16
- 17. Questions?