This presentation is about how Hong Leong Bank set up a private cloud for its database services and the rules of engagement of utilising a private cloud and getting funding for expansion.
Beginners Guide to High Availability for PostgresEDB
Highly available databases are essential to organizations depending on mission-critical, 24/7 access to data. PostgreSQL is widely recognized as an excellent open-source database, with critical maturity and features that allow organizations to scale and achieve high availability.
This webinar will explore:
- High availability concepts and workings
- RPO, RTO, and uptime in high availability
- Postgres high availability using:
- Streaming replication
- Logical replication
- Important high availability parameters in Postgres and options to monitor high availability.
- EDB tools (EDB Postgres Failover Manager, BART etc) to create a highly available Postgres architecture
Speaker:
Gaby Schilders
Senior Sales Engineer, EDB
Deep Dive - Usage of on premises data gateway for hybrid integration scenariosSajith C P Nair
Presentation delivered by Sajith C P, Integration Architect at the 2017 Global Integration Bootcamp, Bangalore.
https://www.biztalk360.com/gib2017-india/#speakers[inline]/7/
In this session the speaker talked about ‘on-premises data gateway’ as a secure centralized gateway that can be used for accessing on premise data from various Azure Services. He took a deep dive on how it works, how to install and various methods to troubleshoot connectivity. He concluded the session with few demos of its use in Azure Logic App, Microsoft Flow, Power Apps and Power BI.
Traditionally database systems were optimized either for OLAP either for OLTP workloads. Such mainstream DBMSes like Postgres,MySQL,... are mostly used for OLTP, while Greenplum, Vertica, Clickhouse, SparkSQL,... are oriented on analytic queries. But right now many companies do not want to have two different data stores for OLAP/OLTP and need to perform analytic queries on most recent data. I want to discuss which features should be added to Postgres to efficiently handle HTAP workload.
Red Hat® Ceph Storage and Network Solutions for Software Defined InfrastructureIntel® Software
This document discusses Intel's vision for software defined infrastructure (SDI) and provides examples of how their technology enables SDI. The key points are:
1. Intel's SDI vision is to provide dynamic, policy-driven management of compute, storage, and networking resources through abstraction, orchestration, and standards-based solutions.
2. Red Hat Ceph Storage is presented as an open source, scalable storage solution optimized for SDI through the use of commodity servers and SSDs.
3. Intel is contributing to open standards and growing an ecosystem of partners through their Network Builders program to accelerate the SDI transformation.
Public Sector Virtual Town Hall: High Availability for PostgreSQLEDB
Highly available databases are essential to organizations depending on mission-critical, 24/7 access to data. PostgreSQL is widely recognized as an excellent open-source database, with critical maturity and features that allow organizations to scale and achieve high availability.
This webinar will explore:
High availability concepts and workings
RPO, RTO, and uptime in high availability
Postgres high availability using streaming replication and logical replication
Important high availability parameters in PostgreSQL and options to monitor high availability
EDB tools (EDB Postgres Failover Manager, BART etc) to create a highly available Postgres architecture
Highly available databases are essential to organizations depending on mission-critical, 24/7 access to data. Postgres is widely recognized as an excellent open-source database, with critical maturity and features that allow organizations to scale and achieve high availability.
This webinar will explore:
- Evolution of replication in Postgres
- Streaming replication
- Logical replication
- Replication for high availability
- Important high availability parameters
- Options to monitor high availability
- HA infrastructure to patch the database with minimal downtime
- EDB Postgres Failover Manager (EFM)
- EDB tools to create a highly available Postgres architecture
Hadoop {Submarine} Project: Running Deep Learning Workloads on YARNDataWorks Summit
Deep learning is useful for enterprises tasks in the field of speech recognition, image classification, AI chatbots and machine translation, just to name a few.
In order to train deep learning/machine learning models, applications such as TensorFlow / MXNet / Caffe / XGBoost can be leveraged. And sometimes these applications will be used together to solve different problems.
To make distributed deep learning/machine learning applications easily launched, managed, monitored. Hadoop community has introduced Submarine project along with other improvements such as first-class GPU support, container-DNS support, scheduling improvements, etc. These improvements make distributed deep learning/machine learning applications run on YARN as simple as running it locally, which can let machine-learning engineers focus on algorithms instead of worrying about underlying infrastructure. Also, YARN can better manage a shared cluster which runs deep learning/machine learning and other services/ETL jobs with these improvements.
In this session, we will take a closer look at Submarine project as well as other improvements and show how to run these deep learning workloads on YARN with demos. Audiences can start trying running these workloads on YARN after this talk.
Speakers:
Sunil Govindan, Staff Engineer
Hortonworks
Zhankun Tank, Staff Engineer
Hortonworks
Future Architecture of Streaming Analytics: Capitalizing on the Analytics of ...DataWorks Summit
The proliferation of connected devices and sensors is leading the Digital Transformation. By 2020 there will be over 20 billion connected devices. Data from these devices need to be ingested at extreme speeds in order to be analyzed before the data decays. The life cycle of the data is critical in revealing what insight can be revealed and how quickly they can be acted upon.
In this session we will look at the past, present and future architecture trends streaming analytics. We will look at how to turn all the data from devices into actionable insights and dive into recommendations for streaming architecture depending on the data streams and time factor of the data. We will also discuss how to manage all the sensor data, understand the life cycle cost of the data, and how to scale capacity and capability easily with a modern infrastructure strategy.
Beginners Guide to High Availability for PostgresEDB
Highly available databases are essential to organizations depending on mission-critical, 24/7 access to data. PostgreSQL is widely recognized as an excellent open-source database, with critical maturity and features that allow organizations to scale and achieve high availability.
This webinar will explore:
- High availability concepts and workings
- RPO, RTO, and uptime in high availability
- Postgres high availability using:
- Streaming replication
- Logical replication
- Important high availability parameters in Postgres and options to monitor high availability.
- EDB tools (EDB Postgres Failover Manager, BART etc) to create a highly available Postgres architecture
Speaker:
Gaby Schilders
Senior Sales Engineer, EDB
Deep Dive - Usage of on premises data gateway for hybrid integration scenariosSajith C P Nair
Presentation delivered by Sajith C P, Integration Architect at the 2017 Global Integration Bootcamp, Bangalore.
https://www.biztalk360.com/gib2017-india/#speakers[inline]/7/
In this session the speaker talked about ‘on-premises data gateway’ as a secure centralized gateway that can be used for accessing on premise data from various Azure Services. He took a deep dive on how it works, how to install and various methods to troubleshoot connectivity. He concluded the session with few demos of its use in Azure Logic App, Microsoft Flow, Power Apps and Power BI.
Traditionally database systems were optimized either for OLAP either for OLTP workloads. Such mainstream DBMSes like Postgres,MySQL,... are mostly used for OLTP, while Greenplum, Vertica, Clickhouse, SparkSQL,... are oriented on analytic queries. But right now many companies do not want to have two different data stores for OLAP/OLTP and need to perform analytic queries on most recent data. I want to discuss which features should be added to Postgres to efficiently handle HTAP workload.
Red Hat® Ceph Storage and Network Solutions for Software Defined InfrastructureIntel® Software
This document discusses Intel's vision for software defined infrastructure (SDI) and provides examples of how their technology enables SDI. The key points are:
1. Intel's SDI vision is to provide dynamic, policy-driven management of compute, storage, and networking resources through abstraction, orchestration, and standards-based solutions.
2. Red Hat Ceph Storage is presented as an open source, scalable storage solution optimized for SDI through the use of commodity servers and SSDs.
3. Intel is contributing to open standards and growing an ecosystem of partners through their Network Builders program to accelerate the SDI transformation.
Public Sector Virtual Town Hall: High Availability for PostgreSQLEDB
Highly available databases are essential to organizations depending on mission-critical, 24/7 access to data. PostgreSQL is widely recognized as an excellent open-source database, with critical maturity and features that allow organizations to scale and achieve high availability.
This webinar will explore:
High availability concepts and workings
RPO, RTO, and uptime in high availability
Postgres high availability using streaming replication and logical replication
Important high availability parameters in PostgreSQL and options to monitor high availability
EDB tools (EDB Postgres Failover Manager, BART etc) to create a highly available Postgres architecture
Highly available databases are essential to organizations depending on mission-critical, 24/7 access to data. Postgres is widely recognized as an excellent open-source database, with critical maturity and features that allow organizations to scale and achieve high availability.
This webinar will explore:
- Evolution of replication in Postgres
- Streaming replication
- Logical replication
- Replication for high availability
- Important high availability parameters
- Options to monitor high availability
- HA infrastructure to patch the database with minimal downtime
- EDB Postgres Failover Manager (EFM)
- EDB tools to create a highly available Postgres architecture
Hadoop {Submarine} Project: Running Deep Learning Workloads on YARNDataWorks Summit
Deep learning is useful for enterprises tasks in the field of speech recognition, image classification, AI chatbots and machine translation, just to name a few.
In order to train deep learning/machine learning models, applications such as TensorFlow / MXNet / Caffe / XGBoost can be leveraged. And sometimes these applications will be used together to solve different problems.
To make distributed deep learning/machine learning applications easily launched, managed, monitored. Hadoop community has introduced Submarine project along with other improvements such as first-class GPU support, container-DNS support, scheduling improvements, etc. These improvements make distributed deep learning/machine learning applications run on YARN as simple as running it locally, which can let machine-learning engineers focus on algorithms instead of worrying about underlying infrastructure. Also, YARN can better manage a shared cluster which runs deep learning/machine learning and other services/ETL jobs with these improvements.
In this session, we will take a closer look at Submarine project as well as other improvements and show how to run these deep learning workloads on YARN with demos. Audiences can start trying running these workloads on YARN after this talk.
Speakers:
Sunil Govindan, Staff Engineer
Hortonworks
Zhankun Tank, Staff Engineer
Hortonworks
Future Architecture of Streaming Analytics: Capitalizing on the Analytics of ...DataWorks Summit
The proliferation of connected devices and sensors is leading the Digital Transformation. By 2020 there will be over 20 billion connected devices. Data from these devices need to be ingested at extreme speeds in order to be analyzed before the data decays. The life cycle of the data is critical in revealing what insight can be revealed and how quickly they can be acted upon.
In this session we will look at the past, present and future architecture trends streaming analytics. We will look at how to turn all the data from devices into actionable insights and dive into recommendations for streaming architecture depending on the data streams and time factor of the data. We will also discuss how to manage all the sensor data, understand the life cycle cost of the data, and how to scale capacity and capability easily with a modern infrastructure strategy.
This document discusses using data virtualization to accelerate application projects by 50%. It outlines some common problems with physical data copies, such as bottlenecks, bugs due to old data, difficulty creating subsets, and delays. The document then introduces the concept of using a data virtualization appliance to take snapshots of production data and create thin clones for development and testing environments. This allows for fast, full-sized, self-service clones that can be refreshed quickly. Use cases discussed include improved development and testing workflows, faster production support like recovery and migration, and enabling continuous business intelligence functions.
Integrating data stored in rdbms and hadoopleorick lin
This document discusses two approaches to integrating data from an RDBMS and Hadoop using Spark: (1) Spark with JdbcRDD, which allows reading from an RDBMS into RDDs but has complex system configuration, and (2) Spark with JDBC Client, which has simpler system configuration but a more complex programming process. It also provides examples of using each approach to perform joins between the data sources.
During this webinar, we will review best practices and lessons learned from working with large and mid-size companies on their deployment of PostgreSQL. We will explore the practices that helped industry leaders move through these stages quickly, and get as much value out of PostgreSQL as possible without incurring undue risk.
We have identified a set of levers that companies can use to accelerate their success with PostgreSQL:
- Application Tiering
- Collaboration between DBAs and Development Teams
- Evangelizing
- Standardization and Automation
- Balance of Migration and New Development
This document summarizes the roles of servers in a Hadoop cluster, including manager, name nodes, edge nodes, and data nodes. It discusses hardware considerations for Hadoop cluster design like CPU to memory to disk ratios for different use cases. It also provides an overview of Dell's Hadoop solutions that integrate PowerEdge servers, Dell Networking switches, and support from Etu for analytic software and Dell Professional Services for implementation. It briefly discusses futures around in-memory processing and virtualized Hadoop deployments.
Automating a PostgreSQL High Availability Architecture with AnsibleEDB
Highly available databases are essential to organizations depending on mission-critical, 24/7 access to data. Postgres is widely recognized as an excellent open-source database, with critical maturity and features that allow organizations to scale and achieve high availability.
EDB reference architectures are designed to help new and existing users alike to quickly design a deployment architecture that suits their needs. Users can use these reference architectures as a blueprint or as the basis for a design that enhances and extends the functionality and features offered.
This webinar will explore:
- Concepts of High Availability
- Quick review of EDB reference architectures
- EDB tools to create a highly available PostgreSQL architecture
- Options for automating the deployment of reference architectures
- EDB Ansible® roles helping in automating the deployment of reference architectures
- Features and capabilities of Ansible roles
- Automating the provisioning of the resources in the cloud using Terraform™
Beginner's Guide to High Availability for PostgresEDB
Highly available databases are essential to organizations depending on mission-critical, 24/7 access to data. PostgreSQL is widely recognized as an excellent open-source database, with critical maturity and features that allow organizations to scale and achieve high availability.
This webinar will explore:
- High availability concepts and workings
- RPO, RTO, and uptime in high availability
- Postgres high availability using
- Streaming replication
- Logical replication
- Important high availability parameters in Postgres and options to monitor high availability.
- EDB tools (EDB Postgres Failover Manager, BART etc) to create a highly available Postgres architecture
Lessons and Observations Scaling a Time Series DatabaseInfluxData
InfluxData builds a Time Series Platform primarily deployed for DevOps and IoT monitoring. This talk presents several lessons learned while scaling the platform across a large number of deployments—from single server open source instances to highly available high-throughput clusters.
This talk presents a number of failure conditions that informed subsequent design choices. Ryan will discuss designing backpressure in an AP system with 10’s of thousands of resource-limited writers; trade-offs between monolithic and service-oriented database implementations; and lessons learned implementing multiple query processing systems.
This presentation discusses how data constraints negatively impact IT and businesses. It introduces Delphix as a solution to virtualize data and eliminate the data constraint. The key points covered are:
I. The problem of data constraints in IT which include strains on IT resources, huge costs, and lack of awareness of the problem by companies.
II. Delphix addresses this by providing virtual data using file-level snapshots and compression to deliver fast, parallel environments without moving large amounts of data.
III. The solution enables use cases such as accelerated development, QA, disaster recovery, real-time business intelligence, and application modernization.
Flex Cloud - Conceptual Design - ver 0.2David Pasek
The Energy
=========
The cost of energy is increasing. A significant part of electrical energy cost is the cost of distribution. That's the reason why the popularity of small home solar systems increases. That's the way how to generate and consume electricity locally and be independent of the distribution network. However, we have a problem. "Green Energy" from solar, wind, and hydroelectric power stations is difficult to distribute via the electrical grid. Energy accumulation (batteries, pumped storage power plant, etc.) is costly and for the traditional electrical grid is very difficult to automatically manage the distribution of so many energy sources.
The Cloud Computing
=================
The demand for cloud (computing and storage) capacity is increasing year by year. Internet bandwidth increases and cost decreases every year. 5G Networks and SD-WANs are on the radar. Cloud Computing is operated on data centers. The significant part of data center costs is the cost of energy.
The potential synergy between Energetics and Cloud Computing
=================================================
The solution is to consume electricity in the proximity of green power generators. Excess electricity is accumulated into batteries but batteries capacity is limited. We should treat batteries like a cache or buffer to overcome times when green energy does not generate energy but we have local demand. However, when we have excess electricity and the battery (cache/buffer) is full, instead of providing the energy into the electrical grid, the excess electricity can be consumed by a computer system providing compute resources to cloud computing consumers over the internet. This is the form of Distributed Cloud Computing.
Cloud-Native Applications
====================
So, let's assume we will have Distributed Cloud Computing with so-called Spot Compute Resource Pools". Spot Compute Resource Pools are computing resources that can appear or disappear within hours or minutes. This is not optimal IT infrastructure for traditional software applications which are not infrastructure aware. For such distributed cloud computing the software applications must be designed and developed with infrastructure resources ephemerality in mind. In other words, Cloud-Native Applications must be able to leverage ephemeral compute resource pools and know how to use "Spot Compute Resource Pools".
This was co-presented at the OpenStack Summit 2013 in Portland by Kamesh Pemmaraju, Product Manager from Dell and Neil Levine Inktank.
Inktank Ceph is a transformational open source storage solution fully integrated into OpenStack providing scalable object and block storage (via Cinder) using commodity servers. The Ceph solution is resilient to failures, uses storage efficiently, and performs well under a variety of VM Workloads.
Dell Crowbar is an open source software framework that can automatically deploy Ceph and OpenStack on bare metal servers in a matter of hours. The Ceph team worked with Dell to create a Ceph barclamp (a crowbar extention) that integrates Glance, Cinder, and Nova-Volume. As a result, it is lot faster and easier to install, configure, and manage a sizable OpenStack and Ceph cluster that is tightly integrated and cost- optimized.
Hear how OpenStack users can address their storage deployment challenges:
Considerations when selecting a cloud storage system
Overview of the Ceph architecture with unique features and benefits
Overview of Dell Crowbar and how it can automate and simplify Ceph/OpenStack deployments Best practices in deploying cloud storage with Ceph and OpenStack
Next Generation Scheduling for YARN and K8s: For Hybrid Cloud/On-prem Environ...DataWorks Summit
Scheduler of a container orchestration system, such as YARN and K8s, is a critical component that users rely on to plan resources and manage applications.
And if we assess where we are today, in YARN effectively it had two power schedulers (Fair and Capacity scheduler) and both serve many strong use cases in big data ecosystem. It can scale up to 50k nodes per cluster, and schedule 20k containers per second, and extremely efficient to manage batch workloads.
K8s default scheduler is an industry-proven solution to efficiently manage long-running services. As more big data apps are moving to K8s and cloud world, but many features like hierarchical queues to support multi-tenancy better, fairness resource sharing, and preemption, etc. are either missing or not mature enough at this point of time to support big data apps running on K8s.
At this point, there is no solution that exists to address the needs of having a unified resource scheduling experiences across platforms. That makes it extremely difficult to manage workloads running on different environments, from on-premise to cloud.
Hence evolving a common scheduler powered from YARN and K8s’s legacy capabilities and improving towards cloud use cases will focus more on use cases like:
Better bin-packing scheduling (and gang scheduling)
Autoscale up and shrink policy management
Effectively run batch workloads and services with clear SLA’s
In summary, we are improving core scheduling capabilities to manage both K8s and YARN cluster which is cloud aware as a separate initiative and above-mentioned cases will be the core focus of this initiative. More details of our works will be presented in this talk.
The document discusses GemFire, a memory-oriented key-value data store from Pivotal. It provides three use cases where GemFire was used to scale online ticket sales, global electronic trading systems, and the largest railway in China. GemFire enabled significant performance improvements like 50-100x faster queries and the ability to scale elastically with data growth. The document also summarizes GemFire's features like data partitioning, replication, off-heap memory, and integration with Apache Geode.
Red Hat Storage Day Boston - Red Hat Gluster Storage vs. Traditional Storage ...Red_Hat_Storage
Red Hat Gluster Storage provides a software-defined storage solution that is more cost efficient and flexible than traditional storage appliances. It leverages standard x86 hardware and has open source architecture with no vendor lock-in. A comparison shows Gluster Storage outperforms EMC Isilon on factors like cost, scalability, data protection methods, access protocols, and management capabilities. Gluster Storage is positioned to go beyond traditional storage by supporting containers, disaster recovery in cloud environments, and its roadmap includes additional advanced features.
- POWER9 delivers 2x the compute resources per socket through new cores optimized for stronger thread performance and efficiency.
- It features direct memory attach with up to 8 DDR4 ports and buffered memory with 8 channels for scale-out and scale-up configurations.
- The processor provides leadership hardware acceleration through enhanced on-chip acceleration, NVLink 2.0, CAPI 2.0, and a new open CAPI interface using 25G signaling for high bandwidth and low latency attachment of accelerators.
IBM World of Watson 2016 - DB2 Analytics Accelerator on CloudDaniel Martin
IBM is introducing a new deployment option for the DB2 Analytics Accelerator on Cloud using dashDB as the acceleration engine. This provides customers with a hybrid cloud offering that gives the flexibility of running the Accelerator either on-premises or in the cloud. The Cloud deployment offers benefits like monthly pricing, hardware provisioning by IBM, and fast provisioning time. Initial focus areas include basic Accelerator functionality for offloading queries to the cloud, with a roadmap to continuously expand features and functionality.
This document discusses strategies for performing database upgrades with zero downtime. It explains how to optimize parallelism during the upgrade process by adjusting parameters like the number of parallel processes and processors per PDB. Refreshing database statistics before the upgrade and tuning the post-upgrade recompilation can reduce downtime. Benchmark results show that increasing the parallelism through higher CPU_COUNT values can significantly decrease the total upgrade time.
Building Scalable Applications using Pivotal Gemfire/Apache Geodeimcpune
This document discusses using Pivotal GemFire/Apache Geode to build scalable applications. It provides an overview of GemFire concepts like distributed caching and integration with traditional databases. It also presents a case study of how the Indian Railways used GemFire to improve performance and scalability of its online ticket booking system, allowing it to support over 200,000 concurrent purchases. The document concludes by outlining GemFire's roadmap and providing information on how to get involved with the GemFire community.
Using Databases and Containers From Development to DeploymentAerospike, Inc.
This document discusses using containers and databases together from development to production. It addresses challenges like data redundancy, dynamic cluster formation and healing when containers start and stop. It proposes that existing architectures are broken and presents Aerospike as a solution, being self-organizing, self-healing and optimized for flash storage. It demonstrates building an app with Python, Aerospike and Docker, deploying to a Swarm cluster, and scaling the database and web tiers through containers.
Faster, more Secure Application Modernization and Replatforming with PKS - Ku...VMware Tanzu
Faster, more Secure Application Modernization and Replatforming with PKS - Kubernetes for the Enterprise - London
Alex Ley
Associate Director, App Transformation, Pivotal EMEA
28th March 2018
FlexPod delivers new integrated infrastructure validated designs with NetApp All-Flash and Cisco ACI that deliver new levels of performance and the ability to meet business objectives
This document discusses using data virtualization to accelerate application projects by 50%. It outlines some common problems with physical data copies, such as bottlenecks, bugs due to old data, difficulty creating subsets, and delays. The document then introduces the concept of using a data virtualization appliance to take snapshots of production data and create thin clones for development and testing environments. This allows for fast, full-sized, self-service clones that can be refreshed quickly. Use cases discussed include improved development and testing workflows, faster production support like recovery and migration, and enabling continuous business intelligence functions.
Integrating data stored in rdbms and hadoopleorick lin
This document discusses two approaches to integrating data from an RDBMS and Hadoop using Spark: (1) Spark with JdbcRDD, which allows reading from an RDBMS into RDDs but has complex system configuration, and (2) Spark with JDBC Client, which has simpler system configuration but a more complex programming process. It also provides examples of using each approach to perform joins between the data sources.
During this webinar, we will review best practices and lessons learned from working with large and mid-size companies on their deployment of PostgreSQL. We will explore the practices that helped industry leaders move through these stages quickly, and get as much value out of PostgreSQL as possible without incurring undue risk.
We have identified a set of levers that companies can use to accelerate their success with PostgreSQL:
- Application Tiering
- Collaboration between DBAs and Development Teams
- Evangelizing
- Standardization and Automation
- Balance of Migration and New Development
This document summarizes the roles of servers in a Hadoop cluster, including manager, name nodes, edge nodes, and data nodes. It discusses hardware considerations for Hadoop cluster design like CPU to memory to disk ratios for different use cases. It also provides an overview of Dell's Hadoop solutions that integrate PowerEdge servers, Dell Networking switches, and support from Etu for analytic software and Dell Professional Services for implementation. It briefly discusses futures around in-memory processing and virtualized Hadoop deployments.
Automating a PostgreSQL High Availability Architecture with AnsibleEDB
Highly available databases are essential to organizations depending on mission-critical, 24/7 access to data. Postgres is widely recognized as an excellent open-source database, with critical maturity and features that allow organizations to scale and achieve high availability.
EDB reference architectures are designed to help new and existing users alike to quickly design a deployment architecture that suits their needs. Users can use these reference architectures as a blueprint or as the basis for a design that enhances and extends the functionality and features offered.
This webinar will explore:
- Concepts of High Availability
- Quick review of EDB reference architectures
- EDB tools to create a highly available PostgreSQL architecture
- Options for automating the deployment of reference architectures
- EDB Ansible® roles helping in automating the deployment of reference architectures
- Features and capabilities of Ansible roles
- Automating the provisioning of the resources in the cloud using Terraform™
Beginner's Guide to High Availability for PostgresEDB
Highly available databases are essential to organizations depending on mission-critical, 24/7 access to data. PostgreSQL is widely recognized as an excellent open-source database, with critical maturity and features that allow organizations to scale and achieve high availability.
This webinar will explore:
- High availability concepts and workings
- RPO, RTO, and uptime in high availability
- Postgres high availability using
- Streaming replication
- Logical replication
- Important high availability parameters in Postgres and options to monitor high availability.
- EDB tools (EDB Postgres Failover Manager, BART etc) to create a highly available Postgres architecture
Lessons and Observations Scaling a Time Series DatabaseInfluxData
InfluxData builds a Time Series Platform primarily deployed for DevOps and IoT monitoring. This talk presents several lessons learned while scaling the platform across a large number of deployments—from single server open source instances to highly available high-throughput clusters.
This talk presents a number of failure conditions that informed subsequent design choices. Ryan will discuss designing backpressure in an AP system with 10’s of thousands of resource-limited writers; trade-offs between monolithic and service-oriented database implementations; and lessons learned implementing multiple query processing systems.
This presentation discusses how data constraints negatively impact IT and businesses. It introduces Delphix as a solution to virtualize data and eliminate the data constraint. The key points covered are:
I. The problem of data constraints in IT which include strains on IT resources, huge costs, and lack of awareness of the problem by companies.
II. Delphix addresses this by providing virtual data using file-level snapshots and compression to deliver fast, parallel environments without moving large amounts of data.
III. The solution enables use cases such as accelerated development, QA, disaster recovery, real-time business intelligence, and application modernization.
Flex Cloud - Conceptual Design - ver 0.2David Pasek
The Energy
=========
The cost of energy is increasing. A significant part of electrical energy cost is the cost of distribution. That's the reason why the popularity of small home solar systems increases. That's the way how to generate and consume electricity locally and be independent of the distribution network. However, we have a problem. "Green Energy" from solar, wind, and hydroelectric power stations is difficult to distribute via the electrical grid. Energy accumulation (batteries, pumped storage power plant, etc.) is costly and for the traditional electrical grid is very difficult to automatically manage the distribution of so many energy sources.
The Cloud Computing
=================
The demand for cloud (computing and storage) capacity is increasing year by year. Internet bandwidth increases and cost decreases every year. 5G Networks and SD-WANs are on the radar. Cloud Computing is operated on data centers. The significant part of data center costs is the cost of energy.
The potential synergy between Energetics and Cloud Computing
=================================================
The solution is to consume electricity in the proximity of green power generators. Excess electricity is accumulated into batteries but batteries capacity is limited. We should treat batteries like a cache or buffer to overcome times when green energy does not generate energy but we have local demand. However, when we have excess electricity and the battery (cache/buffer) is full, instead of providing the energy into the electrical grid, the excess electricity can be consumed by a computer system providing compute resources to cloud computing consumers over the internet. This is the form of Distributed Cloud Computing.
Cloud-Native Applications
====================
So, let's assume we will have Distributed Cloud Computing with so-called Spot Compute Resource Pools". Spot Compute Resource Pools are computing resources that can appear or disappear within hours or minutes. This is not optimal IT infrastructure for traditional software applications which are not infrastructure aware. For such distributed cloud computing the software applications must be designed and developed with infrastructure resources ephemerality in mind. In other words, Cloud-Native Applications must be able to leverage ephemeral compute resource pools and know how to use "Spot Compute Resource Pools".
This was co-presented at the OpenStack Summit 2013 in Portland by Kamesh Pemmaraju, Product Manager from Dell and Neil Levine Inktank.
Inktank Ceph is a transformational open source storage solution fully integrated into OpenStack providing scalable object and block storage (via Cinder) using commodity servers. The Ceph solution is resilient to failures, uses storage efficiently, and performs well under a variety of VM Workloads.
Dell Crowbar is an open source software framework that can automatically deploy Ceph and OpenStack on bare metal servers in a matter of hours. The Ceph team worked with Dell to create a Ceph barclamp (a crowbar extention) that integrates Glance, Cinder, and Nova-Volume. As a result, it is lot faster and easier to install, configure, and manage a sizable OpenStack and Ceph cluster that is tightly integrated and cost- optimized.
Hear how OpenStack users can address their storage deployment challenges:
Considerations when selecting a cloud storage system
Overview of the Ceph architecture with unique features and benefits
Overview of Dell Crowbar and how it can automate and simplify Ceph/OpenStack deployments Best practices in deploying cloud storage with Ceph and OpenStack
Next Generation Scheduling for YARN and K8s: For Hybrid Cloud/On-prem Environ...DataWorks Summit
Scheduler of a container orchestration system, such as YARN and K8s, is a critical component that users rely on to plan resources and manage applications.
And if we assess where we are today, in YARN effectively it had two power schedulers (Fair and Capacity scheduler) and both serve many strong use cases in big data ecosystem. It can scale up to 50k nodes per cluster, and schedule 20k containers per second, and extremely efficient to manage batch workloads.
K8s default scheduler is an industry-proven solution to efficiently manage long-running services. As more big data apps are moving to K8s and cloud world, but many features like hierarchical queues to support multi-tenancy better, fairness resource sharing, and preemption, etc. are either missing or not mature enough at this point of time to support big data apps running on K8s.
At this point, there is no solution that exists to address the needs of having a unified resource scheduling experiences across platforms. That makes it extremely difficult to manage workloads running on different environments, from on-premise to cloud.
Hence evolving a common scheduler powered from YARN and K8s’s legacy capabilities and improving towards cloud use cases will focus more on use cases like:
Better bin-packing scheduling (and gang scheduling)
Autoscale up and shrink policy management
Effectively run batch workloads and services with clear SLA’s
In summary, we are improving core scheduling capabilities to manage both K8s and YARN cluster which is cloud aware as a separate initiative and above-mentioned cases will be the core focus of this initiative. More details of our works will be presented in this talk.
The document discusses GemFire, a memory-oriented key-value data store from Pivotal. It provides three use cases where GemFire was used to scale online ticket sales, global electronic trading systems, and the largest railway in China. GemFire enabled significant performance improvements like 50-100x faster queries and the ability to scale elastically with data growth. The document also summarizes GemFire's features like data partitioning, replication, off-heap memory, and integration with Apache Geode.
Red Hat Storage Day Boston - Red Hat Gluster Storage vs. Traditional Storage ...Red_Hat_Storage
Red Hat Gluster Storage provides a software-defined storage solution that is more cost efficient and flexible than traditional storage appliances. It leverages standard x86 hardware and has open source architecture with no vendor lock-in. A comparison shows Gluster Storage outperforms EMC Isilon on factors like cost, scalability, data protection methods, access protocols, and management capabilities. Gluster Storage is positioned to go beyond traditional storage by supporting containers, disaster recovery in cloud environments, and its roadmap includes additional advanced features.
- POWER9 delivers 2x the compute resources per socket through new cores optimized for stronger thread performance and efficiency.
- It features direct memory attach with up to 8 DDR4 ports and buffered memory with 8 channels for scale-out and scale-up configurations.
- The processor provides leadership hardware acceleration through enhanced on-chip acceleration, NVLink 2.0, CAPI 2.0, and a new open CAPI interface using 25G signaling for high bandwidth and low latency attachment of accelerators.
IBM World of Watson 2016 - DB2 Analytics Accelerator on CloudDaniel Martin
IBM is introducing a new deployment option for the DB2 Analytics Accelerator on Cloud using dashDB as the acceleration engine. This provides customers with a hybrid cloud offering that gives the flexibility of running the Accelerator either on-premises or in the cloud. The Cloud deployment offers benefits like monthly pricing, hardware provisioning by IBM, and fast provisioning time. Initial focus areas include basic Accelerator functionality for offloading queries to the cloud, with a roadmap to continuously expand features and functionality.
This document discusses strategies for performing database upgrades with zero downtime. It explains how to optimize parallelism during the upgrade process by adjusting parameters like the number of parallel processes and processors per PDB. Refreshing database statistics before the upgrade and tuning the post-upgrade recompilation can reduce downtime. Benchmark results show that increasing the parallelism through higher CPU_COUNT values can significantly decrease the total upgrade time.
Building Scalable Applications using Pivotal Gemfire/Apache Geodeimcpune
This document discusses using Pivotal GemFire/Apache Geode to build scalable applications. It provides an overview of GemFire concepts like distributed caching and integration with traditional databases. It also presents a case study of how the Indian Railways used GemFire to improve performance and scalability of its online ticket booking system, allowing it to support over 200,000 concurrent purchases. The document concludes by outlining GemFire's roadmap and providing information on how to get involved with the GemFire community.
Using Databases and Containers From Development to DeploymentAerospike, Inc.
This document discusses using containers and databases together from development to production. It addresses challenges like data redundancy, dynamic cluster formation and healing when containers start and stop. It proposes that existing architectures are broken and presents Aerospike as a solution, being self-organizing, self-healing and optimized for flash storage. It demonstrates building an app with Python, Aerospike and Docker, deploying to a Swarm cluster, and scaling the database and web tiers through containers.
Faster, more Secure Application Modernization and Replatforming with PKS - Ku...VMware Tanzu
Faster, more Secure Application Modernization and Replatforming with PKS - Kubernetes for the Enterprise - London
Alex Ley
Associate Director, App Transformation, Pivotal EMEA
28th March 2018
FlexPod delivers new integrated infrastructure validated designs with NetApp All-Flash and Cisco ACI that deliver new levels of performance and the ability to meet business objectives
This document discusses strategies for modernizing applications and moving workloads to Kubernetes and container platforms like Pivotal Container Service (PKS). It recommends identifying candidate applications using buckets based on factors like programming language, dependencies, and access to source code. It outlines assessing applications' business value and technical quality using Gartner's TIME methodology to prioritize efforts. The document provides an overview of PKS and how it can provide benefits like increased speed, stability, scalability and cost savings. It recommends starting projects by pushing a few applications to production on PKS to measure ROI metrics.
This document discusses strategies for modernizing applications and moving workloads to Kubernetes and container platforms like Pivotal Container Service (PKS). It recommends identifying candidate applications using buckets based on factors like programming language, dependencies, and access to source code. It outlines assessing applications' business value and technical quality using Gartner's TIME methodology to prioritize efforts. The document provides an overview of PKS and how it can provide benefits like increased speed, security, scalability and cost savings. It recommends starting projects by pushing a few applications to production on PKS to measure ROI metrics.
Oracle Database Consolidation with FlexPod on Cisco UCSNetApp
Cisco and Oracle as technology front-runners provide YOU the tools you need to optimize your Oracle environments! John McAbel, Senior Product Manager - Oracle Solutions on UCS at Cisco Systems, explains how NetApp and Cisco are providing a flexible infrastructure that helps prepare organizations for today, and for future business growth and change.
This document provides a summary of Chaitanya Prati's work experience and qualifications. He has over 10 years of experience as an Oracle DBA providing support for multi-terabyte Oracle databases. Currently he works as an onsite technical lead for Wipro Technologies providing Oracle database administration support to Citigroup. His responsibilities include managing critical compliance applications, implementing GoldenGate replication, and resolving performance issues. He is proficient in technologies like Oracle RAC, ASM, GoldenGate and tools like SQL*Plus and Toad.
NetApp FlexPod continues its Converged Infrastructure leadership with new validated designs and program enhancements -- here's the latest FlexPod news as of Winter 2013/14.
Lily Craps, responsible for the Mainframe outsourcing project at SDWorx, explains how the moving of their mainframe to a shared environment at NRB, enabled ‘economies of scale’ on infrastructure costs for hardware and software. She describes the process, from starting the outsourcing study, over the RFI/RFP process, the selection of the provider, the contract negotiations and the migration project, next to the criteria for choosing NRB and an Infrastructure As A Service –cloud model.
Service-Level Objective for Serverless Applicationsalekn
Deploying commercial applications that meet their expected business needs is challenging due to the differences between how business goals are specified and how the system is evaluated. Furthermore, business goals are dynamic, requiring deployment to change constantly over time. Such difficulties make it costly to maintain application quality as the underlying infrastructure is not always fast enough to keep up with business changes. Nowadays, serverless opens a new approach to build application. By abstracting out the deployment details, serverless application can be implemented with minimum deployment efforts. Serverless also reduces maintenance cost with auto-scaling and pay-as-you-go. Such abilities make us believe that by adopting serverless, we can build application that can meet and quickly adapt to business goals.
However, simply writing applications with serverless is not sufficient. Due to best-effort invocation mechanisms and the lack of application structure awareness, serverless performance is highly variable and often fails to support applications with rigorous quality of service requirements. In this study, we aim to mitigate such limitations by coupling serverless deployment with business needs. In particular, we define an Serverless Service-Level Objective (SLO) interface that allows developers to describe their application structure and business goals in terms of software-level objectives. We implement an SLO enforcer, which uses this information in combination with the system performance metrics to decide a proper serverless deployment and resource allocation for meeting business goals. The Serverless SLO leverages blueprint model, which allow developers to describe applications' architecture and runtime characteristics needs, to map application description to serverless function deployment on the top of Knative. We deploy our proposed system on KinD, a tool to run Kubernetes cluster over our local Docker container, and evaluate it with different system configurations. Evaluation results showed that SLO definition and enforcement helps serverless application use resources in accordance with business goals.
Cloud-Native Patterns and the Benefits of MySQL as a Platform Managed ServiceVMware Tanzu
You can’t have cloud-native applications without a modern approach to databases and backing services. Data professionals are looking for ways to transform how databases are provisioned and managed.
In this webinar, we’ll cover practical strategies you can employ to deliver improved business agility at the data layer. We’ll discuss the impact that microservices are having in the enterprise, and what this means for MySQL and other popular databases. Join us and learn the answers to these common questions:
● How can you meet the operational challenge of scaling the number of MySQL database instances and managing the fleet?
● Adding to this scale challenge, how can your MySQL instances maintain availability in a world where the underlying IT infrastructure is ephemeral?
● How can you secure data in motion?
● How can you enable self-service while maintaining control and governance?
We’ll cover these topics and share how enterprises like yours are delivering greater outcomes with our Pivotal Platform managed MySQL.
Now you can scale without fear of failure.
Presenters:
Judy Wang, Product Management
Jagdish Mirani, Product Marketing
The document discusses how Oracle Application Express (APEX) is the fastest route to developing applications in the cloud. It was designed for cloud computing with a simple architecture that allows easy deployment on Oracle's cloud infrastructure or on-premise. APEX requires only a browser to build powerful data-driven web apps using SQL and PL/SQL. It has a proven track record of being used by thousands of customers for cloud apps since 2002. The document argues that APEX's low-code development, portability across environments, and leveraging of Oracle databases and cloud services makes it ideal for cloud development.
Dayasekhar Babu has over 6 years of experience as an Oracle DBA with expertise in Oracle, RAC, and Golden Gate. He has experience installing, configuring, and administering Oracle databases. He is proficient in database backup, recovery, performance monitoring and tuning. He also has experience with MongoDB, MySQL, and Microsoft SQL Server databases.
Government Technology & Services Coalition & InfraGard NCR's Program: Cyber Security: Securing the Federal Cyber Domain by Strengthening Public-Private Partnership
Presentation: How do we Protect our Systems and Meet Compliance in a Rapidly Changing Environment: Cyber Security Information and Event Management
Presenter: Dr. Jim Murray, Technical Staff, HBB Systems, LLC
Description: With all the constant innovation in cyber, what is “cutting edge”? What constraints hinder innovation? How is technology being used to address the Executive Orders, comply to standards, and other meet other mandates? What areas still need resources, ideas and innovation? Join us to hear advances in cyber security technology and ways to protect and monitor systems that will provide for resilient infrastructures and incorporate new solutions.
VMworld 2013: Strategic Reasons for Classifying Workloads for Tier 1 Virtuali...VMworld
This document discusses the importance of classifying workloads before virtualizing tier 1 applications. Workload classification involves measuring existing application and database workloads to properly size and place them in a new virtualized environment. This reduces risks and speeds up implementation by providing the proper analysis. The document outlines challenges, opportunities, models, metrics, tools and an example MolsonCoors used workload classification to virtualize their SAP landscape.
How the Development Bank of Singapore solves on-prem compute capacity challen...Alluxio, Inc.
The Development Bank of Singapore (DBS) has evolved its data platforms over three generations to address big data challenges and the explosion of data. It now uses a hybrid cloud model with Alluxio to provide a unified namespace across on-prem and cloud storage for analytics workloads. Alluxio enables "zero-copy" cloud bursting by caching hot data and orchestrating analytics jobs between on-prem and cloud resources like AWS EMR and Google Dataproc. This provides dynamic scaling of compute capacity while retaining data locality. Alluxio also offers intelligent data tiering and policy-driven data migration to cloud storage over time for cost efficiency and management.
EMC Symmetrix VMAX: An Introduction to Enterprise Storage: Brian Boyd, Varrow...Brian Boyd
This session gives an overview of the EMC Symmetrix VMAC enterprise storage array. We will discuss the appropriate time to start looking at enterprise storage in your datacenter, the benefits and difference in technology between VMAC and other storage arrays, and give specific examples of how VMAX has helped out customers in their environments
Cisco Enhances Data Protection, Increases Bandwidth and Simplifies End to End Storage Management
Protect
• Enhance disaster recovery and Business Continuance
• Integrated FCIP on Director Class
Scale
• Nexus 9K for Storage Networking
• 100G /50G/25G IP Storage connectivity
Simplify Operations
• DCNM Connect
• Storage End-to-end Provisioning
Similar to Hlb private cloud rules of engagement idc (20)
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...Neo4j
Leonard Jayamohan, Partner & Generative AI Lead, Deloitte
This keynote will reveal how Deloitte leverages Neo4j’s graph power for groundbreaking digital twin solutions, achieving a staggering 100x performance boost. Discover the essential role knowledge graphs play in successful generative AI implementations. Plus, get an exclusive look at an innovative Neo4j + Generative AI solution Deloitte is developing in-house.
Dr. Sean Tan, Head of Data Science, Changi Airport Group
Discover how Changi Airport Group (CAG) leverages graph technologies and generative AI to revolutionize their search capabilities. This session delves into the unique search needs of CAG’s diverse passengers and customers, showcasing how graph data structures enhance the accuracy and relevance of AI-generated search results, mitigating the risk of “hallucinations” and improving the overall customer journey.
AI 101: An Introduction to the Basics and Impact of Artificial IntelligenceIndexBug
Imagine a world where machines not only perform tasks but also learn, adapt, and make decisions. This is the promise of Artificial Intelligence (AI), a technology that's not just enhancing our lives but revolutionizing entire industries.
“An Outlook of the Ongoing and Future Relationship between Blockchain Technologies and Process-aware Information Systems.” Invited talk at the joint workshop on Blockchain for Information Systems (BC4IS) and Blockchain for Trusted Data Sharing (B4TDS), co-located with with the 36th International Conference on Advanced Information Systems Engineering (CAiSE), 3 June 2024, Limassol, Cyprus.
UiPath Test Automation using UiPath Test Suite series, part 5DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 5. In this session, we will cover CI/CD with devops.
Topics covered:
CI/CD with in UiPath
End-to-end overview of CI/CD pipeline with Azure devops
Speaker:
Lyndsey Byblow, Test Suite Sales Engineer @ UiPath, Inc.
In the rapidly evolving landscape of technologies, XML continues to play a vital role in structuring, storing, and transporting data across diverse systems. The recent advancements in artificial intelligence (AI) present new methodologies for enhancing XML development workflows, introducing efficiency, automation, and intelligent capabilities. This presentation will outline the scope and perspective of utilizing AI in XML development. The potential benefits and the possible pitfalls will be highlighted, providing a balanced view of the subject.
We will explore the capabilities of AI in understanding XML markup languages and autonomously creating structured XML content. Additionally, we will examine the capacity of AI to enrich plain text with appropriate XML markup. Practical examples and methodological guidelines will be provided to elucidate how AI can be effectively prompted to interpret and generate accurate XML markup.
Further emphasis will be placed on the role of AI in developing XSLT, or schemas such as XSD and Schematron. We will address the techniques and strategies adopted to create prompts for generating code, explaining code, or refactoring the code, and the results achieved.
The discussion will extend to how AI can be used to transform XML content. In particular, the focus will be on the use of AI XPath extension functions in XSLT, Schematron, Schematron Quick Fixes, or for XML content refactoring.
The presentation aims to deliver a comprehensive overview of AI usage in XML development, providing attendees with the necessary knowledge to make informed decisions. Whether you’re at the early stages of adopting AI or considering integrating it in advanced XML development, this presentation will cover all levels of expertise.
By highlighting the potential advantages and challenges of integrating AI with XML development tools and languages, the presentation seeks to inspire thoughtful conversation around the future of XML development. We’ll not only delve into the technical aspects of AI-powered XML development but also discuss practical implications and possible future directions.
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdfMalak Abu Hammad
Discover how MongoDB Atlas and vector search technology can revolutionize your application's search capabilities. This comprehensive presentation covers:
* What is Vector Search?
* Importance and benefits of vector search
* Practical use cases across various industries
* Step-by-step implementation guide
* Live demos with code snippets
* Enhancing LLM capabilities with vector search
* Best practices and optimization strategies
Perfect for developers, AI enthusiasts, and tech leaders. Learn how to leverage MongoDB Atlas to deliver highly relevant, context-aware search results, transforming your data retrieval process. Stay ahead in tech innovation and maximize the potential of your applications.
#MongoDB #VectorSearch #AI #SemanticSearch #TechInnovation #DataScience #LLM #MachineLearning #SearchTechnology
Building Production Ready Search Pipelines with Spark and MilvusZilliz
Spark is the widely used ETL tool for processing, indexing and ingesting data to serving stack for search. Milvus is the production-ready open-source vector database. In this talk we will show how to use Spark to process unstructured data to extract vector representations, and push the vectors to Milvus vector database for search serving.
Full-RAG: A modern architecture for hyper-personalizationZilliz
Mike Del Balso, CEO & Co-Founder at Tecton, presents "Full RAG," a novel approach to AI recommendation systems, aiming to push beyond the limitations of traditional models through a deep integration of contextual insights and real-time data, leveraging the Retrieval-Augmented Generation architecture. This talk will outline Full RAG's potential to significantly enhance personalization, address engineering challenges such as data management and model training, and introduce data enrichment with reranking as a key solution. Attendees will gain crucial insights into the importance of hyperpersonalization in AI, the capabilities of Full RAG for advanced personalization, and strategies for managing complex data integrations for deploying cutting-edge AI solutions.
UiPath Test Automation using UiPath Test Suite series, part 6DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 6. In this session, we will cover Test Automation with generative AI and Open AI.
UiPath Test Automation with generative AI and Open AI webinar offers an in-depth exploration of leveraging cutting-edge technologies for test automation within the UiPath platform. Attendees will delve into the integration of generative AI, a test automation solution, with Open AI advanced natural language processing capabilities.
Throughout the session, participants will discover how this synergy empowers testers to automate repetitive tasks, enhance testing accuracy, and expedite the software testing life cycle. Topics covered include the seamless integration process, practical use cases, and the benefits of harnessing AI-driven automation for UiPath testing initiatives. By attending this webinar, testers, and automation professionals can gain valuable insights into harnessing the power of AI to optimize their test automation workflows within the UiPath ecosystem, ultimately driving efficiency and quality in software development processes.
What will you get from this session?
1. Insights into integrating generative AI.
2. Understanding how this integration enhances test automation within the UiPath platform
3. Practical demonstrations
4. Exploration of real-world use cases illustrating the benefits of AI-driven test automation for UiPath
Topics covered:
What is generative AI
Test Automation with generative AI and Open AI.
UiPath integration with generative AI
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2024/06/building-and-scaling-ai-applications-with-the-nx-ai-manager-a-presentation-from-network-optix/
Robin van Emden, Senior Director of Data Science at Network Optix, presents the “Building and Scaling AI Applications with the Nx AI Manager,” tutorial at the May 2024 Embedded Vision Summit.
In this presentation, van Emden covers the basics of scaling edge AI solutions using the Nx tool kit. He emphasizes the process of developing AI models and deploying them globally. He also showcases the conversion of AI models and the creation of effective edge AI pipelines, with a focus on pre-processing, model conversion, selecting the appropriate inference engine for the target hardware and post-processing.
van Emden shows how Nx can simplify the developer’s life and facilitate a rapid transition from concept to production-ready applications.He provides valuable insights into developing scalable and efficient edge AI solutions, with a strong focus on practical implementation.
Infrastructure Challenges in Scaling RAG with Custom AI modelsZilliz
Building Retrieval-Augmented Generation (RAG) systems with open-source and custom AI models is a complex task. This talk explores the challenges in productionizing RAG systems, including retrieval performance, response synthesis, and evaluation. We’ll discuss how to leverage open-source models like text embeddings, language models, and custom fine-tuned models to enhance RAG performance. Additionally, we’ll cover how BentoML can help orchestrate and scale these AI components efficiently, ensuring seamless deployment and management of RAG systems in the cloud.
Removing Uninteresting Bytes in Software FuzzingAftab Hussain
Imagine a world where software fuzzing, the process of mutating bytes in test seeds to uncover hidden and erroneous program behaviors, becomes faster and more effective. A lot depends on the initial seeds, which can significantly dictate the trajectory of a fuzzing campaign, particularly in terms of how long it takes to uncover interesting behaviour in your code. We introduce DIAR, a technique designed to speedup fuzzing campaigns by pinpointing and eliminating those uninteresting bytes in the seeds. Picture this: instead of wasting valuable resources on meaningless mutations in large, bloated seeds, DIAR removes the unnecessary bytes, streamlining the entire process.
In this work, we equipped AFL, a popular fuzzer, with DIAR and examined two critical Linux libraries -- Libxml's xmllint, a tool for parsing xml documents, and Binutil's readelf, an essential debugging and security analysis command-line tool used to display detailed information about ELF (Executable and Linkable Format). Our preliminary results show that AFL+DIAR does not only discover new paths more quickly but also achieves higher coverage overall. This work thus showcases how starting with lean and optimized seeds can lead to faster, more comprehensive fuzzing campaigns -- and DIAR helps you find such seeds.
- These are slides of the talk given at IEEE International Conference on Software Testing Verification and Validation Workshop, ICSTW 2022.
Threats to mobile devices are more prevalent and increasing in scope and complexity. Users of mobile devices desire to take full advantage of the features
available on those devices, but many of the features provide convenience and capability but sacrifice security. This best practices guide outlines steps the users can take to better protect personal devices and information.
Unlocking Productivity: Leveraging the Potential of Copilot in Microsoft 365, a presentation by Christoforos Vlachos, Senior Solutions Manager – Modern Workplace, Uni Systems
HCL Notes and Domino License Cost Reduction in the World of DLAUpanagenda
Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-and-domino-license-cost-reduction-in-the-world-of-dlau/
The introduction of DLAU and the CCB & CCX licensing model caused quite a stir in the HCL community. As a Notes and Domino customer, you may have faced challenges with unexpected user counts and license costs. You probably have questions on how this new licensing approach works and how to benefit from it. Most importantly, you likely have budget constraints and want to save money where possible. Don’t worry, we can help with all of this!
We’ll show you how to fix common misconfigurations that cause higher-than-expected user counts, and how to identify accounts which you can deactivate to save money. There are also frequent patterns that can cause unnecessary cost, like using a person document instead of a mail-in for shared mailboxes. We’ll provide examples and solutions for those as well. And naturally we’ll explain the new licensing model.
Join HCL Ambassador Marc Thomas in this webinar with a special guest appearance from Franz Walder. It will give you the tools and know-how to stay on top of what is going on with Domino licensing. You will be able lower your cost through an optimized configuration and keep it low going forward.
These topics will be covered
- Reducing license cost by finding and fixing misconfigurations and superfluous accounts
- How do CCB and CCX licenses really work?
- Understanding the DLAU tool and how to best utilize it
- Tips for common problem areas, like team mailboxes, functional/test users, etc
- Practical examples and best practices to implement right away
GraphRAG for Life Science to increase LLM accuracyTomaz Bratanic
GraphRAG for life science domain, where you retriever information from biomedical knowledge graphs using LLMs to increase the accuracy and performance of generated answers
TrustArc Webinar - 2024 Global Privacy SurveyTrustArc
How does your privacy program stack up against your peers? What challenges are privacy teams tackling and prioritizing in 2024?
In the fifth annual Global Privacy Benchmarks Survey, we asked over 1,800 global privacy professionals and business executives to share their perspectives on the current state of privacy inside and outside of their organizations. This year’s report focused on emerging areas of importance for privacy and compliance professionals, including considerations and implications of Artificial Intelligence (AI) technologies, building brand trust, and different approaches for achieving higher privacy competence scores.
See how organizational priorities and strategic approaches to data security and privacy are evolving around the globe.
This webinar will review:
- The top 10 privacy insights from the fifth annual Global Privacy Benchmarks Survey
- The top challenges for privacy leaders, practitioners, and organizations in 2024
- Key themes to consider in developing and maintaining your privacy program
1. 5 & 6 March 2015
Marina Bay Sands
Singapore
Rules of Engagement
on Cloud
2. Traditional Silo Setup
2
2 x P740 (PRD)
DB2:
• CV-RIB MY
2 x P720 (DR)
DB2:
• CV-RIB MY
1 x P740 (UAT/SIT)
DB2:
• CV-RIB MY
1 x Wintel (PRD)
MSSQL:
• Siebel 6
1 x Wintel (UAT/SIT)
MSSQL:
• Siebel 6
1 x P5 (PRD)
DB2:
• FPX/ePay
1 x P5 (UAT/SIT)
DB2:
• FPX/ePay
1 x RS6000 (PRD)
DB2:
• BScore
1 x RS6000 (UAT/SIT)
DB2:
• BScore
4 x T3-2 (PRD)
Oracle 11g SE:
• LOAD$ MG, CC & PL
2 x T3-2 (UAT/SIT)
Oracle 11g SE:
• LOAD$ MG, CC & PL
2 x P720 (PRD)
Oracle 11g SE:
• AML MY & VN
1 x P720 (UAT/SIT)
Oracle 11g SE:
• AML MY & VN
c1
c2
c4
c5
1 x P5 (PRD)
Oracle 10g:
• HL Wealth Planner
1 x P5 (UAT/SIT)
Oracle 10g:
• HL Wealth Planner
1 x Intel (PRD)
Oracle 9i on Linux:
• UTNS
1 x Intel (SIT/UAT)
Oracle 9i on Linux:
• UTNS
1 x Intel (PRD)
MySQL on CentOS:
• 8i Token MY
1 x Intel (UAT/SIT)
MySQL on CentOS:
• 8i Token MY
2 x M4000 (PRD)
DB2:
• CIB MY
1 x M4000 (UAT/SIT)
DB2:
• CIB MY
c3
p3
p4
c5
c6
w1
w2
w3
w4
p1
p2
p5
p6
p9
p10
p7
p8
Legend:
• Tech refresh completed
• Tech refresh WIP
• Tech refresh in planning
• New initiative completed
• New initiative WIP
• New initiative in planning
c1
w1
p1
1. Projects CER includes Capex/Opex for
hardware, software & professional services
for 3 or 5 years
2. Charge-out based on agreed percentage to
business unit 1, 2, 3 etc
1. Projects CER includes Capex/Opex for
hardware, software & professional services
for 3 or 5 years
2. Charge-out based on agreed percentage to
business unit 4, 5, 6 etc
1. Projects CER includes Capex/Opex for
hardware, software & professional services
for 3 or 5 years
2. Charge-out based on agreed percentage to
business unit 7, 8, 9 etc
1. Projects CER includes Capex/Opex for
hardware, software & professional services
for 3 or 5 years
2. Charge-out based on agreed percentage to
business unit 10, 11, 12 etc
No Challenges
1 High Capex/Opex for silo hardware
2 Long time-to-market (e-bidding,
procurement etc) for each project
3 Data center exhausted with
heterogeneous hardware, increase
facility cost
4 Decrease operation efficiency
3. Oracle
Infiniband
Switch
Oracle
Infiniband
Switch
1)No Single Point of Failure System
Infrastructure which have high
bandwidth & low latency interconnect
Fabric.
2) Database Infrastructure which can
deliver near 0 downtime & able to
scale out of the box.
3) Integrate both Hardware &
Software into single point of
Monitoring, Management, Metering &
Chargeback Tool.
40GbpsInfinibandFabric
Oracle SPARC T5-2 Servers
Oracle ZFS Clustered
Storage
4) DB Cloud tuning from DB down to
Storage through Infiniband Fabric.Clustered
5) Start migrating or consolidating
database’s into DB Cloud.
RIB MY CRELOAD$ Siebel RIB VN WLL G.Earth IB.Fraud
What did we do? We built a Private Cloud
4. Before, few database’s in silo-ed environment always led to performance issue especially
CPU bottleneck.
After consolidating 24 Database's into the DB Cloud, CPU Utilization shows less than 10% at
most of the time & peak is still below 20%.
The private cloud facilitates better utilisation
5. Scale according to the needs of
more computing power or more
storage capacity.
•Every Rack come with 2 Infiniband
Switches to Join the Cloud
Infiniband Fabric.
•12 x Compute Nodes of T5-2 for
Compute Node Expansion.
•2 x T5-2 Compute Nodes, 2 x
Backup Nodes, 1 x ZFS Storage for
Storage Node Expansion.
The private cloud facilitates elasticity and scalability
6. ■Shareable resources with others
while idle.
■Increase systems efficiency.
■10 CPU Processing Power only.
■Saving up to 60%.
The private cloud facilitated shared resources
8. ■Coexist different generation of
SPARC Servers in DB Private
Cloud Architecture
• Private Database Cloud
Startup.
The private cloud facilitated shared resources
9. • Long project timeline, slow to market.
• Cut down project timeline from 6 months to 1 month with DB
Private Cloud.
• 5x Faster to Market.
Design & Planning
Purchasing & Delivery
Implementation
Integration
User Acceptance Test
Go Live
Month 1 Month 2 Month 3 Month 4 Month 5 Month 6
Design & Planning
Purchasing & Delivery
Implementation
Integration
User Acceptance Test
Go Live
Month 6Month 1 Month 2 Month 3 Month 4 Month 5
Design & Planning
Purchasing & Delivery
Implementation
Provisioning
User Acceptance Test
Go Live
Month 1 Month 2 Month 3 Month 4 Month 5 Month 6
Provision
Database
Service within
Hours with
Private Cloud
The private cloud facilitated business agility
15. 1
5
Initial Projects (Jun’13)
1. Fuzion RIB/ESS MY
2. Fuzion RIB VN
3. Siebel 8
4. LOAD$-HP/MG/CC/PL
5. CRE
Additional Projects (Jan’14)
6. WLL
7. Green Earth
8. IB Fraud
Cloud Expansion Projects (Sep’14)
9. FPX/ePay
10.EAI
11.Falcon
12.Basel 2
13.Fuzion RIB KH
Additional Projects (Nov’14)
14.Fuzion RIB SG
15.8i Token SG
16.HLIB WinOps & CMS
New Projects (Jan’15)
17.8i Token MY
18.MVI Payment Gateway SG
1. Bulk purchase cloud hardware,
software & professional services for
3 or 5 years
2. Total cost distributed to respective
projects CER
3. Charge-out based on agreed
percentage to business unit 1, 2, 3 etc
1. Each project CER only includes
additional storage, software &
professional services for 3 or 5 years
2. Charge-out based on agreed
percentage to business unit 4, 5, 6 etc
1. Bulk purchase cloud expansion
hardware, software & professional
services for 3 or 5 years
2. Total cost distributed to respective
projects CER
3. Charge-out based on agreed
percentage to business unit 7, 8, 9 etc
1. Each project CER only includes
additional storage, software &
professional services for 3 or 5 years
2. Charge-out based on agreed
percentage to business unit 1, 2, 3 etc
No Challenges
1 High Capex/Opex for silo hardware
Lower Capex/Opex overall
2 Long time-to-market (e-bidding,
procurement etc) for each project
Shorten Lower Capex/Opex for
overall projects
3 Data center exhausted with
heterogeneous hardware, increase
facility cost
Standardize infrastructure
4 Decrease operation efficiency
Improve operation efficiency
5 Difficult to align bulk purchase
timeline for multiple projects
6 Un-fair business model as only
initial project fully bear the
Capex/Opex base infrastructure
7 Might not able to consolidate few
projects to contribute for future
expansion
Challenges – Chargeback
16. Proposed To-Be Cloud Chargeback Methodology 1
6
Initial Projects (Jun’13)
1. Fuzion RIB/ESS MY
2. Fuzion RIB VN
3. Siebel 8
4. LOAD$-HP/MG/CC/PL
5. CRE
Additional Projects (Jan’14)
6. WLL
7. Green Earth
8. IB Fraud
Cloud Expansion Projects (Sep’14)
9. FPX/ePay
10.EAI
11.Falcon
12.Basel 2
13.Fuzion RIB KH
Additional Projects (Nov’14)
14.Fuzion RIB SG
15.8i Token SG
16.HLIB WinOps & CMS
New Projects (Jan’15)
17.8i Token MY
18.MVI Payment Gateway SG
No Challenges
1 High Capex/Opex for silo hardware
Lower Capex/Opex overall
2 Long time-to-market (e-bidding,
procurement etc) for each project
Shorten Lower Capex/Opex for
overall projects
3 Data center exhausted with
heterogeneous hardware, increase
facility cost
Standardize infrastructure
4 Decrease operation efficiency
Improve operation efficiency
5 Difficult to align bulk purchase
timeline for multiple projects
Reserved capacity
6 Un-fair business model as only initial
project fully bear the Capex/Opex bas
infrastructure
Chargeback by allocation
7 Might not able to consolidate few
projects to contribute for future
expansion
Cloud Cost Center
Reserved capacities
under Cloud Cost Center
for upcoming projects or
organic business growth
Proposed chargeback process:
1. ITF to summarize the existing cloud total
investment for hardware & software
2. ITF/GITA to calculate the chargeback
block for hardware & software into a cloud
catalogue
3. GITA/GITI to provide the capacity
allocation & derive the chargeback cost
by project
4. ITF/Finance to manage the chargeback to
respective business units into Cloud Cost
Center
Chargeback for new projects:
1. Monthly chargeback of Capex/Opex for
cloud infrastructure to business unit
2. Chargeback the professional services to
project
3. IT to purchase new Cloud expansion in
advance and keep the capacities for next
business project using Cloud Cost Center
Chargeback for existing projects:
1. Monthly chargeback of Opex only for
cloud infrastructure to business unit for
next 2 years and chargeback Capex/Opex
thereafter
Chargeback (Now)
17. 1
7
Proposed chargeback process:
1. ITF to summarize the existing cloud total
investment for hardware & software
2. ITF/GITA to calculate the chargeback
block for hardware & software into a cloud
catalogue
3. GITA/GITI to provide the capacity
utilization & derive the chargeback cost
by project
4. ITF/Finance to manage the chargeback to
respective business units into Cloud Cost
Center
Chargeback for new projects:
1. Monthly chargeback of Capex/Opex for
cloud infrastructure to business unit
2. Chargeback the professional services to
project
3. IT to purchase new Cloud expansion in
advance and keep the capacities for next
business project using Cloud Cost Center
Chargeback for existing projects:
1. Monthly chargeback of Opex only for
cloud infrastructure to business unit for
another 2 years and chargeback
Capex/Opex thereafter
Initial Projects (Jun’13)
1. Fuzion RIB/ESS MY
2. Fuzion RIB VN
3. Siebel 8
4. LOAD$-HP/MG/CC/PL
5. CRE
Additional Projects (Jan’14)
6. WLL
7. Green Earth
8. IB Fraud
Cloud Expansion Projects (Sep’14)
9. FPX/ePay
10.EAI
11.Falcon
12.Basel 2
13.Fuzion RIB KH
Additional Projects (Nov’14)
14.Fuzion RIB SG
15.8i Token SG
16.HLIB WinOps & CMS
New Projects (Jan’15)
17.8i Token MY
18.MVI Payment Gateway SG
Reserved capacities
under Cloud Cost Center
for upcoming projects or
organic business growth
Upon maturity of
cloud monitoring
& metering
No Challenges
1 High Capex/Opex for silo hardware
Lower Capex/Opex overall
2 Long time-to-market (e-bidding,
procurement etc) for each project
Shorten Lower Capex/Opex for
overall projects
3 Data center exhausted with
heterogeneous hardware, increase
facility cost
Standardize infrastructure
4 Decrease operation efficiency
Improve operation efficiency
5 Difficult to align bulk purchase
timeline for multiple projects
Reserved capacity
6 Un-fair business model as only initial
project fully bear the Capex/Opex bas
infrastructure
Chargeback by utilization
7 Might not able to consolidate few
projects to contribute for future
expansion
Cloud Cost Center
Chargeback (Future)
18. 1
8
No Component Oracle DB
Cloud
PureApp WAS &
DB2 Cloud
Wintel Cloud
1 Server CPU/Core
2 Server Memory
3 Storage Controller -
4 Storage Drive/Disk
(include OS only)
(App data from 3PAR)
5 Software License -
(free for WAS & DB2)
(include OS only)
6 Professional Services
(by project)
-
(deploy from pattern)
(by project)
7 Operation Maintenance
Cost
Chargeback – working out the components
19. 1. For PJC & WHL, each Oracle DB Cloud is architected with 2 units of T5-2 server, 2 Infiniband switches & 2
storage controller.
2. T5-2 servers provide CPU & memory for database processing and the storage controllers host the
storage drive for OS & data. T5-2 servers & storage controllers are connected by Infiniband switches as
the network backbone.
3. The Oracle DB will be setup as RAC (active-active clustering) on both servers to achieve high availability.
Each Oracle DB software license allows activation of 2 cores.
4. Each project will contribute Oracle DB software license to a pool which can be shared across multiple
application based on different peak period to achieve maximum license savings. 1
9
Oracle Infiniband
Switch
Oracle Infiniband
Switch
40GbpsInfinibandFabric
Oracle SPARC T5-2 Servers
Oracle ZFS Clustered Storage
Clustered
Chargeback – components of a database private cloud
20. 20
1 unit Server Rack
2 units Infiniband Switches
12 units T5-2 Servers
For each T5-2 server
• 512 GB memory per server
• 64GB memory reserved for Control Domain
• Each DB VM block is 8GB memory
• Total 56 blocks available [448/8=56]
For each Server Rack
• Total 12 servers
• Total blocks: 12*56=672
Standard allocation is 0.5 core per 8GB memory
1 unit Storage Ra
2 units Infiniband Switches
4 units ZFS
Storage Controller
(36 Drives)
For each Storage Drive
• 1TB reserved for workarea
• 8TB usable for DB
• Each storage block is 100GB
• Total 80 blocks available [8000/100=80]
For each Storage Rack
• Total 4 storage controller (36 drives each)
• Total blocks: 4*36*80=11,520
Proposed hardware chargeback block:
1. Oracle DB Cloud consists of 2 key components: servers &
storage drives
2. Server will be chargeback by core & memory allocation
3. Storage drive will be chargeback by storage allocation
Chargeback – components of a database private cloud
21. Core License Core License Core License Core License
1 Fuzion ESS 2
2 Fuzion MY 2
3 Fuzion VN 2
4 LOAD$ HP 8
5 LOAD$ MG 4
6 LOAD$ CC/PL/RCB/SME 4
7 Siebel 4
8 CRE 4
9 WLL 4 2
10 Green Earth 4 2 2 1
11 IB Fraud 4 2 2 1
12 FPX 2 2
13 EAI (WMB & DataPower) 2 2
14 Falcon 2 -
15 Basel II 4 4
16 Fuzion KH 2 2
17 Fuzion SG 2 2
18 8i Token SG 2 2
19 WinOps & CMS 4 2 4 2 1 1
20 8i Token MY - - - - - - - -
21 MVI Payment Gateway SG - - - - - - - -
22 LOAD$ VN - - - - - - - -
23 Basel II VN - - - - - - - -
24 UTNS MY - - - - - - - -
25 CMS - - - - - - - -
62 26 26 10 10 5 10 5
SIT
10
2
2
PRD
4 2
4
DR
10
UAT
Total
10
8
2
2
No Projects
Proposed software chargeback block:
1. Chargeback 1 OPL for every 4 cores allocation (50% savings
from Oracle standard licensing model)
2. GITA/GITI to monitor the actual core utilization before next OPL
purchase for more core activation
For Oracle DB EE License (OPL)
• Each OPL allows activation of 2 core
• Initial projects started with 16 OPL
• Additional projects contribute some OPL
to the Cloud
• Cloud expansion projects contributed
additional 16 OPL
Server Core Allocation & Shared Pool
• Each project to contribute minimum
software license to enable more cores
into the shared pool
• Each application able to utilize unused
core within the shared pool
• Designed based on the concept of
maximizing cores sharing across multiple
applications by leveraging on the
differences during idle/peak hours
Chargeback – actual in action