These are a series of presentations and knowledge collected from the web to help knowledge sharing at the government of Puerto Rico, created with the hope of helping transform government culture by engaging key personnel in diverse areas of central government IT. We discussed design and development methodologies as well as implementation, network and server technologies that led to the successful launch of the most popular online service in PR.gov, in the hope that the knowledge is retained and used to prevent problems that have plagued digital services of the past.
How did Puerto Rico build the New Good standing Certificate Online Service? How did it scale to handle millions of visitors while having 0 licensing costs? This is the technical overview of the design, philosophy and implementation.
- Good standing certificate knowledge transfer presentation by Andrés Colón
Note on attribution: some content such as logos and designs were used from the web. Rights remain with their original authors. Thanks for sharing with the world.
What is Kafka & why is it Important? (UKOUG Tech17, Birmingham, UK - December...Lucas Jellema
Fast data arrives in real time and potentially high volume. Rapid processing, filtering and aggregation is required to ensure timely reaction and actual information in user interfaces. Doing so is a challenge, make this happen in a scalable and reliable fashion is even more interesting. This session introduces Apache Kafka as the scalable event bus that takes care of the events as they flow in and Kafka Streams and KSQL for the streaming analytics. Both Java and Node applications are demonstrated that interact with Kafka and leverage Server Sent Events and WebSocket channels to update the Web UI in real time. User activity performed by the audience in the Web UI is processed by the Kafka powered back end and results in live updates on all clients.
This presentation includes a demonstration of remote database synchronization through Twitter.
This beginner session is about advanced network services for private/hybrid clouds, particularly focusing on Load Balancing as a Service (LBaaS) and Global Service Load Balancing. In physical world of enterprise data centers, load balancers have evolved over time to do more than simple spraying of connections to a set of backend servers. We will delve into some of the core requirements including session persistence, SSL termination, geo-based routing, and rule based switching. We will present the evolution of OpenStack LBaaS APIs as they incorporated these requirements.
On the implementation side, we compare and contrast the most common architectures of LBaaS deployments: (i) process-based (reference architecture), (ii) appliance-based, and (iii) service-VM-based. We will focus on service-VM architecture's elastic scalability and high availability.
L4-L7 services for SDN and NVF by Youcef Laribibuildacloud
In this talk, we will discuss how L4-L7 devices can integrate in various SDN architectures, discuss benefits and some of the challenges that such integration represents. We will also talk about how SDN and NFV relate, and what are the different challenges to successfully deploy L4-L7 devices as Virtual Network Functions (VNFs) or provide such services to the NFV Infrastructure (VIM).
Bio
Youcef Laribi is a Principal Architect in the Delivery Networks BU at Citrix. He is responsible for driving the integration projects of the NetScaler ADC product with several Cloud, SDN and Automation environments including OpenStack, CloudStack, VMware NSX and Cisco ACI. He is also the Citrix representative on the OpenDaylight Technical Steering Committee. His background is mainly in Operating Systems and Distributed Systems, and he worked on several middleware technologies from DCE and CORBA in the early days, to J2EE and .NET to SOA and micro-services today. Youcef speaks 4 languages and holds a PhD and an MSc in Computer Science from the French INPG Institute in Grenoble, France.
ThaiNOG Day 2019: Internet Number Registry Services, the Next GenerationAPNIC
APNIC Director General Paul Wilson gives a presentation on Internet number registry services - the next generation at ThaiNOG 2019, held with BKNIX 2019 in Bangkok, Thailand from 7 to 8 May 2019.
CERN is expanding its computing infrastructure to support growing data and computing needs. It is adopting open source tools like Puppet for configuration management and OpenStack for cloud computing. CERN plans to deploy OpenStack into production in 2013 to manage over 15,000 hypervisors and 100,000 VMs across its data centers by 2015, supporting both traditional and cloud-based workflows. This will enable CERN to more efficiently manage resources and better support dynamic workloads and temporary spikes in demand.
Rolling the Root Zone DNSSEC Key Signing Key, by Edward Lewis.
A presentation given at APNIC 42's DNS and INR Security session on Monday, 3 October 2016.
What is Kafka & why is it Important? (UKOUG Tech17, Birmingham, UK - December...Lucas Jellema
Fast data arrives in real time and potentially high volume. Rapid processing, filtering and aggregation is required to ensure timely reaction and actual information in user interfaces. Doing so is a challenge, make this happen in a scalable and reliable fashion is even more interesting. This session introduces Apache Kafka as the scalable event bus that takes care of the events as they flow in and Kafka Streams and KSQL for the streaming analytics. Both Java and Node applications are demonstrated that interact with Kafka and leverage Server Sent Events and WebSocket channels to update the Web UI in real time. User activity performed by the audience in the Web UI is processed by the Kafka powered back end and results in live updates on all clients.
This presentation includes a demonstration of remote database synchronization through Twitter.
This beginner session is about advanced network services for private/hybrid clouds, particularly focusing on Load Balancing as a Service (LBaaS) and Global Service Load Balancing. In physical world of enterprise data centers, load balancers have evolved over time to do more than simple spraying of connections to a set of backend servers. We will delve into some of the core requirements including session persistence, SSL termination, geo-based routing, and rule based switching. We will present the evolution of OpenStack LBaaS APIs as they incorporated these requirements.
On the implementation side, we compare and contrast the most common architectures of LBaaS deployments: (i) process-based (reference architecture), (ii) appliance-based, and (iii) service-VM-based. We will focus on service-VM architecture's elastic scalability and high availability.
L4-L7 services for SDN and NVF by Youcef Laribibuildacloud
In this talk, we will discuss how L4-L7 devices can integrate in various SDN architectures, discuss benefits and some of the challenges that such integration represents. We will also talk about how SDN and NFV relate, and what are the different challenges to successfully deploy L4-L7 devices as Virtual Network Functions (VNFs) or provide such services to the NFV Infrastructure (VIM).
Bio
Youcef Laribi is a Principal Architect in the Delivery Networks BU at Citrix. He is responsible for driving the integration projects of the NetScaler ADC product with several Cloud, SDN and Automation environments including OpenStack, CloudStack, VMware NSX and Cisco ACI. He is also the Citrix representative on the OpenDaylight Technical Steering Committee. His background is mainly in Operating Systems and Distributed Systems, and he worked on several middleware technologies from DCE and CORBA in the early days, to J2EE and .NET to SOA and micro-services today. Youcef speaks 4 languages and holds a PhD and an MSc in Computer Science from the French INPG Institute in Grenoble, France.
ThaiNOG Day 2019: Internet Number Registry Services, the Next GenerationAPNIC
APNIC Director General Paul Wilson gives a presentation on Internet number registry services - the next generation at ThaiNOG 2019, held with BKNIX 2019 in Bangkok, Thailand from 7 to 8 May 2019.
CERN is expanding its computing infrastructure to support growing data and computing needs. It is adopting open source tools like Puppet for configuration management and OpenStack for cloud computing. CERN plans to deploy OpenStack into production in 2013 to manage over 15,000 hypervisors and 100,000 VMs across its data centers by 2015, supporting both traditional and cloud-based workflows. This will enable CERN to more efficiently manage resources and better support dynamic workloads and temporary spikes in demand.
Rolling the Root Zone DNSSEC Key Signing Key, by Edward Lewis.
A presentation given at APNIC 42's DNS and INR Security session on Monday, 3 October 2016.
HKIX IPv4 Address Renumbering from /23 to /21 - Experience SharingAPNIC
HKIX IPv4 Address Renumbering from /23 to /21 - Experience Sharing, by Che-Hoo Cheng.
A presentation given at the APNIC 40 Lightning Talks session on Tue, 8 Sep 2015.
This presentation provides an introduction to Apache Kafka and describes best practices for working with fast data streams in Kafka and MapR Streams.
The code examples used during this talk are available at github.com/iandow/design-patterns-for-fast-data.
Author:
Ian Downard
Presented at the Portland Java User Group on Tuesday, October 18 2016.
The document discusses OpenStack Quantum and OpenFlow/SDN. It provides an overview of Quantum, which allows network connectivity as a service in OpenStack. It describes how Quantum works by creating networks and ports and plugging interface devices. It also lists several Quantum plugins that can be used, such as plugins for Cisco, Linux bridge, NVP, and Open vSwitch. Finally, it introduces OpenFlow/SDN and provides basics on the OpenFlow protocol and how OpenFlow switching works.
IPv6 Deployment Case on a Korean Governmental WebsiteAPNIC
IPv6 Deployment Case on a Korean Governmental Website, by Jean Ryu.
A presentation given at APNIC 42's Network Operations session on Tuesday, 4 October 2016.
This document discusses achieving horizontal scaling for enterprise messaging using Fabric8. It provides an introduction to Fabric8 and enterprise messaging concepts. It then describes how Fabric8MQ, which is built on Vert.x, provides horizontal scaling and load balancing for ActiveMQ by implementing features like protocol conversion, Camel routing, API management, multiplexing, and destination sharding across Kubernetes pods and nodes. The document concludes with a demo of Fabric8MQ's capabilities.
Configuration Management Evolution at CERNGavin McCance
The document discusses CERN's evolution to using Puppet, Foreman, OpenStack and other open source tools for configuration management and infrastructure automation. It summarizes the key benefits of the new approach, including improved stability, scalability, and the ability to hire staff with relevant skills. Some ongoing challenges discussed are ensuring stability and scalability as the number of managed systems grows, integrating the new tools with existing site systems, and developing processes for collaborative development while preventing conflicts between teams.
A New Internet? Introduction to HTTP/2, QUIC and DOHAPNIC
This document discusses recent changes and improvements to core internet protocols like HTTP, DNS, and TCP. It introduces HTTP/2, which improves performance over HTTP/1.1 by allowing multiple requests per connection and header compression. It also discusses the development of QUIC, an experimental UDP-based protocol that aims to improve latency compared to TCP. Additionally, it covers DNS over HTTPS (DOH) which aims to increase privacy and censorship resistance by encrypting DNS queries over HTTPS. The document concludes that these protocols help accelerate the web by reducing round trips and blocking while securing more internet traffic.
Presented at the CloudStack Silicon Valley User Group in September 2015 at Nuage Networks. Discussed impact of containers, emerging software defined networking platforms, NFV, IPv6 and performance.
Kafka Summit SF 2017 - Kafka and the Polyglot Programmerconfluent
An Overview of the Kafka clients ecosystem. APIs – wire protocol clients – higher level clients (Streams) – REST Languages (with simple snippets – full examples in GitHub) – the most developed clients – Java and C/C++ – the librdkafka wrappers node-rdkafka, python, GO, C# – why use wrappers Shell scripted Kafka ( e.g. custom health checks) kafkacat Platform gotchas (e.g. SASL on Win32)
Presented at Kafka Summit SF 2017 by Edoardo Comar and Andrew Schofield, IBM
Messaging For the Cloud and MicroservicesRob Davies
Utilising messaging in cloud deployments isn't straightforward, particularly if you want to take advantage of auto scaling. This talk covers the general problems of scaling for cloud deployments, and messaging for faster inter-service communication for Microservices
Protecting your data at rest with Apache Kafka by Confluent and Vormetricconfluent
This document discusses securing Apache Kafka deployments with Vormetric and Confluent Platform. It begins with an introduction to Apache Kafka and Confluent Platform. It then provides an overview of Vormetric's policy-driven security solution and how it can be used to encrypt Kafka data at rest. The document outlines the typical Confluent Platform deployment architecture and various security considerations, such as authentication, authorization, and data encryption. Finally, it provides steps for implementing secure deployments using SSL, Kerberos, and Vormetric encryption policies.
APNIC Training Manager Tashi Phuntsho explains why securing Internet routing is so important at the first Mongolian Network Operators Group meeting in Ulaanbaatar, Mongolia from 16 to 20 September 2019.
Gli apparati preposti all’accelerazione del traffico geografico sono spesso sottovalutati a fronte di linee ad alta capacità. Nella presentazione vedremo che la realtà è ben diversa: gli ottimizzatori di banda hanno una loro utilità ancora oggi e sono in grado di risolvere problemi che altrimenti sarebbero di difficile soluzione.
Dopo un’analisi dei punti di forza di questa tecnologia, verrà analizzato un case study reale, evidenziando gli immediati vantaggi che questi apparati possono dare.
The Next Generation Internet Number Registry ServicesMyNOG
This document provides an overview of registry services, including the Registration Data Access Protocol (RDAP) and the Resource Public Key Infrastructure (RPKI). RDAP is designed to replace the aging WHOIS protocol by providing structured query and response formats to enable automation. RDAP also supports access control, internationalization, redirection and extensibility. RPKI is a PKI framework that adds Internet number resource information to certificates to cryptographically validate resource ownership and authorization of routing announcements. It enables applications like route origin validation to secure the routing system. The document discusses how RDAP and RPKI work and provide benefits like improved security, automation and verification of registry data.
Design and Implementation of Incremental Cooperative Rebalancingconfluent
Watch this talk here: https://www.confluent.io/online-talks/design-and-implementation-of-incremental-cooperative-rebalancing-on-demand
Since its initial release, the Kafka group membership protocol has offered Connect, Streams and Consumer applications an ingenious and robust way to balance resources among distributed processes. The process of rebalancing, as it’s widely known, allows Kafka APIs to define an embedded protocol for load balancing within the group membership protocol itself.
Until now, rebalancing has been working under the simple assumption that every time a new group generation is created, the members join after first releasing all of their resources, getting a whole new load assignment by the time the new group is formed. This allows Kafka APIs to provide task fault-tolerance and elasticity on top of the group membership protocol.
However, due to its side-effects on multi-tenancy and scalability this simple approach in rebalancing, also known as stop-the-world effect, is limiting larger scale deployments. Because of stop-the-world, application tasks get interrupted only for most of them to receive the same resources after rebalancing. In this technical deep dive, we’ll discuss the proposition of Incremental Cooperative Rebalancing as a way to alleviate stop-the-world and optimize rebalancing in Kafka APIs.
This talk will cover:
-The internals of Incremental Cooperative Rebalancing
-Uses cases that benefit from Incremental Cooperative Rebalancing
-Implementation in Kafka Connect
-Performance results in Kafka Connect clusters
Apache NiFi: latest developments for flow management at scaleAbdelkrim Hadjidj
The document discusses Apache NiFi, an open source dataflow management platform. It provides an overview of NiFi's capabilities including over 225 processors for common data access, transformation, and management tasks. The presentation demonstrates NiFi and its web-based user interface, zero-master clustering architecture, and extensibility via custom processors and controllers. New features discussed include component versioning, change data capture from MySQL, and a record-based processing mechanism for improved data handling.
MiNiFi is a recently started sub-project of Apache NiFi that is a complementary data collection approach which supplements the core tenets of NiFi in dataflow management, focusing on the collection of data at the source of its creation. Simply, MiNiFi agents take the guiding principles of NiFi and pushes them to the edge in a purpose built design and deploy manner. This talk will focus on MiNiFi's features, go over recent developments and prospective plans, and give a live demo of MiNiFi.
The config.yml is available here: https://gist.github.com/JPercivall/f337b8abdc9019cab5ff06cb7f6ff09a
Building high performance microservices in finance with Apache ThriftRX-M Enterprises LLC
Apache Roadshow Chicago Talk on May 14, 2019
In this talk we’ll look at the ways Apache Thrift can solve performance problems commonly facing next generation applications deployed in performance sensitive capital markets and banking environments. The talk will include practical examples illustrating the construction, performance and resource utilization benefits of Apache Thrift. Apache Thrift is a high-performance cross platform RPC and serialization framework designed to make it possible for organizations to specify interfaces and application wide data structures suitable for serialization and transport over a wide variety of schemes. Due to the unparalleled set of languages supported by Apache Thrift, these interfaces and structs have similar interoperability to REST type services with an order of magnitude improvement in performance. Apache Thrift services are also a perfect fit for container technology, using considerably fewer resources than traditional application server style deployments. Decomposing applications into microservices, packaging them into containers and orchestrating them on systems like Kubernetes can bring great value to an organization; however, it can also take a very fast monolithic application and turn it into a high latency web of slow, resource hungry services. Apache Thrift is a perfect solution to the performance and resource ills of many microservice based endeavors.
Adding Real-time Features to PHP ApplicationsRonny López
It's possible to introduce real-time features to PHP applications without deep modifications of the current codebase.
Using WAMP you can build distributed systems out of application components which are loosely coupled and communicate in (soft) real-time.
There is no need to learn a whole new language, with the implications it has.
It also opens the door to write reactive, event-based, distributed architectures and to achieve easier scalability by distributing messages to multiple systems.
HKIX IPv4 Address Renumbering from /23 to /21 - Experience SharingAPNIC
HKIX IPv4 Address Renumbering from /23 to /21 - Experience Sharing, by Che-Hoo Cheng.
A presentation given at the APNIC 40 Lightning Talks session on Tue, 8 Sep 2015.
This presentation provides an introduction to Apache Kafka and describes best practices for working with fast data streams in Kafka and MapR Streams.
The code examples used during this talk are available at github.com/iandow/design-patterns-for-fast-data.
Author:
Ian Downard
Presented at the Portland Java User Group on Tuesday, October 18 2016.
The document discusses OpenStack Quantum and OpenFlow/SDN. It provides an overview of Quantum, which allows network connectivity as a service in OpenStack. It describes how Quantum works by creating networks and ports and plugging interface devices. It also lists several Quantum plugins that can be used, such as plugins for Cisco, Linux bridge, NVP, and Open vSwitch. Finally, it introduces OpenFlow/SDN and provides basics on the OpenFlow protocol and how OpenFlow switching works.
IPv6 Deployment Case on a Korean Governmental WebsiteAPNIC
IPv6 Deployment Case on a Korean Governmental Website, by Jean Ryu.
A presentation given at APNIC 42's Network Operations session on Tuesday, 4 October 2016.
This document discusses achieving horizontal scaling for enterprise messaging using Fabric8. It provides an introduction to Fabric8 and enterprise messaging concepts. It then describes how Fabric8MQ, which is built on Vert.x, provides horizontal scaling and load balancing for ActiveMQ by implementing features like protocol conversion, Camel routing, API management, multiplexing, and destination sharding across Kubernetes pods and nodes. The document concludes with a demo of Fabric8MQ's capabilities.
Configuration Management Evolution at CERNGavin McCance
The document discusses CERN's evolution to using Puppet, Foreman, OpenStack and other open source tools for configuration management and infrastructure automation. It summarizes the key benefits of the new approach, including improved stability, scalability, and the ability to hire staff with relevant skills. Some ongoing challenges discussed are ensuring stability and scalability as the number of managed systems grows, integrating the new tools with existing site systems, and developing processes for collaborative development while preventing conflicts between teams.
A New Internet? Introduction to HTTP/2, QUIC and DOHAPNIC
This document discusses recent changes and improvements to core internet protocols like HTTP, DNS, and TCP. It introduces HTTP/2, which improves performance over HTTP/1.1 by allowing multiple requests per connection and header compression. It also discusses the development of QUIC, an experimental UDP-based protocol that aims to improve latency compared to TCP. Additionally, it covers DNS over HTTPS (DOH) which aims to increase privacy and censorship resistance by encrypting DNS queries over HTTPS. The document concludes that these protocols help accelerate the web by reducing round trips and blocking while securing more internet traffic.
Presented at the CloudStack Silicon Valley User Group in September 2015 at Nuage Networks. Discussed impact of containers, emerging software defined networking platforms, NFV, IPv6 and performance.
Kafka Summit SF 2017 - Kafka and the Polyglot Programmerconfluent
An Overview of the Kafka clients ecosystem. APIs – wire protocol clients – higher level clients (Streams) – REST Languages (with simple snippets – full examples in GitHub) – the most developed clients – Java and C/C++ – the librdkafka wrappers node-rdkafka, python, GO, C# – why use wrappers Shell scripted Kafka ( e.g. custom health checks) kafkacat Platform gotchas (e.g. SASL on Win32)
Presented at Kafka Summit SF 2017 by Edoardo Comar and Andrew Schofield, IBM
Messaging For the Cloud and MicroservicesRob Davies
Utilising messaging in cloud deployments isn't straightforward, particularly if you want to take advantage of auto scaling. This talk covers the general problems of scaling for cloud deployments, and messaging for faster inter-service communication for Microservices
Protecting your data at rest with Apache Kafka by Confluent and Vormetricconfluent
This document discusses securing Apache Kafka deployments with Vormetric and Confluent Platform. It begins with an introduction to Apache Kafka and Confluent Platform. It then provides an overview of Vormetric's policy-driven security solution and how it can be used to encrypt Kafka data at rest. The document outlines the typical Confluent Platform deployment architecture and various security considerations, such as authentication, authorization, and data encryption. Finally, it provides steps for implementing secure deployments using SSL, Kerberos, and Vormetric encryption policies.
APNIC Training Manager Tashi Phuntsho explains why securing Internet routing is so important at the first Mongolian Network Operators Group meeting in Ulaanbaatar, Mongolia from 16 to 20 September 2019.
Gli apparati preposti all’accelerazione del traffico geografico sono spesso sottovalutati a fronte di linee ad alta capacità. Nella presentazione vedremo che la realtà è ben diversa: gli ottimizzatori di banda hanno una loro utilità ancora oggi e sono in grado di risolvere problemi che altrimenti sarebbero di difficile soluzione.
Dopo un’analisi dei punti di forza di questa tecnologia, verrà analizzato un case study reale, evidenziando gli immediati vantaggi che questi apparati possono dare.
The Next Generation Internet Number Registry ServicesMyNOG
This document provides an overview of registry services, including the Registration Data Access Protocol (RDAP) and the Resource Public Key Infrastructure (RPKI). RDAP is designed to replace the aging WHOIS protocol by providing structured query and response formats to enable automation. RDAP also supports access control, internationalization, redirection and extensibility. RPKI is a PKI framework that adds Internet number resource information to certificates to cryptographically validate resource ownership and authorization of routing announcements. It enables applications like route origin validation to secure the routing system. The document discusses how RDAP and RPKI work and provide benefits like improved security, automation and verification of registry data.
Design and Implementation of Incremental Cooperative Rebalancingconfluent
Watch this talk here: https://www.confluent.io/online-talks/design-and-implementation-of-incremental-cooperative-rebalancing-on-demand
Since its initial release, the Kafka group membership protocol has offered Connect, Streams and Consumer applications an ingenious and robust way to balance resources among distributed processes. The process of rebalancing, as it’s widely known, allows Kafka APIs to define an embedded protocol for load balancing within the group membership protocol itself.
Until now, rebalancing has been working under the simple assumption that every time a new group generation is created, the members join after first releasing all of their resources, getting a whole new load assignment by the time the new group is formed. This allows Kafka APIs to provide task fault-tolerance and elasticity on top of the group membership protocol.
However, due to its side-effects on multi-tenancy and scalability this simple approach in rebalancing, also known as stop-the-world effect, is limiting larger scale deployments. Because of stop-the-world, application tasks get interrupted only for most of them to receive the same resources after rebalancing. In this technical deep dive, we’ll discuss the proposition of Incremental Cooperative Rebalancing as a way to alleviate stop-the-world and optimize rebalancing in Kafka APIs.
This talk will cover:
-The internals of Incremental Cooperative Rebalancing
-Uses cases that benefit from Incremental Cooperative Rebalancing
-Implementation in Kafka Connect
-Performance results in Kafka Connect clusters
Apache NiFi: latest developments for flow management at scaleAbdelkrim Hadjidj
The document discusses Apache NiFi, an open source dataflow management platform. It provides an overview of NiFi's capabilities including over 225 processors for common data access, transformation, and management tasks. The presentation demonstrates NiFi and its web-based user interface, zero-master clustering architecture, and extensibility via custom processors and controllers. New features discussed include component versioning, change data capture from MySQL, and a record-based processing mechanism for improved data handling.
MiNiFi is a recently started sub-project of Apache NiFi that is a complementary data collection approach which supplements the core tenets of NiFi in dataflow management, focusing on the collection of data at the source of its creation. Simply, MiNiFi agents take the guiding principles of NiFi and pushes them to the edge in a purpose built design and deploy manner. This talk will focus on MiNiFi's features, go over recent developments and prospective plans, and give a live demo of MiNiFi.
The config.yml is available here: https://gist.github.com/JPercivall/f337b8abdc9019cab5ff06cb7f6ff09a
Building high performance microservices in finance with Apache ThriftRX-M Enterprises LLC
Apache Roadshow Chicago Talk on May 14, 2019
In this talk we’ll look at the ways Apache Thrift can solve performance problems commonly facing next generation applications deployed in performance sensitive capital markets and banking environments. The talk will include practical examples illustrating the construction, performance and resource utilization benefits of Apache Thrift. Apache Thrift is a high-performance cross platform RPC and serialization framework designed to make it possible for organizations to specify interfaces and application wide data structures suitable for serialization and transport over a wide variety of schemes. Due to the unparalleled set of languages supported by Apache Thrift, these interfaces and structs have similar interoperability to REST type services with an order of magnitude improvement in performance. Apache Thrift services are also a perfect fit for container technology, using considerably fewer resources than traditional application server style deployments. Decomposing applications into microservices, packaging them into containers and orchestrating them on systems like Kubernetes can bring great value to an organization; however, it can also take a very fast monolithic application and turn it into a high latency web of slow, resource hungry services. Apache Thrift is a perfect solution to the performance and resource ills of many microservice based endeavors.
Adding Real-time Features to PHP ApplicationsRonny López
It's possible to introduce real-time features to PHP applications without deep modifications of the current codebase.
Using WAMP you can build distributed systems out of application components which are loosely coupled and communicate in (soft) real-time.
There is no need to learn a whole new language, with the implications it has.
It also opens the door to write reactive, event-based, distributed architectures and to achieve easier scalability by distributing messages to multiple systems.
Cloud Services Powered by IBM SoftLayer and NetflixOSSaspyker
This presentation covers our work starting with Acme Air web scale and transitioning to operational lessons learned in HA, automatic recovery, continuous delivery, and operational visibility. It shows the port of the Netflix OSS cloud platform to IBM's cloud - SoftLayer and use of RightScale.
Moving to microservices – a technology and organisation transformational journeyBoyan Dimitrov
Moving to microservices was a transformational journey for the company as their system grew rapidly. They started with a monolithic architecture which became difficult to maintain and scale. This led them to redesign their system using microservices built with Go running on AWS (Amazon Web Services). They developed core platform capabilities to support automated provisioning, routing, discovery, monitoring and more. This allowed them to deploy new services rapidly and operate their distributed system more efficiently. The transition required changes to both their technology and organizational culture.
REST is a lightweight architecture for building client-server applications. It uses standard HTTP methods to allow requesting and modifying resource state representations. While SOAP and web services will continue to be used, REST is better suited for mobile and web applications. Organizations are realizing they cannot replace existing technologies and instead focus on integrating technologies to leverage their respective strengths. Exposing existing systems through a REST API gateway allows for coexistence while providing a clean interface. Security, caching, throttling and monitoring are important when managing REST APIs at an enterprise scale.
Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with A...confluent
Microservices, events, containers, and orchestrators are dominating our vernacular today. As operations teams adapt to support these technologies in production, cloud-native platforms like Pivotal Cloud Foundry and Kubernetes have quickly risen to serve as force multipliers of automation, productivity and value.
Apache Kafka® is providing developers a critically important component as they build and modernize applications to cloud-native architecture.
This talk will explore:
• Why cloud-native platforms and why run Apache Kafka on Kubernetes?
• What kind of workloads are best suited for this combination?
• Tips to determine the path forward for legacy monoliths in your application portfolio
• Demo: Running Apache Kafka as a Streaming Platform on Kubernetes
HAProxy is a free, open source load balancer and reverse proxy that is fast, reliable and offers high availability. It can be used to load balance HTTP and TCP traffic, provide failover, and maximize throughput and availability. Many large companies use HAProxy to load balance their websites and applications. It offers out of band health checks, hot reconfiguration, and works on Linux, BSD, Solaris and AIX. When implementing HAProxy, you configure backends, frontends and load balancing algorithms in its configuration file and can monitor it with tools like Nagios.
HAProxy is a free, open-source load balancer and reverse proxy that is fast, reliable and offers high availability. It can be used to load balance HTTP and TCP-based applications. Some key features include out-of-band health checks, hot reconfiguration, and multiple load balancing algorithms. Many large companies use HAProxy to load balance their websites and applications. It runs on Linux, BSD, and Solaris and can be used to load balance applications across servers on-premises or in the cloud.
This document discusses when a service mesh may be needed and provides an overview of the current service mesh landscape. It begins with why microservices are adopted and the challenges of operating distributed applications. It then describes a maturity journey where a service mesh is not initially needed but may become useful for applications that become more complex, distributed, and interdependent. The document outlines some current major service mesh implementations and notes that the technology is still new and changing rapidly. It recommends investigating service meshes through proof of concepts but cautions that production usage requires significant resources. It profiles F5 Aspen Mesh and NGINX solutions for service meshes and microservices.
HPC control systems are evolving into the future. This presentation looks at where this evolution may lead, and describes how the control system of the future might be constructed.
Netflix uses a microservices architecture and immutable infrastructure approach. It loads content across multiple AWS regions for high availability and scales services dynamically. Netflix employs techniques like caching, adaptive streaming, and content delivery networks to optimize the user experience of streaming video globally to over 140 million subscribers.
Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with K...confluent
Microservices, events, containers, and orchestrators are dominating our vernacular today. As operations teams adapt to support these technologies in production, cloud-native platforms like Cloud Foundry and Kubernetes have quickly risen to serve as force multipliers of automation, productivity and value. Kafka is providing developers a critically important component as they build and modernize applications to cloud-native architecture. This talk will explore:
• Why cloud-native platforms and why run Kafka on Kubernetes?
• What kind of workloads are best suited for this combination?
• Tips to determine the path forward for legacy monoliths in your application portfolio
• Running Kafka as a Streaming Platform on Container Orchestration
The 12 Factor App methodology provides guidelines for building software-as-a-service applications in the cloud. It advocates for codebases that are tracked in revision control, explicit declaration of dependencies, separation of configuration from code, treating backing services as attached resources, and strict separation between build, release, and run stages. The methodology also includes guidelines for processes, port binding, concurrency, disposability, keeping development and production environments similar, and treating logs as event streams. Following the 12 factors can help applications maximize portability, be more robust and agile, and scale smoothly by avoiding reliance on implicit tools or behaviors.
This document discusses a partnership between FileCatalyst and Square Box Systems (CatDV). FileCatalyst provides accelerated file transfer solutions, while CatDV provides media asset management software. The document outlines FileCatalyst's technology for improving file transfer speeds compared to standard TCP/IP protocols. It also describes how FileCatalyst integrates with CatDV to allow automated ingest of remote media assets into the CatDV system and sharing of assets out to remote locations at high speeds.
In this session we will talk about the history of NGINX and NGINX Plus and the role it has played in the development of the internet.
We will discuss some of the most recent changes and additions to the popular software project and touch base on some planned feature enhancements coming in the next months
Weaveworks discusses Microservices and best practices
Visit Weave Cloud: https://www.weave.works/product/cloud/
For more free talks, join our Weave Online User Group: https://www.meetup.com/Weave-User-Group/
- Debugging microservices presents key challenges due to their distributed nature across multiple processes. Observability techniques like logging, monitoring and tracing are important to gain visibility.
- Telepresence allows debugging services locally by intercepting requests to emulate the environment without needing to deploy to the cluster. Telepresence v1 swaps the deployment entirely for local debugging, while v2 intercepts specific ports/requests.
- Choosing between Telepresence v1 and v2 depends on use cases - v1 is better for consuming messages while v2 is better for intercepting specific ports/requests without a full deployment swap. Both provide useful debugging capabilities for microservices running in Kubernetes.
Debugging Microservices - key challenges and techniques - Microservices Odesa...Lohika_Odessa_TechTalks
Microservice architecture is widespread our days. It comes with a lot of benefits and challenges to solve. Main goal of this talk is to go through troubleshooting and debugging in the distributed micro-service world. Topic would cover:
main aspects of the logging,
monitoring,
distributed tracing,
debugging services on the cluster.
About speaker:
Andrеy Kolodnitskiy is Staff engineer in the Lohika and his primary focus is around distributed systems, microservices and JVM based languages.
Majority of time engineers spend debugging and fixing the issues. This talk will be dedicated to best practicies and tools Andrеys team uses on its project which do help to find issues more efficiently.
Similar to Building Modern Digital Services on Scalable Private Government Infrastructures using Open Source Technologies for Public Service (20)
Dandelion Hashtable: beyond billion requests per second on a commodity serverAntonios Katsarakis
This slide deck presents DLHT, a concurrent in-memory hashtable. Despite efforts to optimize hashtables, that go as far as sacrificing core functionality, state-of-the-art designs still incur multiple memory accesses per request and block request processing in three cases. First, most hashtables block while waiting for data to be retrieved from memory. Second, open-addressing designs, which represent the current state-of-the-art, either cannot free index slots on deletes or must block all requests to do so. Third, index resizes block every request until all objects are copied to the new index. Defying folklore wisdom, DLHT forgoes open-addressing and adopts a fully-featured and memory-aware closed-addressing design based on bounded cache-line-chaining. This design offers lock-free index operations and deletes that free slots instantly, (2) completes most requests with a single memory access, (3) utilizes software prefetching to hide memory latencies, and (4) employs a novel non-blocking and parallel resizing. In a commodity server and a memory-resident workload, DLHT surpasses 1.6B requests per second and provides 3.5x (12x) the throughput of the state-of-the-art closed-addressing (open-addressing) resizable hashtable on Gets (Deletes).
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-EfficiencyScyllaDB
Freshworks creates AI-boosted business software that helps employees work more efficiently and effectively. Managing data across multiple RDBMS and NoSQL databases was already a challenge at their current scale. To prepare for 10X growth, they knew it was time to rethink their database strategy. Learn how they architected a solution that would simplify scaling while keeping costs under control.
Main news related to the CCS TSI 2023 (2023/1695)Jakub Marek
An English 🇬🇧 translation of a presentation to the speech I gave about the main changes brought by CCS TSI 2023 at the biggest Czech conference on Communications and signalling systems on Railways, which was held in Clarion Hotel Olomouc from 7th to 9th November 2023 (konferenceszt.cz). Attended by around 500 participants and 200 on-line followers.
The original Czech 🇨🇿 version of the presentation can be found here: https://www.slideshare.net/slideshow/hlavni-novinky-souvisejici-s-ccs-tsi-2023-2023-1695/269688092 .
The videorecording (in Czech) from the presentation is available here: https://youtu.be/WzjJWm4IyPk?si=SImb06tuXGb30BEH .
HCL Notes and Domino License Cost Reduction in the World of DLAUpanagenda
Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-and-domino-license-cost-reduction-in-the-world-of-dlau/
The introduction of DLAU and the CCB & CCX licensing model caused quite a stir in the HCL community. As a Notes and Domino customer, you may have faced challenges with unexpected user counts and license costs. You probably have questions on how this new licensing approach works and how to benefit from it. Most importantly, you likely have budget constraints and want to save money where possible. Don’t worry, we can help with all of this!
We’ll show you how to fix common misconfigurations that cause higher-than-expected user counts, and how to identify accounts which you can deactivate to save money. There are also frequent patterns that can cause unnecessary cost, like using a person document instead of a mail-in for shared mailboxes. We’ll provide examples and solutions for those as well. And naturally we’ll explain the new licensing model.
Join HCL Ambassador Marc Thomas in this webinar with a special guest appearance from Franz Walder. It will give you the tools and know-how to stay on top of what is going on with Domino licensing. You will be able lower your cost through an optimized configuration and keep it low going forward.
These topics will be covered
- Reducing license cost by finding and fixing misconfigurations and superfluous accounts
- How do CCB and CCX licenses really work?
- Understanding the DLAU tool and how to best utilize it
- Tips for common problem areas, like team mailboxes, functional/test users, etc
- Practical examples and best practices to implement right away
Have you ever been confused by the myriad of choices offered by AWS for hosting a website or an API?
Lambda, Elastic Beanstalk, Lightsail, Amplify, S3 (and more!) can each host websites + APIs. But which one should we choose?
Which one is cheapest? Which one is fastest? Which one will scale to meet our needs?
Join me in this session as we dive into each AWS hosting service to determine which one is best for your scenario and explain why!
GraphRAG for Life Science to increase LLM accuracyTomaz Bratanic
GraphRAG for life science domain, where you retriever information from biomedical knowledge graphs using LLMs to increase the accuracy and performance of generated answers
Ivanti’s Patch Tuesday breakdown goes beyond patching your applications and brings you the intelligence and guidance needed to prioritize where to focus your attention first. Catch early analysis on our Ivanti blog, then join industry expert Chris Goettl for the Patch Tuesday Webinar Event. There we’ll do a deep dive into each of the bulletins and give guidance on the risks associated with the newly-identified vulnerabilities.
Monitoring and Managing Anomaly Detection on OpenShift.pdfTosin Akinosho
Monitoring and Managing Anomaly Detection on OpenShift
Overview
Dive into the world of anomaly detection on edge devices with our comprehensive hands-on tutorial. This SlideShare presentation will guide you through the entire process, from data collection and model training to edge deployment and real-time monitoring. Perfect for those looking to implement robust anomaly detection systems on resource-constrained IoT/edge devices.
Key Topics Covered
1. Introduction to Anomaly Detection
- Understand the fundamentals of anomaly detection and its importance in identifying unusual behavior or failures in systems.
2. Understanding Edge (IoT)
- Learn about edge computing and IoT, and how they enable real-time data processing and decision-making at the source.
3. What is ArgoCD?
- Discover ArgoCD, a declarative, GitOps continuous delivery tool for Kubernetes, and its role in deploying applications on edge devices.
4. Deployment Using ArgoCD for Edge Devices
- Step-by-step guide on deploying anomaly detection models on edge devices using ArgoCD.
5. Introduction to Apache Kafka and S3
- Explore Apache Kafka for real-time data streaming and Amazon S3 for scalable storage solutions.
6. Viewing Kafka Messages in the Data Lake
- Learn how to view and analyze Kafka messages stored in a data lake for better insights.
7. What is Prometheus?
- Get to know Prometheus, an open-source monitoring and alerting toolkit, and its application in monitoring edge devices.
8. Monitoring Application Metrics with Prometheus
- Detailed instructions on setting up Prometheus to monitor the performance and health of your anomaly detection system.
9. What is Camel K?
- Introduction to Camel K, a lightweight integration framework built on Apache Camel, designed for Kubernetes.
10. Configuring Camel K Integrations for Data Pipelines
- Learn how to configure Camel K for seamless data pipeline integrations in your anomaly detection workflow.
11. What is a Jupyter Notebook?
- Overview of Jupyter Notebooks, an open-source web application for creating and sharing documents with live code, equations, visualizations, and narrative text.
12. Jupyter Notebooks with Code Examples
- Hands-on examples and code snippets in Jupyter Notebooks to help you implement and test anomaly detection models.
Introduction of Cybersecurity with OSS at Code Europe 2024Hiroshi SHIBATA
I develop the Ruby programming language, RubyGems, and Bundler, which are package managers for Ruby. Today, I will introduce how to enhance the security of your application using open-source software (OSS) examples from Ruby and RubyGems.
The first topic is CVE (Common Vulnerabilities and Exposures). I have published CVEs many times. But what exactly is a CVE? I'll provide a basic understanding of CVEs and explain how to detect and handle vulnerabilities in OSS.
Next, let's discuss package managers. Package managers play a critical role in the OSS ecosystem. I'll explain how to manage library dependencies in your application.
I'll share insights into how the Ruby and RubyGems core team works to keep our ecosystem safe. By the end of this talk, you'll have a better understanding of how to safeguard your code.
This presentation provides valuable insights into effective cost-saving techniques on AWS. Learn how to optimize your AWS resources by rightsizing, increasing elasticity, picking the right storage class, and choosing the best pricing model. Additionally, discover essential governance mechanisms to ensure continuous cost efficiency. Whether you are new to AWS or an experienced user, this presentation provides clear and practical tips to help you reduce your cloud costs and get the most out of your budget.
Best 20 SEO Techniques To Improve Website Visibility In SERPPixlogix Infotech
Boost your website's visibility with proven SEO techniques! Our latest blog dives into essential strategies to enhance your online presence, increase traffic, and rank higher on search engines. From keyword optimization to quality content creation, learn how to make your site stand out in the crowded digital landscape. Discover actionable tips and expert insights to elevate your SEO game.
WeTestAthens: Postman's AI & Automation Techniques
Building Modern Digital Services on Scalable Private Government Infrastructures using Open Source Technologies for Public Service
1. Certificate of Good Standing
Knowledge Transfer
Presentation
New PR.gov Infrastructure
Good Standing Certificate Service
By Andrés Colón Pérez
2. About me
• Architect for the Good Standing Certificate
System
• Office of Management and Budget Employee
previously assigned to the Chief Information
Officer Office
• Worked with a multi-agency team to identify the
problems and solutions
• Designed and developed the PR.gov
infrastructure, networks, applications and
deployment automation
3. About this Presentation
• An attempt to familiarize government
personnel with:
– building modern digital services
– using free open source software to reduce
licensing costs
– designing networks, building and deploying
applications
– designing scalable government systems
4. What you’ll learn over the weeks:
• Project Overview (in spanish)
• Overview of concepts and technologies used in design, app
development, and infrastructure, including:
– HaProxy
– Keepalived
– Bind9
– Redis
– Ruby
– Resque
– Padrino, Grape, Puma, EventMachine and other gems
– HTML5 and Twitter’s Bootstrap
– Git
– Ansible
– PhoneGap
6. What’s the Plan?
• Overview of Knowledge Transfer Plan
• Knowledge Transfer Talk Topics: Week 1
7. Before we get started:
• Topics serve as an introduction
• Feel free to ask questions
• You’re expected to read up more on your own
• Five minute breaks
9. Knowledge Transfer Plan
• Weekly Presentations
– Save your calendar: Fridays 2:30pm
– June to September
– Starts June 19, end September 11 2015
• Weekly Q&A, Thursdays 9:30am – 10:30am
• Multiple Topics
10. Week 1 Topics
• Overview of concepts and technologies in:
– Development
– PR.gov Infrastructure / Network Design
11. Development Topics
• Software Architecture Style (Micro-Services)
• Application Programming Interfaces
• Redundancy and Scalability
• Asynchronous vs Synchronous Services
• Open Data (data.pr.gov)
• Version Control
• Open Source (Github / Application Stack)
• Software Development Method (DevOps)
20. API’s for Short
• Set of Protocol, Routines & tools for building
software applications
• Goal: interoperability
• A Web Service is a type of API
• Most popular Web Services:
– Simple Object Access Protocol (SOAP)
– Representational State Transfer (REST)
• SOAP often referred to as WebService
• REST often referred to as REST API
21.
22.
23.
24.
25. REST APIs
• Versioning is important
– /v1/apiname/resource
– /v2/apiname/resource
• Must properly implement HTTP error codes
• Commonly return JSON, YML.
43. Version Control
• System that records changes to files and set of
files
• You can easily recall specific versions
• Great for collaboration:
– Branch
– Merge
– Revisions
• Popular version control: GIT, Mercurial
• GITHUB != GIT
46. DevOps
• Short for Development and Operations
• As Systems Scale, automation is critical
• Do more with less
• Consistent and easily deploy servers, manage
networks and applications
60. What’s the Plan?
• Last week Q&A
• Overview of Knowledge Transfer Plan Week 2
• Knowledge Transfer Talk Topics: Week 2
61. Last week we saw:
• CAP Project Overview
• Overview of concepts in:
– Development
– PR.gov Infrastructure / Network Design
62. Last Week: Development Topics Q&A
• Software Architecture Style (Micro-Services)
• Application Programming Interfaces
• Redundancy and Scalability
• Asynchronous vs Synchronous Services
• Open Data (data.pr.gov)
• Version Control
• Open Source (Github / Application Stack)
• Software Development Method (DevOps)
73. Keepalived
• Free Open Source Software
• Zero licensing costs
• Written in pure C
• Used for High-availability
• Implements VRRP
• I/O Multiplexer provides realtime networking
• Robust and Stable
74. Keepalived & VRRP
• VRRP is IETF protocol
• Allows two or more routers to act as a virtual
router
• Routers present a Virtual IP Address (VIP) that
corresponds to a Virtual Mac Address (VMAC)
• Each router has a real hardware and IP
address
75. Keepalived & VRRP
• Linux does not support Virtual MACs.
• Keepalived only implements VIPs, works fine
on all modern networks
• Requires a network that allows gratitious
Advanced Resolution Protocol (ARP) requests
– An advanced notification
– Updates cache ARP cache before other systems
ask for them
76. Keepalived & VRRP
• Backup Server(s) monitor continuously
– Listens for multicast advertisements
– Expects them from the current master server
• If master disappears
– An election process ocurrs
– The highest priority backup wins
– Winning backup announces gratitious ARP for that
VMAC, and takes over
– Happens almost instantly
79. • IP Virtual Server: provides transport-layer load
balancing inside the Linux kernel
• Layer-4 switching (OSI)
• Allows things Linux Virtual Servers (LVS)
– Cluster of servers
– Appears as single server to user
– Layer 4 balancing
• Note: we dont use LVS
Keepalived & IPVS
89. HAProxy: What is it?
• Realible, High Performance Load Balancer
• Can Load-balance both TCP and HTTP
• Can handle massive amounts of traffic
• Can queue up requests for a server
• Can be configured to send specific amounts of
traffic to an application:
– Configure to never sends more than you can
handle
– Helpful for heavy apps
90. HAProxy: Who uses it?
• Who uses it? Internet Giants:
– Twitter
– Instagram
– Reddit
– Tumblr
– Airbnb
– Farmville
– Imgur
– Github
– TaoBao: Largest Picture Content Distribution Network in
the World
– <insert censored site> uses it
91. HAProxy:
• How do we use it?
– HTTP Load-Balancing
– TCP Load-Balancing
– Health Checks
– Statistics
101. Transport Layer Security
• TLS is a protocol that ensures privacy between
communicating applications and their users
• When a server and client communicate, TLS
aims to ensure that no third party may
eavesdrop or tamper with any message.
• TLS is the successor to the Secure Sockets
Layer (SSL).
102. Transport Layer Security
• SSLv1, SSLv2, SSLv3 and TLSv1 are all
vulnerable
• Weakest protocol supported by the system is
used for attacks
– POODLE
– HEARTBLEED
103. Transport Layer Security
• Check your servers for insecure cyphers and
protocols:
– https://www.ssllabs.com
• Man in the middle attacks can decrypt HTTPS
data
109. Secure Shell
• SSH is a cryptographic (encrypted) network
protocol
• Used for initiating text-based shell sessions
• Used for administering remote machines in a
secure way
– Network Switches
– Servers
• Can authenticate using:
– Username / Password (less secure)
– SSH Keys (more secure)
113. SSH: Generate your Keys
• Default is 2048 bit key
• Use: ssh-keygen -t rsa -b 4096
114. SSH: Transfer your Keys
• One way:
– ssh-copy-id <username>@<host>
– Example: ssh-copy-id acolon@192.168.1.1
• Or:
– Copy your public key to the server
– cp authorized_keys authorized_keys_Backup
– cat id_rsa.pub >> authorized_keys
119. What’s the Plan?
• Last week Q&A
• Overview of Knowledge Transfer Plan Week 3
• Knowledge Transfer Talk Topics: Week 3
120. Q&A - Last week we saw:
Infrastructure Services:
• Keepalived (high-availability with VRRP in Linux)
• HaProxy (Load-Balancing in Linux)
– Introduction
– HTTP Load-Balancing
– TCP Load-Balancing
– Health Checks
• Transport Layer Security (TLS) & SSL vulnerabilities
• Passwordless Authentication – SSH Key Authentication
121. Overview of technologies in
Operations:
Infrastructure Services:
• Advanced Key Value Store: Redis
• PostFix
• Bind9
122. Databases
• There isn’t a “one-size fits all”
• Choosing the right tech, hinges on use case
• If your data doesn’t change and has moderate
manageable growth: SQL not dead for you
• High throughput and growth, efficient scaling,
rapid data change: NoSQL
123. RDBMS
• Scalability:
– Scability is vertical
– More data usually means bigger servers
– Scaling across multiple servers is possible but time-
consuming
• Fixed Schema
– Must be decided and locked before data entry
• ACID compliance
• Stored in Relational Model
– Rows: contain all information about specific entity
– Columns: contain all the seperate data points about entity.
125. Polyglot Persistence
“Use the right tool for what you’re trying to
accomplish”
CAP:
Our project uses both SQL and NoSQL
– PR.gov successfully implements NoSQL (Redis)
– RCI uses both SQL and NoSQL (MSSQL, MongoDB)
126.
127. What is NoSQL?
• An alternate way of thinking about databases
• NoSQL = “Not Only SQL”
• Not a Relational Database
• Data not modeled in terms of tabular relations
• Some NoSQL databases are ACID compliant,
but some sacrifice compliancy for
performance and scalability
128. How many types of NoSQL?
There are plenty of NoSQL flavours:
• Key-Value Stores
• Document databases
• Graph Databases
• Wide Column Databases
129. NoSQL Adoption
• Size Matters:
– When working with large datasets, consistently scaling
is easier to achieve with many NoSQL family
• Speed:
– NoSQL is usually faster and sometimes extremely
speedier in writes
– Reads can also be very fast depending on the NoSQL
DB used and data being queried
• NoSQL has seen rapid adoption in web-
technologies
131. Data in NoSQL
• Key-Value Stores
– Associative Array of key-value Pairs
• Document databases
– Stored as collection of document, structure can vary
• Graph Databases
– Data is stored in nodes, properties and lines
• Wide Column Databases
– Data is stored in column families, rows can have
different columns
137. Introduction to Redis
• Most important feature: high-performance
• Advanced Key-Value Store
• Often referred to as a Data Structure Server
• Open Source (BSD license)
• Built-in replication
• Multiple Persistence Options
• Read and Write speeds obsessively
documented
138. Redis: high-performance
• In-memory database
• Small code-base (20k lines in C)
• Connection via TCP or Unix Socket
• No nested data structures
• Persistence via Snapshotting and/or Journaling
• Master/Slave chain database replication
• Sentinel Server Monitoring – real clustering now
in beta
139. Redis: Uses Cases
• Caching
• Statistics collection (downloads, hits, time
benchmarks)
• Log buffers
• Tasks Queues
• Share state between processes
• Inter-proccess communication in a distributed
network
• Built-in Publish Subscribe
141. Redis: Why we love it!
• Automatic Key Expiration
• Great for both caching and storage
• Scales for millions of requests
• Used for fast, self-expiring sessions on Web App
• Used for fast, self-expiring transactions on GMQ
• Powerful Libraries available for EventMachine
(Reactor Pattern) used by our GMQ API for Redis
• Redis used by Github’s Resque for asynchronous
workers
142. Cap Redis
• Used primarily as:
– Web Session Storage
– Transaction Storage
– Workers Coordination
– Statistics
143. Redis: Data Structure
• Often referred to as a Data Structure Server
– Can contain Strings
– Hashes
– Lists
– Sets
– Sorted sets
– Bitmaps
145. Redis: Master the Data Types
• Redis can be used as Key Value storage
• But to get the most out of it, think of it as a tool set
• When designing for efficiency, think how to best model
your data using the myriad of available data types
• Think of how you want to store your data, including the
key
161. Redis: Persistence
• In-memory
– No storage on disks. Useful for caches
• RDB
– Favors performance over persistence
– Very compact single-file representation
– Perfect for Backups (backup daily, keep snapshots for months)
– Very good for disaster recovery (compact, easily transferable)
– RDB maximizes performance since all the parent process needs to do in order to persist is
forking a child that will do all the rest. The parent instance will never perform disk I/O or alike.
– Can save every X seconds or if more than Y number of transactions have been changed
• AOF
– Favors persistence over performance
– Much more durable than RDB
– An append-only log, there are no seeks nor corruption problems if power failure
– If log ends with half-written command (disk-failure, etc), redis-check-aof tool fixes it easily
– Much bigger than RDB
– More aggressive storage, as it favors persistence
164. Email
• One of the most popular internet services to
date
• Facilitates communication
• MTA – mail transfer agents, move mail from
one mail system to another
• MDA – mail delivery agents, move mail from
one system to the user’s mailbox
165. Postfix
• Free open source mail transfer agent (MTA)
• Handles routing and delivery of email
• Solid Email Server for Linux
• The default MTA for a number linux
distributions including Ubuntu
• Very useful for SMTP Relay
• Quick setup, very reliable
166. Postfix Queues
• Incoming Queue:
– Receives mail from other hosts
– As long as emails are arriving and it hasn’t been
processed, it is kept in this queue
• Active queue:
– The queue that actually deliver messages
– It has a limited size and messages are accepted if
there is space for them. Other queues must wait
for the active queue to be ready to accept items.
167. Postfix Queues
• Deferred queue:
– Email that cannot be delivered
– Prevents the system from continously trying to deliver
email
– Keeps the active queue short, by storing failed emails,
and thus newer messages get priority
– Enhances stability
– If MTA cannot reach a domain, emails are stored here
– Retry is scheduled with an increasing waiting time.
– After wait, the item is put on the active queue.
169. Postfix: how we use it
• GMQ provides a REST interface for mailing
• GMQ workers queue jobs in relay server
• Postfix mail server is not exposed to the
internet
• Traffic is only outbound, not inbound
170. Postfix: Installation
• sudo apt-get install postfix
• Select “Internet Site”
• Enter name of your domain
• Additional configuration:
– Edit: /etc/postfix/main.cf
• Sender Policy Framework (SPF) record is
important for the domain you wish to relay
171. Postfix: Installation
• Postfix start – starts the server
• Postfix stop – stop the server
• Postfix reload – reloads configuration without
downtime
172. Postfix: Important commands:
• check queue size:
– mailq
• Check current queue:
– postqueue –q
• Flush the queue (force resend):
– postqueue –f
• Show number of emails being sent to each domain:
– qshape active
• Same as above but for deferred queue
– qshape deferred
174. DNS
• Allows not to hard-code IPs in our network
• Possible to associate multiple names to the
same machine to update the different
available services
• Makes our infrastructure more resilient to
future changes
175. Bind9
• Free Open Source DNS Server
• Massively popular in the linux community
• Resilient and easy to install and configure
• Allows for master and slaves DNS
• Allows for zone transfers
188. Topics
• Some thoughts on Programming Languages
• Ruby History
• Who uses Ruby
• Ruby Basics
• Learn Ruby by Example
189. The Principles of Languages
• Thinking is Important for Programmers, we
can only code what we can think
• But how do we think?
– In words of specific languages
– We grasp the world by language and express
ourselves with them
– Languages are not only tools to communicate but
also tools to Think
190. Programmer’s Thoughts
• Natural languages are:
– too ambiguos,
– too verbose
– too indirect
• In code, written down thoughts become
programs
196. About Ruby
Ruby is a dynamic, object-oriented,
general-purpose programming
language.
197. About Ruby
Ruby is a:
• dynamic,
• object-oriented,
• general-purpose programming
language
198. Dynamic Languages
High-level programming languages which
at runtime, execute many common
programming behaviors that static
programming languages perform during
compilation.
199. Object Oriented Programming (OOP)
A programming paradigm based on the
concept of "objects”, which are data
structures that contain data, in the form of
fields, often known as attributes; and code,
in the form of procedures, often known as
methods.
200. General Purpose Language
In computer software a general-purpose
programming language is a programming
language designed to be used for writing
software in a wide variety of application
domains.
201. About Ruby: History
• Relatively young, 1995
• From Japan
• Designed to be Natural
• Grew hugely in popularity with the Rails
Framework (Ruby on Rails)
202. About Ruby
• Free:
– Developed as open source with a very open
license
– Freedom to learn from the source
– Freedom to extend and modify
203. About Ruby
• Ruby is strong in scripting as Perl
– Built in regular expressions
– Almost all equivalent functionality
• Can access all system calls on the Operating
System via a standard library
– Ruby/DL (Dynamic Loading)
– Explicit libraries: syscall, Win32API
• Useful for scripting, but not limited to it
204. About Ruby
• Ruby’s OOP Features:
– Object
• Everything is an object
– Class
• Every class is an object
– Methods
• Every procedure is a method;
206. Hello World: LotusScript
%INCLUDE "symphonylsx.lss"
Dim application As SymphonyApplication
Dim documents As SymphonyDocuments
Dim document As SymphonyDocument
Dim range As SymphonyTextRange
Set application = New SymphonyApplication
Set documents = application.Documents
Set document = documents.AddDocument("",True,True)
Set range = document.content.End
Call range.InsertBefore("Hello World")
213. Ruby is focused on
programmer productivity
over machine optimization
214.
215.
216. Ruby Uses
• Simulations
• 3D Modeling
• Business
• Robotics
• Networking
• Game Development
• System Administration
• Web Applications
• Security
217. Who is Using it?
• NASA (Langely Research Center)
• Google (Sketchup)
• Lucent (3G wireless telephony product)
• Level 3 Communications (central data collection
for over 1,700 global servers)
• 37Signals (Basecamp)
• Twitter
• AT&T (YellowPages.com)
• StateFarm (R&D Center)
218. Ruby Features
• Cross Platform
• Object Oriented
• Powerful string operations
• Variables are not typed
• Regular Expressions
219. Ruby Features
• Class Inheritance
• Garbage Collection
• Threads
• Iterators and Closures
• Exception Handling
• Operator Overloading
• Introspection, Reflection, Meta Programming
226. Basics: Types of Variables
• Constant variables
– Cannot be changed.
• Local variables
– Local to a specific scope. Such as a method.
• Global variables
– Accessible through the entire progarm
• Class variables
– Accessible to the class.
• Instance variables
– Specific to each instance of a class
258. Hands-on Experience
• Now we’re going to have a hands-on
experience with Ruby
• Let’s see some examples
• Let’s modify them real-time based on Q&A
• Let’s catch up on some basic Git
259. Learn Ruby by Example - Follow me to github:
https://github.com/mindware/cap_ruby_training.git
260. Let’s Learn by Example
• Loops
– Basics
– Hashes
– Arrays
• Classes
– Instances
– Methods
– Getters and Setters
– Inheritance
– Namespaces
261. Let’s Learn by Example
• Gems
• DotEnv and Environment Variables
• Rake
• Bundler
• Internationalization
262. …Done! What we did:
• We practiced Git for version control
• We learned some ruby basics
• We saw some cool ruby examples
• Now let’s learn about Ruby Gems
274. Rack is the foundation for all modern Ruby Web
Frameworks
275. Rack provides a common interface between
server and Applications.
276. By wrapping HTTP requests and responses in the
simplest way possible, it unifies and distills the
API for web servers, web frameworks, and
software in between (the so-called middleware)
into a single method call.
277. Rack allows you to write once and run
everywhere:
• Puma
• Goliath
• Thin
• Webrick
285. Detailed Demo
• Let’s see some demos and practice:
– EventMachine
– Goliath
– Grape
– Sinatra
– Padrino
– Redis-Rb
– Hi-Redis
– EM::Synchrony
• Q&A
286. Detailed Demo
• Let’s see some demos and practice:
– EventMachine
– Goliath
– Grape
– Sinatra
– Padrino
– Redis-Rb
– Hi-Redis
– EM::Synchrony
• Q&A
287. Now let’s see how we used these:
• Let’s review our Github Source Code for:
– CAP Web App
– GMQ CAP API
– GMQ Workers
• Head over to:
– https://github.com/commonwealth-of-puerto-rico
• Q&A
298. • En la prensa se publicaron algunos de los
problemas, pero no todos. Tip of the
Iceberg.
• El certificado anterior incorporaba
información que no habia sido validada
con otras agencias.
• Muchos patronos utilizaban información
en el certificado que la Policia no habia
validado correctamente.
299. En el antiguo sistema: si se entraba un seguro social
inventado, con la información (falsa) de Homero Simpson,
la Policia le emitia un certificado sin validar la identidad.
300.
301. En las profundidades….
• Habia personas cometiendo
fraude con estos certificados;
• Sistema se apagaba 8 horas para
hacer backup;
• Solicitudes no se reintentaban si
ocurrían un fallos básicos en el
302. • En los intentos de fraude, el
Registro Demográfico de la Policía
almacenaba información
incorrecta y luego no emitía
certificado verdadero dueño del
seguro social;
• No habia forma de consultar
datos de delitos menos graves de
la Policia, por lo que los
certificados no cumplian con la
ley.
303. Mas abajo:
• Los datos se encontraba en 4 bancos de datos
distintos, que no se hablaban entre si.
• RCC, el sistema de Justicia que alimentaba el
banco de datos principal de la Policia, seria
decomisado en semanas.
• Policia no tenia forma de sincronizar los delitos
Graves de forma automatizada y necesitaban
ayuda
318. Código Disponible en Github
https://github.com/commonwealth-of-puerto-rico/prgov_cap_webapp/
319. START
Ready!
Modulos para la
Policia
Desarrollo de APIs y Micro Services (RCI) Equipo
Componentes Técnicos
Desarollo de Web App
Desarollo de Sistema
De Mensajeria Gubernamental PR.Gov
Personal Técnico,
Agilidad en
Contratación y Accesos
325. …Antes:
• En fallas, las solicitudes no se reintentaban automaticamente
• No se emitian certificados positivos
• No se validaba la información de identidad previo a la emisión
• Certificados emitidos en ventanilla no era posible invalidarlos
posteriormente una vez emitido, aún si contenian errores.
• Certificado de PR.gov era aceptado por patronos, pero se imprimia en hoja
de papel regular.
• Policia emitia en papel especial con un alto costo para la agencia.
• La seguridad del papel no era funcional toda vez que si emitian
incorrectamente un certificado, no podian cancelarlo.
• Certificado no expiraba.
332. Validación de Identidad:
• Aceptamos con o sin acentos, mayuzcula o
minuscula.
• La información de las agencias es la utilizada.
• El sistema tiene inteligencia para detectar
posibles errores en los nombres e identificar
apropiadamente.
• Si toda la validación es correcta, se emite
certificado.
• De requerir evaluación humana, se envia a
analista de la Policia
334. Retos del Beta
• Algunas personas tienen su información
incorrecta en DTOP y estamos colaborando
interagencialmente en el particular
• Interesamos incorporar nuevos métodos de
validación en PR.gov (licencias de otros países, y
pasaporte)
335. Logros (Alpha)
• Consolidamos cuatro sistemas de datos
criminales en uno, adoptando RCI
• Validación identidad del ciudadano en DTOP
• Integración del registro de ofensores sexuales
• Integración de modulo para entrada de los
delitos menos grave de la Policía
• Un mismo proceso de validación, para
solicitudes presenciales en las ventanillas de la
Policía y en línea en PR.gov
• Funciona en celulares y tabletas
• Sistema escalable y con $0 costos de
licenciamiento
336. Logros (Beta)
• Se emiten certificados positivos por primera vez
por PR.Gov
• Certificados que antes salían negativos, ahora
salen positivos correctamente.
• Se valida la identidad del ciudadano previo a la
emisión.
• Se incorporó más allá que la tecnología, un
análisis de los procesos operacionales de la
Policía, para atender sus necesidades.
• Servicio en español e inglés
337. Logros del Prototipo (Beta)
• Servicio en español e inglés
• Por primera vez, personal que es sentenciado,
busca certificado el mismo día, y se le emite
positivo.
347. Resumen:
• Nuevo Certificado es más seguro y rápido.
• Require un ID de DTOP
• Se trámita rápido y de forma segura
• Funciona en tu móvil, tabletas y PCs
• Versión beta está disponible en:
– http://servicios.pr.gov/cap
• App de solicitud y validación disponible para
Android y Iphone (keyword: PRGOV)
Editor's Notes
July 2015
WARNING – debido a restricciones de tiempo, muchos temas van a estar presentados sustancialmente de forma simplificada.
Queda de cada uno de los participantes,
Five minute break
Multiple versions can return different data, the developers knows exactly what to expect.
Cuando veamos VRRP van a ver que se vé parecido a esto.
WARNING – debido a restricciones de tiempo, muchos temas van a estar presentados sustancialmente de forma simplificada.
Queda de cada uno de los participantes,
Keep your SERVICE alive
Via VRRP provides you with Virtual IP that is assigned to the server available
Via VRRP provides you with Virtual IP that is assigned to the server available
IETF Internet Engineering Task Force
The Address Resolution Protocol (ARP) is a telecommunication protocol used for resolution of network layer addresses into link layer addresses, a critical function in multiple-access networks. ARP was defined by RFC 826 in 1982.
he gratuitous ARP packet has the following characteristics: Both source and destination IP in the packet are the IP of the host issuing the gratuitous ARP The destination MAC address is the broadcast MAC address (ff:ff:ff:ff:ff:ff) This means the packet will be flooded to all ports on a switch No reply is expected Gratuitous ARP is used for some reasons: Update ARP tables after a MAC address for an IP changes (failover, new NIC, etc.) Update MAC address tables on L2 devices (switches) that a MAC address is now on a different port Send gratuitous ARP when interface goes up to notify other hosts about new MAC/IP bindings in advance so that they don't have to use ARP requests to find out When a reply to a gratuitous ARP request is received you know that you have an IP address conflict in your network
HSRP, VRRP etc. use gratuitous ARP to update the MAC address tables on L2 devices (switches). Also there is the option to use the burned-in MAC address for HSRP instead of the "virtual"one. In that case the gratuitous ARP would also update the ARP tables on L3 devices/hosts.
Via VRRP provides you with Virtual IP that is assigned to the server available
Via VRRP provides you with Virtual IP that is assigned to the server available
No need to focus too much on this. Just a general view.
Via VRRP provides you with Virtual IP that is assigned to the server available
NOTE WE DONT USE LVS
Refresh their memory.
How many layers? 7
What does OSI mean? Open Systems Interconnection
Name the layers.
Use sudo before the commands to specify super user
Load-balancing we redirect traffic and can scale
But single point of failure
Use sudo before the commands to specify super user
Use sudo before the commands to specify super user
Use sudo before the commands to specify super user
You can configure multiple backends and front ends with multiple servers
VRRP for high-availability in Proxy Servers, Routers
Or Linux firewalls like: pfsense, ipfire, smoothwall.
Use sudo before the commands to specify super user
WARNING – debido a restricciones de tiempo, muchos temas van a estar presentados sustancialmente de forma simplificada.
Queda de cada uno de los participantes,
, there really isn’t a ‘one-system-fits-all’ approach; choosing the right technology hinges on the use case. If your data needs are changing rapidly, you need high throughput to handle viral growth, or your data is growing fast and you need to be able to scale out quickly and efficiently, maybe NoSQL is for you. But if the data you have isn’t changing in structure and you’re experiencing moderate, manageable growth, your needs may be best met by SQL technologies. Certainly, SQL is not dead yet.
In computer science, ACID (Atomicity, Consistency, Isolation, Durability) is a set of properties that guarantee that database transactions are processed reliably. In the context of databases, a single logical operation on the data is called a transaction.
different data storage technologies for different kinds of data
Remind them of ACID:
Atomicity, Consistency, Isolation, Durability
To facilitate this kind of analysis, OLAP data is stored in a multidimensional database. Whereas a relational database can be thought of as two-dimensional, a multidimensional database considers each data attribute (such as product, geographic sales region, and time period) as a separate "dimension."
You can configure multiple backends and front ends with multiple servers
WARNING – debido a restricciones de tiempo, muchos temas van a estar presentados sustancialmente de forma simplificada.
Queda de cada uno de los participantes,
Perhaps your frustration was not with Programming, but with a Programming Lanage!
False.
Operator Overloading: methods perform different on classes, and you can define your own
Operator Overloading: methods perform different on classes, and you can define your own
Operator Overloading: methods perform different on classes, and you can define your own
Operator Overloading: methods perform different on classes, and you can define your own
Operator Overloading: methods perform different on classes, and you can define your own
Operator Overloading: methods perform different on classes, and you can define your own
Operator Overloading: methods perform different on classes, and you can define your own
Operator Overloading: methods perform different on classes, and you can define your own
Operator Overloading: methods perform different on classes, and you can define your own
There are more operators, but we leave them as an assignment to look.
There are more operators, but we leave them as an assignment to look.
There are more operators, but we leave them as an assignment to look.
Operator Overloading: methods perform different on classes, and you can define your own
Operator Overloading: methods perform different on classes, and you can define your own
Operator Overloading: methods perform different on classes, and you can define your own
Operator Overloading: methods perform different on classes, and you can define your own
Operator Overloading: methods perform different on classes, and you can define your own
Operator Overloading: methods perform different on classes, and you can define your own
Operator Overloading: methods perform different on classes, and you can define your own
Operator Overloading: methods perform different on classes, and you can define your own
Operator Overloading: methods perform different on classes, and you can define your own
Operator Overloading: methods perform different on classes, and you can define your own
Operator Overloading: methods perform different on classes, and you can define your own
(note: This training happened during a time where we were in a severe drought. The audience picked up on the examples quickly. In this example we were showing the amounts of days without water)
Operator Overloading: methods perform different on classes, and you can define your own
Operator Overloading: methods perform different on classes, and you can define your own
Operator Overloading: methods perform different on classes, and you can define your own
Operator Overloading: methods perform different on classes, and you can define your own
Operator Overloading: methods perform different on classes, and you can define your own
Operator Overloading: methods perform different on classes, and you can define your own
Operator Overloading: methods perform different on classes, and you can define your own
Operator Overloading: methods perform different on classes, and you can define your own
Operator Overloading: methods perform different on classes, and you can define your own
Here we did some hands on training and exmamples. We did a Q&A
-t = threads
-w workers
Allows for concurrency
Uses Copy on Write when creating processes
En general veremos esto
Tip of the Iceberg:
Se incorporaba información para el certificado que no habia sido validada con otras agencias.
La informacion suministrada por el usuario aparecia tal el alegaba.
Muchos patronos esperaban información en el certificado que la policia en ese momento no podia validar.
En otras palabras….si se entraba la informacion de Homero Simpson.
Si esa información no tenia record criminal…
Se le emitia un certificado negativo.
En el fondo….
Esta era la manera en que se sincronizaba Tribunales con la Policia de Puerto Rico.
Y asi…
Certificado online se tardaba demasiado en llegar;
Necesidad de ir físicamente a Policía para agilizar el proceso;
We used the federal playbook to build digital servces, developed by the United States Digital Service.
Office of the Chief Information Officer of the Government of Puerto Rico Team at PR.gov:
Andrés Colón Pérez
Arelies Rivera
Giancarlo Gonzales
Alberto Colón
Sasha Mendez
David Acevedo
Ruth Silva
Line of Sight at the DOJ:
Omar Cruz
Osvaldo Ferrero
Simmone Mago
Juan Jimenez
Lizjacnira Martinez
DOJ CJIS:
Juan Marin
Heriberto Luna
Edwin
Puerto Rico Police Department (Policia):
Walter Lamela
Juan Carlos Rivera
OGP:
Frank Hernandez
Nilda Lebron
Roberto Clausell
Juan Cabrera
Angel Ayala