Writing Well-behaved Unix Utilities
A talk given at the London Ruby Users Group in October 2014. It's about what makes a good Unix utility, and how we can use Ruby to write our own.
It covers things like working as part of text processing pipelines, reading from ARGF, handling command-line arguments, and lots more.
Node collaboration - Exported Resources and PuppetDBm_richardson
Node Collaboration - How can your servers share information with each other. Exploring Exported Resources, PuppetDB and other methods.
This talk was given at Sydney Puppet Users Meetup on 14/08/2014.
Elasticsearch, Logstash, Kibana. Cool search, analytics, data mining and more...Oleksiy Panchenko
In the age of information and big data, ability to quickly and easily find a needle in a haystack is extremely important. Elasticsearch is a distributed and scalable search engine which provides rich and flexible search capabilities. Social networks (Facebook, LinkedIn), media services (Netflix, SoundCloud), Q&A sites (StackOverflow, Quora, StackExchange) and even GitHub - they all find data for you using Elasticsearch. In conjunction with Logstash and Kibana, Elasticsearch becomes a powerful log engine which allows to process, store, analyze, search through and visualize your logs.
Video: https://www.youtube.com/watch?v=GL7xC5kpb-c
Scripts for the Demo: https://github.com/opanchenko/morning-at-lohika-ELK
Writing Well-behaved Unix Utilities
A talk given at the London Ruby Users Group in October 2014. It's about what makes a good Unix utility, and how we can use Ruby to write our own.
It covers things like working as part of text processing pipelines, reading from ARGF, handling command-line arguments, and lots more.
Node collaboration - Exported Resources and PuppetDBm_richardson
Node Collaboration - How can your servers share information with each other. Exploring Exported Resources, PuppetDB and other methods.
This talk was given at Sydney Puppet Users Meetup on 14/08/2014.
Elasticsearch, Logstash, Kibana. Cool search, analytics, data mining and more...Oleksiy Panchenko
In the age of information and big data, ability to quickly and easily find a needle in a haystack is extremely important. Elasticsearch is a distributed and scalable search engine which provides rich and flexible search capabilities. Social networks (Facebook, LinkedIn), media services (Netflix, SoundCloud), Q&A sites (StackOverflow, Quora, StackExchange) and even GitHub - they all find data for you using Elasticsearch. In conjunction with Logstash and Kibana, Elasticsearch becomes a powerful log engine which allows to process, store, analyze, search through and visualize your logs.
Video: https://www.youtube.com/watch?v=GL7xC5kpb-c
Scripts for the Demo: https://github.com/opanchenko/morning-at-lohika-ELK
This slides are used to present the following Twitter pipeline using the ELK stack (Elasticsearch, Logstash, Kibana): https://github.com/melvynator/ELK_twitter It shows how to integrate Machine Learning into your Twitter pipeline.
Redis - for duplicate detection on real time streamCodemotion
Roberto "frank" Franchini presenta a Codemotion Techmeetup Torino Redis, un data structure server che può utilizzare come chiavi stringhe, hashes, lists, sets, sorted sets, bitmaps e hyperloglogs
.
For the Docker users out there, Sematext's DevOps Evangelist, Stefan Thies, goes through a number of different Docker monitoring options, points out their pros and cons, and offers solutions for Docker monitoring. Webinar contains actionable content, diagrams and how-to steps.
Ontopic's presentation on Storm-Crawler for ApacheCon North America 2015.
Storm-Crawler is a next-generation web crawler that discovers and processes content on the Web, in real-time with low latency. This open source (and Apache Licensed) project is built on the Apache Storm framework, which provides a great foundation for a distributed real-time web crawler.
A presentation about the deployment of an ELK stack at bol.com
At bol.com we use Elasticsearch, Logstash and Kibana in a logsearch system that allows our developers and operations people to easilly access and search thru logevents coming from all layers of its infrastructure.
The presentations explains the initial design and its failures. It continues with explaining the latest design (mid 2014). Its improvements. And finally a set of tips are giving regarding Logstash and Elasticsearch scaling.
These slides were first presented at the Elasticsearch NL meetup on September 22nd 2014 at the Utrecht bol.com HQ.
This slides are used to present the following Twitter pipeline using the ELK stack (Elasticsearch, Logstash, Kibana): https://github.com/melvynator/ELK_twitter It shows how to integrate Machine Learning into your Twitter pipeline.
Redis - for duplicate detection on real time streamCodemotion
Roberto "frank" Franchini presenta a Codemotion Techmeetup Torino Redis, un data structure server che può utilizzare come chiavi stringhe, hashes, lists, sets, sorted sets, bitmaps e hyperloglogs
.
For the Docker users out there, Sematext's DevOps Evangelist, Stefan Thies, goes through a number of different Docker monitoring options, points out their pros and cons, and offers solutions for Docker monitoring. Webinar contains actionable content, diagrams and how-to steps.
Ontopic's presentation on Storm-Crawler for ApacheCon North America 2015.
Storm-Crawler is a next-generation web crawler that discovers and processes content on the Web, in real-time with low latency. This open source (and Apache Licensed) project is built on the Apache Storm framework, which provides a great foundation for a distributed real-time web crawler.
A presentation about the deployment of an ELK stack at bol.com
At bol.com we use Elasticsearch, Logstash and Kibana in a logsearch system that allows our developers and operations people to easilly access and search thru logevents coming from all layers of its infrastructure.
The presentations explains the initial design and its failures. It continues with explaining the latest design (mid 2014). Its improvements. And finally a set of tips are giving regarding Logstash and Elasticsearch scaling.
These slides were first presented at the Elasticsearch NL meetup on September 22nd 2014 at the Utrecht bol.com HQ.
Basic of and Unix and Command. More presentation you can find on www.scmGalaxy.com.
scmGalaxy.com is dedicated to software configuration, build and Release management. This covers CVS, VSS (Visual Source Safe),Perforce, SVN(Subversion) MKS Integrity, ClearCase,TFS,CM Synergy, Best Practices ,AnthillPro, Apache Ant, Maven, Bamboo, Cruise Control and many more tools.
Jenkins and Chef: Infrastructure CI and Automated DeploymentDan Stine
This presentation discusses two key components of our deployment pipeline: Continuous integration of Chef code and automated deployment of Java applications. CI jobs for Chef code run static analysis and then provision, configure and test EC2 instances. Release jobs publish new cookbook versions to the Chef server. Deployment jobs identify target EC2 and VMware nodes and orchestrate Chef client runs. The flexibility of Jenkins is essential to our overall delivery architecture.
Solution de transfert mobile - Formats d'échangeOCTO Technology
Les solutions de transfert mobiles sont nombreuses. Mais en pratique, c’est le format d’échange JSON qui est régulièrement utilisé. Déjà intégré pour le web, connu et lisible, ce format a tout pour plaire. Mais est-il le seul? Est-il réellement adapté au mobile?
Le poids de la données ainsi que sa vitesse d'encodage/décodage sont des facteurs importants pour diminuer le temps de chargement. De plus, une simplicité d'usage d'un format garantit une mise en place plus rapide, plus simple. La manipulation d'un JSON, par exemple, n'est pas si aisée en Android.
La présentation ci-dessous présente les trois formats comparés: JSON, Message Pack et Protocol Buffers.
Atmosphere 2014: Really large scale systems configuration - Phil DibowitzPROIDEA
For many years, Facebook managed its systems with cfengine2. With many individual clusters over 10k nodes in size, a slew of different constantly-changing system configurations, and small teams, this system was showing its age and the complexity was steadily increasing, limiting its effectiveness and usability. It was difficult to integrate with internal systems, testing was often impractical, and it provided no isolation of configurations, among many other problems. After an extensive evaluation of the tools and paradigms in modern systems configuration management – open source, proprietary, and a potential home-grown solution – we built a system based on the open-source project Chef. The evaluation process involved understanding the direction we wanted to take in managing the next many iterations of systems, clusters, and teams. More importantly, we evaluated the various paradigms behind effective configuration management and the different kinds of scale they provide. What we ended up with is an extremely flexible system that allows a tiny team to manage an incredibly large number of systems with a variety of unique configuration needs. In this talk we will look at the paradigms behind the system we built, the software we chose and why, and the system we built using that software. Further, we will look at how the philosophies we followed can apply to anyone wanting to scale their systems infrastructure.
Phil Dibowitz - Phil Dibowitz has been working in systems engineering for 12 years and is currently a production engineer at Facebook. Initially, he worked on the traffic infrastructure team, automating load balancer configuration management, as well as designing and building the production IPv6 infrastructure. He now leads the team responsible for rebuilding the configuration management system from the ground up. Prior to Facebook, he worked at Google, where he managed the large Gmail environment, and at Ticketmaster, where he co-authored and open sourced a configuration management tool called Spine. He also contributes to, and maintains, various open source projects and has spoken at conferences and LUG’s on a variety of topics from Path MTU Discovery to X509.
This is story of our journey from SaltStack to Puppet and beyond. This talk will answer following questions:
- why we moved from SaltStack
- why Puppet was chosen
- how to use Puppet OpenSource in painless way
- which orchestration tool to use with Puppet
- what is next
Don’t Forget About Your Past—Optimizing Apache Druid Performance With Neil Bu...HostedbyConfluent
Don’t Forget About Your Past—Optimizing Apache Druid Performance With Neil Buesing | Current 2022
Businesses need to react to results immediately; to achieve this, real-time processing is becoming a requirement in many analytic verticals. But sometimes, the move from batch to real-time can leave you in a pinch. How do you handle and correct mistakes in your data? How do you migrate a new system to real-time along with historical data?
Let’s start with how to run Apache Druid locally with your containerized-based development environment. While streaming real-time events from Kafka into Druid, an S3 Complaint Store captures messages via Kafka Connect, for historical processing. An exploration of performance implications when the real-time stream of events contains historical data and how that affects performance and the techniques to prevent those issues, leaving a high-performance analytic platform supporting real-time and historical processing.
You’ll leave with the tools of doing real-time analytic processing and historical batch processing from a single source of truth. Your Druid cluster will have better rollups (pre-computed aggregates) and fewer segments, which reduces cost and improves query performance.
Kubernetes intro public - kubernetes user group 4-21-2015reallavalamp
Kubernetes Introduction - talk given by Daniel Smith at Kubenetes User Group meetup #2 in Mountain View on 4/21/2015.
Explains the basic concepts and principles of the Kubernetes container orchestration system.
TXLF: Chef- Software Defined Infrastructure Today & TomorrowMatt Ray
The open source configuration management and automation framework Chef is used to configure, deploy and manage infrastructure of every sort. In addition to managing Linux, Windows and many other operating systems; Chef may be used to manage network hardware and storage systems. This session will provide an overview of the concepts and capabilities of Chef and discuss upcoming projects and how they fit into the Chef ecosystem.
Docker Service Registration and Discoverym_richardson
This talk covers some basic concepts of Service Registry and Discovery with Docker. Consul, Registrator and consul-template are discussed.
It was presented at the Sydney Docker meetup in April 2015
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionAggregage
Join Maher Hanafi, VP of Engineering at Betterworks, in this new session where he'll share a practical framework to transform Gen AI prototypes into impactful products! He'll delve into the complexities of data collection and management, model selection and optimization, and ensuring security, scalability, and responsible use.
UiPath Test Automation using UiPath Test Suite series, part 3DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 3. In this session, we will cover desktop automation along with UI automation.
Topics covered:
UI automation Introduction,
UI automation Sample
Desktop automation flow
Pradeep Chinnala, Senior Consultant Automation Developer @WonderBotz and UiPath MVP
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
A tale of scale & speed: How the US Navy is enabling software delivery from l...sonjaschweigert1
Rapid and secure feature delivery is a goal across every application team and every branch of the DoD. The Navy’s DevSecOps platform, Party Barge, has achieved:
- Reduction in onboarding time from 5 weeks to 1 day
- Improved developer experience and productivity through actionable findings and reduction of false positives
- Maintenance of superior security standards and inherent policy enforcement with Authorization to Operate (ATO)
Development teams can ship efficiently and ensure applications are cyber ready for Navy Authorizing Officials (AOs). In this webinar, Sigma Defense and Anchore will give attendees a look behind the scenes and demo secure pipeline automation and security artifacts that speed up application ATO and time to production.
We will cover:
- How to remove silos in DevSecOps
- How to build efficient development pipeline roles and component templates
- How to deliver security artifacts that matter for ATO’s (SBOMs, vulnerability reports, and policy evidence)
- How to streamline operations with automated policy checks on container images
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Ramesh Iyer
In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
16. Chef Server
Chef Search is fantastic for
• Searching for nodes with particular
attributes
• Searching for nodes with particular profiles
• Using search results to configure services on
other nodes.
17. Chef Server
Common use cases
• My application talks to a database. What’s its FQDN/IP?
• How many memcache nodes are there for my application? What are
there FQDN/IPs?
• Where should I send my log files? What’s the FQDN of the syslog
server?
• I want to provide a white-list of IP’s that can request information from a
node.
• What are the IP addresses of the webservers behind my load balancer?
• My Jenkins slaves need to connect to a Jenkins master. What’s it’s IP?
• What are the SSH Host keys for particular hosts.
19. Exported Resources
Definition:
An exported resource declaration specifies a
desired state for a resource, does not manage the
resource on the target system, and publishes the
resource for use by other nodes. Any node
(including the node that exported it) can then
collect the exported resource and manage its own
copy of it. *
* https://docs.puppetlabs.com/puppet/latest/reference/lang_exported.html
20. Exported Resources
Think of it as
• Node(s) “publish” resources to the Puppet Master
• Other Node(s) consume those resources
* https://docs.puppetlabs.com/puppet/latest/reference/lang_exported.html
21.
22. Exported Resources
Common Examples
• Backend servers configured in a load balancer
• Monitoring Servers updated with monitoring
clients (nagios_host / nagios_service)
• Distribute public keys (ssh and openssl)
24. PuppetDB
Definition:
collects data generated by Puppet. It enables
advanced Puppet features like the inventory service
and exported resources, and can be the foundation
for other applications that use Puppet’s data. *
25. PuppetDB
PuppetDB stores:
• The most recent facts from every node
• The most recent catalog for every node
• Optionally, 14 days of event reports for
every node
• Exported Resources
26. PuppetDB
PuppetDB stores:
• The most recent facts from every node
• The most recent catalog for every node
• Optionally, 14 days of event reports for
every node
• Exported Resources
27. PuppetDB
PuppetDB REST API
V3 API includes the following endpoints:
• Facts
• Resources
• Nodes
• Fact-names,
• Metrics
• Reports
• Events
• And more
30. PuppetDB
But this talk is about Node Collaboration.
How can I make use of this data?
31. PuppetDB
checkout
Puppetdbquery module
by Erik Dalén
https://forge.puppetlabs.com/dalen/puppetdbquery
https://github.com/dalen/puppet-puppetdbquery
32. PuppetDB
Puppetdbquery module
• Command line tools
• Puppet functions to query PuppetDB
• Hiera backend to return query results
from PuppetDB
33. PuppetDB
Puppetdbquery module
• Command line tools
• Puppet functions to query PuppetDB
• Hiera backend to return query results
from PuppetDB
37. PuppetDB
PuppetDBquery
• Works great and very similar to Chef
Search (searching for specific hosts, and
use their facts for configuring other
hosts).
• If you use Roles+Profiles pattern, this is
a fantastic fit for profiles.
40. Chef Search and PuppetDB are great!
but what if…
• You want your infrastructure to react more quickly than
your Puppet agent / Chef-client run intervals (are you
even running the agents continuously).
41. Chef Search and PuppetDB are great!
but what if…
• You want your infrastructure to react more quickly than
your Puppet agent / Chef-client run intervals (are you
even running the agents continuously).
• You appreciate “desired state” AND “running state” aren’t
always the same thing.
42. Chef Search and PuppetDB are great!
but what if…
• You want your infrastructure to react more quickly than
your Puppet agent / Chef-client run intervals (are you
even running the agents continuously).
• You appreciate “desired state” AND “running state” aren’t
always the same thing.
• You don’t want to run a Puppet Master or Chef Server
(masterless Puppet / Chef-solo).
43. Chef Search and PuppetDB are great!
but what if…
• You want your infrastructure to react more quickly than
your Puppet agent / Chef-client run intervals (are you
even running the agents continuously).
• You appreciate “desired state” AND “running state” aren’t
always the same thing.
• You don’t want to run a Puppet Master or Chef Server
(masterless Puppet / Chef-solo).
• You don’t use Puppet or Chef.
45. Confd
• “Lightweight configuration management tool”
• https://github.com/kelseyhightower/confd
• Written in go
• Manage local application configuration files using
templates and data from etcd or consul.
46. etcd
• Highly-available key/value store for shared
configuration and service discovery
• https://github.com/coreos/etcd
• Written in go
• Typical cluster size or 3-9 peers.
• CAP theorem
47. Confd + etcd
Distributed etcd cluster
etcd1
etcd2 etcd3 etcd4 etcd5
Hosts running applications
and services configured
with confd
48. Confd + etcd
• Hosts/Services that wish to share
information post data to etcd.
• Confd is configured on other hosts
to use this data to manage local
configuration files and services.
49. Confd + etcd
Distributed etcd cluster
etcd1
etcd2 etcd3 etcd4 etcd5
1 2
App server DB server
Example
1. App server runs Confd which is polling the following keys.
1. /myapp/db_ip
2. /myapp/db_port
3. /myapp/db_adapter
4. /myapp/db_name
2. Once up and running DB Server, writes details to etcd
cluster.
3. Confd reads key/values. Uses template file to write new
application configuration file and reloads application.
50. Confd + etcd
Demo
A “working” demo is worth a thousand powerpoint slides…
51. Summary
• Share information between your servers.
• Let them discover each other.
• Let your nodes collaborate with each
other.
Query_node - Accepts 2 arguments
Query to discover nodes
Fact that should be returned (optional)
Returns array of certnames or fact values
Query_facts – requires 2 arguments
query
list of facts to return in a nested hash
Polls key/value databases for data (etcd or consul).
Updates local configuration files with polled data via template files.
Reloads applications to pick up config file changes.