Conda is a cross-platform package manager that lets you quickly and easily build environments containing complicated software stacks. It was built to manage the NumPy stack in Python but can be used to manage any complex software dependencies.
The Docker network overlay driver relies on several technologies: network namespaces, VXLAN, Netlink and a distributed key-value store. This talk will present each of these mechanisms one by one along with their userland tools and show hands-on how they interact together when setting up an overlay to connect containers.
The talk will continue with a demo showing how to build your own simple overlay using these technologies.
2022.03.23 Conda and Conda environments.pptxPhilip Ashton
A presentation for the African Pathogen Genomics initiative at KEMRI-Wellcome in Kilifi Kenya on Conda and Conda environments. Includes a practical exercise.
The Docker network overlay driver relies on several technologies: network namespaces, VXLAN, Netlink and a distributed key-value store. This talk will present each of these mechanisms one by one along with their userland tools and show hands-on how they interact together when setting up an overlay to connect containers.
The talk will continue with a demo showing how to build your own simple overlay using these technologies.
2022.03.23 Conda and Conda environments.pptxPhilip Ashton
A presentation for the African Pathogen Genomics initiative at KEMRI-Wellcome in Kilifi Kenya on Conda and Conda environments. Includes a practical exercise.
MongoDB SoCal 2020: Migrate Anything* to MongoDB AtlasMongoDB
During this talk we'll navigate through a customer's journey as they migrate an existing MongoDB deployment to MongoDB Atlas. While the migration itself can be as simple as a few clicks, the prep/post effort requires due diligence to ensure a smooth transfer. We'll cover these steps in detail and provide best practices. In addition, we’ll provide an overview of what to consider when migrating other cloud data stores, traditional databases and MongoDB imitations to MongoDB Atlas.
Systemd: the modern Linux init system you will learn to loveAlison Chaiken
The talk combines a design overview of systemd with some tutorial incofrmation about how to configure it. Systemd's features and pitfalls are illustrated by short demos and real-life examples. Files used in the demos are listed under "Presentations" at http://she-devel.com/
Video of the live presentation will appear here:
http://www.meetup.com/Silicon-Valley-Linux-Technology/events/208133972/
Introducing envoy-based service mesh at Booking.comIvan Kruglov
Service mesh is a dedicated layer of a company's infrastructure which should simplify communication between services and make it reliable, secure and observable.
In this talk, we'll go deep into Booking.com's case study of introducing service mesh. We will discover the reasons and objectives of the project. Why Envoy was selected as the base rather than other available options. Find out what is the setup and features of the homegrown control plane. We will expand on what service is provided for developers and how they safely deploy potentially dangerous configuration changes. Finally, we will talk about pitfalls met along the way.
En esta charla vamos hablar cerca de una estrategia de release llamada Blue-green deployment.
Cuales son sus vantages y custos que podremos tener y como eso se queda en practica.
Presentation slides of the workshop on "Introduction to Pig" at Fifth Elephant, Bangalore, India on 26th July, 2012.
http://fifthelephant.in/2012/workshop-pig
MongoDB for Coder Training (Coding Serbia 2013)Uwe Printz
Slides of my MongoDB Training given at Coding Serbia Conference on 18.10.2013
Agenda:
1. Introduction to NoSQL & MongoDB
2. Data manipulation: Learn how to CRUD with MongoDB
3. Indexing: Speed up your queries with MongoDB
4. MapReduce: Data aggregation with MongoDB
5. Aggregation Framework: Data aggregation done the MongoDB way
6. Replication: High Availability with MongoDB
7. Sharding: Scaling with MongoDB
An Introduction to Makefile.
about 23 slides to present you a quick start to the make utility, its usage and working principles. Some tips/examples in order to understand and write your own
Makefiles.
In this presentation you will learn why this utility continues to hold its top position in project build software, despite many younger competitors.
Visit Do you know Magazine : https://www.facebook.com/douknowmagazine
These are slides from our Big Data Warehouse Meetup in April. We talked about NoSQL databases: What they are, how they’re used and where they fit in existing enterprise data ecosystems.
Mike O’Brian from 10gen, introduced the syntax and usage patterns for a new aggregation system in MongoDB and give some demonstrations of aggregation using the new system. The new MongoDB aggregation framework makes it simple to do tasks such as counting, averaging, and finding minima or maxima while grouping by keys in a collection, complementing MongoDB’s built-in map/reduce capabilities.
For more information, visit our website at http://casertaconcepts.com/ or email us at info@casertaconcepts.com.
MongoDB SoCal 2020: Migrate Anything* to MongoDB AtlasMongoDB
During this talk we'll navigate through a customer's journey as they migrate an existing MongoDB deployment to MongoDB Atlas. While the migration itself can be as simple as a few clicks, the prep/post effort requires due diligence to ensure a smooth transfer. We'll cover these steps in detail and provide best practices. In addition, we’ll provide an overview of what to consider when migrating other cloud data stores, traditional databases and MongoDB imitations to MongoDB Atlas.
Systemd: the modern Linux init system you will learn to loveAlison Chaiken
The talk combines a design overview of systemd with some tutorial incofrmation about how to configure it. Systemd's features and pitfalls are illustrated by short demos and real-life examples. Files used in the demos are listed under "Presentations" at http://she-devel.com/
Video of the live presentation will appear here:
http://www.meetup.com/Silicon-Valley-Linux-Technology/events/208133972/
Introducing envoy-based service mesh at Booking.comIvan Kruglov
Service mesh is a dedicated layer of a company's infrastructure which should simplify communication between services and make it reliable, secure and observable.
In this talk, we'll go deep into Booking.com's case study of introducing service mesh. We will discover the reasons and objectives of the project. Why Envoy was selected as the base rather than other available options. Find out what is the setup and features of the homegrown control plane. We will expand on what service is provided for developers and how they safely deploy potentially dangerous configuration changes. Finally, we will talk about pitfalls met along the way.
En esta charla vamos hablar cerca de una estrategia de release llamada Blue-green deployment.
Cuales son sus vantages y custos que podremos tener y como eso se queda en practica.
Presentation slides of the workshop on "Introduction to Pig" at Fifth Elephant, Bangalore, India on 26th July, 2012.
http://fifthelephant.in/2012/workshop-pig
MongoDB for Coder Training (Coding Serbia 2013)Uwe Printz
Slides of my MongoDB Training given at Coding Serbia Conference on 18.10.2013
Agenda:
1. Introduction to NoSQL & MongoDB
2. Data manipulation: Learn how to CRUD with MongoDB
3. Indexing: Speed up your queries with MongoDB
4. MapReduce: Data aggregation with MongoDB
5. Aggregation Framework: Data aggregation done the MongoDB way
6. Replication: High Availability with MongoDB
7. Sharding: Scaling with MongoDB
An Introduction to Makefile.
about 23 slides to present you a quick start to the make utility, its usage and working principles. Some tips/examples in order to understand and write your own
Makefiles.
In this presentation you will learn why this utility continues to hold its top position in project build software, despite many younger competitors.
Visit Do you know Magazine : https://www.facebook.com/douknowmagazine
These are slides from our Big Data Warehouse Meetup in April. We talked about NoSQL databases: What they are, how they’re used and where they fit in existing enterprise data ecosystems.
Mike O’Brian from 10gen, introduced the syntax and usage patterns for a new aggregation system in MongoDB and give some demonstrations of aggregation using the new system. The new MongoDB aggregation framework makes it simple to do tasks such as counting, averaging, and finding minima or maxima while grouping by keys in a collection, complementing MongoDB’s built-in map/reduce capabilities.
For more information, visit our website at http://casertaconcepts.com/ or email us at info@casertaconcepts.com.
The big picture view of the IPython architecture. Based on this document: http://ipython.org/ipython-doc/dev/development/how_ipython_works.html
A lightning talk for SFPython November 2014
Keynote of HadoopCon 2014 Taiwan:
* Data analytics platform architecture & designs
* Lambda architecture overview
* Using SQL as DSL for stream processing
* Lambda architecture using SQL
Using Anaconda to light up dark data. My talk given to the Berkeley Institute of Data Science describing Anaconda and the Blaze ecosystem for bringing a virtual analytical database to your data.
Six Feet Up's senior developer Clayton Parker will lead you on a journey to become a Python zen master. Your code should be as fashionable as it is functional. To quote the Zen of Python, "Beautiful is better than ugly". This talk will teach you about the Python style guide and why it is important. The talk will show you examples of well written Python and how to analyze your current code to make Guido proud.
Big Data and Fast Data - Lambda Architecture in ActionGuido Schmutz
Big Data (volume) and real-time information processing (velocity) are two important aspects of Big Data systems. At first sight, these two aspects seem to be incompatible. Are traditional software architectures still the right choice? Do we need new, revolutionary architectures to tackle the requirements of Big Data?
This presentation discusses the idea of the so-called lambda architecture for Big Data, which acts on the assumption of a bisection of the data-processing: in a batch-phase a temporally bounded, large dataset is processed either through traditional ETL or MapReduce. In parallel, a real-time, online processing is constantly calculating the values of the new data coming in during the batch phase. The combination of the two results, batch and online processing is giving the constantly up-to-date view.
This talk presents how such an architecture can be implemented using Oracle products such as Oracle NoSQL, Hadoop and Oracle Event Processing as well as some selected products from the Open Source Software community. While this session mostly focuses on the software architecture of BigData and FastData systems, some lessons learned in the implementation of such a system are presented as well.
Installing Anaconda Distribution of PythonJatin Miglani
This document is useful when use with Video session I have recorded today with execution, This is document no. 1 of course "Introduction of Data Science using Python". Which is a prerequisite of Artificial Intelligence course at Ethans Tech.
Disclaimer: Some of the Images and content have been taken from Multiple online sources and this presentation is intended only for Knowledge Sharing
>>WATCH THE WEBINAR HERE: https://codefresh.io/docker-based-pipelines-with-codefresh/
Most people think that Docker adoption means deploying Docker images. In this webinar, we will see the alternative way of adopting Docker in a Continuous Integration Pipeline, by packaging all build tools inside Docker containers. This makes it very easy to use different tool versions on the same build and puts an end to version conflicts in build machines. We will use Codefresh as a CI/CD solution as it fully supports pipelines where each build step is running on its own container image.
Sign up for FREE Codefresh account (120 builds/month) at Codefresh.io/codefresh-signup
Presentació a càrrec de Víctor Pérez, tècnic de Càlcul i Aplicacions del CSUC, duta a terme a la "4a Jornada de formació sobre l'ús del servei de càlcul" celebrada el 17 de març de 2021 en format virtual.
Most people think "adopting containers" means deploying Docker images to production. In practice, adopting containers in the continuous integration process provides visible benefits even if the production environment are VMs.
In this webinar, we will explore this pattern by packaging all build tools inside Docker containers.
Container-based pipelines allow us to create and reuse building blocks to make pipeline creation and management MUCH easier. It's like building with Legos instead of clay.
This not only makes pipeline creation and maintenance much easier, it also solves a myriad of classic CI/CD problems such as:
Putting an end to version conflicts in build machines
Eliminating build machine management in general
Step portability and maintenance
In a very real sense, Docker-based pipelines reflect lessons learned from microservices in CI/CD pipelines. We will share tips and tricks for running these kinds of pipelines while using Codefresh as a CI/CD solution as it fully supports pipelines where each build step is running on its own Docker image.
Slides from Thom Cuddihy for the 12 July 2018 EMBL-ABR webinar about Bioconda and Conda.
Thom, a bioinformatician from QFAB and the Research Computing Centre at the University of Queensland talked about Bioconda, which is the most popular and widely used bioinformatics channel for Conda (the package, dependency and environment management tool for any language—Python, R, Ruby, Lua, Scala, Java, JavaScript, C/ C++, FORTRAN). By adding a tool to the Bioconda ecosystem, it becomes widely available as an installable tool package for various operating systems and hardware that is stored in a fully-supported, global repository of bioinformatics tools.
The webinar covered Bioconda basics and how Australian researchers can use it to streamline bioinformatics tool wrapping for use on various systems.
A recording of the webinar is available here: https://youtu.be/lGa9PCSH5IU
The Latest Status of CE Workgroup Shared Embedded Linux Distribution ProjectYoshitake Kobayashi
The CE workgroup of Linux Foundation has started a project to share the work of maintaining long-term support for an embedded distribution, by leveraging the work of the Debian and Debian LTS project. Debian gives you pre-compiled binary packages but the meta-debian layer enables to install customized packages to create similar or smaller images. If both usecases are able to share the source code, it is good to share the maintenance effort.
In this talk, Yoshitake will describe the details of meta-debian which provides a meta layer for the Poky build system. This talk will to gives the latest status, technical details and lessons learned from its development.
All source code are available on GitHub and related document also available on Github and elinux wiki.
Undine: Turnkey Drupal Development EnvironmentsDavid Watson
Undine is a cross-platform, fully-featured development VM (virtual machine) for Drupalistas of all experience levels. Sponsored by Stevens Institute of Technology, it is a turnkey solution to many of the common pain points encountered in developing for Drupal.
Download Undine: http://drupal.org/project/undine
Managing Software Dependencies and the Supply Chain_ MIT EM.S20.pdfAndrew Lamb
Slides presented at an MIT seminar: Wrangling Software Projects
Provides both a commercial and an open-source perspective on the benefits, costs, and risks of taking on dependencies.
Building distribution packages with DockerBruno Cornec
This presentation demonstrate how to use Docker in order to build upstream Fedora or Mageia distribution packages much more easily than usual. It was given during Fosdem 2016
Leverage the power of Open Source in your company Guillaume POTIER
Open source is a major tech key nowadays for companies. In this presentation I try to explain how to carefully choose your OS libraries and how to share some bits of your company code to the OS world.
Presented at useR! 2014, July 2, 2014.
The R ecosystem is in a state of near constant change. While a new version of the R engine is now released just once a year, 2-3 patches are usually released in the interim. On top of that, new versions of R packages on CRAN are released at rate of several per day (and that’s not counting packages that are part of the BioConductor project or hosted elsewhere on the Web).
While this rapid change is a boon for the advancement of R, it can cause problems for package authors[1] and also for scientists and their peers who may need to reliably reproduce the results of an R script (possibly dependent on a number of packages) months or even years down the line. In this talk we propose a downstream distribution of CRAN packages that provides for the reproducibility of R scripts and reduces the impact of dependencies for packages authors.
A survey of problems involved in building containers and build tools such as:
buildah
nixos-container
ansible-container
Smith
Distroless
Buildkit
Source to Image (s2i)
Habitat
Leonid Vasilyev "Building, deploying and running production code at Dropbox"IT Event
Reproducible builds, fast and safe deployment process together with self-healing services form the basis of stable and maintainable infrastructure. In this talk I’d like to cover, from the Site Reliability Engineering (SRE) perspective, how Dropbox addresses above challenges, what technologies are used and what lessons were learnt during implementation process.
Slides from Ben Golub's (Docker CEO, @golubbe) opening day keynote at the DockerCon EU conference in Amsterdam on December 4, 2014 (http://europe.dockercon.com/)
At my first visit to SciPy in Latin America, I was able to review the history of PyData, SciPy, and NumFOCUS, and discuss how to grow its communities and cooperate in the future. I also introduce OpenTeams as a way for open-source contributors to grow their reputation and build businesses.
Keynote talk at PyCon Estonia 2019 where I discuss how to extend CPython and how that has led to a robust ecosystem around Python. I then discuss the need to define and build a Python extension language I later propose as EPython on OpenTeams: https://openteams.com/initiatives/2
Talk given at first OmniSci user conference where I discuss cooperating with open-source communities to ensure you get useful answers quickly from your data. I get a chance to introduce OpenTeams in this talk as well and discuss how it can help companies cooperate with communities.
A lecture given for Stats 285 at Stanford on October 30, 2017. I discuss how OSS technology developed at Anaconda, Inc. has helped to scale Python to GPUs and Clusters.
Python is dominating the fast-growing data-science landscape. This talk provides a foundational overview of the practice of data science and some of the most popular Python libraries for doing data science. It also provides an overview of how Anaconda brings it all together.
With Dask and Numba, you can NumPy-like and Pandas-like code and have it run very fast on multi-core systems as well as at scale on many-node clusters.
Making NumPy-style and Pandas-style code faster and run in parallel. Continuum has been working on scaled versions of NumPy and Pandas for 4 years. This talk describes how Numba and Dask provide scaled Python today.
With Anaconda (in particular Numba and Dask) you can scale up your NumPy and Pandas stack to many cpus and GPUs as well as scale-out to run on clusters of machines including Hadoop.
Blaze: a large-scale, array-oriented infrastructure for PythonTravis Oliphant
This talk gives a high-level overview of the motivation, design goals, and status of the Blaze project from Continuum Analytics which is a large-scale array object for Python.
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Tobias Schneck
As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other?
Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Connector Corner: Automate dynamic content and events by pushing a buttonDianaGray10
Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to:
Create a campaign using Mailchimp with merge tags/fields
Send an interactive Slack channel message (using buttons)
Have the message received by managers and peers along with a test email for review
But there’s more:
In a second workflow supporting the same use case, you’ll see:
Your campaign sent to target colleagues for approval
If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team
But—if the “Reject” button is pushed, colleagues will be alerted via Slack message
Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors.
And...
Speakers:
Akshay Agnihotri, Product Manager
Charlie Greenberg, Host
JMeter webinar - integration with InfluxDB and GrafanaRTTS
Watch this recorded webinar about real-time monitoring of application performance. See how to integrate Apache JMeter, the open-source leader in performance testing, with InfluxDB, the open-source time-series database, and Grafana, the open-source analytics and visualization application.
In this webinar, we will review the benefits of leveraging InfluxDB and Grafana when executing load tests and demonstrate how these tools are used to visualize performance metrics.
Length: 30 minutes
Session Overview
-------------------------------------------
During this webinar, we will cover the following topics while demonstrating the integrations of JMeter, InfluxDB and Grafana:
- What out-of-the-box solutions are available for real-time monitoring JMeter tests?
- What are the benefits of integrating InfluxDB and Grafana into the load testing stack?
- Which features are provided by Grafana?
- Demonstration of InfluxDB and Grafana using a practice web application
To view the webinar recording, go to:
https://www.rttsweb.com/jmeter-integration-webinar
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualityInflectra
In this insightful webinar, Inflectra explores how artificial intelligence (AI) is transforming software development and testing. Discover how AI-powered tools are revolutionizing every stage of the software development lifecycle (SDLC), from design and prototyping to testing, deployment, and monitoring.
Learn about:
• The Future of Testing: How AI is shifting testing towards verification, analysis, and higher-level skills, while reducing repetitive tasks.
• Test Automation: How AI-powered test case generation, optimization, and self-healing tests are making testing more efficient and effective.
• Visual Testing: Explore the emerging capabilities of AI in visual testing and how it's set to revolutionize UI verification.
• Inflectra's AI Solutions: See demonstrations of Inflectra's cutting-edge AI tools like the ChatGPT plugin and Azure Open AI platform, designed to streamline your testing process.
Whether you're a developer, tester, or QA professional, this webinar will give you valuable insights into how AI is shaping the future of software delivery.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
Neuro-symbolic is not enough, we need neuro-*semantic*Frank van Harmelen
Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”.
All of this illustrated with link prediction over knowledge graphs, but the argument is general.
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Ramesh Iyer
In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
UiPath Test Automation using UiPath Test Suite series, part 3DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 3. In this session, we will cover desktop automation along with UI automation.
Topics covered:
UI automation Introduction,
UI automation Sample
Desktop automation flow
Pradeep Chinnala, Senior Consultant Automation Developer @WonderBotz and UiPath MVP
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
Transcript: Selling digital books in 2024: Insights from industry leaders - T...
Effectively using Open Source with conda
1. Effectively using Open Source
with conda
Travis E. Oliphant, PhD
Continuum Analytics, Inc
2. The Opportunity
• Millions of
projects that can
be used in the
enterprise
• Not enough to
just adopt once
— these projects
change rapidly
• Effective use
requires a plan
for managing
updates
4. The Challenge
• Different “entry-points” (end-user applications or
scripts) can have different dependencies. Often many
of the dependencies are shared but a few applications
need different versions of some packages.
• Not specific to any particular language or ecosystem.
Python, Ruby, Node.Js, C/C++, .NET, Java, all have the
same problem: How do you manage software life-cycle
effectively?
• Production deployments need stability. IT managers
want ease of deployment and testing. Developers want
agility and ease of development.
5. The Challenge
How can developers and domain
experts in an organization quickly and
easily take advantage of the latest
software developments yet still have
stable production deployments of
complex software?
You cannot take full advantage of the
pace of open-source development if
you don’t address this!
6. Case Study: SciPy
There was this thing called the
Internet and one could make a
web-page and put code up on
it and people started using it ...
Facebook for Hackers
I started SciPy in 1999 while I was in grad-
school at the Mayo Clinic
(it was called Multipack back then)
7. Case Study: SciPy
Packaging circa 1999: Source tar ball and
make file (users had to build)
SciPy is basically a bunch of C/C++/Fortran routines
with Python interfaces
Observation: Popularity of Multipack (Early SciPy)
grew significantly when Robert Kern made pre-
built binaries for Windows
8. Case Study: SciPy
• Difficulty of producing binaries plus the desire to avoid
the dependency chain and lack of broad packaging
solutions led to early SciPy being a “distribution” instead
of separate inter-related libraries.
• There were (and are) too many different projects in
SciPy (projects need 1-5 core contributors for
communication dynamic reasons related to team-sizes)
9. Case Study: NumPy
I started writing NumPy in 2005 while I
was teaching at BYU (it was a merger
of Numeric and Numarray)
NumPy ABI has not changed
“officially” since 1.0 came out in 2006
Presumably extension modules (SciPy, scikit-learn, matplotlib,
etc.) compiled against NumPy 1.0 will still work on NumPy 1.8.1
This was not a design goal!!!
10. Case Study: NumPy
This was a point of some contention and
community difficulty when date-time was
added in version 1.4 (impossible without
changing the ABI in some way) but not
really settled until version 1.7
The fundamental reason was a user-driven
obsession with keeping ABI compatibility.
Windows users lacked useful packaging
solution in face of NumPy-Stack
11. NumPy Stack (cry for conda...)
NumPy
SciPy Pandas Matplotlib
scikit-learnscikit-image statsmodels
PyTables
OpenCV
Cython
Numba SymPy NumExpr
astropy BioPython GDALPySAL
... many many more ...
12. Fundamental Principles
•Complex things are built out of simple things
•Fundamental principle of software engineering is
“separation of concerns” (modularity)
•Reusability is enhanced when you “do one thing
and do it well”
•But, to deploy you need to bring the pieces back
together.
!
•This means you need a good packaging system for
binary artifacts — with multiple-environments.
13. Continuum Solutions (Free)
Conda
binstar.org Anaconda
Free all-in-one distribution of Python for
Analytics andVisualization
• numpy, scipy, ipython
• matplotlib, bokeh,
• pandas, statsmodels, scikit-learn
• many, many more… 100+
Miniconda
Python + conda — with these you can
install exactly what you want…
• Binary repository of packages (public)
• Multiple package types
• Free public build queue
• Current focus on:
• Python pypi-compatible packages
(source distributions)
• conda packages (binary distributions)
$ conda install anaconda
• Cross-platform package manager
• Dependency management (uses SAT
solver to resolve all dependencies)
• System-level virtual environments (more
flexible than virtualenv)
14. Continuum Solutions (Premium)
Anaconda
Server
• Binary repository for private package
Premium features:
• hosting of private packages (public
packages are free)
• access to priority build queue
• $10 / month (individuals)
• 25 private packages
• 5 GB disk space
• $50 / month (organizations)
• 200 private packages
• 30 GB disk space
• right to have private packages in
organizations
• $1500 / year
• unlimited private packages
• 100 GB of disk space
binstar.org
• Internal mirror of public repositories
• Mix private internal packages with public
repositories
• Build customized versions of Anaconda
installers
• Environment to .exe and .rpm tools
• Comprehensive licensing
• Comprehensive support
• On-premise version of binstar.org
15. System Packaging solutions
yum (rpm)
apt-get (dpkg)
Linux OSX
macports
homebrew
Windows
chocolatey
npackd
Cross-platform
conda
With virtual environments conda provides a modern, cross-
platform, system-level packaging and deployment solution
16. Conda Features
• Excellent support for “system-level” environments (like
having miniVMs but much lighter weight than docker.io)
• Minimizes code-copies (uses hard/soft links if possible)
• Dependency solver using fast satisfiability solver (SAT
solver)
• Simple format binary tar-ball + meta-data
• Meta-data allows static analysis of dependencies
• Easy to create multiple “channels” which are repositories
for binary packages
• User installable (no root privileges needed)
• Can still use tools like pip --- conda fills in where they
fail.
17. Examples
Setup a test environment
$ conda update conda
$ conda create -n test python pip
$ source activate test
Install another package
(test)$ conda install scikit-learn
$ activate test
Windows
18. First steps
$ conda create -n py3k python=3.3
$ source activate py3k
Create an environment
Install IPython notebook
(py3k) $ conda install ipython-notebook
$ conda create -n py3k python=3.3 ipython-notebook
$ source activate py3k
All in One
19. Anaconda installation
ROOT_DIR!
The directory that Anaconda was installed into; for
example, /opt/Anaconda or C:Anaconda!
/pkgs!
Also referred to as PKGS_DIR. This directory contains
exploded packages, ready to be linked in conda
environments. Each package resides in a subdirectory
corresponding to its canonical name.!
/envs!
The system location for additional conda environments to be
created.!
!
the default, or root, environment!
/bin!
/include!
/lib!
/share
20. Look at conda package --- a simple .tar.bz2
http://docs.continuum.io/conda/intro.html
21. Anatomy of unpacked conda package
/lib
/include
/bin
/man
/info
files
index.json
bzipped tarfile of all the files comprising
the package at the full-paths they would
be installed to relative to a “system”
install or “chroot jail”
an environment is just a “union” of these paths
All conda packages have this info directory
which contains meta-data for tracked files,
dependency information, etc.
22. Environments
One honking great idea!
Let’s do more of those!
Easy to make
Easy to throw away
Uses:
• Testing (python 2.6, 2.7, 3.3)
• Development
• Trying new packages from PyPI
• Separating deployed apps with
different dependency needs
• Trying new versions of Python
• Reproducing someone’s work conda create -h
23. conda info -e
Getting System information
Basic info
conda info
Named-environment info
conda info --all
System info
conda info --system
24. conda install -n py3k scipy pip
http://repo.continuum.io/pkgs/dev
Experimental or developmental versions of packages
http://repo.continuum.io/pkgs/gpl
GPL licensed packages
http://repo.continuum.io/pkgs/free
non GPL open source packages
Default package repositories (configurable)
Installing packages
25. How it works
Channel 1
Channel 2
Channel N
metadata
metadata
metadata
conda
merged
metadata
l
l
l
26. Create channels
• Create a directory of conda packages
• Run conda index <dirname>
• Either use file:///path/to/dir in .condarc or
use simple web server on the /path/to/dir
Option 1
Option 2
Use binstar.org (also available as on-premise solution
with Anaconda Server)
27. Binstar.org — channels (request invite)
conda install -c
<channel name>
<pkg name>
!
will install from
binstar channel
!
or you can add
channel to your
config file
free for public packages
28. conda list also includes packages installed via pip!
List Installed packages
conda create -n py3k scipy pip
source activate py3k
pip install pint
$ conda list
# packages in environment at /Users/travis/anaconda/envs/py3k:
#
numpy 1.8.1 py27_0
openssl 1.0.1g 0
pint 0.4.2 <pip>
pip 1.5.4 py27_0
python 2.7.6 1
readline 6.2 2
scipy 0.13.3 np18py27_0
setuptools 3.1 py27_0
sqlite 3.7.13 1
tk 8.5.13 1
wsgiref 0.1.2 <pip>
zlib 1.2.7 1
Output
29. Update a package to latest
conda update pandas
get the latest pandas from the
channels you are subscribed to
conda update anaconda
change to the latest released anaconda
including its specific dependencies
this can downgrade packages if
they are newer than those in
the “released” Anaconda
conda update --all
To update all the packages in an
environment to the latest
versions use the --all option
30. conda search <regex>Search for a package
Find packages and channels they are in
conda search --outdated sympy
Only show packages matching regex
that are installed but outdated
conda search typo
typogrify * 2.0.0 py27_0 http://conda.binstar.org/travis/osx-64/
2.0.0 py33_1 http://conda.binstar.org/asmeurer/osx-64/
2.0.0 py26_1 http://conda.binstar.org/asmeurer/osx-64/
sympy 0.7.1 py27_0 defaults
!
0.7.4 py26_0 defaults
0.7.4.1 py33_0 defaults
* 0.7.4.1 py27_0 defaults
0.7.4.1 py26_0 defaults
0.7.5 py34_0 defaults
0.7.5 py33_0 defaults
l
l
l
l
l
l
31. conda remove -n py3k scipy matplotlib
Removing files and environments
Removing Packages
Removing Environment
conda remove -n py3k --all
Note: packages are
just “unlinked” from
environment. All the
files are still available
unpacked in a
package cache.
Removing unused packages
conda clean -t
conda clean -p
Remove unused tarballs
Remove unused directories
32. conda package -u
conda package --pkg-name bulk --pkg-version 0.1
Untracked Files
Easy way to install into an environment using
anything (pip, make, setup.py, etc.) and then package
up all of it into a binary tar-ball deployable via
conda install <pkg-name>.tar.bz2
!
pickle for binary code!
33. # This is a sample .condarc file
!
# channel locations. These override conda defaults, i.e., conda will
# search *only* the channels listed here, in the order given. Use "default" to
# automatically include all default channels.
!
channels:
- defaults
- http://some.custom/channel
!
# Proxy settings
# http://[username]:[password]@[server]:[port]
proxy_servers:
http: http://user:pass@corp.com:8080
https: https://user:pass@corp.com:8080
!
envs_dirs:
- /opt/anaconda/envs
- /home/joe/my-envs
!
pkg_dirs:
- /home/joe/user-pkg-cache
- /opt/system/pkgs
!
changeps1: False
!
# binstar.org upload (not defined here means ask)
binstar_upload: True
Conda configuration
Scripting interface
conda config —add KEY VALUE
conda config —remove-key KEY
conda config —get KEY
conda config —set KEY BOOL
conda config —remove KEY VALUE
35. Conda Recipe is a directory
build.sh BASH build commands (POSIX)
bld.bat CMD build commands (Win)
meta.yaml extended yaml declarative meta-data
Required
Optional
run_test.py will be executed during test phase
*.patch patch-files for the source
* any other resources needed by build but not included
in sources described in meta.yaml file
36. Recipe MetaData
package:
name: # name of package
version: # version of package
about:
home: # home-page
license: # license
!
# All optional from here....
source:
fn: # filename of source
url: # url of source
md5: # hash of source
# or from git:
git_url:
git_tag:
patches: # list of patches to source
- fix.patch
build:
entry_points: # entry-points (binary commands or scripts)
- name = module:function
number: # defaults to 0
requirements: # lists of requirements
build: # requirements for build (as a list)
run: # requirements for running (as a list)
test:
requires: # list of requirements for testing
commands: # commands to run for testing (entry-points)
imports: # modules to import for testing
http://docs.continuum.io/conda/build.html
37. Converting to another platform
Conda packages are specific to a particular
platform. However, if there are no platform-
specific binary files in a package, it can be
converted automatically to a package that can
be installed on another platform.
conda convert --output-dir win32 --platform win-32 <package-file>
Example
38. Binstar.org (request invite)
Once you
have built a
conda
package, you
can share it
with the
world on
binstar.org
!
conda install
-c <name>
<pkgname>
free for public packages
39. Binstar
$ conda config --add channels
'http://conda.binstar.org/travis'
$ conda config --add channels
'http://conda.binstar.org/asmuerer'
Adding channels
Uploading packages
binstar upload /full/path/to/package.tar.bz2
binstar register /full/path/to/package.tar.bz2
if package never uploaded before
40. Binstar Package Types
Permissions Description
Private
Only people given permission can see this
package.
Personal
Everyone will be able to see this package in
your user repository.
Publish
This package will be published in the global
public repository.