This document discusses managing Python dependencies at Beeswax using Bazel and related tools. It has 30 engineers working on a monorepo of 100 applications totaling 300 kLoC of Python code. Previously they used a global requirements.txt file, pipenv, and rules_python, but now use Bazel, pip-tools, and virtualenvs to manage dependencies. This new approach provides fully correct and minimal dependencies across the monorepo, works well with the monorepo structure, and maintains a history of dependency changes through lock files, though it has made their development and CI workflows more complex.
"TCP Input Text" implements the Google SOAP Search API and Bing API v2 to extract TCP Ports and Fully Qualified Domain Names (FQDN) from Google Search Results into a .csv file and individual shell scripts for maltego, nmap and nc aka netcat to provide assurance of a listening TCP service since the time that has past of the last crawl performed by the GoogleBot and BingBot
"TCP Input Text" implements the Google SOAP Search API and Bing API v2 to extract TCP Ports and Fully Qualified Domain Names (FQDN) from Google Search Results into a .csv file and individual shell scripts for maltego, nmap and nc aka netcat to provide assurance of a listening TCP service since the time that has past of the last crawl performed by the GoogleBot and BingBot
images oriented of c++ it can be easily learn to c++, it can be eagerly understand for students.and it can be easily understand the programming in c++. I have attached the simple programs in my slide it is very useful for students.
Presented at PyCon UK 2018 (18 September 2018, Cardiff).
The slides are incomplete.
Recording available at:
https://www.youtube.com/watch?v=-weU0Zy4Yd8
images oriented of c++ it can be easily learn to c++, it can be eagerly understand for students.and it can be easily understand the programming in c++. I have attached the simple programs in my slide it is very useful for students.
Presented at PyCon UK 2018 (18 September 2018, Cardiff).
The slides are incomplete.
Recording available at:
https://www.youtube.com/watch?v=-weU0Zy4Yd8
Python 2 is dead! Drag your old code into the modern ageBecky Smith
The clock is ticking on Python 2.7, with support to be dropped in January 2020. With major dependencies such as Django, NumPy and pandas moving to Python 3 only, the time has come for even big established codebases to consider upgrading. Many organisations are still postponing for various reasons; we will attempt to demonstrate that with a bit of planning and perseverance, and the assistance of some handy tools, we can embrace the future!
This session will provide a first-hand perspective on how we upgraded a large (~65,000 lines of python code) 8-year-old Django project with multiple external dependencies from Python 2.7 to Python 3.6.
We will briefly discuss the benefits of upgrading to Python 3, and architectural considerations. The session will primarily focus on the practicalities of upgrading the code itself. We will not try to provide a single “best” solution for upgrading to Python 3, but rather will introduce some of the available tools, provide an insight into how we used them, and their advantages and disadvantages from our experience. We will discuss preparatory steps and approaches, strategies for dealing with external dependencies, and “gotchas” that we encountered during the process.
The aim of this session is to provide an example of how a Python 3 upgrade on an established commercial product can be successfully completed, and to furnish audience members with a set of tools and strategies to help them with their own projects.
MongoDB Days UK: Using MongoDB and Python for Data Analysis PipelinesMongoDB
Presented by Eoin Brazil, Proactive Technical Services Engineer, MongoDB
Experience level: Advanced
MongoDB offers a flexible, scalable, and easy way to store your large data set. Python provides many useful data science tools (e.g. NumPy, SciPy, Scikit-learn, etc.). This talk will discuss the concerns for creating operational data analytic pipelines, introduce Monary as alternative for loading data into NumPy, and give examples of accessing data with Monary, as well as how to build scalable data analysis pipelines using these open source tools.
2018 jPrime Deconstructing and Evolving REST SecurityDavid Blevins
The learning curve for security is severe and unforgiving. Specifications promise infinite flexibility, habitually give old concepts new names, are riddled with extensions, and almost seem designed to deliberately confuse. For a back-end REST developer, choking all this down for the first time is mission impossible. With an aggressive distaste for fancy terminology, this session delves into OAuth 2.0 as it pertains to REST and shows how it falls into two camps: stateful and stateless. We then detail a competing Amazon-style approach called HTTP Signatures, ideal for B2B scenarios and similar to what is use to secure all Amazon AWS API calls. Each approach will be explored analyzing the architectural differences, with a heavy focus on the wire, showing actual HTTP messages and enough detail to have you thinking, "I could write this myself."
As a bonus at the end, well peak into a new IETF Internet Draft launched this year that combines JWT and HTTP Signatures into the perfect two-factor system that could provide a one-stop shop for business as well as mobile REST scenarios. Come to this session if you want to go from novice to expert with a bit of humor, a big picture perspective and wire-level detail.
Video: https://www.hashicorp.com/resources/operating-consul-at-scale
With more than 35k machines and first external Contributor, Criteo is a very large Consul User. This presentation describes how we operate Consul at Scale at Criteo.
Presentation at HashiTalks 2019.
Python se ha convertido en el lenguaje más usado para desarrollar herramientas dentro del ámbito de la seguridad. Esta charla se centrará en las diferentes formas en que un analista puede aprovechar el lenguaje de programación Python tanto desde el punto de vista defensivo como ofensivo.
Desde el punto de vista defensivo Python es una de las mejores opciones como herramienta de pentesting por la gran cantidad de módulos que nos pueden ayudar a desarrollar nuestras propias herramientas con el objetivo de realizar un análisis de nuestro objetivo.
Desde el punto de vista ofensivo podemos utilizar Python para recolección de información de nuestro objetivo de forma pasiva y activa. El objetivo final es obtener el máximo conocimiento posible en el contexto que estamos auditando. Entre los principales puntos a tratar podemos destacar:
1.Introducción a Python para proyectos de ciberseguridad(5 min)
2.Herramientas de pentesting(10 min)
3.Herramientas Python desde el punto de vista defensivo(10 min)
4.Herramientas Python desde el punto de vista ofensivo(10 min)
September 6 2017 Niklas Bivald held a PyPy talk at PyCon Sweden. The name of the talk was "Using PyPy - a faster version of Python for long running applications - as a first step to speed up your application."
It shows synthetic and real world usage of using PyPy to speed up your application.
Writing and debugging code is hard, but testing shouldn't be. This talk will discuss common techniques for writing simpler tests that still exercise your production code while preventing you from spending time debugging test code.
Similar to Python Dependencies at Beeswax - BazelCon 2019 Lightning Talk - Ron Rothman (20)
top nidhi software solution freedownloadvrstrong314
This presentation emphasizes the importance of data security and legal compliance for Nidhi companies in India. It highlights how online Nidhi software solutions, like Vector Nidhi Software, offer advanced features tailored to these needs. Key aspects include encryption, access controls, and audit trails to ensure data security. The software complies with regulatory guidelines from the MCA and RBI and adheres to Nidhi Rules, 2014. With customizable, user-friendly interfaces and real-time features, these Nidhi software solutions enhance efficiency, support growth, and provide exceptional member services. The presentation concludes with contact information for further inquiries.
TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERRORTier1 app
Even though at surface level ‘java.lang.OutOfMemoryError’ appears as one single error; underlyingly there are 9 types of OutOfMemoryError. Each type of OutOfMemoryError has different causes, diagnosis approaches and solutions. This session equips you with the knowledge, tools, and techniques needed to troubleshoot and conquer OutOfMemoryError in all its forms, ensuring smoother, more efficient Java applications.
Code reviews are vital for ensuring good code quality. They serve as one of our last lines of defense against bugs and subpar code reaching production.
Yet, they often turn into annoying tasks riddled with frustration, hostility, unclear feedback and lack of standards. How can we improve this crucial process?
In this session we will cover:
- The Art of Effective Code Reviews
- Streamlining the Review Process
- Elevating Reviews with Automated Tools
By the end of this presentation, you'll have the knowledge on how to organize and improve your code review proces
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I ...Juraj Vysvader
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I didn't get rich from it but it did have 63K downloads (powered possible tens of thousands of websites).
We describe the deployment and use of Globus Compute for remote computation. This content is aimed at researchers who wish to compute on remote resources using a unified programming interface, as well as system administrators who will deploy and operate Globus Compute services on their research computing infrastructure.
Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...Globus
The Earth System Grid Federation (ESGF) is a global network of data servers that archives and distributes the planet’s largest collection of Earth system model output for thousands of climate and environmental scientists worldwide. Many of these petabyte-scale data archives are located in proximity to large high-performance computing (HPC) or cloud computing resources, but the primary workflow for data users consists of transferring data, and applying computations on a different system. As a part of the ESGF 2.0 US project (funded by the United States Department of Energy Office of Science), we developed pre-defined data workflows, which can be run on-demand, capable of applying many data reduction and data analysis to the large ESGF data archives, transferring only the resultant analysis (ex. visualizations, smaller data files). In this talk, we will showcase a few of these workflows, highlighting how Globus Flows can be used for petabyte-scale climate analysis.
Top Features to Include in Your Winzo Clone App for Business Growth (4).pptxrickgrimesss22
Discover the essential features to incorporate in your Winzo clone app to boost business growth, enhance user engagement, and drive revenue. Learn how to create a compelling gaming experience that stands out in the competitive market.
Enhancing Research Orchestration Capabilities at ORNL.pdfGlobus
Cross-facility research orchestration comes with ever-changing constraints regarding the availability and suitability of various compute and data resources. In short, a flexible data and processing fabric is needed to enable the dynamic redirection of data and compute tasks throughout the lifecycle of an experiment. In this talk, we illustrate how we easily leveraged Globus services to instrument the ACE research testbed at the Oak Ridge Leadership Computing Facility with flexible data and task orchestration capabilities.
Prosigns: Transforming Business with Tailored Technology SolutionsProsigns
Unlocking Business Potential: Tailored Technology Solutions by Prosigns
Discover how Prosigns, a leading technology solutions provider, partners with businesses to drive innovation and success. Our presentation showcases our comprehensive range of services, including custom software development, web and mobile app development, AI & ML solutions, blockchain integration, DevOps services, and Microsoft Dynamics 365 support.
Custom Software Development: Prosigns specializes in creating bespoke software solutions that cater to your unique business needs. Our team of experts works closely with you to understand your requirements and deliver tailor-made software that enhances efficiency and drives growth.
Web and Mobile App Development: From responsive websites to intuitive mobile applications, Prosigns develops cutting-edge solutions that engage users and deliver seamless experiences across devices.
AI & ML Solutions: Harnessing the power of Artificial Intelligence and Machine Learning, Prosigns provides smart solutions that automate processes, provide valuable insights, and drive informed decision-making.
Blockchain Integration: Prosigns offers comprehensive blockchain solutions, including development, integration, and consulting services, enabling businesses to leverage blockchain technology for enhanced security, transparency, and efficiency.
DevOps Services: Prosigns' DevOps services streamline development and operations processes, ensuring faster and more reliable software delivery through automation and continuous integration.
Microsoft Dynamics 365 Support: Prosigns provides comprehensive support and maintenance services for Microsoft Dynamics 365, ensuring your system is always up-to-date, secure, and running smoothly.
Learn how our collaborative approach and dedication to excellence help businesses achieve their goals and stay ahead in today's digital landscape. From concept to deployment, Prosigns is your trusted partner for transforming ideas into reality and unlocking the full potential of your business.
Join us on a journey of innovation and growth. Let's partner for success with Prosigns.
Innovating Inference - Remote Triggering of Large Language Models on HPC Clus...Globus
Large Language Models (LLMs) are currently the center of attention in the tech world, particularly for their potential to advance research. In this presentation, we'll explore a straightforward and effective method for quickly initiating inference runs on supercomputers using the vLLM tool with Globus Compute, specifically on the Polaris system at ALCF. We'll begin by briefly discussing the popularity and applications of LLMs in various fields. Following this, we will introduce the vLLM tool, and explain how it integrates with Globus Compute to efficiently manage LLM operations on Polaris. Attendees will learn the practical aspects of setting up and remotely triggering LLMs from local machines, focusing on ease of use and efficiency. This talk is ideal for researchers and practitioners looking to leverage the power of LLMs in their work, offering a clear guide to harnessing supercomputing resources for quick and effective LLM inference.
First Steps with Globus Compute Multi-User EndpointsGlobus
In this presentation we will share our experiences around getting started with the Globus Compute multi-user endpoint. Working with the Pharmacology group at the University of Auckland, we have previously written an application using Globus Compute that can offload computationally expensive steps in the researcher's workflows, which they wish to manage from their familiar Windows environments, onto the NeSI (New Zealand eScience Infrastructure) cluster. Some of the challenges we have encountered were that each researcher had to set up and manage their own single-user globus compute endpoint and that the workloads had varying resource requirements (CPUs, memory and wall time) between different runs. We hope that the multi-user endpoint will help to address these challenges and share an update on our progress here.
Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...Globus
The U.S. Geological Survey (USGS) has made substantial investments in meeting evolving scientific, technical, and policy driven demands on storing, managing, and delivering data. As these demands continue to grow in complexity and scale, the USGS must continue to explore innovative solutions to improve its management, curation, sharing, delivering, and preservation approaches for large-scale research data. Supporting these needs, the USGS has partnered with the University of Chicago-Globus to research and develop advanced repository components and workflows leveraging its current investment in Globus. The primary outcome of this partnership includes the development of a prototype enterprise repository, driven by USGS Data Release requirements, through exploration and implementation of the entire suite of the Globus platform offerings, including Globus Flow, Globus Auth, Globus Transfer, and Globus Search. This presentation will provide insights into this research partnership, introduce the unique requirements and challenges being addressed and provide relevant project progress.
In software engineering, the right architecture is essential for robust, scalable platforms. Wix has undergone a pivotal shift from event sourcing to a CRUD-based model for its microservices. This talk will chart the course of this pivotal journey.
Event sourcing, which records state changes as immutable events, provided robust auditing and "time travel" debugging for Wix Stores' microservices. Despite its benefits, the complexity it introduced in state management slowed development. Wix responded by adopting a simpler, unified CRUD model. This talk will explore the challenges of event sourcing and the advantages of Wix's new "CRUD on steroids" approach, which streamlines API integration and domain event management while preserving data integrity and system resilience.
Participants will gain valuable insights into Wix's strategies for ensuring atomicity in database updates and event production, as well as caching, materialization, and performance optimization techniques within a distributed system.
Join us to discover how Wix has mastered the art of balancing simplicity and extensibility, and learn how the re-adoption of the modest CRUD has turbocharged their development velocity, resilience, and scalability in a high-growth environment.
Providing Globus Services to Users of JASMIN for Environmental Data AnalysisGlobus
JASMIN is the UK’s high-performance data analysis platform for environmental science, operated by STFC on behalf of the UK Natural Environment Research Council (NERC). In addition to its role in hosting the CEDA Archive (NERC’s long-term repository for climate, atmospheric science & Earth observation data in the UK), JASMIN provides a collaborative platform to a community of around 2,000 scientists in the UK and beyond, providing nearly 400 environmental science projects with working space, compute resources and tools to facilitate their work. High-performance data transfer into and out of JASMIN has always been a key feature, with many scientists bringing model outputs from supercomputers elsewhere in the UK, to analyse against observational or other model data in the CEDA Archive. A growing number of JASMIN users are now realising the benefits of using the Globus service to provide reliable and efficient data movement and other tasks in this and other contexts. Further use cases involve long-distance (intercontinental) transfers to and from JASMIN, and collecting results from a mobile atmospheric radar system, pushing data to JASMIN via a lightweight Globus deployment. We provide details of how Globus fits into our current infrastructure, our experience of the recent migration to GCSv5.4, and of our interest in developing use of the wider ecosystem of Globus services for the benefit of our user community.
Into the Box Keynote Day 2: Unveiling amazing updates and announcements for modern CFML developers! Get ready for exciting releases and updates on Ortus tools and products. Stay tuned for cutting-edge innovations designed to boost your productivity.
Paketo Buildpacks : la meilleure façon de construire des images OCI? DevopsDa...Anthony Dahanne
Les Buildpacks existent depuis plus de 10 ans ! D’abord, ils étaient utilisés pour détecter et construire une application avant de la déployer sur certains PaaS. Ensuite, nous avons pu créer des images Docker (OCI) avec leur dernière génération, les Cloud Native Buildpacks (CNCF en incubation). Sont-ils une bonne alternative au Dockerfile ? Que sont les buildpacks Paketo ? Quelles communautés les soutiennent et comment ?
Venez le découvrir lors de cette session ignite
4. # requirements.lock
boto3==1.10.2
botocore==1.13.2 # via boto3, s3transfer
certifi==2019.9.11 # via requests
chardet==3.0.4 # via requests
docutils==0.15.2 # via botocore
futures==3.3.0 # via s3transfer
idna==2.8 # via requests
jmespath==0.9.4 # via boto3, botocore
python-dateutil==2.8.0 # via botocore
requests==2.22.0
s3transfer==0.2.1 # via boto3
six==1.12.0 # via python-dateutil
urllib3==1.25.6 # via botocore,
requests
py_binary(
name = "my_app",
deps = ["lib_a"],
requires = ["requests>=2.21"],
)
py_library(
name = "lib_a",
deps = ["lib_b"],
requires = ["requests>=2.18"],
)
py_library(
name = "lib_b",
deps = ["lib_c"],
requires = ["boto3~=1.9"],
)
py_venv(
name = "my_app_env",
targets = [":my_app"],
)
# requirements.in
boto3~=1.9
requests>=2.18
requests>=2.21
5. Needs improvement:
● Dev/CI workflow more complicated than before
● Bazel cache misses
● Dependency changes are not propagated automatically
Working well:
● 100% correct (and minimal) dependencies!
● Monorepo-friendly
● History of dependency changes (lock files)
● Wheels are easy
We're 30 / monorepo / 100 / 300k
The problem... how to manage our Py 3p deps
By third-party...
[C]When we bāzel-test or bāzel-run some target...
also deps of every other target in its xtive clo
same in prod
[C]When had one or two apps, we solved... reqs.txt
...this didn't scale well once critical mass by multiple engs
rules_python - can't guarantee import correct version
pipenv
Which led us to roll...
Our solution, which we call Beeve... set of tools
pip-tools is...
composed of starlark macros/rules/aspects + command line tools
[C]Beeve's output is...
A lockfile is a set of 3p deps, each pinned
an application's reqs.txt, but autogenerated as needed
We leveraged: Bāzel's... walk the graph, collect 3pd; pip-tools's... resolve deps
[C]We take each app's lockfile and use it...
Here's an example. toy BUILD
Note we've annotated each py rule... new attr requires... dep spec
a dep spec is just a constraint on
py_venv - tells Beeve we want lockfile for the...
When run Bāzel on py_venv rule, our Starlark code... [C] requirements.in... union
[C]then calls pip-compile to generate...
the lockfile contains pinned versions of...
as well as their transitive deps...
ie it's the complete set of Py pkgs that our py_binary...
not hermetic; nor reproducible; current state PyPI
if you run... different lockfiles
guaranteed that any resulting lockfile conforms to constraints... !!!
one downside is... devs must now be aware of py envs, switch between them Bāzel-test
error prone
Bāzel caching less effective
3p dep changes are not propagated automatically to downstream...
[C]on plus side... most important goal... reliably know precise set of 3p deps...
contains everything/nothing
monorepo-friendly... go without saying
nice side benefit... track dep changes over time, since...
can easily produce wheels for our Py apps, has made deployments...
I'm happy to talk more about Beeve or Python dependencies
...email me with any questions or just find me at the conference.
Thank you!