Solving BIG problems with Open Source: Condor

•

0 likes•443 views

This is an introduction to the queue distribution system Condor and how are we using it at the I3A (http://i3a.unizar.es)

Technology Business

> Solving Big problems with OS: Condor
> Antonio Sanz (ansanz@unizar.es)
> 09 / Nov / 11

3
> Antonio Sanz
> I3A System Manager
> HERMES HPC cluster sysadmin
> ansanz@unizar.es
> @antoniosanzalc

Problema inicial 6
> Dr. Good
> Neurologist
> Alzheimer research
> Process 20000
brain image scans
(1h/image)
> A thousand times.
Maybe two.

3. Sistemas de gestión de colas : Condor

Problema inicial 7
> Mrs. Nice
> Santa’s Logistic Officer
> Gift transportation
> Analize 6x10e7 possible
load/reindeers/routes
(10min/analysis)
> Before Christmas!

Condor Basics 10

Single queue

3. Sistemas de gestión de colas : Condor

Condor Basics 12

Multiple queues

3. Sistemas de gestión de colas : Condor

Problem can be broken into independent pieces 15

Condor Basics 17

For loops are
your best friends

3. Sistemas de gestión de colas : Condor

Condor Basics 18

While loops …
can sometimes be
convinced

3. Sistemas de gestión de colas : Condor

Condor Basics 19

Do it yourself !
3. Sistemas de gestión de colas : Condor

Condor Basics 21

Heterogeneous computing

Condor Basics 25

Fair use of resources

3. Sistemas de gestión de colas : Condor

Condor Basics 32

3. Sistemas de gestión de colas : Condor

Condor Basics 36

Job list ClassAd

3. Sistemas de gestión de colas : Condor

Condor Basics 41

Job running
3. Sistemas de gestión de colas : Condor

45

#!/bin/sh
# I’m hola.sh
echo Hola mundo desde `hostname`

#
# A Hello World .. In Condor!
#
# I’m hello.sub
Universe = vanilla
Executable = hola.sh
Log = hola.log
Output = hola.out
Error = hola.err
Queue

Hello, World !!

Lanzar el cálculo 46

condor_submit

4. Condor Basics – Un cálculo fácil

$48 Something tastier… #!/bin/sh # I’m hello2.sh OUTPUT=hello${1}.result cat hello.input >> $OUTPUT cat echo Hello world, I’m job $1 here from `hostname` > $OUTPUT # Execute n times with different outputs Universe = vanilla Executable = hello2.sh Transfer_input_files = hello.input WhenToTransferOutput = ON_EXIT_OR_EVICT Arguments = $(Process) Log = hello.log Output = hello.out Queue 10$

49

Perfect
Simulation

4. Condor Basics – Un cálculo fácil

Condor Basics 55

Hadoop Integration

3. Sistemas de gestión de colas : Condor

Condor Basics 63

3. Sistemas de gestión de colas : Condor

Condor Basics 64

> Conclusiones

3. Sistemas de gestión de colas : Condor

66

Antonio Sanz
ansanz@unizar.es
@antoniosanzalc

Slides here:
http://web.hermes.cps.unizar.es/doc/condor.pdf

Viewers also liked

Print™ Sample Analysis

Sociagility

Domain driven design in a nutshell

Toni Esteves

Informatica forense

mausmr

Por qué un Pecha Kucha es mejor que un Iphone (y casi tanto como el chocolate)

Antonio Sanz Alcober

Manual oms[1]

Elizza Quezzad

La ciber guerra en las relaciones internacionales

carinna_07

Capacidades de China para la ciberguerra

Antonio Sanz Alcober

07 decreto ap.julio 2007.joaquin

mariajoseadamuzgracia

Phising

Rossy Carhuaricra Celis

Seguridad para Pymes "Guerra de Guerrillas"

Tuvika Tuvika

Zen y el arte de pescar APT

Antonio Sanz Alcober

Administracion electronica: Tecnologías al servicio del ciudadano

Antonio Sanz Alcober

Propuestas para mejorar la calidad de vida de los enfermos terminales

EIDEC

Seguridad de la banca online - Consejos para usuarios

Antonio Sanz Alcober

Ciberguerra y terrorismo digital

Hacking Bolivia

(H)ac(k)tivismo, Ciberterrorismo y Ciberguerra

Joel A. Gómez Treviño

Pablo González & Juan Antonio Calles – Cyberwar: Looking for… touchdown! [Roo...

RootedCON

Triagedeurgencias MÉDICA Y ENFERMERÍA

Anelly May

Viewers also liked (18)

Print™ Sample Analysis

Domain driven design in a nutshell

Informatica forense

Por qué un Pecha Kucha es mejor que un Iphone (y casi tanto como el chocolate)

Manual oms[1]

La ciber guerra en las relaciones internacionales

Capacidades de China para la ciberguerra

07 decreto ap.julio 2007.joaquin

Phising

Seguridad para Pymes "Guerra de Guerrillas"

Zen y el arte de pescar APT

Administracion electronica: Tecnologías al servicio del ciudadano

Propuestas para mejorar la calidad de vida de los enfermos terminales

Seguridad de la banca online - Consejos para usuarios

Ciberguerra y terrorismo digital

(H)ac(k)tivismo, Ciberterrorismo y Ciberguerra

Pablo González & Juan Antonio Calles – Cyberwar: Looking for… touchdown! [Roo...

Triagedeurgencias MÉDICA Y ENFERMERÍA

Similar to Solving BIG problems with Open Source: Condor

condor_user_tutorial.ppt

vlaaade

Basics_of_Kernel_Panic_Hang_and_ Kdump.pdf

stroganovboris

Michal will take you on a journey all the way to 90’s and back, sharing the Mozilla detection framework - a systematic way to detect and hunt down threat actors. Why did we spend hours digging through some old Phrack issues? How does a blue team's member approach writing rootkits? What is better - a fail negative or a false positive? I will share answers to these questions plus a lot of alerting and evil-doing code.

"A rootkits writer’s guide to defense" - Michal Purzynski

PROIDEA

Solaris Kernel Debugging V1.0

Jarod Wang

Complexity is increasing. Trust eroding. In the wake of Spectre and Meltdown, when it seems that things cannot get any darker for processor security, the last light goes out. This talk will demonstrate what everyone has long feared but never proven: there are hardware backdoors in x86 processors, and they’re buried deeper than we ever imagined possible. In this talk, we walk through how we discovered a privilege escalation backdoor in a family of x86 CPUs, that allows an unprivileged user, on an unmodified system, to circumvent all processor security checks and escalate from ring 3 to ring 0 – permitting an unprivileged, arbitrary userland program to directly modify and execute code inside of the kernel, regardless of the operating system, security patches, antivirus, firmware, etc. Speakers: Christopher Domas, Cyber Security Researcher

GOD MODE Unlocked: Hardware backdoors in x86 CPUs

Priyanka Aash

100 bugs in Open Source C/C++ projects

Andrey Karpov

Ako využiť (ukradnúť) kolegove CPU cykly

cloudcampsk

Code profilers are used to identify execution bottlenecks and understand the cause of a slowdown. Execution sampling is a monitoring technique commonly employed by code profilers because of its low impact on execution. Regularly sampling the execution of an application estimates the amount of time the interpreter, hardware or software, spent in each method execution time. Nevertheless, this execution time estimation is highly sensitive to the execution environment, making it non reproductive, non-deterministic and not comparable across platforms. On our platform, we have observed that the number of messages sent per second remains within tight (±7%) bounds across a basket of 16 applications. Using principally the Pharo platform for experimentation, we show that such a proxy is stable, reproducible over multiple executions, profiles are comparable, even when obtained in different execution contexts. We have produced Compteur, a new code profiler that does not suffer from execution sampling limitations and have used it to extend the SUnit testing framework for execution comparison.

2011 ecoop

bergel

Intro to Arduino Programming.pdf

HimanshuDon1

This talk will cover the problems currently with why applications are not being sandboxed to lessen the attack surface. Mostly this is based upon the existing tools being not user friendly and requiring a low level knowledge of syscalls that is hard to find in application developers. Seccomp is one of these tools. It defines syscall filters that allow an application to define what syscalls it allows or denies. It is commonly used in the highly-regarded Chrome sandbox. Integrating things like seccomp filters into programming languages at build time could allow for creating a perfect set of filters based off the application code. In practice, some try to mock this behavior at runtime but it often fails due to certain functions not being called during testing and missing specific syscalls. Therefore causing the user to turn it off completely. By integrating it into the code at build time we can ensure that all the syscalls are accounted for. This talk will also show a proof of concept with this in Golang.

Making Security Invisible

J On The Beach

Slides from my talk at AWS ComSum 2022 in Manchester. If you are writing code in Python which communicates with AWS APIs you are more than likely using the boto3 library. boto3 isn't extended via inheritance or callbacks, instead it offers an event system which allows you to intercept and modify calls at different stages in their lifecycle. This is a very powerful mechanism but it can be hard to write code for very rare events. Using a tool like mitmproxy you can rewrite responses boto3 receives and simulate rarer events to aid writing code to handle them. In this talk I'll show: - How to write basic code to listen to events - An example of some events you might see in typical S3 requests - How to use mitmproxy to intercept, understand and ultimately rewrite HTTP requests between boto3 and AWS to simulate different scenarios - An example of the events you'll see during request retries - An example putting this together to insert monitoring into the boto3 retry mechanism to diagnose network issues Full source here: https://github.com/micktwomey/exploring-boto3-events-with-mitmproxy

Exploring Boto3 Events With Mitmproxy

Michael Twomey

Lab Manual Combaring Redis with Relational

Amazon Web Services

Ruun commands

Queen Lumanta

Monitoring a program that monitors computer networks

PVS-Studio

PythonBrasil[8] - CPython for dummies

Tatiana Al-Chueyr

In a world where microservices are more and more a standard architecture for Java based applications running in the cloud, the JVM warmup time can become a limitation. Especially when you look at spinning up new instances of an app as response to changes in load, the warmup time can be a problem. Native images are one solution to solve these problems because their statically ahead of time compiled code simply doesn’t have to warmup and so has short startup time. But even with the shorter startup time and smaller footprint it doesn’t come without a drawback. The overall performance might be slower because of the missing JIT optimizations at runtime. There is a new OpenJDK project called CRaC (Coordinated Restore at Checkpoint) which goal it is to address the JVM warmup problem with a different approach. The idea is to take a snapshot of the running JVM, store it in files and restore the JVM at a later point in time (or even on another machine). This session will give you a short overview of the CRaC project and shows some results from a proof of concept implementation.

What the CRaC - Superfast JVM startup

Gerrit Grunwald

Skiron - Experiments in CPU Design in D

Mithun Hunsur

Stealthy, Hypervisor-based Malware Analysis

Tamas K Lengyel

WWCode Dallas - Kubernetes: Learning from Zero to Production

Rosemary Wang

by Jeff Duffy, Database Specialist Solution Architect, AWS Database Week at the AWS Loft is an opportunity to learn about Amazon’s broad and deep family of managed database services. These services provide easy, scalable, reliable, and cost-effective ways to manage your data in the cloud. We explain the fundamentals and take a technical deep dive into Amazon RDS and Amazon Aurora relational databases, Amazon DynamoDB non-relational databases, Amazon Neptune graph databases, and Amazon ElastiCache managed Redis, along with options for database migration, caching, search and more. You'll will learn how to get started, how to support applications, and how to scale.

Hands-on Lab - Combaring Redis with Relational

Amazon Web Services

More from Antonio Sanz Alcober

Ciberamenazas - ¿A qué nos enfrentamos?

Antonio Sanz Alcober

China lleva más de una década incrementando su presencia en Internet, siendo una de las potencias más activas en ciberataques y estando bajo sospecha de algunas de las APT más sonadas. Pero para conocer los objetivos de China es necesario realizar un análisis global, que más allá de la tecnología tenga en cuenta su particular entorno histórico, social y político. El propósito de la charla es conformar la visión que China tiene como potencia mundial y su aplicación en el dominio del “quinto espacio”. Gracias a este análisis podremos analizar correctamente el pasado de los ciberataques, sabremos detectar mejor los ataques presentes y estaremos mejor preparados para futuros vectores de ataque.

China y el ciberespionaje: Sun Tzu, APT1 y los tiempos interesantes.

Antonio Sanz Alcober

Cyberthreats: Are we screwed ?

Antonio Sanz Alcober

SEA - Syrian Electronic Army - Cyberterrorism

Antonio Sanz Alcober

System administration is tough. HPC system administration is tougher. Most of the time you have to juggle computing, storage, networks, software and code in order to get the performance required (of course with 100% availability & security). Do you have an screwdriver and some chew gum?. Technology is only a leg of the tripod. You have to deal with users (who always want more), bosses (who always want to pay less) and consultants (who always want your money). People have to be sheph...sorry, managed properly. And don't forget that all the big changes that you make to your HPC infrastructure can (and should) be treated as projects. Although project management can be hell, well managed can be also the key to heaven (or at least to some inner zen-like peace). I've been managing a midsized HPC cluster for more than 10 years, and I'd like to give back some tips and tricks learned (most of the time, by trial/error or utmost failure) to make this challenging task lighter. The tips will be 50%/50% split between technology and management, and black humour will be all around.

Sex lies and queues: Confessions of an HPC sysadmin

Antonio Sanz Alcober

Cómo comprar seguro en internet

Antonio Sanz Alcober

Apple compra Microsoft : Respuesta ante incidentes de seguridad en redes soci...

Antonio Sanz Alcober

Harry el Sucio te enseña Seguridad en Wordpress

Antonio Sanz Alcober

Sex, lies and magnetic tapes: Confessions of a sysadmin

Antonio Sanz Alcober

Supercomputación: Haciendo HOY la ciencia del mañana

Antonio Sanz Alcober

Historia de la inteligencia china

Antonio Sanz Alcober

Análisis de la capacidad científica, tecnológica e innovadora de la república...

Antonio Sanz Alcober

Seguridad en redes sociales

Antonio Sanz Alcober

Cloud computing security

Antonio Sanz Alcober

Construyendo un cluster de supercomputación - Guía paso a paso

Antonio Sanz Alcober

More from Antonio Sanz Alcober (15)

Ciberamenazas - ¿A qué nos enfrentamos?

China y el ciberespionaje: Sun Tzu, APT1 y los tiempos interesantes.

Cyberthreats: Are we screwed ?

SEA - Syrian Electronic Army - Cyberterrorism

Sex lies and queues: Confessions of an HPC sysadmin

Cómo comprar seguro en internet

Apple compra Microsoft : Respuesta ante incidentes de seguridad en redes soci...

Harry el Sucio te enseña Seguridad en Wordpress

Sex, lies and magnetic tapes: Confessions of a sysadmin

Supercomputación: Haciendo HOY la ciencia del mañana

Historia de la inteligencia china

Análisis de la capacidad científica, tecnológica e innovadora de la república...

Seguridad en redes sociales

Cloud computing security

Construyendo un cluster de supercomputación - Guía paso a paso

Recently uploaded

Heather Hedden, Senior Consultant at Enterprise Knowledge, presented “The Role of Taxonomy and Ontology in Semantic Layers” at a webinar hosted by Progress Semaphore on April 16, 2024. Taxonomies at their core enable effective tagging and retrieval of content, and combined with ontologies they extend to the management and understanding of related data. There are even greater benefits of taxonomies and ontologies to enhance your enterprise information architecture when applying them to a semantic layer. A survey by DBP-Institute found that enterprises using a semantic layer see their business outcomes improve by four times, while reducing their data and analytics costs. Extending taxonomies to a semantic layer can be a game-changing solution, allowing you to connect information silos, alleviate knowledge gaps, and derive new insights. Hedden, who specializes in taxonomy design and implementation, presented how the value of taxonomies shouldn’t reside in silos but be integrated with ontologies into a semantic layer. Learn about: - The essence and purpose of taxonomies and ontologies in information and knowledge management; - Advantages of semantic layers leveraging organizational taxonomies; and - Components and approaches to creating a semantic layer, including the integration of taxonomies and ontologies

The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf

Enterprise Knowledge

Automating Google Workspace (GWS) & more with Apps Script

wesley chun

In this session, we will delve into strategic approaches for optimizing knowledge management within Microsoft 365, amidst the evolving landscape of Copilot. From leveraging automatic metadata classification and permission governance with SharePoint Premium, to unlocking Viva Engage for the cultivation of knowledge and communities, you will gain actionable insights to bolster your organization's knowledge-sharing initiatives. In this session, we will also explore how to facilitate solutions to enable your employees to find answers and expertise within Microsoft 365. You will leave equipped with practical techniques and a deeper understanding of how there is more to effective knowledge management than just enabling Copilot, but building actual solutions to prepare the knowledge that Copilot and your employees can use.

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

Drew Madelung

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Neo4j

Imagine a world where information flows as swiftly as thought itself, making decision-making as fluid as the data driving it. Every moment is critical, and the right tools can significantly boost your organization’s performance. The power of real-time data automation through FME can turn this vision into reality. Aimed at professionals eager to leverage real-time data for enhanced decision-making and efficiency, this webinar will cover the essentials of real-time data and its significance. We’ll explore: FME’s role in real-time event processing, from data intake and analysis to transformation and reporting An overview of leveraging streams vs. automations FME’s impact across various industries highlighted by real-life case studies Live demonstrations on setting up FME workflows for real-time data Practical advice on getting started, best practices, and tips for effective implementation Join us to enhance your skills in real-time data automation with FME, and take your operational capabilities to the next level.

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

Safe Software

GenAI Risks & Security Meetup 01052024.pdf

lior mazor

Handwritten Text Recognition for manuscripts and early printed texts

Maria Levchenko

MySQL Webinar, presented on the 25th of April, 2024. Summary: MySQL solutions enable the deployment of diverse Database Architectures tailored to specific needs, including High Availability, Disaster Recovery, and Read Scale-Out. With MySQL Shell's AdminAPI, administrators can seamlessly set up, manage, and monitor these solutions, ensuring efficiency and ease of use in their administration. MySQL Router, on the other hand, provides transparent routing from the application traffic to the backend servers in the architectures, requiring minimal configuration. Completely built in-house and supported by Oracle, these solutions have been adopted by enterprises of all sizes for their business-critical applications. In this presentation, we'll delve into various database architecture solutions to help you choose the right one based on your business requirements. Focusing on technical details and the latest features to maximize the potential of these solutions.

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

Miguel Araújo

With more memory available, system performance of three Dell devices increased, which can translate to a better user experience Conclusion When your system has plenty of RAM to meet your needs, you can efficiently access the applications and data you need to finish projects and to-do lists without sacrificing time and focus. Our test results show that with more memory available, three Dell PCs delivered better performance and took less time to complete the Procyon Office Productivity benchmark. These advantages translate to users being able to complete workflows more quickly and multitask more easily. Whether you need the mobility of the Latitude 5440, the creative capabilities of the Precision 3470, or the high performance of the OptiPlex Tower Plus 7010, configuring your system with more RAM can help keep processes running smoothly, enabling you to do more without compromising performance.

Boost PC performance: How more available memory can improve productivity

Principled Technologies

Evaluating the top large language models.pdf

ChristopherTHyatt

Presentation on how to chat with PDF using ChatGPT code interpreter

naman860154

Boost Fertility New Invention Ups Success Rates.pdf

sudhanshuwaghmare1

How to Troubleshoot Apps for the Modern Connected Worker

ThousandEyes

As privacy and data protection regulations evolve rapidly, organizations operating in multiple jurisdictions face mounting challenges to ensure compliance and safeguard customer data. With state-specific privacy laws coming up in multiple states this year, it is essential to understand what their unique data protection regulations will require clearly. How will data privacy evolve in the US in 2024? How to stay compliant? Our panellists will guide you through the intricacies of these states' specific data privacy laws, clarifying complex legal frameworks and compliance requirements. This webinar will review: - The essential aspects of each state's privacy landscape and the latest updates - Common compliance challenges faced by organizations operating in multiple states and best practices to achieve regulatory adherence - Valuable insights into potential changes to existing regulations and prepare your organization for the evolving landscape

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

TrustArc

Three things you will take away from the session: • How to run an effective tenant-to-tenant migration • Best practices for before, during, and after migration • Tips for using migration as a springboard to prepare for Copilot in Microsoft 365 Main ideas: Migration Overview: The presentation covers the current reality of cross-tenant migrations, the triggers, phases, best practices, and benefits of a successful tenant migration Considerations: When considering a migration, it is important to consider the migration scope, performance, customization, flexibility, user-friendly interface, automation, monitoring, support, training, scalability, data integrity, data security, cost, and licensing structure Next Wave: The next wave of change includes the launch of Copilot, which requires businesses to be prepared for upcoming changes related to Copilot and the cloud, and to consolidate data and tighten governance ShareGate: ShareGate can help with pre-migration analysis, configurable migration tool, and automated, end-user driven collaborative governance

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

sammart93

2024: Domino Containers - The Next Step. News from the Domino Container commu...

Martijn de Jong

Histor y of HAM Radio presentation slide

vu2urc

Abhishek Deb(1), Mr Abdul Kalam(2) M. Des (UX) , School of Design, DIT University , Dehradun. This paper explores the future potential of AI-enabled smartphone processors, aiming to investigate the advancements, capabilities, and implications of integrating artificial intelligence (AI) into smartphone technology. The research study goals consist of evaluating the development of AI in mobile phone processors, analyzing the existing state as well as abilities of AI-enabled cpus determining future patterns as well as chances together with reviewing obstacles as well as factors to consider for more growth.

Exploring the Future Potential of AI-Enabled Smartphone Processors

debabhi2

[2024]Digital Global Overview Report 2024 Meltwater.pdf

hans926745

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

The Digital Insurer

Recently uploaded (20)

The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf

Automating Google Workspace (GWS) & more with Apps Script

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

GenAI Risks & Security Meetup 01052024.pdf

Handwritten Text Recognition for manuscripts and early printed texts

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

Boost PC performance: How more available memory can improve productivity

Evaluating the top large language models.pdf

Presentation on how to chat with PDF using ChatGPT code interpreter

Boost Fertility New Invention Ups Success Rates.pdf

How to Troubleshoot Apps for the Modern Connected Worker

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

2024: Domino Containers - The Next Step. News from the Domino Container commu...

Histor y of HAM Radio presentation slide

Exploring the Future Potential of AI-Enabled Smartphone Processors

[2024]Digital Global Overview Report 2024 Meltwater.pdf

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

Solving BIG problems with Open Source: Condor

1. > Solving Big problems with OS: Condor > Antonio Sanz (ansanz@unizar.es) > 09 / Nov / 11

2. 2

3. 3 > Antonio Sanz > I3A System Manager > HERMES HPC cluster sysadmin > ansanz@unizar.es > @antoniosanzalc

4. 4

5. 5 Show / Know / Use

6. Problema inicial 6 > Dr. Good > Neurologist > Alzheimer research > Process 20000 brain image scans (1h/image) > A thousand times. Maybe two. 3. Sistemas de gestión de colas : Condor

7. Problema inicial 7 > Mrs. Nice > Santa’s Logistic Officer > Gift transportation > Analize 6x10e7 possible load/reindeers/routes (10min/analysis) > Before Christmas!

8. 8 Hey … ! It’s a 64K one !

9. 9 Queue distribution systems

10. Condor Basics 10 Single queue 3. Sistemas de gestión de colas : Condor

11. 11

12. Condor Basics 12 Multiple queues 3. Sistemas de gestión de colas : Condor

13. 13

14. 14 Problem partitioning

15. Problem can be broken into independent pieces 15

16. Condor Basics 16 Oh Yeah!

17. Condor Basics 17 For loops are your best friends 3. Sistemas de gestión de colas : Condor

18. Condor Basics 18 While loops … can sometimes be convinced 3. Sistemas de gestión de colas : Condor

19. Condor Basics 19 Do it yourself ! 3. Sistemas de gestión de colas : Condor

20. Condor Basics 20 >

21. Condor Basics 21 Heterogeneous computing

22. 22 Resource harvesting

23. 23 Requirements

24. 24 Job Surveillance

25. Condor Basics 25 Fair use of resources 3. Sistemas de gestión de colas : Condor

26. 26 Checkpoints

27. Condor Basics 27 Nested jobs (DAG)

28. Condor Basics 28 Email Notifications

29. 29 Grid & Cloud Computing

30. Condor Basics 30 Flexibility

31. 31 … with Hadoop, MPI, OpenMP, GPU

32. Condor Basics 32 3. Sistemas de gestión de colas : Condor

33. 33 How Condor works

34. 34 Management [Hello, Dave]

35. 35 Compute

36. Condor Basics 36 Job list ClassAd 3. Sistemas de gestión de colas : Condor

37. 37 Resource list ClassAd

38. 38 Matchmaking

39. Condor Basics 39 Priority Management

40. 40 Data Transfer

41. Condor Basics 41 Job running 3. Sistemas de gestión de colas : Condor

42. 42 Job Monitoring

43. 43 Job End

44. 44 Example

45. 45 #!/bin/sh # I’m hola.sh echo Hola mundo desde `hostname` # # A Hello World .. In Condor! # # I’m hello.sub Universe = vanilla Executable = hola.sh Log = hola.log Output = hola.out Error = hola.err Queue Hello, World !!

46. Lanzar el cálculo 46 condor_submit 4. Condor Basics – Un cálculo fácil

47. Lanzar el cálculo 47 condor_q

48. 48 Something tastier… #!/bin/sh # I’m hello2.sh OUTPUT=hello${1}.result cat hello.input >> $OUTPUT cat echo Hello world, I’m job $1 here from `hostname` > $OUTPUT # Execute n times with different outputs Universe = vanilla Executable = hello2.sh Transfer_input_files = hello.input WhenToTransferOutput = ON_EXIT_OR_EVICT Arguments = $(Process) Log = hello.log Output = hello.out Queue 10

49. 49 Perfect Simulation 4. Condor Basics – Un cálculo fácil

50. 50 Extra Bonus

51. 51 Dynamic Partitioning

52. Condor Basics 52 Configurable Jobs

53. 53 Advanced Accounting

54. 54 Dynamic Checkpointing

55. Condor Basics 55 Hadoop Integration 3. Sistemas de gestión de colas : Condor

56. 56 Green Computing

57. Condor Basics 57 GPU Integration

58. 58 I3A & Condor

59. 59 Gaming IA

60. 60 MRI Brain Analysis