Delivering and using Grid service with Nordugrid/ARC

•

2 likes•283 views

NorduGrid is a collaboration that develops the free grid middleware known as the Advanced Resource Connector (ARC). ARC started as version 0 but required a new architecture and design, leading to the birth of ARC 1. ARC 1 uses a common container called HED to simplify management and configuration of grid services like the A-REX resource execution service. It supports standards like GLUE 2 for resource description and JSDL for job submission and integrates with local resource managers from systems like PBS, SGE, and LSF. Advanced features include distributed storage, security delegation, and utilizing cloud resources while supporting monitoring, accounting and remote management.

Technology Business

Delivering and using
grid services with
NorduGrid/ARC

Ferenc Szalai
“one of the main developers of ARC1”

“Nordugrid is a Grid
Research and
Development
collaboration aiming at
development, maintenance and
support of the free Grid middleware,
known as the Advance Resource
Connector (ARC)”

Long history of production grid
operation and development

... and evening and morning were the sixth day
and finally ARC 1 was born.

Simple Secure

Elegant

Modular
Standard
Based

simplify the management and
configuration of the services

A-REX
(ARC Resource Coupled EXecution
Service)

GLUE 2 JSDL
(job submission description
(resource description) language)

WS-RP BES
(resource property) (basic execution service)

Condor
Fork

Load-Leveler

Local Resource Manager Support

PBS

SGE
LSF

Who cares about your jobs when
you are in the mountains?

Infinite
hierarchy

rescheduling

persistent
job queue

no file-store!

no listening
port

not required to
modify the
applications

rd
What can I do with my 3 party
storages?

Fine grained delegation using
policies in the credentials

Application specific portals and
workflow engines

KnowARC
http://www.knowarc.eu

Nordugrid
http://www.nordugrid.org

The world has changed and having one huge server won’t do the job anymore, when you’re talking about vast amounts of data, growing all the time the ability to Scale Out would be your saviour. This lecture will be about the basics of Apache Spark and distributed computing and the development tools needed to have a functional environment. Bio: Demi Ben-Ari, Sr. Data Engineer @Windward, Ofek Alumni Has over 9 years of experience in building various systems both from the field of near real time applications and Big Data distributed systems. Co-Founder of the “Big Things” Big Data community: http://somebigthings.com/big-things-i...

Spark 101

Shahaf Azriely {TopLinked} ☁

Spark 101 - First steps to distributed computing

Demi Ben-Ari

Apache Spark e AWS Glue

Laercio Serra

An overview of Apache Spark and AWS Glue. Apache Spark is a fast and general-purpose cluster computing system. It provides high-level APIs in Java, Scala, Python and R, and an optimized engine that supports general execution graphs. It also supports a rich set of higher-level tools including Spark SQL for SQL and structured data processing, MLlib for machine learning, GraphX for graph processing, and Spark Streaming. AWS Glue is a fully managed extract, transform, and load (ETL) service that makes it easy for customers to prepare and load their data for analytics. You can create and run an ETL job with a few clicks in the AWS Management Console. You simply point AWS Glue to your data stored on AWS, and AWS Glue discovers your data and stores the associated metadata (e.g. table definition and schema) in the AWS Glue Data Catalog. Once cataloged, your data is immediately searchable, queryable, and available for ETL.

Open Source Storage at Scale: Ceph @ GRNET

Nikos Kormpakis

Viewers also liked

Presentación Hinojosa del Duquemaviconv

Grid Underground projekt

Ferenc Szalai

reportKatie Morfoot

Oficio 021 - SMPE - 2012Diogo Morais

Condor és Globus együttműködési lehetőségei

Ferenc Szalai

Oficio 076 - SMPE - 2012Diogo Morais

Defining Immigration Law

steadfastimplem31

Softlayer Bluemix User Summit 2015 Keynote

Jesse Proudman

Scaffolding Autobiographical writingMagda EnriquezBeitler

Virtualizáció Linuxon: XEN

Ferenc Szalai

Grid és adattárolás

Ferenc Szalai

Miért vagyok Python rajongó? - avagy kalandozások egy nyílt forrású programoz...

Ferenc Szalai

Cockpit紹介

atk1234

Properties water

Shania Cotiangco

Cuadro Sinóptico - Cultura Digital

Yudy_Yohana

The past of be

Ingles Corporativo

Viewers also liked (16)

Presentación Hinojosa del Duque

Grid Underground projekt

report

Oficio 021 - SMPE - 2012

Condor és Globus együttműködési lehetőségei

Oficio 076 - SMPE - 2012

Defining Immigration Law

Softlayer Bluemix User Summit 2015 Keynote

Scaffolding Autobiographical writing

Virtualizáció Linuxon: XEN

Grid és adattárolás

Miért vagyok Python rajongó? - avagy kalandozások egy nyílt forrású programoz...

Cockpit紹介

Properties water

Cuadro Sinóptico - Cultura Digital

The past of be

Similar to Delivering and using Grid service with Nordugrid/ARC

SDN, OpenFlow, NFV, and Virtual Network

Tim4PreStartup

Spark 101 – First Steps To Distributed Computing - Demi Ben-Ari @ Ofek Alumni

Demi Ben-Ari

Spark 101

Shahaf Azriely {TopLinked} ☁

Spark 101 - First steps to distributed computing

Demi Ben-Ari

Apache Spark e AWS Glue

Laercio Serra

Open Source Storage at Scale: Ceph @ GRNET

Nikos Kormpakis

Fast Data Analytics with Spark and Python

Benjamin Bengfort

In this one day workshop, we will introduce Spark at a high level context. Spark is fundamentally different than writing MapReduce jobs so no prior Hadoop experience is needed. You will learn how to interact with Spark on the command line and conduct rapid in-memory data analyses. We will then work on writing Spark applications to perform large cluster-based analyses including SQL-like aggregations, machine learning applications, and graph algorithms. The course will be conducted in Python using PySpark.

Bring the Spark To Your Eyes

Demi Ben-Ari

Accelerating TensorFlow with RDMA for high-performance deep learning

DataWorks Summit

Google’s TensorFlow is one of the most popular deep learning (DL) frameworks. In distributed TensorFlow, gradient updates are a critical step governing the total model training time. These updates incur a massive volume of data transfer over the network. In this talk, we first present a thorough analysis of the communication patterns in distributed TensorFlow. Then we propose a unified way of achieving high performance through enhancing the gRPC runtime with Remote Direct Memory Access (RDMA) technology on InfiniBand and RoCE. Through our proposed RDMA-gRPC design, TensorFlow only needs to run over the gRPC channel and gets the optimal performance. Our design includes advanced features such as message pipelining, message coalescing, zero-copy transmission, etc. The performance evaluations show that our proposed design can significantly speed up gRPC throughput by up to 1.5x compared to the default gRPC design. By integrating our RDMA-gRPC with TensorFlow, we are able to achieve up to 35% performance improvement for TensorFlow training with CNN models. Speakers Dhabaleswar K (DK) Panda, Professor and University Distinguished Scholar, The Ohio State University Xiaoyi Lu, Research Scientist, The Ohio State University

LOD2: State of Play WP6 - LOD2 Stack Architecture

LOD2 Creating Knowledge out of Interlinked Data

Accelerating Apache Hadoop through High-Performance Networking and I/O Techno...

DataWorks Summit/Hadoop Summit

Near Real time Indexing Kafka Messages to Apache Blur using Spark Streaming

Dibyendu Bhattacharya

Apache Spark 101 - Demi Ben-Ari - Panorays

Demi Ben-Ari

Demi Ben Ari - Apache Spark 101 - First Steps into distributed computing: The world has changed, having one huge server won’t do the job, the ability to Scale Out would be your savior. Apache Spark is a fast and general engine for big data processing, with streaming, SQL, machine learning and graph processing. Showing the basics of Apache Spark and distributed computing. Demi is a Software engineer, Entrepreneur and an International Tech Speaker. Demi has over 10 years of experience in building various systems both from the field of near real time applications and Big Data distributed systems. Co-Founder of the “Big Things” Big Data community and Google Developer Group Cloud. Big Data Expert, but interested in all kinds of technologies, from front-end to backend, whatever moves data around.

Realizing the promise of portable data processing with Apache Beam

DataWorks Summit

The world of big data involves an ever changing field of players. Much as SQL stands as a lingua franca for declarative data analysis, Apache Beam aims to provide a portable standard for expressing robust, out-of-order data processing pipelines in a variety of languages across a variety of platforms. In a way, Apache Beam is a glue that can connect the Big Data ecosystem together; it enables users to "run-anything-anywhere". This talk will briefly cover the capabilities of the Beam model for data processing, as well as the current state of the Beam ecosystem. We'll discuss Beam architecture and dive into the portability layer. We'll offer a technical analysis of the Beam's powerful primitive operations that enable true and reliable portability across diverse environments. Finally, we'll demonstrate a complex pipeline running on multiple runners in multiple deployment scenarios (e.g. Apache Spark on Amazon Web Services, Apache Flink on Google Cloud, Apache Apex on-premise), and give a glimpse at some of the challenges Beam aims to address in the future.

Apache Spark Overview

Dharmjit Singh

Spark to Production @Windward

Demi Ben-Ari

What you need to know about ceph

Emma Haruka Iwao

2014 carlos gzlez florido nksip the erlang sip application server

VOIP2DAY

G rpc talk with intel (3)

Intel

Google and Intel speak on NFV and SFC service delivery The slides are as presented at the meet up "Out of Box Network Developers" sponsored by Intel Networking Developer Zone Here is the Agenda of the slides: How DPDK, RDT and gRPC fit into SDI/SDN, NFV and OpenStack Key Platform Requirements for SDI SDI Platform Ingredients: DPDK, IntelⓇRDT gRPC Service Framework IntelⓇ RDT and gRPC service framework

Scaling Spark Workloads on YARN - Boulder/Denver July 2015

Mac Moore

Similar to Delivering and using Grid service with Nordugrid/ARC (20)

SDN, OpenFlow, NFV, and Virtual Network

Spark 101 – First Steps To Distributed Computing - Demi Ben-Ari @ Ofek Alumni

Spark 101

Spark 101 - First steps to distributed computing

Apache Spark e AWS Glue

Open Source Storage at Scale: Ceph @ GRNET

Fast Data Analytics with Spark and Python

Bring the Spark To Your Eyes

Accelerating TensorFlow with RDMA for high-performance deep learning

LOD2: State of Play WP6 - LOD2 Stack Architecture

Accelerating Apache Hadoop through High-Performance Networking and I/O Techno...

Near Real time Indexing Kafka Messages to Apache Blur using Spark Streaming

Apache Spark 101 - Demi Ben-Ari - Panorays

Realizing the promise of portable data processing with Apache Beam

Apache Spark Overview

Spark to Production @Windward

What you need to know about ceph

2014 carlos gzlez florido nksip the erlang sip application server

G rpc talk with intel (3)

Scaling Spark Workloads on YARN - Boulder/Denver July 2015

Recently uploaded

Enhancing Performance with Globus and the Science DMZ

Globus

Essentials of Automations: The Art of Triggers and Actions in FME

Safe Software

In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation. We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios. Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!

Assure Contact Center Experiences for Your Customers With ThousandEyes

ThousandEyes

Leading Change strategies and insights for effective change management pdf 1.pdf

OnBoard

Generative AI Deep Dive: Advancing from Proof of Concept to Production

Aggregage

UiPath Community Day Dubai: AI at Work..

UiPathCommunity

Welcome to the first live UiPath Community Day Dubai! Join us for this unique occasion to meet our local and global UiPath Community and leaders. You will get a full view of the MEA region's automation landscape and the AI Powered automation technology capabilities of UiPath. Also, hosted by our local partners Marc Ellis, you will enjoy a half-day packed with industry insights and automation peers networking. 📕 Curious on our agenda? Wait no more! 10:00 Welcome note - UiPath Community in Dubai Lovely Sinha, UiPath Community Chapter Leader, UiPath MVPx3, Hyper-automation Consultant, First Abu Dhabi Bank 10:20 A UiPath cross-region MEA overview Ashraf El Zarka, VP and Managing Director MEA, UiPath 10:35: Customer Success Journey Deepthi Deepak, Head of Intelligent Automation CoE, First Abu Dhabi Bank 11:15 The UiPath approach to GenAI with our three principles: improve accuracy, supercharge productivity, and automate more Boris Krumrey, Global VP, Automation Innovation, UiPath 12:15 To discover how Marc Ellis leverages tech-driven solutions in recruitment and managed services. Brendan Lingam, Director of Sales and Business Development, Marc Ellis

GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...

Sri Ambati

GraphRAG is All You need? LLM & Knowledge Graph

Guy Korland

Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs. 1. Unifying Large Language Models and Knowledge Graphs: A Roadmap. https://arxiv.org/abs/2306.08302 2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs: https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/

Monitoring Java Application Security with JDK Tools and JFR Events

Ana-Maria Mihalceanu

State of ICS and IoT Cyber Threat Landscape Report 2024 preview

Prayukth K V

The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development. The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers: State of global ICS asset and network exposure Sectoral targets and attacks as well as the cost of ransom Global APT activity, AI usage, actor and tactic profiles, and implications Rise in volumes of AI-powered cyberattacks Major cyber events in 2024 Malware and malicious payload trends Cyberattack types and targets Vulnerability exploit attempts on CVEs Attacks on counties – USA Expansion of bot farms – how, where, and why In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East Why are attacks on smart factories rising? Cyber risk predictions Axis of attacks – Europe Systemic attacks in the Middle East Download the full report from here: https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/

PCI PIN Basics Webinar from the Controlcase Team

ControlCase

SAP Sapphire 2024 - ASUG301 building better apps with SAP Fiori.pdf

Peter Spielvogel

Building better applications for business users with SAP Fiori. • What is SAP Fiori and why it matters to you • How a better user experience drives measurable business benefits • How to get started with SAP Fiori today • How SAP Fiori elements accelerates application development • How SAP Build Code includes SAP Fiori tools and other generative artificial intelligence capabilities • How SAP Fiori paves the way for using AI in SAP apps

DevOps and Testing slides at DASA Connect

Kari Kakkonen

By Design, not by Accident - Agile Venture Bolzano 2024

Pierluigi Pugliese

The Metaverse and AI: how can decision-makers harness the Metaverse for their...

Jen Stirrup

The Metaverse is popularized in science fiction, and now it is becoming closer to being a part of our daily lives through the use of social media and shopping companies. How can businesses survive in a world where Artificial Intelligence is becoming the present as well as the future of technology, and how does the Metaverse fit into business strategy when futurist ideas are developing into reality at accelerated rates? How do we do this when our data isn't up to scratch? How can we move towards success with our data so we are set up for the Metaverse when it arrives? How can you help your company evolve, adapt, and succeed using Artificial Intelligence and the Metaverse to stay ahead of the competition? What are the potential issues, complications, and benefits that these technologies could bring to us and our organizations? In this session, Jen Stirrup will explain how to start thinking about these technologies as an organisation.

Video Streaming: Then, Now, and in the Future

Alpen-Adria-Universität

In his public lecture, Christian Timmerer provides insights into the fascinating history of video streaming, starting from its humble beginnings before YouTube to the groundbreaking technologies that now dominate platforms like Netflix and ORF ON. Timmerer also presents provocative contributions of his own that have significantly influenced the industry. He concludes by looking at future challenges and invites the audience to join in a discussion.

Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...

UiPathCommunity

💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™: See how to accelerate model training and optimize model performance with active learning Learn about the latest enhancements to out-of-the-box document processing – with little to no training required Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath. Speakers: 👨‍🏫 Andras Palfi, Senior Product Manager, UiPath 👩‍🏫 Lenka Dulovicova, Product Program Manager, UiPath

FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf

FIDO Alliance

Transcript: Selling digital books in 2024: Insights from industry leaders - T...

BookNet Canada

The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more. Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/ Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.

Removing Uninteresting Bytes in Software Fuzzing

Aftab Hussain

Imagine a world where software fuzzing, the process of mutating bytes in test seeds to uncover hidden and erroneous program behaviors, becomes faster and more effective. A lot depends on the initial seeds, which can significantly dictate the trajectory of a fuzzing campaign, particularly in terms of how long it takes to uncover interesting behaviour in your code. We introduce DIAR, a technique designed to speedup fuzzing campaigns by pinpointing and eliminating those uninteresting bytes in the seeds. Picture this: instead of wasting valuable resources on meaningless mutations in large, bloated seeds, DIAR removes the unnecessary bytes, streamlining the entire process. In this work, we equipped AFL, a popular fuzzer, with DIAR and examined two critical Linux libraries -- Libxml's xmllint, a tool for parsing xml documents, and Binutil's readelf, an essential debugging and security analysis command-line tool used to display detailed information about ELF (Executable and Linkable Format). Our preliminary results show that AFL+DIAR does not only discover new paths more quickly but also achieves higher coverage overall. This work thus showcases how starting with lean and optimized seeds can lead to faster, more comprehensive fuzzing campaigns -- and DIAR helps you find such seeds. - These are slides of the talk given at IEEE International Conference on Software Testing Verification and Validation Workshop, ICSTW 2022.

Recently uploaded (20)

Enhancing Performance with Globus and the Science DMZ

Essentials of Automations: The Art of Triggers and Actions in FME

Assure Contact Center Experiences for Your Customers With ThousandEyes

Leading Change strategies and insights for effective change management pdf 1.pdf

Generative AI Deep Dive: Advancing from Proof of Concept to Production

UiPath Community Day Dubai: AI at Work..

GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...

GraphRAG is All You need? LLM & Knowledge Graph

Monitoring Java Application Security with JDK Tools and JFR Events

State of ICS and IoT Cyber Threat Landscape Report 2024 preview

PCI PIN Basics Webinar from the Controlcase Team

SAP Sapphire 2024 - ASUG301 building better apps with SAP Fiori.pdf

DevOps and Testing slides at DASA Connect

By Design, not by Accident - Agile Venture Bolzano 2024

The Metaverse and AI: how can decision-makers harness the Metaverse for their...

Video Streaming: Then, Now, and in the Future

Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...

FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf

Transcript: Selling digital books in 2024: Insights from industry leaders - T...

Removing Uninteresting Bytes in Software Fuzzing

Delivering and using Grid service with Nordugrid/ARC

1. Delivering and using grid services with NorduGrid/ARC

2. Ferenc Szalai “one of the main developers of ARC1”

3. “Nordugrid is a Grid Research and Development collaboration aiming at development, maintenance and support of the free Grid middleware, known as the Advance Resource Connector (ARC)”

4. “What are you doing?”

5. “We are developing CLEVER services”

6. Since 2002

7. Long history of production grid operation and development

8. In the beginning there was ARC 0

10. Limitations

11. new architecture and design required

12. ... and evening and morning were the sixth day and finally ARC 1 was born.

13. Simple Secure Elegant Modular Standard Based

14. Common container for ALL services: HED

15.

16. simplify the management and configuration of the services