This document discusses the CineGrid Exchange, which aims to build a globally distributed testbed for managing and preserving digital media assets. It involves geographically distributed storage nodes connected by fast networks. The CineGrid Exchange uses iRODS middleware to automate management of distributed assets across the network according to defined workflows. Phase 1 goals are to implement basic ingest, distribution, and deletion workflows using iRODS microservices and test high-speed file transfers between nodes.
Preservation Metadata Initiatives and StandardsKEEP_project
Presentation of Getaneh Alemu (UPHEC) at the seminar "The Digital Media Collection +100 Years" in Bristol (16/09/2009) organised by the Joint Information Systems Committee (JISC)
Introduction to the Open Grid Forum community and the document production process, as well as several primary application arenas for OGF specifications, given at the co-located International Conference on Cloud and Autonomic Computing (CAC 2014), IEEE International Conference on Self-Adaptive and Self-Organizing Systems (SASO 2014) and the IEEE International Conference on Peer-to-Peer Computing (P2P’14) conferences, September 8-12, 2014 at Imperial College in London, UK.
Invited talk on Open Grid Forum standards, focusing specifically on the current status of the Open Cloud Computing Interface (OCCI), given at the US National Institute of Standards and Technology Cloud Computing Forum and Workshop VIII, July 7-10, 2015.
Overview and introductory remarks for the OGF sessions held May 21-22, 2015 co-located with the European Grid Initiative 2015 conference that took place the week of May 18-22, 2015 in Lisbon, Portugal. For details, see https://www.ogf.org/ogf/doku.php/events/ogf-44
Cloud Testbeds for Standards Development and InnovationAlan Sill
Invited talk given at the 2014 Chip-to-Cloud Security Forum "Advances in Securing Embedded, Mobile and Cloud Services and Ecosystems" in the seminar session on "Procurement, SLAs, and Standardisation on a Global Scale." In this talk, Dr. Sill reviews the history of cloud and grid computing, the formation and charter description for Phases I and II of the US National Institute of Standards and Technology (NIST) "SAJACC" working group, and brings the discussion up to date with an overview of current "DevOps"-oriented cloud standards and software interoperability hands-on testing efforts worldwide.
Preservation Metadata Initiatives and StandardsKEEP_project
Presentation of Getaneh Alemu (UPHEC) at the seminar "The Digital Media Collection +100 Years" in Bristol (16/09/2009) organised by the Joint Information Systems Committee (JISC)
Introduction to the Open Grid Forum community and the document production process, as well as several primary application arenas for OGF specifications, given at the co-located International Conference on Cloud and Autonomic Computing (CAC 2014), IEEE International Conference on Self-Adaptive and Self-Organizing Systems (SASO 2014) and the IEEE International Conference on Peer-to-Peer Computing (P2P’14) conferences, September 8-12, 2014 at Imperial College in London, UK.
Invited talk on Open Grid Forum standards, focusing specifically on the current status of the Open Cloud Computing Interface (OCCI), given at the US National Institute of Standards and Technology Cloud Computing Forum and Workshop VIII, July 7-10, 2015.
Overview and introductory remarks for the OGF sessions held May 21-22, 2015 co-located with the European Grid Initiative 2015 conference that took place the week of May 18-22, 2015 in Lisbon, Portugal. For details, see https://www.ogf.org/ogf/doku.php/events/ogf-44
Cloud Testbeds for Standards Development and InnovationAlan Sill
Invited talk given at the 2014 Chip-to-Cloud Security Forum "Advances in Securing Embedded, Mobile and Cloud Services and Ecosystems" in the seminar session on "Procurement, SLAs, and Standardisation on a Global Scale." In this talk, Dr. Sill reviews the history of cloud and grid computing, the formation and charter description for Phases I and II of the US National Institute of Standards and Technology (NIST) "SAJACC" working group, and brings the discussion up to date with an overview of current "DevOps"-oriented cloud standards and software interoperability hands-on testing efforts worldwide.
In networked publics, power based on network positionality replaces media power. How can we design networked publics to improve parity? I present @TheTweetserve, a prototype solution.
Order to Cash Project ( Time to Invoice), TTI for a leading oil and gas company based in Libya. Diagnostics, current state presentation, used to drive the workshop, to drive team toward
solution.
In networked publics, power based on network positionality replaces media power. How can we design networked publics to improve parity? I present @TheTweetserve, a prototype solution.
Order to Cash Project ( Time to Invoice), TTI for a leading oil and gas company based in Libya. Diagnostics, current state presentation, used to drive the workshop, to drive team toward
solution.
The BlueBRIDGE approach to collaborative researchBlue BRIDGE
Gianpaolo Coro, ISTI-CNR, at BlueBRIDGE workshop on "Data Management services to support stock assessement", held during the Annual ICES Science conference 2016
Federating Infrastructure as a Service cloud computing systems to create a un...David Wallom
This paper details the state of the art, the design, development and deployment of the EGI Federated Cloud platform, an e-infrastructure offering scalable and flexible models of utilization to the European research community. While continuing support for the traditional High Throughput Computing model, the EGI Cloud Platform extends its reach to other models of utilization such as long-lived services and on demand computation. Following a two-year period of development, the EGI Federated Cloud platform was officially launched in May 2014 offering resources provided by trusted academic and research organisations from within the user communities and consistently with their standard funding regime. Since then, the use cases supported have significantly increased both in total number and diversity of model of service required, validating both the choice of enforcing cloud technology agnosticism and of supporting service mobility and portability by means of open standards. These design choices have also allowed for the inclusion of commercial cloud providers into an infrastructure previously supported only by academic institutions. This contributes to a wider goal of funding agencies to create economic and social impact from supported research activities.
Between 5-7 September 2022, in Salamanca was held the ON-SITE Conference - International Joint Conferences HAIS-SOCO-CISIS-ICEUTE & STARTUP OLÉ and our colleague Oliviu Matei has presented an article about the MUSHNOMICS Platform Architecture.
Cloud Standards in the Real World: Cloud Standards Testing for DevelopersAlan Sill
Learn about standards studied in the US National Science Foundation Cloud and Autonomic Computing Industry/University Cooperative Research Center Cloud Standards Testing Lab and how you can get involved to extend the successes from these results in your own cloud software settings. Presented at the O'Reilly OSCON 2014 Open Cloud Day.
Video available at https://www.youtube.com/watch?v=eD2h0SqC7tY
Future Internet: Managing Innovation and TestbedShinji Shimojo
Innovation is a big key word for ICT research and development. However, a road toward innovation is facing full of uncertainties and there are many obstacles. key elements to overcome these obstacles seems to be agile management of people, software and hardware. In addition, we think involvement of users in R&D will have much effect on the management of uncertainty in R&D. In this talk, I talk on our approach to this user involvement in JGN-X, an international future internet testbed and Knowledge Capital, Osaka, an smart city experimental testbed.
UK e-Infrastructure: Widening Access, Increasing ParticipationNeil Chue Hong
A talk given at the ICHEC Annual Seminar by Neil Chue Hong, reflecting on the rise of Grid and Web 2.0, and how this might enable increased participation and use of computing infrastructure for e-Science and research.
Using a Widely Distributed Federated Cloud System to Support Multiple Dispara...inside-BigData.com
In this deck from the 2014 ISC Cloud Conference, David Wallom from the University of Oxford presents:
Using a Widely Distributed Federated Cloud System to Support Multiple Disparate User Communities.
"The EGI federated cloud, which has been in development for the past 3 years has now entered production. Building on the tried and trusted EGI core services we have added federated IaS compute and storage services, utilising open standards to support more than 10 pilot communities. We will discuss the model of federation, and the different application design models that the users use and why cloud will be a success when compared with grid due to this inherent flexibility."
Learn more: http://www.isc-events.com/cloud14/schedule.html
Watch the video presentation: http://wp.me/p3RLHQ-daY
Cloud Computing Needs for Earth Observation Data Analysis: EGI and EOSC-hubBjörn Backeberg
This presentation was given during the Japan Geosciences Union 2019. Session details can be found at http://www.jpgu.org/meeting_e2019/SessionList_en/detail/M-GI31.htm
Presentation from Networkshop46.
GÉANT, by Mian Usman, GÉANT.
Campus network refresh, by David Stockdale, Imperial College London.
Multicast QUIC for video content delivery, by Richard Bradbury, BBC Research & Development.
Montana State, Research Networking and the Outcomes from the First National R...Jerry Sheehan
Presentation at Educause 17 with our Partner Cisco on Research networking, covers our campus experience and the first National Research Platform Workshop findings
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
A tale of scale & speed: How the US Navy is enabling software delivery from l...sonjaschweigert1
Rapid and secure feature delivery is a goal across every application team and every branch of the DoD. The Navy’s DevSecOps platform, Party Barge, has achieved:
- Reduction in onboarding time from 5 weeks to 1 day
- Improved developer experience and productivity through actionable findings and reduction of false positives
- Maintenance of superior security standards and inherent policy enforcement with Authorization to Operate (ATO)
Development teams can ship efficiently and ensure applications are cyber ready for Navy Authorizing Officials (AOs). In this webinar, Sigma Defense and Anchore will give attendees a look behind the scenes and demo secure pipeline automation and security artifacts that speed up application ATO and time to production.
We will cover:
- How to remove silos in DevSecOps
- How to build efficient development pipeline roles and component templates
- How to deliver security artifacts that matter for ATO’s (SBOMs, vulnerability reports, and policy evidence)
- How to streamline operations with automated policy checks on container images
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
Le nuove frontiere dell'AI nell'RPA con UiPath Autopilot™UiPathCommunity
In questo evento online gratuito, organizzato dalla Community Italiana di UiPath, potrai esplorare le nuove funzionalità di Autopilot, il tool che integra l'Intelligenza Artificiale nei processi di sviluppo e utilizzo delle Automazioni.
📕 Vedremo insieme alcuni esempi dell'utilizzo di Autopilot in diversi tool della Suite UiPath:
Autopilot per Studio Web
Autopilot per Studio
Autopilot per Apps
Clipboard AI
GenAI applicata alla Document Understanding
👨🏫👨💻 Speakers:
Stefano Negro, UiPath MVPx3, RPA Tech Lead @ BSP Consultant
Flavio Martinelli, UiPath MVP 2023, Technical Account Manager @UiPath
Andrei Tasca, RPA Solutions Team Lead @NTT Data
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
PHP Frameworks: I want to break free (IPC Berlin 2024)Ralf Eggert
In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development.
This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Ramesh Iyer
In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
Cine grid exchange@cenic2010-5
1. CineGrid ExchangeBuilding A Global Networked Testbed for Distributed Media Management and Preservation Shaofeng Liu, UCSD/CSE Jurgen Schulze, UCSD/Calit2 Laurin Herr, Pacific Interface CENIC March 9, 2010
2. What is CineGrid? Formed 2004 – non-profit international membership organization Mission – to build an interdisciplinary community Focus – the research, development & demonstration of networked collaborative tools To enable – the production, use, preservation & exchange of very high-quality digital media over high-speed photonic networks Members – media arts schools, research universities, scientific labs, post-production facilities & hardware and software developers, network operators from around the world (~50 members worldwide) Connected via 1 G Ethernet & 10 G Ethernet networks For – research & education
3. “Learning by Doing” Early CineGrid Projects CineGrid @ iGrid 2005 CineGrid @ AES 2006 CineGrid @ GLIF 2007 CineGrid @ Holland Festival 2007
4. CineGrid ExchangeMotivations TERABYTES PILING UP. Need to store & distribute own collection of digital media assets. Members access materials for experiments and demonstrations. THE DIGITAL DILEMMA. Report published by AMPAS – lessons learned by NDIPP and NARA + pioneering research at Stanford (LOCKSS) & UCSD (SRB and iRODS) Create global-scale testbed = high quality media assets + distributed storage + fast networks. Explore strategic issues in digital media storage, access, distribution and preservation
5. CineGrid Exchange 2009Build on the Basic Concept Receive “seed” funding from AMPAS to start building multi-layer open source asset management & user access framework for distributed digital media repository Establish Working Group: AMPAS, PII, Ryerson, UCSD, UIC, Naval Postgraduate School, UW, UvA, Keio, NTT, CESNET Write CineGrid Exchange functional requirements document Expand CineGrid Exchange storage capacity
6. CineGrid Exchange 2009Phase 1 Development Goals Define basic workflows for ingest, distribution, & deletion Define CineGrid Exchange metadata schema Implement basic workflows using iRODS rules-based microservices to automate the management of CineGrid Exchange assets Deploy iRODS to all CineGrid Exchange nodes Enable large amounts of data transfers between nodes Prepare for integration of CollectiveAccess cataloging application atop iRODS-managed distributed repository
11. CineGrid Exchange MiddleWareiRODS (Integrated Rules Oriented Data Services) Programmable micro-services to implement management policies for automated operations across distributed storage repositories Transparent management of files in a distributed repository Centralized catalog (iCAT) File transfer using parallel TCP & UDP Suitable for experimenting with digital media preservation strategies
19. Register files into iRODS database For all copies of each file, set the right access permissions
20.
21.
22. CineGrid Exchange testing suggests checksums may be a potential bottleneck
23.
24. Acknowledgements Academy of Motion Picture Arts and Sciences, Science and Technology Council Naval Postgraduate School, MOVES Institute UCSD/Calit2 UIC/EVL University of Washington, Research Channel Ryerson University CESNET NTT Network Innovation Laboratory Keio University/DMC University of Amsterdam Pacific Interface DICE Team at UNC & SDSC
26. CineGrid Exchange DevelopmentMulti-layer Open Software Stack User Interface & Access CineGrid Exchange Access Portal For CineGrid by CineGrid Open Source Application for Collections Management & On-Line Access Collective AccessExtended by Whirl-I-Gig for AMPAS Open Source Digital Repository Interface iRODS by DICE For CineGrid By Calit2 Open Source Middleware for Rule-based Management of Distributed Digital Repository Resources & Asset Testbed Infrastructure of Distributed Storage and Network Links Resource Description Framework For GLIF and CineGrid by UvA
27. CineGrid ExchangeDistribution Workflow Upon request for content, the CX metadata management interface (CollectiveAccess) will be searched to identify assets for distribution (pending integration) CollectiveAccess will query the iRODS resource management module to confirm availability of asset within CX distributed repository iRODS module will identify the best location from which to fetch the asset, set parameters and start the transfer using either TCP or UDP Integrity of all fetched files confirmed by checksum comparison Distribution permissions associated with asset checked Member access authorization confirmed Assets distributed via network or HDD
Editor's Notes
These are the goals of the first phase CX design and development. Basically the primary goal is to integrate all the distributed storages provided by CineGrid members andProvide a transparent way to access it; to define the right way to manage and distribute digital contents; to design the metadata management scheme.
In 2008,Cinegrid Exchange started from three major sites in San Diego, Tokyo and Amsterdam. These three sites are connected by 10Gb networks.
And in 2009, we have expanded CX to include 7 major sites worldwide including San Dieog, Los Angeles, Chicago, Tokyo, Toronto, Prague, and Amsterdam. Still, the connections Between them are 10Gbps networks.
This table listed the current storage provided by those sites. There are about 250TB currently managed by Cinegrid Exchange.
We designed CX as a three level hierarchical structures. The lowest level is the physical distributed content repositories, connected by high speed optical networks. The middle ware mange the resources and provide support for applications. iRODS is the major middleware in CX, it is datagrid software that can manages distributed storages. It is also programmable through it’s rule system. CollectiveAccess is a metadata management software that we are working on. Our partners in AMPASAnd NPS are working on binding iRODS and CA together.
Here are some features of iRODS. It’s programmable, it’s transparent. It uses a centralized catalog to maintain file location, checksum, and some metadata. We integrated UDP into iRODS, so that it now support Very fast file transfer within CX IT forms the base of the test bed we want to build in CX. I have a short demo of how this works.
We have defined a few workflows in CX to start with. Here is an examples. I will give a small demo of the ingest workflow. I will put a file into the dropBox, and that file will be automatically replicated into two other storage system and then moved to a central points. This is our first phase implementation only, so if you are not surprised at the demo, I am not surprised either.
A distributed testbed like CX requires a good network, and good protocols. We integrated both TCP and UDP into CX. And performance is satisfactory. This figures shows the file transfer speed with TCP and UDP. The x-asix is the number of flows, and y-axis is the overall transfer speed. With 3 UDP streams, we can transfer files at a speed of 350MB/sec from Japan to US, and with TCP, we can achieve 250MB/s with 15 streams or so.
The performance of ingest/distribution are subjected to other factors, like checksum, we found with checksum performed with the transfer, the overall speed is significantly slowed down. We are investigating ways to solve these problems. For example, we can do delayed checksum, or do pipelined checksum, or do FEC coding. This will be solved.