Big Lab Problems Solved with Spectrum Scale: Innovations for the Coral Programinside-BigData.com
In this video from the DDN User Group at SC16, Sven Oehme Chief Research Strategist, IBM, presents "Big Lab Problems Solved with Spectrum Scale: Innovations for the Coral Program."
Watch the video presentation: http://wp.me/p3RLHQ-g52
Sign up for our insideHPC Newsletter: http://wp.me/p3RLHQ-g52
Naprostá bezpečnost vašich dat díky jednoduchému, škálovatelnému, flexibilním...MarketingArrowECS_CZ
Petr Vančák, Senior Presales Consultant Symantec, Arrow ECS
Virtualization Forum 2014, Prague, 22.10.2014
Jestliže SlideShare nezobrazí prezentaci korektně, můžete si ji stáhnout ve formátu .ppsx nebo .pdf (kliknutím na tlačitko v dolní liště snímků).
Big Lab Problems Solved with Spectrum Scale: Innovations for the Coral Programinside-BigData.com
In this video from the DDN User Group at SC16, Sven Oehme Chief Research Strategist, IBM, presents "Big Lab Problems Solved with Spectrum Scale: Innovations for the Coral Program."
Watch the video presentation: http://wp.me/p3RLHQ-g52
Sign up for our insideHPC Newsletter: http://wp.me/p3RLHQ-g52
Naprostá bezpečnost vašich dat díky jednoduchému, škálovatelnému, flexibilním...MarketingArrowECS_CZ
Petr Vančák, Senior Presales Consultant Symantec, Arrow ECS
Virtualization Forum 2014, Prague, 22.10.2014
Jestliže SlideShare nezobrazí prezentaci korektně, můžete si ji stáhnout ve formátu .ppsx nebo .pdf (kliknutím na tlačitko v dolní liště snímků).
Google and Intel speak on NFV and SFC service delivery
The slides are as presented at the meet up "Out of Box Network Developers" sponsored by Intel Networking Developer Zone
Here is the Agenda of the slides:
How DPDK, RDT and gRPC fit into SDI/SDN, NFV and OpenStack
Key Platform Requirements for SDI
SDI Platform Ingredients: DPDK, IntelⓇRDT
gRPC Service Framework
IntelⓇ RDT and gRPC service framework
Hitachi Unified Storage and Hitachi NAS Platform 4000 Series -- DatasheetHitachi Vantara
HUS and HNAS 4000 series product overview, key features and technical specifications. For more information on Hitachi Unified Storage and Hitachi NAS Platform 4000 Series please visit: http://www.hds.com/products/file-and-content/network-attached-storage/?WT.ac=us_mg_pro_hnasp
2018 Infortrend EonStor GSe Pro Family Introductioninfortrendgroup
EonStor GSe Pro is a single controller solution with support for SATA and SSD drives exclusively designed for SMBs, integrating SAN and NAS services as well as Cloud Gateway features to be deployed under a general NAS architecture for file sharing, backup, cloud synchronization, video editing, and surveillance purposes. Its complete product line features a hardware design of rackmount or desktop enclosures and flexible host boards to choose from; as for software, it comes with complete data services and simple, intuitive management interfaces, so that you can find the perfect storage device according to your performance or budget needs.
More Information:
https://www.infortrend.com/global/products/GSePro
Recommended Series:
https://www.infortrend.com/global/products/families/GSePro/pro3000
https://www.infortrend.com/global/products/families/GSePro/pro1000
https://www.infortrend.com/global/products/families/GSePro/Pro%20200
Dimension Data Cloud Business Unit - Solution OfferingRifaHaryadi
Dimension Data - Cloud Business Unit Solution Offering. This presentation will take you through Dimension Data Solution Offering and Roadmap to the Future of Cloud Computing. Dimension Data Cloud Computing Solution are fully control by Manage Cloud Platform - Dimension Data Propretiary Orchestration and Automation Tools
Hitachi Unified Storage and Hitachi NAS Platform, 4000 series, product overview, key features, business value description and technical specifications.
IBM Cloud Object Storage System, presented Oct 16, 2017 at IBM Systems Technical University in New Orleans, LA. This covers the object storage from IBM from the acquisition of Cleversafe, formerly known as DSnet product.
Summit 16: Deploying Virtualized Mobile Infrastructures on OpenstackOPNFV
Service Provider is evolving and competing with leaner Over the Top Providers (OTP) providers such as Google and Amazon to provide mobile services. Furture SP network has ot be agile, resilient and auto salable. SPs are leaning towards using COTS infra, open networking (OPNFV, ONOS) and VNF to run routers, switches, mobile gateways, firewall, NAT, DPI functions. Session covers design and deployment of virtualizing the mobile infra such as Virtual Evolved Packet Core, GiLAN and VoLTE as well as 5G core. We will also cover performance fine tuning using DPDK, SR-IOV etc. WE will present case study using Cisco (VNF Manager and NFVO), Redhat (NFVI), Openstack and block storage using CEPH technology. Participants will be able to understand complexities of mobile packet core, evolution NFV based solution and architecture framework for 5G mobile packet core.
Google and Intel speak on NFV and SFC service delivery
The slides are as presented at the meet up "Out of Box Network Developers" sponsored by Intel Networking Developer Zone
Here is the Agenda of the slides:
How DPDK, RDT and gRPC fit into SDI/SDN, NFV and OpenStack
Key Platform Requirements for SDI
SDI Platform Ingredients: DPDK, IntelⓇRDT
gRPC Service Framework
IntelⓇ RDT and gRPC service framework
Hitachi Unified Storage and Hitachi NAS Platform 4000 Series -- DatasheetHitachi Vantara
HUS and HNAS 4000 series product overview, key features and technical specifications. For more information on Hitachi Unified Storage and Hitachi NAS Platform 4000 Series please visit: http://www.hds.com/products/file-and-content/network-attached-storage/?WT.ac=us_mg_pro_hnasp
2018 Infortrend EonStor GSe Pro Family Introductioninfortrendgroup
EonStor GSe Pro is a single controller solution with support for SATA and SSD drives exclusively designed for SMBs, integrating SAN and NAS services as well as Cloud Gateway features to be deployed under a general NAS architecture for file sharing, backup, cloud synchronization, video editing, and surveillance purposes. Its complete product line features a hardware design of rackmount or desktop enclosures and flexible host boards to choose from; as for software, it comes with complete data services and simple, intuitive management interfaces, so that you can find the perfect storage device according to your performance or budget needs.
More Information:
https://www.infortrend.com/global/products/GSePro
Recommended Series:
https://www.infortrend.com/global/products/families/GSePro/pro3000
https://www.infortrend.com/global/products/families/GSePro/pro1000
https://www.infortrend.com/global/products/families/GSePro/Pro%20200
Dimension Data Cloud Business Unit - Solution OfferingRifaHaryadi
Dimension Data - Cloud Business Unit Solution Offering. This presentation will take you through Dimension Data Solution Offering and Roadmap to the Future of Cloud Computing. Dimension Data Cloud Computing Solution are fully control by Manage Cloud Platform - Dimension Data Propretiary Orchestration and Automation Tools
Hitachi Unified Storage and Hitachi NAS Platform, 4000 series, product overview, key features, business value description and technical specifications.
IBM Cloud Object Storage System, presented Oct 16, 2017 at IBM Systems Technical University in New Orleans, LA. This covers the object storage from IBM from the acquisition of Cleversafe, formerly known as DSnet product.
Summit 16: Deploying Virtualized Mobile Infrastructures on OpenstackOPNFV
Service Provider is evolving and competing with leaner Over the Top Providers (OTP) providers such as Google and Amazon to provide mobile services. Furture SP network has ot be agile, resilient and auto salable. SPs are leaning towards using COTS infra, open networking (OPNFV, ONOS) and VNF to run routers, switches, mobile gateways, firewall, NAT, DPI functions. Session covers design and deployment of virtualizing the mobile infra such as Virtual Evolved Packet Core, GiLAN and VoLTE as well as 5G core. We will also cover performance fine tuning using DPDK, SR-IOV etc. WE will present case study using Cisco (VNF Manager and NFVO), Redhat (NFVI), Openstack and block storage using CEPH technology. Participants will be able to understand complexities of mobile packet core, evolution NFV based solution and architecture framework for 5G mobile packet core.
VIOPS07: OSMと地理空間情報
The OpenStreetMap Foundation Japan
Georepublic Japan G.K.
関 治之
2012年7月20日(金曜日)10:00-18:00
VIOPS07 WORKSHOP
Virtualized Infrastructures Workshop [07]
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Essentials of Automations: The Art of Triggers and Actions in FMESafe Software
In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation.
We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios.
Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!
Pushing the limits of ePRTC: 100ns holdover for 100 daysAdtran
At WSTS 2024, Alon Stern explored the topic of parametric holdover and explained how recent research findings can be implemented in real-world PNT networks to achieve 100 nanoseconds of accuracy for up to 100 days.
Climate Impact of Software Testing at Nordic Testing DaysKari Kakkonen
My slides at Nordic Testing Days 6.6.2024
Climate impact / sustainability of software testing discussed on the talk. ICT and testing must carry their part of global responsibility to help with the climat warming. We can minimize the carbon footprint but we can also have a carbon handprint, a positive impact on the climate. Quality characteristics can be added with sustainability, and then measured continuously. Test environments can be used less, and in smaller scale and on demand. Test techniques can be used in optimizing or minimizing number of tests. Test automation can be used to speed up testing.
Welcome to the first live UiPath Community Day Dubai! Join us for this unique occasion to meet our local and global UiPath Community and leaders. You will get a full view of the MEA region's automation landscape and the AI Powered automation technology capabilities of UiPath. Also, hosted by our local partners Marc Ellis, you will enjoy a half-day packed with industry insights and automation peers networking.
📕 Curious on our agenda? Wait no more!
10:00 Welcome note - UiPath Community in Dubai
Lovely Sinha, UiPath Community Chapter Leader, UiPath MVPx3, Hyper-automation Consultant, First Abu Dhabi Bank
10:20 A UiPath cross-region MEA overview
Ashraf El Zarka, VP and Managing Director MEA, UiPath
10:35: Customer Success Journey
Deepthi Deepak, Head of Intelligent Automation CoE, First Abu Dhabi Bank
11:15 The UiPath approach to GenAI with our three principles: improve accuracy, supercharge productivity, and automate more
Boris Krumrey, Global VP, Automation Innovation, UiPath
12:15 To discover how Marc Ellis leverages tech-driven solutions in recruitment and managed services.
Brendan Lingam, Director of Sales and Business Development, Marc Ellis
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionAggregage
Join Maher Hanafi, VP of Engineering at Betterworks, in this new session where he'll share a practical framework to transform Gen AI prototypes into impactful products! He'll delve into the complexities of data collection and management, model selection and optimization, and ensuring security, scalability, and responsible use.
SAP Sapphire 2024 - ASUG301 building better apps with SAP Fiori.pdfPeter Spielvogel
Building better applications for business users with SAP Fiori.
• What is SAP Fiori and why it matters to you
• How a better user experience drives measurable business benefits
• How to get started with SAP Fiori today
• How SAP Fiori elements accelerates application development
• How SAP Build Code includes SAP Fiori tools and other generative artificial intelligence capabilities
• How SAP Fiori paves the way for using AI in SAP apps
The Metaverse and AI: how can decision-makers harness the Metaverse for their...Jen Stirrup
The Metaverse is popularized in science fiction, and now it is becoming closer to being a part of our daily lives through the use of social media and shopping companies. How can businesses survive in a world where Artificial Intelligence is becoming the present as well as the future of technology, and how does the Metaverse fit into business strategy when futurist ideas are developing into reality at accelerated rates? How do we do this when our data isn't up to scratch? How can we move towards success with our data so we are set up for the Metaverse when it arrives?
How can you help your company evolve, adapt, and succeed using Artificial Intelligence and the Metaverse to stay ahead of the competition? What are the potential issues, complications, and benefits that these technologies could bring to us and our organizations? In this session, Jen Stirrup will explain how to start thinking about these technologies as an organisation.
A tale of scale & speed: How the US Navy is enabling software delivery from l...sonjaschweigert1
Rapid and secure feature delivery is a goal across every application team and every branch of the DoD. The Navy’s DevSecOps platform, Party Barge, has achieved:
- Reduction in onboarding time from 5 weeks to 1 day
- Improved developer experience and productivity through actionable findings and reduction of false positives
- Maintenance of superior security standards and inherent policy enforcement with Authorization to Operate (ATO)
Development teams can ship efficiently and ensure applications are cyber ready for Navy Authorizing Officials (AOs). In this webinar, Sigma Defense and Anchore will give attendees a look behind the scenes and demo secure pipeline automation and security artifacts that speed up application ATO and time to production.
We will cover:
- How to remove silos in DevSecOps
- How to build efficient development pipeline roles and component templates
- How to deliver security artifacts that matter for ATO’s (SBOMs, vulnerability reports, and policy evidence)
- How to streamline operations with automated policy checks on container images
Elevating Tactical DDD Patterns Through Object CalisthenicsDorra BARTAGUIZ
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdfPaige Cruz
Monitoring and observability aren’t traditionally found in software curriculums and many of us cobble this knowledge together from whatever vendor or ecosystem we were first introduced to and whatever is a part of your current company’s observability stack.
While the dev and ops silo continues to crumble….many organizations still relegate monitoring & observability as the purview of ops, infra and SRE teams. This is a mistake - achieving a highly observable system requires collaboration up and down the stack.
I, a former op, would like to extend an invitation to all application developers to join the observability party will share these foundational concepts to build on:
2. 魂▪創▪通
22
IV
III
Introduction to the GLORY-FSII
Future RoadmapV
Highlights of the GLORY-FS
GLORY-FS use in Korea
About ETRII
Other ProjectsAppendix
3. 魂▪創▪通
33
I. About ETRI
Electronics and Telecommunications Research Institute (www.etri.re.kr)
• Government funded Research Institute (since 1976)
• Personnel status: Total 1,894 (Researcher/Technical Staff: 1,737)
• 8 Division (Bigdata SW, SW-SOC, Broadcasting & Communication, …)
• Project Status: 547 Projects/547,618 million won
(http://www.etri.re.kr)
2011 Transparent display
2010 4G LTE-advanced technology
2007 3.6Gbps 4th generation mobile communication system
2006 Wireless Home Network (UWB)
2005 Terrestrial DMB service
2004 WiBro
1999 IMT 2000(CDMA2000) STP system
1996 ATM Exchanging Machine
1995 Commercialization of CDMA
1991 TDX-10, TiCOM II
1989 256M DRAM
1988 8bit Educational Computer
1982 Korea’s first semiconductor product “32K ROM chip”
Daejon (大田)
4. 魂▪創▪通
44
II. Introduction to the GLORY-FS
Multimedia metadata
management/retrieval
Multimedia Data distribution
Large-scale distributed data
management
Large-scale distributed parallel
processing
Large-scale File management
Large-scale cluster management
Low-power platform OS & HW
Retrieval: thousands of pages per
second(for hundreds of millions
of web pages)
Data parallel processing: 3,000
nodes
Storage capacity: Up to Petabytes
I/O performance: Up to 100Gb/sec
Cluster management: 10,000
nodes
Power saving: 20% reduction
Contents-based retrieval for
large-scale video data
10 ,000-node distributed data
processing middleware
Global cluster management
Global file management
Linux based 20% power saving
Internet service solution test
Provide global internet service solution specialized in UCC & IPTV
by developing open source-based GLORY platform
Global internet service solution S/W (GLORY-FS, GLORY-DB, GLORY-DP, GLORY-CL)
Multi-IDC testbed (256 nodes * 3 data centers)
Overview of the GLORY project (2007~2012)
This work was supported by the IT R&D program of MKE/KEIT.
[K1001703, Development of Cost Effective and Large Scale Global Internet Service Solution]
5. 魂▪創▪通
55
UCC Retrieval Service IPTV Service E-learning Service
Authoring Tagging Storing Retrieval Delivery
Dynamic Service
Management
Internet Services
Video Data Management Components
Internet
Services
Community
Components
Distributed job Scheduling
Job Partitioning & Merging
Data Distribution
Service Data Management
Data Access and Recovery
File Metadata
Management
Distributed Data
Storing & Replication
Remote
Backup & Archiving
Low-Cost Server Platform Node Manager Low-Power OS & H/W
Resource
Monitoring
Cluster
Orchestration
Automatic
Provisioning
II. Introduction to the GLORY-FS
SW Architecture of GLORY platform
6. 魂▪創▪通
66
GLORY-FS
Metadata Server GLORY-FS Data Server GLORY-FS Data Server
GLORY-FS
Metadata Server
GLORY-FS
Client Filesystem
GLORY-FS
Client Filesystem
Global Namespace
Storage TCO minimization by utilizing commodity servers as storage servers
High Performance by linearly scalable I/O performance
High Availability by efficient failure management
High Compatibility by supporting POSIX-FS standards
II. Introduction to the GLORY-FS
7. 魂▪創▪通
77
Internet Applications Service based on Videos (UCC/IPTV/e-Learning/etc.)
Large-scale and highly available File Services
Lustre
PVFS2
Panasas
Google FS
Mogile FS
GLORY-FS
(SW)
GlusterFS
Isilon
(Appliance)
II. Introduction to the GLORY-FS
HPC
Storage
Object
Store
Hadoop FS
Swift FS
Scale-out
NAS
High Performance
High Cost
Full POSIX Compliance
High Capacity
Low Cost
No POSIX Compliance
High Throughput
Low Cost
Near POSIX Compliance
8. 魂▪創▪通
88
Scalability in performance and capacity (up to 150 GB/s, Petabytes)
Availability with commodity HW (x86 server, SATA HDD)
High Compatibility with no kernel dependency (compatibility with any existing SW including web server)
Minimize management overhead
I/O and file sharing optimized to the internet services (web-disk, video, image content services)
III. Highlight of the GLORY-FS
POSIX API
Online storage
server expansion/
maintenance
*Storage reconfiguration
(Migration, Rebalance)
Multiple I/O data path
I/O for large, seq, read-
intensive workload
Lock-free
Cache Consistency Control
(NFS sharing semantic)
Sync, updatable
N-way data
replication
Parallel Replica
consistency
Asynchronous
MDS H/A
Web-based
management tool
*Highly scalable
metadata
cluster server
(3 billion scale)
*Synchronous
MDS H/A
*M+N striped
storage
(RAID Double
Parity)
Software Development
Kit
Scalable Capacity Scalable Performance Scalable Availability High Compatibility Simple Management
Virtual Metadata
Management
System
Windows Support
(*): Experimental
Over Petabyte
storage capacity
Up to 100Gb/s Data
input/output
performance
Up to 1 billion
files management
Disk Relocation
for fast rebalance
User-defined
Event handling
Unattended Recovery
Self-Diagnosis
on the SATA HDD
Hot-spot
Avoidance
(Self-tuning)
Private Replication
Network support
9. 魂▪創▪通
99
GLORY-FS Data Server
GLORY-FS Client File System
GLORY-FS Metadata Server
1g/10g Ethernet
Switch
Volume //
homehome shareshare
big.avi
Data
Metadata
Data
III. Highlight of the GLORY-FS
10. 魂▪創▪通
1010
III. Highlight of the GLORY-FS
#Client
Performance Scalability Test Metadata Clustering Effect Test
Data Server Cache Hit
Data Server Disk Hit
Client Cache Hit
Linearly scales-out
Linearly scales-out with multiple metadata servers
11. 魂▪創▪通
1111
III. Highlight of the GLORY-FS
Data Server Data Server Data Server Data Server
Each file is sliced into pieces, called CHUNK, and stored across multiple data servers
While CHUNKs are stored, REPLICA chunks are made synchronously to different data servers
When data server failure occurs, RECOVERS lost chunks from their replicas
All REPLICAs are used for file Read Access (Read load balance)
Arbitrary range of file are UPDATABLE at any time (Hadoop FS don’t allow file update operation)
Write performance with synchronous replication
Less than 5% replication overhead
C0
C1
C2
File
C0 C0C0 C0 C0
Memory Buffer
12. 魂▪創▪通
1212
III. Highlight of the GLORY-FS
REPLICAted file may become a Distributed RAID file when their access rate decreases
Generate PARITY chunks to different data servers from existing CHUNK
Remove REPLICA to save storage usage after PARITY coding completes
When data server failure occurs, RECOVERS lost chunks from their CHUNK & PARITY
Currently read-only access is allowed to RAID files . (for update access, revert to REPLICAed file)
Data Server Data Server Data Server Data Server
C0
Memory Buffer
C1 C2
P
R0 R1 R2C0 C1 C2
100GB files (1GB each) conversion time Read performance
13. 魂▪創▪通
1313
III. Highlight of the GLORY-FS
Filesystem Client
Data Servers
Service I/O Traffic Data Replication Traffic
Gigabit Switch 1/10 Gigabit Switch
Data Servers
New Data Server
(Empty)
Old Data Server
(Full of data)
Capacity Balanced
14. 魂▪創▪通
1414
III. Highlight of the GLORY-FS
Data Server Data Server Data Server Data Server Data Server
H H
File “H” is HOT
H HH HH HH HH H
File “H” is REPLICATED
For instant explosive read access such as hot movie
Hot file will be detected and replicated among more data servers automatically
to distribute load to other servers
Youtube Hot File Rank
16. 魂▪創▪通
1616
III. Highlight of the GLORY-FS
Built-in CPU, Disk, NIC, file access statistics monitoring
Accumulated statistics are visualized with MRTG-like chart
(daily, weekly, monthly, yearly statistics are also provided)
17. 魂▪創▪通
1717
Category References within Korea Year Capacity (TB)
Service
Company
KTH
('09)
Internet portal/UCC service '09~ 630
14,418
UCC service (Image/Video) '09.04~ 190
5GB mail attachment service '09.10 80
Mail service '09.12~ 300
N-Screen service '10.10 60
LG U+
('10)
LG U+ Internet Portal/UCC '10.11~ 8,700
Multimedia N-Screen service (U+ Box) '10.11~ 8,000
Dacom web-disk service '10.12 700
SKT ('10) Cloud storage service similar to the Amazon S3 (Ez-storage) '12.4~ 4,000
GS Neotech Storage for Content Delivery Network Service ‘12.1~ 200
Storage
Company
PSPACE
('07, '08, '09, '10)
InfiniStore (Appliance) '07~ 570
3D render farm storage '07~ 210
KBSn (IPTV, VOD) '10 72
MBN (IPTV, VOD) '10 48
KT (IPTV, VOD) '10 80
Neowiz (game portal) '10 60
BBMC (internet broadcasting) '10 100
MacroImpact
('09)
Sanique SFS '09~ 318
Storage for Content Delivery Network Service '09.10 318
Gluesys
('11)
Cluster NAS (Applicance) '11~
-
IV. GLORY-FS References within Korea
2007: NAS for High Capacity
2008~10: Storage intensive internet Services like web-mail, web-disk and image/video hosting service
2011~: Cloud Storage Service, Cloud CDN Service
18. 魂▪創▪通
1818
V. Future Roadmap
Project Name Period Status
Global Internet Service Solution (MKE)
–Global File System(GLORY-FS)
’07.3~’12.2
(5 year)
Closed
Supercomputing System for Genome Analysis (MKE)
– High Performance File System (MAHA-FS)
’11.3~’15.2
(5 year)
Open
Unified Storage Solution for Peta-scale (MKE)
’11.12~’13.11
(2 year)
Open
File System SW for Large-Scale Virtual Desktop Infrastructure (MKE)
‘12.5~’15.4
(3 year)
Open
GLORY-FS
Unified Storage
High Performance File System
VDI File System
07 08 09 10 11 12 13 14 15
Current Projects related with GLORY-FS
21. 魂▪創▪通
21
Unified Storage Solution for Peta-Scale
- Commercialization Project for GLORY-FS (World-best SW Program, MKE)
§ Low Cost/Large Scale à Low Cost/Large Scale/Higher Efficiency (Higher Storage Utilization)
§ iSCSI, NFS/CIFS support
§ Amazon S3 like API support (Restful API)
§ Sophisticated Management
Objectives
Appendix
22. 魂▪創▪通
22
File System SW for Large-scale virtual desktop infrastructure (VDI) service
- Commodity storage strategy for large-scale VDI service
§ Cost Wall: SAN storage à Commodity storage (50% cost saving)
§ Performance Wall: Low latency storage for VDI service (less than 20ms of VDI experience)
§ Scalability Wall: Up to 10,000 VDI user support
Objectives
Source Users
Boot
(IOPS)
Login
(IOPS)
Steady
(IOPS)
NetApp 2,500 46,000 29,000 10,000
EMC 500 63,500 14,500 10,500
Nimble 200 12,400 - 4,000
Estimated
(Worst)
1 127.0 112.0 32.0
10,000 1,270,000 1,120,000 320,000
Appendix
23. 魂▪創▪通
23
High Performance File System for Genome Analysis
- Overcome performance limitation of GLORY-FS for Petabyte-scale Genome Analysis
§ Hybrid Use of SSD + HDD (100 Million IOPS)
§ Totally Re-design of I/O subsystem for HPC
- Lower storage power consumption with the MAIS and MAID technology
§ MAIS: Massive Array of Idle Server (power on/off un-accessed storage server actively)
§ MAID: Massive Array of Idle Disk (power on/off un-accessed disks actively)
Objectives
Bandwidth
IOPS
Built-in Data Availability
(Replica, Distributed RAID)
Metadata OPS
Sharing Level
(POSIX Compliance, Locking)
TCO
Lustre MAHA-FS
CloudFS
Appendix