SFScon18 - Martin Palma - Building a petabyte-scale storage system based on free software

•

1 like•145 views

This lightning talk introduces the challenges in implementing a free software petabyte-scale storage system based on Ceph. The rationale it was chosen in favor of a non-free commercial solution and the difficulties in operating and managing such a system in an enterprise like environment. It will also provide a short technical overview of Ceph and its current usage inside Eurac Research.

Technology

Building a petabyte-scale storage
system based on free software

martin palma / infrastructure engineer / eurac research

What we want…
scalability
economically
reliability

What we want…
scalability
economically
reliability
persistence

What we want…
scalability
economically
reliability
persistence
performance

a uniﬁed, distributed storage system designed for excellent
performance, reliability and scalability.

• Created by Sage Weil for his PHD dissertation at the University of California, Santa Cruz in
2007
• From fall 2007 he worked full-time on Ceph at Dreamhost (he is one of the co-founder of
Dreamhost)
• In 2012 founded Inktank Storage for professional services and support of Ceph
• First release Argonaut on July 2, 2012
• April 2014 Red Hat purchased Inktank
• October 2015 the Ceph Community Advisory Board was formed. (Canonical, CERN, Cisco,
Fujitsu, Intel, Red Hat, SanDisk, and SUSE.)
• November 2018 The Ceph Foundation (funded under the Linux Foundation)
• Until now 13 releases (Mimic is the latest)

• All components must scale horizontally
• There can be no single point of failure
• The solution must be hardware agnostic
• Should use commodity hardware
• Self-Managed whenever possible
• Open Source (LGPL)

RGW
A web services
gateway for object
storage, compatible
with S3 and Swift
LIBRADOS
A library allowing apps to directly access RADOS (C, C++, Java, Python, Ruby, PHP)
RADOS
A software-based, reliable, autonomous, distributed object store comprised of
self-healing, self-managing, intelligent storage nodes and lightweight monitors
RBD
A reliable, fully-
distributed block
device with cloud
platform integration
CEPHFS
A distributed >le
system with POSIX
semantics and scale-
out metadata
management
APP HOST/VM CLIENT

• Maintain cluster membership and state
• Provide consensus for distributed decision-making
• Small and odd number
• Not part of the data path
• Storing and serving data to clients
• On per disk (HDD, SSD, NVMe)
• 10s to 1000s in a cluster
• Intelligently peer for replication & recovery
Monitors
Object Storage Daemon (OSD)

CRUSH Controlled, Scalable, Decentralized Placement of Replicated Data
Pseudo-random placement algorithm
• Fast calculation, no lookup
• Repeatable, deterministic
Statistically uniform distribution
Rule-base conﬁguration
• Infrastructure topology aware (CRUSH map)
• Adjustable replication
• Weighted devices (different sizes)
Data Placement

Why Ceph?
scalability
economically
reliability
persistence
performance

Why Ceph and not X?
scalability
economically
reliability
persistence
performance

• 39 Storage nodes (12 * 4 TB SATA, 4 * 400 GB SSD)
• 546 Object Storage Daemon
• 5 Monitor nodes
• 2 Metadata nodes
• Dedicated public and cluster network
• Storage nodes with 2 x 10 Gbit/s for public & 2 x 40 Gbit/s for cluster connectivity
• Monitor & Metadata have 2 x 10 Gbit/s public connectivity
• Hardware: Supermicro, Mellanox
• OS: CentOS 7.4
• Ceph version: Luminous 12.2.4

• Mainly CephFS and RBD used
• Integrated into internal Kubernetes and LXD infrastructure
• Started in 2014 with Hammer LTS
• Done 3 major version upgrades (Hammer -> Jewel -> Luminous)
• Scaled from initial 600 TB to 1,7 PB
• Raw used: ~1 PB
• Usage:
RBD
23%
CephFS
77%

Challenges
documentation
operation
experience

Challenges
experience
backup
operation
documentation

Thank you
Building a petabyte-scale storage system based on free
software
Martin Palma
Eurac Research
martin.palma@eurac.ed
http://www.eurac.edu

What's hot

When to select hyper converged 2016 Sydney VMUG

Keith Townsend

Cost Effectively Run Multiple Oracle Database Copies at Scale

NetApp

How Lenovo and Nutanix are delivering the invisible infrastructure

Lenovo Data Center

Clouds in Your Coffee Session with Cleversafe & Avere

Avere Systems

An Easy Path to Kubernetes on Nutanix

NEXTtour

7 steps to storage freedom and avoiding vendor lock in - io fabric 2017

Greg Wyman

IDC Nutanix - Hyperconvergence and the Pulling Forces in the Datacenter

NEXTtour

Experiencing the hyperconverged

ICT-Partners

ConverBiz offering for database services

Sridhar Pai

Přehled portfolia ODA a praktických případů v regionu EMEA

MarketingArrowECS_CZ

VMUG.IT UserCon 2015 - Nutanix

VMUG IT

The HPC storage performance tier is well defined: scale-out solid state storage systems. But the capacity tier is up for debate. Should you use a high end NAS file system or make the switch to object storage? More importantly: How do you move data from the performance tier to the capacity tier without placing additional burden on already overworked IT personnel? We answer these questions and provide designs that solve the HPC storage-tug-of-war in our webinar with Caringo. Listen as experts on HPC, NAS and Object Storage discuss the HPC storage challenge, debate the potential solutions and provide you guidance on how to create the right architecture.

Webinar: Performance vs. Cost - Solving The HPC Storage Tug-of-War

Storage Switzerland

SplunkLive! Nutanix Session - Turnkey and scalable infrastructure for Splunk ...

Splunk

4 hp converged_cloud

openstackindia

Modern storage for modern business: get to know Hedvig

Eric Carter

Enterprise File Share and Sync with CleverSafe

jimliddle

Infrastructure is getting smaller through convergence. Cisco UCS exploded in the market by reducing infrastructure components and footprint. Hyperconverged vendors like Nutanix are taking that concept even further – talking about “invisible infrastructure.” The shifts in delivery of storage, compute, networking and virtualization create huge opportunities to improve delivery and TCO, but they can also make you want to tear your hair out trying to protect it all.

Protect the Hype: Backup Best Practices for Converged & Hyperconverged Infras...

marketingunitrends

Cloudera Federal Forum 2014: Hadoop's Impact on the Future of Data Management

Cloudera, Inc.

Scale up is history! is scale out the future for storage

StarWind Software

Webinar: Don't believe the hype, you don't need dedicated storage for VDI

NetApp

What's hot (20)

When to select hyper converged 2016 Sydney VMUG

Cost Effectively Run Multiple Oracle Database Copies at Scale

How Lenovo and Nutanix are delivering the invisible infrastructure

Clouds in Your Coffee Session with Cleversafe & Avere

An Easy Path to Kubernetes on Nutanix

7 steps to storage freedom and avoiding vendor lock in - io fabric 2017

IDC Nutanix - Hyperconvergence and the Pulling Forces in the Datacenter

Experiencing the hyperconverged

ConverBiz offering for database services

Přehled portfolia ODA a praktických případů v regionu EMEA

VMUG.IT UserCon 2015 - Nutanix

Webinar: Performance vs. Cost - Solving The HPC Storage Tug-of-War

SplunkLive! Nutanix Session - Turnkey and scalable infrastructure for Splunk ...

4 hp converged_cloud

Modern storage for modern business: get to know Hedvig

Enterprise File Share and Sync with CleverSafe

Protect the Hype: Backup Best Practices for Converged & Hyperconverged Infras...

Cloudera Federal Forum 2014: Hadoop's Impact on the Future of Data Management

Scale up is history! is scale out the future for storage

Webinar: Don't believe the hype, you don't need dedicated storage for VDI

Similar to SFScon18 - Martin Palma - Building a petabyte-scale storage system based on free software

Fortissimo Foundation is a clustered, pervasive, global direct-remote I/O access system that linearly scales I/O bandwidth, memory, Flash and hard disk storage capacity and server performance to provide an “in-memory” scale-out solution that intelligently aggregates all resources of a data center cluster into a massive global name space, bridging all remote compute and storage resources to look and act as if they were local. By providing a complete set of hardware and software building blocks through Fortissimo, A3Cube enables organizations to broadly deploy the power of high-end HPC clusters using low-cost, commodity servers and storage and without the high complexity, cost and fundamental limitations of traditional scale out systems." Learn more: http://www.a3cube-inc.com/fortissimo-foundation-1.html Watch the video presentation: http://wp.me/p3RLEV-2XF

Fortissimo Foundation A Clustered, Pervasive, Global Direct-remote I/O Access...

inside-BigData.com

Cisco hyperflex software defined storage and ucs unite

Cisco Canada

Excelero was inspired by how Tech Giants like Amazon, Facebook and Google have redefined IT for web-scale applications, leveraging standard servers and shared-nothing architectures to ensure maximum operational efficiency and flexibility, and to provide the highest reliability. Their approach is referred to as the Software-defined Data Center (SDDC). Based on the same architectural principles and with the same objectives, Excelero has designed a Software-Defined Block Storage solution utilizing client-side services for scale-out applications. Visit http://www.excelero.com to learn more

Excelero overview for slideshare

Tom Leyden

Red Hat Storage Day Boston - OpenStack + Ceph Storage

Red_Hat_Storage

Nexenta transtec

TTEC

NAVER Ceph Storage on ssd for Container

Jangseon Ryu

End of RAID as we know it with Ceph Replication

Ceph Community

These are the *updated* slides (InnoDB clusters and MySQL Enterprise Monitor 3.4 are now GA) from the following webinar, which you can now watch on demand: https://www.mysql.com/news-and-events/web-seminars/why-mysql-high-availability-matters/ ----------------------------------------------------- MySQL high availability matters because your data matters. If your database goes down, whether due to human error, catastrophic network failure, or planned maintenance, the accessibility and accuracy of your data can be compromised with disastrous results. We'll examine the critical elements of a high availability solution, including: - Data redundancy - Data consistency - Automatic fault detection and resolution - No single point of failure And how you can achieve these things more easily than ever before using MySQL's new native HA solution.

Why MySQL High Availability Matters

Matt Lord

Have you heard about Inktank Ceph and are interested to learn some tips and tricks for getting started quickly and efficiently with Ceph? Then this is the session for you! In this two part session you learn details of: • the very latest enhancements and capabilities delivered in Inktank Ceph Enterprise such as a new erasure coded storage back-end, support for tiering, and the introduction of user quotas. • best practices, lessons learned and architecture considerations founded in real customer deployments of Dell and Inktank Ceph solutions that will help accelerate your Ceph deployment.

New Ceph capabilities and Reference Architectures

Kamesh Pemmaraju

Software Defined Storage, Big Data and Ceph - What Is all the Fuss About? By: Kamesh Pemmaraju,Neil Levine Have you heard about Inktank Ceph and are interested to learn some tips and tricks for getting started quickly and efficiently with Ceph?  Then this is the session for you! In this two part session you learn details of: • the very latest enhancements and capabilities delivered in Inktank Ceph Enterprise such as a new erasure coded storage back-end, support for tiering, and the introduction of user quotas. • best practices, lessons learned and architecture considerations founded in real customer deployments of Dell and Inktank Ceph solutions that will help accelerate your Ceph deployment.

Software Defined Storage, Big Data and Ceph - What Is all the Fuss About?

Red_Hat_Storage

TECHunplugged Austin 2016

Chris Evans

Red hat ceph storage customer presentation

Rodrigo Missiaggia

For years vendors have been trying to drive down the cost of flash so that the all-flash data center can become reality. The problem is that even the rapidly declining price of flash storage can’t keep pace with the rapidly declining price of hard disk. As a result data that does not need to be on flash storage has to be stored on something less expensive. But does that less expensive storage need to be another hard disk array or could it be stored in the cloud? In this webinar join Storage Switzerland’s founder George Crump and Avere Systems CEO, Ron Bianchini for an interactive webinar Using the Cloud to Create an All-Flash Data Center.

Share on LinkedIn Share on Twitter Share on Facebook Share on Google+ Share b...

Avere Systems

Webinar - Introduction to Ceph and OpenStack

Ceph Community

The Briefing Room with Dez Blanchfield and Red Hat Think of containers as the drones of modern computing. They're small, agile, and can carry a significant payload. In many ways, they represent the fruition of the last two major paradigm shifts in enterprise software: SOA and virtualization. However, for companies to fully leverage this innovative approach, a persistent storage platform is needed that is as flexible and scalable as containers themselves. Register for this episode of The Briefing Room to hear Bloor Group Data Scientist Dez Blanchfield, who will explain the significance of container technology, and the relevance of software-defined storage (SDS) in a constantly evolving IT world. He'll be briefed by Steve Watt and Sayan Saha of Red Hat, who will demonstrate how open-source technology can help organizations take advantage of this brave new world of enterprise computing. They will explain how containers are the next step in the evolution of the operating system, and why SDS is now the optimal solution.

A Tight Ship: How Containers and SDS Optimize the Enterprise

Eric Kavanagh

Accelerate Your Migration to "Application-Centric" Storage-as-a-Service from ...

VMware

start_your_datacenter_sds_v3

David Byte

Cloud-Ready, Scale-Out Storage

ryanwakeling

Cleversafe august 2016

Joe Krotz

Percona Live 4/14/15: Leveraging open stack cinder for peak application perfo...

Tesora

Similar to SFScon18 - Martin Palma - Building a petabyte-scale storage system based on free software (20)

Fortissimo Foundation A Clustered, Pervasive, Global Direct-remote I/O Access...

Cisco hyperflex software defined storage and ucs unite

Excelero overview for slideshare

Red Hat Storage Day Boston - OpenStack + Ceph Storage

Nexenta transtec

NAVER Ceph Storage on ssd for Container

End of RAID as we know it with Ceph Replication

Why MySQL High Availability Matters

New Ceph capabilities and Reference Architectures

Software Defined Storage, Big Data and Ceph - What Is all the Fuss About?

TECHunplugged Austin 2016

Red hat ceph storage customer presentation

Share on LinkedIn Share on Twitter Share on Facebook Share on Google+ Share b...

Webinar - Introduction to Ceph and OpenStack

A Tight Ship: How Containers and SDS Optimize the Enterprise

Accelerate Your Migration to "Application-Centric" Storage-as-a-Service from ...

start_your_datacenter_sds_v3

Cloud-Ready, Scale-Out Storage

Cleversafe august 2016

Percona Live 4/14/15: Leveraging open stack cinder for peak application perfo...

More from South Tyrol Free Software Conference

The complexity of agricultural droughts requires a consistent, reliable, and systematic method for monitoring and reporting. Amongst the various indices used to monitor this phenomenon, the soil moisture anomaly has been proven to be a more reliable predictor. However, the datasets required for computing this index are often large and computationally demanding. To address this challenge, we have developed SMODEX, a Python package that enables scalable, fast, and open-source standard-compliant computation and visualization of soil moisture anomalies. SMODEX simplifies the computation and visualization of time-series for soil moisture and soil moisture anomalies from high-dimensional climate datasets. It allows for quick and easy parallelization of the computation on a daily, weekly, and monthly timescale. Additionally, SMODEX implements a straightforward workflow for automating the use of FAIR (Findable, Accessible, Interoperable, and Reusable) principles in producing and sharing outputs by leveraging the open source STAC API. The package is extendible and provides information on how to contribute to the project, test suites, test coverage, and a use case for the South Tyrol region, all provided in the package repository. In the future, additional agricultural drought indices and indicators would be included to serve to even larger community of researchers, policy makers, and individual users.

SFSCON23 - Rufai Omowunmi Balogun - SMODEX – a Python package for understandi...

South Tyrol Free Software Conference

The Open Hardware PowerPC Notebook designed around GNU/Linux will be showed at NOI Techpark. We had presented here its motherboard design in 2018. We will updates regarding last developments for u-boot AMD video drivers, re-design of heat pipes, and CE test certification process. We will give future availability milestones of this notebook and details regarding the GNU/Linux distributions or other OS that could runs on it.

SFSCON23 - Roberto Innocenti - From the design to reality is here the Communi...

South Tyrol Free Software Conference

Tracking aeroplanes in real time with Open Source Software is possible. Aircrafts must continuously send their current flight parameters to air traffic controllers on the ground and to other aircrafts. This generates a lot of data, especially when planes are being tracked by multiple sensors. The Open Data Hub on the other hand offers a great backbone for data storing and processing, where the correct datasets have to be identified and filtered. After all transformation on the data is done, it will be exposed via API to be further used by a web application. Bringing together sensor generated data, the Open Data Hub and custom web applications, is a showcase on how the Open Data Hub can be used as a service: OaaS.

SFSCON23 - Martin Rabanser - Real-time aeroplane tracking and the Open Data Hub

South Tyrol Free Software Conference

The transition from Web 2.0 to Web 3.0 has fueled the need for a secure and decentralized cloud storage solution for digital assets. Web 2.0 was characterized by centralized platforms where user data was under the control of companies. In contrast, Web 3.0 aims to empower individuals and foster a decentralized web that supports and benefits the Free Software and Open Data Communities. Blockchain technologies facilitate seamless collaboration and interoperability among diverse stakeholders in the Free Software and Open Data communities. Developers can establish open and transparent ecosystems where data can be shared, verified, and integrated across multiple platforms. Beez, with its own blockchain infrastructure, offers a secure and transparent platform for digital asset exchanges, bolstering transaction integrity and trust. By distributing data across a network of nodes, Beez ensures security and mitigates the risk of single points of failure. Users retain control over their data, safeguard their privacy, and can take advantage of the incentive mechanisms offered by blockchain networks. During our presentation, we will explore the role of AI within Beez's ecosystem, facilitating accelerated data processing, correlation, and intelligent automation. AI unlocks valuable insights from blockchain data, and we will touch upon the use of Inductive Logic Programming (ILP) to enhance programming performance. The integration of Blockchain and AI technologies holds great potential for advancing the safety and efficiency of the Open Data ecosystem. By combining decentralized data storage, trust-building mechanisms, and intelligent data processing, Beez is paving the way for a more secure, transparent, and user-centric digital landscape.

SFSCON23 - Marianna d'Atri Enrico Zanardo - How can Blockchain technologies i...

South Tyrol Free Software Conference

We are becoming more and more dependent on the Internet for our work, education, communication, personal relations and entertainment. Our digital devices conquered an unprecedented level of importance in our life. However, we are facing a loss of control over our smartphones, tablets and other devices for internet connection. It's time to resolve monopolies and re-establish democratic control over the technology we most depend upon. This talk will present the challenges end-users are facing to get more control over their devices and how Free Software is key for a consumer re-empowerement. The talk will present real-life examples of policy demands against gatekeepers on digital markets, such as the struggle for Router Freedom in the last years and how Device Neutrality can serve as an important instrument for pushing forward end-user-oriented digital policies.

SFSCON23 - Lucas Lasota - The Future of Connectivity, Open Internet and Human...

South Tyrol Free Software Conference

MOSH and MOAH are the abbreviation of two groups of chemical compounds found in mineral oils. “MOSH” stands for Mineral Oil Saturated Hydrocarbons. MOAH stands for Mineral Oil Aromatic Hydrocarbons. Both of them are under European deeply evaluation because there are two food contaminants. According to the current state of scientific knowledge, there is no sufficient toxicological evidence to prove a health risk to humans from saturated mineral oil fractions (MOSH). Meanwhile, MOAH are suspected to be carcinogenic (especially PAH-like compounds with 3-7 ring systems), therefore their levels in food should be reduced according to the ALARA-principle (as low as reasonably achievable). Gruppo FOS with CNR ( MOSH and MOAH are the abbreviation of two groups of chemical compounds found in mineral oils. “MOSH” stands for Mineral Oil Saturated Hydrocarbons. MOAH stands for Mineral Oil Aromatic Hydrocarbons. Both of them are under European deeply evaluation because there are two food contaminants. According to the current state of scientific knowledge, there is no sufficient toxicological evidence to prove a health risk to humans from saturated mineral oil fractions (MOSH). Meanwhile, MOAH are suspected to be carcinogenic (especially PAH-like compounds with 3-7 ring systems), therefore their levels in food should be reduced according to the ALARA-principle (as low as reasonably achievable). Gruppo FOS with CNR (Consiglio Nazionale delle Ricerche), Santagata 1907 and Enginius are searching the system for finding and trace their presence in the virgin and extra virgin olive oils by using open fingerprints methods, open hardware and open source blockchain and AI technologies.

SFSCON23 - Giovanni Giannotta - Intelligent Decision Support System for trace...

South Tyrol Free Software Conference

Up-to date measurements of surface meteorological variables are essential to monitor weather conditions, their spatio-temporal variability and the potential effects on a wide range of sectors and applications. Moreover, when included in continuous records of long historical observations spanning several decades, they become essential for assessing long-term climate variability and change locally and on a regional level. Automated pipelines capable of retrieving and processing near-real time meteorological data satisfy the primary prerequisites towards the development and advancement of effective and operational climate services. With a public and operational near real-time monitoring web platform in mind, we present automated pipelines to collect and process up-to-date daily temperature and precipitation records for Trentino South Tyrol (Italy) and surrounding areas, and to derive their spatially interpolated fields at sub-km scale. Our pipelines are composed by multiple steps including data download, sanity checks, reconstruction of missing daily records, integration into the historical archive, spatial interpolation and publication onto online FAIR catalogues as (openEO) “datacubes”. The different APIs, data formats and structure across the various data sources, and the need to merge the data onto harmonized meteorological layers, make this a typical case of the so-called Extract, Transform and Load (ETL) pipelines, and, in order to follow the principles of data reproducibility and Open Science, we embraced open-source automated workflow management through GitLab’s Continuous Integration / Continuous Development (CI/CD) capabilities. CI/CD workflows greatly help the management of the relatively complex graphs of tasks required for our climate application, ensuring seamless orchestration with thorough flow monitoring, application logs, transactions rollbacks, and exception handling in general. Native pipeline-oriented software development also fosters a clean separation of roles among the tasks, and a more modular architecture. This effectively reduces barriers to collaborative development and paves the way for robust operational climate services for researchers and decision makers in the face of the changing climate.

SFSCON23 - Elena Maines - Embracing CI/CD workflows for building ETL pipelines

South Tyrol Free Software Conference

The Open Science movement aims to increase the transparency, reproducibility and inclusiveness of academic research. One of its central goals is therefore to make research outputs broadly available, e.g., manuscripts (Open Access) or research data (Open Data). While software/code created in the course of scientific research is a key artifact of scientific research that is clear distinct from the latter two, it has until recently not received the same attention as manuscripts or data, although it follows its own set of paradigms. In this talk I will present an overview on how the core concepts of Free Software and the FAIR (findable, accessible, interoperable, reuseable) Principles intersect, what this means for managing code as research output and recent initiatives on the European level that will provide support for these issues.

SFSCON23 - Christian Busse - Free Software and Open Science

South Tyrol Free Software Conference

Software freedom can be defined in many ways but in legal terms it is squarely defined by a set of approved FSF and OSI software licenses. Yet everyone realizes that beyond these licenses the goal of software freedom and digital sovereignty cannot be achieved without the ability to master and create hardware components and systems - and beyond that, to rely on open digital infrastructure (servers, datacenters, and resources) . This talk will present the challenges around these topics and what we, collectively in Europe already do and can do to ensure our independence and our freedoms.

SFSCON23 - Charles H. Schulz - Why open digital infrastructure matters

South Tyrol Free Software Conference

EDP-portal is the access point to the Environmental Data Platform of Eurac Research since 2021 to achieve FAIRness of our datasets. It allows to publish data and metadata and provides APIs and web services for data access. In the last 2 years the EDP improved the findability and accessibility of the data collected throughout the curation of metadata that was improved with the DOI registration for datasets. The result is a higher metadata quality where the final user can easily find how to properly cite datasets with a persistent identifier. The portal itself and main data repositories are registered in FAIR-sharing portal with their own DOI. The SW components of the EDP are totally based on open source projects.

SFSCON23 - Andrea Vianello - Achieving FAIRness with EDP-portal

South Tyrol Free Software Conference

This lightning talk will explore the transformative potential of integrating Internet of Things (IoT) and Artificial Intelligence (AI) in Mass Customization (MC). There is a significant collective impact of these technologies on businesses, enabling the delivery of personalized products and exceptional customer experiences. Besides giving an overview of MC and the potential ways of integrating IoT and AI, the focus will be on the process of real-time data collection and facilitation of the customization process by IoT on one hand, and on the role of AI in data analysis and generation of personalized recommendations on the other hand. By presenting real-world case studies to demonstrate the practical implementation of IoT and AI in providing customized products and seamless customer experiences, attendees will gain insights into the future of customization and learn actionable strategies to effectively leverage IoT and AI.

SFSCON23 - Thomas Aichner - How IoT and AI are revolutionizing Mass Customiza...

South Tyrol Free Software Conference

SFSCON23 - Stefan Mutschlechner - Smart Werke Meran

South Tyrol Free Software Conference

As open source software becomes the foundation to build digital products, to run the backbones of ICT infrastructure and to ensure digital sovereignty and cyber resilience, both the technology as well as the communities that develop it inevitably move into the focus of regulators. The European Union is advancing a number of policy initiatives that regulate liability, cyber security, data handling and AI applications in digital products, among others. This is a challenge for the still quite decentralised and globally operating open source community. How could the open source community participate in legislative processes, and what may be the potential impacts of the upcoming regulation on the open source development process and community dynamics?

SFSCON23 - Mirko Boehm - European regulators cast their eyes on maturing OSS ...

South Tyrol Free Software Conference

SFSCON23 - Marco Pavanelli - Monitoring the fleet of Sasa with free software

South Tyrol Free Software Conference

AICS is the Italian Agency for Development Cooperation that started operating in 2016 with the ambition of aligning Italy with the main European and international partners in the commitment to development. KNOWAGE Labs are developing for AICS a platform that is probably unique in the world and will allow both the Agency and the public to access all the major indicators on the UN Sustainable Development Goals provided by international sources (World Bank, WTO, ILO..) and easily compare them. The solution will allow analysis to start from 3 different touch points: the infographic of SDG goals, the advanced search criteria, and the virtual assistant. Then, a customized dashboard will be provided to the user, allowing to further expand the analysis by interacting with charts, maps, tables, etc. This talk will show the state of art of the solution, highlighting objectives and expected results of the project, but also the new developments of KNOWAGE related to AI.

SFSCON23 - Marco Cortella - KNOWAGE and AICS for 2030 agenda SDG goals monito...

South Tyrol Free Software Conference

Interoperability is a core element of the ongoing digitalisation of Europe. With the Interoperable Europe Act, the EU is aiming to create a dedicated legal framework for interoperability and to enhance cross-border digital public services across the European Union. This talk will give an overview of the state of play of this proposed regulation in the ongoing EU legislative process, some of its flaws, and the important role that Free Software and its community can play in it.

SFSCON23 - Lina Ceballos - Interoperable Europe Act - A real game changer

South Tyrol Free Software Conference

How to sharpen the demand for public code across Europe and monitor progress with TEDective For six years, the Free Software Foundation Europe has been calling with a broad alliance for publicly funded software to be published as Free Software. This initiative has become a great success: Our demand "Public Money? Public Code!" has found its way into government strategy papers, party programs, as well as coalition treaties, and is being discussed in public administrations across Europe. At the same time, we see less progress than expected and vendor lock ins remain a crucial issue. Digital sovereignty is redefined bypassing Free Software. There is openwashing in publicly funded companies, and government projects in favour of Free Software remain empty words. Public statistics on the procurement of Free Software are largely unavailable. It is therefore no longer enough to promote the idea of "Public Money? Public Code!". We as the Free Software community should be even more vigilant than before – continuing to praise small steps in the right direction, but pointing out and criticising omissions and lack of implementation. We should become more like watchdogs. In the talk we will look at some examples of lack of implementation of Free Software policies. We will discuss how we, as civil society, can identify such shortcomings and how to deal with them. We will present our initiative TEDective – a free-software solution that makes European public procurement data explorable for non-experts, aiming to provide you with a powerful tool to keep an eye on real progress towards "Public Money? Public Code!" across Europe.

SFSCON23 - Johannes Näder Linus Sehn - Let’s monitor implementation of Free S...

South Tyrol Free Software Conference

The Internet today forms the backbone of the digitisation of our society and economy. As connectivity increases, the boundaries between the real and digital world get increasingly blurred. However, there has been an erosion of trust in the Internet following revelations about the exploitation of personal data, large-scale cybersecurity and data breaches, and growing awareness of the proliferation and impacts of online disinformation. What can be done to improve the Internet as a platform for future generations? What initiatives are currently in place to build key technological blocks of an Internet that supports human-centric values, such as privacy, security, and inclusion, while reflecting the values and norms all citizens should enjoy in Europe? This talk will explore why the current state of the internet must be re-imagined and re-engineered in order to support healthy societies, the existing European Commission initiative to work towards doing so, and the role of Free Software in accomplishing these goals.

SFSCON23 - Gabriel Ku Wei Bin - Why Do We Need A Next Generation Internet

South Tyrol Free Software Conference

2023 saw the launch, after a long and well-structured revision and development process, all based on a fruitful collaboration between several departments of the Autonomous Province of Bolzano, most of the township in South Tyrol, Informatica Alto Adige (SIAG - Technical partner) and the Consortium of Municipalities of the Province of Bolzano, of the new version of the integrated geographic data management system IGis Maps. In use for years in South Tyrol, has in the Consortium one of its most enthusiastic contributors and supporters. The very first version was released about eight years ago and its implementation was based on the idea of creating a multi-purpose GIS management system that could support different types of users, that was highly customizable, and, above all, that could be widely shared among the various management entities, both public and private, present within our territory. After years of use and ad-hoc developments, we can finally present the new version of the IGis Maps system, which incorporates all the technical and technological improvements we realized the system needed. It was not just a major update together with new functionalities combined inside the previous software structure, but a true re-engineering that led, among other things, to a new and more efficient user interface, a major advancement regarding the internal security, an optimization and improvement of the entire editing section as well as an optimization of the section regarding the automatic geo-processes. A mobile version is currently under development to better support any field activities, for which a very powerful option will be included, the possibility of creating special work sessions in off-line mode so as to be able to operate even in areas without a proper cellular line network coverage. Other very important peculiarities concern that the system is developed using a totally free software code and infrastructure, that a detailed documentation has been produced to ensure sustainability to any further future evolution, even in case of technical partner turnover, and finally, that by taking advantage of the high standards and levels of security access can be guaranteed to any type of user. From professional users, through dedicated access and qualifications or, using the ordinary SPID, to the private citizen. We will show examples of how different types of users and stakeholders now permanently use the system for the management of a variety of tasks related to their activities, and how it was possible to customize IGis Maps to create visualization and data management contexts that best meet their needs. We will also present a related project concerning the updating and the correction of the new technical basal cartography, built upon the new Basic Core specification, achieved through the automatic conversion implemented by the SIAG team starting from the previous National Core cartography. With the new IGis Maps it was possible to create an a

SFSCON23 - Edoardo Scepi - The Brand-New Version of IGis Maps

South Tyrol Free Software Conference

KNOWAGE is the open source analytics and business intelligence suite made in Italy. KNOWAGE aims to provide company and organizations with analytical capabilities to exploit data to increase their efficiency and sustainability. Also thanks to the open source community support, the suite is constantly evolving combining the reliability of the most popular business intelligence solutions with the security and the transparency guaranteed by open source. This talk will show the last year advancements and new features towards a more mobile, accessible and user-friendly product, focusing on the newly rewritten dashboarding tool.

SFSCON23 - Davide Vernassa - Empowering Insights Unveiling the latest innova...

South Tyrol Free Software Conference

More from South Tyrol Free Software Conference (20)