Presentation at the MOC Workshop, at Boston University.
Cloud Dataverse will be a new service for accessing and processing public data sets in a the Massachusetts Open Cloud (MOC). It is based on Dataverse, a popular software framework for sharing, archiving, and analyzing research data. Cloud Dataverse extends Dataverse to replicate datasets from institution repositories to a cloud-based repository and store their data files in Swift, making data processing faster for in-situ application running in the cloud.
Cloud Dataverse is a collaborative effort between two open source projects: Massachusetts Open Cloud (MOC) and Dataverse. The Dataversesoftware is being developed at Harvard's Institute for Quantitative Social Science (IQSS) with contributors worldwide providing 21 Dataverse installations. The Harvard Dataverse installation alone hosts more than 60,000 datasets from 300 institutions by 15,000 data authors. The MOC is a collaboration between higher education (BU, NEU, Harvard, MIT and UMass), government, and industry. Its mission is to create a self-sustaining at-scale public cloud based on the Open Cloud eXchange model.
This talk describes our experiences from hosting scientific research application in the Microsoft Cloud. Covers an overview of Microsoft Azure capabilities, examples of big data analysis for science, data collections, science gateways and science virtual machine libraries.
This talk explains how cloud computing to speed up your research, and provide capabilities that are otherwise out of reach. Big data, data science, machine learning, high-performance computing are all available on-demand using Microsoft Azure.
Researchers around the world can apply for free cloud computing time for their projects at www.Azure4Research.com
Research results in peer-reviewed publications are reproducible, right? If only it was so clear cut. With high profile paper retractions and pushes for better data sharing by funders, publishers and the community, the spotlight is now focussing on the whole way research is conducted around the world.
This talk from the Software Sustainability Institute's Collaborations Workshop 2014 describes how cloud computing, with Microsoft Azure, is helping researchers realize the goals of scientific reproducibility.
Find out more at www.azure4research.com
A presentation at the NIH Workshop on Advanced Networking for Data-Intensive Biomedical Research. The talk covers our work with the science community on using cloud computing to enhance and improve basic research for data analysis and scientific discovery
How cloud computing can accelerate your research. Presentation given at Moscow State University on 19th May 2015.
Apply for Azure for Research Awards at http://research.microsoft.com/en-US/projects/azure/awards.aspx
This talk describes our experiences from hosting scientific research application in the Microsoft Cloud. Covers an overview of Microsoft Azure capabilities, examples of big data analysis for science, data collections, science gateways and science virtual machine libraries.
This talk explains how cloud computing to speed up your research, and provide capabilities that are otherwise out of reach. Big data, data science, machine learning, high-performance computing are all available on-demand using Microsoft Azure.
Researchers around the world can apply for free cloud computing time for their projects at www.Azure4Research.com
Research results in peer-reviewed publications are reproducible, right? If only it was so clear cut. With high profile paper retractions and pushes for better data sharing by funders, publishers and the community, the spotlight is now focussing on the whole way research is conducted around the world.
This talk from the Software Sustainability Institute's Collaborations Workshop 2014 describes how cloud computing, with Microsoft Azure, is helping researchers realize the goals of scientific reproducibility.
Find out more at www.azure4research.com
A presentation at the NIH Workshop on Advanced Networking for Data-Intensive Biomedical Research. The talk covers our work with the science community on using cloud computing to enhance and improve basic research for data analysis and scientific discovery
How cloud computing can accelerate your research. Presentation given at Moscow State University on 19th May 2015.
Apply for Azure for Research Awards at http://research.microsoft.com/en-US/projects/azure/awards.aspx
How Cyverse.org enables scalable data discoverability and re-useMatthew Vaughn
Cyverse.org designs, builds, and operates an innovative, integrated life sciences cyberinfrastructure. It provides data management and analysis capabilities with point and click, cloud, API, and command-line interfaces that engage users of any computing proficiency and is based on an extensible platform that integrates local and national-scale HPC, storage, and cloud resources. Cyverse directly supports thousands of users who store and access over 2PB of research data, use millions of compute hours annually, and participate in the platform's improvement, plus a secondary user community from partner projects that have built atop it. Cyverse is organized around "Data Store" and "App Catalog" services, each of which enables users to upload digital research assets that can be kept private, shared, or made public. Recently, Cyverse has been transitioning from passively enabling digital sharing towards active facilitation. It is partnering with repositories like NCBI SRA to enable direct submission from Cyverse applications, adopting commonly-used ontologies, enabling import/export of virtual machine images, developing metadata-driven persistent landing pages for data sets, and providing DOI (and other identifier) services. These new features are expected to further catalyze growth of an interoperable, interconnected network of shared research infrastructure across the biological sciences.
These are the slides from a plenary panel that I participated in at IEEE Cloud 2011 on July 5, 2011 in Washington, D.C. I discussed the Open Science Data Cloud and concluded the talk by three research questions
Observations on a whole lot of Things learned through the 23 (research data) ...ARDC
The 23 (research data) Things program has enabled hundreds of Librarians, data managers, repository managers, research impact staff and others to learn and apply best practice data management principles through examples, activities and stories.
UK Research Data Discovery Service metadata schemaJisc RDM
An overview of the metadata schema being developed for the UK research data discovery service. Dom Fripp at the Research Data Network event at Cardiff University, May 2016.
ANDS Applications Program: Building Tools to Facilitate Data ReuseAndrew Treloar
Presentation accompanying talk on the ANDS Applications program at IDCC 2016. Discusses the outputs of the program, but also focusses on issues of sustainability of such eresearch tools
This presentation sets out some of the challenges around citing and identifying datasets and introduces DataCite, the international data citation initiative. DataCite was founded on 1-December 2009 to support researchers by
providing methods for them to locate, identify, and cite
research datasets with confidence.
This presentation was given by Adam Farquhar at the STM Publishers Association Innovation Conference on 4-Dec-2009.
Cloud Dataverse: A Data repository platform for an OpenStack CloudMerce Crosas
In the last 10 years, the Dataverse project has been a leader in open-source repository software for sharing and archiving research data. Dataverse has an active, growing community of developers and users, with 22 installations of the software around the world. The Harvard Dataverse repository alone hosts 70,000 datasets, 330,000 data files, with contributions from more than 500 institutions.
Cloud Dataverse combines Dataverse and OpenStack by storing datasets in OpenStack’s Swift Object storage and replicating datasets from Dataverse repositories world-wide to the cloud(s) -- offering enormous value to both the Dataverse and OpenStack communities. It provides Dataverse users the ability to host larger datasets and efficiently compute on data from around the world using OpenStack’s compute services. It provides OpenStack users with a repository system that is much richer than Amazon’s Public Datasets service.
Dataverse, Cloud Dataverse, and DataTagsMerce Crosas
Talk given at Two Sigma:
The Dataverse project, developed at Harvard's Institute for Quantitative Social Science since 2006, is a widely used software platform to share and archive data for research. There are currently more than 20 Dataverse repository installations worldwide, with the Harvard Dataverse repository alone hosting more than 60,000 datasets. Dataverse provides incentives to researchers to share their data, giving them credit through data citation and control over terms of use and access. In this talk, I'll discuss the Dataverse project, as well as related projects such as DataTags to share sensitive data and Cloud Dataverse to share Big Data.
How Cyverse.org enables scalable data discoverability and re-useMatthew Vaughn
Cyverse.org designs, builds, and operates an innovative, integrated life sciences cyberinfrastructure. It provides data management and analysis capabilities with point and click, cloud, API, and command-line interfaces that engage users of any computing proficiency and is based on an extensible platform that integrates local and national-scale HPC, storage, and cloud resources. Cyverse directly supports thousands of users who store and access over 2PB of research data, use millions of compute hours annually, and participate in the platform's improvement, plus a secondary user community from partner projects that have built atop it. Cyverse is organized around "Data Store" and "App Catalog" services, each of which enables users to upload digital research assets that can be kept private, shared, or made public. Recently, Cyverse has been transitioning from passively enabling digital sharing towards active facilitation. It is partnering with repositories like NCBI SRA to enable direct submission from Cyverse applications, adopting commonly-used ontologies, enabling import/export of virtual machine images, developing metadata-driven persistent landing pages for data sets, and providing DOI (and other identifier) services. These new features are expected to further catalyze growth of an interoperable, interconnected network of shared research infrastructure across the biological sciences.
These are the slides from a plenary panel that I participated in at IEEE Cloud 2011 on July 5, 2011 in Washington, D.C. I discussed the Open Science Data Cloud and concluded the talk by three research questions
Observations on a whole lot of Things learned through the 23 (research data) ...ARDC
The 23 (research data) Things program has enabled hundreds of Librarians, data managers, repository managers, research impact staff and others to learn and apply best practice data management principles through examples, activities and stories.
UK Research Data Discovery Service metadata schemaJisc RDM
An overview of the metadata schema being developed for the UK research data discovery service. Dom Fripp at the Research Data Network event at Cardiff University, May 2016.
ANDS Applications Program: Building Tools to Facilitate Data ReuseAndrew Treloar
Presentation accompanying talk on the ANDS Applications program at IDCC 2016. Discusses the outputs of the program, but also focusses on issues of sustainability of such eresearch tools
This presentation sets out some of the challenges around citing and identifying datasets and introduces DataCite, the international data citation initiative. DataCite was founded on 1-December 2009 to support researchers by
providing methods for them to locate, identify, and cite
research datasets with confidence.
This presentation was given by Adam Farquhar at the STM Publishers Association Innovation Conference on 4-Dec-2009.
Cloud Dataverse: A Data repository platform for an OpenStack CloudMerce Crosas
In the last 10 years, the Dataverse project has been a leader in open-source repository software for sharing and archiving research data. Dataverse has an active, growing community of developers and users, with 22 installations of the software around the world. The Harvard Dataverse repository alone hosts 70,000 datasets, 330,000 data files, with contributions from more than 500 institutions.
Cloud Dataverse combines Dataverse and OpenStack by storing datasets in OpenStack’s Swift Object storage and replicating datasets from Dataverse repositories world-wide to the cloud(s) -- offering enormous value to both the Dataverse and OpenStack communities. It provides Dataverse users the ability to host larger datasets and efficiently compute on data from around the world using OpenStack’s compute services. It provides OpenStack users with a repository system that is much richer than Amazon’s Public Datasets service.
Dataverse, Cloud Dataverse, and DataTagsMerce Crosas
Talk given at Two Sigma:
The Dataverse project, developed at Harvard's Institute for Quantitative Social Science since 2006, is a widely used software platform to share and archive data for research. There are currently more than 20 Dataverse repository installations worldwide, with the Harvard Dataverse repository alone hosting more than 60,000 datasets. Dataverse provides incentives to researchers to share their data, giving them credit through data citation and control over terms of use and access. In this talk, I'll discuss the Dataverse project, as well as related projects such as DataTags to share sensitive data and Cloud Dataverse to share Big Data.
December 9, 2015 NISO Webinar: Two-Part Webinar: Emerging Resource Types - Pa...DeVonne Parks, CEM
Addressing the New Challenges in Data Sharing: Large-Scale Data and Sensitive Data
Mercè Crosas, Chief Data Science and Technology Officer, IQSS, Harvard University
The Dataverse repository framework (http://dataverse.org and http://dataverse.harvard.edu) helps Journals make the data accompanying scholarly articles accessible and citable.
More information at: http://scholar.harvard.edu/mercecrosas/presentations/dataverse-journals
Data Publishing at Harvard's Research Data Access SymposiumMerce Crosas
Data Publishing: The research community needs reliable, standard ways to make the data produced by scientific research available to the community, while giving credit to data authors. As a result, a new form of scholarly publication is emerging: data publishing. Data publishing - or making data reusable, citable, and accessible for long periods - is more than simply providing a link to a data file or posting the data to the researcher’s web site. We will discuss best practices, including the use of persistent identifiers and full data citations, the importance of metadata, the choice between public data and restricted data with terms of use, the workflows for collaboration and review before data release, and the role of trusted archival repositories. The Harvard Dataverse repository (and the Dataverse open-source software) provides a solution for data publishing, making it easy for researchers to follow these best practices, while satisfying data management requirements and incentivizing the sharing of research data.
Data Publishing Workflows with DataverseMicah Altman
By: Mercè Crosas, Director of Data Science at the Institute for Quantitative Social Science (IQSS) at Harvard University
The Dataverse software provides multiple workflows for data publishing to support a wide range of data policies and practices established by journals, as well as data sharing needs from various research communities. This talk will describe these workflows from the user experience and from the system's technical implementation.
This talk was presented as part of the Information Science Brown Bag talks, hosted by the Program on Information Science. (See http://drmaltman.wordpress.com)
Time to Science/Time to Results: Transforming Research in the CloudAmazon Web Services
This session demonstrates how cloud can accelerate breakthroughs in scientific research by providing on-demand access to powerful computing. You will gain insight into how scientific researchers are using the cloud to solve complex science, engineering, and business problems that require high bandwidth, low latency networking and very high compute capabilities. You will hear how leveraging the cloud reduces the costs and time to conduct large scale, worldwide collaborative research. Researchers can then access computational power, data storage, and supercomputing resources, and data sharing capabilities in a cost-efficient manner without implementation delays. Disease research can be accomplished in a fraction of the time, and innovative researchers in small schools or distant corners of the world have access to the same computing power as those at major research institutions by leveraging Amazon EC2, Amazon S3, optimizing C3 instances and more to increase collaboration. This session will provide best practices and insight from UC Berkeley AMP Lab on the services used to connect disparate sets of data to drive meaningful new insight and impact.
FAIR Data Management and FAIR Data SharingMerce Crosas
Presentation at the Critical Perspective on the Practice of Digiral Archeology symposium: http://archaeology.harvard.edu/critical-perspectives-practice-digital-archaeology
Big Data Europe SC6 WS 3: Ron Dekker, Director CESSDA European Open Science A...BigData_Europe
Slides for keynote talk at the Big Data Europe workshop nr 3 on 11.9.2017 in Amsterdam co-located with SEMANTiCS2017 conference by Ron Dekker, Director CESSDA: European Open Science Agenda: where we are and where we are going?
XSEDE National Cyberinfrastructure, NIST, and Supporting NCSI Objectives John Towns
This presentation will provide an overview of the NSF-funded XSEDE (Extreme Science and Engineering Discovery Environment) project. XSEDE provides advanced research computing and data resources and services to the national community supporting open research funded by universities as well as federal agencies and other sources. We will discuss the resources, services and support available to the research community. These resources are also available to NIST researchers and a few recent examples will be highlighted. Finally, we will also look at how XSEDE, as an exemplar research infrastructure, can support the objectives of the NSCI with a focus on the roles the NSF is tasked with.
A Data Ecosystem to Support Machine Learning in Materials ScienceGlobus
This presentation was given at the 2019 GlobusWorld Conference in Chicago, IL by Ben Blaiszik from University of Chicago and Argonne National Laboratory Data Science and Learning Division.
This presentation was provided by Karen Baker, University of Illinois - Urbana-Champaign, during a NISO Virtual Conference on the topic of data curation, held on Wednesday, August 31, 2016
Open Science is a movement to make scientific research, its data and dissemination accessible to all levels of society. This movement considers aspects such as Open Access, Open Data, Reproducible Research and Open Software.
Each of these aspects presents discreteness that need to be evaluated and discussed by the scientific community so that guidelines are established that facilitate the dissemination of scientific information.
The great challenge is to establish effective and efficient practices that allow journals to add these demands in their editorial processes, so as not only to allow data, software and methods to be accessible, but also to encourage the community to do so.
Considering these questions, this panel has as a proposal to discuss important aspects about the advancement of research communication. Some of these aspects are placed in the SciELO indexing criteria, as is the case of referencing research materials in favor of transparency and reproducibility.
Syllabus
FAIR criteria, concepts and implementation; challenges for the publication of data and methods; institutional policies for open data; adoption of TOP guidelines (Transparency and Openness Promotion); software repositories; thematic areas data repositories.
A short review of the new initiatives related to research data management at Harvard University for the CRADLE workshop at IASSIST 2017 (http://www.iassist2017.org/).
Presentation for the workshop on "6 Reasons Fake News is the End of the World as we know it" at Harvard University, organized by the Center for Research on Computation and Society https://crcs.seas.harvard.edu/event/fakenews
Since modern science began, data have been a critical part of the scientific enterprise, not only for conducting science but also for communicating and validating scientific results. From the beginning, it was clear that for the scientific community to continually verify scientific results, the underlying data had to be made accessible. But that has not been, and is still not, always the case. In recent years however, public data repositories have grown significantly, making many research data sets easily accessible to others. The Dataverse project, an open-source software for building repositories to share research data (such as the Harvard Dataverse), has played an important role in making this happen, by giving incentives to researchers to share their own data. In this talk, I will discuss how we got here, and introduce current projects that extend Dataverse to address the next challenges in sharing research data. In particular, I'll present a project that, through integrating Dataverse with remote computing sites, makes large-scale structural biology data widely accessible and helps validate previous results.
Presentation for Harvard's ABCD Technology in Education group:
The Institute for Quantitative Social Science (IQSS) is a unique entity at Harvard - it combines research, software development, and specialized services to provide innovative solutions to research and scholarship problems at Harvard and beyond. I will talk about the software projects that IQSS is currently working on (Dataverse, Zelig, Consilience, and OpenScholar), including the research and development processes, the benefits provided to the Harvard community, and the impacts on research and scholarship.
The DataTags System: Sharing Sensitive Data with ConfidenceMerce Crosas
This talk was part of a session at the Research Data Alliance (RDA) 8th Plenary on Privacy Implications of Research Data Sets, during International Data Week 2016:
https://rd-alliance.org/rda-8th-plenary-joint-meeting-ig-domain-repositories-wg-rdaniso-privacy-implications-research-data
Slides in Merce Crosas site:
http://scholar.harvard.edu/mercecrosas/presentations/datatags-system-sharing-sensitive-data-confidence
Open Source Tools Facilitating Sharing/Protecting Privacy: Dataverse and Data...Merce Crosas
Presentation for the NFAIS Webinar series: Open Data Fostering Open Science: Meeting Researchers' Needs
http://www.nfais.org/index.php?option=com_mc&view=mc&mcid=72&eventId=508850&orgId=nfais
The Rise of Data Publishing in the Digital World (and how Dataverse and DataT...Merce Crosas
Presentation at the National Library of Medicine, in a Symposium organized by the National Data Stewardship Residency, funded by the Library of Congress and the Institute of Museum and Library Services, on "Digital Frenemies: Closing the Gap in Born-Digital and Made-Digital Curation”.
https://ndsr2016.wordpress.com/
A very Brief History of Communicating ScienceMerce Crosas
Mercè Crosas (IQSS, HarvardUniversity) @mercecrosas
An introduction to Force2016 panel on Communicating Science with Steven Pinker, César Hidalgo, and Christie Nicholson
Data Citation Implementation at DataverseMerce Crosas
Presentation at the Data Citation Implementation Pilot Workshop in Boston, February 3rd, 2016.
https://www.force11.org/group/data-citation-implementation-pilot-dcip/pilot-project-kick-workshop
Talk for the workshop on the Future of the Commons, November 18, 2015: http://cendievents.infointl.com/CENDI_NFAIS_RDA_2015/
Slides distributed under under CC-by license: https://creativecommons.org/licenses/by/2.0/
Collaboration in science and technology it summitMerce Crosas
Talk given at Harvard IT Summit, June 4, 2015.
Until recently, the criteria used in assessing and engaging people for the advancement of science and technology have been focused on skills and contributions of single individuals in these fields, and not been carefully evaluated based on their success. As science and technology are increasingly becoming collaborative and social ventures, and it is now seldom the case that the impact of a single individual is crucial, the criteria for and stereotypes of the successful scientific or technical leader should change accordingly. Changing the criteria and stereotypes results in a larger and more diverse talent pool available to advance and lead science and technology, creating teams that not only leverage diverse perspectives, but also are collectively smarter.
Techniques to optimize the pagerank algorithm usually fall in two categories. One is to try reducing the work per iteration, and the other is to try reducing the number of iterations. These goals are often at odds with one another. Skipping computation on vertices which have already converged has the potential to save iteration time. Skipping in-identical vertices, with the same in-links, helps reduce duplicate computations and thus could help reduce iteration time. Road networks often have chains which can be short-circuited before pagerank computation to improve performance. Final ranks of chain nodes can be easily calculated. This could reduce both the iteration time, and the number of iterations. If a graph has no dangling nodes, pagerank of each strongly connected component can be computed in topological order. This could help reduce the iteration time, no. of iterations, and also enable multi-iteration concurrency in pagerank computation. The combination of all of the above methods is the STICD algorithm. [sticd] For dynamic graphs, unchanged components whose ranks are unaffected can be skipped altogether.
Levelwise PageRank with Loop-Based Dead End Handling Strategy : SHORT REPORT ...Subhajit Sahu
Abstract — Levelwise PageRank is an alternative method of PageRank computation which decomposes the input graph into a directed acyclic block-graph of strongly connected components, and processes them in topological order, one level at a time. This enables calculation for ranks in a distributed fashion without per-iteration communication, unlike the standard method where all vertices are processed in each iteration. It however comes with a precondition of the absence of dead ends in the input graph. Here, the native non-distributed performance of Levelwise PageRank was compared against Monolithic PageRank on a CPU as well as a GPU. To ensure a fair comparison, Monolithic PageRank was also performed on a graph where vertices were split by components. Results indicate that Levelwise PageRank is about as fast as Monolithic PageRank on the CPU, but quite a bit slower on the GPU. Slowdown on the GPU is likely caused by a large submission of small workloads, and expected to be non-issue when the computation is performed on massive graphs.
Enhanced Enterprise Intelligence with your personal AI Data Copilot.pdfGetInData
Recently we have observed the rise of open-source Large Language Models (LLMs) that are community-driven or developed by the AI market leaders, such as Meta (Llama3), Databricks (DBRX) and Snowflake (Arctic). On the other hand, there is a growth in interest in specialized, carefully fine-tuned yet relatively small models that can efficiently assist programmers in day-to-day tasks. Finally, Retrieval-Augmented Generation (RAG) architectures have gained a lot of traction as the preferred approach for LLMs context and prompt augmentation for building conversational SQL data copilots, code copilots and chatbots.
In this presentation, we will show how we built upon these three concepts a robust Data Copilot that can help to democratize access to company data assets and boost performance of everyone working with data platforms.
Why do we need yet another (open-source ) Copilot?
How can we build one?
Architecture and evaluation
Learn SQL from basic queries to Advance queriesmanishkhaire30
Dive into the world of data analysis with our comprehensive guide on mastering SQL! This presentation offers a practical approach to learning SQL, focusing on real-world applications and hands-on practice. Whether you're a beginner or looking to sharpen your skills, this guide provides the tools you need to extract, analyze, and interpret data effectively.
Key Highlights:
Foundations of SQL: Understand the basics of SQL, including data retrieval, filtering, and aggregation.
Advanced Queries: Learn to craft complex queries to uncover deep insights from your data.
Data Trends and Patterns: Discover how to identify and interpret trends and patterns in your datasets.
Practical Examples: Follow step-by-step examples to apply SQL techniques in real-world scenarios.
Actionable Insights: Gain the skills to derive actionable insights that drive informed decision-making.
Join us on this journey to enhance your data analysis capabilities and unlock the full potential of SQL. Perfect for data enthusiasts, analysts, and anyone eager to harness the power of data!
#DataAnalysis #SQL #LearningSQL #DataInsights #DataScience #Analytics
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Data and AI
Discussion on Vector Databases, Unstructured Data and AI
https://www.meetup.com/unstructured-data-meetup-new-york/
This meetup is for people working in unstructured data. Speakers will come present about related topics such as vector databases, LLMs, and managing data at scale. The intended audience of this group includes roles like machine learning engineers, data scientists, data engineers, software engineers, and PMs.This meetup was formerly Milvus Meetup, and is sponsored by Zilliz maintainers of Milvus.
Adjusting OpenMP PageRank : SHORT REPORT / NOTESSubhajit Sahu
For massive graphs that fit in RAM, but not in GPU memory, it is possible to take
advantage of a shared memory system with multiple CPUs, each with multiple cores, to
accelerate pagerank computation. If the NUMA architecture of the system is properly taken
into account with good vertex partitioning, the speedup can be significant. To take steps in
this direction, experiments are conducted to implement pagerank in OpenMP using two
different approaches, uniform and hybrid. The uniform approach runs all primitives required
for pagerank in OpenMP mode (with multiple threads). On the other hand, the hybrid
approach runs certain primitives in sequential mode (i.e., sumAt, multiply).
Unleashing the Power of Data_ Choosing a Trusted Analytics Platform.pdfEnterprise Wired
In this guide, we'll explore the key considerations and features to look for when choosing a Trusted analytics platform that meets your organization's needs and delivers actionable intelligence you can trust.
1. Cloud
Dataverse
Mercè
Crosas
(IQSS,
Harvard
University)
with
Orran
Krieger,
Piyanai
SaowaraBtada,
Ata
Turk,
Anuj
Thakur,
Gustavo
Durand,
Leonid
Andreev
MassachuseIs
Open
Cloud
(MOC)Workshop,
December
6
-‐7,
2016,
Boston
University
2. Dataverse
IncenUvizes
Data
Sharing
• A
widely
used
open-‐source
plaWorm
for
building
data
repositories
• Gives
essenUal
incenUves
to
data
authors:
– get
aIribuUon
and
credit
through
data
citaUon
– retain
control
over
data
published
in
the
repository
• Fosters
a
community
to:
– build
new
standards
and
best
pracUces
– increase
research
in
data
sharing
3. Dataverse
repositories
are
installed
in
21
sites
around
the
world
Harvard
Dataverse
repository:
• 63,000
datasets;
12
new
datasets
published
per
day
• 2
Million
data
downloads;
1,500
downloads
per
day
• 15,000
registered
users
• 5000
data
authors
from
500
insUtuUons
4. Data
depositor
Data
users
Metadata
Data
files
Data
+
metadata
Access
object
in
Swi`
+
Compute
with
Sahara/Hadoop
download
Swi`
Object
Store
Dataverse
Now
with
Cloud
Dataverse
Repository
Publish
dataset
Data
Replica3on
5. Cloud
Dataverse
will
be
available
to
any
Dataverse
repository
Each
Dataverse
repository
can
choose
to
enable
the
Cloud
Dataverse
opUon
6. Dataverse
+
MOC
are
being
expanded
to
support
Cloud
Dataverse
2016
Summer
Pilot:
ü
Dataverse
supports
an
external
object
store
ü
Data
are
replicated
from
a
Dataverse
repo
to
MOC
Working
on:
q
From
Dataverse,
user
can
access
object
in
Swi`/S3
+
compute
with
Sahara
and
Hadoop
q
Single
authenUcaUon
to
data
access
and
compute
7. Cloud
Access
+
Compute
This
dataset
has
been
enabled
in
Cloud
Dataverse
