Cloud libraries April 2008

cloud computing concepts

cloud computing concepts

get fuzzier over time ... many different broadly similar definitions abroad

get fuzzier over time ...

many different

broadly similar

definitions abroad

generally means: widely distributed, network based - - storage, - computation, - SaaS models

generally means:

widely distributed,

network based -

- storage,

- computation,

- SaaS models

single datacenter or -hosted apps > moving to network based solutions

single datacenter or -hosted apps >

moving to network based solutions

usually provided by external parties on a quanta basis, e.g. compute cycle or byte traffic

usually provided by

external parties on

a quanta basis, e.g.

compute cycle or byte traffic

a concept of “ virtualization” of infrastructure or web service or application

a concept of

“ virtualization”

of infrastructure

or web service

or application

Poster Child: Amazon’s services -- S3 (simple storage system) and EC2 (elastic compute cloud)

Poster Child:

Amazon’s services --

S3 (simple storage system) and

EC2 (elastic compute cloud)

NY Times converted its TIFF backlist to PDF with Amazon services (EC2 / S3) and bill was ~ $500

NY Times

converted its

TIFF backlist to PDF

with Amazon services (EC2 / S3)

and bill was ~ $500

“ I then began some rough calculations and determined that if I used only four machines, it could take some time to generate all 11 million article PDFs. But thanks to the swell people at Amazon, I got access to a few more machines and churned through all 11 million articles in just under 24 hours using 100 EC2 instances, and generated another 1.5TB of data to store in S3.” - Derek Gottfrid, NYT

“ I then began some rough calculations and determined that if I used only four machines, it could take some time to generate all 11 million article PDFs. But thanks to the swell people at Amazon, I got access to a few more machines and churned through all 11 million articles in just under 24 hours using 100 EC2 instances, and generated another 1.5TB of data to store in S3.” - Derek Gottfrid, NYT

services are offered or increasingly defined by Google, Amazon, Microsoft (maybe), IBM and others.

services are offered

or increasingly

defined by

Google,

Amazon,

Microsoft (maybe),

IBM and others.

Google’s MapReduce break up a problem allocate to many machines reassemble for use

Google’s

MapReduce

break up a problem

allocate to many machines

reassemble for use

[1, 2, 3].map{|i| i * 10} map{} returns new array [10, 20, 30]

[1, 2, 3].map{|i| i * 10}

map{}

returns new array

[10, 20, 30]

map{} could be {image processing} {text search} {metadata extraction}

map{} could be

{image processing}

{text search}

{metadata extraction}

reduce{} transforms output of map{}

reduce{} transforms

output of map{}

It doesn’t matter how many parameters in list every item handled independently from every other item

It doesn’t matter

how many parameters in list

every item handled independently

from every other item

means computation over each item in list can be distributed to many computers eg one VM per item

means computation

over each item in list

can be distributed

to many computers

eg one VM per item

reduce{} combines results

reduce{} combines results

“ ... a new super-group of tech companies [will] dominate the cloud computing market, each of them managing what amounts to a giant centralised computer made up of a number of big datacentres.” - Steve Ballmer, MSFT

“ ... a new super-group of tech companies [will] dominate the cloud computing market, each of them managing what amounts to a giant centralised computer made up of a number of big datacentres.”

- Steve Ballmer, MSFT

“ NSF Partners With Google and IBM to Enhance Academic Research Opportunities” (Press Release 08-031) even very high end universities lag behind corporate computing

“ NSF Partners With Google and IBM to Enhance Academic Research Opportunities”

(Press Release 08-031)

even very high end universities

lag behind corporate computing

“ Fundamental changes in computer architecture and increases in network capacity are encouraging software developers to take new approaches to computer-science problem solving.”

“ Fundamental changes in computer architecture and increases in network capacity are encouraging software developers to take new approaches to computer-science problem solving.”

“ In order to bridge the gap between industry and academia, it is imperative that academic researchers are exposed to the emerging computing paradigm behind the growth of "Internet-scale" applications.”

“ In order to bridge the gap between industry and academia, it is imperative that academic researchers are exposed to the emerging computing paradigm behind the growth of "Internet-scale" applications.”

IBM is deeply committed

IBM is deeply committed

IBM Kittyhawk http://domino.research.ibm.com/ comm/ research_projects.nsf/ pages/ kittyhawk.index.html

http://domino.research.ibm.com/

comm/

research_projects.nsf/

pages/

kittyhawk.index.html

IBM Kittyhawk "We postulate that efficient, balanced machines with high-performance internal networks such as Blue Gene are not only significantly better choices for web-scale companies but can form the building blocks of one global-scale shared computer. Such a computer would be capable of hosting not only individual web-scale workloads but the entire Internet. ...”

"We postulate that efficient, balanced machines with high-performance internal networks such as Blue Gene are not only significantly better choices for web-scale companies but can form the building blocks of one global-scale shared computer. Such a computer would be capable of hosting not only individual web-scale workloads but the entire Internet. ...”

in other words: not your father’s datacenter

in other words:

not your father’s

datacenter

“Hiding in Plain Sight, Google Seeks More Power” Photo Credit: Melanie Conner for The New York Times

cloud computing has already revolutionized many content based information services

cloud computing

has already

revolutionized

many content based

information services

Google Books virtualizes libraries

Google Books

virtualizes

libraries

Flickr virtualizes image collections

Flickr

virtualizes

image collections

iTunes virtualizes audio recordings

iTunes

virtualizes

audio recordings

YouTube virtualizes video

YouTube

virtualizes

video

Is there space for HE to contribute to virtualization layers ?

Is there space for HE

to contribute to

virtualization layers ?

Most new initiatives do not directly involve research libraries as central constituents

Most new initiatives

do not directly involve

research libraries as

central constituents

OpenLibrary

OpenLibrary

Zotero and Internet Archive

Zotero and Internet Archive

LibraryThing

LibraryThing

If (ARTStor) did (Flickr) ...

If (ARTStor) did (Flickr) ...

libraries provide fairly lightweight applications or “soft infrastructure”

libraries provide fairly

lightweight applications

or “soft infrastructure”

e.g. faculty repository systems online publishing platforms boutique special collections

e.g.

faculty repository systems

online publishing platforms

boutique special collections

not fundamental infrastructure provision

not

fundamental

infrastructure

provision

power is in aggregation

power is in aggregation

but there are problems with library collaboration at scale

but there are

problems with

library collaboration

at scale

(it doesn’t happen)

(it doesn’t happen)

similar to classic political science issues on sovereignty

similar to classic

political science issues

on sovereignty

EU one of the most successful trans-national political efforts

EU one of the most successful

trans-national political efforts

Europe tightly integrated due to - physical and - historical and - economic issues

Europe

tightly integrated

due to

- physical and

- historical and

- economic issues

consider the difficulties establishing League of Nations

consider

the difficulties

establishing

League of Nations

and post WW-II, United Nations

and post WW-II,

United Nations

United Nations (as it exists today) not (at all) what its planners wished

United Nations

(as it exists today)

not (at all) what its

planners wished

Tremendous conflict arose in San Francisco between Russia (Molotov) other Allied war parties

Tremendous conflict

arose in San Francisco

between Russia (Molotov)

other Allied war parties

Even in the face of the possibility of world-ending NUCLEAR WAR

Even in the face

of the possibility

of world-ending

NUCLEAR WAR

Libraries have little beyond established workflow interoperation like ILL

Libraries have little beyond

established workflow

interoperation

like ILL

a few dominance hierarchies like the CIC or OhioLink or the CDL which do well in core operations

a few dominance hierarchies

like the CIC

or OhioLink

or the CDL

which do well in core operations

OCLC I see primarily as a vendor in important but established bibliographic service space

I see primarily as a vendor

in important but established

bibliographic service space

Difficult to establish - much less maintain - a  of large libraries with serious long term resource commitments

Difficult to establish

- much less maintain -

a  of large libraries

with serious long term

resource commitments

maybe the best model is support of the equivalent of something like the IAEA

maybe the

best model is

support of

the equivalent of

something like

the IAEA

contribute - dollars - staff - expertise - technology

contribute

- dollars

- staff

- expertise

- technology

working with others but relying on an independent organizational framework for action

working

with others

but relying on

an independent

organizational

framework

for action

a possible role for the DLF “ marriage broker”

a possible role

for the DLF

“ marriage broker”

make introductions between nascent efforts and potential partners

make introductions

between nascent efforts

and potential partners

like libraries and museums and archives and tech firms and societies

like libraries

and museums

and archives

and tech firms

and societies

figure out domain- or discipline- based early-stage faculty teams seeking to develop new services

figure out domain- or discipline-

based early-stage faculty teams

seeking to develop new services

aid and abet -- encourage faculty and professional societies to assume leadership

aid and abet --

encourage faculty and

professional societies

to assume leadership

current thrusts for the DLF ...

current thrusts

for the DLF ...

mobile apps virtual worlds moving images science CI

mobile apps

virtual worlds

moving images

science CI

focus today on mobile : fundamentally reliant on net- scale applications

focus today on mobile :

fundamentally reliant on

net- scale applications

mantra connect communicate collaborate (CCC)

connect

communicate

collaborate

(CCC)

mobile apps iPhone Android wholly game changing development platforms

iPhone

Android

wholly game changing

development platforms

coming : social collaborative learning environments

coming :

social collaborative

learning environments

wild assed designs portable networked computing interfaces

wild assed designs

portable

networked

computing

interfaces

petiteinvention

areas to watch Rudy de Waele Plugg.eu - March 19, 2008 dot.Open

sensoring biometrics transactions lifestreaming recommendation image recognition augmented reality mobile connected games location based social media retail proximity media consumption Rudy de Waele Plugg.eu - March 19, 2008 dot.Open

sensoring

biometrics

transactions

lifestreaming

recommendation

image recognition

augmented reality

mobile connected games

location based social media

retail proximity media consumption

Rudy de Waele Plugg.eu - March 19, 2008 dot.Open

in  ... where ? libraries in the future

in  ...

university libraries will not band together in virtual organizations to establish bigger, better libraries offering cloud based information services

university libraries will not band

together in virtual organizations

to establish

bigger, better libraries

offering cloud based

information services

probably this is against their institutional nature

probably this

is against their

institutional nature

a significant difficulty in moving into wholly different domain areas (like mobile)

a significant difficulty in

moving into wholly different

domain areas (like mobile)

IN NEW ecological niches zero platform expertise no collaboration : industry a paradigm shift in IT

IN NEW ecological niches

zero platform expertise

no collaboration : industry

a paradigm shift in IT

 libraries  MUST STOP defining solutions (in terms of)  libraries 

 libraries 

MUST

STOP defining

solutions

(in terms of)

 libraries 

libraries must learn how to : partner || collaborate || contribute || coordinate

libraries must

learn how to :

partner ||

collaborate ||

contribute ||

coordinate

via organizational frameworks - external to us - friendly to us - supported by us - sponsored by us (but not us.

via organizational frameworks

- external to us

- friendly to us

- supported by us

- sponsored by us

(but not us.

that’s a different kind of library more diverse more opportunistic more open

that’s a different

kind of library

more diverse

more opportunistic

more open

it’s a smaller library it’s a more nimble library it’s a library of pieces broken up from something unwieldy lethargic stifling

it’s a smaller library

it’s a more nimble library

it’s a library of pieces

broken up from something

unwieldy lethargic stifling

( It should be a more fun LIBRARY )

( It should be

a

more fun

LIBRARY )

thanks peter at diglib.org naypinya {twitter, twine, skype}