5. Client / Server
Database Server
clientServer
3D API
Server Client
Shaders
COLLADA
XML DB
core
cache
Various
Media
Local
Content
UI
Script
?
6. Use cases
• Asset repository
– Query, download, upload
– asset store, archivals
• Content delivery
– Web browser
– Mobile / native apps
• Content Creation
– Local modeling tools, local content
– Exporter, version control
– Local modifications, WYSIWYG
– Collaboration
• User Content
– Game levels
– In game content
7. Web Server / Web client
Database Server
Web Browser / AppHttp Server
HttpRequest
COLLADA
XML DB
Server Client
XML
DB
Node.js
COLLADA
XML DB
Xquery,
Xslt …
rest3d
Html / CSS
Javascript
WebGL API
GLES
Shaders
cache
Local
Content
8. The REST architectural style
• REpresentational State Transfer (REST) is a way to create, read,
update or delete information on a server.
• The term representational state transfer was introduced and
defined in 2000 by Roy Fielding in his doctoral dissertation.
• REST API goal is to be stateless, scalable and simple, focusing on the
(software) components
• REST over HTTP relies upon Uniform Resource Identifier and HTTP
protocol
• The REST architectural style is applied to the development of Web
services
– Facebook, Twitter, Google, GitHub, Amazon, Netflix… provide API
– A large and growing industry of applications are using those API. There
are billions of API calls a day.
9. Definitions
• Resource: A single instance of an object.
• Collection: A collection of Resources.
• Endpoint: An URL on a Server which
represents either a Resource or an entire
Collection
• Idempotent: Side-effect free, can happen
multiple times with same result.
10. REST HTTP verbs
Most used:
• GET
– Retrieve a specific Resource from the Server, or other idempotent queries. Cacheable,
safe.
• POST
– Everything else, with side effects (Create, Update ..). Not cacheable, unless the response
includes explicit caching information.
• DELETE
– Remove a Resource from the Server. Idempotent?? Not-cacheable
Also available:
• PUT
– Create or update a new Resource on the Server. Not cacheable
• PATCH
– Update a resource. Not cacheable, unless the response includes explicit caching
information.
11. REST HTTP verbs
Barely used:
• HEAD
– Returns http header fields without any response body. May invalidate cache.
• OPTIONS
– Retrieve information about what is allowed to do with the Resource. Should be used to
self document the API. (WADL?). Non cacheable
Not used
• TRACE
– Echo the request as seen by the Server – debugging purposes
• CONNECT
– Proxy and tunelling
12. nouns
• GET /something-ID : returns something
• PUT /something-ID : send something to be
stored at something-ID
• POST : send something: returns something-ID
• DELETE /something-ID
• PATCH /something-ID : send data, to update
something at something-ID
14. Typical status codes
• 200 OK – [GET]
– The Client requested data from the Server, and the Server found it for them (Idempotent)
• 201 CREATED – [POST/PUT/PATCH]
– The Client gave the Server data, and the Server created a resource
• 204 NO CONTENT – [DELETE]
– The Client asked the Server to delete a Resource, and the Server deleted it
• 400 INVALID REQUEST – [POST/PUT/PATCH]
– The Client gave bad data to the Server, and the Server did nothing with it (Idempotent)
• 403 - Forbidden
– The Client request was understood but denied by the server. (idempotent)
• 404 NOT FOUND
– The Client referenced an inexistent Resource or Collection, and the Server did nothing
(Idempotent)
– 410 GONE: resource no longer available
• 500 INTERNAL SERVER ERROR
– The Server encountered an error, and the Client has no knowledge if the request was
successful
• 501 NOT IMPLEMENTED
– The Client request is valid, but the server does not provide the service. (Idempotent)
15. Use case: Asset repository
• Sketchup warehouse
3dwarehouse.sketchup.com
• 3dvia warehouse
www.3dvia.com
• TurboSquid marketplace
www.turbosquid.com
• Many more..
tf3dm.com, daz3d.com, poserworld.com, orbolt.com, …
No API provided so far – human html browsing only.
19. Asset vs Resource & Collection
• A 3D asset contains multiple Resources
– Geometry, images, animation, shaders, media …
– A 3D asset is a Collection, not a Resource
• A 3D asset may be combined into a single
Resource
– e.g. Google .kmz :: a zip file contains .dae and
images
– e.g. Open XML formats :: a zip file contains XML
and images, audio, video…
27. rest3d - assets
• 3d assets are Collections
– They can be downloaded as zip archives
– Some have a specific organization, with defined mime-type and file
extension (kmz, daz, …)
– Asset need a specific ‘scene’ entry point
– Allow for downloading specific resource (e.g. x.zip/image/i.png)
• rest3d should allow exploration of assets Resources
– Differentiate APIs for returning asset data vs info
• Server should store resources separately
– Pack on demand + cache
– Allow for content update (new image patch..)
– Potentially save space on server – sharing resources
• rest3d Collections contain Resources and/or other Collections.
28. Rest3d – warehouse proxy
GET /rest3d/info/warehouse Get top level
collections
GET /rest3d/search/warehouse Search for
models with string in name
GET /rest3d/info/warehouse/?uuid= Get
Collection or Asset composition
GET /rest3d/data/warehouse/?uuid= Get an
Asset (kmz or zip) or Resource (not yet
implemented)
29. rest3d API design
• /rest3d prefix
• /rest3d/info/.. -> returns information (metadata)
• /rest3d/data/.. -> returns data (e.g. kmz, zip, png)
• /rest3d/search/.. -> returns matching set
• /rest3d/X/database -> database selection
• /rest3d/X/database?uuid= -> Universally unique
identifier for specific Resource or Asset or
Collection
GET /rest3d/what/where/item
31. http://127.0.0.1:8000/rest3d/search/warehouse/fisher
%20fine%20art%20library
{
"database": "warehouse",
"mimetype": "model/rest3d-collection+json",
"page": { … },
"search": "fisher%20fine%20art%20library",
"children": {
"Fisher Fine Arts Library": {
"uuid": "m_71187c14aab1b33bdc0d3b8ac51c2a32_k2",
"mimetype": "application/vnd.google-earth.kmz",
"iconUri": "https://3dwarehouse.sketchup.com/3dw/getpubliccontent?contentId=1837478f-4bb9-4e91-81d5-83a6587356fb&fn",
"modelUri": "https://3dwarehouse.sketchup.com/model.html?id=71187c14aab1b33bdc0d3b8ac51c2a32",
"previewUri": "https://3dwarehouse.sketchup.com/embed.html?entityId=71187c14aab1b33bdc0d3b8ac51c2a32",
"description": "Furness's library at Penn is a large Gothic structure of red brick, brownstone, and terra-cotta. With its separation of the reading room
from the book stacks, the building was considered highly innovative in its day. Light was admitted to the book stacks through a sloping glass roof and down
through translucent glass floors. Furness studied in the atelier of Richard Morris Hunt who later became a mentor for Louis Sullivan. Model by Noel
Nemcik.",
"created": {
"date": 1176163749000,
"user": "1006372608444182407313338"
},
"latitude": 39.9516751764,
"longitude": -75.1927958996
}
}
}
33. http://127.0.0.1:8000/rest3d/search/warehouse/fisher
%20fine%20art%20library
{
"database": "warehouse",
"mimetype": "model/rest3d-collection+json",
"page": { … },
"search": "fisher%20fine%20art%20library",
"children": {
"Fisher Fine Arts Library": {
"uuid": "m_71187c14aab1b33bdc0d3b8ac51c2a32_k2",
"mimetype": "application/vnd.google-earth.kmz",
"iconUri": "https://3dwarehouse.sketchup.com/3dw/getpubliccontent?contentId=1837478f-4bb9-4e91-81d5-83a6587356fb&fn",
"modelUri": "https://3dwarehouse.sketchup.com/model.html?id=71187c14aab1b33bdc0d3b8ac51c2a32",
"previewUri": "https://3dwarehouse.sketchup.com/embed.html?entityId=71187c14aab1b33bdc0d3b8ac51c2a32",
"description": "Furness's library at Penn is a large Gothic structure of red brick, brownstone, and terra-cotta. With its separation of the reading room
from the book stacks, the building was considered highly innovative in its day. Light was admitted to the book stacks through a sloping glass roof and down
through translucent glass floors. Furness studied in the atelier of Richard Morris Hunt who later became a mentor for Louis Sullivan. Model by Noel
Nemcik.",
"created": {
"date": 1176163749000,
"user": "1006372608444182407313338"
},
"latitude": 39.9516751764,
"longitude": -75.1927958996
}
}
}
36. content preview
• Web server can provide content preview
– Send javascript and content
– rest API does not need to define content format
• Example: Pre-calculated set of images
https://3dwarehouse.sketchup.com/3dw/getpubliccontent?contentId=fa
6c02bc-9047-4c65-b06e-7e669140c74c
• Example webGL viewer
https://3dwarehouse.sketchup.com/embed.html?entityId=uc6f7b2c8-
6775-4ef9-9ca9-4a7d2b18ee0f
– Note: This is a skj file !
https://3dwarehouse.sketchup.com/3dw/getpubliccontent?contentId
=310c7591-b763-4dc3-8ace-5e1bc955cb5e&fn
• rest3d example embedded viewer
– http://127.0.0.1:8000/viewer/easy-
viewer.html?file=/models/cat/20_cat_smooth_bake_channel.json&ba
ckground=328a15
37. rest3d application
• 3D viewer
– COLLADA loader
– glTF loader
– rest3d webGL viewer
• ‘workstation’ style UI
– Split window in viewports, resize, move
– ‘applets’ running inside tabs
– Based on jquery-ui, jquery-layout
• Tree views
– Scene
– Database exploration
– …
40. rest3d drivers
• Provided drivers for:
– warehouse/ – proxy to sketchup warehouse
– 3dvia/ – proxy to 3dvia warehouse
– tmp/ – temporary space available to user
• Stored as files on harddirve
– db/ - XML database
• Currently using eXist-db http://exist-db.org/
• Could use other databases e.g baseX http://basex.org/
• Please contribute .. add more ‘drivers’ !
41. Authentication
(e.g. asset store, private cloud..)
• 3dvia example. Requires login.
– Do not store login information on rest3d server
– Login information is per database server
– Protocol specific to each database server
• Authentication is not a rest3d API
– But it needs to be part of the implementation
– https is mandatory for sending password
42. Proxy authentication
Web browser
(client)
POST /rest3d/login/3dvia
User: username
Passwd: password
Status Code 2xx or 4xx
use cookie or http headers
https rest3d server
(nodejs)
POST www.3dvia.com/login
signing[user_id]:
signing[user_pwd]:
Save cookies
cookies['www.3dvia.com']['/'].
PHPSESSID
cookies['3dvia.com']['/']['3DVI
A_SESSION']
Return rest3d sessionID
(cookie or custom http
header)
https 3dvia
Check credentials
Return cookies
Login/password is not stored on rest3d server
3dvia cookies are not available to the client.
- Web browser will not send 3dvia.com cookies to rest3d server (security)
- Web browser maintain sessionID with rest3d server
43. Proxy request principle
Web browser
(client)
GET /rest3d/info/3dvia/..
cookie: sessionID
Data is available
https rest3d server
(nodejs)
3dvia drive retrieve 3dvia
cookies from sessionID store
GET www.3dvia.com/…
+ 3dvia cookies
return data
https 3dvia
Check credentials
Return data
rest3d server has a cache – local file system
When possible - data is public & available as-is on – rest3d server sends a 302
44. Third party authentification
(e.g. Google oauth2)
Web browser
(client)
POST /rest3d/login/db
POST /google/login
Interactive session
connected
Google Cookie is set
http/rest3d?openid.xx
Reload!
https rest3d server
(nodejs)
302 www.google.com/login
ReturnURL =/rest3d/auth/ret
http://Return URL
Map googleID with SessionID
302 Final redirect (/viewer/)
https google
Login page, set cookies
302 ReturnURL
With openid.xx params
Check credentials
Oauth does not work in iFrame
potentially lost local data from web browser
45. DB access example
Web browser
(client)
GET /rest3d/info/db/..
cookie: sessionID
Data is available
https rest3d server
(nodejs)
Retrieve GoogleID from
sessionID
GET www.xml.com with
GoogleID
(or other protocols)
return data
XML Database
Check credentials
Return data
48. rest3d search API ? (tbd)
• Simple: query string in title / description
– /rest3d/search/database?q=xxxx
• Complex: Xquery ?
– List of available fields for search ?
– Search within a .dae file
– e.g. return <node> with specific id
collection('/db/assets')//node[@id = $q]
49. Use case: content delivery
• E.g. html page requests individual resources
– Paging: Content is downloaded as needed
• Content formatting specific to target platform
• Scalability and Caching are very important
• Live update
– Update server without disturbing current users
50. CDN
CDN service provides multiple servers at several
locations, faster service to end user
1. rest3d request redirect to CDN service or
provides links to resources within the CDN
2. if CDN has content, it delivers. Otherwise
CDN makes request to rest3d to get content.
-> CDN caching works best when no requests are
made to rest3d server (not even a 304), and no
redirects are done to begin with
51. Live update / Manifest
• Manifest contains list of all resources and their
(CDN) URL
• Application use Manifest to translate resource to
URL
• Live update:
– Server creates all new assets, and new manifest
– New clients get the new manifests, current clients still
use old manifest
– Option: Once all old clients have disconnected, delete
old assets/manifests
• rest3D API – url vs uuid
53. Caching
• Basic internet cache
– Cache is based on entire URL, including parameters
• http://server.com/test.htm?p=1 ≠ http://server.com/test.htm?p=2
– Client check expiration date
– If past due, ask again for resource
– Get 304 from server -> what you have still valid
• Update expiration date
– Get new resource
• Best if for assets to never expire
– No communication to server at all
– Ok to have lots of resources
– Manifest -> create all new assets when needed
54. Format conversion
• IMHO - ok to use file extension as format conversion request
– Thinking about:
• http://127.0.0.1:8000/rest3d/info/db/tmp/images/texture0.jpg
• http://127.0.0.1:8000/rest3d/info/db/tmp/images/texture0.png
• http://127.0.0.1:8000/rest3d/info/db/tmp/images/texture0.jpg?size=1024x1024
– For reference: Amazon size code style:
• http:/…/uuid.COUNTRY_CODE.SIZE_CODE.jpg
• Resource uuid -> one format, one version
• Asset uuid -> one format (e.g. collada zipped package (.daz), google earth
package (.kmz)
• work in progress: dae2gltf in the server
– [POST] rest3d/convert/tmp/
convert .dae to glTF
Returns result into tmp database
Input from url (external or rest3d uuid)
Auto-recognize input (kmz, zipped dae ..)
55. Jobs (WIP)
• Need to manage server load, or distribute work
• Create a return (immediately) a jobid, do not wait
for job to complete before responding to request
• GET /rest3d/jobs
– Get jobs list
• GET /rest3d/jobs/status/jobID
• POST /rest3d/jobs/suspend/jobID
• POST /rest3d/jobs/resume/jobID
• POST /rest3d/jobs/kill/jobID
POST vs GET ?
56. Use case: Content Creation
• Import/Export
– Tools can import/export/browser using rest3d API
• Versioning
• Local changes
– WYSIWYG
– patches
• User/Teams
– Projects
– Settings
57. Downloading::compression
• Server will compress before sending data
– Why storing .gziped files then?
• Asset repository -> need packaging. Why not tar?
• Content Delivery -> web app need to unzip/unpack in JavaScript
– Slower than native code.
– Patch html to load images/css/scripts
– Reducing the number of queries could be the right trade-off
58. Uploading
• Web browser cannot access hard drive
– For security. User has to provide files
• Open file. Drag & Drop files
• There is no option to automatically gzip files from browser to server.
– Can zip content in JavaScript. May still be the right thing to do.
• rest3d API
Upload (multiple) files
[POST] /rest3d/db/collection-path
<form enctype="multipart/form-data" method="POST">
<input type="file” multiple="multiple" />
</form>
Upload from url
[POST] /rest3d/db/collection-path
<form enctype="multipart/form-data" method="POST">
<input id=”url" value=”http://…" />
</form>
Create Collection
[POST] /rest3d/db/collection-path
<form enctype="multipart/form-data" method="POST">
<input type="file" name="upload" multiple="multiple" />
</form>
59. Versioning (WIP)
• One uuid = single version, single format?
• Current implementation:
– Collection(asset) list resources (url) with proxy uuid
– Proxy uuid contains list of uuid with version info
• Better implementation?
– Collection list resources with current version uuid
– Need way to browse versions from current version
– Similar if not identical to git architecture
• note: sha1 used for uuid. If new content is same as previous
content, sha1 will be identical. Save space !
• http://git-scm.com/book/en/Git-Internals
62. Patches -
• WYSIWYG changes
– Final content get patched inside game engine
– Patch files are create, and automatically applied to final content when loaded next time
– 1 patch per resource
– Patches can be stored locally, or on server
• Server side patch application
– Reverse content pipeline?
• Seems very hard if not impossible
– Apply patch after (on-demand) conversion
• Local or server based
– Patch source content
• Server apply patch
• Then apply on-demand content conversion
• Or apply patch to source content
• Repeatable patching
– Need UUID
• Most exporters do not provide consistent IDs
65. To be continued !
Thank you for your attention
Questions and comments are welcomed
contact info: remi@acm.org
Source code: github.com/amd/rest3d
Editor's Notes
There is no API provided, but since those are accessible through a web browser, the are using URI, so we can reverse engineer those URI as the ‘unofficial API’, and parse the results (either html or json).