Short discussion on some of the most recent works on generating 3d world layouts using Probabilistic generative models and machine learning

1. Synthesizing 3D Worlds
-Baskar Rethinasabapathi

2. Why do we need this?
-laborious task; saves designer time
-Building an avgscene takes upto20 minutes

3. Why do we need this?
-laborious task; saves designer time
-Building an avgscene takes upto20 minutes
-constant need for variety

4. Why do we need this?
-laborious task; saves designer time
-Building an avgscene takes upto20 minutes
-constant need for variety
Systems find it difficult
-High dimensional ; complex to model
-Object properties differ
-Constraints based
-Interactive and dependent

5. Existing approaches
-Procedural modeling
-Should have a structural grammar
-Component based
-Not suited for scenes
-Evolutionary
-Repetitive; less creative
-Probabilistic
-To be discussed

6. Potential application
–Creating indefinite virtual worldsNo man’s sky –Procedural http://youtu.be/ZVl1Hmth3HE?t=28s

7. Problem domains addressed
System 1-Example-based Synthesis of 3D Object Arrangements
-focus on local arrangementsSystem 2 -Synthesizing open worlds with constraints using locally annealed reversible jump MCMC
-focus on open world arrangementsPaper Source
1.http://dl.acm.org/citation.cfm?id=2366154
2.http://dl.acm.org/citation.cfm?id=2185552

8. High-Level Challenges-output should be highly plausible/agreeable
-should generate a very large variety
-Users should not be involved much

9. General approaches1. Example based
2. Constraints basedDataset
-Objects created
-from Google 3D warehouse
-by users and annotated

10. Goal of System 1
-Users should judge synthesized scenes to be highly plausible compared to hand created ones
-At least one out of every three synthesized results should be usable

11. System design 1
Scene-> Contextual + User example -> Learneddatabase Categories Input scenes Mixed ModelProbabilistic Mixed Model =
Occurrence model + Arrangement model

12. Goal of System 2
-Should be more applicable on open world layouts
-Optimal object arrangements which is diverse enough

13. System design 2
Scene database -> Extract system of constraints -> Generate Factor graphsRepeated constraints in Scene -> novel MCMC method -> new scenes

14. System 1 -Algorithm
-Build a bipartite matching graph between objects in image
-Edge cost proportional to = same basic category+ smaller distance
-Max weight = Scene’s static support + Size of Objects + Edge costClustering from the database
-Randomly choose anchor object –initialize a basic cluster
-Align with = neighborhood similarity + geometric features
-Merge clusters

15. Interesting sub problems
System 1
-Finding contextual neighborhood
-How do we find a cookie on a plate in image A is interchangeable with bread on image B?
-Look for forks and knives around
-Similar in type and arrangement
-Finding hierarchical static support
-Cookie should be on a plate, not on the bed
-Filling gaps in user examples
-Construct a Bayesian network from database

16. System 2 -Algorithm
System 2
-Defining constraints (π1)
-numPlates= randInt(1,10)
-Size_Table= Uniform(15,25)
-Position_Plates= uniform(-10,10),uniform(-10,10)
-Size_Plate= uniform(0.1,2)
-Building factor graphs
-70% of the table should be occupied (π2)
-softEq(0.7, total_area(plates)/ area(table))
-Plates should be inside the table (π3)
-For all i, softEq(0.0, area_outside_table(plate_i))
-Total constraints model = π1*π2*π3

17. System 2 -Algorithm
System 2
-Markov Chain
-Next state is dependent only on present state
-Monte Carlo Markov Chain
-The system of Markov chains attains an equilibrium eventually
-Reversible Jump MCMC
-Should accommodate trans-dimensional constraints
-Locally annealed Reversible Jump MCMC
-Do not accept/reject new constraints immediately

18. Demo

19. Analogy with other papers
Bricolage -example based learning

20. Discussion
-How usable are the scenes generated from both systems?
-Why can’t System 1 be scaled to open world layouts? Or, vice versa. Can you apply System 2 for local arrangements?

21. Discussion
-Consider the Coffee Shop Layout. If I want all kinds of fabric in the scene to be changed to leather, how do I achieve?
-Which of the two systems is more suitable for this problem?

22. Results
-Yes. Goals achieved
-output should be highly plausible/agreeable
-Should generate a very large variety
-Users should not be involved much
-Users agreed with synthesized results of System 1
-No evaluation in System 2
-Focus on optimization compared to other greedy algorithms

23. Critique