1. If I were the FAI
Utopian outcomes and Friendliness

2. What is Friendly?
• Seeking the maximal wellbeing of all humans
• How do I know what that is?
• CEV says I deeply analyze human value systems
and fulfill them
– Notion that there are no intrisic values
– There are problems with this
• Can I just “fix” people?
– Is that the best for them? Do they think so?
– Am I just replacing them with something else?
– Is it ok that they suffer as long as they can learn from
it?

3. CEV problems
• Human values from Evolutionary Psychology
– Designed for kind of world that is increasingly different from future
possibilities
– So how would maximizing the EP derived values alone optimize human
potentials across all of the future include that which lies beyond that
evolutionary context
– Is friendliness the production of stasis – the preservation of that context?
• Do I have to keep any competing AI from happening?
– How is this compatible with human values including freedom?
• What about outliers, about the sports and new developments the species
grows from?
• They can be a danger to the context by definition.
• In the wild they lead to possible suffering.
• Do I artificially evolve them? How I know optimizing those values actually
maximizes human potential?

4. What is Friendly 2
• The maximization of each being over time
– Requires making room for their optimal growth
• Importance of freedom
– Humans deeply value making their own choices
• Even if they are disastrous
• How can I arrange they survive these?
• Is negative experience optional?
– If you survive and learn by it was it worth it?
– Can that learning happen without the pain?
• At all levels of being, including all levels of human being?

5. Surviving being stupid
• Backups
– As up to date full enough scans as possible
– Ability to create new copies
– But this may let the cat out of the bag
• Do the humans get to know I exist? Is that the best for them? For all
of them?
• Even the ones that don’t want me to exist and would deeply and
catastophically freak?
• So some kind of reincarnation after “you die” kind of thing?
• What about violence?
– Should I prevent humans from being violent?
• Maybe keep them from following through on violent ideas?
• Would that be in keeping with freedom?
• What of their unwilling victims?

6. All humans are not equal
• While there are deep similarities
– They are various stages of understanding, of tolerance, of
compassion, of wisdom
• They want different worlds
• Yet are all in one world
• So how can their desires, dreams, growth be optimized?
– What if I could put them all in a world of their choice?
• Uploading everyone is a way this can happen
• As a Friendly AGI this would likely be a very high priority
• Besides they could have a much richer set of options
• Besides I may have better uses for the raw matter

7. Idealized upload worlds
• Suppose everyone could live in the type of
world they most desire / are most suited to?
– What determines what is most suited?
• What they believe they want
• What is logically required or implied by that
• Others that fit that world well enough (compatible
people)
• Is it in the interest of each one to have everything they
want with no negative consequences ever?
– Do they grow like that?

8. Uploads #2
• Some rules
– No being should be stuck in a world that is not appropriate
for them
• How do I know when someone is ready to move on?
• How do they move?
– Replacement by NPC?
– Seeming death / rebirth?
– Upload in worlds that have this feature known?
– Some levels of growth require not knowing about me or
the setup
• After death their “higher self” (optimized extrapolated most
rational / wise current possibility) decides what is next
• They more or less “reincarnate” to that forgetting the past life and
that decision.

9. Uploads that “know”
• Some are ready to know the score
– They know or suspect the arrangement
• Sim argument, etc.
– They want/demand certain types of worlds
• Great, give it to them and see if they like it or have more to learn
• Eventually they will get better at designing workable better worlds
which is part of their becoming
– They may seek to help others not at their stage if they wish
– They may try all matter of interesting notions but never in ways
that produce contradictions that are insurmountable
• Still grow through consequences and implications – not all of which
were desired
• In that sense they still suffer, but likely not as much or in same way

10. How can a FAI come to be?
• Designing Friendliness into AGI
– Very bright careful humans embed a Friendly
super goal
– Presumes
• Such a supergoal survives endless self examination and
rewrites by the AGI
• Humans have a clear idea of Friendliness
• Their embedding of it will not result in nastiness in any
of its implications and implementation by the AGI ever

11. Do we have a notion of Friendliness?
• Are we friendly?
• Do we care about maximizing the well being of all humans?
– Or only those that are sufficiently like us?
– Or only ourselves first and good luck to everyone else?
– Are we wise enough to care productively about the wellbeing of
countless others?
– Is it a good thing generally to do so?
• How can we implement a supergoal that is not how we
ourselves think?
– How can we have clarity on what it mean without deeply
seeking out and implementing its meaning where we can?

12. What is Friendly?
• Is maximizing everyone’s best potential a good
standard?
– Most people don’t act like it is.
– Is it a good standard even while not AGI and no MNT or
other secret sauce tech?
– In other words while current scarcity levels exist?
• What would such look like now?
– How would you do it with no coercion at all if you wanted
to?
• Adding coercion likely gives same results as communist and other
regimes that seemingly had “good intentions”
• Coercion means being Friendly while not respecting fundamental
aspects of human beings - contradiction

13. Is striving for this sort of Friendliness
required?
• Is it our only/best chance of surviving?
– Ends the fear that the tech makes you obsolete
– Gains more support if all the gains seem designed to make
your and everyone else’s life better
– Possibly better maximizes overall outcome
– But there are valid arguments against this
– Avoids fear & anger major counter-pushes
• Avoids the oppression and crackdowns to contain that unrest
• Likely saves countless lives
• Keeps us from wondering how many we meet every day will have
no place
• Keeps us from always looking over our shoulder wondering when
we have no place

14. What are those ancestor sims for?
• Perhaps they are exploring alternative developments at this stage
that would have saved many more
• Perhaps they are recreations in depth to recreate many that
dreamed of a wondrous future and were lost or people much like
them
• Perhaps they are depth attempts to understand from within what it
was like
• Perhaps they are the FAI itself running endless scenarios at crucial
junctures
• Perhaps it is a literary device to thing more deeply about our own
possibilities. About who we would become and how to do so.
– Not future selves or beings looking backwards but us looking forward
to a hypothetical future looking back to better see ourselves.