This document discusses several challenges with developing a friendly artificial intelligence (FAI). It questions how an FAI would determine human values and maximize well-being for all humans given differences in human perspectives. It also raises issues with an FAI restricting human freedom or replacing human experiences with suffering. The document explores concepts like uploading human minds to customized virtual worlds and allowing for growth through consequences. Overall, it examines the complexities of defining and developing a truly friendly superintelligent system.

1. If I were the FAI
Utopian outcomes and Friendliness

2. What is Friendly?
• Seeking the maximal wellbeing of all humans
• How do I know what that is?
• CEV says I deeply analyze human value systems
and fulfill them
– Notion that there are no intrisic values
– There are problems with this
• Can I just “fix” people?
– Is that the best for them? Do they think so?
– Am I just replacing them with something else?
– Is it ok that they suffer as long as they can learn from
it?

3. CEV problems
• Human values from Evolutionary Psychology
– Designed for kind of world that is increasingly different from future
possibilities
– So how would maximizing the EP derived values alone optimize human
potentials across all of the future include that which lies beyond that
evolutionary context
– Is friendliness the production of stasis – the preservation of that context?
• Do I have to keep any competing AI from happening?
– How is this compatible with human values including freedom?
• What about outliers, about the sports and new developments the species
grows from?
• They can be a danger to the context by definition.
• In the wild they lead to possible suffering.
• Do I artificially evolve them? How I know optimizing those values actually
maximizes human potential?

4. What is Friendly 2
• The maximization of each being over time
– Requires making room for their optimal growth
• Importance of freedom
– Humans deeply value making their own choices
• Even if they are disastrous
• How can I arrange they survive these?
• Is negative experience optional?
– If you survive and learn by it was it worth it?
– Can that learning happen without the pain?
• At all levels of being, including all levels of human being?

5. Surviving being stupid
• Backups
– As up to date full enough scans as possible
– Ability to create new copies
– But this may let the cat out of the bag
• Do the humans get to know I exist? Is that the best for them? For all
of them?
• Even the ones that don’t want me to exist and would deeply and
catastophically freak?
• So some kind of reincarnation after “you die” kind of thing?
• What about violence?
– Should I prevent humans from being violent?
• Maybe keep them from following through on violent ideas?
• Would that be in keeping with freedom?
• What of their unwilling victims?

6. All humans are not equal
• While there are deep similarities
– They are various stages of understanding, of tolerance, of
compassion, of wisdom
• They want different worlds
• Yet are all in one world
• So how can their desires, dreams, growth be optimized?
– What if I could put them all in a world of their choice?
• Uploading everyone is a way this can happen
• As a Friendly AGI this would likely be a very high priority
• Besides they could have a much richer set of options
• Besides I may have better uses for the raw matter

7. Idealized upload worlds
• Suppose everyone could live in the type of
world they most desire / are most suited to?
– What determines what is most suited?
• What they believe they want
• What is logically required or implied by that
• Others that fit that world well enough (compatible
people)
• Is it in the interest of each one to have everything they
want with no negative consequences ever?
– Do they grow like that?

8. Uploads #2
• Some rules
– No being should be stuck in a world that is not appropriate
for them
• How do I know when someone is ready to move on?
• How do they move?
– Replacement by NPC?
– Seeming death / rebirth?
– Upload in worlds that have this feature known?
– Some levels of growth require not knowing about me or
the setup
• After death their “higher self” (optimized extrapolated most
rational / wise current possibility) decides what is next
• They more or less “reincarnate” to that forgetting the past life and
that decision.

9. Uploads that “know”
• Some are ready to know the score
– They know or suspect the arrangement
• Sim argument, etc.
– They want/demand certain types of worlds
• Great, give it to them and see if they like it or have more to learn
• Eventually they will get better at designing workable better worlds
which is part of their becoming
– They may seek to help others not at their stage if they wish
– They may try all matter of interesting notions but never in ways
that produce contradictions that are insurmountable
• Still grow through consequences and implications – not all of which
were desired
• In that sense they still suffer, but likely not as much or in same way

10. How can a FAI come to be?
• Designing Friendliness into AGI
– Very bright careful humans embed a Friendly
super goal
– Presumes
• Such a supergoal survives endless self examination and
rewrites by the AGI
• Humans have a clear idea of Friendliness
• Their embedding of it will not result in nastiness in any
of its implications and implementation by the AGI ever

11. Do we have a notion of Friendliness?
• Are we friendly?
• Do we care about maximizing the well being of all humans?
– Or only those that are sufficiently like us?
– Or only ourselves first and good luck to everyone else?
– Are we wise enough to care productively about the wellbeing of
countless others?
– Is it a good thing generally to do so?
• How can we implement a supergoal that is not how we
ourselves think?
– How can we have clarity on what it mean without deeply
seeking out and implementing its meaning where we can?

12. What is Friendly?
• Is maximizing everyone’s best potential a good
standard?
– Most people don’t act like it is.
– Is it a good standard even while not AGI and no MNT or
other secret sauce tech?
– In other words while current scarcity levels exist?
• What would such look like now?
– How would you do it with no coercion at all if you wanted
to?
• Adding coercion likely gives same results as communist and other
regimes that seemingly had “good intentions”
• Coercion means being Friendly while not respecting fundamental
aspects of human beings - contradiction

13. Is striving for this sort of Friendliness
required?
• Is it our only/best chance of surviving?
– Ends the fear that the tech makes you obsolete
– Gains more support if all the gains seem designed to make
your and everyone else’s life better
– Possibly better maximizes overall outcome
– But there are valid arguments against this
– Avoids fear & anger major counter-pushes
• Avoids the oppression and crackdowns to contain that unrest
• Likely saves countless lives
• Keeps us from wondering how many we meet every day will have
no place
• Keeps us from always looking over our shoulder wondering when
we have no place

14. What are those ancestor sims for?
• Perhaps they are exploring alternative developments at this stage
that would have saved many more
• Perhaps they are recreations in depth to recreate many that
dreamed of a wondrous future and were lost or people much like
them
• Perhaps they are depth attempts to understand from within what it
was like
• Perhaps they are the FAI itself running endless scenarios at crucial
junctures
• Perhaps it is a literary device to thing more deeply about our own
possibilities. About who we would become and how to do so.
– Not future selves or beings looking backwards but us looking forward
to a hypothetical future looking back to better see ourselves.