NATS is used to build a scalable, location-independent, and resilient augmented reality platform. NATS topics and subjects are used to distribute messages about avatars, props, and world state changes across devices and locations. Messages are serialized using MessagePack for fast serialization and to define the protocol. NATS is integrated into Unity using a services class to connect and publish/subscribe to handle messages asynchronously.

1. NATS &
Augmented Reality
The Journey Creating
Worlds’ Augmented Reality Platform
ron@britvich.com github.com/Britvich @Britvich

2. Problem: Create Social AR Platform for
Handheld Devices and AR Glasses
Scalable Location
independent
Low latency
Authoritative
Throughput
Resilient
self-healing
Secure
Distributed Computing Problem

3. Handheld devices and AR glasses
Unity’s
ARFoundation
Apple’s
ARKit
Microsoft’s
HoloLens
Google’s
ARCore

4. Unity’s Networking?
Unity’s
UNet
Unity’s
NetCode
1-24 users
P2P
Deprecated
1-500 users
Not avail

5. gRPC Networking?
Not location independent
Not easy to auto-scale
Not resilient - self-healing

6. NATS Networking?
Location independent
Easy to auto-scale
Resilient - self-healing

7. AvatarAvatarAvatar
United States Europe
AvatarAvatarProp
AvatarAvatarClient
AvatarAvatarCitizen
AvatarAvatarNATS
AvatarAvatarClient
AvatarAvatarNATS
AvatarAvatarCitizen AvatarAvatarProp AvatarAvatarAvatar
NATS Topology

8. Citizen. Avatar. Prop.
[On]Immigrate
Login
Offer.World[.Cell]
Pose.World[.Cell]
Say.World[.Cell]
Select.World[.Cell]
[On]Add.World[.Cell]
Look.World[.Cell]
[On]Move.World[.Cell]
[On]Remove.World[.Cell]
NATS Subjects
Subject Example Description
World
GarageSale
RealEstate
Pothole
Garage sale items
Homes for sale
Database of potholes
Cell
33811N117919W
37819N122479W
48858N2294E
SE-corner of Disneyland’s hub
Middle of Golden Gate Bridge
Eiffel Tower

9. Subscribed Cells
33812N117920W 33812N117919W 33812N117918W
33811N117920W 33811N117919W 33811N117918W
33810N117920W 33810N117919W 33810N117918W

10. Citizen. Avatar. Prop.
Queue
groups
Immigrate
Login
Add.>
Look.>
Move.>
Remove.>
Subs OnImmigrate OnAdd.>
OnMove.>
OnRemove.>
Pubs
(NoEcho)
OnImmigrate OnAdd.World[.Cell]
OnMove.World[.Cell]
OnRemove.World[.Cell]
Service Messaging

11. Citizen. Avatar. Prop.
Subs Offer.World[.Cell]
Pose.World[.Cell]
Say.World[.Cell]
Select.World[.Cell]
OnAdd.World[.Cell]
OnMove.World[.Cell]
OnRemove.World[.Cell]
Request
Reply
Immigrate
Login
Add.World[.Cell]
Look.World[.Cell]
Move.World[.Cell]
Remove.World[.Cell]
Pubs
(NoEcho)
Offer.World[.Cell]
Pose.World[.Cell]
Say.World[.Cell]
Select.World[.Cell]
Client Messaging

12. Using NATS in Unity
public class Preload : MonoBehaviour {
void Awake() {
DontDestroyOnLoad(gameObject);
Screen.sleepTimeout = SleepTimeout.NeverSleep;
CompositeResolver.RegisterAndSetAsDefault(AvatarResolver.Instance, CitizenResolver.Instance,
PropResolver.Instance, UnityResolver.Instance, BuiltinResolver.Instance);
Server.services = new Services(URL, PORT);
Server.citizen = new CitizenService(Server.services);
SceneManager.LoadScene("Startup");
}
}
public class Services {
readonly IConnection con;
public Services(string url, int port) {
var opt = ConnectionFactory.GetDefaultOptions();
opt.Url = $"nats://{url}:{port}";
opt.NoEcho = true;
opt.MaxReconnect = Options.ReconnectForever;
opt.ReconnectBufferSize = Options.ReconnectBufferDisabled;
opt.ReconnectWait = 10000;
con = new ConnectionFactory().CreateConnection(opt);
}
}

13. MessagePack for C# Protocol (De)serialization

14. Defining the Protocol
[MessagePackObject]
public struct AddEvent {
[Key(0)] public ulong sessionId { get; set; }
[Key(1)] public ulong propId { get; set; }
[Key(2)] public Prop prop { get; set; }
}
[MessagePackObject]
public struct AddRequest {
[Key(0)] public ulong sessionId { get; set; }
[Key(1)] public Prop prop { get; set; }
}
[MessagePackObject]
public struct AddReply {
[Key(0)] public RC rc { get; set; }
[Key(1)] public ulong propId { get; set; }
}
[Union(0, typeof(PropSimple))]
[Union(1, typeof(PropLight))]
[MessagePackObject]
public abstract class Prop {
[Key(0)] public Vector3 position { get; set; }
[Key(1)] public Quaternion rotation { get; set; }
[Key(2)] public string name { get; set; }
}
[MessagePackObject]
public class PropSimple : Prop {
}
[MessagePackObject]
public class PropLight : Prop {
[Key(3)] public bool on { get; set; }
}

15. Async Messaging
public void Handle<Message>(string subject, Action<string,
Message> handler) {
con.SubscribeAsync(subject, (sender, args) => {
var message = MessagePackSerializer.Deserialize<Message>
(args.Message.Data);
handler(args.Message.Subject, message);
});
}
public void Handle<Request, Reply>(string subject, Func<string,
Request, Reply> handler) {
con.SubscribeAsync(subject, (sender, args) => {
var request = MessagePackSerializer.Deserialize<Request>
(args.Message.Data);
var reply = handler(args.Message.Subject, request);
var bytes = MessagePackSerializer.Serialize<Reply>(reply);
con.Publish(args.Message.Reply, bytes);
});
}
public void Publish<Message>(string subject, Message msg) {
var bytes = MessagePackSerializer.Serialize<Message>(msg);
con.Publish(subject, bytes);
}
public async Task<Reply> Request<Request, Reply>(string subject,
Request request) {
var bytes = MessagePackSerializer.Serialize<Request>(request);
var msg = await con.RequestAsync(subject, bytes);
return MessagePackSerializer.Deserialize<Reply>(msg.Data);
}