The world is rapidly moving towards containerizing everything, mostly by using linux cgroups and namespaces. But, Windows has kind of been a second-class citizen in this ecosystem. While great progress has been made in the Windows Container space, it has only recently become generally available. In 2016, when we started our migration to Nomad as our Microservice scheduler, Windows containers were in still their infancy, and the only Nomad driver able to run .Net was raw_exec. In short, running and scheduling .NET Framework microservices isn't easy. In this talk, you'll learn: - A little more about Windows Containers and their limitations. - How we at Jet constrain our Windows raw executables running on Nomad without Container support. - A little bit about the Windows 32 API and how you can hook into it in your Go programs without resorting to cgo.

1. Containment
without
Containers
Running Windows Microservices on Nomad
Justen Walker

2. Justen Walker
Software Engineer
Microservice Platform Team
Some stuff we work with
@

3. Overview
• Containment on Windows
• What is a Job Object
• Containing with Damon
• Go + Win32 API
• Lots of weird-looking Go
WARNING: There are QR Codes in the slides
Calibrate your sensors

4. Containment on Windows

5. What is a container
Source: https://msdn.microsoft.com/en-us/magazine/mt797649
bit.ly/2ymVaVL

6. Containment on Windows
• Windows Container Options
• “Process” Containers
(Windows Server Containers)
• Hyper-V Containers
Windows Server & Docker
The Internals Behind Bringing
Docker & Containers to Windows
- John Starks & Taylor Brown
@ DockerCon16
bit.ly/2QWMtIw

7. Windows Server Containers
• Highly Integrated System
• DLLS are the API, not syscalls
• Tightly coupled dependencies
• System Services + Hidden RPCs
• Process Containers…

8. WSC Portability
• Container Windows Build must match Host
• Larger images
• No FROM scratch
Windows Container Version
Compatibility
bit.ly/2pVTnlC

9. Windows Hyper-V Containers
• VM per container
• Optimized to run containers
• More Portable
• More Overhead
• Back to VMs
• More supporting processes
• Nested Virtualization
Hypervisor Running your Cloud VM

10. Docker for Windows
+ +
“Just Works”*

11. But, Wait!
Do I need…
- Overhead for isolation?
- VM inception?
- Painful Windows Upgrades?

12. Job Objects

13. Job Objects
Job Objects
“A job object allows groups of
processes to be managed as a unit …
enforcing limits such as working set
size and process priority or terminating
all processes associated with a job.”
bit.ly/2CmG6Kz

14. Job Objects
• Sort of like Linux cgroups
• Used by Windows Server Containers
• Do provide Resource Constraints
• Do Not provide Isolation

15. Using Job Objects
Win32 API (C, .NET, PowerShell?)
• CreateJobObjectW => hJob
• SetInformationJobObject(hJob, info) // Set Constraints
• CreateProcessW(CREATE_SUSPENDED) => hProc + hThread // Suspended
• AssignProcessToJobObject(hProc)
• ResumeThread(hThread)
Process constrained within Job!

17. Damon
damon is a supervisor program to constrain windows exe that are run with raw_exec on nomad.
• 100% Golang (no CGO)
• JobObject Resource Limits
• Restricted Tokens
• Prometheus Metrics
github.com/jet/damon
AS SEEN ON
GitHub

18. Damon + Nomad via raw_exec

19. Damon + Nomad + Prometheus

20. Go + Win32 API

21. Go + Win32 API
without CGO
• Creating a Job Object from Go
• Loading Windows DLL
• Discovering API Call Signature
• Creating C strings and Wide Strings
• Constructing a C-Compatible Go Structure
• Calling the Windows API procedures
Blog: Breaking all the rules:
Using Go to call Windows API
bit.ly/2AYJidz

22. Windows DLLs in Go
var kernel32DLL = syscall.NewLazyDLL("kernel32.dll")
var procCreateJobObjectA =
kernel32DLL.NewProc("CreateJobObjectA")
var procCreateJobObjectW =
kernel32DLL.NewProc("CreateJobObjectW")
Server Core Functions by DLL
bit.ly/2PbkGXx

23. API Call Signatures
#define LPCSTR const char*
typedef struct _SECURITY_ATTRIBUTES {
DWORD nLength;
LPVOID lpSecurityDescriptor;
BOOL bInheritHandle;
} SECURITY_ATTRIBUTES,
*PSECURITY_ATTRIBUTES, *LPSECURITY_ATTRIBUTES;
HANDLE CreateJobObjectA(
LPSECURITY_ATTRIBUTES lpJobAttributes,
LPCSTR lpName
);
CreateJobObjectA reference
bit.ly/2yoPPgD

24. Creating C strings in Go
func StringToCharPtr(str string) *uint8 {
chars := append([]byte(str), 0) // null terminated
return &chars[0]
}
import "unicode/utf16"
func StringToUTF16Ptr(str string) *uint16 {
wchars := utf16.Encode([]rune(str + "x00"))
return &wchars[0]
}

25. Creating a C Struct
// C
struct SECURITY_ATTRIBUTES {
DWORD nLength;
LPVOID lpSecurityDescriptor;
BOOL bInheritHandle;
};
// Go
type SecurityAttributes struct {
Length uint32
SecurityDescriptor uintptr
InheritHandle uint32
}
Windows Data Types Reference
bit.ly/2lz2lXb

26. Calling API from Go
func CreateJobObject(attr *syscall.SecurityAttributes, name string)
(syscall.Handle, error) {
ret, _, err := procCreateJobObjectA.Call(
uintptr(unsafe.Pointer(attr)),
uintptr(unsafe.Pointer(StringToCharPtr(name))),
)
if err != syscall.Errno(0) {
return 0, err
}
return syscall.Handle(ret), nil
}
godoc: unsafe.Pointer
bit.ly/2q3Hmup

27. Recap
• Docker + Nomad on Windows works*
• Job Objects: light weight constraints
• Use Damon to constrain exes on Nomad
• DIY! Go + Win32 API hacking

28. Reach me at
justenwalker
@justenwalker
Please check out github.com/jet/damon

30. Working with Raw Memory
var (buflen int;buffer []byte;err error;ret uintptr)
for {
ret, _, err = procSomeWindowsAPI.Call(
uintptr(unsafe.Pointer(&buffer[0])),
uintptr(unsafe.Pointer(&buflen)),
)
if err == syscall.ERROR_INSUFFICIENT_BUFFER {
buffer = make([]byte, buflen)
continue
}
break
}
// check err and ret before this
result := (*WinAPIGoStruct)(unsafe.Pointer(&buffer[0]))

31. Working with Raw Memory

32. Working with ANY_SIZE arrays
type GroupResult struct {
Count uint32
Groups [1]Group
}
gr := (*GroupResult)unsafe.Pointer(&buffer[0])
groups := (*[1 << 30]Group)
(unsafe.Pointer(&gr.Groups))
[0:gr.Count:gr.Count]
offset Length = Capacity

33. Working with ANY_SIZE arrays

34. Thank You Again!
Reach me at
justenwalker
@justenwalker
Please check out github.com/jet/damon