Presentation of the research paper "Integration of Image Processing Techniques into the Unity Game Engine" at the 1st International Conference on Image Processing and Vision Engineering (IMPROVE 2021).

1. Integration of Image Processing Techniques into the Unity Game Engine
Leon Masopust1, Sebastian Pasewaldt2, Jürgen Döllner1 and Matthias Trapp1
1Hasso Plattner Institute, Faculty of Digital Engineering, University of Potsdam, Germany
2Digital Masterpieces GmbH, Potsdam, Germany

2. Results

4. Approach

5. Unity Engine UnityPackage Generic Unity Interface Custom Image Processor
OnRenderImage() GL.IssuePluginEvent() DoProcessing()
Unity Engine UnityPackage Generic Unity Interface Image Processor
OnRenderImage() GL.IssuePluginEvent() DoEffects()
Basic Integration Workflow

6. Unity Engine UnityPackage Generic Unity Interface Custom Image Processor
OnRenderImage() GL.IssuePluginEvent() DoProcessing()
Architecture Overview
Unity Engine
CustomPlugin
CustomUnityParser
+GetEffectString() : String
+GetPresetString() : String
+GetParameterString() : String
UnityProcessor
-instances : List<UnityPluginHandler>
+Init() : int
+ShutDown(id : int)
+PassRenderToTextureInfos(id: int, sourceTextureID : uint, targetTextureID : uint)
+RenderEventCallback(id : int)
+SetEffect(id : int, effect : String)
+SetPreset(id : int, preset : String)
+SetParameter(id : int, parameter : String, value : String)
+GetEffectString() : String
+GetPresetString() : String
+GetParameterString() : String
CustomTextureDisplayer
+SetSourceTextureID(sourceTextureID : uint)
+SetTargetTextureID(targetTextureID : uint)
+Blit()
CustomUnityRenderer
+UntiyRenderer()
+Render()
+SetTexture(source : uint)
+SetEffect(effect : String)
+SetPreset(preset : String)
+SetParameter(name : String, value : Any)
UnityPluginHandler
-effect : String
-preset : String
-parameters : Map<String, String>
+Init()
+ShutDown()
+PassRenderToTextureInfos(sourceTextureID : uint, targetTextureID : uint)
+RenderEventCallback()
+SetEffect(effect : String)
+SetPreset(preset : String)
+SetParameter(parameter : String, value : String)
+GetEffectString() : String
+GetPresetString() : String
+GetParameterString() : String
UnityPackage
GenericPostProcessor
+Init()
+ShutDown()
+OnRender()
CustomMaterial
-Effect : String
-Preset : String
+UpdateInformation()
CustomPostProcessor
-Effect : String
-Preset: String
-Parameters : List<Parameter>
+UpdateInformation()
CustomMaterialProcessor
-materials : List<CustomMaterial>
+Init()
+ShutDown()
+OnRender()
CustomProcessor
+Init() : int
+ShutDown(id : int)
+PassRenderToTextureInfos(id : int, sourceTextureID : uint, targetTextureID : uint)
+Render(id : int)
+SetEffect(id : int, effect : String)
+SetPreset(id : int, preset : String)
+SetParameter(id : int, parameter : String, value : String)
+GetEffects() : List<String>
+GetPresets(asset : String) : List<String>
+GetParameters(asset : String) : List<Parameter>
GenericMaterial
+GenericMaterial(texture: RenderTexture)
+~GenericMaterial()
+ApplyEffect()

7. Initialization Phase
alt
[FirstFrame]
Unity Engine UnityPackage : UnityPluginHandler : CustomUnityRenderer : CustomTextureDisplayer
1.1: Init
2.1.3: SetTargetTextureID
2.1.2: SetSourceTextureID
2.1.1.1: LoadEffect
2.1.1: SetEffect
2.1: GL.IssuePluginEvent
2: MonoBehavior.OnRenderImage
1.3: PassRenderToTextureInfos
1.2: SetEffect
1: MonoBehavior.Start

8. Rendering Phase
alt
[updateRequired]
Unity Scene UnityPackage : UnityPluginHandler : CustomUnityRenderer : CustomTextureDisplayer
1.5.2.2: ApplyEffect
1.5.2.1: SetOpenGLState
1.5.5: Blit
1.5.2: Render
1.5.1: SetTexture
1.4: PassRenderToTextureInfos
1.3: SetParameter
1.2: SetPreset
1.1: SetEffect
1.5.4: SetTargetTextureID
1.5.3: SetSourceTextureID
1.5: GL.IssuePluginEvent
1: MonoBehavior.OnRenderImage

9. Application:
Scene Post-Processing

11. Application:
Material Processing

13. Integration of Image Processing Techniques into the Unity Game Engine
Leon Masopust1, Sebastian Pasewaldt2, Jürgen Döllner1 and Matthias Trapp1
1Hasso Plattner Institute, Faculty of Digital Engineering, University of Potsdam, Germany
2Digital Masterpieces GmbH, Potsdam, Germany