About Neural Radiance Field & Related publications.

1. Neural Radiance Fields
Neural Fields in Computer Vision

2. Field
스칼라장 : 온도, 압력, 오디오 등 벡터장 : 유체속도장, 자기장 등 텐서장 : 영상 등

3. Neural Field

4. 3D Synthesize : COLMAP - Structure-from-Motion and Multi-View Stereo
https://colmap.github.io/

5. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
(ECCV 2020)
https://neuralfields.cs.brown.edu/cvpr22
https://keras.io/examples/vision/nerf/

6. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
(ECCV 2020)
https://github.com/bmild/nerf
https://github.com/NakuraMino/Minimal-NeRF
1. MLP 기반의 NeRF
* input의 Spherical Vector(2D)를 Cartesian(3D)
으로 변경
* input의 viewing direction : sigma, pi
(Cartesian)가 3*2*L_embed로 확장 ( positional
encoding )
* output은 RGBa
(a는 volume density)
=> !!중요한점, 1 view는 1 pixel에 대응
8 Depth FC (with 256 dim - Relu)

7. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
(ECCV 2020)
https://github.com/bmild/nerf/blob/20a91e764a28816ee2234fcadb73bd59a613a44c/load_blender.py
MLP 기반의 NeRF (Dataset from Blender)

8. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
(ECCV 2020)
https://github.com/bmild/nerf/blob/20a91e764a28816ee2234fcadb73bd59a613a44c/load_blender.py
https://docs.blender.org/api/blender_python_api_2_71_release/
https://github.com/bmild/nerf/issues/78
https://www.3dgep.com/understanding-the-view-matrix/#:~:text=The%20world%20transformation%20matrix%20is,%2Dspace%20to%20view%2Dspace.
MLP 기반의 NeRF (Dataset from Blender)
camera_angle_x : Camera lens horizontal field of view <- 카메라의 Focal 값
(모두 동일값)
rotation : radians (degree) <- 카메라 제자리 회전 값… (모두 동일값)
transform_matrix (Worldspace transformation matrix, matrix_world (blender))
<- Camera 월 좌표계 및 상대 Rotation 정보

9. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
(ECCV 2020)
https://www.scratchapixel.com/lessons/mathematics-physics-for-computer-graphics/lookat-function
MLP 기반의 NeRF (Dataset from Blender)
transform_matrix (Look-At 행렬) - Camera 2 World
focal = .5 * W / np.tan(.5 * camera_angle_x)

10. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
(ECCV 2020)
https://www.scratchapixel.com/lessons/mathematics-physics-for-computer-graphics/lookat-function
MLP 기반의 NeRF (Dataset from Blender) -
rays_o는 rays_original (position) : 각 HxWx3 pixel의 original ray point를 의미 == 모든 점이
camera x,y,z로 똑같음 (각 픽셀마다의 camera position)
rays_d는 direction: dirs는 HxW내에서의 상대 좌표. c2w를 곱해줌으로서 camera viewing
direction(+depth)을 계산 (각 픽셀마다의 camera view direction)

11. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
(ECCV 2020)
http://sgvr.kaist.ac.kr/~sungeui/GCG_S22/Student_Presentations/CS580_PaperPresentation_dgkim.pdf

12. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
(ECCV 2020)
https://www.scratchapixel.com/lessons/mathematics-physics-for-computer-graphics/lookat-function
MLP 기반의 NeRF (Dataset from Blender) -
z_vals <- Random Accumulate Point 지정
pts <- z_vals위치의 point들을 생성
pts input에 대한 network output 값의 합산이 rgb가
될 수 있도록 함
(weights 값 합산은 near 1)
=> raw[:3]는 rgb_map과 mse, raw[4]는 직접 mse가
없고 raw[:3]와 곱해져 간접 업데이트
(본 코드에는 raw[4] - a는 업데이트 없음 fix z_vals)

13. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
(ECCV 2020)
https://m.blog.naver.com/jetarrow82/221252045857
1. MLP 기반의 NeRF - Viewing Direction의 중요성
실제 물체는 non-Lambertian effects 특성을 가짐
Lambertian Surface는 ViewPoint에 상관없이
똑같은 빛의 양을 반사함

14. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
(ECCV 2020)
https://www.matthewtancik.com/nerf
1. MLP 기반의 NeRF - Positional Encoding

15. NeRF Result
https://phog.github.io/snerg/viewer/index.html?dir=https://storage.googleapis.com/snerg/750/lego
<= 같이 보자!

16. NeRF: Synthesizer로서의 가치
Small Size Model Rendering 도메인이 한정되어 있지 않음

17. Texture
https://learnopengl.com/Advanced-OpenGL/Cubemaps
https://lifeisforu.tistory.com/375
CubeMap - 보통 ray
영향을 크게 받지 않는
원경에 사용
Spherical Harmonics( SH, 구면
조화 ) - irradiance map(난반사
정도)을 저장하는 효율적 방식

18. Plenoxels : ReRF without Neural Networks
NeRF key point is differentiable Volume Renderer
MLP Volume Rendering을 Voxel Grid와 Interpolation의 조합으로 치환

19. Beyond Synthesize
- iNeRF: Inverting Neural Radiance Fields for Pose Estimation (2020)
Neural Fields in Visual Computing and Beyond (2022 EUROGRPAHICS)
https://yenchenlin.me/inerf/
Trained Nerf를 통해
Generation하고 Pose Estimation을
Iterative하게 Gradient Based
Optimization을 하여 Estimated
Pose를 취득

20. Beyond Synthesize
- iNeRF: Inverting Neural Radiance Fields for Pose Estimation (2020)
Neural Fields in Visual Computing and Beyond (2022 EUROGRPAHICS)
https://yenchenlin.me/inerf/

21. Beyond Synthesize
- Learning Dense Object Descriptors from Neural Radiance Fields (2022 ICRA)
https://yenchenlin.me/nerf-supervision/
Keypoint detection과 유사.
Keypoint는 sparse하게 pose
정보를 추출한다면, Dense
Descriptor는 Dense하게 Pose
정보를 추출

22. Beyond Synthesize
- Learning Dense Object Descriptors from Neural Radiance Fields (2022 ICRA)
https://yenchenlin.me/nerf-supervision/

23. Beyond Synthesize
- NERF-SOS: ANY-VIEW SELF-SUPERVISED OBJECT SEGMENTATION ON COMPLEX SCENES (2022 Pre)
- 셀링포인트는 다양한
각도에서 영상을 촬영하면
NeRF 샘플에서
Unsupervised로
Segmentation을 만들어줌
=> 한계점
- K-Means/Pre-trained
Network에 의존적 (BG가
명확하게 분리되고, Object의
Semantic 정보가 명확해야 할
것으로 보임)

24. NeRFs…
https://kakaobrain.github.io/NeRF-Factory/ <= Nerf 성능 비교
Mega-NeRF: Scalable Construction of Large-Scale NeRFs for Virtual Fly-Throughs <= 넓은 공간 모델링
Instant Neural Graphics Primitive
<= NeRF, SDF(signed distance functions), Gigapixel image등 (Computer graphics primitives)을 처리할 수
있는 네트워크
https://nvlabs.github.io/instant-ngp/assets/mueller2022instant.mp4
Dex-NeRF : Using a Neural Radiance Field to Grasp Transparent Objects
- Transparent, Light-sensitive한 물체를 잘 다룰 수 있지 않을까?
https://sites.google.com/view/dex-nerf
vision-only robot navigation in a neural radiance world - Vision Based Planning
https://mikh3x4.github.io/nerf-navigation/
Relighting the Real World With Neural Rendering -
https://www.unite.ai/adobe-neural-rendering-relighting-2021/
NeRF-Studio
https://docs.nerf.studio/en/latest/index.html