How do camera images become a space you can explore?
- InputCapture the measured world
72 RGB images + K + 4×4 poses + depth (m)CARLA supplies known camera geometry. Reserve every eighth view.
- ProcessOptimize a radiance field
63 training images → 345,318 GaussiansfVDB adjusts position, scale, opacity and color. Held-out depth is excluded from initialization.
- OutputRender another viewpoint
full PLY → MP4 / 345k-point SPZ → SparkThe browser view loads on demand. No physics or collision mesh is included.
The interactive scene below is a real fVDB Gaussian Splatting reconstruction of a CARLA fountain plaza. It is not a panorama on a sphere or a video playing behind a camera. The renderer projects 3D Gaussians from your current viewpoint. Use the 3D button to orbit, pan and move closer; use the video if your browser does not support the viewer.
Images, calibration and depth have different roles
We captured 72 RGB images at 640 × 360, along with camera intrinsics K, world-to-camera matrices, and simulated depth in meters. The camera's horizontal FOV is 80°, giving fx = fy ≈ 381.36 px. A pixel ray is derived from (u-cx)/fx, (v-cy)/fy, and its depth. Transforming that point by the camera-to-world matrix places it in the shared 3D scene. This example uses known simulator cameras, so it does not need COLMAP to estimate poses.
{
"image_path": "view_000.png",
"width": 640,
"height": 360,
"camera_intrinsics": [[381.361, 0, 320], [0, 381.361, 180], [0, 0, 1]],
"world_to_camera": "4 x 4 matrix; see evidence/cameras.json"
}
The matrix values above are rounded for reading. The saved cameras contain full values. CARLA uses a left-handed world; our exporter reflects world Y and converts camera axes to OpenCV's right/down/forward convention. A wrong sign here would mirror the reconstructed scene.
For a concrete point, pixel (320,180) in view_000.png has depth 21.237 m. Its camera-space point is approximately [0,0,21.237] m; applying the saved camera pose gives world position [0.958,0,2.131] m. The actual sample includes full precision, and check.py recomputes the transformation.
What is a Gaussian?
Each Gaussian stores a center [x,y,z], three scales, a rotation quaternion, an opacity and color coefficients. It is a soft ellipsoid in space. fVDB projects and blends these ellipsoids, compares the result with a training image, and adjusts them to reduce rendering error. It also splits or removes Gaussians during refinement. The final model has 345,318 Gaussians. fVDB Reality Capture
The 6.15 MB browser SPZ keeps all 345,318 Gaussians with quantized parameters and degree-1 color coefficients. The full recording uses the original degree-3 spherical-harmonic model. Appearance can therefore differ slightly between the interactive preview and the video. A radiance field describes appearance; collision surfaces, mass and a robot's control loop require separate data.
How is this different from a generated world?
This experiment reconstructs a supplied scene. It does not invent unseen streets from a text prompt. World Labs Marble and NVIDIA Lyra address generative 3D worlds; Infinigen constructs geometry from procedural rules. The 3D worlds comparison separates those outputs and their execution requirements.
Two additional clips below use Apple SHARP on a single CARLA camera image. We supply the known 90° intrinsics instead of relying on missing EXIF. SHARP predicts 1,179,648 Gaussians per image; fVDB then renders a small nearby camera movement. It does not reconstruct the back of every object or produce a complete 360° world. Its weights are for noncommercial research, unlike its separately licensed code.
First time? Get the lab files
Open a terminal on the Isaac Sim machine for GPU labs, or on your own computer for CPU-only steps. Press Enter after each line. You can copy a multi-line command block as a whole.
cd "$HOME"
git clone https://github.com/james-joobs/awesome-nvidia-physical-ai.git
cd awesome-nvidia-physical-ai
pwd
This example clones into your home folder. If you already cloned the repo, skip git clone and enter your project folder. In each new terminal, check that pwd ends in awesome-nvidia-physical-ai so commands can find the labs/ files.
Capture and reconstruct a new scene
1. Capture known camera views
Follow CARLA setup to start the pinned 0.9.16 server in terminal A. In the CARLA Python environment, from the repository in terminal B:
python labs/spatial-reconstruction/capture.py --output runs/plaza-input
The command moves RGB and depth sensors around a static Town03 plaza. Wait for Saved view 72/72. images/ stores color PNGs, depth/ stores meter-valued NumPy arrays, and cameras.json records the exact camera poses. prepare_seed.py builds the initial point cloud without using held-out images or their depth. Index 0,8,16,… is reserved for evaluation.
Stop your CARLA container before starting the next GPU job. The Python environments are separate: a CARLA wheel is not an fVDB installation.
2. Optimize the 3D Gaussians
Install fVDB and fVDB Reality Capture using their official installation guide, matching PyTorch/CUDA to your GPU. Activate that environment and install plyfile, Pillow and SciPy if missing. Our verified run used fVDB 0.6.0 and PyTorch 2.13.0+cu130 on an RTX 5090; see the runtime in recorded.json. Run:
python labs/spatial-reconstruction/reconstruct.py \
--input runs/plaza-input --output runs/plaza-3dgs --ffmpeg ffmpeg
The script uses 63 RGB training views for 40 epochs. It leaves camera poses fixed, exports the full PLY and the lighter .splat, renders a new camera path, and writes evaluation metrics. In the recorded environment, cuDNN library versions conflicted, so this script disables cuDNN and uses native PyTorch CUDA convolutions; fVDB training and rendering still run on the GPU. That workaround is explicit in the source and record.
For the web preview, keep every Gaussian and compress the full PLY with a pinned PlayCanvas CLI:
python3 labs/spatial-reconstruction/compress_web.py runs/plaza-3dgs/town03.ply runs/plaza-3dgs/town03.spz
3. Optional single-image comparison
Install SHARP and its dependencies in an environment where fVDB also imports. Review the model's research-only license before downloading the official checkpoint. Put the ego JPEGs from the CARLA exercise in runs/sharp-input/, then run:
python labs/spatial-reconstruction/reconstruct_single.py \
--input runs/sharp-input --output runs/sharp-output \
--checkpoint /path/to/sharp_2572gikvuh.pt --ffmpeg ffmpeg
This helper assumes those inputs are our 90° CARLA cameras. Change the focal length for other images. It uses a ±0.35 m lateral camera movement; large translations would reveal unsupported regions.
Inspect the result without a GPU

python3 labs/spatial-reconstruction/check.py
The bundled checker verifies the split, camera rotations, finite model parameters, preview size and recorded video hashes. The recorded run has 9 held-out RGB views with a mean 23.434 dB PSNR. The photometric fit excludes those RGBs, and the seed excludes their depth and colors. This is a small synthetic scene measurement, not a cross-dataset benchmark. Water reflections, distant backgrounds, holes and views outside the captured region remain imperfect.
The repository includes camera metadata, code, metrics, compressed recordings and the browser preview. Media provenance and license scope · Generate an indoor scene · fVDB theory.
Explore the reconstructed scene
Explore a CARLA plaza reconstructed with fVDB. The model and viewer load only when requested. This radiance field has no physics or collision mesh.

Drag to orbit, scroll or pinch to zoom, right-drag to pan. Focus the scene and use W/A/S/D to move. Gaps and blurred water are reconstruction artifacts.
The SPZ keeps all 345,318 Gaussians with degree-1 color. The recording and PSNR use the full degree-3 model. Spark / World Labs (MIT) · CARLA assets / CC-BY
Example run and recording
City from one image
SHARP predicts 3D Gaussians from our CARLA image. fVDB renders nearby views. Research-only model weights; no complete 360° scene.
Suburb from one image
SHARP predicts 3D Gaussians from our CARLA image. fVDB renders nearby views. Research-only model weights; no complete 360° scene.
Source code
capture.py
View files on GitHub"""Capture a static CARLA neighborhood with known cameras for fVDB reconstruction.
This is a measured RGB-D dataset, not a generated world or an SfM estimate.
"""
import argparse
import json
import math
import queue
from pathlib import Path
import carla
import numpy as np
from PIL import Image
from prepare_seed import prepare
def read_frame(q, frame):
while True:
image = q.get(timeout=30)
if image.frame == frame:
return np.frombuffer(image.raw_data, np.uint8).reshape(image.height, image.width, 4).copy()
if image.frame > frame:
raise RuntimeError("Camera timestamps do not match")
def main():
parser = argparse.ArgumentParser()
parser.add_argument("--output", required=True, type=Path)
parser.add_argument("--port", type=int, default=24000)
parser.add_argument("--views", type=int, default=72)
args = parser.parse_args()
output = args.output
(output / "images").mkdir(parents=True, exist_ok=True)
(output / "depth").mkdir(exist_ok=True)
client = carla.Client("127.0.0.1", args.port)
client.set_timeout(90)
world = client.load_world("Town03")
previous = world.get_settings()
settings = world.get_settings()
settings.synchronous_mode = True
settings.fixed_delta_seconds = .05
world.apply_settings(settings)
world.set_weather(carla.WeatherParameters.ClearNoon)
width, height, fov = 640, 360, 80
focal = width / (2*math.tan(math.radians(fov/2)))
intrinsics = np.array([[focal,0,width/2],[0,focal,height/2],[0,0,1]])
sensors = []
cameras = []
# Reflect the world Y axis to make a right-handed reconstruction world.
world_basis = np.diag([1.,-1.,1.,1.])
camera_basis = np.array([[0,0,1,0],[1,0,0,0],[0,-1,0,0],[0,0,0,1.]])
try:
for kind in ("rgb", "depth"):
bp = world.get_blueprint_library().find(f"sensor.camera.{kind}")
for key,value in {"image_size_x":width,"image_size_y":height,"fov":fov,"sensor_tick":0}.items():
bp.set_attribute(key,str(value))
if kind == "rgb":
bp.set_attribute("motion_blur_intensity","0")
sensor = world.spawn_actor(bp, carla.Transform())
q=queue.Queue()
sensor.listen(q.put)
sensors.append((sensor,q))
for i in range(args.views):
# Two interleaved rings; every eighth pose is held out during training.
angle=2*math.pi*i/args.views
radius=22+2*math.sin(3*angle)
x,y=radius*math.cos(angle),radius*math.sin(angle)
z=5+2*math.sin(2*angle)
yaw=math.degrees(math.atan2(-y,-x))
pitch=math.degrees(math.atan2(2-z, radius))
transform=carla.Transform(carla.Location(x=x,y=y,z=z),carla.Rotation(pitch=pitch,yaw=yaw))
for sensor,_ in sensors:
sensor.set_transform(transform)
for _ in range(6):
frame=world.tick()
data=[read_frame(q,frame) for _,q in sensors]
rgb=data[0][:,:,:3][:,:,::-1].copy()
raw=data[1].astype(np.float64)
depth=(raw[:,:,2]+raw[:,:,1]*256+raw[:,:,0]*65536)/16777215*1000
name=f"view_{i:03}.png"
Image.fromarray(rgb).save(output/"images"/name)
np.save(output/"depth"/f"view_{i:03}.npy",depth.astype(np.float32))
c2w=world_basis@np.array(transform.get_matrix())@camera_basis
assert np.isclose(np.linalg.det(c2w[:3,:3]),1,atol=1e-5)
cameras.append({"camera_name":name,"image_path":name,"width":width,"height":height,
"camera_intrinsics":intrinsics.tolist(),"world_to_camera":np.linalg.inv(c2w).tolist(),
"sensor_frame":frame})
print(f"Saved view {i+1}/{args.views}",flush=True)
(output/"cameras.json").write_text(json.dumps(cameras,indent=2)+"\n")
seed_points=prepare(output)
report={"backend":"CARLA 0.9.16 / Unreal Engine 4.26","map":"Town03","weather":"ClearNoon",
"views":len(cameras),"resolution":[width,height],"fov_degrees":fov,
"seed_points":seed_points,"seed_views":[i for i in range(args.views) if i%3==0 and i%8!=0],"held_out_modulo":8,
"depth":"CARLA RGB24 depth decoded in meters",
"camera_convention":"OpenCV x-right y-down z-forward; right-handed world: CARLA Y reflected",
"source_license":"CARLA-specific assets: CC-BY; see CARLA 0.9.16 README licenses"}
(output/"capture.json").write_text(json.dumps(report,indent=2)+"\n")
finally:
for sensor,_ in sensors:
sensor.stop()
sensor.destroy()
world.apply_settings(previous)
if __name__=="__main__":
main()
prepare_seed.py
View files on GitHub"""Seed 3DGS only from training-view depth and colors; reserve indices 0,8,16,..."""
import argparse
import json
from pathlib import Path
import numpy as np
from PIL import Image
def prepare(folder):
cameras=json.loads((folder/"cameras.json").read_text())
clouds,colors=[],[]
for i,camera in enumerate(cameras):
if i%3 or i%8==0:
continue
rgb=np.array(Image.open(folder/"images"/camera["image_path"]).convert("RGB"))
depth=np.load(folder/"depth"/f"view_{i:03}.npy")
h,w=depth.shape
K=np.array(camera["camera_intrinsics"])
c2w=np.linalg.inv(camera["world_to_camera"])
v,u=np.mgrid[0:h:6,0:w:6]
d=depth[v,u]
good=(d>.5)&(d<80)
cv=np.stack([(u-K[0,2])/K[0,0]*d,(v-K[1,2])/K[1,1]*d,d],-1)[good]
clouds.append(cv@c2w[:3,:3].T+c2w[:3,3])
colors.append(rgb[v,u][good])
xyz,rgb=np.concatenate(clouds),np.concatenate(colors)
_,indices=np.unique(np.floor(xyz/.12).astype(int),axis=0,return_index=True)
xyz,rgb=xyz[indices],rgb[indices]
with (folder/"pointcloud.ply").open("w") as f:
f.write("ply\nformat ascii 1.0\nelement vertex %d\n"%len(xyz))
f.write("property float x\nproperty float y\nproperty float z\nproperty uchar red\nproperty uchar green\nproperty uchar blue\nend_header\n")
for p,c in zip(xyz,rgb):
f.write("%.5f %.5f %.5f %d %d %d\n"%(*p,*c))
return len(xyz)
if __name__=="__main__":
parser=argparse.ArgumentParser()
parser.add_argument("folder",type=Path)
args=parser.parse_args()
count=prepare(args.folder)
report=json.loads((args.folder/"capture.json").read_text())
report.update(seed_points=count,seed_views=[i for i in range(len(json.loads((args.folder/"cameras.json").read_text()))) if i%3==0 and i%8!=0],held_out_modulo=8)
(args.folder/"capture.json").write_text(json.dumps(report,indent=2)+"\n")
print(f"Seeded {count} points; held-out RGB and depth excluded.")
reconstruct.py
View files on GitHub"""Train an fVDB 3DGS scene from known CARLA cameras; hold out every eighth image."""
import argparse
import hashlib
import json
import math
import subprocess
import time
from pathlib import Path
import fvdb
import numpy as np
import torch
from PIL import Image
from plyfile import PlyData
from scipy.spatial.transform import Rotation, Slerp
from fvdb_reality_capture.radiance_fields import GaussianSplatReconstruction, GaussianSplatReconstructionConfig, GaussianSplatOptimizerConfig
from fvdb_reality_capture.sfm_scene import SfmScene
from fvdb_reality_capture.sfm_scene.sfm_scene import SpatialScaleMode
def export_browser_splat(source, target, maximum=200_000):
"""32-byte .splat, DC color only, deterministic opacity-weighted subset.
Browser preview is deliberately smaller than the original SH-degree-3 model.
"""
vertices = PlyData.read(source)["vertex"].data
opacity = 1/(1+np.exp(-np.clip(vertices["opacity"],-40,40)))
valid = np.flatnonzero(opacity>.04)
if len(valid)>maximum:
rng=np.random.default_rng(42)
weights=opacity[valid].astype(np.float64)
valid=np.sort(rng.choice(valid,maximum,replace=False,p=weights/weights.sum()))
v=vertices[valid]
packed=np.empty(len(v),dtype=[("position","<f4",3),("scale","<f4",3),("rgba","u1",4),("rotation","u1",4)])
packed["position"]=np.stack([v[k] for k in ("x","y","z")],-1)
packed["scale"]=np.exp(np.stack([v[f"scale_{i}"] for i in range(3)],-1))
color=.5+.28209479177387814*np.stack([v[f"f_dc_{i}"] for i in range(3)],-1)
packed["rgba"][:,:3]=np.clip(color*255,0,255).astype(np.uint8)
packed["rgba"][:,3]=np.clip(opacity[valid]*255,0,255).astype(np.uint8)
quat=np.stack([v[f"rot_{i}"] for i in range(4)],-1)
quat/=np.linalg.norm(quat,axis=1,keepdims=True).clip(1e-8)
packed["rotation"]=np.clip(quat*128+128,0,255).astype(np.uint8)
packed.tofile(target)
return {"gaussians":len(v),"bytes":target.stat().st_size,"sha256":hashlib.sha256(target.read_bytes()).hexdigest(),
"format":"32-byte .splat; DC color; opacity > .04; deterministic weighted sampling if needed"}
def main():
parser=argparse.ArgumentParser()
parser.add_argument("--input",type=Path,required=True)
parser.add_argument("--output",type=Path,required=True)
parser.add_argument("--epochs",type=int,default=40)
parser.add_argument("--ffmpeg",default="ffmpeg")
args=parser.parse_args()
args.output.mkdir(parents=True,exist_ok=True)
torch.backends.cudnn.enabled=False # GPU workstation cuDNN library mismatch; native CUDA convolutions still work.
scene=SfmScene.from_simple_directory(args.input)
config=GaussianSplatReconstructionConfig(max_epochs=args.epochs,optimize_camera_poses=False,
refine_stop_epoch=args.epochs,near_plane=.05,far_plane=200)
optimizer=GaussianSplatOptimizerConfig(spatial_scale_mode=SpatialScaleMode.MAX_CAMERA_TO_CENTROID,max_gaussians=750_000)
runner=GaussianSplatReconstruction.from_sfm_scene(scene,config=config,optimizer_config=optimizer,use_every_n_as_val=8)
start=time.monotonic()
runner.optimize()
seconds=time.monotonic()-start
model=runner.model
model_path=args.output/"town03.ply"
runner.save_ply(model_path)
browser=export_browser_splat(model_path,args.output/"town03.splat")
width,height=640,360
K=np.array(scene.images[0].camera_metadata.projection_matrix,dtype=np.float32)
def render(c2w):
with torch.inference_mode():
rgb,alpha=model.render_images(torch.tensor(np.linalg.inv(c2w)[None],dtype=torch.float32,device="cuda"),
torch.tensor(K[None],dtype=torch.float32,device="cuda"),width,height,.05,200)
return rgb[0,:,:,:3].clamp(0,1).cpu().numpy(),float(alpha.mean())
metrics=[]
for i in range(0,len(scene.images),8):
pred,alpha=render(scene.camera_to_world_matrices[i])
ref=np.array(Image.open(scene.images[i].image_path).convert("RGB"))/255
mse=np.mean((pred-ref)**2)
metrics.append({"view":i,"psnr_db":round(-10*math.log10(max(float(mse),1e-12)),3),"mean_alpha":round(alpha,4)})
if i==0:
Image.fromarray((pred*255).astype(np.uint8)).save(args.output/"held-out-render.png")
Image.fromarray((ref*255).astype(np.uint8)).save(args.output/"held-out-reference.png")
poses=scene.camera_to_world_matrices
keys=np.arange(len(poses)+1,dtype=float)
closed=np.concatenate([poses,poses[:1]],axis=0)
rotations=Slerp(keys,Rotation.from_matrix(closed[:,:3,:3]))
fps,frames=24,384
movie=args.output/"flythrough.mp4"
encoder=subprocess.Popen([args.ffmpeg,"-hide_banner","-loglevel","error","-y","-f","rawvideo","-pix_fmt","rgb24",
"-s",f"{width}x{height}","-r",str(fps),"-i","pipe:0","-an","-c:v","libx264","-crf","22",
"-pix_fmt","yuv420p","-movflags","+faststart",str(movie)],stdin=subprocess.PIPE)
alphas=[]
for t in np.linspace(0,len(poses),frames,endpoint=False):
pose=np.eye(4)
pose[:3,:3]=rotations(float(t)).as_matrix()
pose[:3,3]=[np.interp(t,keys,closed[:,axis,3]) for axis in range(3)]
rgb,alpha=render(pose)
alphas.append(alpha)
encoder.stdin.write((rgb*255).astype(np.uint8).tobytes())
encoder.stdin.close()
assert encoder.wait()==0
report={"mode":"multi-view reconstruction, not generative world-model inference",
"input":json.loads((args.input/"capture.json").read_text()),"input_views":len(poses),
"training_views":len(poses)-len(metrics),"held_out_views":len(metrics),"epochs":args.epochs,
"optimization_seconds":round(seconds,3),"gaussians":int(model.num_gaussians),
"held_out_psnr_db":round(float(np.mean([m["psnr_db"] for m in metrics])),3),"per_view":metrics,
"runtime":{"fvdb":fvdb.__version__,"torch":torch.__version__,"gpu":torch.cuda.get_device_name(0),"cudnn_enabled":False},
"browser":browser,"render":{"frames":frames,"fps":fps,"resolution":[width,height],
"median_alpha":float(np.median(alphas)),"sha256":hashlib.sha256(movie.read_bytes()).hexdigest()},
"limitations":["Static radiance field; no collision mesh or dynamic traffic", "CARLA water and distant sky are difficult to reconstruct", "Intermediate .splat uses sampled DC-only data; use compress_web.py on the full PLY for the published SPZ"]}
(args.output/"result.json").write_text(json.dumps(report,indent=2)+"\n")
print(json.dumps(report,indent=2),flush=True)
if __name__=="__main__":
main()
reconstruct_single.py
View files on GitHub"""Run SHARP on our CARLA inputs, using the known 90-degree camera intrinsics.
SHARP weights are research-only. This predicts nearby views, not a complete 360 world.
"""
import argparse
import hashlib
import json
import math
import subprocess
import time
from pathlib import Path
import numpy as np
import torch
from PIL import Image
from fvdb_reality_capture import GaussianSplat3d
from sharp.cli.predict import predict_image
from sharp.models import PredictorParams, create_predictor
from sharp.utils.gaussians import save_ply
def main():
parser=argparse.ArgumentParser()
parser.add_argument("--input",type=Path,required=True)
parser.add_argument("--output",type=Path,required=True)
parser.add_argument("--checkpoint",type=Path,required=True)
parser.add_argument("--ffmpeg",default="ffmpeg")
args=parser.parse_args()
args.output.mkdir(parents=True,exist_ok=True)
torch.backends.cudnn.enabled=False
predictor=create_predictor(PredictorParams())
predictor.load_state_dict(torch.load(args.checkpoint,weights_only=True))
predictor.eval().cuda()
reports=[]
for image_path in sorted(args.input.glob("*.jpg")):
image=np.array(Image.open(image_path).convert("RGB"))
height,width=image.shape[:2]
f_px=width/2 # Our CARLA camera has a known horizontal FOV of 90 degrees.
torch.cuda.synchronize()
start=time.monotonic()
with torch.inference_mode():
gaussians=predict_image(predictor,image,f_px,torch.device("cuda"))
torch.cuda.synchronize()
seconds=time.monotonic()-start
ply=args.output/f"{image_path.stem}.ply"
save_ply(gaussians,f_px,(height,width),ply)
del gaussians
splats,_=GaussianSplat3d.from_ply(str(ply))
w,h,fps,frames=640,360,24,192
K=np.array([[f_px*w/width,0,(w-1)/2],[0,f_px*h/height,(h-1)/2],[0,0,1]])
movie=args.output/f"{image_path.stem}.mp4"
encoder=subprocess.Popen([args.ffmpeg,"-hide_banner","-loglevel","error","-y","-f","rawvideo","-pix_fmt","rgb24",
"-s",f"{w}x{h}","-r",str(fps),"-i","pipe:0","-an","-c:v","libx264","-crf","22",
"-pix_fmt","yuv420p","-movflags","+faststart",str(movie)],stdin=subprocess.PIPE)
for i in range(frames):
phase=i/(frames-1)*2*math.pi
eye=np.array([.35*math.sin(phase),0,.1*(1-math.cos(phase))])
forward=np.array([0,0,12])-eye
forward/=np.linalg.norm(forward)
right=np.cross([0,1,0],forward);right/=np.linalg.norm(right)
pose=np.eye(4);pose[:3,:3]=np.stack([right,np.cross(forward,right),forward],axis=1);pose[:3,3]=eye
with torch.inference_mode():
rgb,_=splats.render_images(torch.tensor(np.linalg.inv(pose)[None],dtype=torch.float32,device="cuda"),
torch.tensor(K[None],dtype=torch.float32,device="cuda"),w,h,.01,1000)
rgb=rgb[0,:,:,:3].clamp(0,1).cpu().numpy()
encoder.stdin.write((rgb*255).astype(np.uint8).tobytes())
if i==0:
Image.fromarray((rgb*255).astype(np.uint8)).save(args.output/f"{image_path.stem}.jpg",quality=90)
encoder.stdin.close()
assert encoder.wait()==0
reports.append({"input":image_path.name,"input_sha256":hashlib.sha256(image_path.read_bytes()).hexdigest(),
"input_resolution":[width,height],"focal_px":f_px,"inference_seconds":round(seconds,3),
"gaussians":int(splats.num_gaussians),"video":movie.name,"video_sha256":hashlib.sha256(movie.read_bytes()).hexdigest(),
"frames":frames,"fps":fps,"camera_motion_m":{"x_amplitude":.35,"z_range":[0,.2]}})
del splats
(args.output/"result.json").write_text(json.dumps({"tool":"Apple SHARP","mode":"single-view reconstruction; nearby novel views",
"renderer":"fVDB","weights_license":"Apple ML Research Model License: noncommercial research purposes",
"runtime":{"torch":torch.__version__,"gpu":torch.cuda.get_device_name(0),"cudnn_enabled":False},"cases":reports},indent=2)+"\n")
if __name__=="__main__":
main()
compress_web.py
View files on GitHub"""Compress the complete 3DGS PLY for Spark without dropping Gaussians.
Requires Node/npm. Uses a pinned MIT-licensed PlayCanvas CLI; no data is uploaded.
"""
import argparse
import gzip
import hashlib
import json
from pathlib import Path
import struct
import subprocess
def main():
parser=argparse.ArgumentParser()
parser.add_argument("ply",type=Path)
parser.add_argument("output",type=Path)
args=parser.parse_args()
subprocess.run(["npm","exec","--yes","--package=@playcanvas/splat-transform@3.12.0","--","splat-transform",
str(args.ply),"--filter-harmonics","1",str(args.output),"--spz-version","3"],check=True)
data=args.output.read_bytes()
magic,version,count,degree,fractional_bits,flags,reserved=struct.unpack_from("<IIIBBBB",gzip.decompress(data))
report={"file":args.output.name,"format":"SPZ v3","gaussians":count,"sh_degree":degree,"bytes":len(data),
"sha256":hashlib.sha256(data).hexdigest(),"converter":"PlayCanvas splat-transform 3.12.0 (MIT)",
"modification":"Keep all Gaussians; quantize SPZ and retain SH through degree 1; full PLY remains degree 3"}
args.output.with_suffix(".json").write_text(json.dumps(report,indent=2)+"\n")
print(json.dumps(report,indent=2))
if __name__=="__main__":
main()
check.py
View files on GitHub"""Inspect 3DGS evidence with Python's standard library; no CUDA or NumPy required."""
import hashlib
import gzip
import json
import math
from pathlib import Path
import struct
def main():
root=Path(__file__).resolve().parents[2]
folder=root/"labs/spatial-reconstruction"
record=json.loads((folder/"recorded.json").read_text())
assert record["input_views"]==record["training_views"]+record["held_out_views"]==72
assert record["held_out_views"]==9
assert all(i%8!=0 for i in record["input"]["seed_views"])
cameras=json.loads((folder/"evidence/cameras.json").read_text())
assert len(cameras)==72
for camera in cameras:
m=camera["world_to_camera"]
assert len(m)==4 and all(len(row)==4 and all(math.isfinite(x) for x in row) for row in m)
det=(m[0][0]*(m[1][1]*m[2][2]-m[1][2]*m[2][1])-m[0][1]*(m[1][0]*m[2][2]-m[1][2]*m[2][0])+m[0][2]*(m[1][0]*m[2][1]-m[1][1]*m[2][0]))
assert math.isclose(det,1,abs_tol=1e-5), "Camera axes are not a right-handed rotation"
sample=json.loads((folder/"evidence/point-sample.json").read_text())
u,v=sample["pixel_uv"];K=sample["intrinsics"];d=sample["depth_m"]
point=[(u-K[0][2])/K[0][0]*d,(v-K[1][2])/K[1][1]*d,d]
assert all(math.isclose(a,b,abs_tol=1e-6) for a,b in zip(point,sample["point_camera_m"]))
m=sample["camera_to_world"]
world=[sum(m[i][j]*point[j] for j in range(3))+m[i][3] for i in range(3)]
assert all(math.isclose(a,b,abs_tol=1e-5) for a,b in zip(world,sample["point_world_m"]))
media=root/"site/public/showcase"
raw=(media/"town03.spz").read_bytes()
assert len(raw)==record["web_preview"]["bytes"]
assert hashlib.sha256(raw).hexdigest()==record["web_preview"]["sha256"]
unpacked=gzip.decompress(raw)
magic,version,count,degree,_,_,_=struct.unpack_from('<IIIBBBB',unpacked)
assert unpacked[:4]==b'NGSP' and version==3 and degree==1
assert count==record["gaussians"]==record["web_preview"]["gaussians"]
assert len(unpacked)==16+count*(20+((degree+1)**2-1)*3), "SPZ payload is incomplete"
assert hashlib.sha256((media/"spatial-reconstruction.mp4").read_bytes()).hexdigest()==record["render"]["sha256"]
sharp=json.loads((folder/"sharp-recorded.json").read_text())
for case in sharp["cases"]:
assert case["gaussians"]==1_179_648 and case["focal_px"]==case["input_resolution"][0]/2
assert hashlib.sha256((media/f'sharp-{case["video"]}').read_bytes()).hexdigest()==case["video_sha256"]
print(f'{record["training_views"]} training / {record["held_out_views"]} held-out views; no held-out depth in seed')
print(f'Full model: {record["gaussians"]:,} Gaussians, held-out PSNR {record["held_out_psnr_db"]} dB')
print(f'Camera transforms, RGB-D sample, {len(raw):,}-byte browser model and 3 videos verified.')
if __name__=="__main__":
main()

