Reconstruct an explorable 3DGS plaza

CARLA camera views become an fVDB radiance field. Move the viewpoint in the browser, or watch the full model rendered along a new camera path.

  • fVDB Reality Capture
  • CARLA
  • Spark / World Labs
  • Apple SHARP
Explore a plaza reconstructed with 3DGS
Recorded runExample run and recording

How do camera images become a space you can explore?

How the data moves
  1. InputCapture the measured world72 RGB images + K + 4×4 poses + depth (m)

    CARLA supplies known camera geometry. Reserve every eighth view.

  2. ProcessOptimize a radiance field63 training images → 345,318 Gaussians

    fVDB adjusts position, scale, opacity and color. Held-out depth is excluded from initialization.

  3. OutputRender another viewpointfull PLY → MP4 / 345k-point SPZ → Spark

    The browser view loads on demand. No physics or collision mesh is included.

The interactive scene below is a real fVDB Gaussian Splatting reconstruction of a CARLA fountain plaza. It is not a panorama on a sphere or a video playing behind a camera. The renderer projects 3D Gaussians from your current viewpoint. Use the 3D button to orbit, pan and move closer; use the video if your browser does not support the viewer.

Input and output flow

Images, calibration and depth have different roles

We captured 72 RGB images at 640 × 360, along with camera intrinsics K, world-to-camera matrices, and simulated depth in meters. The camera's horizontal FOV is 80°, giving fx = fy ≈ 381.36 px. A pixel ray is derived from (u-cx)/fx, (v-cy)/fy, and its depth. Transforming that point by the camera-to-world matrix places it in the shared 3D scene. This example uses known simulator cameras, so it does not need COLMAP to estimate poses.

JSON
{
  "image_path": "view_000.png",
  "width": 640,
  "height": 360,
  "camera_intrinsics": [[381.361, 0, 320], [0, 381.361, 180], [0, 0, 1]],
  "world_to_camera": "4 x 4 matrix; see evidence/cameras.json"
}

The matrix values above are rounded for reading. The saved cameras contain full values. CARLA uses a left-handed world; our exporter reflects world Y and converts camera axes to OpenCV's right/down/forward convention. A wrong sign here would mirror the reconstructed scene.

For a concrete point, pixel (320,180) in view_000.png has depth 21.237 m. Its camera-space point is approximately [0,0,21.237] m; applying the saved camera pose gives world position [0.958,0,2.131] m. The actual sample includes full precision, and check.py recomputes the transformation.

What is a Gaussian?

Each Gaussian stores a center [x,y,z], three scales, a rotation quaternion, an opacity and color coefficients. It is a soft ellipsoid in space. fVDB projects and blends these ellipsoids, compares the result with a training image, and adjusts them to reduce rendering error. It also splits or removes Gaussians during refinement. The final model has 345,318 Gaussians. fVDB Reality Capture

The 6.15 MB browser SPZ keeps all 345,318 Gaussians with quantized parameters and degree-1 color coefficients. The full recording uses the original degree-3 spherical-harmonic model. Appearance can therefore differ slightly between the interactive preview and the video. A radiance field describes appearance; collision surfaces, mass and a robot's control loop require separate data.

How is this different from a generated world?

This experiment reconstructs a supplied scene. It does not invent unseen streets from a text prompt. World Labs Marble and NVIDIA Lyra address generative 3D worlds; Infinigen constructs geometry from procedural rules. The 3D worlds comparison separates those outputs and their execution requirements.

Two additional clips below use Apple SHARP on a single CARLA camera image. We supply the known 90° intrinsics instead of relying on missing EXIF. SHARP predicts 1,179,648 Gaussians per image; fVDB then renders a small nearby camera movement. It does not reconstruct the back of every object or produce a complete 360° world. Its weights are for noncommercial research, unlike its separately licensed code.

First time? Get the lab files

Open a terminal on the Isaac Sim machine for GPU labs, or on your own computer for CPU-only steps. Press Enter after each line. You can copy a multi-line command block as a whole.

Shell
cd "$HOME"
git clone https://github.com/james-joobs/awesome-nvidia-physical-ai.git
cd awesome-nvidia-physical-ai
pwd

This example clones into your home folder. If you already cloned the repo, skip git clone and enter your project folder. In each new terminal, check that pwd ends in awesome-nvidia-physical-ai so commands can find the labs/ files.

Capture and reconstruct a new scene

1. Capture known camera views

Follow CARLA setup to start the pinned 0.9.16 server in terminal A. In the CARLA Python environment, from the repository in terminal B:

Shell
python labs/spatial-reconstruction/capture.py --output runs/plaza-input

The command moves RGB and depth sensors around a static Town03 plaza. Wait for Saved view 72/72. images/ stores color PNGs, depth/ stores meter-valued NumPy arrays, and cameras.json records the exact camera poses. prepare_seed.py builds the initial point cloud without using held-out images or their depth. Index 0,8,16,… is reserved for evaluation.

Stop your CARLA container before starting the next GPU job. The Python environments are separate: a CARLA wheel is not an fVDB installation.

2. Optimize the 3D Gaussians

Install fVDB and fVDB Reality Capture using their official installation guide, matching PyTorch/CUDA to your GPU. Activate that environment and install plyfile, Pillow and SciPy if missing. Our verified run used fVDB 0.6.0 and PyTorch 2.13.0+cu130 on an RTX 5090; see the runtime in recorded.json. Run:

Shell
python labs/spatial-reconstruction/reconstruct.py \
  --input runs/plaza-input --output runs/plaza-3dgs --ffmpeg ffmpeg

The script uses 63 RGB training views for 40 epochs. It leaves camera poses fixed, exports the full PLY and the lighter .splat, renders a new camera path, and writes evaluation metrics. In the recorded environment, cuDNN library versions conflicted, so this script disables cuDNN and uses native PyTorch CUDA convolutions; fVDB training and rendering still run on the GPU. That workaround is explicit in the source and record.

For the web preview, keep every Gaussian and compress the full PLY with a pinned PlayCanvas CLI:

Shell
python3 labs/spatial-reconstruction/compress_web.py runs/plaza-3dgs/town03.ply runs/plaza-3dgs/town03.spz

3. Optional single-image comparison

Install SHARP and its dependencies in an environment where fVDB also imports. Review the model's research-only license before downloading the official checkpoint. Put the ego JPEGs from the CARLA exercise in runs/sharp-input/, then run:

Shell
python labs/spatial-reconstruction/reconstruct_single.py \
  --input runs/sharp-input --output runs/sharp-output \
  --checkpoint /path/to/sharp_2572gikvuh.pt --ffmpeg ffmpeg

This helper assumes those inputs are our 90° CARLA cameras. Change the focal length for other images. It uses a ±0.35 m lateral camera movement; large translations would reveal unsupported regions.

Inspect the result without a GPU

CARLA RGB / fVDB 3DGS

Shell
python3 labs/spatial-reconstruction/check.py

The bundled checker verifies the split, camera rotations, finite model parameters, preview size and recorded video hashes. The recorded run has 9 held-out RGB views with a mean 23.434 dB PSNR. The photometric fit excludes those RGBs, and the seed excludes their depth and colors. This is a small synthetic scene measurement, not a cross-dataset benchmark. Water reflections, distant backgrounds, holes and views outside the captured region remain imperfect.

The repository includes camera metadata, code, metrics, compressed recordings and the browser preview. Media provenance and license scope · Generate an indoor scene · fVDB theory.

Explore the reconstructed scene

Explore a CARLA plaza reconstructed with fVDB. The model and viewer load only when requested. This radiance field has no physics or collision mesh.

Fountain plaza reconstructed with fVDB

No model download until you choose to load it.

Drag to orbit, scroll or pinch to zoom, right-drag to pan. Focus the scene and use W/A/S/D to move. Gaps and blurred water are reconstruction artifacts.

The SPZ keeps all 345,318 Gaussians with degree-1 color. The recording and PSNR use the full degree-3 model. Spark / World Labs (MIT) · CARLA assets / CC-BY

Example run and recording

City from one image

SHARP predicts 3D Gaussians from our CARLA image. fVDB renders nearby views. Research-only model weights; no complete 360° scene.

Suburb from one image

SHARP predicts 3D Gaussians from our CARLA image. fVDB renders nearby views. Research-only model weights; no complete 360° scene.

Source code

capture.pyView files on GitHub
Python
"""Capture a static CARLA neighborhood with known cameras for fVDB reconstruction.

This is a measured RGB-D dataset, not a generated world or an SfM estimate.
"""
import argparse
import json
import math
import queue
from pathlib import Path

import carla
import numpy as np
from PIL import Image
from prepare_seed import prepare


def read_frame(q, frame):
    while True:
        image = q.get(timeout=30)
        if image.frame == frame:
            return np.frombuffer(image.raw_data, np.uint8).reshape(image.height, image.width, 4).copy()
        if image.frame > frame:
            raise RuntimeError("Camera timestamps do not match")


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--output", required=True, type=Path)
    parser.add_argument("--port", type=int, default=24000)
    parser.add_argument("--views", type=int, default=72)
    args = parser.parse_args()
    output = args.output
    (output / "images").mkdir(parents=True, exist_ok=True)
    (output / "depth").mkdir(exist_ok=True)
    client = carla.Client("127.0.0.1", args.port)
    client.set_timeout(90)
    world = client.load_world("Town03")
    previous = world.get_settings()
    settings = world.get_settings()
    settings.synchronous_mode = True
    settings.fixed_delta_seconds = .05
    world.apply_settings(settings)
    world.set_weather(carla.WeatherParameters.ClearNoon)
    width, height, fov = 640, 360, 80
    focal = width / (2*math.tan(math.radians(fov/2)))
    intrinsics = np.array([[focal,0,width/2],[0,focal,height/2],[0,0,1]])
    sensors = []
    cameras = []
    # Reflect the world Y axis to make a right-handed reconstruction world.
    world_basis = np.diag([1.,-1.,1.,1.])
    camera_basis = np.array([[0,0,1,0],[1,0,0,0],[0,-1,0,0],[0,0,0,1.]])
    try:
        for kind in ("rgb", "depth"):
            bp = world.get_blueprint_library().find(f"sensor.camera.{kind}")
            for key,value in {"image_size_x":width,"image_size_y":height,"fov":fov,"sensor_tick":0}.items():
                bp.set_attribute(key,str(value))
            if kind == "rgb":
                bp.set_attribute("motion_blur_intensity","0")
            sensor = world.spawn_actor(bp, carla.Transform())
            q=queue.Queue()
            sensor.listen(q.put)
            sensors.append((sensor,q))
        for i in range(args.views):
            # Two interleaved rings; every eighth pose is held out during training.
            angle=2*math.pi*i/args.views
            radius=22+2*math.sin(3*angle)
            x,y=radius*math.cos(angle),radius*math.sin(angle)
            z=5+2*math.sin(2*angle)
            yaw=math.degrees(math.atan2(-y,-x))
            pitch=math.degrees(math.atan2(2-z, radius))
            transform=carla.Transform(carla.Location(x=x,y=y,z=z),carla.Rotation(pitch=pitch,yaw=yaw))
            for sensor,_ in sensors:
                sensor.set_transform(transform)
            for _ in range(6):
                frame=world.tick()
                data=[read_frame(q,frame) for _,q in sensors]
            rgb=data[0][:,:,:3][:,:,::-1].copy()
            raw=data[1].astype(np.float64)
            depth=(raw[:,:,2]+raw[:,:,1]*256+raw[:,:,0]*65536)/16777215*1000
            name=f"view_{i:03}.png"
            Image.fromarray(rgb).save(output/"images"/name)
            np.save(output/"depth"/f"view_{i:03}.npy",depth.astype(np.float32))
            c2w=world_basis@np.array(transform.get_matrix())@camera_basis
            assert np.isclose(np.linalg.det(c2w[:3,:3]),1,atol=1e-5)
            cameras.append({"camera_name":name,"image_path":name,"width":width,"height":height,
                "camera_intrinsics":intrinsics.tolist(),"world_to_camera":np.linalg.inv(c2w).tolist(),
                "sensor_frame":frame})
            print(f"Saved view {i+1}/{args.views}",flush=True)
        (output/"cameras.json").write_text(json.dumps(cameras,indent=2)+"\n")
        seed_points=prepare(output)
        report={"backend":"CARLA 0.9.16 / Unreal Engine 4.26","map":"Town03","weather":"ClearNoon",
            "views":len(cameras),"resolution":[width,height],"fov_degrees":fov,
            "seed_points":seed_points,"seed_views":[i for i in range(args.views) if i%3==0 and i%8!=0],"held_out_modulo":8,
            "depth":"CARLA RGB24 depth decoded in meters",
            "camera_convention":"OpenCV x-right y-down z-forward; right-handed world: CARLA Y reflected",
            "source_license":"CARLA-specific assets: CC-BY; see CARLA 0.9.16 README licenses"}
        (output/"capture.json").write_text(json.dumps(report,indent=2)+"\n")
    finally:
        for sensor,_ in sensors:
            sensor.stop()
            sensor.destroy()
        world.apply_settings(previous)


if __name__=="__main__":
    main()
prepare_seed.pyView files on GitHub
Python
"""Seed 3DGS only from training-view depth and colors; reserve indices 0,8,16,..."""
import argparse
import json
from pathlib import Path

import numpy as np
from PIL import Image


def prepare(folder):
    cameras=json.loads((folder/"cameras.json").read_text())
    clouds,colors=[],[]
    for i,camera in enumerate(cameras):
        if i%3 or i%8==0:
            continue
        rgb=np.array(Image.open(folder/"images"/camera["image_path"]).convert("RGB"))
        depth=np.load(folder/"depth"/f"view_{i:03}.npy")
        h,w=depth.shape
        K=np.array(camera["camera_intrinsics"])
        c2w=np.linalg.inv(camera["world_to_camera"])
        v,u=np.mgrid[0:h:6,0:w:6]
        d=depth[v,u]
        good=(d>.5)&(d<80)
        cv=np.stack([(u-K[0,2])/K[0,0]*d,(v-K[1,2])/K[1,1]*d,d],-1)[good]
        clouds.append(cv@c2w[:3,:3].T+c2w[:3,3])
        colors.append(rgb[v,u][good])
    xyz,rgb=np.concatenate(clouds),np.concatenate(colors)
    _,indices=np.unique(np.floor(xyz/.12).astype(int),axis=0,return_index=True)
    xyz,rgb=xyz[indices],rgb[indices]
    with (folder/"pointcloud.ply").open("w") as f:
        f.write("ply\nformat ascii 1.0\nelement vertex %d\n"%len(xyz))
        f.write("property float x\nproperty float y\nproperty float z\nproperty uchar red\nproperty uchar green\nproperty uchar blue\nend_header\n")
        for p,c in zip(xyz,rgb):
            f.write("%.5f %.5f %.5f %d %d %d\n"%(*p,*c))
    return len(xyz)


if __name__=="__main__":
    parser=argparse.ArgumentParser()
    parser.add_argument("folder",type=Path)
    args=parser.parse_args()
    count=prepare(args.folder)
    report=json.loads((args.folder/"capture.json").read_text())
    report.update(seed_points=count,seed_views=[i for i in range(len(json.loads((args.folder/"cameras.json").read_text()))) if i%3==0 and i%8!=0],held_out_modulo=8)
    (args.folder/"capture.json").write_text(json.dumps(report,indent=2)+"\n")
    print(f"Seeded {count} points; held-out RGB and depth excluded.")
reconstruct.pyView files on GitHub
Python
"""Train an fVDB 3DGS scene from known CARLA cameras; hold out every eighth image."""
import argparse
import hashlib
import json
import math
import subprocess
import time
from pathlib import Path

import fvdb
import numpy as np
import torch
from PIL import Image
from plyfile import PlyData
from scipy.spatial.transform import Rotation, Slerp
from fvdb_reality_capture.radiance_fields import GaussianSplatReconstruction, GaussianSplatReconstructionConfig, GaussianSplatOptimizerConfig
from fvdb_reality_capture.sfm_scene import SfmScene
from fvdb_reality_capture.sfm_scene.sfm_scene import SpatialScaleMode


def export_browser_splat(source, target, maximum=200_000):
    """32-byte .splat, DC color only, deterministic opacity-weighted subset.

    Browser preview is deliberately smaller than the original SH-degree-3 model.
    """
    vertices = PlyData.read(source)["vertex"].data
    opacity = 1/(1+np.exp(-np.clip(vertices["opacity"],-40,40)))
    valid = np.flatnonzero(opacity>.04)
    if len(valid)>maximum:
        rng=np.random.default_rng(42)
        weights=opacity[valid].astype(np.float64)
        valid=np.sort(rng.choice(valid,maximum,replace=False,p=weights/weights.sum()))
    v=vertices[valid]
    packed=np.empty(len(v),dtype=[("position","<f4",3),("scale","<f4",3),("rgba","u1",4),("rotation","u1",4)])
    packed["position"]=np.stack([v[k] for k in ("x","y","z")],-1)
    packed["scale"]=np.exp(np.stack([v[f"scale_{i}"] for i in range(3)],-1))
    color=.5+.28209479177387814*np.stack([v[f"f_dc_{i}"] for i in range(3)],-1)
    packed["rgba"][:,:3]=np.clip(color*255,0,255).astype(np.uint8)
    packed["rgba"][:,3]=np.clip(opacity[valid]*255,0,255).astype(np.uint8)
    quat=np.stack([v[f"rot_{i}"] for i in range(4)],-1)
    quat/=np.linalg.norm(quat,axis=1,keepdims=True).clip(1e-8)
    packed["rotation"]=np.clip(quat*128+128,0,255).astype(np.uint8)
    packed.tofile(target)
    return {"gaussians":len(v),"bytes":target.stat().st_size,"sha256":hashlib.sha256(target.read_bytes()).hexdigest(),
        "format":"32-byte .splat; DC color; opacity > .04; deterministic weighted sampling if needed"}


def main():
    parser=argparse.ArgumentParser()
    parser.add_argument("--input",type=Path,required=True)
    parser.add_argument("--output",type=Path,required=True)
    parser.add_argument("--epochs",type=int,default=40)
    parser.add_argument("--ffmpeg",default="ffmpeg")
    args=parser.parse_args()
    args.output.mkdir(parents=True,exist_ok=True)
    torch.backends.cudnn.enabled=False  # GPU workstation cuDNN library mismatch; native CUDA convolutions still work.
    scene=SfmScene.from_simple_directory(args.input)
    config=GaussianSplatReconstructionConfig(max_epochs=args.epochs,optimize_camera_poses=False,
        refine_stop_epoch=args.epochs,near_plane=.05,far_plane=200)
    optimizer=GaussianSplatOptimizerConfig(spatial_scale_mode=SpatialScaleMode.MAX_CAMERA_TO_CENTROID,max_gaussians=750_000)
    runner=GaussianSplatReconstruction.from_sfm_scene(scene,config=config,optimizer_config=optimizer,use_every_n_as_val=8)
    start=time.monotonic()
    runner.optimize()
    seconds=time.monotonic()-start
    model=runner.model
    model_path=args.output/"town03.ply"
    runner.save_ply(model_path)
    browser=export_browser_splat(model_path,args.output/"town03.splat")
    width,height=640,360
    K=np.array(scene.images[0].camera_metadata.projection_matrix,dtype=np.float32)

    def render(c2w):
        with torch.inference_mode():
            rgb,alpha=model.render_images(torch.tensor(np.linalg.inv(c2w)[None],dtype=torch.float32,device="cuda"),
                torch.tensor(K[None],dtype=torch.float32,device="cuda"),width,height,.05,200)
        return rgb[0,:,:,:3].clamp(0,1).cpu().numpy(),float(alpha.mean())

    metrics=[]
    for i in range(0,len(scene.images),8):
        pred,alpha=render(scene.camera_to_world_matrices[i])
        ref=np.array(Image.open(scene.images[i].image_path).convert("RGB"))/255
        mse=np.mean((pred-ref)**2)
        metrics.append({"view":i,"psnr_db":round(-10*math.log10(max(float(mse),1e-12)),3),"mean_alpha":round(alpha,4)})
        if i==0:
            Image.fromarray((pred*255).astype(np.uint8)).save(args.output/"held-out-render.png")
            Image.fromarray((ref*255).astype(np.uint8)).save(args.output/"held-out-reference.png")
    poses=scene.camera_to_world_matrices
    keys=np.arange(len(poses)+1,dtype=float)
    closed=np.concatenate([poses,poses[:1]],axis=0)
    rotations=Slerp(keys,Rotation.from_matrix(closed[:,:3,:3]))
    fps,frames=24,384
    movie=args.output/"flythrough.mp4"
    encoder=subprocess.Popen([args.ffmpeg,"-hide_banner","-loglevel","error","-y","-f","rawvideo","-pix_fmt","rgb24",
        "-s",f"{width}x{height}","-r",str(fps),"-i","pipe:0","-an","-c:v","libx264","-crf","22",
        "-pix_fmt","yuv420p","-movflags","+faststart",str(movie)],stdin=subprocess.PIPE)
    alphas=[]
    for t in np.linspace(0,len(poses),frames,endpoint=False):
        pose=np.eye(4)
        pose[:3,:3]=rotations(float(t)).as_matrix()
        pose[:3,3]=[np.interp(t,keys,closed[:,axis,3]) for axis in range(3)]
        rgb,alpha=render(pose)
        alphas.append(alpha)
        encoder.stdin.write((rgb*255).astype(np.uint8).tobytes())
    encoder.stdin.close()
    assert encoder.wait()==0
    report={"mode":"multi-view reconstruction, not generative world-model inference",
        "input":json.loads((args.input/"capture.json").read_text()),"input_views":len(poses),
        "training_views":len(poses)-len(metrics),"held_out_views":len(metrics),"epochs":args.epochs,
        "optimization_seconds":round(seconds,3),"gaussians":int(model.num_gaussians),
        "held_out_psnr_db":round(float(np.mean([m["psnr_db"] for m in metrics])),3),"per_view":metrics,
        "runtime":{"fvdb":fvdb.__version__,"torch":torch.__version__,"gpu":torch.cuda.get_device_name(0),"cudnn_enabled":False},
        "browser":browser,"render":{"frames":frames,"fps":fps,"resolution":[width,height],
        "median_alpha":float(np.median(alphas)),"sha256":hashlib.sha256(movie.read_bytes()).hexdigest()},
        "limitations":["Static radiance field; no collision mesh or dynamic traffic", "CARLA water and distant sky are difficult to reconstruct", "Intermediate .splat uses sampled DC-only data; use compress_web.py on the full PLY for the published SPZ"]}
    (args.output/"result.json").write_text(json.dumps(report,indent=2)+"\n")
    print(json.dumps(report,indent=2),flush=True)


if __name__=="__main__":
    main()
reconstruct_single.pyView files on GitHub
Python
"""Run SHARP on our CARLA inputs, using the known 90-degree camera intrinsics.

SHARP weights are research-only. This predicts nearby views, not a complete 360 world.
"""
import argparse
import hashlib
import json
import math
import subprocess
import time
from pathlib import Path

import numpy as np
import torch
from PIL import Image
from fvdb_reality_capture import GaussianSplat3d
from sharp.cli.predict import predict_image
from sharp.models import PredictorParams, create_predictor
from sharp.utils.gaussians import save_ply


def main():
    parser=argparse.ArgumentParser()
    parser.add_argument("--input",type=Path,required=True)
    parser.add_argument("--output",type=Path,required=True)
    parser.add_argument("--checkpoint",type=Path,required=True)
    parser.add_argument("--ffmpeg",default="ffmpeg")
    args=parser.parse_args()
    args.output.mkdir(parents=True,exist_ok=True)
    torch.backends.cudnn.enabled=False
    predictor=create_predictor(PredictorParams())
    predictor.load_state_dict(torch.load(args.checkpoint,weights_only=True))
    predictor.eval().cuda()
    reports=[]
    for image_path in sorted(args.input.glob("*.jpg")):
        image=np.array(Image.open(image_path).convert("RGB"))
        height,width=image.shape[:2]
        f_px=width/2 # Our CARLA camera has a known horizontal FOV of 90 degrees.
        torch.cuda.synchronize()
        start=time.monotonic()
        with torch.inference_mode():
            gaussians=predict_image(predictor,image,f_px,torch.device("cuda"))
        torch.cuda.synchronize()
        seconds=time.monotonic()-start
        ply=args.output/f"{image_path.stem}.ply"
        save_ply(gaussians,f_px,(height,width),ply)
        del gaussians
        splats,_=GaussianSplat3d.from_ply(str(ply))
        w,h,fps,frames=640,360,24,192
        K=np.array([[f_px*w/width,0,(w-1)/2],[0,f_px*h/height,(h-1)/2],[0,0,1]])
        movie=args.output/f"{image_path.stem}.mp4"
        encoder=subprocess.Popen([args.ffmpeg,"-hide_banner","-loglevel","error","-y","-f","rawvideo","-pix_fmt","rgb24",
            "-s",f"{w}x{h}","-r",str(fps),"-i","pipe:0","-an","-c:v","libx264","-crf","22",
            "-pix_fmt","yuv420p","-movflags","+faststart",str(movie)],stdin=subprocess.PIPE)
        for i in range(frames):
            phase=i/(frames-1)*2*math.pi
            eye=np.array([.35*math.sin(phase),0,.1*(1-math.cos(phase))])
            forward=np.array([0,0,12])-eye
            forward/=np.linalg.norm(forward)
            right=np.cross([0,1,0],forward);right/=np.linalg.norm(right)
            pose=np.eye(4);pose[:3,:3]=np.stack([right,np.cross(forward,right),forward],axis=1);pose[:3,3]=eye
            with torch.inference_mode():
                rgb,_=splats.render_images(torch.tensor(np.linalg.inv(pose)[None],dtype=torch.float32,device="cuda"),
                    torch.tensor(K[None],dtype=torch.float32,device="cuda"),w,h,.01,1000)
            rgb=rgb[0,:,:,:3].clamp(0,1).cpu().numpy()
            encoder.stdin.write((rgb*255).astype(np.uint8).tobytes())
            if i==0:
                Image.fromarray((rgb*255).astype(np.uint8)).save(args.output/f"{image_path.stem}.jpg",quality=90)
        encoder.stdin.close()
        assert encoder.wait()==0
        reports.append({"input":image_path.name,"input_sha256":hashlib.sha256(image_path.read_bytes()).hexdigest(),
            "input_resolution":[width,height],"focal_px":f_px,"inference_seconds":round(seconds,3),
            "gaussians":int(splats.num_gaussians),"video":movie.name,"video_sha256":hashlib.sha256(movie.read_bytes()).hexdigest(),
            "frames":frames,"fps":fps,"camera_motion_m":{"x_amplitude":.35,"z_range":[0,.2]}})
        del splats
    (args.output/"result.json").write_text(json.dumps({"tool":"Apple SHARP","mode":"single-view reconstruction; nearby novel views",
        "renderer":"fVDB","weights_license":"Apple ML Research Model License: noncommercial research purposes",
        "runtime":{"torch":torch.__version__,"gpu":torch.cuda.get_device_name(0),"cudnn_enabled":False},"cases":reports},indent=2)+"\n")


if __name__=="__main__":
    main()
compress_web.pyView files on GitHub
Python
"""Compress the complete 3DGS PLY for Spark without dropping Gaussians.

Requires Node/npm. Uses a pinned MIT-licensed PlayCanvas CLI; no data is uploaded.
"""
import argparse
import gzip
import hashlib
import json
from pathlib import Path
import struct
import subprocess


def main():
    parser=argparse.ArgumentParser()
    parser.add_argument("ply",type=Path)
    parser.add_argument("output",type=Path)
    args=parser.parse_args()
    subprocess.run(["npm","exec","--yes","--package=@playcanvas/splat-transform@3.12.0","--","splat-transform",
        str(args.ply),"--filter-harmonics","1",str(args.output),"--spz-version","3"],check=True)
    data=args.output.read_bytes()
    magic,version,count,degree,fractional_bits,flags,reserved=struct.unpack_from("<IIIBBBB",gzip.decompress(data))
    report={"file":args.output.name,"format":"SPZ v3","gaussians":count,"sh_degree":degree,"bytes":len(data),
        "sha256":hashlib.sha256(data).hexdigest(),"converter":"PlayCanvas splat-transform 3.12.0 (MIT)",
        "modification":"Keep all Gaussians; quantize SPZ and retain SH through degree 1; full PLY remains degree 3"}
    args.output.with_suffix(".json").write_text(json.dumps(report,indent=2)+"\n")
    print(json.dumps(report,indent=2))


if __name__=="__main__":
    main()
check.pyView files on GitHub
Python
"""Inspect 3DGS evidence with Python's standard library; no CUDA or NumPy required."""
import hashlib
import gzip
import json
import math
from pathlib import Path
import struct


def main():
    root=Path(__file__).resolve().parents[2]
    folder=root/"labs/spatial-reconstruction"
    record=json.loads((folder/"recorded.json").read_text())
    assert record["input_views"]==record["training_views"]+record["held_out_views"]==72
    assert record["held_out_views"]==9
    assert all(i%8!=0 for i in record["input"]["seed_views"])
    cameras=json.loads((folder/"evidence/cameras.json").read_text())
    assert len(cameras)==72
    for camera in cameras:
        m=camera["world_to_camera"]
        assert len(m)==4 and all(len(row)==4 and all(math.isfinite(x) for x in row) for row in m)
        det=(m[0][0]*(m[1][1]*m[2][2]-m[1][2]*m[2][1])-m[0][1]*(m[1][0]*m[2][2]-m[1][2]*m[2][0])+m[0][2]*(m[1][0]*m[2][1]-m[1][1]*m[2][0]))
        assert math.isclose(det,1,abs_tol=1e-5), "Camera axes are not a right-handed rotation"
    sample=json.loads((folder/"evidence/point-sample.json").read_text())
    u,v=sample["pixel_uv"];K=sample["intrinsics"];d=sample["depth_m"]
    point=[(u-K[0][2])/K[0][0]*d,(v-K[1][2])/K[1][1]*d,d]
    assert all(math.isclose(a,b,abs_tol=1e-6) for a,b in zip(point,sample["point_camera_m"]))
    m=sample["camera_to_world"]
    world=[sum(m[i][j]*point[j] for j in range(3))+m[i][3] for i in range(3)]
    assert all(math.isclose(a,b,abs_tol=1e-5) for a,b in zip(world,sample["point_world_m"]))
    media=root/"site/public/showcase"
    raw=(media/"town03.spz").read_bytes()
    assert len(raw)==record["web_preview"]["bytes"]
    assert hashlib.sha256(raw).hexdigest()==record["web_preview"]["sha256"]
    unpacked=gzip.decompress(raw)
    magic,version,count,degree,_,_,_=struct.unpack_from('<IIIBBBB',unpacked)
    assert unpacked[:4]==b'NGSP' and version==3 and degree==1
    assert count==record["gaussians"]==record["web_preview"]["gaussians"]
    assert len(unpacked)==16+count*(20+((degree+1)**2-1)*3), "SPZ payload is incomplete"
    assert hashlib.sha256((media/"spatial-reconstruction.mp4").read_bytes()).hexdigest()==record["render"]["sha256"]
    sharp=json.loads((folder/"sharp-recorded.json").read_text())
    for case in sharp["cases"]:
        assert case["gaussians"]==1_179_648 and case["focal_px"]==case["input_resolution"][0]/2
        assert hashlib.sha256((media/f'sharp-{case["video"]}').read_bytes()).hexdigest()==case["video_sha256"]
    print(f'{record["training_views"]} training / {record["held_out_views"]} held-out views; no held-out depth in seed')
    print(f'Full model: {record["gaussians"]:,} Gaussians, held-out PSNR {record["held_out_psnr_db"]} dB')
    print(f'Camera transforms, RGB-D sample, {len(raw):,}-byte browser model and 3 videos verified.')


if __name__=="__main__":
    main()