How does a prompt become a place you can look around?
- InputDescribe or show a place
prompt / RGB + 3×3 camera K + fill maskText creates a full panorama. An image fixes observed pixels and generates its surroundings.
- ProcessGenerate appearance and infer distance
FLUX + panorama LoRA → RGB 1600×800 → DA-2The distance map has one inferred ray distance per pixel. Max 20 is a normalization, not measured meters.
- OutputBuild and render a persistent scene
1,280,000 Gaussians + mesh → PLY / SPZ / MP4XYZ = unit ray × distance. Translated cameras reveal parallax and hidden-surface artifacts.
WorldGen first generates the appearance of a 360° scene, then estimates how far its surfaces are from the camera. Those two arrays become a persistent Gaussian scene and a colored mesh. A renderer can then move the camera through this saved geometry. In this lab, you will create a robotics laboratory from text, then expand an image into a surrounding room. An additional forest run shows what changes outdoors.
1. Generate appearance, then estimate geometry
For text input, FLUX.1-dev and WorldGen's panorama LoRA generate an RGB array of shape [800, 1600, 3]. The horizontal axis covers 360° and the vertical axis covers 180°. A panorama is still an image; it has no explicit 3D surface positions yet.
DA-2 reads that panorama and predicts a distance array [800, 1600]. Each value describes distance along the pixel's viewing ray, unlike camera-plane Z depth in the Replicator lab. WorldGen scales this estimate so its maximum is 20. This is an arbitrary normalization of inferred geometry, not a measurement in meters.

The distance preview is brighter for farther inferred surfaces. It is a visualization of model output, not ground truth. A convincing panorama can still have bent tables, uncertain distances or inconsistent hidden surfaces.
2. Turn each ray into a Gaussian
For each panorama pixel, compute a unit direction ray = [x, y, z]. Multiply it by the predicted distance:
XYZ = ray × distance
RGB pixel → Gaussian color
angular pixel footprint + distance → Gaussian scale
local surface frame → rotation quaternion [w, x, y, z]
Here the camera is at [0, 0, 0], with X right, Y down and Z forward. The full result has 1,280,000 Gaussians, one per panorama pixel. Each Gaussian carries center, scale, rotation, opacity and color. The PLY stores log scale and logit opacity, so consumers must decode those fields. Our exporter corrects the upstream opacity encoding and clamps degenerate pole scales; it does not train a new model. The mesh connects neighboring back-projected pixels into triangles. It is an inferred surface shell, not a set of separately modeled, articulated objects.
A real pixel from our generated laboratory follows below. Values are rounded; the audit retains full precision. The near-forward ray places this Gaussian mostly along +Z. Distance remains in normalized WorldGen units.
{
"pixel": [
800,
400
],
"rgb": [
179,
175,
163
],
"ray": [
0.001963,
0.001963,
0.999996
],
"distance": 7.208282,
"xyz": [
0.014154,
0.014153,
7.208254
]
}
3. What changes when the input is an image?
We use our own Infinigen synthetic render, not a real photograph, and its known pinhole camera matrix:
{
"resolution": [640, 360],
"K": [[160, 0, 320], [0, 160, 180], [0, 0, 1]]
}
fx=fy=160 controls angular magnification; (cx,cy)=(320,180) is the principal point. Our calibrated adapter projects panorama rays into this source image. A black mask pixel (0) preserves observed RGB, while white (255) requests generated content from FLUX.1-Fill-dev plus WorldGen's image LoRA. We composite the observed pixels back exactly after generation. Preservation applies to the reprojected, bilinearly sampled panorama pixels, not lossless recovery of every original camera pixel.
In our run, 16.30% of panorama pixels came from the calibrated image projection; the remaining 83.70% was generated. The saved keep-region RGB error is zero.

This adapter replaces upstream's ordinary-image-to-panorama mapping; the FLUX panorama model, DA-2 and WorldGen geometry conversion remain upstream implementations. Newly filled regions are invented. Keeping RGB does not guarantee that DA-2 estimates the original object's geometry correctly.
First time? Get the lab files
Open a terminal on the Isaac Sim machine for GPU labs, or on your own computer for CPU-only steps. Press Enter after each line. You can copy a multi-line command block as a whole.
cd "$HOME"
git clone https://github.com/james-joobs/awesome-nvidia-physical-ai.git
cd awesome-nvidia-physical-ai
pwd
This example clones into your home folder. If you already cloned the repo, skip git clone and enter your project folder. In each new terminal, check that pwd ends in awesome-nvidia-physical-ai so commands can find the labs/ files.
Create your own scene step by step
1. Prepare a separate Linux environment
Start at the root of this repository. Use an NVIDIA GPU with a working driver, Git and uv. Run nvidia-smi first: it should list the GPU without an error. This recipe was executed on an RTX 5090 with 32 GB VRAM and 30 GiB system RAM. Reserve about 100 GB of disk space for environments, model downloads and output. The FP4 recipe is for Linux x86_64 and this Blackwell setup; other architectures need their own compatible Nunchaku build.
bash labs/worldgen-scenes/setup.sh runs/worldgen-runtime
runs/worldgen-runtime/.venv/bin/hf auth login
The first command creates Python 3.11 with PyTorch 2.7/cu128, Nunchaku 0.2 and fixed WorldGen/DA-2 source revisions. It does not install into Isaac Sim. The second saves your Hugging Face login locally; do not paste a token into a script or commit it. Before downloading, accept access terms for both FLUX.1-dev and FLUX.1-Fill-dev using the same account.
The runner uses Nunchaku FP4 and CPU offload to fit the machine. DA-2's built-in PyTorch SDPA fallback avoids the older xFormers kernel that fails on RTX 5090. PyTorch3D is installed without native extensions: this workflow only needs its pure-PyTorch quaternion conversion. Native PyTorch3D rasterization is unavailable in this environment. Source and model hashes are in model-pins.json.
2. Generate a place from text
Paste the complete block into a terminal at the repository root. The backslash joins the following line to the same command. Wait for the final JSON message before opening the output.
runs/worldgen-runtime/.venv/bin/python labs/worldgen-scenes/generate.py \
--output runs/worldgen/robot-lab --seed 42 \
--prompt "A spacious realistic robotics laboratory with collaborative robot arms on workbenches, yellow safety markings on a polished concrete floor, metal storage racks, large windows, natural daylight, no people, a coherent modern industrial interior"
The first run downloads fixed model snapshots; later runs reuse the cache. panorama.png appears first, followed by distance.npy, scene.ply, scene-mesh.ply, scene-mesh.glb and result.json. An image alone does not mean 3D conversion succeeded: check for the final result file. On this machine, the successful robotics-lab run took about 44 seconds after cached downloads, including model load; this is a single run, not a hardware benchmark.
3. Expand an image with known camera information
The repository includes the input and calibration used in our run. The camera file's resolution must match the image. An arbitrary photograph needs suitable intrinsics; do not reuse these numbers for another camera.
runs/worldgen-runtime/.venv/bin/python labs/worldgen-scenes/generate.py \
--output runs/worldgen/livingroom \
--image labs/worldgen-scenes/evidence/input.png \
--intrinsics labs/worldgen-scenes/evidence/input-camera.json \
--prompt "A coherent spacious living room continuing the input image, matching the pale walls, wooden furniture, indoor materials and natural window light, realistic interior, no people"
Open generation-mask.png and conditioning.png before judging the result. They identify which regions came from the input and which the model filled. To try the outdoor run, omit the image arguments and use seed 17 with the forest prompt in recorded.json.
4. Render genuinely different viewpoints
Use a separate compatible fVDB Reality Capture environment to run the renderer; do not add it to the WorldGen Torch 2.7 environment. The following path represents that separate interpreter. It must already import fvdb_reality_capture and have ffmpeg available.
runs/fvdb-runtime/bin/python labs/worldgen-scenes/render.py \
--scene runs/worldgen/robot-lab/scene.ply \
--output runs/worldgen/robot-lab/render --ffmpeg ffmpeg
The renderer writes 480 frames at 24 FPS into a 20-second MP4, plus camera matrices in cameras.json. It rotates 360° while changing the camera center by 0.4 / 0.09 / 0.3 world units across X/Y/Z. This is re-rendering from a 3D PLY, rather than cropping a flat panorama. Movement stays near the source viewpoint because unseen surfaces are not reliably reconstructed. All three recordings below use the full Gaussian scene. The browser previews use 320,000 Gaussians, one in four, with widened footprints to reduce download size; load only the scene you want.
Check the data and understand the limits
You can watch the videos and inspect the bundled evidence without CUDA:
python3 labs/worldgen-scenes/check.py
This checks published media hashes, compressed scene headers, the image's generation mask, actual camera translation and saved numeric ray examples. To check a new result, use the generation environment on CPU:
runs/worldgen-runtime/.venv/bin/python labs/worldgen-scenes/audit.py \
--folder runs/worldgen/robot-lab
It opens the real distance array and PLY, checks finite positive distances, normalized rotations and XYZ = ray × distance, and writes geometry-audit.json. For the image route it also verifies exact preservation of observed panorama pixels. A changed prompt or package version can change the output; compare the data checks, not just an expected filename.
The forest recording exposes much more transparency and gaps than the indoor scenes. Thin leaves, depth discontinuities and surfaces uncovered by translation are difficult for a single RGB-distance shell. Alpha coverage is recorded as a render diagnostic, not a geometric accuracy score. We do not have ground-truth geometry or a held-out PSNR benchmark for these generated worlds. None of the results includes validated collisions, metric scale, joints or a robot controller.
Licenses: WorldGen code, its LoRA card and DA-2 weights list Apache-2.0. FLUX base and derived quantized weights retain separate FLUX.1-dev Non-Commercial model terms; output permissions have their own scope. The repository distributes our example outputs and code, not those model weights. Media provenance · Measured fVDB reconstruction · Editable procedural house.
Next: import this laboratory into Isaac Sim, check scale and collisions, and drive a robot.
Generated robotics laboratory
A generated WorldGen surface, without measured scale or collisions. Load the 3D scene only when you want it.

Drag to look around; two-finger or right-drag to pan. Focus the view and use W/A/S/D to translate. Moving far from the source exposes gaps and stretched surfaces.
SPZ preview: 320,000 of 1,280,000 Gaussians with wider footprints. Videos use the full model. Forest gaps are more visible than indoor gaps. Spark / World Labs (MIT) · WorldGen / FLUX terms separate
Expanded living room
A generated WorldGen surface, without measured scale or collisions. Load the 3D scene only when you want it.

Drag to look around; two-finger or right-drag to pan. Focus the view and use W/A/S/D to translate. Moving far from the source exposes gaps and stretched surfaces.
SPZ preview: 320,000 of 1,280,000 Gaussians with wider footprints. Videos use the full model. Forest gaps are more visible than indoor gaps. Spark / World Labs (MIT) · WorldGen / FLUX terms separate
Generated forest trail
A generated WorldGen surface, without measured scale or collisions. Load the 3D scene only when you want it.

Drag to look around; two-finger or right-drag to pan. Focus the view and use W/A/S/D to translate. Moving far from the source exposes gaps and stretched surfaces.
SPZ preview: 320,000 of 1,280,000 Gaussians with wider footprints. Videos use the full model. Forest gaps are more visible than indoor gaps. Spark / World Labs (MIT) · WorldGen / FLUX terms separate
Example run and recording
Expand one image into a 360° living room
Actual model inference on an RTX GPU, followed by a translated camera path rendered from the Gaussian PLY. Distance is inferred and not metric ground truth.
Generate a forest trail from text
Actual model inference on an RTX GPU, followed by a translated camera path rendered from the Gaussian PLY. Distance is inferred and not metric ground truth.
Source code
setup.sh
View files on GitHub#!/usr/bin/env bash
# Linux x86_64 / RTX 5090 recipe, isolated from Isaac Sim and the fVDB renderer.
set -euo pipefail
if command -v uv >/dev/null 2>&1; then
worldgen_uv="$(command -v uv)"
elif [[ -x "$HOME/.local/bin/uv" ]]; then
worldgen_uv="$HOME/.local/bin/uv"
else
echo 'Install uv first: https://docs.astral.sh/uv/getting-started/installation/' >&2
exit 1
fi
worldgen_root="${1:-runs/worldgen-runtime}"
mkdir -p "$worldgen_root"
worldgen_root="$(cd "$worldgen_root" && pwd)"
if [[ ! -x "$worldgen_root/.venv/bin/python" ]]; then "$worldgen_uv" venv --python 3.11 "$worldgen_root/.venv"; fi
worldgen_python="$worldgen_root/.venv/bin/python"
"$worldgen_uv" pip install --python "$worldgen_python" torch==2.7.0 torchvision==0.22.0 --index-url https://download.pytorch.org/whl/cu128
"$worldgen_uv" pip install --python "$worldgen_python" xformers==0.0.30 \
'https://github.com/nunchux-ai/nunchaku/releases/download/v0.2.0/nunchaku-0.2.0%2Btorch2.7-cp311-cp311-linux_x86_64.whl'
"$worldgen_uv" pip install --python "$worldgen_python" diffusers==0.33.1 transformers==4.51.3 peft==0.15.2 \
accelerate==1.7.0 huggingface-hub==0.35.3 timm==1.0.15 opencv-python==4.12.0.88 open3d==0.19.0 \
py360convert einops pillow scikit-image sentencepiece trimesh plyfile imageio imageio-ffmpeg \
matplotlib omegaconf ninja scipy tqdm tensorboard tabulate fvcore iopath viser==1.1.1 setuptools wheel
if [[ ! -d "$worldgen_root/source/.git" ]]; then git clone https://github.com/ZiYang-xie/WorldGen.git "$worldgen_root/source"; fi
git -C "$worldgen_root/source" checkout 7ce7b2767fdf31e2727b69a2e61e2e950e3a017f
if [[ ! -d "$worldgen_root/da2/.git" ]]; then git clone https://github.com/EnVision-Research/DA-2.git "$worldgen_root/da2"; fi
git -C "$worldgen_root/da2" checkout d659838585f2bc9967c7e9367af573795271dbbb
if [[ ! -d "$worldgen_root/pytorch3d/.git" ]]; then git clone https://github.com/facebookresearch/pytorch3d.git "$worldgen_root/pytorch3d"; fi
git -C "$worldgen_root/pytorch3d" checkout ddfc55127fed1a74514cd24db08a7c2e62e76b08
PYTORCH3D_NO_EXTENSION=1 "$worldgen_uv" pip install --python "$worldgen_python" --no-build-isolation --no-deps "$worldgen_root/pytorch3d"
"$worldgen_uv" pip install --python "$worldgen_python" --no-deps \
'git+https://github.com/EasternJournalist/utils3d.git@3913c65d81e05e47b9f367250cf8c0f7462a0900' \
-e "$worldgen_root/source" -e "$worldgen_root/da2/src"
"$worldgen_python" -c 'import torch; from worldgen import WorldGen; from nunchaku.utils import get_precision; print(torch.__version__, torch.cuda.get_device_name(0), get_precision())'
generate.py
View files on GitHub"""Generate a real WorldGen panorama, DA-2 distance field, Gaussian PLY and mesh.
Use the dedicated Python 3.11 runtime in setup.sh. FLUX approvals are required.
The image adapter uses known pinhole calibration instead of treating a normal
camera photograph as an equirectangular panorama. No model weights are changed.
"""
import argparse
import hashlib
import importlib.metadata
import json
import resource
import subprocess
import time
from pathlib import Path
import numpy as np
from PIL import Image
import torch
import torch.nn.functional as F
from huggingface_hub import snapshot_download
from plyfile import PlyData, PlyElement
def project_image(image, intrinsics, width, height, device="cuda"):
"""Inverse-project the panorama's rays into the calibrated source camera."""
from worldgen.utils.general_utils import pano_unit_rays
source = np.array(image.convert("RGB"))
sh, sw = source.shape[:2]
K = np.asarray(intrinsics, dtype=float)
rays = pano_unit_rays(height, width, device)
z = rays[..., 2]
u = K[0, 0] * rays[..., 0] / z.clamp(min=1e-6) + K[0, 2]
v = K[1, 1] * rays[..., 1] / z.clamp(min=1e-6) + K[1, 2]
valid = (z > 0) & (u >= 0) & (u <= sw-1) & (v >= 0) & (v <= sh-1)
grid = torch.stack((u/(sw-1)*2-1, v/(sh-1)*2-1), dim=-1)
rgb = torch.tensor(source, device=device, dtype=torch.float32).permute(2, 0, 1)[None]/255
mapped = F.grid_sample(rgb, grid[None], mode="bilinear", align_corners=True, padding_mode="zeros")[0].permute(1, 2, 0)
mapped = (mapped * valid[..., None] * 255).round().clamp(0, 255).to(torch.uint8).cpu().numpy()
mask = (~valid).to(torch.uint8).cpu().numpy()*255
return Image.fromarray(mapped), Image.fromarray(mask), float(valid.float().mean())
def export_ply(splat, path):
"""Export standard 3DGS fields: log scale, logit opacity and wxyz rotation."""
fields = ["x", "y", "z", "nx", "ny", "nz", "f_dc_0", "f_dc_1", "f_dc_2", "opacity",
"scale_0", "scale_1", "scale_2", "rot_0", "rot_1", "rot_2", "rot_3"]
values = np.concatenate((splat.centers, np.zeros_like(splat.centers),
(splat.rgbs-.5)/.28209479177387814,
np.log(np.clip(splat.opacities, 1e-6, 1-1e-6)/np.clip(1-splat.opacities, 1e-6, 1)),
np.log(np.maximum(splat.scales, 1e-5)), splat.rotations), axis=1).astype(np.float32)
assert np.isfinite(values).all(), "Non-finite Gaussian attributes"
assert np.allclose(np.linalg.norm(splat.rotations, axis=1), 1, atol=1e-3)
vertices = np.empty(len(values), dtype=[(field, "f4") for field in fields])
for i, field in enumerate(fields):
vertices[field] = values[:, i]
PlyData([PlyElement.describe(vertices, "vertex")], text=False).write(path)
return {"gaussians": len(values), "all_finite": True,
"bounds": [splat.centers.min(0).tolist(), splat.centers.max(0).tolist()]}
def main():
parser = argparse.ArgumentParser()
parser.add_argument("--output", type=Path, required=True)
parser.add_argument("--prompt", required=True)
parser.add_argument("--image", type=Path)
parser.add_argument("--intrinsics", type=Path)
parser.add_argument("--pins", type=Path, default=Path(__file__).with_name("model-pins.json"))
parser.add_argument("--width", type=int, default=1600)
parser.add_argument("--steps", type=int, default=50)
parser.add_argument("--seed", type=int, default=42)
args = parser.parse_args()
assert args.width % 32 == 0
if args.image and not args.intrinsics:
parser.error("The image route requires the camera's intrinsics JSON")
output = args.output
output.mkdir(parents=True, exist_ok=True)
pins = json.loads(args.pins.read_text())
snapshots = {r["repo"]: snapshot_download(r["repo"], revision=r["revision"],
ignore_patterns=r.get("omitted") or None, max_workers=3) for r in pins["models"]}
from da2.model.spherevit import SphereViT
import da2.model.dinov2.attention as da2_attention
# DA-2 already provides this PyTorch SDPA fallback. Old xFormers has no sm_120 kernel.
da2_attention.XFORMERS_AVAILABLE = False
from nunchaku import NunchakuFluxTransformer2dModel
from worldgen import WorldGen
import worldgen.worldgen as wg
from worldgen.pano_depth import DA2_CONFIG, pred_pano_depth
from worldgen.pano_gen import gen_pano_image, gen_pano_fill_image
from worldgen.models.flux_pano_gen_pipeline import FluxPipeline
from worldgen.models.flux_pano_fill_pipeline import FluxFillPipeline
from worldgen.utils.lora_utils import load_and_fix_lora
import open3d as o3d
torch.backends.cudnn.enabled = False
torch.manual_seed(args.seed)
torch.cuda.reset_peak_memory_stats()
mode = "i2s" if args.image else "t2s"
base = "black-forest-labs/FLUX.1-Fill-dev" if args.image else "black-forest-labs/FLUX.1-dev"
quant = "mit-han-lab/svdq-fp4-flux.1-fill-dev" if args.image else "mit-han-lab/svdq-fp4-flux.1-dev"
lora_file = "worldgen_img2scene.safetensors" if args.image else "worldgen_text2scene.safetensors"
def depth_model(device):
return SphereViT.from_pretrained(snapshots["haodongli/DA-2"], config=DA2_CONFIG).eval().to(device)
def panorama_model(lora_path=None, device="cuda", low_vram=True):
transformer = NunchakuFluxTransformer2dModel.from_pretrained(snapshots[quant], offload=True)
pipe_type = FluxFillPipeline if args.image else FluxPipeline
pipe = pipe_type.from_pretrained(snapshots[base], transformer=transformer, torch_dtype=torch.bfloat16)
state_dict, _ = load_and_fix_lora(str(Path(snapshots["LeoXie/WorldGen"])/"models--WorldGen-Flux-Lora"/lora_file))
transformer.update_lora_params(state_dict)
pipe.enable_model_cpu_offload()
pipe.enable_vae_tiling()
return pipe
# Keep the upstream WorldGen generation, depth, splat and mesh implementations.
# The builders above select immutable model snapshots and FP4/offload explicitly.
wg.build_depth_model = depth_model
wg.build_pano_gen_model = panorama_model
wg.build_pano_fill_model = panorama_model
started = time.monotonic()
model = WorldGen(mode=mode, resolution=args.width, device=torch.device("cuda"), low_vram=True)
loaded_seconds = time.monotonic()-started
print("Models loaded:", mode, "RTX 5090 / FP4 / CPU offload", flush=True)
height = args.width//2
stage = time.monotonic()
conditioning = None
with torch.inference_mode():
if args.image:
image = Image.open(args.image).convert("RGB")
intrinsics = json.loads(args.intrinsics.read_text())
assert list(image.size) == intrinsics["resolution"]
mapped, mask, coverage = project_image(image, intrinsics["K"], args.width, height)
mapped.save(output/"conditioning.png")
mask.save(output/"generation-mask.png")
image.save(output/"input.png")
panorama = gen_pano_fill_image(model.pano_gen_model, mapped, mask, prompt=args.prompt,
seed=args.seed, num_inference_steps=args.steps, height=height, width=args.width)
generated = np.array(panorama)
keep = np.array(mask)==0
generated[keep] = np.array(mapped)[keep]
panorama = Image.fromarray(generated)
conditioning = {"source_sha256": hashlib.sha256(args.image.read_bytes()).hexdigest(),
"source_resolution": list(image.size), "intrinsics": intrinsics["K"],
"observed_panorama_fraction": coverage,
"max_preserved_rgb_error": int(np.abs(generated[keep].astype(int)-np.array(mapped)[keep].astype(int)).max()),
"projection": "known pinhole K to equirectangular rays; white mask is generated"}
else:
panorama = gen_pano_image(model.pano_gen_model, prompt=args.prompt, seed=args.seed,
num_inference_steps=args.steps, height=height, width=args.width)
torch.cuda.synchronize()
generation_seconds = time.monotonic()-stage
panorama.save(output/"panorama.png")
print("Panorama saved", list(panorama.size), flush=True)
stage = time.monotonic()
predictions = pred_pano_depth(model.depth_model, panorama)
depth = predictions["distance"].float().cpu().numpy()
assert np.isfinite(depth).all() and depth.min()>0
np.save(output/"distance.npy", depth)
splat = model.depth2gs(predictions)
result = export_ply(splat, output/"scene.ply")
mesh = model.depth2mesh(predictions)
assert o3d.io.write_triangle_mesh(str(output/"scene-mesh.ply"), mesh)
assert o3d.io.write_triangle_mesh(str(output/"scene-mesh.glb"), mesh)
torch.cuda.synchronize()
geometry_seconds = time.monotonic()-stage
report = {"status": "success", "mode": mode, "prompt": args.prompt, "seed": args.seed,
"steps": args.steps, "resolution": list(panorama.size), "conditioning": conditioning,
"distance": {"shape": list(depth.shape), "min": float(depth.min()), "max": float(depth.max()),
"scale": "DA-2 scale-invariant estimate normalized to max 20; not measured metric distance"},
"gaussians": result, "mesh": {"vertices": len(mesh.vertices), "triangles": len(mesh.triangles)},
"seconds": {"load": loaded_seconds, "panorama": generation_seconds, "geometry": geometry_seconds},
"runtime": {"gpu": torch.cuda.get_device_name(0), "torch": torch.__version__, "cuda": torch.version.cuda,
"packages": {n: importlib.metadata.version(n) for n in ["nunchaku", "diffusers", "transformers", "da2", "worldgen", "pytorch3d"]},
"peak_gpu_gb": torch.cuda.max_memory_allocated()/1e9, "max_rss_gb": resource.getrusage(resource.RUSAGE_SELF).ru_maxrss/1e6},
"model_revisions": pins["models"], "source_revisions": pins["source_revisions"]}
for filename in ["panorama.png", "distance.npy", "scene.ply", "scene-mesh.ply", "scene-mesh.glb"]:
path = output/filename
report.setdefault("artifacts", {})[filename] = {"bytes": path.stat().st_size, "sha256": hashlib.sha256(path.read_bytes()).hexdigest()}
(output/"result.json").write_text(json.dumps(report, indent=2)+"\n")
print(json.dumps({"mode": mode, "seconds": report["seconds"], "gaussians": result["gaussians"]}), flush=True)
if __name__ == "__main__":
main()
render.py
View files on GitHub"""Render a translated 360-degree camera path through WorldGen's actual 3D PLY.
Run with the separate fVDB Reality Capture runtime, not the WorldGen environment.
"""
import argparse
import hashlib
import json
import math
import subprocess
from pathlib import Path
import numpy as np
import torch
from PIL import Image
from fvdb_reality_capture import GaussianSplat3d
def main():
p=argparse.ArgumentParser()
p.add_argument('--scene',type=Path,required=True)
p.add_argument('--output',type=Path,required=True)
p.add_argument('--ffmpeg',default='ffmpeg')
p.add_argument('--frames',type=int,default=480)
p.add_argument('--fps',type=int,default=24)
a=p.parse_args()
a.output.mkdir(parents=True,exist_ok=True)
torch.backends.cudnn.enabled=False
model,_=GaussianSplat3d.from_ply(str(a.scene),device='cuda')
width,height=960,540
focal=width/(2*math.tan(math.radians(80)/2))
K=np.array([[focal,0,width/2],[0,focal,height/2],[0,0,1]],dtype=np.float32)
camera_json={'convention':'OpenCV world-to-camera; x right, y down, z forward; arbitrary WorldGen distance scale',
'resolution':[width,height],'K':K.tolist(),'fps':a.fps,'frames':[]}
video=a.output/'free-view.mp4'
encoder=subprocess.Popen([a.ffmpeg,'-hide_banner','-loglevel','error','-y','-f','rawvideo','-pix_fmt','rgb24',
'-s',f'{width}x{height}','-r',str(a.fps),'-i','pipe:0','-an','-c:v','libx264','-preset','fast',
'-crf','23','-pix_fmt','yuv420p','-movflags','+faststart',str(video)],stdin=subprocess.PIPE)
for i,t in enumerate(np.linspace(0,1,a.frames,endpoint=False)):
yaw=2*math.pi*t
# Translation really changes the optical center; this is not a 2D panorama crop.
center=np.array([.20*math.sin(2*yaw),.045*math.sin(yaw),.15*math.sin(yaw)])
forward=np.array([math.sin(yaw),.055*math.sin(yaw),math.cos(yaw)])
forward/=np.linalg.norm(forward)
right=np.cross([0,1,0],forward); right/=np.linalg.norm(right)
down=np.cross(forward,right)
c2w=np.eye(4,dtype=np.float32); c2w[:3,:3]=np.stack([right,down,forward],axis=1); c2w[:3,3]=center
w2c=np.linalg.inv(c2w)
with torch.inference_mode():
rgb,alpha=model.render_images(torch.tensor(w2c[None],device='cuda'),torch.tensor(K[None],device='cuda'),width,height,.005,100.)
pixels=(rgb[0,:,:,:3].clamp(0,1).cpu().numpy()*255).astype(np.uint8)
encoder.stdin.write(pixels.tobytes())
camera_json['frames'].append({'frame':i,'center':center.tolist(),'world_to_camera':w2c.tolist(),'mean_alpha':float(alpha.mean())})
if i in (0,a.frames//4,a.frames//2,a.frames*3//4):
Image.fromarray(pixels).save(a.output/f'view-{i:04}.jpg',quality=90)
print('Rendered',i,'/',a.frames,flush=True)
encoder.stdin.close()
if encoder.wait(): raise RuntimeError('ffmpeg failed')
assert np.mean([f['mean_alpha'] for f in camera_json['frames']])>.5
(a.output/'cameras.json').write_text(json.dumps(camera_json,indent=2)+'\n')
report={'frames':a.frames,'fps':a.fps,'duration_seconds':a.frames/a.fps,'resolution':[width,height],
'renderer':'fVDB Reality Capture','torch':torch.__version__,'gaussians':int(model.num_gaussians),
'center_range':np.ptp([f['center'] for f in camera_json['frames']],axis=0).tolist(),
'mean_alpha':float(np.mean([f['mean_alpha'] for f in camera_json['frames']])),
'mp4_sha256':hashlib.sha256(video.read_bytes()).hexdigest(),'mp4_bytes':video.stat().st_size,
'cameras_sha256':hashlib.sha256((a.output/'cameras.json').read_bytes()).hexdigest()}
(a.output/'render.json').write_text(json.dumps(report,indent=2)+'\n')
print(json.dumps(report),flush=True)
if __name__=='__main__': main()
audit.py
View files on GitHub"""Validate newly generated PLY/distance files on CPU, and save concrete samples."""
import argparse
import json
import math
from pathlib import Path
import numpy as np
from PIL import Image
from plyfile import PlyData
def audit(folder):
record=json.loads((folder/'result.json').read_text())
depth=np.load(folder/'distance.npy');h,w=depth.shape
vertices=PlyData.read(folder/'scene.ply')['vertex'].data
assert len(vertices)==h*w==record['gaussians']['gaussians']
assert np.isfinite(depth).all() and (depth>0).all()
assert all(np.isfinite(vertices[k]).all() for k in vertices.dtype.names)
rgb=np.array(Image.open(folder/'panorama.png'))
samples=[]
for y,x in [(h//2,w//2),(3*h//8,5*w//16),(5*h//8,11*w//16),(h//8,w//8)]:
v=vertices[y*w+x];phi=((x+.5)/w)*2*math.pi-math.pi;theta=((y+.5)/h)*math.pi-math.pi/2
ray=np.array([math.cos(theta)*math.sin(phi),math.sin(theta),math.cos(theta)*math.cos(phi)])
xyz=np.array([float(v[k]) for k in ['x','y','z']]);distance=float(depth[y,x])
assert np.max(np.abs(xyz-ray*distance))<1e-4
samples.append({'pixel':[x,y],'rgb':list(map(int,rgb[y,x])),'ray':ray.tolist(),'distance':distance,'xyz':xyz.tolist()})
norms=np.linalg.norm(np.stack([vertices[f'rot_{i}'] for i in range(4)],-1),axis=1)
assert np.allclose(norms,1,atol=1e-3)
result={'samples':samples,'distance_finite':True,'distance_positive':True,'finite_ply_fields':True,
'opacity_min':float((1/(1+np.exp(-vertices['opacity']))).min()),
'quaternion_norm_range':[float(norms.min()),float(norms.max())]}
if record['mode']=='i2s':
mask=np.array(Image.open(folder/'generation-mask.png'))
conditioning=np.array(Image.open(folder/'conditioning.png'))
assert np.any(mask==0) and np.any(mask==255)
assert np.array_equal(conditioning[mask==0],rgb[mask==0])
assert record['conditioning']['max_preserved_rgb_error']==0
(folder/'geometry-audit.json').write_text(json.dumps(result,indent=2)+'\n')
print(f'{folder.name}: finite geometry, unit rotations, ray×distance positions and conditioning verified.')
if __name__=='__main__':
p=argparse.ArgumentParser();p.add_argument('--folder',type=Path,required=True)
audit(p.parse_args().folder)
preview.py
View files on GitHub"""Create a labeled quarter-density browser preview; full PLY stays untouched."""
import argparse
import json
from pathlib import Path
import numpy as np
from plyfile import PlyData, PlyElement
p=argparse.ArgumentParser()
p.add_argument('--folder',type=Path,required=True)
a=p.parse_args()
r=json.loads((a.folder/'result.json').read_text())
w,h=r['resolution']
v=PlyData.read(a.folder/'scene.ply')['vertex'].data
assert len(v)==h*w
# Retain one of four panorama pixels and widen its tangent footprint.
small=v.reshape(h,w)[::2,::2].copy().reshape(-1)
for field in ('scale_0','scale_1'): small[field]+=np.log(2.)
PlyData([PlyElement.describe(small,'vertex')],text=False).write(a.folder/'preview.ply')
print(json.dumps({'source_gaussians':len(v),'preview_gaussians':len(small),'method':'2x2 image-grid stride, tangent scales doubled; DC color unchanged'}))
model-pins.json
View files on GitHub{
"source_revisions": {
"WorldGen": "7ce7b2767fdf31e2727b69a2e61e2e950e3a017f",
"DA-2": "d659838585f2bc9967c7e9367af573795271dbbb",
"PyTorch3D": "ddfc55127fed1a74514cd24db08a7c2e62e76b08"
},
"models": [
{
"repo": "black-forest-labs/FLUX.1-dev",
"revision": "45d95c497444bbaebe766e1528792fc9158c0c3a",
"omitted": [
"transformer/*"
]
},
{
"repo": "black-forest-labs/FLUX.1-Fill-dev",
"revision": "7326a96224b2513681b5d97281c83af967926bb6",
"omitted": [
"transformer/*"
]
},
{
"repo": "mit-han-lab/svdq-fp4-flux.1-dev",
"revision": "70ecf6cfde1525d88ec386fbec5a3a8cdfc33ea4",
"omitted": []
},
{
"repo": "mit-han-lab/svdq-fp4-flux.1-fill-dev",
"revision": "cec73b4a4e208b28df9e4f977d775e70edff71a8",
"omitted": []
},
{
"repo": "LeoXie/WorldGen",
"revision": "2e2b2cb4dc3144a65481f0ca26b892584436868c",
"omitted": []
},
{
"repo": "haodongli/DA-2",
"revision": "0d55ccb5e46b8ed4715fae3a4c04fc897f1689f3",
"omitted": []
}
]
}
check.py
View files on GitHub"""Inspect published WorldGen evidence with Python's standard library; no GPU."""
import gzip
import hashlib
import json
import math
from pathlib import Path
import struct
def main():
root=Path(__file__).resolve().parents[2]
folder=root/'labs/worldgen-scenes'
record=json.loads((folder/'recorded.json').read_text())
for name,digest in record['implementation_sha256'].items():
assert hashlib.sha256((folder/name).read_bytes()).hexdigest()==digest,name
assert {row['mode'] for row in record['cases']}=={'t2s','i2s'}
assert len(record['cases'])==3
for case in record['cases']:
evidence=folder/'evidence'/case['id']
result=json.loads((evidence/'result.json').read_text())
audit=json.loads((evidence/'geometry-audit.json').read_text())
cameras=json.loads((evidence/'cameras.json').read_text())
assert result['status']=='success' and result['resolution']==[1600,800]
assert result['gaussians']['gaussians']==1_280_000 and result['gaussians']['all_finite']
assert result['distance']['shape']==[800,1600] and result['distance']['max']==20
assert audit['distance_finite'] and audit['distance_positive'] and audit['finite_ply_fields']
assert all(abs(v-1)<1e-3 for v in audit['quaternion_norm_range'])
for sample in audit['samples']:
assert math.isclose(sum(v*v for v in sample['ray']),1,abs_tol=1e-5)
assert max(abs(x-r*sample['distance']) for x,r in zip(sample['xyz'],sample['ray']))<1e-4
assert len(cameras['frames'])==480 and cameras['fps']==24
centers=[f['center'] for f in cameras['frames']]
assert max(c[0] for c in centers)-min(c[0] for c in centers)>.39
for frame in cameras['frames']:
# W2C must map the documented optical center back to the origin.
for row in frame['world_to_camera'][:3]:
assert abs(sum(row[j]*frame['center'][j] for j in range(3))+row[3])<1e-5
assert hashlib.sha256((evidence/'cameras.json').read_bytes()).hexdigest()==case['render']['cameras_sha256']
for name,meta in case['published'].items():
data=(root/'site/public/showcase'/name).read_bytes()
assert len(data)==meta['bytes'] and hashlib.sha256(data).hexdigest()==meta['sha256'],name
spz=gzip.decompress((root/'site/public/showcase'/case['web_preview']['file']).read_bytes())
magic,version,count,degree,_,_,_=struct.unpack_from('<IIIBBBB',spz)
assert magic==int.from_bytes(b'NGSP','little') and version==3 and count==320_000 and degree==0
assert len(spz)==16+count*20
assert .5<case['render']['mean_alpha']<=1
if case['mode']=='i2s':
assert result['conditioning']['max_preserved_rgb_error']==0
assert .1<result['conditioning']['observed_panorama_fraction']<.3
assert hashlib.sha256((folder/'evidence/input.png').read_bytes()).hexdigest()==result['conditioning'].get('published_source_sha256', result['conditioning']['source_sha256'])
print('WorldGen: 3 generated scenes, 1,440 translated camera poses, 3 MP4s and 3 compressed previews verified.')
if __name__=='__main__': main()