Picasso: Holistic Scene Reconstruction with Physics-Constrained Sampling

3 min read

Picasso takes per-object pose and shape estimates (from SAM3D or CRISP) and corrects them so the whole scene is physically plausible: no two objects interpenetrate, nothing sinks into the table or into space the camera saw as empty, and nothing floats. It writes multi-object pose estimation as maximum likelihood (a Chamfer fit to the masked depth points, with a Geman-McClure robust loss) subject to those constraints, and solves it by coarse-to-fine rejection sampling over Sim(3) on a GPU. Sampling all objects jointly would blow up with the object count, so a VLM (Gemini) reads the image and returns a contact scene graph; Picasso turns that into a DAG by breadth-first search from the table node and estimates one object at a time with its parents held fixed. The authors also release a 10-scene contact-rich dataset (iPad LiDAR, 2 to 8 objects per scene) and a Scene Plausibility Score: the kinetic energy of the reconstructed digital twin after a 20-step PyBullet rollout.

Findings

  • YCB-V with CRISP: a CRISP model trained only on synthetic images, plus Picasso, reaches 0.008 m mean ADD-S and 0.52 AUC at 1 cm. That beats CRISP trained on real images (0.009 m, 0.44), so the correction closes the sim-to-real gap on this benchmark. Picasso takes about 0.35 s per object here; GICP takes 0.03 s but is less accurate.
  • YCB-V with SAM3D over 12 trajectories: ADD-S falls from 21.68 mm to 10.56 mm, the plausibility score from 5.84 to 1.85, and penetration from 9.29 mm to 2.79 mm. The same sampler without physics constraints already reaches 10.29 mm and 1.91, so most of the geometric gain comes from the search itself. The authors say as much: constraints are only checked on the top 16 candidates.
  • Picasso dataset: SAM3D’s mean ADD-S drops from 11.71 mm to 7.57 mm. GICP on SAM3D with point-map input gets the best mean ADD-S (7.23 mm) but a worse plausibility score (8.58 against 6.59) and more penetration.
  • Runtime on the Picasso dataset: 4.15 s per object with the physics corrector, 1.97 s without. One SAM3D call is 7.66 s and one Gemini contact query 11.71 s.
  • A Mechanical Turk study finds that human plausibility rankings agree with the score and prefer Picasso’s reconstructions over raw SAM3D.

Limits

  • Static scenes from a single RGB-D frame, with objects resting on one plane found by RANSAC.
  • The contact graph is forced acyclic and solved greedily, so the global optimum is not guaranteed (the authors admit this). A wrong parent pose is inherited by every child.
  • Contact topology comes from a cloud VLM in about 12 s per query. A wrong contact edge silently changes which constraints apply.
  • If all 16 candidates violate the constraints, Picasso returns the best-scoring pose anyway, so the output is not always plausible.
  • Tabletop and sink-sized scenes of household objects only.

Relevance

Here the scene graph drives inference (contact edges decide the order in which objects are estimated), which suggests an object layer for 3d-scene-graphs whose support relations a manipulation planner could rely on. The silent fallback when no sample is feasible is the kind of degraded output a runtime monitor should catch. Compare Hickory, which also post-processes SAM3D, for navigation and map alignment.