CRISP: Object Pose and Shape Estimation with Test-Time Adaptation

3 min read

CRISP estimates both the pose and the shape of an object from one RGB-D crop, without being told the object’s category. A ViT backbone with a DPT head predicts pose-normalised coordinates for each depth point, and a FiLM-conditioned decoder turns a latent shape code into a signed distance field. When the test data looks different from training, a corrector fixes the estimate by optimisation: the shape decoder is approximated by an active shape model (a convex combination of known shapes), which turns shape correction into a constrained linear least-squares problem. An observable-correctness check (the depth points must sit near the zero level set of the corrected SDF, at a chosen quantile) decides which corrected outputs become pseudo-labels for self-training at test time.

Findings

  • YCBV: mean shape error (Chamfer) 0.026 for CRISP trained on real data against 0.099 for Shap-E, a 73% reduction. Self-training the synthetic-only model improves mean shape error by 12% to 17% and mean ADD-S by 17% to 20%, depending on the corrector.
  • The oc check works as a filter: the Chamfer distance distribution of outputs that pass it shifts sharply towards zero (their Fig. 6).
  • SPE3R satellites, test objects unseen in training: self-training improves mean L1 shape error by 13% and pose error by 23%, and beats SQRECON on all metrics.
  • NOCS: 67% lower shape error than the next baseline (CASS). Rotation error is worse than category-level methods.
  • Runtime (mean): 125 ms for CRISP itself, 251 ms for the least-squares corrector, 3.8 s for the block coordinate descent corrector at 25 iterations. So the fast corrector is usable online and the slow one is not.

Limits

  • The authors assume ground-truth data association (which detection is which object) at test time, and flag tracking errors as future work.
  • The corrector’s latent-code projection grows linearly with the number of training shapes.
  • The active shape model only covers the convex hull of known shapes. A truly new shape outside it can only be approximated.
  • The oc check is a threshold on residuals, so it says “consistent with the data”, which for a heavily occluded object may not mean “correct”. C-3PO’s separate non-degeneracy certificate is not carried over here.

Relevance

CRISP keeps the correct-and-certify loop of C-3PO and drops the need for CAD models, as a robot in an unfamiliar place (a warehouse, a farm, a disaster site) would need. The self-training only learns from outputs that pass a consistency check, which is a runtime monitor used as a teacher. Related to outlier-robust estimation and multi-robot perception.