FunGraph: Functionality Aware 3D Scene Graphs for Language-Prompted Scene Interaction

3 min read

FunGraph adds a layer below objects to a ConceptGraphs-style scene graph: the knobs, handles, buttons and switches a robot actually touches, linked to their parent object by a “has-part” edge that carries an affordance label (rotate, hook pull, key press). No 2D dataset of such parts existed, so the authors made one by projecting SceneFun3D’s 3D laser-scan annotations into the RGB-D frames, keeping only visible, large-enough projections. Detectors trained on that data find parts, YOLO-World finds objects, and parts are tied to objects by bounding-box overlap. Each object gets a CLIP vector from its crop, and GPT-4o, shown the object with its parts boxed, turns the closed affordance label into a specific name (“freezer handle”). GPT-4o also labels object-to-object spatial edges from highlighted images. To answer a task query, the graph is written to JSON (ids, centres, boxes, labels, relations, affordances) and GPT-4o picks the node ids.

Findings

  • The generated 2D dataset has 274,022 annotations across 132,635 images. Zero-shot detectors fail on parts (YOLO-World 0.0 and Grounding DINO 0.7 mAP50), while RT-DETR fine-tuned on sliced images reaches 21.0 mAP50:95 and 26.2 mAP50.
  • Lifted to 3D on 10 SceneFun3D scenes, part segmentation averages 5.9 AP, 16.0 AP50 and 30.3 AP25, close to the reported Mask3D-F figures (7.9, 18.3, 26.6) that segment the laser scan directly. The authors warn the comparison is loose because Mask3D-F’s split and weights are not public.
  • On 99 task-driven affordance queries (“open the trash bin”), FunGraph succeeds on 33.3% at IoU 25% and 58.6% with any overlap. ConceptGraphs returns whole objects and gets 0.0% and 31.3%. The best SceneFun3D model reports 17.5 AP25.
  • Naming parts with context matters: on 100 hand-checked parts, GPT-4o shown the parent object is correct 78% of the time, GPT-4o shown only the part 52%, and CLIP classification 30%.

Limits

  • Parts link straight to the object (a knob to the cabinet, skipping the drawer). Parts shared between objects, like the knobs between an oven and a stove, cause most naming and association errors; the authors say so.
  • Only spatial relations are kept. The authors note that functional links such as a switch to the light it controls may break the hierarchy’s disjoint-children property.
  • Inaccurate dataset poses smear small parts during merging. Parts are smaller than a hand, so the method needs high-resolution RGB; the authors judge centimetre-resolution LiDAR too coarse.
  • Every object and part group triggers a GPT-4o call for labels and edges, and the whole pipeline ran on a desktop RTX 4090. No timing is reported, and node labels and filtering are finalised only after the whole sequence is processed.

Relevance

FunGraph shows the open-vocabulary recipe breaking at small scale: CLIP names a knob-sized crop correctly only 30% of the time, and the useful label comes from a VLM that sees the parent object. For multi-robot multimodal perception, the same question arises for any small but important thing, such as a valve on a pipe or a latch on a hatch. It extends ConceptGraphs and uses the same JSON-to-LLM query as SayPlan; see 3D scene graphs and language-grounded planning.