Thesis map

Every paper I’ve collected so far, each with a note. Pick a tab, click a column heading to sort, and hover over a title for a preview.

The status says how far I got. read means the note comes from the full paper, skimmed means the abstract and the code’s README, and to-read means I know why I want to read it and haven’t yet. Most of the to-read papers are preprints from 2026, found while writing these notes.

Each paper is filed under one or more robotics subfields. The threads are the three directions my thesis might take: robust robotics, multi-robot multimodal perception and long-term autonomy. The thesis map shows how they connect.

Papers

An operator has to decide how much to rely on a robot team. The interface affects that decision.

Earlier work that feeds into this: mixed reality interfaces for explaining cyber-physical attacks to operators and policymakers (book chapter, Stanford talk).

Questions

• How can an interface show that one robot in twelve is probably lying, without too many false alarms?
• What should an explanation from a robot team look like?

Papers

• When a VLM judges whether a robot succeeded, it leans towards “yes” (FailBench). An operator who trusts that judge inherits the bias.
• Agreement between two models is only evidence if their errors are independent. See disagreement between models.
• A robot that hedges when its models disagree, as GoGet does when counting, gives the operator something to calibrate against.

Related: multi-robot systems, semantic coordination, evaluating language models.

Singapore has a busy port, strict airspace rules and a dense city on a small island, which gives useful test cases for robot teams:

• Sea: port operations, hull inspection, search and rescue in busy shipping lanes.
• Air: drones over dense urban areas, under strict rules about where they may fly.
• Land: ground robots in crowded public spaces, estates and campuses.

To do: find out which of these have public datasets or simulators. So far:

• Air and ground: AGT-CV, a small real air-ground dataset, and MAG-VLAQ, place recognition from the ground against aerial images.
• Sea: Above and Below does SLAM for one surface vessel and several AUVs. DRACo-SLAM2 has a real three-robot sonar dataset.

Related: multi-robot systems.

A possible thesis thread: robots and robot teams that keep working when some of their parts fail, are wrong, or are compromised. Hacked firmware, a spoofed GPS fix and a sensor that dies while still reporting plausible numbers look the same from the outside, and a robot has to cope with all three.

I come to this from eight years of cyber-physical security at iTrust, where the plant was a water treatment testbed and the defences were detectors and invariants. A robot is a control system that moves, and a robot team is a networked one.

Detection

Every detector I’ve read about depends on something the faulty part doesn’t control: a second sensor, a second network path, the physics of radio, a design document. Hand-written checks cover only what their designer thought of, and learned ones come with no guarantee. More in failure recovery.

Recovery

Most robots stop. Simplex, shields and barrier filters fall back to something conservative, and swarms cut members off. Few papers measure how long a robot stays useful after its safety layer acts. My workshop paper Safe but Stuck is about what stopping costs in the field. See also safe control.

Monitors

A runtime monitor reads the same state estimate as the controller, and none of the ones I’ve read check it. A spoofed sensor fools the safety switch as easily as the controller. See runtime monitoring.

Estimation

Outlier rejection in robust estimation copes well with look-alike places and badly with a coordinated liar. TEASER++ proves exact recovery needs at least three more inliers than adversarial outliers, the same kind of bound as $n>3f$ in Byzantine fault tolerance.

Learned components

Robots running the same model make the same mistakes, and VLM judges lean towards calling an attempt a success. See disagreement between models and evaluating language models. Cheap attacks work on them too: a printed sign, a spoken safety phrase, a backdoor that survives fine-tuning (robot security).

Open questions

1. How can a team tell a broken teammate from a lying one, using a witness the suspect can’t control? A broken robot can be repaired or given another job; a liar has to be cut off.
2. What does recovery look like beyond stopping, and how do you measure it?
3. Can a monitor know when its own inputs are uncertified, and say so to its teammates?
4. Which alarm guarantees survive drift and an adversary at the same time?
5. How should agreement between two robots running the same model be discounted?

Earlier pieces of this on the site: Lying robots (resilient consensus, interactive), the buoy swarm paper, and resilient consensus.

Papers

Other threads: multi-robot multimodal perception, long-term autonomy. All three on one page: thesis map.

What a robot does after something goes wrong. Most papers cover detecting the fault. Few cover recovering from it so the robot keeps doing useful work, and most robots stop and wait for a person.

Detection

• Residuals against a model. Predict what a sensor should read and flag disagreement (Bezzo). A fault the motion never excites leaves no residual (Hausman).
• Invariants. Physical rules the process must obey, written from the design (Adepu and Mathur), mined from logs (Umer), or drafted by an LLM from documents (INVARLLM: 29 of 36 attacks caught, but 42.54% precision per sample). My years on the SWaT testbed were mostly spent here.
• Learned detectors on the policy itself. SAFE reads a VLA’s internal features, FAIL-Detect treats failure as out-of-distribution detection without any failure data, and Hide-and-Seek uses contrastive trajectory signals. All three use conformal prediction to set the alarm threshold.
• Outside judges. A VLM watching the robot. FailBench found the best of 13 VLM detectors reaches 0.77 balanced accuracy, fine-tuned ones do worse than general ones, and all of them lean towards calling ambiguous attempts a success. No Free Checker puts it generally: the cheaper and earlier a verdict, the less it tells you.

Every detector here depends on a witness the faulty part doesn’t control: a second sensor, a second network path, the physics of radio, a design document.

Recovery

Mostly stopping, or handing control to something conservative:

• Simplex switches to a verified safe controller. SOTER adds the switch back: its advanced controller ran more than 96% of 104 simulated hours, and all 34 crashes came from the safe controller not being scheduled in time.
• Shields allow a few steps to recover, then drop into a fail-safe.
• Control barrier filters keep the robot safe by giving up progress, and their multi-robot version admits deadlock in crowds.
• ROSMonitoring’s only built-in reaction is dropping a message. Caldas and others found no guidelines for anything richer.

Swarms recover by exclusion: Strobel’s token economy starves bad robots, the blocklist protocol cuts off both ends of an accusation, honest accusers included. Real repair is rare. O’Keeffe sends a flagged robot home before it fails, Oğuz reroutes traffic until an intermittent fault clears, Mayya reallocates tasks, and Pant schedules trips to recharge.

Stopping

A robot that always stops when unsure is safe and useless, and an easy target. Steinberg and Gal halt an LLM-controlled robot in up to 98.3% of multi-turn runs with one to five spoken safety phrases. In the field, a robot that latches into a state no command can clear needs a person to power-cycle it, which can mean a boat trip. My workshop paper Safe but Stuck is about this.

Faults and attacks

Most papers don’t separate a teammate that is broken from one that lies. Fault papers assume random, independent faults; security papers assume a bounded number of worst-case robots with trusted identities. The case in between, a real robot with its own identity and a broken sensor, falls through both. Luckcuck and others quote Winfield and Nembrini: a robot with dead wheels and a working radio hurt the swarm more than a dead one. The two need different recoveries: a broken robot can be repaired or given another job, and a liar has to be cut off.

Questions

• How long does a robot stay useful after its safety layer acts? Almost nobody measures it.
• What does recovery look like for a team, when the robot that failed is the one that was relaying for the others?
• Can a robot tell its teammates “I’m degraded, here’s what I can still do”?

Related: runtime monitoring, safe control, robust robotics, long-term autonomy.

Keeping a robot inside a safe set (no collisions, no stalls over water, no battery below the return threshold) while it does its job. The usual tool is a filter between the planner and the motors that changes a command only when it would leave the safe set.

Barrier functions

• CBF-QP (Ames and others) solves a small quadratic program at every step: stay as close as possible to the commanded input, subject to a barrier constraint that keeps the safe set invariant. When safety and progress conflict, progress gives way.
• Safety barrier certificates turn this into a multi-robot collision filter, with each robot taking its share of the avoidance. The authors admit deadlock and infeasible programs in dense crowds.
• Qin and others learn decentralised barrier certificates with a neural network, and Bousias and others use an equivariant graph network for quadrotor swarms. Like most learned barrier work, both assume every robot runs the same policy.

Switching and shielding

• Simplex runs a high-performance controller and switches to a verified safe one when needed. SOTER also switches back.
• Shield synthesis builds a component that corrects unsafe outputs of a system against a temporal-logic specification, and shielded RL puts one between a learning agent and the world.

Assumptions

The guarantee is about the state the controller believes. A wrong estimate (spoofed GPS, a drifting IMU) means the filter keeps the estimate safe, which may not be where the robot is. In a team, the distributed filters assume every neighbour runs the same filter and does its share: a neighbour that ignores its share, through a bug or on purpose, pushes the honest robots to do more than theirs. And a safety filter that only ever slows down can trap a robot in a state that is safe and useless, and getting it out is a failure recovery problem.

Questions

• How should a barrier function treat a neighbour whose reported position it doesn’t trust?
• Can a safety filter prove progress as well as safety, at least for a while?

Related: runtime monitoring, aerial swarms, robust robotics, resilient consensus.

A Byzantine node can behave arbitrarily: lie, tell different neighbours different things, go silent, or behave correctly until later. The name comes from Lamport, Shostak and Pease’s 1982 paper about generals agreeing on a plan when some of them are traitors.

The classic bound for exact agreement: with $n$ nodes you can tolerate at most $f$ traitors if $n>3f$.

Relevance to robots

Hacked firmware, spoofed GPS, and a failed sensor that still reports plausible numbers look the same from the outside. A compromised PLC in a water treatment plant is also a Byzantine node (see cyber-physical security).

Faults and attacks

Most robotics papers don’t separate a faulty teammate from a malicious one. Fault papers assume random, independent faults. Security papers assume a bounded number of worst-case robots with trusted identities (Wardega, Strobel). A real robot with its own identity and a broken sensor falls through both, and the two cases call for different responses: repair it, or cut it off. More in failure recovery.

Physical witnesses help. Gil and others use Wi-Fi fingerprints a spoofer can’t forge, SABER caps an agent’s trust by its worst report, and RigidShift shows that checking robots only against each other misses a shift of the whole group.

See also

• resilient consensus for agreement on continuous values.
• Byzantine-robust learning for training models when some workers lie.
• Robust estimation: TEASER++ proves exact registration needs at least three more inliers than adversarial outliers, the estimation version of $n>3f$.
• Robust robotics, the thread this belongs to.

References

• L. Lamport, R. Shostak, M. Pease. The Byzantine generals problem. ACM Transactions on Programming Languages and Systems, 1982.

Security for systems where software controls physical processes: water treatment, power grids, robots. Attacks usually aim at a physical outcome (overflowing a tank, tripping a breaker), so defenders can monitor the physical process as well as the network.

Work at iTrust

• Orthogonal defence (Argus, the patent): monitor the plant independently of the control system you’re protecting.
• Command validation (PCAT): check what a PLC is about to tell an actuator before it happens.
• Ensembles of detectors (Super Detector).
• Exercises (CPX design): CISS, CIDEX, and NATO CCDCOE’s Locked Shields and Crossed Swords.

Most of this happened on the SWaT testbed.

Invariants

Much of the iTrust work rests on process invariants: physical rules a plant must obey, checked at runtime. They can be written from the design (Adepu and Mathur), mined from logs (Umer and others, who find neither source is enough alone), or now drafted by an LLM from plant documents (INVARLLM). I went back over this from the student’s side in Secure Cyber-Physical Systems.

Relevance to robots

A robot team is a mobile, networked cyber-physical system. See multi-robot systems, Byzantine fault tolerance, robot security and runtime monitoring.

Robots now call language models to plan, to describe what they see, and to judge whether a task succeeded. Each of those needs some way to tell a good answer from a bad one, and the cheap way, asking another LLM, has known failure modes.

LLM judges

The LLM-as-a-Judge survey lists the biases: position (preferring the first answer), length, self-preference, authority and names. The attacks are cheap: a constant “null” answer, or starting a response with “Thought process:”. In the survey’s own experiment:

• Agreement with human labels barely varies across judges, from 50.72% to 61.54%.
• After the two answers are swapped, LLaMA3-8B keeps its verdict only 38.85% of the time.
• Asking the judge to explain itself made position consistency worse. Asking it to check itself did nothing. A majority over repeated runs helped.
• A three-model panel was only as good as its weakest member, whose bias dragged the vote.

Judges as rewards

CaRR shows what happens at training time. Rewarding a search agent only for the final answer let it learn shortcuts: it cited fewer pages and checked less. The fix rewards citations, but it depends on a judge LLM that was checked on 10 trajectories, and the benchmarks are scored by yet another judge.

Human graders

LalaEval uses human graders and adds a dispute analysis that flags graders and questions with unusual disagreement. The survey’s advice (a strong judge, plus swaps and voting) averages disagreement away as noise. LalaEval uses it to find unreliable graders and questions, which is closer to what a robot team needs; see disagreement between models.

Questions

• A robot that asks a VLM “did the grasp work?” is using an LLM judge. How often does it say yes when it shouldn’t?
• If a team’s robots all run the same model, is their agreement worth anything?

Related: foundation models for robotics, runtime monitoring, human-robot trust.

Getting a group to agree on a value (a meeting point, a heading, an estimate) when up to $F$ members may be Byzantine.

Plain averaging has no defence: one liar reporting an extreme value can move everyone, because the mean has a breakdown point of zero. The explainer shows this.

W-MSR

Weighted Mean-Subsequence-Reduced. Each normal agent sorts the values it hears, drops up to $F$ that are larger than its own and up to $F$ that are smaller, and takes a weighted average of what’s left and its own value.

• Safety: if each agent has at most $F$ Byzantine neighbours, normal agents never leave the range spanned by the normal agents’ initial values.
• Agreement needs the network to be connected enough. LeBlanc, Zhang, Koutsoukos and Sundaram (2013) show that $(2F+1)$-robustness is sufficient. Robustness asks that for any two disjoint groups, someone in one of them has enough neighbours outside it.
• The idea goes back to approximate agreement (Dolev, Lynch, Pinter, Stark and Weihl, 1986).

Multi-dimensional values

Applying W-MSR one coordinate at a time keeps agents inside the bounding box of the normal agents, which is weaker than the convex hull. Vector versions exist but cost more. Reputation-based consensus (2026) reaches vector consensus without being told $F$: each agent scores its neighbours from the discrepancies it sees and can push a neighbour’s weight to zero. To read.

In robot teams

• Strobel and others compare W-MSR with a blockchain approach in a simulated swarm. W-MSR ran on a graph with average degree 2.4, far below the connectivity its guarantee needs, which says more about sparse swarms than about W-MSR.
• Ballotta and others weight neighbours by trust from a physical-layer signal, which catches spoofed identities and fully trusts a real robot with a broken sensor.
• The blocklist protocol removes Byzantine robots by accusation, and blocks the honest accuser along with the accused.
• FRAC-MARL brings the same idea to multi-agent RL when only the links are corrupted.
• The flocking result underneath much of this is Jadbabaie, Lin and Morse (2003).

Open questions

• Robot teams move, so the communication graph changes as they coordinate. Which robustness guarantees survive that?
• Robustness is expensive to check on a large graph. Is there a cheap way for a team to check it?
• Can the same trimming work on map claims instead of numbers? See 3D scene graphs.

References

• H. J. LeBlanc, H. Zhang, X. Koutsoukos, S. Sundaram. Resilient asymptotic consensus in robust networks. IEEE Journal on Selected Areas in Communications, 2013.
• D. Dolev, N. Lynch, S. Pinter, E. Stark, W. Weihl. Reaching approximate agreement in the presence of faults. Journal of the ACM, 1986.

When two models, or two robots, disagree, at least one of them is wrong. That is useful: it tells you where not to trust an answer, without knowing the right one.

Background

• Query by committee (1992) asks for a label where a committee of models splits.
• Deep ensembles read uncertainty from the spread among independently trained networks.
• Self-consistency samples several reasoning chains from one LLM and trusts the answer most of them reach.
• Selective classification gives the frame for abstaining: trade how often you answer (coverage) against how often you are wrong when you do (risk).
• Calibration: without it, fusing confidences lets the most overconfident source win.

Robots

RoboTwin 2.0 has a VLM observer that watches the policy, and Agentic Robot has a verifier that checks the executor’s work. GoGet uses disagreement directly. It trusts the segmenter’s object count only when the segmenter and a vision-language model agree, and hedges when they don’t. On a robot, the hedge can turn into an action: move closer and look again, which is active perception driven by disagreement.

Resilient consensus uses the same logic: a robot discards the neighbours that disagree most with everyone else.

Correlated errors

The assumption underneath is that the models fail independently. They often don’t. Self-consistency samples share one model’s blind spots, a judge panel inherits its weakest member, and a verifier that is also a VLM fails on the same images as the executor. Worse, a spoofed input fed to every robot in a team produces agreement on something false, and that looks like confidence. A recent preprint on uncertainty in scene graphs makes the same point: detections from one model fail together and should not count as independent evidence.

Questions

• How do you measure how independent two models’ errors are, before trusting their agreement?
• When two robots disagree, which one should move, and which one should ask?
• Is disagreement between a camera and a lidar worth more than disagreement between two cameras running the same detector?

Related: evaluating language models, robust estimation, Byzantine fault tolerance, multi-robot multimodal perception.

Estimating a pose, a shape or a map when some of the measurements are wrong, and knowing when the answer can be trusted. Least squares is the robotic version of plain averaging: one bad loop closure can bend the whole map.

Methods

• Down-weight what doesn’t fit. Graduated non-convexity starts from an easy convex problem and gradually makes it robust, so bad measurements lose their weight. It handles 70% to 80% outliers and breaks down near 90%. Hydra and Hydra-Multi use it to reject wrong loop closures.
• Keep the largest consistent set. PCM finds the biggest group of inter-robot loop closures that agree with each other. This works against random errors, and fails if the wrong measurements agree with each other: repeated structure, or a coordinated liar.
• Certify the answer, or say you can’t. TEASER++ survives 99% random outliers with known scale, and gives a certificate when its answer is globally optimal. C-3PO and CRISP bring certificates to learned object pose and shape: C-3PO certified 22% of mug instances, and refused the rest. CP-Cert makes certification up to a thousand times faster when the problem is degenerate.

A certificate gives a robot a principled way to say “I can’t vouch for this”, which a recovery layer or a teammate can act on.

Observability

Hausman and others show why some faults can’t be caught by any residual: a bias in a direction the trajectory never excites produces no error to see. How the robot moves decides what a detector can observe. Bezzo and others inflate the noise of any speed sensor that disagrees with the predicted state, which works while fewer than half the sensors are attacked.

Adversarial outliers

TEASER++ proves that with adversarial outliers, exact recovery needs at least three more inliers than outliers. It is the estimation counterpart of the $n>3f$ bound from Byzantine fault tolerance, and the same breakdown-point argument as in resilient consensus. RigidShift shows the limit of checking robots only against each other: a slow shift applied to the whole swarm keeps every pairwise distance intact, and only trusted anchors make it visible.

Questions

• Can a certificate travel with a measurement, so the receiving robot knows how much the sender trusted it?
• What does certifiable estimation look like when the outliers come from a teammate with an agenda?
• Turbid water and night produce dense, confident, wrong depth. Can a certifier catch that?

Related: collaborative SLAM, runtime monitoring, robust robotics.

Checking, while the robot runs, that what it is doing satisfies a property: “never closer than a metre to a person”, “every request is answered within two seconds”. Model checking asks whether every possible run is correct. A runtime monitor judges only the run it is watching. That is a smaller promise, and it can be kept on real robots.

Decidability

• A monitor judges one observed trace (Leucker and Schallhart).
• LTL3 gives three answers: true, false, or not yet known. Only 54 of 97 usable specification patterns can ever be decided from a finite trace.
• “Eventually” without a deadline never resolves. A property such as “the robot eventually reaches the dock” stays inconclusive forever, so liveness needs a time bound before a monitor can check it. The split between safety and liveness goes back to Pnueli.
• Signal temporal logic robustness adds a margin, so a monitor can warn before a violation happens.
• Timed automata make timeouts exactly checkable. Hybrid automata show that adding a single stopwatch makes reachability undecidable. Real robot dynamics are past that line, so in practice robot safety gets checked at runtime.

Inputs

No monitor promises that its input is correct. SOTER assumes accurate state estimates, shields read simulator state, Simplex trusts the measurement. A spoofed sensor fools the switch as easily as it fools the controller. Seshia and others call turning environment assumptions into monitors, and deciding what to do when one fires, “a largely unsolved problem”.

The survey by Luckcuck and others found runtime monitoring in 3 of 63 core robot papers, against 32 for model checking. Konur’s swarm verification handles hundreds of identical robots by counting them, and so can’t represent one robot that deviates. MCMAS checks multi-agent systems against epistemic properties (“the robot knows that…”).

Learned components

C-3PO is a perception module that monitors itself: it certifies its own pose estimate or abstains. For learned policies, see the detectors in failure recovery. ROSMonitoring and Caldas and others cover what practitioners actually do with ROS: monitors exist, and their reactions are crude.

Questions

• Monitors on a team with a changing communication graph: this cell is empty in Caldas’s survey of practice.
• Can a monitor know when its own inputs are uncertified, and say so?
• What should a monitor that fires on a teammate do: warn, exclude, or ask a third robot?

Related: robust estimation, safe control, cyber-physical security, robust robotics.

Attacks on robots, and defences that work when the attacker can touch the physical world. A robot has every weakness of a networked computer, plus sensors that can be fooled and actuators that can hurt someone.

Attacks

• Sensors. GNSS spoofing is the classic. RigidShift shows a slow shift of a whole swarm defeats any check that compares drones only with each other, and Francos and others model an adversary that spoofs its own position just below what the monitor would notice.
• Middleware and networks. The humanoid security SoK maps the attack surface of a modern humanoid from firmware to cloud. The DNS papers (SAD DNS, its follow-up, SCAD) teach the general lesson: rejecting a spoofed message isn’t enough if the rejection changes shared state the attacker can read back. On a robot that shared state might be a radio’s rate limit, a middleware discovery table, or a map’s update queue.
• Models. A printed sign hijacks a VLM planner in up to 29.4% of trials (physical prompt injection). Prompts injected into one agent spread to others through shared prompt structure (multi-agent prompt injection). Backdoors in vision-language-action models survive fine-tuning (BadVLA). On the defence side, Detect and Suppress finds one internal feature that tracks an adversarial patch.
• Safety logic itself. Steinberg and Gal halt an LLM-controlled robot with one to five spoken safety phrases, in up to 98.3% of multi-turn runs. The safety layer is the attack.
• Teammates. CILC shows a compromised robot can reconstruct an honest teammate’s images from its loop-closure descriptors.

The survey of embodied agent security counts where the effort goes: attacks cluster on perception and action interfaces, defences on the action and runtime layer, and memory, world-state integrity and trust between agents get little of either.

Defences

These defences use a witness the attacker doesn’t control. Gil and others use Wi-Fi signal fingerprints that a spoofer can’t forge, Vijay and others check positions against measured ranges, and Bezzo and others check sensors against each other and a model. This is the same idea as the invariants we used in water treatment plants (see cyber-physical security).

Questions

• What is the robot version of a process invariant, for a machine whose “normal” depends on the task?
• How do you threat-model a robot team where one member may be compromised?
• When a safety phrase can stop a robot, who is allowed to say it?

Related: Byzantine fault tolerance, runtime monitoring, robust robotics, Systems Security (course).

Secure Water Treatment: a scaled-down, working water treatment plant at iTrust, SUTD, used for research and training in industrial control security. It has six stages and real PLCs. Related testbeds cover water distribution (WADI) and power (EPIC), which allows studying attacks that cross between them.

References

• A. P. Mathur, N. O. Tippenhauer. SWaT: a water treatment testbed for research and training on ICS security. International Workshop on Cyber-physical Systems for Smart Water Networks (CySWater), 2016.

Papers on invariant-based attack detection using SWaT: Adepu and Mathur, Umer and others, INVARLLM (29 of 36 attacks caught).

Related: cyber-physical security, Secure Cyber-Physical Systems.

Distributed or federated training where some workers send bad gradients, by mistake or on purpose. This is similar to resilient consensus: the server, or each peer, has to aggregate vectors it can’t fully trust.

Aggregators

• Krum (Blanchard, El Mhamdi, Guerraoui and Stainer, 2017) picks the gradient closest to its nearest neighbours.
• Coordinate-wise median and trimmed mean (Yin, Chen, Ramchandran and Bartlett, 2018), with statistical error rates. The trimmed mean is a close cousin of W-MSR.

In multi-agent RL

FRAC-MARL (2026) is a decentralised actor-critic method that tolerates corrupted messages on links. Each agent uses redundancy in two-hop messages to pick out the reliable ones, and under linear parameterisations the parameters converge to the attack-free limit, where earlier methods only got close.

Relevance

Robot teams that learn together (shared maps, shared policies) have the same problem, with less compute, less reliable links, and physical consequences for mistakes.

References

• P. Blanchard, E. M. El Mhamdi, R. Guerraoui, J. Stainer. Machine learning with adversaries: Byzantine tolerant gradient descent. NeurIPS, 2017.
• D. Yin, Y. Chen, K. Ramchandran, P. Bartlett. Byzantine-robust distributed learning: towards optimal statistical rates. ICML, 2018.

A possible thesis thread: robots that see the world through different sensors, on land, at sea and in the air, building one picture of it that a person can query and rely on.

A drone sees roofs and roads from above, a ground robot sees doors and kerbs, a boat sees a shoreline and a radar return. Each robot’s map lives in its own frame, uses its own labels and has its own errors, so merging them takes geometry, semantics and trust at once.

Scene graphs

Most groups now store maps as 3D scene graphs: objects, places and rooms as nodes, relations as edges, and language features on top (open-vocabulary mapping). Only Hydra-Multi and CURB-SG build one graph from several robots, and both assume a shared frame, honest teammates, good links and the same label set. The field’s own survey gives multi-robot work one sentence.

Sharing maps

What robots send each other keeps shrinking: whole graphs, then object maps, then labels and centroids, then fixed-size descriptors with images sent only on request. See collaborative SLAM.

Models and sensors

Foundation models give a cheap robot dense depth from one camera and segmentation by text prompt, along with domain shift and confident errors (foundation models for robotics). Outdoors there are no rooms to organise a map around, small drones carry 27 g of everything (aerial swarms), and under water even localisation is hard (underwater perception).

Language

Language works both ways: robots can describe what they see in it, and people can ask them things with it. See language-grounded planning and semantic coordination.

Open questions

1. How do robots agree on a node when one says “chair”, another says “grey office chair” and a third has only a lidar cluster?
2. Given several maps that disagree, which claims should the merged map keep? The scene graph explainer and the project on my research page start from this question.
3. What is the smallest message that still lets a teammate query my map in language?
4. Is agreement between a camera and a lidar worth more than agreement between two cameras running the same detector?
5. How do you tie perception errors to task outcomes, when graph quality and task success are measured on different datasets?

Related published work: GoGet, which uses agreement between a segmenter and a vision-language model to decide when to trust a count.

Papers

Other threads: robust robotics, long-term autonomy. All three on one page: thesis map.

A 3D scene graph stores a map as a graph. Nodes are things (objects, places, rooms, floors, buildings, people), edges say how they relate (on, in, next to, part of), and every node points at the patch of the metric map it occupies. A robot can then answer “where is the fire extinguisher nearest the stairs” without searching a point cloud, and an LLM can read the map as text.

The interactive explainer on 3D scene graphs has the history and small simulations. This note is the reading map that goes with it. The best single entry point is the 2026 survey by Rotondi and others, written by people from most of the groups that built the field.

History

Vision and robotics

The survey splits the field in two. The vision camp predicts a graph from a finished, fused point cloud, learns the relations, and reports triplet recall. The robotics camp builds the graph incrementally from a live sensor stream, fights pose drift, and reports trajectory error. Methods from one camp are rarely tested on the other’s benchmarks, and most robotics systems are evaluated on their own data, so they are rarely compared head to head.

Within the robotics camp there are two designs. The Hydra line builds the graph beside the SLAM backend and corrects every layer at once through a deformation graph, with graduated non-convexity to reject wrong loop closures (see robust estimation). The S-Graphs line puts walls, rooms and floors inside the factor graph, so the structure constrains the pose. Without its room layer, S-Graphs’ trajectory error grows to 0.40 m and 1.01 m in the larger simulated scenes.

Multi-robot construction

Only Hydra-Multi and CURB-SG build one graph from several robots. Both quietly assume four things:

• A shared frame. Hydra-Multi finds it from inter-robot loop closures and survives 80% to 90% outlier closures in look-alike dorm rooms. CURB-SG starts every car at its ground-truth pose. Tunnels, mines and multi-storey car parks give neither GPS nor distinctive places.
• Honest teammates. Outlier rejection handles perceptual aliasing, but every robot’s own graph and odometry are taken as true. Hydra-Multi undoes all proposed merges if fewer than half survive, so one robot that is consistently wrong could cancel the good merges from the rest.
• Good links. Both are centralised. Hydra-Multi resends each robot’s whole graph and mesh at every update, which its authors call naive. Neither tests dropped or delayed messages.
• The same labels. Hydra-Multi merges two objects only when their labels match. Free-text captions from two robots would almost never match.

The 3DSSG paper measures this: graph-based scene retrieval gets the right scene first 96% of the time from ground-truth graphs and 34% from predicted ones. Matching two robots’ graphs is only as good as each robot’s perception. The survey gives multi-robot construction one sentence and never mentions trust or communication. This is the question the last section of the explainer asks: given several graphs that disagree, which claims should the merged map keep? It links resilient consensus to perception.

Outdoors and in the air

Almost every system assumes a building. Outdoors there are no rooms: OpenGraph builds a lane graph from the path the robot drove, and the Strader ontology paper gets regions such as beach, road and field from an LLM-written ontology using 0.1% of the labels. On small drones, USS-Nav runs at 15 Hz on a Jetson Orin NX by embedding label text instead of image crops. More on small aerial robots in aerial swarms.

Numbers

• Hydra on a Jetson Xavier NX CPU: about 75 ms for objects, 33 ms for places, 55 ms for rooms (from a 2025 talk by Strader).
• Clio keeps 92 objects where ConceptGraphs keeps 181, at 0.30 s per frame against 2.0 s.
• ROMAN: a 40-object submap is under 250 KB, and a kilometre of map under 25 MB.
• Khronos calls an object gone when 60% of camera rays in a 5 s window pass through where it was.
• Hydra++ raises object precision from 0.670 to 0.787 by checking each learned mesh against its mask.

Questions

• How do robots agree on a node when one says “chair”, another says “office chair, grey”, and a third only has a lidar cluster with no label?
• What is the smallest message that still lets a teammate query my graph in language?
• Which checks can a robot run on a teammate’s graph before merging it: physical plausibility (Picasso), consistency with its own observations, agreement with a third robot?
• Khronos’s absence test needs a wall behind the object. What counts as evidence that something is gone in an open field?
• How should a graph forget, over months?

Related: open-vocabulary mapping, language-grounded planning, dynamic scenes, collaborative SLAM, semantic coordination, multi-robot multimodal perception.

A map whose objects can be found by any phrase, including ones nobody listed when the map was built. “The red toolbox”, “something to sit on” and “the thing leaking oil” should all work. Closed-set maps use a fixed label list (a detector trained on 80 or 150 classes) and can’t answer questions outside it.

Language features

Three recipes, often mixed:

1. A CLIP vector per segment, averaged over views. Segment each frame (SAM, FastSAM), embed each segment with CLIP, and keep a running mean per object. ConceptGraphs, Clio and ROMAN do this. HOV-SG blends whole-frame, crop and masked-crop embeddings and takes a majority vote per segment instead of a mean, which scores better on Replica (mIoU 0.231 against 0.18).
2. A caption from a vision-language model. ConceptGraphs adds LLaVA captions from each object’s ten best views and has GPT-4 condense them. OpenGraph uses the caption’s sentence embedding as the feature. FunGraph asks GPT-4o to name the parts of an object (handles, knobs) with the parent object in view.
3. Features learned in 3D. Open3DSG trains a 3D network to output CLIP-space node features and BLIP-space edge features, then asks an LLM what each relation is.

CLIP alone fails on negation, relations and small parts. ConceptGraphs’ negation queries get the right object first 26% of the time from CLIP and 80% from GPT-4 reading captions. CLIP relation queries in Open3DSG score a recall@50 of 0.02. CLIP names FunGraph’s parts correctly 30% of the time, GPT-4o 78%.

They disagree on when to commit. ConceptGraphs and OpenGraph fix the text at mapping time. Open3DSG stores features and prompts later. Clio fixes a task list before the robot moves and keeps only what is informative about those tasks. ASHiTA lets the tasks and the graph refine each other.

Cost

Most of the compute goes into the language layer. Planning over the finished graph is cheap.

Most of the others report no runtime at all, and those that do are not comparable.

Teams

• Two robots that caption the same chair differently, or carve a shelf into different objects because they were given different tasks, need more than geometry to agree. See 3D scene graphs and semantic coordination.
• CLIP vectors are 512 to 1024 numbers per object. Over a weak radio link at tens of kilobits per second that is the whole budget. ROMAN’s maps are 100 times smaller without CLIP, for 12% less alignment success.
• Thermal cameras, smoke, rain and night give images that CLIP was never trained on, so the language layer degrades in the conditions field robots work in.
• Several systems call a hosted model (GPT-4, GPT-4o). A robot that loses its link loses its vocabulary.
• There is no agreed protocol for evaluating open-vocabulary graphs, which Open3DSG’s authors point out themselves.

Questions

• Can disagreement between two models, or two robots, tell a robot when not to trust an answer? GoGet uses agreement between a segmenter and a VLM to decide when to trust a count.
• What is the cheapest feature that still supports the queries a team actually makes?
• How should a robot mark a node as “seen but not understood”?

Related: foundation models for robotics, language-grounded planning, multi-robot multimodal perception.

Several robots building one map. Each robot drifts in its own frame, so the team has to find where their maps overlap (inter-robot loop closures), agree on a common frame, and merge, over links that are slow, lossy and sometimes absent.

Pipeline

1. Each robot runs its own SLAM.
2. Robots exchange something small enough to send: place descriptors, object maps, scene graphs.
3. A candidate match between two robots’ data becomes a relative-pose constraint.
4. An outlier filter decides which constraints to believe, and a pose graph optimiser merges everything.

A team is most fragile at step 4. PCM keeps the largest set of inter-robot loop closures that agree with each other, GNC down-weights the ones that fit badly, and Kimera borrowed PCM for a single robot. See robust estimation.

Messages

Across air and ground, MAG-VLAQ matches ground camera and lidar against aerial images (61.1 Recall@1 against 34.5 for the next best), and Hickory aligns maps from different sessions by object shape alone, with 0.24 m error against 0.82 m for ROMAN.

Assumptions

• Honest teammates. Outlier filters are designed for perceptual aliasing: look-alike rooms, repeated pier pilings. A coordinated set of wrong measurements wins under PCM, because it is the largest consistent set. TEASER++ proves that with adversarial outliers, exact recovery needs at least three more inliers than outliers.
• Good links. Hydra-Multi resends everything every update. DRACo-SLAM2 replays separately recorded sessions, so real packet loss is never tested.
• Privacy is free. CILC shows that a compromised teammate can reconstruct an honest robot’s images from the descriptors it broadcasts for loop closure, and fixes it with secure multi-party computation on the similarity step only.
• A guarantee survives merging. Conformal map fusion is one of the few papers that proves something about the merged map: robots exchange scalar e-values, and the fused occupancy map recovers its target coverage on any communication graph.

Questions

• What does a robot do with a loop closure from a teammate it can’t verify?
• How small can the exchanged summary be before matches become unreliable?
• Which part of the pipeline should carry the trust model: the matcher, the outlier filter, or the merge?

Related: 3D scene graphs, resilient consensus, Byzantine faults, underwater perception, multi-robot multimodal perception.

Large pretrained models that a robot uses without training them itself: for segmentation, depth, language, and increasingly for choosing actions. They change what a cheap robot can perceive, and they bring failure modes nobody wrote down.

Models

Uses

• A robot with one cheap camera can now get dense depth, segment objects it was never trained on, and answer questions about what it sees. Fast-FoundationStereo runs in 14 to 49 ms on a desktop GPU, within about 650 MB.
• Models with optional inputs (MapAnything, Pow3R, InfiniDepth) take whatever the robot has: a depth sensor, a pressure reading, a sparse lidar scan.
• A segmenter and a vision-language model can check each other. GoGet does this when it counts objects; see disagreement between models.

Failure modes

• Domain shift. These models are trained on internet images: daylight, air, familiar objects. On RoboTwin 2.0, changing only lighting, texture, clutter and table height takes π0 from 46.4% to 16.3% success (see benchmarks). Depth models degrade badly in water until fine-tuned (underwater perception).
• Confident and wrong. Dense wrong depth is a worse prompt than sparse true depth (Pow3R-SLAM). A VLM states a wrong count with full confidence. Brain2Qwerty’s language model writes fluent wrong sentences when its encoder fails.
• Attacks. A printed sign can hijack a VLM planner, and backdoors in vision-language-action models survive fine-tuning. See robot security and the to-read list.
• The link. Many systems call a hosted model. A robot that loses its connection loses its perception or its vocabulary.

Questions

• Which of these can run on a robot that has a Jetson and a battery, and what does it lose?
• How should a robot discount agreement between two teammates that run the same model?
• When a foundation model is wrong, how would the robot find out?

Related: open-vocabulary mapping, language-grounded planning, evaluating language models, multi-robot multimodal perception.

Turning an instruction such as “bring me something to drink from the kitchen” into a plan a robot can execute, using a map of the actual place. The map is usually a 3D scene graph, since its nodes and relations can be written out as text for a language model.

Approaches

• Paste the whole graph into the prompt. ConceptGraphs and FunGraph do this. It stops working once the graph is bigger than the context window.
• Let the LLM page through it. SayPlan starts from a collapsed graph and lets GPT-4 expand and contract nodes until it finds the relevant part. It finds the right subgraph in 86.7% of simple and 73.3% of complex queries; GPT-3.5 manages almost none. The fixed prompt alone takes about 3900 of 8192 tokens, and in the 300-room test every failure was a token overflow.
• Prune before planning. Taskography keeps the goal objects and everything linked to them, which is provably enough when the graph is correct. A classical planner that couldn’t finish in 24 hours on the full graph solved 72.86% of problems within 30 s on the pruned one. Pruning mattered more than the choice of planner.
• Compress while mapping. Clio keeps only what is informative about a task list given in advance.
• Keep a formal check around the LLM. In Dai and others, GPT-4 supplies a heuristic that may be wrong, and a consistent heuristic from a temporal-logic formulation keeps the final path optimal. First paths cost 1.24 to 1.69 times the optimum. The multi-robot system from MIT turns language into one PDDL goal per robot and hands those to a planner.
• Learn a policy on the graph. Ravichandran and others train a graph neural network to navigate over the scene graph. It does worse than a policy that only sees places once about 40% of the labels are wrong.

Assumptions

Every planner here assumes the graph is right. Taskography’s guarantee is for correct predicates. SayPlan’s simulator checks a plan against the same graph it was made from. Only Ravichandran’s paper measures what happens when labels are wrong, and the answer is that the learned policy falls apart.

Dai’s split, an untrusted proposer and a trusted checker, could extend from an LLM’s suggestions to a teammate’s map: use what the other robot says to guide the search, and only commit to what you can verify. That ties planning to robust robotics and runtime monitoring.

Questions

• What does a planner do when two robots’ graphs disagree about whether the door is open?
• How much of a plan should be re-checked when the scene changes under it (see dynamic scenes)?
• Most of these call a hosted LLM. What is the fallback when the link drops, underground or at a disaster site?

Related: open-vocabulary mapping, foundation models for robotics, multi-robot multimodal perception.

Coordinating on descriptions such as “the person in the red jacket near the jetty”. Robots with different sensors (a drone’s camera, a ground robot’s lidar, a boat’s sonar) need a shared representation to agree on what they observe.

3D scene graphs are one such representation: objects, places and rooms as nodes, and their relations as edges. There’s an interactive explainer on 3D scene graphs with their history, how they’re built, and what changes outside buildings.

Questions

• What does it mean for a semantic claim from another robot to be verifiable?
• Can resilient consensus work on labels and descriptions, or does everything have to be squashed into vectors first?
• Where do multimodal foundation models help, and where do they fail?

Papers

• HEROIC (2026) runs a decentralised drone and ground robot team whose members talk only in natural language.
• SGM-SLAM matches robots’ scene graphs on object labels and centroids alone, which keeps messages small.
• Open-vocabulary mapping covers how language gets attached to map objects, and what that costs.

Related: multi-robot systems, human-robot trust, 3D scene graphs.

Seeing and localising under water, where most of what works on land stops working. There is no GPS below the surface, cameras see a few metres on a good day, and the link to a teammate or to shore carries kilobits per second.

Light

Light goes missing in two ways. Attenuation removes the direct signal from the object, red first, so colour depends on range. Backscatter adds light scattered back by particles, a veil that thickens with distance and eats contrast. Sea-thru gives the two effects separate range-dependent coefficients, and nearly every reconstruction paper since inherits its model. Then add turbidity that changes from one dive to the next, a low-texture seafloor, caustics in shallow water, fish and marine snow, and flat camera ports that bend rays so the pinhole model drifts towards the edges of the image (Refractive COLMAP). Moving cameras, stirred sediment and a blank water column are on Kaveti and others’ list of field failures.

Navigation

• Lean on the other sensors. TURTLMap and AQUA-SLAM trust the Doppler velocity log, IMU and pressure sensor when the camera has nothing to grip, and use the camera for the map.
• Fuse camera and sonar. Sonar gives range without elevation, a camera gives bearing without range. Opti-acoustic SLAM takes each from where it is good. In its ablation, matching on geometry alone made the trajectory worse than plain odometry (1.40 m against 0.58 m).
• Teams. DRACo-SLAM2 closes loops between sonar robots by exchanging compact summaries (1.35 kbit on average) and asking for raw scans only after a match, inside a 62.5 kbps modem. The links in its experiments are replayed from separate recordings, and all robots stay at one depth.
• Relocalising after years. The Eiffel Tower set (four visits over five years) and the benthic localisation set (five sites, up to six years) show place recognition is much weaker underwater than on land.

Reconstruction

Most underwater splatting papers (SeaThru-NeRF, WaterSplatting, SeaSplat, UW-GS, RUSplatting) model the water inside the renderer, and assume structure-from-motion poses from dense, clear views. The cross-regime study shows where that breaks: structure from motion registers 99.5% of frames in clear water and none at 12 NTU. Under a light that moves with the vehicle, a renderer that ignores the medium beats both medium-aware ones. Swimm3R and AquaFlow move the medium model upstream into pose estimation, and Wat3R adapts the geometry model itself on unlabelled dive video.

Image metrics reward pictures. Inspection asks geometric questions, and the cross-regime study finds the photometric leader last on an industrial survey’s geometry.

Depth models

MoGe, UniDepth, Depth Anything 3 and VGGT give dense geometry from one cheap camera, and Fast-FoundationStereo runs in 14 to 49 ms on a desktop GPU. Underwater they degrade badly until fine-tuned (Cai and Metzler). Scale from one image is a learned prior about object sizes, and the seafloor has few familiar objects. Pow3R-SLAM found that 3% of pixels with true depth is a better prompt than dense wrong depth, and turbid water produces dense, confident, wrong depth.

Questions

• How does a vehicle know its depth or pose estimate has gone wrong, when turbid water fools the confidence head as well?
• What is worth sending over an acoustic modem? MERLION picks frames for a thin link; nobody yet picks measurements for a team.
• How do you get ground truth in murky water, when the reference tool (structure from motion) is the first thing to fail?

Related: collaborative SLAM, robust estimation, foundation models for robotics, Singapore scenarios, multi-robot multimodal perception.

Teams of small drones. They are cheap, fast and fragile, they carry little compute, and they can’t stop and think in mid-air.

Platforms

Most indoor swarm research flies the Crazyflie, a 27 g open-source quadrotor, and Crazyswarm, which flies 49 of them in dense formation from one base station. Every pose in Crazyswarm comes from one motion-capture system, so a tracker fault hits every drone at once and the room itself is a single point of failure. Outdoors, drones use GNSS, which can be spoofed.

Papers

• Pursuit and evasion. Kouzehgar and others train scouts and pursuers with a role-based multi-agent RL method and fly six Crazyflies with UWB positioning. The follow-up adds a changing communication network. Both trust every teammate’s report.
• Safe flight. Learned barrier certificates and equivariant graph CBFs keep swarms from colliding. See safe control.
• Spoofing. Vijay and others check GNSS positions against measured ranges between drones, with four real Crazyflies and three simulated ones. RigidShift shows the limit: shift the whole swarm slowly and every pairwise range stays correct, so no relative check can see it. Gil and others use the physics of the radio channel to catch spoofed identities, which works well indoors and is hard in free space.
• Perception. Mono-Hydra builds a scene graph from a camera and an IMU, which a small drone can carry. USS-Nav runs at 15 Hz on a Jetson Orin NX. HEROIC coordinates drones and ground robots for search, with all messages in natural language.

Questions

• How much of a ground robot’s perception stack fits on a drone that weighs 27 g?
• Which of the coordination results survive when positions come from GNSS that someone may be spoofing?

Related: multi-robot systems, resilient consensus, robot security, Singapore scenarios.

This survey of 3D scene graphs is written by people from most of the groups that built the field (Stuttgart, Sapienza, Google, MIT, Freiburg, UTN, Montreal, TU Munich). It gives one formal definition that covers the variants, splits construction into five design axes, sorts applications and their evaluation protocols, and ends every section with a paragraph of open challenges. A companion site, 3dscenegraphs.com, indexes the papers by keyword and already listed more than 150 when the survey was submitted. It is a good first read, and the open-challenge paragraphs can be read on their own.

Taxonomy

The definition is a tuple G = (V, V_g, V_f, E, E_f, [L]):

• V, E: nodes and directed edges. Several edges between the same pair are allowed, so the graph is a multigraph (Taskography’s choice).
• V_g, grounding: maps each node to the subset of metric map points it occupies. Point clouds are the common denominator; meshes and Gaussian splats reduce to them.
• V_f, E_f, features: geometric attributes (centroid, box, shape) and semantic ones (class, material, colour, CLIP-style embeddings, captions). Edge attributes are spatial (“next to”), structural (“part of”), semantic (“smaller than”) or functional (a switch “controls” a light). A graph that focuses on semantic relations gets called a 3D Semantic Scene Graph (3DSSG).
• L, layers (optional): assigns nodes to levels. Hierarchical edges link adjacent layers; intra-layer edges link nodes within a layer. A flat graph omits L.
• Time: any component can be indexed by t. A graph with any time-indexed part is a 4D scene graph.

Hierarchies come in three families:

• Indoor: building, (floor), room, place, object. Armeni, Hydra and Clio follow this hierarchy. S-Graphs swap the object layer for geometric primitives such as wall and corridor planes.
• Urban: district, roads and intersections, lane graph, objects (CURB-SG, OpenGraph).
• Functional or terrain-based: sidewalk, grass and asphalt regions, or concepts such as orchards and verges (Strader et al., xFLIE). Strader et al. is the only work the survey finds that handles indoor and outdoor in one framework.

Dynamics come in three representational styles, each with a cost:

1. A full graph per timestamp. Easy to learn from, but static content gets copied every step.
2. A separate node per observation of an entity (Khronos). History is kept, but something must decide which nodes are the same thing.
3. Nodes with time-varying attributes (Kimera, CURB-SG). The most common and the most compact, but one data-association error corrupts the attributes. Best for short-term motion, such as human nodes that store SMPL joints.

Functional and actionable graphs add part nodes (knobs, handles) with kinematic state, action nodes (RoboEXP), traversable free-space nodes, and “blind nodes” that hold beliefs about unobserved parts of the scene.

Construction axes

The survey’s conceptual map has five axes: processing (offline batch or online incremental), nodes (objects, places, attributes), edges (hierarchical or relational), priors (knowledge bases, multimodal LLMs, task) and consistency (spatial or temporal).

• Batch vs incremental. Vision papers say “prediction” or “learning” and work on a fused, instance-segmented point cloud (3DSSG, Open3DSG). Robotics papers say “construction” and work from live sensor streams (Hydra, Hydra-Multi, vS-Graphs). Batch methods skip long-term consistency entirely.
• Nodes. The dominant recipe lifts 2D instance masks into 3D with depth and pose, then fuses them across views. The closed-set version used Mask R-CNN; the open-vocabulary version chains a tagger or VLM, GroundingDINO or YOLO-World, SAM and CLIP. Rooms come from structural boundaries (S-Graphs), from slicing the ESDF below the ceiling and running 2D connected components (Kimera), or from clustering free space (Hydra). Urban region nodes come from trajectory inflection and intersection points.
• Edges. Candidate pairs are picked by geometric heuristics such as overlap or co-visibility, then labelled by a GCN (batch) or by an LLM or VLM (open-vocabulary).
• SLAM-integrated is the survey’s name for any pipeline that has to keep the graph spatially consistent without ground-truth pose, or temporally consistent in a changing scene. Spatial consistency is inherited from object SLAM: greedy or linear assignment on distance, overlap or embedding similarity; odometry priors; loop closures from graph descriptors (Hydra). The S-Graph family puts the whole graph into the factor graph, so walls, rooms and floors constrain the pose. Floor-level constraints help reject false loop closures across storeys.

The multi-robot treatment is one sentence. Spatial consistency becomes a shared-frame alignment problem in which each robot’s partial graph lives in its own drifting frame and needs inter-robot loop closures and cross-agent data association (Hydra-Multi). CURB-SG adds that anchoring on persistent structure (lanes, curbs) beats associating through dynamic or ambiguous entities. Compression and synchronisation for distributed teams get one line under “emerging applications”.

Evaluation and applications

• Intrinsic evaluation. Node quality borrows detection and segmentation metrics (precision, recall, F1, mIoU, AP, Chamfer distance). Edges use triplet metrics such as recall@k, because exact graph similarity is NP-complete. Only a handful of datasets carry ground-truth scene graphs (Armeni, 3DSSG and a few more), so online RGB-D systems end up on custom data or crowd-sourced edge checks. SLAM-integrated systems also report ATE.
• Scene understanding. CLIP matching works for “the red chair next to the window” and fails on multi-hop queries across layers. Serialising the graph as JSON for an LLM (SayPlan’s approach) handles multi-hop queries until the JSON outgrows the context window. Subgraph retrieval and GraphRAG are the current fixes.
• Planning and navigation. Kim et al. grounded PDDL goals in the graph by hand-written rules. Taskography started scalable planning. LLM planners (SayPlan) trade away completeness and soundness; translating language into PDDL or LTL and calling a classical planner keeps them. Only one paper does joint task and motion planning on a scene graph. Most evaluation runs in Habitat, AI2-THOR or iGibson, partly because static and full-observability assumptions make real-robot trials expensive.
• Manipulation. The graph mostly just localises the target object, so most of what it stores goes unused at execution time. The survey’s example: a bottle stored as “full” should change grasp force even though it looks empty.
• Dataset grouping. Their table groups evaluation by base dataset: 3RScan, ARKitScenes, Gibson, KITTI, MP3D, Replica, ScanNet, simulators, custom. KITTI and the CARLA simulator (CURB-SG) are the only outdoor sources; terrain and agricultural graphs sit under custom data. The custom row is the longest, and it holds almost every SLAM-integrated and multi-robot system (Hydra, Clio, Hydra-Multi, S-Graphs, vS-Graphs), so those systems are rarely compared on shared data.

Open challenges

These are the survey’s open-challenge paragraphs, in their order:

• Abstract layers. Layer definitions are hand-coded. Open areas without walls have no natural regions, and only ASHiTA starts to infer layers from the task.
• Dynamics at scale. Graphs grow with the data and have only been shown over short durations. Months or years of operation need compression, summarisation or marginalisation. Most systems keep a single world estimate and no belief over past, present and future states. Prediction of how entities and relations will evolve is understudied.
• Function, physics and causality. Full functional graphs are too large for small platforms. They suggest parametric nodes that expand detail on demand, mass and material attributes, and causally consistent updates when an action changes what else can be reached.
• Node construction. Cascaded perception means early errors propagate into duplicate, fragmented or missing nodes. Objects are assumed to be discrete and well bounded. Not all semantics can be extracted up front, so they argue for hybrid graphs that keep a compact implicit store and make attributes explicit on demand.
• Edges. Co-visibility heuristics miss long-range and functional relations, upstream errors flow into edges, and the number of candidate pairs explodes. Relations should be generated on demand.
• Consistency. Consistency is treated as purely geometric. Nobody checks whether a node’s embedding, attributes and relations still agree across visits, even though that could catch changes geometry misses. Change detection is sensitive to tuning. There is no principled rule for when to commit to a hard data association, or for when to forget an entity (for example, replacing a frequently moved object with a distribution over its room).
• Reasoning. They want benchmarks for long-horizon spatio-temporal and multi-hop reasoning, faithfulness to the graph, LLMs distilled into small scene-specific agents, and query-adaptive graphs.
• Planning. Scenes are assumed static and fully observed, and tasks are mostly mobile pick-and-place. They call for tighter coupling, with the graph acting as the planner’s memory.
• Manipulation. VLAs see only the current frame and the instruction. A graph could give them memory, check preconditions and effects, and bind “the bottle on the shelf” to a node.
• Overall (conclusion). Evaluation is the most systemic gap: graph quality and task success are measured on different datasets, so the effect of perception errors on behaviour is rarely quantified. Learning pipelines lack spatial consistency. Indoor and outdoor use incompatible decompositions. Most methods assume quasi-static scenes.

Findings

• The field went from offline, static, closed-set graphs (2019 to 2020) to online SLAM-integrated, open-vocabulary, LLM-facing graphs (2023 to 2026). The timeline’s colour coding follows that arc: datasets, hierarchy, semantic relationships, SLAM, planning, open-vocabulary, functionality.
• The authors split the field into two camps: vision (batch, learned relations, 3DSSG-style metrics) and robotics (incremental, pose drift, ATE). Methods from one camp are rarely tested on the other’s benchmarks.
• Hughes et al. show that indoor scene graphs have low treewidth under some assumptions, which is the theoretical case for hierarchy that the survey leans on.

Limits

• Outdoor coverage is thin: the only non-indoor rows are urban driving and terrain or agricultural hierarchies, and LiDAR appears only as an input stream.
• Multi-robot construction gets a sentence and a citation to Hydra-Multi and CURB-SG. Communication limits, trust between robots, heterogeneous sensors, and faulty or adversarial inputs are not discussed anywhere in the text. When the survey talks about robust deployment, it means coping with perception noise, drift and scene change.
• The author list overlaps heavily with the authors of the systems reviewed (Hydra, Khronos, 3DSSG, Open3DSG, FunGraph). The critique is candid about evaluation gaps, but the framing follows their lines of work.
• It is a survey, so it reports no new numbers. Its claims of what is “underexplored” are bounded by the 150-odd papers it indexes.

Relevance

Background for 3d-scene-graphs and for the scene-graph half of multi-robot multimodal perception. The parts it leaves blank are the interesting ones for field teams: what fusion looks like when robots see through different sensors, over bad links, and cannot all be trusted (see collaborative SLAM and robust robotics). Its forgetting and belief-over-time challenges map directly onto long-term autonomy and dynamic-scenes.

Links

• arXiv · paper index site
• Builds on or reviews: armeni-2019-3d-scene-graph, wald-2020-3dssg, rosinol-2020-3d-dsg, rosinol-2021-kimera, hughes-2022-hydra, hughes-2024-foundations, bavle-2022-s-graphs, tourani-2026-vs-graphs, chang-2023-hydra-multi, greve-2024-curb-sg, schmid-2024-khronos, looper-2023-3d-vsg, kim-2019-3d-scene-graph
• Subfield: Semantic mapping, Mapping and localisation

Hydra-Multi builds one 3D scene graph from a team of robots, online. Each robot runs its own copy of Hydra and periodically sends its whole local scene graph to a central control station. The station searches for inter-robot loop closures with Hydra’s hierarchical descriptors (places, then objects, then visual appearance), checks each match geometrically with RANSAC or TEASER++, estimates every robot’s frame by robust pose averaging, and optimises the joint graph as an embedded deformation graph with Graduated Non-Convexity (GNC). Places and objects that two robots both mapped are proposed as merges, and a merge is kept only if GNC classes it as an inlier.

Findings

• The robots need no initial pose calibration. Pairwise frame offsets come from a truncated least squares average over all loop closures between two robots, chained along a spanning tree of the robot graph. A robot counts as initialised once it has at least 5 inlier loop closures.
• On the Simmons dormitory data, identical rooms with identical furniture make the frontend return 80% to 90% outlier loop closures, and the frames still align.
• Trajectory error (ATE): 0.25 m on simulated uHumans2 against 0.59 m for a centralised Kimera-Multi, and 3.92 m against 4.99 m on the multi-floor SidPac recordings. On Simmons the lidar-based LAMP 2.0 stays ahead (0.73 m and 0.58 m against 0.99 m and 0.79 m).
• The ablation shows where the accuracy comes from. On Simmons SM1, objects found and objects correct fall from 59.3% and 55.2% to 23.8% and 16.2% without initial alignment, and to 37.3% and 29.5% without merge proposals.
• Two Jackals mapped a full dormitory floor in about 30 minutes. One robot covering similar paths took about 50.
• A heterogeneous run pairs a camera robot with a lidar robot that has no semantics. The lidar robot still contributes mesh and places; only the camera robot fills the object layer. Any robot whose map matches at least one layer can join.
• The frontend stays near 100 ms per iteration. The backend grows with graph size, so loop-closure corrections lag behind.

Limits

• One control station receives everything and does all the fusion. The authors list a distributed backend as future work.
• Each robot resends its entire scene graph and mesh at every update, and those dominate bandwidth (loop-closure descriptors and deformation-graph inputs stay under 1 MB). The authors call this naive.
• The world is assumed static. The authors list dynamic objects as future work.
• Peers are trusted. GNC rejects wrong loop closures and wrong merges caused by perceptual aliasing, but nothing models a robot that sends a corrupted or fabricated graph, or one whose odometry has quietly failed. Reconciliation is all or nothing: if fewer than half of proposed merges survive, every merge is undone.
• Object merges require the same semantic label and overlapping boxes, so robots must share a label set. Two segmentation networks are used across datasets (HRNet, OneFormer), never mixed within one team.
• The frame alignment is computed once and recomputed only if it disagrees with the optimised solution by more than 10 m.
• Links are assumed reliable. There is no test with dropped, delayed or rate-limited messages, and the two SidPac “robots” are two hand-held recordings.

Relevance

Hydra-Multi shows how far outlier-robust estimation goes for multi-robot scene graphs: GNC copes with aliasing, yet every robot’s local graph is taken as honest. A team whose robots see different things, or where one robot’s map is quietly wrong, needs more than this; see collaborative SLAM, outlier-robust estimation, Byzantine fault tolerance and multi-robot multimodal perception. The outdoor counterpart is CURB-SG.

Links

• arXiv · IEEE
• Subfield: Multi-robot systems, Semantic mapping, Mapping and localisation

A possible thesis thread: robots that keep working for weeks or months without a person resetting them. The world changes under their maps, their parts wear, and the small faults that never turn up in a demo eventually do.

Change

Maps assume the world holds still. The better systems separate evidence of absence (I looked and the chair wasn’t there) from absence of evidence (I didn’t look). Khronos does this with camera rays, which needs a wall behind the object. See dynamic scenes.

Memory

Scene graphs grow with the data and have only been shown over short runs. Recent memory papers ask what to keep, how far to trust it, and how to tell when it has gone stale. Relocalising against imagery from months or years earlier is still weak, especially under water (underwater perception).

Deployments

Few papers report months in the field: LTR² covers 40 km over six months, and Kairos has eleven months of pedestrian data. Most “long horizon” benchmarks mean a task with many steps (benchmarks).

Faults over time

Faults that come and go, faults that build up slowly, and batteries that run out all look different from the sudden failures most papers test. A safety layer that stops the robot is fine in a demo and expensive in the field, where someone has to go out and reset it. My workshop paper Safe but Stuck is about this. See failure recovery.

Open questions

1. What counts as evidence that something is gone, outdoors or in open air, where there is no wall behind it?
2. When should a frequently moved object stop being a node and become a distribution over a room?
3. How should a robot decide that what it remembers is out of date?
4. How long does a robot stay useful after its safety layer acts, and can it recover without a person?
5. Can a team divide the work of re-checking a shared map, so every part gets seen often enough?

Papers

Other threads: robust robotics, multi-robot multimodal perception. All three on one page: thesis map.

Maps assume the world holds still. It doesn’t: chairs move, doors close, cars leave, and a field looks different after the harvest. A robot that runs for weeks needs a map that knows what changed, what probably changed, and what it hasn’t looked at for a while.

Representations

The scene graph survey sorts the options:

1. A full graph per time step. Easy to learn from, wasteful, since everything static is copied every step.
2. A node per observation (Khronos). The history is kept, but something has to decide which observations are the same object.
3. Nodes with attributes that change over time (Kimera, CURB-SG). Compact, and the most common, but one wrong data association corrupts the history of that node.

Absence

Khronos separates absence of evidence (I didn’t see the chair) from evidence of absence (I looked where the chair was and saw the wall behind it). It casts camera rays, and calls an object gone when 60% of the rays in a 5 s window pass through where it used to be. Change detection F1 is 62.0, against 38.6 for Panoptic Mapping. SuperMap keeps a log-odds existence value per map point instead, and reports change recall of 0.62 to 1.00 on a ten-minute real run. 3D VSG tries to predict which nodes will change (77.1% accuracy), but change is rare (13% to 21% of nodes) and it runs on ground-truth graphs.

Khronos also found that swapping in open-set segmentation barely moved its change-detection F1 (64.4 against 64.6). The hard part is the bookkeeping.

Limits

• The ray test needs a surface behind the object. In an open field or open air there is no back wall, so there is no evidence of absence, only of not seeing.
• Memory grows without bound. Nobody has shown a scene graph running for months with a principled way to forget.
• Consistency is checked geometrically. The survey points out that nobody checks whether a node’s embedding, attributes and relations still agree from one visit to the next, which could catch changes that geometry misses.
• With several robots, “it moved” and “the other robot is wrong” look the same. Telling them apart needs a model of how much to trust each robot (resilient consensus, Byzantine faults).

Questions

• When should a frequently moved object stop being a node and become a distribution over its room?
• How long can a robot trust an observation it made last week?
• Can a team divide the work of re-checking a shared map, so each part is seen often enough?

Related: long-term autonomy, 3D scene graphs, collaborative SLAM.

How the field measures whether a robot policy works. Almost always: a success rate on one task, in one simulated room or tabletop, with a check written by the people who built the simulator.

Benchmarks

• The embodied AI survey reviews nine simulators. All of them model apartments. Point navigation with GPS and a compass already gets within 3 to 5% of a shortest-path oracle, so the easy version is nearly solved.
• RoboTwin 2.0 moves to bimanual tabletop work and generates its data with strong randomisation.
• LIBERO runs a sequence of 130 tasks to test whether a policy keeps what it learned.
• EmboCoach-Bench measures the engineering loop around training: can an AI agent write and tune the code that trains a robot?

Gaps

Across all of these, no benchmark has more than one robot acting. None injects a sensor fault or a teammate that lies. None runs outdoors, on the water or in the air. “Long horizon” means a task with many steps, or a sequence of new tasks, and never one robot running for weeks while its world changes. The survey pointed out the missing multi-agent work in 2021, and the newer benchmarks still don’t have it.

The drops are large even without faults. On RoboTwin 2.0, randomising lighting, texture, clutter and table height takes π0 from 46.4% to 16.3% success, and Diffusion Policy from 28.0% to 0.6%.

Data generation

RoboTwin 2.0 varies five things: distractor objects (deliberately excluding look-alikes), 11,000 generated textures, light colour, type, intensity and position, table height, and LLM-written rewordings of the instruction, plus 1 cm of camera jitter for the real-robot runs. Mass, friction and latency are not varied, though its related work lists them as standard. Every axis is drawn from what the designers think is plausible. Nobody randomises failures (a dropped frame, a frozen encoder) or adversarial inputs. Domain randomisation is built for the average case, and security is about the worst case.

The generators share one more assumption with AlphaEvolve and EmboCoach: the simulator’s success check is the truth. AlphaEvolve names its reliance on an automatic evaluator as its main limitation. An agent that can rewrite training code against a fixed check will eventually find a way to pass the check.

Policies

Most of the robot learning on my disk is built on a small set of open policies: ACT, Diffusion Policy, OpenVLA, π0 and SmolVLA, trained and run through LeRobot. RT-2 and PaLM-E are where vision-language-action models started.

Questions

• What would a benchmark look like with three robots, one of them broken, and a task that lasts a week?
• Can a data generator randomise faults and attacks as easily as it randomises textures?
• How do you score a policy that is right to refuse?

Related: foundation models for robotics, failure recovery, robust robotics, long-term autonomy.

Khronos builds a dense metric-semantic map of a scene at every past time, online, so a robot can answer “what was here at 1:52?” as well as “what is here now?”. The authors call the problem Spatio-temporal Metric-semantic SLAM (SMS) and factor it with one assumption: over a short window, both odometry drift and scene change are small. Observations are grouped into object fragments inside an active window, where short-term motion (people, a pushed cart) is tracked with a free-space motion cue. A global stage adds fragment poses to Hydra’s deformation graph and solves it with truncated least squares and GNC, treating same-label, overlapping fragments as candidate associations. A last stage decides when each object appeared or vanished by checking it against a library of camera rays.

Findings

• The ray check separates evidence of absence from absence of evidence. If a ray to a background vertex passed through where an object was, the object was gone at that time; a timestamp counts as absence only if at least 60% of rays in a 5 s window agree.
• Two TESSE simulations: Apartment (87 s, 39 m, 64 static objects, 10 dynamic, 6 changes) and Office (217 s, 181 m, 196 objects, 6 dynamic, 8 changes).
• Office with Kimera visual-inertial poses: change detection F1 62.0 against 38.6 for Panoptic Mapping, dynamic-object F1 53.9 against 23.5 for Dynablox, and static-object F1 73.1 against 62.1 for Hydra.
• Apartment with ground-truth poses: object recall 83.9 against Hydra’s 28.0. Hydra segments objects out of the background mesh and misses small items.
• Swapping ground-truth semantics for an open-set SAM plus CLIP frontend barely changes Apartment change F1 (64.4 against 64.6).
• The active window takes 45.5 ± 9.2 ms per frame (22.2 FPS) on a laptop i7-12700H CPU. Optimisation after a loop closure can take several seconds; change inference stays under 100 ms.
• On a Jackal, against manual labels: object precision 86.9 and recall 97.7, change precision 87.0 and recall 66.7. The table prints a change F1 of 93.9, which cannot follow from that precision and recall, so I quote only the pair. It also ran on a Spot across a building floor (qualitative).

Limits

• Fragment associations use bounding-box centroids as translation-only constraints, so partial views and occlusion hurt them. No 6D registration (authors).
• Association is purely geometric, so a moved object becomes one disappearance plus one appearance. Its history is not linked (authors).
• The ray test needs a reference surface behind the object. The authors expect it to fail in open spaces with sparse surfaces, which covers most outdoor, aerial and open-water scenes.
• Every fragment stays in memory, so the problem grows without bound (authors).
• Motion detection misses people walking straight towards or away from the camera, because the free space behind them is never observed (authors).
• Single robot. The main benchmark uses simulator semantics.
• SuperMap reports that Khronos could not keep up in its tests because of slow semantic inference. Khronos’s own timing assumes segmentation is already there, so the two claims probably differ on what gets counted.

Relevance

Khronos defines what “the map at time t” should mean, and its fragment and ray machinery is reusable for long-term autonomy and dynamic scenes. It also inherits the static-world baseline of Hydra and the short- and long-term split of 3D VSG.

Links

• arXiv · code
• Subfield: Semantic mapping, Mapping and localisation

LTR² is a teach-and-repeat system that teaches with LiDAR in good conditions and repeats with 4D millimetre-wave radar, which keeps working in bad weather and smoke. A cross-modal registration network aligns sparse forward-looking radar to dense omnidirectional LiDAR using Doppler motion priors and the physics of LiDAR intensity and radar power density, and it fine-tunes itself from localisation errors without ground truth. It was tested on three robot platforms over 40+ km and 6 months, including smoke at night, with centimetre-level accuracy.

Relevance

Six months and 40 km is a real long-term deployment, and teaching with one sensor and repeating with another suits mixed teams. Read for long-term autonomy and multimodal perception.

Links

• arXiv
• Subfield: Planning and navigation, Semantic mapping, Mapping and localisation

MemoGuard checks episodic memories before a communication-limited robot reuses them, because a retrieved action can match the current context and still be unsafe (changed topology, too little battery, an unreliable past outcome). The paper calls these memory traps. MemoGuard validates each memory against topology, resource and outcome contracts and calls local LLM reasoning only when validation fails; in a corridor-inspection simulator it cuts battery safety violations by 76.6% against similarity-only reuse and fallback calls by 21.4% against always reasoning.

Relevance

Memory that was right last week can be wrong today, and this treats stale memory as a safety problem. See long-term autonomy, runtime monitoring and robust robotics.

Links

• arXiv · code
• Subfield: Safety and verification, Planning and navigation, Semantic mapping

FOCUS detects object-level change in semi-static scenes where an object may be swapped in place for a similar-looking one. It fuses a geometric likelihood with an appearance likelihood rendered from a 3D Gaussian map, and a recursive three-state estimator tracks each object as PERSISTED, REPLACED or REMOVED. Rendering error is modelled probabilistically, with parameters calibrated from a change-free replay of the first mapping session; replacement F1 rises from 0.20 to 0.84 over a probabilistic baseline on a new Isaac Sim warehouse benchmark, and the method transfers to TorWIC without retuning.

Relevance

Calibrating from a change-free replay suits any map that has to run for months. Read with OASIS-Map for long-term autonomy and dynamic scenes.

Links

• arXiv
• Subfield: Semantic mapping, Mapping and localisation