Opti-Acoustic Semantic SLAM with Unknown Objects in Underwater Environments

3 min read

This paper builds object-level semantic SLAM for an underwater vehicle that is never told which object classes are in the scene. MaskCut, an unsupervised segmenter on a DINO backbone, proposes object masks in monocular camera frames, and each mask’s patch features are squeezed to a fixed 384-dimensional vector (patch averaging) or a 3 x 384 UMAP projection. The mask centroid gives bearing and elevation; the multibeam sonar beam closest in bearing gives range. So the camera supplies the elevation the sonar cannot resolve, and the sonar supplies the range a monocular camera lacks. Each observation is matched to an existing landmark by cosine similarity of its feature, then by a chi-square test on Mahalanobis distance, and iSAM2 jointly optimises vehicle poses and object landmarks on top of EKF odometry from a DVL, an IMU and a pressure sensor.

Findings

  • Map-level precision and recall with patch averaging: 0.82/0.82, 0.41/1.0, 0.86/1.0 and 0.73/0.92 on the four indoor pool runs, and 0.75/0.60 on the outdoor tank. A YOLOv8n baseline fine-tuned on the same objects reaches precision 1.0 on every run, with recall from 0.75 to 1.0.
  • Patch averaging (384 numbers) beats UMAP (1,152 numbers) on precision in every run. The authors suspect UMAP’s assumption of data spread uniformly on a manifold does not hold for these features.
  • Outdoor tank trajectory error (APE): odometry alone 0.58 m, odometry plus geometric association 1.40 m, adding uncertainty 0.60 m, adding semantic features 0.56 m. Geometry-only association makes the trajectory worse than no loop closures at all. The closed-set YOLOv8 variant gets 0.59 m.
  • ORB-SLAM3 loses tracking after 27% of the outdoor trajectory because caustics move across the scene, with 2.28 m APE on the part it completes.
  • Off-the-shelf YOLOv8 weights detect none of the objects. The fine-tuned baseline needed 230 hand-labelled images of the exact objects, augmented to 691.
  • Setup: a modified BlueROV2. Indoors, an Oculus M1200d at 2.1 MHz (60 degree horizontal, 12 degree vertical aperture). Outdoors, a roughly 64 m² tank about 4 m deep, three loops, an Oculus M750d at 750 kHz (130 degree horizontal, 20 degree vertical). Ground truth comes from a low-cost iLBL-USBL acoustic beacon system at 1 Hz, with heading from beamforming on the vehicle’s acoustic array because the tank walls corrupt the magnetometer.
  • They release camera, sonar, odometry and ground-truth data plus 691 labelled images over 17 underwater object classes.

Limits

  • Metrics are at map level, so one bad association over 100 or more observations of an object spawns a duplicate landmark. The authors trace false positives to sonar multipath giving very wrong ranges and to lighting or viewpoint changes that break feature similarity. False negatives come from the segmenter missing a white pipe with little contrast against the background.
  • The range lookup takes the strongest return in one beam above a threshold; the similarity threshold is chosen by hand.
  • DINO features are not tied to language, so “classes” are clusters in feature space.
  • Only a pool and a small tank. No open-water or turbid-water runs, no runtime figures, one vehicle. Algae marks on the tank wall were picked up as objects and left out of the scoring.

Relevance

Camera and sonar cover each other’s blind spots: the camera gives bearing and elevation, the sonar gives range. The ablation, where geometry-only association makes the trajectory worse than odometry, shows that wrong data association does more damage than none; see robust robotics, robust estimation and underwater perception.