MERLION: Marine ExploRation with Language guIded Online iNformative Visual Sampling and Enhancement

2 min read

MERLION picks a handful of frames from a long, murky AUV video that match what a user asked for in words. Every frame goes through CLIP; frames whose softmax similarity to a positive text query (scored against one or more negative queries) clears a threshold are passed to an online “surprise” sampler adapted from ROST, which keeps a fixed-size summary of the most mutually distant CLIP embeddings. The MERLION-E variant runs a conditional diffusion enhancer (tang-2023-dm-underwater) only on frames that already passed the text filter, then scores them again, so the slow enhancer sees a small part of the stream.

Findings

  • Scoring uses SRUM (semantic match plus temporal closeness to frames picked by 14 human evaluators, six frames each). On the low-visibility video MERLION-E scores 0.407, against 0.111 for ROST and 0.656 for the human-vs-human score.
  • On the moderate-visibility video MERLION-E scores 0.557, slightly above the human-vs-human 0.522.
  • Enhancement barely moves ROST (0.111 to 0.138 at low visibility, 0.364 to 0.333 at moderate) but helps the CLIP pipeline a lot. The authors put this down to CLIP being trained on clean images: enhancement pushes murky frames back towards its training distribution.
  • On an NVIDIA Jetson Orin AGX 32GB in a 5-inch tube on an upgraded BlueROV2, with the feed subsampled to 5 fps and 128x128 pixels, MERLION-E handles 2 to 3 frames per second. It was deployed off two Singapore islands, one of them in open turbid water.

Limits

  • The similarity threshold is set by hand per dataset (40, 50 and 70 for low, moderate and clear visibility), so the robot has to know how murky the water is before it starts. The authors say the values were found empirically.
  • Three videos, all about fish, taken from public dive footage. The clear-visibility video has no MERLION-E result, so “best on all three” rests on two comparisons.
  • The target is human taste. That suits summaries, but says nothing about whether the chosen frames help mapping or counting.

Relevance

Few underwater papers treat the link to the surface as a design input; this one asks what is worth sending. For a team sharing a thin acoustic channel the question grows: whose frames go first, and how does a robot tell that a teammate’s “interesting” frame came from a fouled lens? See underwater perception and foundation models for robotics.