Runtime Verification and Field-based Testing for ROS-based Robotic Systems

2 min read

Caldas and colleagues write 20 guidelines for building ROS systems that can be runtime-verified and tested in the field: 8 for developers (provide APIs for querying system modes, logging and fault injection; isolate components; keep global time monotonic) and 12 for QA teams (understand monitor overhead, specify properties in a temporal logic or a ROS-specific DSL, record and replay, no GUIs in CI, post-mortem analysis). The guidelines come from a literature review, a sweep of 1088 cloned ROS repositories for exemplars, and two questionnaires.

Findings

  • The literature search returned 262 papers, 132 after merging, and 27 relevant studies: 16 on runtime verification, 9 on field-based testing, 1 on both.
  • 55 questionnaire answers; median ratings sit at Agree or Strongly Agree for applicability, clarity and usefulness. Three guidelines (identifying security or privacy vulnerabilities, test automation, model-based runtime assessment) had weak support and were revised after follow-up with 10 respondents.
  • Some practitioners saw security checks as unnecessary for academic robots, or solved them by keeping the robot off the Internet.
  • The mapping of open challenges to guidelines leaves two runtime verification challenges with no guideline at all: distributed monitoring and “richer reactions” (doing something beyond flagging a violation).

Limits

  • Guidelines are practice advice, validated by opinion. There is no experiment showing that following them finds more bugs.
  • The surveys they start from do not focus on ROS, which the authors admit explains some unmapped challenges.
  • Respondents found property specification useful but hard to apply, and felt it suits senior developers more than QA staff.

Relevance

The two empty cells (distributed monitors and richer reactions) are the main open problems for teams of field robots; see runtime monitoring and failure recovery. Tools named here include ROSMonitoring, and the formal side is surveyed in Luckcuck et al..