System Failure Injection

System failure injection allows you to induce different types of sensor and system failures, either programmatically using the MAVSDK failure plugin, or "manually" via a PX4 console like the MAVLink shell. This enables easier testing of safety failsafe behaviour, and more generally, of how PX4 behaves when systems and sensors stop working correctly.

Failure injection is disabled by default, and can be enabled using the SYS_FAILURE_EN parameter.

Failure injection must also be be supported by the current simulator, and the set of supported failures is simulator-dependent.

INFO

PX4 may accept a command to set a particular failure mode even it that mode is not supported by your simulator.

All MAV_CMD_INJECT_FAILURE commands are handled internally by the failure-injection module, which acknowledges each command and republishes the active failures for the sensor/actuator simulators to apply. The failure-injection module will NACK the command with MAV_RESULT_UNSUPPORTED for failure combinations that are not implemented by PX4 or any simulator. However it the module will accept (respond with MAV_MISSION_ACCEPTED) for any other failure-type, even if it is not supported by your particular simulator.

Failure System Command

Failures can be injected using the failure system command from any PX4 console/shell (such as the QGC MAVLink Console or SITL pxh shell), specifying both the target and type of the failure.

Syntax

The full syntax of the failure command is:

failure <component> <failure_type> [-i <instance_number>]

where:

• component:
  • Sensors:
    • gyro: Gyroscope
    • accel: Accelerometer
    • mag: Magnetometer
    • baro: Barometer
    • gps: Global navigation satellite system
    • optical_flow: Optical flow.
    • vio: Visual inertial odometry
    • distance_sensor: Distance sensor (rangefinder).
    • airspeed: Airspeed sensor
  • Systems:
    • battery: Battery
    • motor: Motor
    • servo: Servo
    • avoidance: Avoidance
    • rc_signal: RC Signal
    • mavlink_signal: MAVLink data telemetry connection
• failure_type:
  • ok: Publish as normal (Disable failure injection)
  • off: Stop publishing
  • stuck: Constantly report the same value which can happen on a malfunctioning sensor
  • garbage: Publish random noise. This looks like reading uninitialized memory
  • wrong: Publish invalid values that still look reasonable/aren't "garbage"
  • slow: Publish at a reduced rate
  • delayed: Publish valid data with a significant delay
  • intermittent: Publish intermittently
• instance number (optional): Instance number of affected sensor. 0 (default) indicates all sensors of specified type.

INFO

The simulated GPS (SITL) implements only the off, stuck, and wrong failure modes; the other failure types have no effect on it.

MAVSDK Failure Plugin

The MAVSDK failure plugin can be used to programmatically inject failures. It is used in PX4 Integration Testing to simulate failure cases (for example, see PX4-Autopilot/test/mavsdk_tests/autopilot_tester.cpp).

The plugin API is a direct mapping of the failure command shown above, with a few additional error signals related to the connection.

Example: GPS

To test the GPS failsafe by stopping GPS:

1. Enable the SYS_FAILURE_EN parameter.

2. Enter the following commands on the MAVLink console or SITL pxh shell:

   # Stop GPS publishing
   failure gps off
   
   # Restart GPS publishing
   failure gps ok

Example: Motor

To stop a motor mid-flight without the system anticipating it or excluding it from allocation effectiveness:

1. Enable the SYS_FAILURE_EN parameter.

2. Enable CA_FAILURE_MODE parameter to allow turning off motors.

3. Enter the following commands on the MAVLink console or SITL pxh shell:

   # Turn off first motor
   failure motor off -i 1
   
   # Turn it back on
   failure motor ok -i 1