ODrive Node Constantly Crashing

[ansarid]

Hello,

I have a large system with 12 ODrive Pros split between 2 can interfaces. I am able to get data from all my ODrives as well as command them. But quite often I'm having an issue where many of the ODrive nodes crash. Here is the log output filtered for messages from the ODrive nodes.

[INFO] [odrive_can_node-4]: process started with pid [10910]
[INFO] [odrive_can_node-5]: process started with pid [10912]
[INFO] [odrive_can_node-6]: process started with pid [10930]
[INFO] [odrive_can_node-7]: process started with pid [10939]
[INFO] [odrive_can_node-8]: process started with pid [10941]
[INFO] [odrive_can_node-9]: process started with pid [10946]
[INFO] [odrive_can_node-10]: process started with pid [10953]
[INFO] [odrive_can_node-11]: process started with pid [10956]
[INFO] [odrive_can_node-12]: process started with pid [10969]
[INFO] [odrive_can_node-13]: process started with pid [10988]
[INFO] [odrive_can_node-14]: process started with pid [10996]
[INFO] [odrive_can_node-15]: process started with pid [11031]
[odrive_can_node-4] �[0m[INFO] [1779588690.057289498] [front_left_top_suspension.odrive_node]: node_id: 1�[0m
[odrive_can_node-4] �[0m[INFO] [1779588690.057546695] [front_left_top_suspension.odrive_node]: interface: can0�[0m
[odrive_can_node-5] �[0m[INFO] [1779588689.605896222] [front_right_top_suspension.odrive_node]: node_id: 2�[0m
[odrive_can_node-5] �[0m[INFO] [1779588689.606111631] [front_right_top_suspension.odrive_node]: interface: can0�[0m
[odrive_can_node-6] �[0m[INFO] [1779588690.379771910] [back_left_top_suspension.odrive_node]: node_id: 3�[0m
[odrive_can_node-6] �[0m[INFO] [1779588690.380074462] [back_left_top_suspension.odrive_node]: interface: can0�[0m
[odrive_can_node-7] �[0m[INFO] [1779588689.623248829] [back_right_top_suspension.odrive_node]: node_id: 4�[0m
[odrive_can_node-7] �[0m[INFO] [1779588689.623506563] [back_right_top_suspension.odrive_node]: interface: can0�[0m
[odrive_can_node-9] �[0m[INFO] [1779588690.367362388] [front_right_steer.odrive_node]: node_id: 12�[0m
[odrive_can_node-9] �[0m[INFO] [1779588690.367582959] [front_right_steer.odrive_node]: interface: can1�[0m
[odrive_can_node-11] �[0m[INFO] [1779588690.163207625] [back_right_steer.odrive_node]: node_id: 14�[0m
[odrive_can_node-11] �[0m[INFO] [1779588690.163530242] [back_right_steer.odrive_node]: interface: can1�[0m
[odrive_can_node-12] �[0m[INFO] [1779588690.231027143] [front_left_drive.odrive_node]: node_id: 21�[0m
[odrive_can_node-12] �[0m[INFO] [1779588690.231287043] [front_left_drive.odrive_node]: interface: can1�[0m
[odrive_can_node-8] �[0m[INFO] [1779588690.557359043] [front_left_steer.odrive_node]: node_id: 11�[0m
[odrive_can_node-8] �[0m[INFO] [1779588690.663750270] [front_left_steer.odrive_node]: interface: can1�[0m
[odrive_can_node-10] �[0m[INFO] [1779588691.117304756] [back_left_steer.odrive_node]: node_id: 13�[0m
[odrive_can_node-10] �[0m[INFO] [1779588691.167078753] [back_left_steer.odrive_node]: interface: can1�[0m
[odrive_can_node-13] �[0m[INFO] [1779588691.365165584] [front_right_drive.odrive_node]: node_id: 22�[0m
[odrive_can_node-13] �[0m[INFO] [1779588691.365437428] [front_right_drive.odrive_node]: interface: can1�[0m
[odrive_can_node-14] �[0m[INFO] [1779588691.576816315] [back_left_drive.odrive_node]: node_id: 23�[0m
[odrive_can_node-14] �[0m[INFO] [1779588691.643345248] [back_left_drive.odrive_node]: interface: can1�[0m
[odrive_can_node-15] �[0m[INFO] [1779588692.164572434] [back_right_drive.odrive_node]: node_id: 24�[0m
[odrive_can_node-15] �[0m[INFO] [1779588692.165921138] [back_right_drive.odrive_node]: interface: can1�[0m
[odrive_can_node-7] terminate called after throwing an instance of 'rclcpp::exceptions::RCLError'
[odrive_can_node-7]   what():  failed to publish message: cannot publish data, at ./src/rmw_publish.cpp:62, at ./src/rcl/publisher.c:250
[ERROR] [odrive_can_node-7]: process has died [pid 10939, exit code -6, cmd '/home/robot/rover_ws/install/odrive_can/lib/odrive_can/odrive_can_node --ros-args -r __node:=odrive_node -r __ns:=/back_right_top_suspension --params-file /tmp/launch_params_wsxrx8b8'].
[odrive_can_node-8] terminate called after throwing an instance of 'rclcpp::exceptions::RCLError'
[odrive_can_node-8]   what():  failed to publish message: cannot publish data, at ./src/rmw_publish.cpp:62, at ./src/rcl/publisher.c:250
[odrive_can_node-5] terminate called after throwing an instance of 'rclcpp::exceptions::RCLError'
[odrive_can_node-5]   what():  failed to publish message: cannot publish data, at ./src/rmw_publish.cpp:62, at ./src/rcl/publisher.c:250
[odrive_can_node-4] terminate called after throwing an instance of 'rclcpp::exceptions::RCLError'
[odrive_can_node-4]   what():  failed to publish message: cannot publish data, at ./src/rmw_publish.cpp:62, at ./src/rcl/publisher.c:250
[odrive_can_node-12] terminate called after throwing an instance of 'rclcpp::exceptions::RCLError'
[odrive_can_node-12]   what():  failed to publish message: cannot publish data, at ./src/rmw_publish.cpp:62, at ./src/rcl/publisher.c:250
[odrive_can_node-14] terminate called after throwing an instance of 'rclcpp::exceptions::RCLError'
[odrive_can_node-14]   what():  failed to publish message: cannot publish data, at ./src/rmw_publish.cpp:62, at ./src/rcl/publisher.c:250
[odrive_can_node-13] terminate called after throwing an instance of 'rclcpp::exceptions::RCLError'
[odrive_can_node-13]   what():  failed to publish message: cannot publish data, at ./src/rmw_publish.cpp:62, at ./src/rcl/publisher.c:250
[ERROR] [odrive_can_node-4]: process has died [pid 10910, exit code -6, cmd '/home/robot/rover_ws/install/odrive_can/lib/odrive_can/odrive_can_node --ros-args -r __node:=odrive_node -r __ns:=/front_left_top_suspension --params-file /tmp/launch_params_y18cv70k'].
[odrive_can_node-9] terminate called after throwing an instance of 'rclcpp::exceptions::RCLError'
[odrive_can_node-9]   what():  failed to publish message: cannot publish data, at ./src/rmw_publish.cpp:62, at ./src/rcl/publisher.c:250
[odrive_can_node-11] terminate called after throwing an instance of 'rclcpp::exceptions::RCLError'
[odrive_can_node-11]   what():  failed to publish message: cannot publish data, at ./src/rmw_publish.cpp:62, at ./src/rcl/publisher.c:250
[odrive_can_node-6] terminate called after throwing an instance of 'rclcpp::exceptions::RCLError'
[odrive_can_node-6]   what():  failed to publish message: cannot publish data, at ./src/rmw_publish.cpp:62, at ./src/rcl/publisher.c:250
[ERROR] [odrive_can_node-12]: process has died [pid 10969, exit code -6, cmd '/home/robot/rover_ws/install/odrive_can/lib/odrive_can/odrive_can_node --ros-args -r __node:=odrive_node -r __ns:=/front_left_drive --params-file /tmp/launch_params_8e8mtirb'].
[odrive_can_node-10] terminate called after throwing an instance of 'rclcpp::exceptions::RCLError'
[odrive_can_node-10]   what():  failed to publish message: cannot publish data, at ./src/rmw_publish.cpp:62, at ./src/rcl/publisher.c:250
[ERROR] [odrive_can_node-14]: process has died [pid 10996, exit code -6, cmd '/home/robot/rover_ws/install/odrive_can/lib/odrive_can/odrive_can_node --ros-args -r __node:=odrive_node -r __ns:=/back_left_drive --params-file /tmp/launch_params_6wul9eyn'].
[ERROR] [odrive_can_node-8]: process has died [pid 10941, exit code -6, cmd '/home/robot/rover_ws/install/odrive_can/lib/odrive_can/odrive_can_node --ros-args -r __node:=odrive_node -r __ns:=/front_left_steer --params-file /tmp/launch_params_x3pxaz92'].
[ERROR] [odrive_can_node-13]: process has died [pid 10988, exit code -6, cmd '/home/robot/rover_ws/install/odrive_can/lib/odrive_can/odrive_can_node --ros-args -r __node:=odrive_node -r __ns:=/front_right_drive --params-file /tmp/launch_params_uz9t4f5l'].
[ERROR] [odrive_can_node-9]: process has died [pid 10946, exit code -6, cmd '/home/robot/rover_ws/install/odrive_can/lib/odrive_can/odrive_can_node --ros-args -r __node:=odrive_node -r __ns:=/front_right_steer --params-file /tmp/launch_params__d3r9cyh'].
[ERROR] [odrive_can_node-5]: process has died [pid 10912, exit code -6, cmd '/home/robot/rover_ws/install/odrive_can/lib/odrive_can/odrive_can_node --ros-args -r __node:=odrive_node -r __ns:=/front_right_top_suspension --params-file /tmp/launch_params_sac1ravv'].
[ERROR] [odrive_can_node-10]: process has died [pid 10953, exit code -6, cmd '/home/robot/rover_ws/install/odrive_can/lib/odrive_can/odrive_can_node --ros-args -r __node:=odrive_node -r __ns:=/back_left_steer --params-f

If I check journalctl I see messages like this.

May 24 00:47:12 rev systemd[1]: Started Session 6 of User robot.
May 24 00:47:12 rev systemd-logind[756]: New session 6 of user robot.
May 24 00:47:35 rev kernel: Pid 4429(odrive_can_node) over core_pipe_limit
May 24 00:47:35 rev kernel: Skipping core dump
May 24 00:49:38 rev kernel: perf: interrupt took too long (3936 > 3935), lowering kernel.perf_event_max_sample_rate to 50750
May 24 00:55:03 rev kernel: Pid 6915(odrive_can_node) over core_pipe_limit
May 24 00:55:03 rev kernel: Skipping core dump
May 24 00:55:03 rev kernel: Pid 6979(odrive_can_node) over core_pipe_limit
May 24 00:55:03 rev kernel: Skipping core dump
May 24 00:55:14 rev systemd[1]: Starting Daily apt upgrade and clean activities...
May 24 00:55:26 rev systemd[1]: apt-daily-upgrade.service: Deactivated successfully.
May 24 00:55:26 rev systemd[1]: Finished Daily apt upgrade and clean activities.
May 24 00:55:26 rev systemd[1]: apt-daily-upgrade.service: Consumed 11.824s CPU time.
May 24 01:01:11 rev kernel: perf: interrupt took too long (4927 > 4920), lowering kernel.perf_event_max_sample_rate to 40500
May 24 01:02:28 rev sshd[2872]: pam_unix(sshd:session): session closed for user robot
May 24 01:02:28 rev systemd[1]: session-5.scope: Deactivated successfully.
May 24 01:02:28 rev systemd-logind[756]: Session 5 logged out. Waiting for processes to exit.
May 24 01:02:28 rev systemd-logind[756]: Removed session 5.
May 24 01:14:19 rev kernel: Pid 9356(odrive_can_node) over core_pipe_limit
May 24 01:14:19 rev kernel: Skipping core dump
May 24 01:14:19 rev kernel: Pid 9350(odrive_can_node) over core_pipe_limit
May 24 01:14:19 rev kernel: Skipping core dump
May 24 01:16:23 rev kernel: Pid 12765(odrive_can_node) over core_pipe_limit
May 24 01:16:23 rev kernel: Skipping core dump
May 24 01:16:23 rev kernel: Pid 12688(odrive_can_node) over core_pipe_limit
May 24 01:16:23 rev kernel: Skipping core dump
May 24 01:17:01 rev CRON[14456]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)

I've had this issue on both ODrive firmware versions 0.6.11-1 and 0.6.12. But this doesn't seem firmware related at all.

At first I thought maybe the ROS middleware was being overwhelmed with messages but that doesn't seem to be the issue after a lot of testing.

Please let me know what other information I need to share so that this issue can be solved quickly.

Thank you!

[samuelsadok]

At first I thought maybe the ROS middleware was being overwhelmed with messages

This is also what Claude suggests when I point it at this issue. Assuming Claude's axioms about how ROS works are correct, the explanation seems very plausible to me: That with rclcpp::KeepAll, the queue falls behind (grows indefinitely) as soon as any active subscriber is too slow.

but that doesn't seem to be the issue after a lot of testing.

What tests did you do?

Either way, it could be worth to give Claude's suggested fix a shot, which is to set a queue limit on the publishers:

rclcpp::QoS publisher_qos(rclcpp::KeepLast(10));
ctrl_publisher_ = rclcpp::Node::create_publisher<ControllerStatus>("controller_status", publisher_qos);
odrv_publisher_ = rclcpp::Node::create_publisher<ODriveStatus>("odrive_status", publisher_qos);

Let me know if that happens to fix the crash.