Node is continuously reset and reconnecting to hive

General
1

Hello @dillon

Have a great day.

My node is running on Ubuntu and getting points as normal. But I observe the connection continuously reset in the console.

I have other nodes also running on Ubuntu with the same versions of OS & apps (different lines, public IP) but they don’t have any issue, just ping & received.

I checked the network connectivity but no packet drop or CRC issue, and the firewall also fine (Cisco devices). :+1:

Kindly advice if this is a normal or not and what I need to do with this.

aios-kernel_console
aios-kernel_console1878×928 33.1 KB

Johnny.

1 Like
2

And below is the aios_kernel log FYR.

Reset state after reconnection
[2024-12-20 03:39:52] INFO [aios_kernel::logger] Successfully reregistered models.
[2024-12-20 03:39:52] INFO [aios_kernel::logger] Reconnecting to hive…
[2024-12-20 03:39:52] INFO [aios_kernel::logger] Message handler spawned
[2024-12-20 03:39:52] INFO [aios_kernel::logger] Resetting current doing infreq
[2024-12-20 03:39:53] INFO [aios_kernel::logger] {“type”:“NEW_CLIENT_CONNECTED”,“data”:{“currentConnectionID”:“DEkJld3ZoAMCJdA=”,“newConnectionID”:“DEkJ_doLoAMCFGw=”}}
[2024-12-20 03:39:53] INFO [aios_kernel::logger] Connection Closed Normally
[2024-12-20 03:39:53] INFO [aios_kernel::logger] connection closed by message!
[2024-12-20 03:39:53] INFO [aios_kernel::logger] connection stream closed!
[2024-12-20 03:39:53] INFO [aios_kernel::logger] Reconnected
[2024-12-20 03:39:53] INFO [aios_kernel::logger] ws_stream set
[2024-12-20 03:39:53] INFO [aios_kernel::logger] ws_sink set
[2024-12-20 03:39:53] INFO [aios_kernel::logger] Set last pong at 2024-12-20 03:39:53.381635087 UTC
Reset state after reconnection
[2024-12-20 03:39:54] INFO [aios_kernel::logger] Successfully reregistered models.
[2024-12-20 03:39:54] INFO [aios_kernel::logger] Reconnecting to hive…
[2024-12-20 03:39:54] INFO [aios_kernel::logger] Message handler spawned
[2024-12-20 03:39:54] INFO [aios_kernel::logger] Resetting current doing infreq
[2024-12-20 03:39:59] INFO [aios_kernel::logger] {“type”:“NEW_CLIENT_CONNECTED”,“data”:{“currentConnectionID”:“DEkJ_doLoAMCFGw=”,“newConnectionID”:“DEkKZfFQoAMCINg=”}}
[2024-12-20 03:39:59] INFO [aios_kernel::logger] Connection Closed Normally
[2024-12-20 03:39:59] INFO [aios_kernel::logger] connection closed by message!
[2024-12-20 03:39:59] INFO [aios_kernel::logger] connection stream closed!
[2024-12-20 03:39:59] INFO [aios_kernel::logger] Reconnected
[2024-12-20 03:39:59] INFO [aios_kernel::logger] ws_stream set
[2024-12-20 03:39:59] INFO [aios_kernel::logger] ws_sink set
[2024-12-20 03:39:59] INFO [aios_kernel::logger] Set last pong at 2024-12-20 03:39:59.167964772 UTC
Reset state after reconnection
[2024-12-20 03:40:00] INFO [aios_kernel::logger] Successfully reregistered models.
[2024-12-20 03:40:00] INFO [aios_kernel::logger] Reconnecting to hive…
[2024-12-20 03:40:00] INFO [aios_kernel::logger] Message handler spawned
[2024-12-20 03:40:00] INFO [aios_kernel::logger] Resetting current doing infreq
[2024-12-20 03:40:01] INFO [aios_kernel::logger] {“type”:“NEW_CLIENT_CONNECTED”,“data”:{“currentConnectionID”:“DEkKZfFQoAMCINg=”,“newConnectionID”:“DEkKycAOIAMCIUw=”}}
[2024-12-20 03:40:01] INFO [aios_kernel::logger] Connection Closed Normally
[2024-12-20 03:40:01] INFO [aios_kernel::logger] connection closed by message!
[2024-12-20 03:40:01] INFO [aios_kernel::logger] connection stream closed!
[2024-12-20 03:40:01] INFO [aios_kernel::logger] Reconnected
[2024-12-20 03:40:01] INFO [aios_kernel::logger] ws_stream set
[2024-12-20 03:40:01] INFO [aios_kernel::logger] ws_sink set
[2024-12-20 03:40:01] INFO [aios_kernel::logger] Set last pong at 2024-12-20 03:40:01.684771326 UTC
Reset state after reconnection
[2024-12-20 03:40:17] INFO [aios_kernel::logger] Finished with hive inference update
[2024-12-20 03:40:29] WARN [aios_kernel::logger] Reregistering on reconnect failed: ResponseError(ResponseContent { status: 503, content: “{"message":"Service Unavailable"}”, entity: Some(UnknownValue(Object {“message”: String(“Service Unavailable”)})) })
[2024-12-20 03:40:29] INFO [aios_kernel::logger] Reconnecting to hive…
[2024-12-20 03:40:29] INFO [aios_kernel::logger] Message handler spawned
[2024-12-20 03:40:29] INFO [aios_kernel::logger] Resetting current doing infreq
[2024-12-20 03:40:33] INFO [aios_kernel::logger] {“type”:“NEW_CLIENT_CONNECTED”,“data”:{“currentConnectionID”:“DEkKycAOIAMCIUw=”,“newConnectionID”:“DEkPzdtuIAMCJ9Q=”}}
[2024-12-20 03:40:33] INFO [aios_kernel::logger] Connection Closed Normally
[2024-12-20 03:40:33] INFO [aios_kernel::logger] connection closed by message!
[2024-12-20 03:40:33] INFO [aios_kernel::logger] connection stream closed!
[2024-12-20 03:40:33] INFO [aios_kernel::logger] Reconnected
[2024-12-20 03:40:33] INFO [aios_kernel::logger] ws_stream set
[2024-12-20 03:40:33] INFO [aios_kernel::logger] ws_sink set
[2024-12-20 03:40:33] INFO [aios_kernel::logger] Set last pong at 2024-12-20 03:40:33.879359747 UTC
Reset state after reconnection
[2024-12-20 03:40:34] INFO [aios_kernel::logger] Successfully reregistered models.
[2024-12-20 03:40:34] INFO [aios_kernel::logger] Reconnecting to hive…
[2024-12-20 03:40:34] INFO [aios_kernel::logger] Message handler spawned
[2024-12-20 03:40:34] INFO [aios_kernel::logger] Resetting current doing infreq
[2024-12-20 03:40:36] INFO [aios_kernel::logger] {“type”:“NEW_CLIENT_CONNECTED”,“data”:{“currentConnectionID”:“DEkPzdtuIAMCJ9Q=”,“newConnectionID”:“DEkQNcZ5oAMCL_A=”}}
[2024-12-20 03:40:36] INFO [aios_kernel::logger] Connection Closed Normally
[2024-12-20 03:40:36] INFO [aios_kernel::logger] connection closed by message!
[2024-12-20 03:40:36] INFO [aios_kernel::logger] connection stream closed!
[2024-12-20 03:40:36] INFO [aios_kernel::logger] Reconnected
[2024-12-20 03:40:36] INFO [aios_kernel::logger] ws_stream set
[2024-12-20 03:40:36] INFO [aios_kernel::logger] ws_sink set
[2024-12-20 03:40:36] INFO [aios_kernel::logger] Set last pong at 2024-12-20 03:40:36.282054085 UTC
Reset state after reconnection
[2024-12-20 03:40:37] INFO [aios_kernel::logger] Successfully reregistered models.
[2024-12-20 03:40:37] INFO [aios_kernel::logger] Reconnecting to hive…
[2024-12-20 03:40:37] INFO [aios_kernel::logger] Message handler spawned
[2024-12-20 03:40:37] INFO [aios_kernel::logger] Resetting current doing infreq
[2024-12-20 03:40:38] INFO [aios_kernel::logger] {“type”:“NEW_CLIENT_CONNECTED”,“data”:{“currentConnectionID”:“DEkQNcZ5oAMCL_A=”,“newConnectionID”:“DEkQmeA0oAMCFxg=”}}
[2024-12-20 03:40:38] INFO [aios_kernel::logger] Connection Closed Normally
[2024-12-20 03:40:38] INFO [aios_kernel::logger] connection closed by message!
[2024-12-20 03:40:38] INFO [aios_kernel::logger] connection stream closed!
[2024-12-20 03:40:38] INFO [aios_kernel::logger] Reconnected
[2024-12-20 03:40:38] INFO [aios_kernel::logger] ws_stream set
[2024-12-20 03:40:38] INFO [aios_kernel::logger] ws_sink set
[2024-12-20 03:40:38] INFO [aios_kernel::logger] Set last pong at 2024-12-20 03:40:38.657223659 UTC
Reset state after reconnection
[2024-12-20 03:40:39] INFO [aios_kernel::logger] Successfully reregistered models.
[2024-12-20 03:40:39] INFO [aios_kernel::logger] Reconnecting to hive…
[2024-12-20 03:40:39] INFO [aios_kernel::logger] Message handler spawned
[2024-12-20 03:40:39] INFO [aios_kernel::logger] Resetting current doing infreq
[2024-12-20 03:40:40] INFO [aios_kernel::logger] {“type”:“NEW_CLIENT_CONNECTED”,“data”:{“currentConnectionID”:“DEkQmeA0oAMCFxg=”,“newConnectionID”:“DEkRBdhGoAMCK6Q=”}}
[2024-12-20 03:40:40] INFO [aios_kernel::logger] Connection Closed Normally
[2024-12-20 03:40:40] INFO [aios_kernel::logger] connection closed by message!
[2024-12-20 03:40:40] INFO [aios_kernel::logger] connection stream closed!
[2024-12-20 03:40:41] INFO [aios_kernel::logger] Reconnected
[2024-12-20 03:40:41] INFO [aios_kernel::logger] ws_stream set
[2024-12-20 03:40:41] INFO [aios_kernel::logger] ws_sink set
[2024-12-20 03:40:41] INFO [aios_kernel::logger] Set last pong at 2024-12-20 03:40:41.028736964 UTC
Reset state after reconnection
[2024-12-20 03:40:42] INFO [aios_kernel::logger] Successfully reregistered models.
[2024-12-20 03:40:42] INFO [aios_kernel::logger] Reconnecting to hive…
[2024-12-20 03:40:42] INFO [aios_kernel::logger] Message handler spawned
[2024-12-20 03:40:42] INFO [aios_kernel::logger] Resetting current doing infreq
[2024-12-20 03:40:43] INFO [aios_kernel::logger] {“type”:“NEW_CLIENT_CONNECTED”,“data”:{“currentConnectionID”:“DEkRBdhGoAMCK6Q=”,“newConnectionID”:“DEkRbeC1oAMCKXA=”}}
[2024-12-20 03:40:43] INFO [aios_kernel::logger] Connection Closed Normally
[2024-12-20 03:40:43] INFO [aios_kernel::logger] connection closed by message!
[2024-12-20 03:40:43] INFO [aios_kernel::logger] connection stream closed!
[2024-12-20 03:40:43] INFO [aios_kernel::logger] Reconnected
[2024-12-20 03:40:43] INFO [aios_kernel::logger] ws_stream set
[2024-12-20 03:40:43] INFO [aios_kernel::logger] ws_sink set
[2024-12-20 03:40:43] INFO [aios_kernel::logger] Set last pong at 2024-12-20 03:40:43.539525614 UTC
Reset state after reconnection
[2024-12-20 03:40:44] INFO [aios_kernel::logger] Successfully reregistered models.
[2024-12-20 03:40:44] INFO [aios_kernel::logger] Reconnecting to hive…
[2024-12-20 03:40:44] INFO [aios_kernel::logger] Message handler spawned
[2024-12-20 03:40:44] INFO [aios_kernel::logger] Resetting current doing infreq
[2024-12-20 03:40:45] INFO [aios_kernel::logger] {“type”:“NEW_CLIENT_CONNECTED”,“data”:{“currentConnectionID”:“DEkRbeC1oAMCKXA=”,“newConnectionID”:“DEkR2ezwIAMCFdQ=”}}
[2024-12-20 03:40:45] INFO [aios_kernel::logger] Connection Closed Normally
[2024-12-20 03:40:45] INFO [aios_kernel::logger] connection closed by message!
[2024-12-20 03:40:45] INFO [aios_kernel::logger] connection stream closed!
[2024-12-20 03:40:46] INFO [aios_kernel::logger] Reconnected
[2024-12-20 03:40:46] INFO [aios_kernel::logger] ws_stream set
[2024-12-20 03:40:46] INFO [aios_kernel::logger] ws_sink set
[2024-12-20 03:40:46] INFO [aios_kernel::logger] Set last pong at 2024-12-20 03:40:46.305825287 UTC
Reset state after reconnection
[2024-12-20 03:40:47] INFO [aios_kernel::logger] Successfully reregistered models.
[2024-12-20 03:40:47] INFO [aios_kernel::logger] Reconnecting to hive…
[2024-12-20 03:40:47] INFO [aios_kernel::logger] Message handler spawned
[2024-12-20 03:40:47] INFO [aios_kernel::logger] Resetting current doing infreq
[2024-12-20 03:40:48] INFO [aios_kernel::logger] {“type”:“NEW_CLIENT_CONNECTED”,“data”:{“currentConnectionID”:“DEkR2ezwIAMCFdQ=”,“newConnectionID”:“DEkSVfphoAMCKJg=”}}
[2024-12-20 03:40:48] INFO [aios_kernel::logger] Connection Closed Normally
[2024-12-20 03:40:48] INFO [aios_kernel::logger] connection closed by message!
[2024-12-20 03:40:48] INFO [aios_kernel::logger] connection stream closed!
[2024-12-20 03:40:49] INFO [aios_kernel::logger] Reconnected
[2024-12-20 03:40:49] INFO [aios_kernel::logger] ws_stream set
[2024-12-20 03:40:49] INFO [aios_kernel::logger] ws_sink set
[2024-12-20 03:40:49] INFO [aios_kernel::logger] Set last pong at 2024-12-20 03:40:49.017941584 UTC
Reset state after reconnection
[2024-12-20 03:40:50] INFO [aios_kernel::logger] Successfully reregistered models.
[2024-12-20 03:40:50] INFO [aios_kernel::logger] Reconnecting to hive…
[2024-12-20 03:40:50] INFO [aios_kernel::logger] Message handler spawned
[2024-12-20 03:40:50] INFO [aios_kernel::logger] Resetting current doing infreq
[2024-12-20 03:40:51] INFO [aios_kernel::logger] {“type”:“NEW_CLIENT_CONNECTED”,“data”:{“currentConnectionID”:“DEkSVfphoAMCKJg=”,“newConnectionID”:“DEkSzd_CoAMCLrA=”}}
[2024-12-20 03:40:51] INFO [aios_kernel::logger] Connection Closed Normally
[2024-12-20 03:40:51] INFO [aios_kernel::logger] connection closed by message!
[2024-12-20 03:40:51] INFO [aios_kernel::logger] connection stream closed!
[2024-12-20 03:40:52] INFO [aios_kernel::logger] Reconnected
[2024-12-20 03:40:52] INFO [aios_kernel::logger] ws_stream set
[2024-12-20 03:40:52] INFO [aios_kernel::logger] ws_sink set
[2024-12-20 03:40:52] INFO [aios_kernel::logger] Set last pong at 2024-12-20 03:40:52.249152792 UTC
Reset state after reconnection
[2024-12-20 03:40:54] INFO [aios_kernel::logger] Successfully reregistered models.
[2024-12-20 03:40:54] INFO [aios_kernel::logger] Reconnecting to hive…
[2024-12-20 03:40:54] INFO [aios_kernel::logger] Message handler spawned
[2024-12-20 03:40:54] INFO [aios_kernel::logger] Resetting current doing infreq
[2024-12-20 03:40:55] INFO [aios_kernel::logger] {“type”:“NEW_CLIENT_CONNECTED”,“data”:{“currentConnectionID”:“DEkSzd_CoAMCLrA=”,“newConnectionID”:“DEkTdcUsIAMCKJw=”}}
[2024-12-20 03:40:55] INFO [aios_kernel::logger] Connection Closed Normally
[2024-12-20 03:40:55] INFO [aios_kernel::logger] connection closed by message!
[2024-12-20 03:40:55] INFO [aios_kernel::logger] connection stream closed!
[2024-12-20 03:40:55] INFO [aios_kernel::logger] Reconnected
[2024-12-20 03:40:55] INFO [aios_kernel::logger] ws_stream set
[2024-12-20 03:40:55] INFO [aios_kernel::logger] ws_sink set
[2024-12-20 03:40:55] INFO [aios_kernel::logger] Set last pong at 2024-12-20 03:40:55.801885730 UTC
Reset state after reconnection
[2024-12-20 03:40:57] INFO [aios_kernel::logger] Successfully reregistered models.
[2024-12-20 03:40:57] INFO [aios_kernel::logger] Reconnecting to hive…
[2024-12-20 03:40:57] INFO [aios_kernel::logger] Message handler spawned
[2024-12-20 03:40:57] INFO [aios_kernel::logger] Resetting current doing infreq
johnny@T7910:~$
johnny@T7910:~$

3

Hello all,

FYI. I figured out the root cause and fixed it.

For interested technician: WSL conflicts with VMware and the solution is hypervisorlaunchtype.

4

Finally, it’s network intermittence because of Hive availability.

5

i have the same issue with VMware installed on my host… on the hypervisorlaunchtype do i need to enable or disable it?

6

I set it to Auto and start WSL first. Once aios-cli connected and registered, power VM on.
Good luck.