Open
Description
Describe the bug
Service has been working for several hours receiving and writing data, then the node went dead.
To Reproduce (Required)
Not sure at this point
Expected behavior (Required)
Node should not go dead
What's your LinDB version? (Required)
# /opt/lind version
LinDB: v0.3.0, BuildDate: 2023-08-29T10:41:33+0000
GOOS="linux"
GOARCH="amd64"
GOVERSION="go1.19.12"
Screenshots
Data writes stopped at around 2h00 (and restarted when I restarted the lindb service:
Logs around that time:
2024-04-18 21:15:37.077 ERROR rpc/write_stream.go:155 [ RPC] [WriteStream]: write stream context is canceled {"target": "155.138.132.24:2891", "error": "context canceled"}
2024-04-18 22:15:37.077 ERROR rpc/write_stream.go:164 [ RPC] [WriteStream]: receive error from write stream {"target": "155.138.132.24:2891", "error": "rpc error: code = Canceled desc = context canceled"}
2024-04-18 22:15:37.078 ERROR rpc/write_stream.go:155 [ RPC] [WriteStream]: write stream context is canceled {"target": "155.138.132.24:2891", "error": "context canceled"}
2024-04-18 23:15:37.074 ERROR rpc/write_stream.go:164 [ RPC] [WriteStream]: receive error from write stream {"target": "155.138.132.24:2891", "error": "rpc error: code = Canceled desc = context canceled"}
2024-04-18 23:15:37.075 ERROR rpc/write_stream.go:155 [ RPC] [WriteStream]: write stream context is canceled {"target": "155.138.132.24:2891", "error": "context canceled"}
2024-04-19 00:15:37.074 ERROR rpc/write_stream.go:164 [ RPC] [WriteStream]: receive error from write stream {"target": "155.138.132.24:2891", "error": "rpc error: code = Canceled desc = context canceled"}
2024-04-19 00:15:37.075 ERROR rpc/write_stream.go:155 [ RPC] [WriteStream]: write stream context is canceled {"target": "155.138.132.24:2891", "error": "context canceled"}
2024-04-19 01:15:37.076 ERROR rpc/write_stream.go:164 [ RPC] [WriteStream]: receive error from write stream {"target": "155.138.132.24:2891", "error": "rpc error: code = Canceled desc = context canceled"}
2024-04-19 01:15:37.078 ERROR rpc/write_stream.go:155 [ RPC] [WriteStream]: write stream context is canceled {"target": "155.138.132.24:2891", "error": "context canceled"}
2024-04-19 02:02:13.134 ERROR monitoring/native_pusher.go:150 [ Monitoring] [Pusher]: failed to post request {"error": "Put \"http://127.0.0.1:9000/api/v1/write?db=_internal\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
2024-04-19 02:02:12.929 ERROR monitoring/native_pusher.go:150 [ Monitoring] [Pusher]: failed to post request {"error": "Put \"http://127.0.0.1:9000/api/v1/write?db=_internal\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
2024-04-19 02:02:18.706 ERROR monitoring/native_pusher.go:150 [ Monitoring] [Pusher]: failed to post request {"error": "Put \"http://127.0.0.1:9000/api/v1/write?db=_internal\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
2024-04-19 02:02:24.537 ERROR monitoring/native_pusher.go:150 [ Monitoring] [Pusher]: failed to post request {"error": "Put \"http://127.0.0.1:9000/api/v1/write?db=_internal\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
2024-04-19 02:02:33.117 ERROR monitoring/native_pusher.go:150 [ Monitoring] [Pusher]: failed to post request {"error": "Put \"http://127.0.0.1:9000/api/v1/write?db=_internal\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
2024-04-19 02:02:38.023 ERROR monitoring/native_pusher.go:150 [ Monitoring] [Pusher]: failed to post request {"error": "Put \"http://127.0.0.1:9000/api/v1/write?db=_internal\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
2024-04-19 02:02:45.865 ERROR monitoring/native_pusher.go:150 [ Monitoring] [Pusher]: failed to post request {"error": "Put \"http://127.0.0.1:9000/api/v1/write?db=_internal\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
2024-04-19 02:02:58.118 ERROR monitoring/native_pusher.go:150 [ Monitoring] [Pusher]: failed to post request {"error": "Put \"http://127.0.0.1:9000/api/v1/write?db=_internal\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
2024-04-19 02:03:14.135 ERROR query/task_handler.go:86 [ Query] [TaskHandler]: task server stream error {"error": "EOF"}
2024-04-19 02:03:14.200 ERROR rpc/task_transport.go:213 [ RPC] [TaskClient]: receive task error from stream {"error": "rpc error: code = Unknown desc = EOF"}
2024-04-19 02:03:14.420 ERROR rpc/write_stream.go:164 [ RPC] [WriteStream]: receive error from write stream {"target": "155.138.132.24:2891", "error": "rpc error: code = Canceled desc = context canceled"}
2024-04-19 02:03:14.420 ERROR rpc/write_stream.go:155 [ RPC] [WriteStream]: write stream context is canceled {"target": "155.138.132.24:2891", "error": "context canceled"}
2024-04-19 02:03:14.420 ERROR rpc/write_stream.go:164 [ RPC] [WriteStream]: receive error from write stream {"target": "155.138.132.24:2891", "error": "rpc error: code = Canceled desc = context canceled"}
2024-04-19 02:03:14.420 ERROR rpc/write_stream.go:155 [ RPC] [WriteStream]: write stream context is canceled {"target": "155.138.132.24:2891", "error": "context canceled"}
2024-04-19 02:03:14.430 ERROR query/task_handler.go:86 [ Query] [TaskHandler]: task server stream error {"error": "EOF"}
2024-04-19 02:03:14.439 ERROR rpc/task_transport.go:213 [ RPC] [TaskClient]: receive task error from stream {"error": "rpc error: code = Unknown desc = EOF"}
2024-04-19 02:03:14.573 ERROR rpc/write_stream.go:164 [ RPC] [WriteStream]: receive error from write stream {"target": "155.138.132.24:2891", "error": "rpc error: code = Canceled desc = context canceled"}
2024-04-19 02:03:14.573 ERROR rpc/write_stream.go:155 [ RPC] [WriteStream]: write stream context is canceled {"target": "155.138.132.24:2891", "error": "context canceled"}
2024-04-19 02:03:14.573 ERROR rpc/write_stream.go:164 [ RPC] [WriteStream]: receive error from write stream {"target": "155.138.132.24:2891", "error": "rpc error: code = Canceled desc = context canceled"}
2024-04-19 02:03:14.573 ERROR rpc/write_stream.go:155 [ RPC] [WriteStream]: write stream context is canceled {"target": "155.138.132.24:2891", "error": "context canceled"}
2024-04-19 02:03:14.573 ERROR rpc/write_stream.go:164 [ RPC] [WriteStream]: receive error from write stream {"target": "155.138.132.24:2891", "error": "rpc error: code = Canceled desc = context canceled"}
Additional context
Pretty much a vanilla install. I simply untared the binaries in /opt installed the unit file, started the service and create a DB from the WebUI.
Then I have a Go client that listens for sensor data and commits it to the LinDB database.