Logs: how to send relevant log messages to a Discord web hook

Node Operators
, , ,
21

yep, seems to be about that for each 24 hour period for the last week atleast
but can’t exclude that its not my scripts failing, but i haven’t have problems with these particular ones, but i haven’t fully checked them… just seemed weird

maybe it is just the zfs compression

NAME               PROPERTY              VALUE                  SOURCE
bitpool/storjlogs  type                  filesystem             -
bitpool/storjlogs  creation              Thu Jun  3 15:46 2021  -
bitpool/storjlogs  used                  3.72G                  -
bitpool/storjlogs  available             5.88T                  -
bitpool/storjlogs  referenced            3.72G                  -
bitpool/storjlogs  compressratio         4.67x                  -
bitpool/storjlogs  mounted               yes                    -

seems like a no the zfs compression is much higher than just 2x
which leads me back to … weird.

scanned through my logs and i can’t see any gaps over a 24 hour period

22

Why does anyone need so many log file data in general? That’s weird to me. :wink:

1 Like
23

Currently I am using these 3 calls to count the audit, error and fatal cases, and alerting via Discord. That’s really working amazingly, I’ll share what I’ve done later some day.

tmp_fatal_errors=$(docker logs storagenode --since "24h" 2>&1 | grep FATAL -c)
tmp_audits_failed=$(docker logs storagenode --since "24h" 2>&1 | grep "GET_AUDIT" | grep "failed" -c)
tmp_rest_of_errors=$(docker logs storagenode --since "24h" 2>&1 | grep ERROR | grep -v -e "collector" -e "piecestore" -c)

Technical question meanwhile: I want to group and count the subsequent causes of the error messages in order to include it in the notification.

I want to immediately see, what’s going wrong and how fast I need to respond. E.g. “ERROR: satellite ping timeout (#1). Is there an easy way to do so? (Linux bash + docker; # = count)

24

I would recommend to update to grep -E "GET_AUDIT|GET_REPAIR", because they both affects audit score.

1 Like
25

published my shell script on GitHub, details in a separate post:

cc @SGC

1 Like