Storagenode was working fine for straight 2 months, didn’t bother touching the node, then this appeared. Here are the logs
FATAL Unrecoverable error {“error”: “rpc error: code = Unavailable desc = transport is closing”}
I tried running docker exec -it storagenode /app/dashboard.sh, the cmd would freeze.
2019-10-11T08:55:25.916Z INFO piecestore downloaded {“Piece ID”: “6DK4OMYPQZOSJ6D5LS6HZF5HPMETAE6HLTBL45GN6IMD5BQDK4IQ”, “SatelliteID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET_AUDIT”}
2019-10-11T08:55:48.122Z ERROR piecestore upload rejected, too many requests {“live requests”: 31}
2019-10-11T08:58:24.071Z ERROR piecestore upload rejected, too many requests {“live requests”: 59}
2019-10-11T08:58:29.401Z INFO version running on version v0.23.2
2019-10-11T09:00:51.647Z ERROR piecestore upload rejected, too many requests {“live requests”: 87}
2019-10-11T09:01:03.937Z INFO piecestore download started {“Piece ID”: “SHYFVKGHA4VTP4HICWRGHP3QDQFBNGJHTNM2SGHB5PDUC4PQOOLQ”, “SatelliteID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “Action”: “GET_AUDIT”}
2019-10-11T09:01:08.215Z ERROR piecestore upload rejected, too many requests {“live requests”: 92}
2019-10-11T09:01:35.008Z ERROR piecestore upload rejected, too many requests {“live requests”: 95}
2019-10-11T09:01:51.805Z ERROR piecestore upload rejected, too many requests {“live requests”: 98}
2019-10-11T09:02:03.980Z INFO piecestore download started {“Piece ID”: “XBXG5QHGGGHFYBL2YKXVSIRPMNKKHF7FF3V7MRI223A3SNCT5GWA”, “SatelliteID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET_AUDIT”}
2019-10-11T09:02:30.077Z ERROR piecestore upload rejected, too many requests {“live requests”: 104}
2019-10-11T09:03:33.806Z ERROR piecestore upload rejected, too many requests {“live requests”: 113}
2019-10-11T09:03:46.761Z ERROR piecestore upload rejected, too many requests {“live requests”: 115}
2019-10-11T09:04:06.264Z ERROR piecestore upload rejected, too many requests {“live requests”: 115}
2019-10-11T09:05:09.567Z ERROR piecestore upload rejected, too many requests {“live requests”: 128}
2019-10-11T09:05:58.097Z ERROR piecestore upload rejected, too many requests {“live requests”: 137}
2019-10-11T09:08:11.891Z INFO piecestore download failed {“Piece ID”: “SHYFVKGHA4VTP4HICWRGHP3QDQFBNGJHTNM2SGHB5PDUC4PQOOLQ”, “SatelliteID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “Action”: “GET_AUDIT”, “error”: “rpc error: code = Internal desc = pieces error: filestore error: unable to open “config/storage/blobs/abforhuxbzyd35blusvrifvdwmfx4hmocsva4vmpp3rgqaaaaaaa/sh/yfvkgha4vtp4hicwrghp3qdqfbngjhtnm2sghb5pduc4pqoolq.sj1”: open config/storage/blobs/abforhuxbzyd35blusvrifvdwmfx4hmocsva4vmpp3rgqaaaaaaa/sh/yfvkgha4vtp4hicwrghp3qdqfbngjhtnm2sghb5pduc4pqoolq.sj1: resource temporarily unavailable”}
2019-10-11T09:08:11.892Z ERROR server gRPC stream error response {“error”: “rpc error: code = Internal desc = pieces error: filestore error: unable to open “config/storage/blobs/abforhuxbzyd35blusvrifvdwmfx4hmocsva4vmpp3rgqaaaaaaa/sh/yfvkgha4vtp4hicwrghp3qdqfbngjhtnm2sghb5pduc4pqoolq.sj1”: open config/storage/blobs/abforhuxbzyd35blusvrifvdwmfx4hmocsva4vmpp3rgqaaaaaaa/sh/yfvkgha4vtp4hicwrghp3qdqfbngjhtnm2sghb5pduc4pqoolq.sj1: resource temporarily unavailable”}
2019-10-11T09:08:11.893Z INFO piecestore download failed {“Piece ID”: “XBXG5QHGGGHFYBL2YKXVSIRPMNKKHF7FF3V7MRI223A3SNCT5GWA”, “SatelliteID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET_AUDIT”, “error”: “rpc error: code = Internal desc = pieces error: filestore error: unable to open “config/storage/blobs/v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa/xb/xg5qhggghfybl2ykxvsirpmnkkhf7ff3v7mri223a3snct5gwa.sj1”: open config/storage/blobs/v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa/xb/xg5qhggghfybl2ykxvsirpmnkkhf7ff3v7mri223a3snct5gwa.sj1: resource temporarily unavailable”}
2019-10-11T09:08:11.893Z ERROR server gRPC stream error response {“error”: “rpc error: code = Internal desc = pieces error: filestore error: unable to open “config/storage/blobs/v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa/xb/xg5qhggghfybl2ykxvsirpmnkkhf7ff3v7mri223a3snct5gwa.sj1”: open config/storage/blobs/v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa/xb/xg5qhggghfybl2ykxvsirpmnkkhf7ff3v7mri223a3snct5gwa.sj1: resource temporarily unavailable”}