Unknown downtime reason

Hey together,

today I woke up and got a ton of notifications from my grafana alert, that my node was down at night.
As you can see, there was a problem from ~11:00pm until ~04:30am.

My Internet-connection is up since December 2019. I have a static IP-Address. Server was up and running seamlessly.

2020-01-20T22:47:27.213Z INFO piecestore download started {"Piece ID": "YSJTXRWHSPISDDZX4P3SP5KLNEU7HDNNT6DJXSLVFA26EA5NXCDQ", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"} 2020-01-20T22:47:27.879Z INFO piecestore upload started {"Piece ID": "NTEVMBFZR43I3GO2TQZ6W6A2YF6FCD3ZP6A2SREUETABQ6BLFFKA", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "PUT"} 2020-01-20T22:47:29.000Z INFO piecestore uploaded {"Piece ID": "NTEVMBFZR43I3GO2TQZ6W6A2YF6FCD3ZP6A2SREUETABQ6BLFFKA", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "PUT"} 2020-01-20T22:47:29.295Z INFO piecestore upload started {"Piece ID": "ZZBMGUU6QBXJXWWEQYZT3YWXVM7KYR26G2EQDGXBZ76VBF4A7OYA", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT"} 2020-01-20T22:47:30.008Z INFO piecestore upload started {"Piece ID": "TKK2RM7P7SD5JJGLYFPWMRVQTUKGEV5EY3RRZSSOSKT2HQLDTOOQ", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "PUT"} 2020-01-20T22:47:30.187Z INFO piecestore uploaded {"Piece ID": "TKK2RM7P7SD5JJGLYFPWMRVQTUKGEV5EY3RRZSSOSKT2HQLDTOOQ", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "PUT"} 2020-01-20T22:48:31.139Z INFO piecestore upload failed {"Piece ID": "ZZBMGUU6QBXJXWWEQYZT3YWXVM7KYR26G2EQDGXBZ76VBF4A7OYA", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT", "error": "piecestore protocol: context canceled", "errorVerbose": "piecestore protocol: context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:389\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:268\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:1066\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:175\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:153\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:114\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:147\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"} 2020-01-20T22:48:37.795Z INFO piecestore download failed {"Piece ID": "YSJTXRWHSPISDDZX4P3SP5KLNEU7HDNNT6DJXSLVFA26EA5NXCDQ", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET", "error": "piecestore: piecestore protocol: write tcp 172.17.0.4:28967->100.70.131.26:37302: write: connection timed out", "errorVerbose": "piecestore: piecestore protocol: write tcp 172.17.0.4:28967->100.70.131.26:37302: write: connection timed out\n\tstorj.io/drpc/drpcstream.(*Stream).pollWrite:189\n\tstorj.io/drpc/drpcwire.SplitN:25\n\tstorj.io/drpc/drpcstream.(*Stream).RawWrite:233\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:266\n\tstorj.io/common/pb.(*drpcPiecestoreDownloadStream).Send:1168\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload.func3:643\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"} 2020-01-20T22:59:02.872Z INFO version running on version v0.29.3 2020-01-20T23:14:02.933Z INFO version running on version v0.29.3 2020-01-20T23:29:02.388Z INFO bandwidth Performing bandwidth usage rollups 2020-01-20T23:29:02.890Z INFO version running on version v0.29.3 2020-01-20T23:29:28.280Z INFO orders.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6 sending {"count": 57} 2020-01-20T23:29:28.280Z INFO orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S sending {"count": 40} 2020-01-20T23:29:28.280Z INFO orders.118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW sending {"count": 101} 2020-01-20T23:29:28.280Z INFO orders.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs sending {"count": 66} 2020-01-20T23:29:28.624Z INFO orders.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs finished 2020-01-20T23:29:28.868Z INFO orders.118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW finished 2020-01-20T23:29:28.896Z INFO orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S finished 2020-01-20T23:29:29.747Z INFO orders.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6 finished 2020-01-20T23:44:02.869Z INFO version running on version v0.29.3 2020-01-20T23:59:02.877Z INFO version running on version v0.29.3 2020-01-21T00:14:02.911Z INFO version running on version v0.29.3 2020-01-21T00:29:02.388Z INFO bandwidth Performing bandwidth usage rollups 2020-01-21T00:29:02.905Z INFO version running on version v0.29.3 2020-01-21T00:44:02.885Z INFO version running on version v0.29.3 2020-01-21T00:59:02.886Z INFO version running on version v0.29.3 2020-01-21T01:14:02.886Z INFO version running on version v0.29.3 2020-01-21T01:29:02.388Z INFO bandwidth Performing bandwidth usage rollups 2020-01-21T01:29:02.876Z INFO version running on version v0.29.3 2020-01-21T01:44:02.904Z INFO version running on version v0.29.3 2020-01-21T01:59:02.871Z INFO version running on version v0.29.3 2020-01-21T02:14:02.877Z INFO version running on version v0.29.3 2020-01-21T02:29:02.388Z INFO bandwidth Performing bandwidth usage rollups 2020-01-21T02:29:02.892Z INFO version running on version v0.29.3 2020-01-21T02:44:02.888Z INFO version running on version v0.29.3 2020-01-21T02:59:02.897Z INFO version running on version v0.29.3 2020-01-21T03:14:02.887Z INFO version running on version v0.29.3 2020-01-21T03:29:02.388Z INFO bandwidth Performing bandwidth usage rollups 2020-01-21T03:29:02.910Z INFO version running on version v0.29.3 2020-01-21T03:44:02.862Z INFO version running on version v0.29.3 2020-01-21T03:59:02.888Z INFO version running on version v0.29.3 2020-01-21T04:14:02.891Z INFO version running on version v0.29.3 2020-01-21T04:15:21.079Z INFO trust Scheduling next refresh {"after": "3h15m8.631613334s"} 2020-01-21T04:29:02.388Z INFO bandwidth Performing bandwidth usage rollups 2020-01-21T04:29:02.903Z INFO version running on version v0.29.3 2020-01-21T04:29:09.553Z INFO piecestore download started {"Piece ID": "RQ75BGWFXJG2EVD2GQETFXR3XWUOKSPPS67FO3CUOBX5YYK7YWLQ", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"} 2020-01-21T04:29:09.750Z INFO piecestore upload started {"Piece ID": "WOWWXKLIIZB2NNK572H7Q3GHT5XATF7FJVNTOSDF3IWSWAXMECFA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"} 2020-01-21T04:29:09.933Z INFO piecestore uploaded {"Piece ID": "WOWWXKLIIZB2NNK572H7Q3GHT5XATF7FJVNTOSDF3IWSWAXMECFA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"} 2020-01-21T04:29:11.672Z INFO piecestore download started {"Piece ID": "HALSVNRYDVWQZRI7WBIBGFR7BSGYOPFXGT4H7FAY5R2FABOZEMLQ", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"} 2020-01-21T04:29:19.022Z INFO piecestore upload started {"Piece ID": "YHZDFTNRL47XQL3AITYWBJJAITVKNNZJKZ6MDKZAXJ5MMQBGRO5Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT"} 2020-01-21T04:29:19.943Z INFO piecestore uploaded {"Piece ID": "YHZDFTNRL47XQL3AITYWBJJAITVKNNZJKZ6MDKZAXJ5MMQBGRO5Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT"} 2020-01-21T04:29:21.204Z INFO piecestore upload started {"Piece ID": "AWVSHEB4WBTAFFMZZ5P535LYFVKHLIVGVUG7VKNEWMSVVHHDFLHQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"} 2020-01-21T04:29:21.362Z INFO piecestore uploaded {"Piece ID": "AWVSHEB4WBTAFFMZZ5P535LYFVKHLIVGVUG7VKNEWMSVVHHDFLHQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"} 2020-01-21T04:29:24.378Z INFO piecestore downloaded {"Piece ID": "RQ75BGWFXJG2EVD2GQETFXR3XWUOKSPPS67FO3CUOBX5YYK7YWLQ", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"} 2020-01-21T04:29:29.470Z INFO piecestore upload started {"Piece ID": "SXHDULD3YPRUJTBQVCJFSKXKUX3D6RE222TEZIQAN2EAZHJCDBNA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT"} 2020-01-21T04:29:29.562Z INFO piecestore downloaded {"Piece ID": "HALSVNRYDVWQZRI7WBIBGFR7BSGYOPFXGT4H7FAY5R2FABOZEMLQ", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"} 2020-01-21T04:29:29.624Z INFO piecestore uploaded {"Piece ID": "SXHDULD3YPRUJTBQVCJFSKXKUX3D6RE222TEZIQAN2EAZHJCDBNA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT"}

image


I dropped from my solid 100% to ~99.6% uptime at all 4 satellites.
Does anybody know what happend?

Do you have a uptimerobot.com monitoring?
It could say is your port was open or not.

1 Like

unfortunatly not yet. Will activate right now.
Thank you.