Changelog v1.44.1

littleskunk · November 30, 2021, 3:47pm

For Storagenodes

Storagenode Updater for FreeBSD
Included in this github release is the storagenode updater for FreeBSD. Shoutout to @svanhoutte for testing the binary.

Email Alerts with Grafana
With the new storage node version, you can now set up additional email alerts. More details here: [Tech Preview] Email alerts with Grafana and Prometheus

For Customers

Geofencing
We can now restrict empty buckets to a specific geolocation. An upload to that bucket will be stored only on storage nodes in that region. Repair and graceful exit will also keep the pieces in that region. Shoutout to @Roberto for participating in our test run.

Satellite UI New Object Flow
The new object flow is enabled on all production satellites. If you open the object browser you will see the list of buckets first. When selecting a bucket you have to enter your encryption key.
Shoutout to @jammerdan and @nerdatwork for giving us great feedback regarding the latest satellite UI changes.

Satellite UI Force Delete Bucket
A nice little side effect of the object flow is that the object browser can now delete buckets with all files in it. If you have forgotten your password you can delete the bucket and start with a new one.

Storgeez · November 30, 2021, 4:23pm

1.43.1 binary uploaded under 1.44.1.

stefanbenten · November 30, 2021, 6:00pm

This is fixed as a couple of minutes ago.

Krystof · November 30, 2021, 6:03pm

Minimum version is now 1.43.1?
My watchtower dont update my node yet… Its not 72 hours from publish latest 1.43.1 on dockerhub…

nerdatwork · December 3, 2021, 8:19am

I have tested this by uploading a file then deleting the bucket without entering passphrase. Here are the results of piece activity on testnode.

4 pieces uploaded ; 0 deleted
2 pieces uploaded; 1 deleted
1 piece uploaded ; 1 deleted
5 pieces uploaded; 0 deleted.

2021-12-03T03:33:25.609Z        INFO    piecestore      upload started  {"Piece ID": "DTGPLWGGSBVFMUA7W65S2SMAI3JKVCDIYUDZB4STVZYMZBCSTE4A", "Satellite ID": "1GGZktUwmMKTwTWNcmGnFJ3n7rjE58QnNcRp98Y23MmbDnVoiU", "Action": "PUT", "Available Space": 998741064448}
2021-12-03T03:33:28.911Z        INFO    piecestore      upload started  {"Piece ID": "J5I3MHTXG6LQYCZL2RV5FGDHM5VUAL45CRNU5G33M45VN4PUXINA", "Satellite ID": "1GGZktUwmMKTwTWNcmGnFJ3n7rjE58QnNcRp98Y23MmbDnVoiU", "Action": "PUT", "Available Space": 998741064448}
2021-12-03T03:35:43.849Z        INFO    piecestore      uploaded        {"Piece ID": "J5I3MHTXG6LQYCZL2RV5FGDHM5VUAL45CRNU5G33M45VN4PUXINA", "Satellite ID": "1GGZktUwmMKTwTWNcmGnFJ3n7rjE58QnNcRp98Y23MmbDnVoiU", "Action": "PUT", "Size": 1515264}
2021-12-03T03:36:11.900Z        INFO    bandwidth       Performing bandwidth usage rollups
2021-12-03T03:36:25.317Z        INFO    piecestore      uploaded        {"Piece ID": "DTGPLWGGSBVFMUA7W65S2SMAI3JKVCDIYUDZB4STVZYMZBCSTE4A", "Satellite ID": "1GGZktUwmMKTwTWNcmGnFJ3n7rjE58QnNcRp98Y23MmbDnVoiU", "Action": "PUT", "Size": 2319360}
2021-12-03T04:36:11.825Z        INFO    bandwidth       Performing bandwidth usage rollups
2021-12-03T05:00:34.269Z        INFO    piecestore      upload started  {"Piece ID": "5U2KT6NB6BJVVRRR7XPRQI4C7AGRTGHPK3HSKRYYQTGN42K3P24Q", "Satellite ID": "1GGZktUwmMKTwTWNcmGnFJ3n7rjE58QnNcRp98Y23MmbDnVoiU", "Action": "PUT", "Available Space": 998737228800}
2021-12-03T05:00:35.621Z        INFO    piecestore      uploaded        {"Piece ID": "5U2KT6NB6BJVVRRR7XPRQI4C7AGRTGHPK3HSKRYYQTGN42K3P24Q", "Satellite ID": "1GGZktUwmMKTwTWNcmGnFJ3n7rjE58QnNcRp98Y23MmbDnVoiU", "Action": "PUT", "Size": 455936}
2021-12-03T05:00:36.517Z        INFO    piecestore      upload started  {"Piece ID": "UOO2PP7NGIKTEWHBIGBU62NMEIKHUNX54LEOA4CEJYWAJYV6E7IQ", "Satellite ID": "1GGZktUwmMKTwTWNcmGnFJ3n7rjE58QnNcRp98Y23MmbDnVoiU", "Action": "PUT", "Available Space": 998736772352}
2021-12-03T05:00:37.579Z        INFO    piecestore      uploaded        {"Piece ID": "UOO2PP7NGIKTEWHBIGBU62NMEIKHUNX54LEOA4CEJYWAJYV6E7IQ", "Satellite ID": "1GGZktUwmMKTwTWNcmGnFJ3n7rjE58QnNcRp98Y23MmbDnVoiU", "Action": "PUT", "Size": 409600}
2021-12-03T05:36:11.855Z        INFO    bandwidth       Performing bandwidth usage rollups
2021-12-03T05:36:21.294Z        INFO    orders.1GGZktUwmMKTwTWNcmGnFJ3n7rjE58QnNcRp98Y23MmbDnVoiU       sending {"count": 5}
2021-12-03T05:36:22.468Z        INFO    orders.1GGZktUwmMKTwTWNcmGnFJ3n7rjE58QnNcRp98Y23MmbDnVoiU       finished
2021-12-03T06:32:26.383Z        INFO    piecestore      upload started  {"Piece ID": "F6L23HKTLFNAQMDNSOQ4PFGKUYRD53RIDXMD6N6KFDQBV267FM4A", "Satellite ID": "1GGZktUwmMKTwTWNcmGnFJ3n7rjE58QnNcRp98Y23MmbDnVoiU", "Action": "PUT", "Available Space": 998736362240}
2021-12-03T06:32:27.062Z        INFO    piecestore      upload started  {"Piece ID": "2W42BYB5WYSNMLJCSCVNUUH2DN7DCJEDNGWFLO6GZZDMUMQAIKLA", "Satellite ID": "1GGZktUwmMKTwTWNcmGnFJ3n7rjE58QnNcRp98Y23MmbDnVoiU", "Action": "PUT", "Available Space": 998736362240}
2021-12-03T06:33:57.577Z        INFO    piecestore      uploaded        {"Piece ID": "2W42BYB5WYSNMLJCSCVNUUH2DN7DCJEDNGWFLO6GZZDMUMQAIKLA", "Satellite ID": "1GGZktUwmMKTwTWNcmGnFJ3n7rjE58QnNcRp98Y23MmbDnVoiU", "Action": "PUT", "Size": 1612032}
2021-12-03T06:34:20.007Z        INFO    piecestore      uploaded        {"Piece ID": "F6L23HKTLFNAQMDNSOQ4PFGKUYRD53RIDXMD6N6KFDQBV267FM4A", "Satellite ID": "1GGZktUwmMKTwTWNcmGnFJ3n7rjE58QnNcRp98Y23MmbDnVoiU", "Action": "PUT", "Size": 2319360}
2021-12-03T06:36:11.833Z        INFO    bandwidth       Performing bandwidth usage rollups
2021-12-03T06:39:48.045Z        INFO    piecedeleter    delete piece sent to trash      {"Satellite ID": "1GGZktUwmMKTwTWNcmGnFJ3n7rjE58QnNcRp98Y23MmbDnVoiU", "Piece ID": "2W42BYB5WYSNMLJCSCVNUUH2DN7DCJEDNGWFLO6GZZDMUMQAIKLA"}
2021-12-03T06:40:32.921Z        INFO    piecestore      upload started  {"Piece ID": "HDKBI2BBUHU274ONMGGIRUFBKWCT52FDT6LWF5PAMIJ3YYXMZZFA", "Satellite ID": "1GGZktUwmMKTwTWNcmGnFJ3n7rjE58QnNcRp98Y23MmbDnVoiU", "Action": "PUT", "Available Space": 998732429824}
2021-12-03T06:43:21.977Z        INFO    piecestore      uploaded        {"Piece ID": "HDKBI2BBUHU274ONMGGIRUFBKWCT52FDT6LWF5PAMIJ3YYXMZZFA", "Satellite ID": "1GGZktUwmMKTwTWNcmGnFJ3n7rjE58QnNcRp98Y23MmbDnVoiU", "Action": "PUT", "Size": 2319360}
2021-12-03T06:45:41.289Z        INFO    piecedeleter    delete piece sent to trash      {"Satellite ID": "1GGZktUwmMKTwTWNcmGnFJ3n7rjE58QnNcRp98Y23MmbDnVoiU", "Piece ID": "HDKBI2BBUHU274ONMGGIRUFBKWCT52FDT6LWF5PAMIJ3YYXMZZFA"}
2021-12-03T06:47:33.402Z        INFO    piecestore      upload started  {"Piece ID": "MOLEH4QSPDI2Y2E5GUNQA4YRQPB2KJ2UT4EEJRXQTTK4WLL2DECA", "Satellite ID": "1GGZktUwmMKTwTWNcmGnFJ3n7rjE58QnNcRp98Y23MmbDnVoiU", "Action": "PUT", "Available Space": 998730109952}
2021-12-03T06:47:33.542Z        INFO    piecestore      upload started  {"Piece ID": "PTRDBV5N6JX6UB7BNMNLT6KK7ZW63QJFKD7ORZLFCERGLPP3CPBQ", "Satellite ID": "1GGZktUwmMKTwTWNcmGnFJ3n7rjE58QnNcRp98Y23MmbDnVoiU", "Action": "PUT", "Available Space": 998730109952}
2021-12-03T06:47:33.664Z        INFO    piecestore      upload started  {"Piece ID": "VQNTCKUON6CQE654XQ76Q7T76KAXBWWV65HVBQ56TYKAW6NCVP3Q", "Satellite ID": "1GGZktUwmMKTwTWNcmGnFJ3n7rjE58QnNcRp98Y23MmbDnVoiU", "Action": "PUT", "Available Space": 998730109952}
2021-12-03T06:53:09.196Z        INFO    piecestore      uploaded        {"Piece ID": "MOLEH4QSPDI2Y2E5GUNQA4YRQPB2KJ2UT4EEJRXQTTK4WLL2DECA", "Satellite ID": "1GGZktUwmMKTwTWNcmGnFJ3n7rjE58QnNcRp98Y23MmbDnVoiU", "Action": "PUT", "Size": 2319360}
2021-12-03T06:53:18.025Z        INFO    piecestore      uploaded        {"Piece ID": "PTRDBV5N6JX6UB7BNMNLT6KK7ZW63QJFKD7ORZLFCERGLPP3CPBQ", "Satellite ID": "1GGZktUwmMKTwTWNcmGnFJ3n7rjE58QnNcRp98Y23MmbDnVoiU", "Action": "PUT", "Size": 2319360}
2021-12-03T06:53:19.766Z        INFO    piecestore      uploaded        {"Piece ID": "VQNTCKUON6CQE654XQ76Q7T76KAXBWWV65HVBQ56TYKAW6NCVP3Q", "Satellite ID": "1GGZktUwmMKTwTWNcmGnFJ3n7rjE58QnNcRp98Y23MmbDnVoiU", "Action": "PUT", "Size": 2319360}
2021-12-03T06:53:23.653Z        INFO    piecestore      upload started  {"Piece ID": "HVWNRLJEUZXUDUEHOPPUP7BSKVIHG6K5V24OHQI2AXYUL36DQS4A", "Satellite ID": "1GGZktUwmMKTwTWNcmGnFJ3n7rjE58QnNcRp98Y23MmbDnVoiU", "Action": "PUT", "Available Space": 998723150336}
2021-12-03T06:53:27.443Z        INFO    piecestore      upload started  {"Piece ID": "XFBXYGELO3XYUI4YOJ5YPWX5B3YQVHZ6LH32WD6YW3RFWG7BNTIQ", "Satellite ID": "1GGZktUwmMKTwTWNcmGnFJ3n7rjE58QnNcRp98Y23MmbDnVoiU", "Action": "PUT", "Available Space": 998723150336}
2021-12-03T06:54:29.770Z        INFO    piecestore      uploaded        {"Piece ID": "XFBXYGELO3XYUI4YOJ5YPWX5B3YQVHZ6LH32WD6YW3RFWG7BNTIQ", "Satellite ID": "1GGZktUwmMKTwTWNcmGnFJ3n7rjE58QnNcRp98Y23MmbDnVoiU", "Action": "PUT", "Size": 238336}
2021-12-03T07:00:26.041Z        INFO    piecestore      uploaded        {"Piece ID": "HVWNRLJEUZXUDUEHOPPUP7BSKVIHG6K5V24OHQI2AXYUL36DQS4A", "Satellite ID": "1GGZktUwmMKTwTWNcmGnFJ3n7rjE58QnNcRp98Y23MmbDnVoiU", "Action": "PUT", "Size": 2319360}

What would happen to the pieces that are supposed to be deleted?

littleskunk · December 3, 2021, 10:55am

They should get deleted ofc. Maybe with a few minutes delay but I would expect that they get deleted.

I tested it myself right now. If I delete a file, folder or bucket my storage node gets the delete message. Can you reproduce your issue?

nerdatwork · December 3, 2021, 3:38pm

This is true IFF you enter the passphrase and access the files. I am specifically mentioning the deletion of bucket without entering the passphrase. You can’t delete file or folder without entering passphrase.

https://link.staging.tardigradeshare.io/jwwn7mfvoif4avlrxvhjrudylbua/second%2Fdeletebucket.mp4

littleskunk · December 3, 2021, 3:44pm

@Andrii can you take a look please ^? I noticed that the delete security question hasn’t been answered. I would expect an error message but it looks like we have broken that part.

nerdatwork · December 3, 2021, 3:47pm

I have mentioned this here too

littleskunk · December 3, 2021, 4:09pm

In my test I did answer that security question and my bucket was deleted inclusive delete messages.

nerdatwork · December 3, 2021, 7:11pm

Even after filling the bucket name only 1 piece out of 2 pieces was deleted.

Andrii · December 3, 2021, 7:22pm

Yes, looks like we broke it. As a hotfix we can disable “Confirm Delete Bucket” button if no data inputted.

littleskunk · December 6, 2021, 2:22pm

I tried to recreate that situation. My bucket gets deleted and my storage node also gets the expected delete messages. I believe the reason for the different observations is the long-tail cancelation. The delete job doesn’t wait until all storage nodes respond to the delete request. The delete job will terminate the remaining connections after deleting 75% of the pieces.

This long tail cancelation seems to work fine in production. Even the remaining 25% should have enough time to receive the delete message and even execute it. We just cut off the callback. Now in testnet most nodes run in a data center. That means we reach the 75% too fast and the side effect would be that your nodes might miss a few more delete messages all the time.

nerdatwork · December 6, 2021, 3:41pm

Could we test this when you execute your test on taking few test nodes offline? I don’t remember the exact post where you mentioned taking test nodes offline.

littleskunk · December 6, 2021, 6:14pm

This specific test is impossible I think. We just don’t have enough storage nodes for that.

SGC · December 6, 2021, 6:50pm

call for more test nodes heard, haven’t gotten around to spinning one up myself, but will do that in the near future.

Vadim · December 7, 2021, 5:47am

for testnodes is there also rule /24 as one node? if not i can make seome.

Alexey · December 9, 2021, 7:00am

7 posts were split to a new topic: Watchtower did not update the node (it’s ended 2 versions behind the current release)