Strange prooblem, need some help

So my node started reboot by it self, every up to 40min.
I found that:

find -maxdepth 1 -iname “*.db” -print0 -exec sqlite3 ‘{}’ ‘PRAGMA integrity_check;’ ‘;’

./reputation.dbok
./piece_expiration.dbok
./pieceinfo.dbok
./piece_spaced_used.db*** in database main ***
Page 5: btreeInitPage() returns error code 11
Error: stepping, database disk image is malformed (11)
./info.dbok
./storage_usage.dbok
./used_serial.dbok
./orders.dbok
./notifications.dbok
./heldamount.dbok
./satellites.dbok
./pricing.dbok
./bandwidth.db*** in database main ***
Page 19845: btreeInitPage() returns error code 11
Page 19181: btreeInitPage() returns error code 11
Page 19879: btreeInitPage() returns error code 11
Page 19866: btreeInitPage() returns error code 11
Page 19738: btreeInitPage() returns error code 11
Error: stepping, database disk image is malformed (11)
./secret.dbok

so i fix it. Now i have:

find -maxdepth 1 -iname “*.db” -print0 -exec sqlite3 ‘{}’ ‘PRAGMA integrity_check;’ ‘;’

./reputation.dbok
./piece_expiration.dbok
./pieceinfo.dbok
./piece_spaced_used.dbok
./info.dbok
./storage_usage.dbok
./used_serial.dbok
./orders.dbok
./notifications.dbok
./heldamount.dbok
./satellites.dbok
./pricing.dbok
./bandwidth.dbok
./secret.dbok

BUT still when i run the node, it stops and gives this error:


2024-02-24T14:50:37Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.96.6"}
2024-02-24T14:50:37Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-updater"}
2024-02-24T14:50:37Z    INFO    server  kernel support for server-side tcp fast open remains disabled.  {"process": "storagenode"}
2024-02-24T14:50:37Z    INFO    server  enable with: sysctl -w net.ipv4.tcp_fastopen=3  {"process": "storagenode"}
2024-02-24T14:50:37Z    INFO    Telemetry enabled       {"process": "storagenode", "instance ID": "12Tna3C4LrMXzUn3GJmwuCAopFPJGmH6mXyzfF5HiY2G5Uapuoh"}
2024-02-24 14:50:37,897 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-02-24 14:50:37,899 INFO success: storagenode entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-02-24 14:50:37,900 INFO success: storagenode-updater entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-02-24T14:50:37Z    INFO    Event collection enabled        {"process": "storagenode", "instance ID": "12Tna3C4LrMXzUn3GJmwuCAopFPJGmH6mXyzfF5HiY2G5Uapuoh"}
2024-02-24T14:50:37Z    INFO    db.migration    Database Version        {"process": "storagenode", "version": 54}
Error: Error during preflight check for storagenode databases: preflight: database "bandwidth": expected schema does not match actual:   &dbschema.Schema{
        Tables: {&{Name: "bandwidth_usage", Columns: {&{Name: "action", Type: "INTEGER"}, &{Name: "amount", Type: "BIGINT"}, &{Name: "created_at", Type: "TIMESTAMP"}, &{Name: "satellite_id", Type: "BLOB"}}}, &{Name: "bandwidth_usage_rollups", Columns: {&{Name: "action", Type: "INTEGER"}, &{Name: "amount", Type: "BIGINT"}, &{Name: "interval_start", Type: "TIMESTAMP"}, &{Name: "satellite_id", Type: "BLOB"}}, PrimaryKey: {"action", "interval_start", "satellite_id"}}},
-       Indexes: []*dbschema.Index{
-               s`Index<Table: bandwidth_usage, Name: idx_bandwidth_usage_created, Columns: created_at, Unique: false, Partial: "">`,
-               s`Index<Table: bandwidth_usage, Name: idx_bandwidth_usage_satellite, Columns: satellite_id, Unique: false, Partial: "">`,
-       },
+       Indexes:   nil,
        Sequences: nil,
  }

        storj.io/storj/storagenode/storagenodedb.(*DB).preflight:429
        storj.io/storj/storagenode/storagenodedb.(*DB).Preflight:376
        main.cmdRun:108
        main.newRunCmd.func1:32
        storj.io/private/process.cleanup.func1.4:393
        storj.io/private/process.cleanup.func1:411
        github.com/spf13/cobra.(*Command).execute:852
        github.com/spf13/cobra.(*Command).ExecuteC:960
        github.com/spf13/cobra.(*Command).Execute:897
        storj.io/private/process.ExecWithCustomOptions:112
        main.main:30
        runtime.main:267
2024-02-24 14:50:38,019 INFO exited: storagenode (exit status 1; not expected)
2024-02-24 14:50:39,027 INFO spawned: 'storagenode' with pid 98
2024-02-24 14:50:39,028 WARN received SIGQUIT indicating exit request
2024-02-24 14:50:39,029 INFO waiting for storagenode, processes-exit-eventlistener, storagenode-updater to die
2024-02-24T14:50:39Z    INFO    Got a signal from the OS: "terminated"  {"Process": "storagenode-updater"}
2024-02-24 14:50:39,041 INFO stopped: storagenode-updater (exit status 0)
2024-02-24 14:50:39,048 INFO stopped: storagenode (terminated by SIGTERM)
2024-02-24 14:50:39,049 INFO stopped: processes-exit-eventlistener (terminated by SIGTERM)

Is there any way to fix it?

/dev/sda1 5.5T 4.9T 316G 95% /home/node/storj/8

Is it to full ? Or where is the problem?
I know even ready to delete all DB’s lost info, but keep the node alive.

Thank you.

1 Like

First check filesystem on errors. Seen this happen more then once when filesystem is juist corrupted.

BTW, did you use search function of the forum?
Let me help you: Search results for 'expected schema does not match actual' - Storj Community Forum (official)

1 Like

But is i know, i can delete then DB’s at all. Recreate them from scratch, i will loose statistics, but will save the node. Or is isn’t?

Yes, i was looking on the forum, and then when to repair my DB’s :expressionless:

I every time got this. Somehow it’s not fixing.
Is my HDD dying?

’ ’ ’
sudo e2fsck -f /dev/sda1

e2fsck 1.46.5 (30-Dec-2021)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Block bitmap differences: -(229146624–229146629) -(229146632–229146638) -(229146645–229146646) -(229146649–229146651) -(229146657–229146658) -(229146680–229146681) -(229146684–229146686) -229146688 -(229146717–229146720) -(229146722–229146723) -(229146728–229146730) -229146732 -(229146734–229146738) -(229146740–229146741) -(229146750–229146752) -(229146756–229146759) -229146761 -229146764 -229146767 -(229146774–229146779) -(229146785–229146786) -(229146789–229146793) -(229146798–229146799) -229146809 -(229146811–229146817) -(229146819–229146826) -(229146832–229146833) -229146835 -(229146837–229146838) -(229146840–229146847) -229146849 -(229146855–229146875) -(229146877–229146885) -(229146887–229146888) -(229146890–229146895) -(229146898–229146928) -229146931 -(229146933–229146940) -(229146950–229146964) -(229146966–229146972) -(229146975–229146977) -(229146984–229146988) -(229146994–229146999) -(229147008–229147015) -(229147024–229147026) -(229147048–229147071) -(229147080–229147095) -(229147104–229147121) -(229147123–229147135) -(229147188–229147197) -(229147204–229147211) -(229147213–229147215) -(229147217–229147231) -(229147234–229147238) -(229147246–229147248) -(229147251–229147255) -(229147301–229147303) -(229147336–229147343) -(229147356–229147359) -(229147404–229147419) -(229147424–229147471) -(229147480–229147481) -(229147492–229147495) -(229147498–229147513) -(229147520–229147525) -(229147580–229147585) -(229147592–229147612) -(229147618–229147623) -(229147638–229147647) -(229147768–229147791) -(229147808–229147820) -(229147840–229147851) -(229147866–229147867) -(229147892–229147894) -(229147896–229147903) -(229147908–229147912) -(229147914–229147935) -(229147956–229147961) -(229147963–229147965) -(229147995–229148032) -(229148046–229148050) -(229148059–229148077) -(229148094–229148095) -(229148124–229148135) -229148156 -(229148160–229148177) -229148182 -(229148197–229148198) -(229148201–229148214) -(229148231–229148234) -(229148244–229148247) -229148279 -(229148288–229148310) -(229148337–229148341) -229148354 -229148356 -229148361 -(229148363–229148373) -(229148378–229148383) -(229148405–229148407) -229148414 -(229148714–229148735) -(229148768–229148783) -(229148901–229148903) -(229148912–229148929) -(229148932–229148934) -229148972 -229148978 -229148989 -229149002 -229149033 -(229149044–229149046) -(229149069–229149076) -(229149078–229149080) -229149086 -(229149089–229149091) -(229149111–229149112) -(229149240–229149245) -(229149280–229149295) -(229149329–229149334) -(229149377–229149386) -(229149440–229149455) -(229149510–229149511) -229149520 -(229149568–229149583) -(229150432–229150463) -(229150720–229150767) -(229150773–229150782) -(229150793–229150802) -(229150836–229150839) -(229150848–229150867) -(229150871–229150880) -(229150884–229150919) -229150930 -(229150939–229150948) -(229150950–229150969) -(229150975–229150984) -(229150991–229151000) -(229151015–229151016) -(229151020–229151023) -(229151027–229151036) -(229151055–229151068) -(229151079–229151137) -(229151147–229151156) -229151178 -(229151189–229151207) -(229151213–229151228) -(229151287–229151288) -(229151319–229151323) -(229151334–229151340) -(229151396–229151487) -(229151744–229151798) -(229151909–229151935) -(229151972–229151999) -(229152256–229152275) -(229152291–229152293) -(229152320–229152416) -(229152426–229152439) -(229152450–229152452) -(229152462–229152473) -(229152475–229152511) -(229152584–229152589) -(229152599–229152626) -(229152685–229152692) -(229152703–229152710) -(229152722–229152733) -(229152744–229152760) -(229152968–229152979) -(229152992473) -(229152475–229152511) -(229152584–229152589) -(229152599–229152626) -(229152685–229152692) -(229152703–229152710) -(229152722–229152733) -(229152744–229152760) -(229152968–229152979) -(229152993–229152994) -(229153023–229153038) -(229153054–229153056) -(229153112–229153146) -(229153238–229153253) -(229153262–229153269) -(229153277–229153280) -229153283 -(229153301–229153303) -(229153333–229153336) -(229153370–229153382) -(229153393–229153394) -229153421 -(229153532–229153747) -(229153755–229153866) -(229153872–229153896) -(229153899–229153901) -(229155840–229155903) -(229155949–229156002) -(229156004–229156011) -(229156015–229156052) -(229156057–229156074) -229156085 -(229156095–229156113) -229156133 -(229156143–229156145) -(229156147–229156159) -(229156199–229156292) -(229156304–229156310) -(229156315–229156316) -(229156322–229156326) -(229156342–229156347) -229156367 -(229156377–229156385) -(229157943–229158062) -(229158089–229158143) -(229158189–229158207) -(229158253–229158271) -(229158343–229158359) -(229158364–229158383) -(229158386–229158390) -(229158397–229158399) -(229158445–229158452) -(229158461–229158469) -(229159076–229159103) -(229159149–229159158) -229159179 -(229159183–229159187) -(229159199–229159216) -(229159223–229159236) -229159245 -(229159252–229159255) -229159262 -(229159278–229159282) -(229159287–229159302) -(229159305–229159351) -229159356 -(229159358–229159545) -(229159550–229159564) -229159569 -(229159575–229159603) -(229159619–229159622) -(229159624–229159649) -(229159659–229159662) -(229159678–229159679) -(229159685–229159700) -(229159702–229159710) -(229159714–229159718) -229159730 -(229159740–229159743) -(229159781–229159799) -(229159814–229159874) -(229159885–229159891) -(229159899–229159906) -(229159908–229159926) -(229159934–229159935) -(229160008–229160011) -(229160015–229160025) -(229160031–229160046) -229160051 -229160063 -(229160482–229160493) -(229160500–229160516) -(229160528–229160544) -229160556 -(229161984–229162037) -(229162093–229162162) -(229162164–229162173) -(229162183–229162205) -(229162211–229162240) -(229162245–229162272) -(229162276–229162285) -(229162287–229162306) -(229162312–229162342) -(229162350–229162351) -(229162357–229162367) -(229162413–229162445) -(229162451–229162460) -(229162462–229162463) -(229162488–229162507) -229162510 -(229162515–229162516) -229162529 -229162542 -(229162553–229162557) -(229162582–229162611) -(229162613–229162617) -(229162634–229162638) -(229162640–229162652) -(229162659–229162665) -(229162668–229162671) -(229162682–229162687) -(229162733–229162751) -(229162941–229163007) -(229163416–229163454) -(229163501–229163551) -(229163554–229163573) -(229164053–229164086) -(229164142–229164159) -(229164249–229164287) -(229164453–229164479) -(229164525–229164543) -(229164562–229164598) -(229165082–229165110) -(229165570–229165588) -(229165598–229165628) -(229165677–229165796) -(229165852–229166079) -(229166244–229166335) -(229166381–229166422) -(229166436–229166447) -(229166486–229166497) -229166510 -(229166515–229166519) -(229166522–229166524) -(229166529–229166536) -(229166560–229166562) -229166583 -(229166645–229166646) -(229167269–229167359) -(229167405–229167551) -(229167587–229167615) -(229168226–229168284) -229168286 -(229168290–229168291) -229168304 -229168314 -(229168318–229168326) -229168332 -229168341 -(229168350–229168352) -(229168354–229168363) -(229168372–229168376) -(229168382–229168396) -(229168400–229168406) -229168410 -(229168412–229168583) -(229168585–229168600) -(229168602–229168699) -(229168712–229168745) -(229168749–229168783) -(229168785–229168806) -(229168808–229168814) -(229168829–229168862) -(229168864–229168889) -(229168906–229168918) -(229168926–229168953) -(229168957–229168980) -(229168987–229169005) -(229169015–229169028) -(229169033–229169062) -(229169068–229169079) -(229169089–229169155) -(229169161–229169163) -(229169165–229169169) -(229169184–229169190) -(229169196–229169215) -(229169233–229169243) -(229169255–229169375) -(229169408–229169424) -229169428 -(229169441–229169450) -(229169455–229169458) -229169473 -(229169481–229169491) -(229169502–229169529) -(229169540–229169544) -(229169563–229169568) -(229169570–229169573) -(229169583–229169603) -(229169626–229169628) -(229169643–229169644) -(229169653–229169656) -(229169662–229169676) -(229169678–229169692) -(229169697–229169714) -(229169716–229169717) -(229169828–229169838) -(229169858–229169860) -(229169875–229169909) -(229169911–229169915) -(229169941–229169946) -(229169953–229169966) -229169971 -229169981 -(229169993–229169996) -(229170000–229170015) -(229170033–229170035) -(229170049–229170051) -(229170069–229170085) -(229170098–229170120) -(229170125–229170130) -(229170133–229170148) -229170155 -229170163 -(229170169–229170175) -(229170318–229170325) -(229170330–229170372) -(229170374–229170376) -(229170392–229170436) -(229170441–229170445) -(229170452–229170484) -(229170497–229170499) -(229170509–229170516) -(229170535–229170541) -(229170558–229170577) -(229170582–229170587) -(229170593–229170614) -(229170620–229170639) -(229170642–229170647) -(229170653–229170656) -229170659 -(229170663–229170666) -(229170680–229170714) -229170727 -(229172224–229172256) -(229172258–229172273) -(229172278–229172286) -(229172290–229172304) -(229172310–229172312) -(229172328–229172339) -(229172350–229172362) -(229172366–229172373) -(229172379–229172388) -(229172398–229172401) -(229172421–229172430) -(229172442–229172478) -(229172491–229172518) -(229172522–229172556) -229172558 -(229172560–229172564) -(229172567–229172572) -(229172574–229172735) -(229173248–229173356) -(229174284–229174463) -(229174509–229174620) -(229174622–229174633) -(229174652–229174653) -(229174667–229174676) -(229174684–229174687) -(229174706–229174749) -229174770 -(229174772–229174791) -(229174807–229174810) -229174831 -(229175808–229175812) -(229175826–229175859) -229175861 -(229175917–229175925) -(229175942–229175954) -(229175956–229175974) -229175994 -(229175996–229175997) -(229175999–229176063) -(229176109–229176252) -(229176262–229176319) -(229177088–229177094) -(229177103–229177120) -(229177128–229177140) -(229177144–229177150) -(229177160–229177175) -(229177185–229177186) -(229177193–229177196) -(229177205–229177215) -(229177263–229177268) -(229177278–229177287) -(229177293–229177294) -(229177296–229177299) -229177330 -(229177339–229177343) -(229177856–229177862) -(229177879–229177905) -(229178368–229178403) -(229178609–229178621) -(229178677–229178723) -437813036 -(437813038–437813048) -(437813084–437813093) -(437813177–437813181) -(441459672–441459677) -(441459681–441459682) -441459688 -(441460540–441460546) -(441460571–441460577) -(441460608–441460613) -(441460635–441460638) -(441460666–441460671) -(441460697–441460698) -441460731 -(441460850–441460855) -441460859 -(441462137–441462142) -(441462164–441462169) -(441462210–441462214) -(441462251–441462254) -441462267 -441462274 -(441462283–441462286) -(441462314–441462320) -(441462329–441462340) -(441462350–441462354) -(441462367–441462371) -(441462384–441462400) -(441462414–441462424) -(441462440–441462442) -441462494 -(441462551–441462553) -441462559 -441462561 -(441463123–441463127) -441463172 -(441463174–441463176) -(441466624–441466629) -(441466632–441466634) -441466721 -(441466723–441466726) -(441466738–441466742) -441466754 -(441466767–441466781) -(441466825–441466826) -(441466842–441466845) -(441466848–441466851) -(441466868–441466871) -441466875 -441466878 -441468701 -(441468744–441468750) -441468767 -(441468827–441468833) -(441468835–441468836) -(441468868–441468869) -441468875 -(441468913–441468919) -441470808 -(441471044–441471048) -441471056 -(441475007–441475013) -(441475040–441475045) -(441475052–441475058) -(441478363–441478364) -(441478372–441478375) -(441478381–441478393) -441478426 -(441478472–441478474) -(441478579–441478585) -441478590 -(441480919–441480920) -441480997 -(441481566–441481568) -(441481625–441481626) -(441481686–441481690) -(441481845–441481851) -441481884 -(441483041–441483048) -(1274059904–1274059992) -(1274062048–1274062066) -(1274062976–1274063064) -(1274068128–1274068146) -(1274071168–1274071212) -(1274071616–1274071634) -(1274074112–1274074130) -(1274076736–1274076754) -(1274077056–1274077074) -(1274077088–1274077105) -(1274077344–1274077362) -(1274079712–1274079730) -(1274108672–1274108760) -(1274122880–1274122948) -(1274126848–1274126902) -(1274131392–1274131436) -(1274143317–1274143326) -(1274143392–1274143401) -(1274143420–1274143429) -(1274143487–1274143496) -(1274147895–1274148351) -(1274151040–1274151084) -(1274163712–1274163766) -(1274185216–1274185243) -(1274187488–1274187506) -(1274189632–1274189676) -(1274200896–1274200940) -(1274223424–1274223468) -(1274224128–1274224147) -(1274224265–1274224273) -(1274224298–1274224301) -(1274224565–1274224574) -(1274224640–1274224689) -(1274224832–1274224876) -(1274230208–1274230270) -(1274234304–1274234348) -(1274235801–1274235810) -(1274236224–1274236268) -1274248881 -(1274248898–1274248907) -(1274248988–1274248996) -(1274249133–1274249140) -(1274249440–1274249460) -(1274249472–1274249539) -(1274259712–1274259788) -(1274264832–1274264944) -(1274281664–1274281708) -(1274286656–1274286691) -(1274290016–1274290038) -(1274290048–1274290117) -(1274291328–1274291416) -(1274296832–1274296841) -(1274297099–1274297108) -(1274297124–1274297133) -(1274297200–1274297209) -(1274324736–1274324913) -(1274366019–1274366027) -(1274366287–1274366296) -(1274380928–1274381002) -(1274386304–1274386337) -(1274401792–1274401801) -(1274402069–1274402078) -(1274428160–1274428334) -(1274450432–1274450441) -(1274450525–1274450533) -(1274450612–1274450621)
Fix? yes to all
/dev/sda1: 7599132/183144448 files (4.7% non-contiguous), 1309273718/1465129488 blocks

’ ’ ’

OK. so i was running same command many times, restarted PC and these problems GONE. Now disks is clear.
Node did not started anyways.

Then i copied(moved) DB’s, restarted node, it kind of created few new DB’s, but it was not running. It stopped by it self. Then i copied back correct DB’s. And ques what. It is still not working. When started i get the same error:

2024-02-24T23:07:43Z INFO db.migration.29 Migrate piece_space_used to add total column {“process”: “storagenode”}
Error: Error creating tables for master database on storagenode: migrate: v29: no such table: piece_space_used
storj.io/storj/private/migrate.(*Migration).Run:209
storj.io/storj/storagenode/storagenodedb.(*DB).MigrateToLatest:370
main.cmdRun:93
main.newRunCmd.func1:32
storj.io/private/process.cleanup.func1.4:393
storj.io/private/process.cleanup.func1:411
github.com/spf13/cobra.(*Command).execute:852
github.com/spf13/cobra.(*Command).ExecuteC:960
github.com/spf13/cobra.(*Command).Execute:897
storj.io/private/process.ExecWithCustomOptions:112
main.main:30
runtime.main:267
2024-02-24 23:07:43,831 INFO exited: storagenode (exit status 1; not expected)
2024-02-24 23:07:46,841 INFO spawned: ‘storagenode’ with pid 73
2024-02-24T23:07:46Z INFO Configuration loaded {“process”: “storagenode”, “Location”: “/app/config/config.yaml”}
2024-02-24T23:07:46Z INFO Anonymized tracing enabled {“process”: “storagenode”}
2024-02-24T23:07:46Z INFO Operator email {“process”: “storagenode”, “Address”: “”}
2024-02-24T23:07:46Z INFO Operator wallet {“process”: “storagenode”, “Address”: “xxxx”}
2024-02-24T23:07:47Z INFO server kernel support for server-side tcp fast open remains disabled. {“process”: “storagenode”}
2024-02-24T23:07:47Z INFO server enable with: sysctl -w net.ipv4.tcp_fastopen=3 {“process”: “storagenode”}
2024-02-24T23:07:47Z INFO Telemetry enabled {“process”: “storagenode”, “instance ID”: “12Tna3C4LrMXzUn3GJmwuCAopFPJGmH6mXyzfF5HiY2G5Uapuoh”}
2024-02-24T23:07:47Z INFO Event collection enabled {“process”: “storagenode”, “instance ID”: “12Tna3C4LrMXzUn3GJmwuCAopFPJGmH6mXyzfF5HiY2G5Uapuoh”}
2024-02-24T23:07:47Z INFO db.migration.29 Migrate piece_space_used to add total column {“process”: “storagenode”}
Error: Error creating tables for master database on storagenode: migrate: v29: no such table: piece_space_used
storj.io/storj/private/migrate.(*Migration).Run:209
storj.io/storj/storagenode/storagenodedb.(*DB).MigrateToLatest:370
main.cmdRun:93
main.newRunCmd.func1:32
storj.io/private/process.cleanup.func1.4:393
storj.io/private/process.cleanup.func1:411
github.com/spf13/cobra.(*Command).execute:852
github.com/spf13/cobra.(*Command).ExecuteC:960
github.com/spf13/cobra.(*Command).Execute:897
storj.io/private/process.ExecWithCustomOptions:112
main.main:30
runtime.main:267
2024-02-24 23:07:47,502 INFO exited: storagenode (exit status 1; not expected)
2024-02-24 23:07:48,504 INFO gave up: storagenode entered FATAL state, too many start retries too quickly
2024-02-24 23:07:49,506 WARN received SIGQUIT indicating exit request
2024-02-24 23:07:49,507 INFO waiting for processes-exit-eventlistener, storagenode-updater to die
2024-02-24T23:07:49Z INFO Got a signal from the OS: “terminated” {“Process”: “storagenode-updater”}
2024-02-24 23:07:49,516 INFO stopped: storagenode-updater (exit status 0)
2024-02-24 23:07:50,519 INFO stopped: processes-exit-eventlistener (terminated by SIGTERM)

Any ideas?

You have to stop the node, delete corrupted databases, move all remained databases, then start the node, it will re-create all of them. Now you need to stop it again and move all databases back except corrupted ones, after that try to start the node.

Right now it’s complain about piece_spaced_used.db, so this one should be deleted, all other - moved.

2 Likes

I of course was stopped the node before doing this procedure.
But now the question should i still delete those two DB’s that was corrupted?
Because now i see this:

find -maxdepth 1 -iname “*.db” -print0 -exec sqlite3 ‘{}’ ‘PRAGMA integrity_check;’ ‘;’

./reputation.dbok
./piece_expiration.dbok
./pieceinfo.dbok
./piece_spaced_used.dbok
./info.dbok
./storage_usage.dbok
./used_serial.dbok
./orders.dbok
./notifications.dbok
./heldamount.dbok
./satellites.dbok
./pricing.dbok
./bandwidth.dbok
./secret.dbok

This means all db’s are OK. But looks like it is still doesn’t like: piece_spaced_used.db. So probably i will NOT move back this DB after recreation.

One more question? Should i change any permissions to these new db’s? Or leave as it?

Thank you.

yes, the integrity is ok, but not the schema, it’s not match the versions sequence. Actually you can recover it with a sqlite3 one version by another, but it will take a lot of time. So, the simpler solution is to just remove the database in question and re-create it empty, it doesn’t affect the node, only a history and stat.

depends. Did you use --user $(id -u):$(id -g) in your docker run command? If the answer - is “yes”, then make sure that all files, include databases are belongs to this owner ($(id -u):$(id -g)), otherwise root:root is probably ok if you use sudo to run the container, otherwise you need to update the owner to the same one, as used by the container.

Alexey please help!
I did a few times already. Same story.
Please see all my steps. Sorry for long text. But where is my mistake?

forum, does not allow long text, so i put it here:

.no, i dont use " Did you use --user $(id -u):$(id -g) in your docker run command?"

During the night, “uptime robot” was showing this node online/offline/online/offline. every like 1 hour.
Now… it looks like the same. on/off/on/off

Now the problem is

Error: piecestore monitor: disk space requirement not met

So, you have less than 500GB free space. Kind of expected, since the DB is gone and is not filled yet with the actual usage…
The only way is to temporary reduce a monitored minimum free space,

storage2.monitor.minimum-disk-space: 0B

save the config and restart the node.
You may also use a command line argument after the image name instead, i.e. --storage2.monitor.minimum-disk-space=0B.
I would recommend to remove it or comment out after the used space is recognized.

1 Like

OMG thank you Alexey!!!
After i put comment in to : config.yaml
it started to work again !!!

I believe my next steps are reduce allocated space. And then question when about to remove the added line to config.yaml?

Thank you very much !!!

1 Like

When you see the usage as expected. The fast check - just restart. If it would show the actual usage - you are good.
However, before that I would suggest to check, that your logs doesn’t contain errors related to a filewalker (search for walk|retain and error|failed case insensitive).
Because if you would have errors, the used space DB will not be updated unfortunately, see Disk usage discrepancy? and Debugging space usage discrepancies for more details

1 Like

Thank you very much!!!
Have a great day !

2 Likes

You are welcome!
Have a great day too!

2 Likes

Still strange things happening with this node.
After i added:

storage2.monitor.minimum-disk-space: 0B

It started to work. And it was and still is working nice. When i look at the log’s i see nice and proper things like: “INFO piecestore download started”

But for some reason, yesterday i received email, saying: “Your Node has been suspended for being offline, Your Storage Node on the US1 satellite has been suspended for being offline for too long”. I quickly checked, it is still online and still working no errors.
Node at the moment have 620GB Free space and dashboard show that is online and working since we had this conversation here.
So yesterday i removed line:

storage2.monitor.minimum-disk-space: 0B

restarted this node, it is working nice, but for some reason i haven’t received email saying that my node is back online on that US1 satellite.

When looking on the web dash board, i see that this node has ±50% online score on all satellites. There another two nodes on the same machine, and they have ±80% online score on all satellites.
But another two nodes have this situation:

Status ONLINE
Uptime 134h19m36s
&
Status ONLINE
Uptime 134h20m17s

This is fiber optics. All these 3 nodes are monitored by “Uptime robot” and i never got notifications. They always online and always working.

Why online score is so low (red). And how to deal with it?

Thank you.

You may check when it was offline:

Any DDOS prevention tools involved? I got the same problem with an ASUS router with DDOS prevention. I was missing out on audits, which become worse over time because the amounts of audits depends on the amount of data you have. So it took me some months to discover the problem.

2 Likes

In your case i believe DDOS prevention was enabled? Once you disabled it you resolved the problem?

For sure, I don’t have options to exclude some ports, regions or source domains from DDOS prevention. So I disabled it totally, especially since I don’t see any benefit for me as a consumer from this functionality.

1 Like