How to monitor all nodes in your LAN using prometheus + grafana [linux using docker]

It looks like this is a very new node, so it will probably just take a few more hours until you get some data from those satellites. It should fix itself soon.

Node age more than one year, but monitoring has been installed yesterday.
I already tried recreate storj and storj-export docker container and double checked prometheus settings :frowning:

Oh… Strange. Then it should have data. What does your dashboard say? Is the node doing fine on those satellites?
Since you got data for the other satellites, the exporter is working.

I think yes, because current month earnings twice as much in dashboard than grafana

@kevink you know of any reason that my prometheus container on a raspberry pi 4 would all of a sudden stop working last night?

No… what does the log say?

So just for reference, here’s my container start cmd:

sudo docker run -d -p 9090:9090 --restart unless-stopped --user 1000:1000 --name prometheus
-v /home/pi/Downloads/prometheus.yml:/etc/prometheus/prometheus.yml
-v /home/pi/Downloads/prometheus:/prometheus
prom/prometheus --storage.tsdb.retention.time=360d --storage.tsdb.retention.size=30GB
–config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/prometheus

here’s the log after starting until it just kills itself and restarts:

level=info ts=2021-05-30T19:18:09.302Z caller=main.go:426 msg=“Starting Prometheus” version=“(version=2.27.1, branch=HEAD, revision=db7f0bcec27bd8aeebad6b08ac849516efa9ae02)”
level=warn ts=2021-05-30T19:18:09.302Z caller=main.go:428 msg=“This Prometheus binary has not been compiled for a 64-bit architecture. Due to virtual memory constraints of 32-bit systems, it is highly recommended to switch to a 64-bit binary of Prometheus.” GOARCH=arm
level=info ts=2021-05-30T19:18:09.302Z caller=main.go:431 build_context=“(go=go1.16.4, user=root@bd14fd30da36, date=20210518-14:12:35)”
level=info ts=2021-05-30T19:18:09.303Z caller=main.go:432 host_details=“(Linux 5.10.17-v7l+ #1421 SMP Thu May 27 14:00:13 BST 2021 armv7l f87d44e1152f (none))”
level=info ts=2021-05-30T19:18:09.303Z caller=main.go:433 fd_limits=“(soft=1048576, hard=1048576)”
level=info ts=2021-05-30T19:18:09.303Z caller=main.go:434 vm_limits=“(soft=unlimited, hard=unlimited)”
level=info ts=2021-05-30T19:18:09.314Z caller=web.go:540 component=web msg=“Start listening for connections” address=0.0.0.0:9090
level=info ts=2021-05-30T19:18:09.315Z caller=main.go:803 msg=“Starting TSDB …”
level=info ts=2021-05-30T19:18:09.318Z caller=repair.go:57 component=tsdb msg=“Found healthy block” mint=1609687043842 maxt=1611381600000 ulid=01EWQED9Y0VXDKSPDHX97T32BC
level=info ts=2021-05-30T19:18:09.320Z caller=tls_config.go:191 component=web msg=“TLS is disabled.” http2=false
level=info ts=2021-05-30T19:18:09.322Z caller=repair.go:57 component=tsdb msg=“Found healthy block” mint=1611381600000 maxt=1613131200000 ulid=01EYBC39K5ZSJQCHPTCZR5X2W8
level=info ts=2021-05-30T19:18:09.324Z caller=repair.go:57 component=tsdb msg=“Found healthy block” mint=1613131202711 maxt=1614880800000 ulid=01EZZQGWDW3VD9AGV60EF0Z5X3
level=info ts=2021-05-30T19:18:09.325Z caller=repair.go:57 component=tsdb msg=“Found healthy block” mint=1614880801459 maxt=1616630400000 ulid=01F1KW2K3RMHCB12GZFA3YPKRN
level=info ts=2021-05-30T19:18:09.325Z caller=repair.go:57 component=tsdb msg=“Found healthy block” mint=1616630402844 maxt=1618380000000 ulid=01F380M41DAHST9SFR8ETT89ZE
level=info ts=2021-05-30T19:18:09.326Z caller=repair.go:57 component=tsdb msg=“Found healthy block” mint=1618380003675 maxt=1620129600000 ulid=01F4WJZ80YFKDBN0P0P4N4CX1N
level=info ts=2021-05-30T19:18:09.327Z caller=repair.go:57 component=tsdb msg=“Found healthy block” mint=1620129603674 maxt=1620712800000 ulid=01F5DHB6N232WY0DQGPD04EJP4
level=info ts=2021-05-30T19:18:09.328Z caller=repair.go:57 component=tsdb msg=“Found healthy block” mint=1620712800000 maxt=1621296000000 ulid=01F5YPNCABRR9FSVFR3WBAWF2A
level=info ts=2021-05-30T19:18:09.328Z caller=repair.go:57 component=tsdb msg=“Found healthy block” mint=1621296000000 maxt=1621879200000 ulid=01F6G2XY0M5SP4MVW35HF2RGXW
level=info ts=2021-05-30T19:18:09.329Z caller=repair.go:57 component=tsdb msg=“Found healthy block” mint=1621879203675 maxt=1622073600000 ulid=01F6P39T92S9YX5RE3T72357TQ
level=info ts=2021-05-30T19:18:09.329Z caller=repair.go:57 component=tsdb msg=“Found healthy block” mint=1622073600000 maxt=1622268000000 ulid=01F6VNWEA021BPA55W2KJZA26R
level=info ts=2021-05-30T19:18:09.329Z caller=repair.go:57 component=tsdb msg=“Found healthy block” mint=1622332800000 maxt=1622340000000 ulid=01F6XKDNG7WGEN4JQVATXR8NR3
level=info ts=2021-05-30T19:18:09.332Z caller=repair.go:57 component=tsdb msg=“Found healthy block” mint=1622340003675 maxt=1622347200000 ulid=01F6XT9FYWTHTNT9M113DBCKVA
level=info ts=2021-05-30T19:18:09.332Z caller=repair.go:57 component=tsdb msg=“Found healthy block” mint=1622268003675 maxt=1622332800000 ulid=01F6XW10AHZBTFHTX1V1JJA6H7
level=info ts=2021-05-30T19:18:09.332Z caller=repair.go:57 component=tsdb msg=“Found healthy block” mint=1622347200000 maxt=1622354400000 ulid=01F6Y1540B0893B7ZSGMDZ6FXS
level=info ts=2021-05-30T19:18:09.332Z caller=repair.go:57 component=tsdb msg=“Found healthy block” mint=1622354403680 maxt=1622361600000 ulid=01F6Y80YDBS35C2V25F71WGN33
level=info ts=2021-05-30T19:18:11.679Z caller=head.go:741 component=tsdb msg=“Replaying on-disk memory mappable chunks if any”
level=info ts=2021-05-30T19:18:11.689Z caller=head.go:755 component=tsdb msg=“On-disk memory mappable chunks replay completed” duration=9.538609ms
level=info ts=2021-05-30T19:18:11.689Z caller=head.go:761 component=tsdb msg=“Replaying WAL, this may take a while”
level=info ts=2021-05-30T19:18:16.933Z caller=head.go:787 component=tsdb msg=“WAL checkpoint loaded”
level=info ts=2021-05-30T19:18:17.731Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1641 maxSegment=2095
level=info ts=2021-05-30T19:18:18.039Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1642 maxSegment=2095
level=info ts=2021-05-30T19:18:18.731Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1643 maxSegment=2095
level=info ts=2021-05-30T19:18:19.529Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1644 maxSegment=2095
level=info ts=2021-05-30T19:18:20.266Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1645 maxSegment=2095
level=info ts=2021-05-30T19:18:20.986Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1646 maxSegment=2095
level=info ts=2021-05-30T19:18:21.747Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1647 maxSegment=2095
level=info ts=2021-05-30T19:18:22.498Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1648 maxSegment=2095
level=info ts=2021-05-30T19:18:23.209Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1649 maxSegment=2095
level=info ts=2021-05-30T19:18:24.058Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1650 maxSegment=2095
level=info ts=2021-05-30T19:18:24.724Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1651 maxSegment=2095
level=info ts=2021-05-30T19:18:25.430Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1652 maxSegment=2095
level=info ts=2021-05-30T19:18:26.144Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1653 maxSegment=2095
level=info ts=2021-05-30T19:18:26.825Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1654 maxSegment=2095
level=info ts=2021-05-30T19:18:27.505Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1655 maxSegment=2095
level=info ts=2021-05-30T19:18:28.252Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1656 maxSegment=2095
level=info ts=2021-05-30T19:18:29.037Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1657 maxSegment=2095
level=info ts=2021-05-30T19:18:29.820Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1658 maxSegment=2095
level=info ts=2021-05-30T19:18:30.650Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1659 maxSegment=2095
level=info ts=2021-05-30T19:18:31.543Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1660 maxSegment=2095
level=info ts=2021-05-30T19:18:32.376Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1661 maxSegment=2095
level=info ts=2021-05-30T19:18:33.214Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1662 maxSegment=2095
level=info ts=2021-05-30T19:18:33.986Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1663 maxSegment=2095
level=info ts=2021-05-30T19:18:34.726Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1664 maxSegment=2095
level=info ts=2021-05-30T19:18:35.496Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1665 maxSegment=2095
level=info ts=2021-05-30T19:18:36.253Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1666 maxSegment=2095
level=info ts=2021-05-30T19:18:37.744Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1667 maxSegment=2095
level=info ts=2021-05-30T19:18:38.564Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1668 maxSegment=2095
level=info ts=2021-05-30T19:18:39.396Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1669 maxSegment=2095
level=info ts=2021-05-30T19:18:40.196Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1670 maxSegment=2095
level=info ts=2021-05-30T19:18:41.070Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1671 maxSegment=2095
level=info ts=2021-05-30T19:18:41.875Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1672 maxSegment=2095
level=info ts=2021-05-30T19:18:42.841Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1673 maxSegment=2095
level=info ts=2021-05-30T19:18:43.721Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1674 maxSegment=2095
level=info ts=2021-05-30T19:18:44.656Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1675 maxSegment=2095
level=info ts=2021-05-30T19:18:45.932Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1676 maxSegment=2095
level=info ts=2021-05-30T19:18:47.234Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1677 maxSegment=2095
level=info ts=2021-05-30T19:18:48.974Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1678 maxSegment=2095
level=info ts=2021-05-30T19:18:50.671Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1679 maxSegment=2095
level=info ts=2021-05-30T19:18:52.092Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1680 maxSegment=2095
level=info ts=2021-05-30T19:18:53.131Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1681 maxSegment=2095
level=info ts=2021-05-30T19:18:54.202Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1682 maxSegment=2095
level=info ts=2021-05-30T19:18:55.264Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1683 maxSegment=2095
level=info ts=2021-05-30T19:18:56.215Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1684 maxSegment=2095
level=info ts=2021-05-30T19:18:57.062Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1685 maxSegment=2095
level=info ts=2021-05-30T19:18:57.881Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1686 maxSegment=2095
level=info ts=2021-05-30T19:18:57.882Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1687 maxSegment=2095
level=info ts=2021-05-30T19:18:57.883Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1688 maxSegment=2095
level=info ts=2021-05-30T19:18:57.884Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1689 maxSegment=2095
level=info ts=2021-05-30T19:18:57.884Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1690 maxSegment=2095
level=info ts=2021-05-30T19:18:57.885Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1691 maxSegment=2095
level=info ts=2021-05-30T19:18:57.886Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1692 maxSegment=2095
level=info ts=2021-05-30T19:18:57.887Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1693 maxSegment=2095
level=info ts=2021-05-30T19:18:57.888Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1694 maxSegment=2095
level=info ts=2021-05-30T19:18:57.889Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1695 maxSegment=2095
level=info ts=2021-05-30T19:18:57.890Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1696 maxSegment=2095
level=info ts=2021-05-30T19:18:58.659Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1697 maxSegment=2095
level=info ts=2021-05-30T19:18:59.510Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1698 maxSegment=2095
level=info ts=2021-05-30T19:19:00.374Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1699 maxSegment=2095
level=info ts=2021-05-30T19:19:01.194Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1700 maxSegment=2095
level=info ts=2021-05-30T19:19:01.195Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1701 maxSegment=2095
level=info ts=2021-05-30T19:19:02.068Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1702 maxSegment=2095
level=info ts=2021-05-30T19:19:02.851Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1703 maxSegment=2095
level=info ts=2021-05-30T19:19:02.852Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1704 maxSegment=2095
level=info ts=2021-05-30T19:19:02.853Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1705 maxSegment=2095
level=info ts=2021-05-30T19:19:02.854Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1706 maxSegment=2095
level=info ts=2021-05-30T19:19:02.855Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1707 maxSegment=2095
level=info ts=2021-05-30T19:19:02.856Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1708 maxSegment=2095
level=info ts=2021-05-30T19:19:02.856Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1709 maxSegment=2095
level=info ts=2021-05-30T19:19:02.857Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1710 maxSegment=2095
level=info ts=2021-05-30T19:19:02.858Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1711 maxSegment=2095
level=info ts=2021-05-30T19:19:02.859Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1712 maxSegment=2095
level=info ts=2021-05-30T19:19:02.860Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1713 maxSegment=2095
level=info ts=2021-05-30T19:19:02.861Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1714 maxSegment=2095
level=info ts=2021-05-30T19:19:02.862Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1715 maxSegment=2095
level=info ts=2021-05-30T19:19:02.864Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1716 maxSegment=2095
level=info ts=2021-05-30T19:19:02.865Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1717 maxSegment=2095
level=info ts=2021-05-30T19:19:02.866Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1718 maxSegment=2095
level=info ts=2021-05-30T19:19:02.867Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1719 maxSegment=2095
level=info ts=2021-05-30T19:19:02.868Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1720 maxSegment=2095
level=info ts=2021-05-30T19:19:02.870Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1721 maxSegment=2095
level=info ts=2021-05-30T19:19:02.871Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1722 maxSegment=2095
level=info ts=2021-05-30T19:19:02.872Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1723 maxSegment=2095
level=info ts=2021-05-30T19:19:02.874Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1724 maxSegment=2095
level=info ts=2021-05-30T19:19:02.875Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1725 maxSegment=2095
level=info ts=2021-05-30T19:19:02.876Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1726 maxSegment=2095
level=info ts=2021-05-30T19:19:02.878Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1727 maxSegment=2095
level=info ts=2021-05-30T19:19:02.879Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1728 maxSegment=2095
level=info ts=2021-05-30T19:19:02.880Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1729 maxSegment=2095
level=info ts=2021-05-30T19:19:02.880Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1730 maxSegment=2095
level=info ts=2021-05-30T19:19:02.881Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1731 maxSegment=2095
level=info ts=2021-05-30T19:19:02.882Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1732 maxSegment=2095
level=info ts=2021-05-30T19:19:02.884Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1733 maxSegment=2095
level=info ts=2021-05-30T19:19:02.885Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1734 maxSegment=2095
level=info ts=2021-05-30T19:19:02.886Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1735 maxSegment=2095
level=info ts=2021-05-30T19:19:02.887Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1736 maxSegment=2095
level=info ts=2021-05-30T19:19:02.888Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1737 maxSegment=2095
level=info ts=2021-05-30T19:19:02.890Z caller=head.go:813 component=tsdb msg=“WAL segment loaded” segment=1738 maxSegment=2095
panic: mmap, size 134217728: cannot allocate memory

goroutine 1014 [running]:
github.com/prometheus/prometheus/tsdb.(*memSeries).mmapCurrentHeadChunk(0x6a400b0, 0x5a2ea20)
/app/tsdb/head.go:2322 +0x22c
github.com/prometheus/prometheus/tsdb.(*memSeries).cutNewHeadChunk(0x6a400b0, 0xbc52a621, 0x179, 0x5a2ea20, 0x0)
/app/tsdb/head.go:2293 +0x24
github.com/prometheus/prometheus/tsdb.(*memSeries).append(0x6a400b0, 0xbc52a621, 0x179, 0x63c72d59, 0x3f3d9bba, 0x0, 0x0, 0x5a2ea20, 0x11de0001)
/app/tsdb/head.go:2449 +0x3a8
github.com/prometheus/prometheus/tsdb.(*Head).processWALSamples(0x5920240, 0xbc497000, 0x179, 0x22b05b00, 0x22b05ac0, 0x0, 0x0)
/app/tsdb/head.go:420 +0x270
github.com/prometheus/prometheus/tsdb.(*Head).loadWAL.func6(0x5920240, 0x219e38e0, 0x219e38f0, 0x22b05b00, 0x22b05ac0)
/app/tsdb/head.go:522 +0x40
created by github.com/prometheus/prometheus/tsdb.(*Head).loadWAL
/app/tsdb/head.go:521 +0x2c4

Maybe it’s a memory issue? there stated at the end of the first block of the log? which seems odd to me since I’ve been running prometheus on this RPi4 (8GB RAM) for months now without issue…

hmm so it can’t load one file. I have no experience with it but if you don’t care much about past records, you could just delete them all and let the db start from zero (or move them all out and move only the healthy ones back).

yes, I saw that part. Although, up until today, I’ve been running prometheus in docker on this raspberry pi, RaspberryPi OS 32bit version for months without issue.

so I’m just really confused as to why this is all of a sudden an issue for me.

yeah, again not sure what happened there, but I went ahead and deleted everything in the “wal” folder in the prometheus repository, and tried restarting and that worked…so looks like I just deleted all of that and probably didn’t need to, but oh well…it’s just graphing data right?

thanks for looking at my logs and responding. As always, appreciate it!

1 Like

Who knows… could be corruption due to an unsafe shutdown or just generally data corruption due to sd-card error or ram error… I must admit, haven’t seen file corruption on a pi yet but it’s not impossible.

1 Like

@Alexey can you please make this topic a wiki? thanks

Done.
You are Regular, you should have these permissions too, especially for your own posts.

Thanks.
Surprisingly I don’t have the perms to do that, there is no wrench symbol under the post where you can change it.

Weird. You should be able to do so… I would ask the admin.
Ah I was wrong. Only moderators can do that for any post. The Regular can do that only in special category…

1 Like

hey there. I was trying to setup prometheus, but the container is constantly restarting and the call of localhost:9090 provides no result. :frowning:

my docker run command looks like the following:

sudo docker run -d -p 9090:9090 --restart unless-stopped --name prometheus -v /mnt/my5tb/prometheus.yml:/etc/prometheus/prometheus.yml -v /mnt/my5tb/prometheus:/prometheus prom/prometheus --storage.tsdb.retention.time=360d --storage.tsdb.retention.size=100GB --config.file=/mnt/my5tb/prometheus.yml --storage.tsdb.path=/mnt/my5tb/prometheus

docker ps results in:

5083ac4be3bd   prom/prometheus                 "/bin/prometheus --s…"   6 minutes ago   Restarting (2) 24 seconds ago                                                                                                  prometheus

OR

5083ac4be3bd   prom/prometheus                 "/bin/prometheus --s…"   4 seconds ago   Up Less than a second   0.0.0.0:9090->9090/tcp, :::9090->9090/tcp                                                      prometheus

curl command with that result:

curl -X POST http://localhost:9090/-/reload

curl: (7) Failed to connect to localhost port 9090: Connection refused

Your docker command doesn’t look right, I would just copy and paste the original one and test that instead.

but then the paths are not correct:

sudo docker run -d -p 9090:9090 --restart unless-stopped --user 1000:1000 --name prometheus -v /sharedfolders/config/prometheus.yml:/etc/prometheus/prometheus.yml -v /sharedfolders/prometheus:/prometheus prom/prometheus --storage.tsdb.retention.time=360d --storage.tsdb.retention.size=30GB --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/prometheus

4d9e8827d82ea71ce9e3d5eacd6674a25b2616e9e693bcd8e3dd694eb9e2d0d5

docker: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:76: mounting "/sharedfolders/config/prometheus.yml" to rootfs at "/etc/prometheus/prometheus.yml" caused: mount through procfd: not a directory: unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type.

Right sorry what I meant to say is copy the original one and fill in your locations for the config.

that’s what I did:

sudo docker run -d -p 9090:9090 --restart unless-stopped --name prometheus -v /mnt/my5tb/prometheus.yml:/etc/prometheus/prometheus.yml -v /mnt/my5tb/prometheus:/prometheus prom/prometheus --storage.tsdb.retention.time=360d --storage.tsdb.retention.size=100GB --config.file=/mnt/my5tb/prometheus.yml --storage.tsdb.path=/mnt/my5tb/prometheus

config path: /mnt/my5tb/prometheus.yml
storage path: /mnt/my5tb/prometheus