So far the only commonly seen cause for memory usage increasing is if your storage isn’t fast enough. A node operates on a large number of small-ish files, hence it needs a decent random read/write performance—raw HDD is fine, but some additional software layers may degrade it too much. For example, using a lazily-allocated qcow image might be the reason. You can test this hypothesis e.g. by running iostat -dmx inside your VM and observing the %util column, if it is >10% I’d be worried.
How is your storage configured inside the VM?
What are the models of your HDDs?
BTW, if this VM is dedicated to hosting a node, you should be fine with 2 GiB and a single core. I’ve got an old server with 2 GiB and two cores operating a total of 12TB in Storj nodes and the CPU usage rarely crosses 10%. What’s more is that using two sockets (as opposed to a single socket with two cores) in Proxmox is discouraged unless the client VM software really needs it—and Storj nodes don’t.