With much excitement, I finally got Storj up and running well for the first time yesterday. But had to reboot TrueNAS. When it restarted, all my apps deployed successfully except Storj which seemed stuck Deploying. When I went to check the logs, the container was gone. Can’t find anything about this in docs.
You may want to ask about it with TrueNAS support,.as they built the interface to run a Storj node in TrueNAS.
2 Likes
Yes, I tried that first without any responses–this was a Hail Mary attempt. I just hate to have to give up on Storj until I master TrueNAS or Docker.
With docker it could be much simpler, because you will be independent on UI, you can control every single parameter, not what they implemented.
However, as I explained it in your support request you need to check logs during deploy to catch an error, or at least check Application Events, they could have some information why is your TrueNAS unable to start the application, or what happened with the containers.
I guess the pod is restarting, but since it’s kubernetes-backed, it will have a different name every restart, so you need to catch the error before the pod is disappear.
You may also configure to redirect logs to the file:
- Add variable
STORJ_LOG_OUTPUTwith value/app/config/storagenode.login the application settings - Start the application
- You may check logs from the TrueNAS Shell in the storage pool for this application. For example:
tail -f /mnt/data/ix-applications/releases/storj/volumes/ix_volumes/ix_data/storagenode.log
Your path may be different, it’s a default path for the dataset named data. You may name it differently, so your actual path would be different too.
1 Like
I’ve been running Storj since TrueNAS Pointed it out. Only thing I will say is immediately after a reboot your TrueNAS Scale App catalog is not running yet, and if you log in quickly and check, it’ll be blank.
It takes a few minutes for services to start up, and then things appear. I don’t know if that is what you experienced but wanted you to be aware of it.
1 Like
I second this. The storj-IXsystem integration is a joke across the board. They did not even provide cloud sync properly — forcing you to use S3, not even giving a choice. From a quality and stability focused company that looks like marketing folks pushed engineering “do something” and they did it in between actual work.
On Scale - you can run storagenode container if you want. On Core (which is superior system in my opinion anyway — why would you want scale?!) — install storj natively in the jail. Because even docker is too much extra unnecessary complexity, let alone this superficial “integration”, that only muddies the waters for no benefits in return.
Now if something happens you have to not only triage node but also docker and/or IX systems “integration”. No thank you. /rant.
Not even sure why, but the node is up and running again!
So I am using Scale. I chose it, since I have some familiarity with Linux and wanted to use it to advance that familiarity, rather than beginning from scratch with another OS. My TrueNAS is basically a playground for my learning (only about a year into computers at all).
Could you say a bit more about “runn[ing] it natively” in Scale?
I don’t use linux, as I strongly prefer BSD, but the idea being storagenode is a go application, self-contained, with no dependencies, it does not benefit from containerization.
As such, you can run it directly with your OS’es system scheduler (systemd on linux) and avoid overhead of containers, issues related to volumes, ports, fastOpens, etc etc. This is how I do it on FreeBSD (TrueNAS core) GitHub - arrogantrabbit/freebsd_storj_installer: Installer script for Storj on FreeBSD
The value of TrueNAS is that the team spent considerable amount of time cherrypicking components, versions, and patches, and tweaking configuration to get a rock solid storage appliance. If you noticed, you can’t upgrade packages by default, because this would destabilize the system. It is recommended to do all applications in jail/container, as to not screw up dependencies on main OS. That said – you don’t have any wiggle room for experimentation. Might as well install Fedora/rhel/oracle and learn linux specifics there.
My advice would be to drop linux (world has adopted linux for all the wrong reasons, the same way world adopted windows for all the wrong reasons). If you want to learn technology I’d recommend learning FreeBSD – the documentation is excellent, and that experience will somewhat transfer to macOS. I worked with linux, I developed kernel drivers on linux as part of my past jobs, and I wholeheartedly dislike everything about it. But that’s my personal take from my personal experience. YMMV
I mean if you want to work for a living having FreeBSD on your resume does not help near as much as Linux.
I agree with the MacOS / FreeBSD connection being useful but surprised at your ‘all the wrong reasons’ statement.
I’m curious why you’d say that ‘all the wrong reasons’. It isn’t a take I’ve seen very often. I entered the IT industry right as Windows 3 came out and have ridden the rollercoaster in the corporate world working at a system vendor and at software companies.
I disagree. I subscribe to and follow an entirely different approach: it’ best to be very good in some niche obscure field, than do whatever everyone else is doing and compete in the same rat race. Yes, someone who can program in Java on Linux will have significantly wider range of potential employers than someone who is a professional Cobol architect. But in the former case you are competing with every college graduate and high schooler on the planet in a race to the bottom. In the latter – you have virtually no competition, and those two employers who want you – will pay anything you ask. And you only need one to begin with.
Linux became popular and gain wide adoption because it was free. That’s it. That’s the only reason. Not on merit. Only recently it started getting features that have been part of Solaris and FreeBSD for ages. One example: proper implementation of multiprocessing was done by Sun microsystem. Another example: dtrace. Recently linux got something similar (I forgot the name) but it took decades to happen. How can you analyze and debug system performance without probes into kernel?! Or FreeBSD Boot Environments. Why is it still not part of linux today? Why do you have to jump through hoops if you want to mess with your system and/or kernel? Linux is a haphazardly put together mess, always was, still is.
Among this chaos RHEL and related swarm of OSes stand out of course, so if I’m forced to use linux - that probably will be my reluctant choice. Mostly due to its stability. It’s still linux kernel.
[Later wider adoption among general public as a desktop OS was facilitated by reverse engineered Nvidia drivers, and opensource AMD, but from the desktop perspective internals don’t matter: you just need a solid, fast, and stable GUI UX. At that point linux “tipped”. [And if you are after user experience – why bother with Gnome, let alone all the other KDEs, just get macOS.]]
People just don’t know what they are missing. And this is just scratching the surface. The display/graphics subsystem design on linux is the mess on the whole new level.
The same reason was responsible for Windows Phone losing to Android in mass market cell phone markets – one had a $15 per phone OEM license fee, and another was free (I can talk about Windows Phone for hours, and there were many more reasons why it tanked, while being years ahead of contemporary iOS, let alone Android, technologically; this one just came to mind in the context of the present discussion)
Software engineering side of my career started with window 95 OSR2 and user-space app development, and gradually shifted closer to hardware and into the kernel, first on Windows, then variety of small OSes – like BREW, then Windows CE kernel, then Android kernel and frameworks, then Windows Phone and RT, then briefly Android again (at the time I worked for a major SoC vendor) and I’m continuing in the same vein, for almost a decade, deep in the kernel of yet another modern operating system, at another company.
This what formed my opinion on the design of each OS; I’ve seen each one from inside, in action, every day, hands-on, for decades. And I can tell you confidently, if someone offers me android/linux job – I will (and did) reject it. Regardless of compensation. That’s how much I disagree with linux (and windows, since after Win8. Vista was the best thing that Microsoft did, and then they botched it with this win10 nonsense few years later. Microsoft is a separate story of course.). Last time I looked at Android was years after Window Phone died, at it was still behind in many aspects, including in the graphics pipeline.
Back to the job market discussion: I’m horrible at any modern programming languages, modern frontend frameworks, web services, and all that – jus have no interest in all of that; C, C++, lldb, dtrace, and maybe a little bit of python, is what I used daily, and as a result I only have pretty much just a handful of employers available to me if I ever wanted to change my current job. But again, I just need one.
In other words, I strongly oppose the “do whatever everyone else is doing because 1000 lemmings can’t be wrong” approach. One should do what they feel passionate about, that is the only criteria. If that happens to be a unique niche – awesome. if this happens to be a current fad – doesn’t matter either. If you love it you’'ll be great at it.
In the context of the resume – having FreeBSD and Windows Phone on your resume will in fact make it stand out among thousands of cookie cutter “linux java docker backend developers”. Ultimately, the companies you really want to work for don’t hire based on skills the person currently happens to possess, but ability to learn and adapt.
As I have mentioned before I also started in *BSD. First on Ultrix (which has BSD roots) and then at various times ran all three of the majors for various uses. (NetBSD,OpenBSD,FreeBSD) NetBSD used to be my major platform. OpenBSD was secondary and FreeBSD was last. I am not going to say I am a fan of every direction Linux goes in. I am not and have never been a fan of the systemd expansion for one. However, if you take the view of TrueNAS that iX does that it is an appliance and in no way a hand customizable system then Linux is the better choice - if only from the perspective of better hardware support at the enterprise level. FreeBSD is just fussier about what hardware it runs on.
I did run TrueNAS Core at home for some years but as soon as Scale was viable I switched. It just gave me support for more of the hardware I had - particularly 10Gbit adapters. Indeed my switch was to resolve an issue with the Broadcomm NIC adapter in my Proliants. I could make it work - but every upgrade would break it again. It became annoying. Linux just didn’t have that issue with that hardware.
I’m not going to say Linux is the be all and end all here. I remember running Tru64/OSF1/Digital Unix on my Alpha;s some years ago and then trying one system on Linux. Digital Unix kicked it’s bum to be honest performance wise. AdvFS was around long before zfs but offered some very neat features for the time. Even using the same gcc compiler on Linux that DU was using the DU version was some 3 to 5 times faster on the same hardware.
But for my NAS boxes, TrueNAS Scale is a better fit for me than Core was. So long as you remember it is an appliance still.
1 Like
I understand, but on the other hand, specifically because it’s an appliance, and iXSystems controls everything – software, hardware, and services – the more important point becomes “what’s easier to stabilize and support”. They can always pick supported and well established hardware and thus minimize support volume and improve reliability.
Drivers for network equipment is rarely a problem today, most reputable vendors either already work right away, or provide driver source (in other words, the fact that Realtek support is horrific speaks mostly about Realtek than anything else; Windows OSes, for example, support Realtek though Microsoft and Realtek drivers allegedly well, but I would never let that silicon be near anything important regardless).
Apple, btw, does exactly the same thing with macOS/darwin, and it seems to work just fine. Intel Macs use mostly off the shelf components for wifi, ethernet, display, even GPUs. Linux on the other hand did not have a way to make a jelly bean wifi adapter work out of the box (or at all) for an obscenely long time… In other words, this is specific to how a company approaches the product development, and not underlying OS.
In the end, everything can be stabilized, the question is only what amount of effort it requires. I’m actually curious why did ixSystem choose freebsd to begin with, and not up and coming linux. Perhaps all those reasons still stand today?
You have mentioned Realtek before but the vendors I have seen issue with are far more Enterprise level and include both Brocade and Broadcom. i.e. Vendors that at least two major Enterprise hardware suppliers use. (Referring to both HP and Dell here)
I know at a minimum the Broadcom issues are yet to be resolved.
Yeah. Well, linux won, I’m not denying it. I just don’t like it 
Of course Broadcom does not have any incentive to fix issues on the OS for three low volume customers, and ixsystem is not in th position to write and support their own drivers either. Once the OS adoption tips, it self-sustains.
Exactly the same as I was not liking that Alpha was shutdown in favor of the sh*tshow that became Itanium. lol Some of the DEC/Compaq Engineers from Alpha went on to work on the Intel Core and I series series so they certainly had an impact.
1 Like