Has Storj ever looked into the cctv market?

Ideas & Suggestions
1

We are seeing some actors on the cctv market and those could be interesting for Storj.
Generally speaking, it is common to use the cloud for cctv recordings.

There are manufacturers of cctvs and recording hardware or closed software, which could be interested to directly implement access to the Tardigrade network and maybe earn some comissions.

Also there are cctv cloud provider like https://www.cameraftp.com/ that need a cloud infrastructure for their service, which also might be interested in learning about Tardigrade. This is only one of many providers it seems. (google “nvr cloud” or “cctv cloud”)

And finally, there are Open Source projects like https://github.com/ispysoftware/iSpy or https://github.com/ZoneMinder/ZoneMinder. A Tardigrade access could be built into it directly or via plugins by the Storj team.
@stefanbenten Or maybe a transfer.sh bridge could be developed that uploads cctv recordings automatically.

Maybe Storj wants to have a look at this specific market as cctv operators have specific but homogeneous requirements: Constant recording, constant storage requirement, secure and resiliant, often off location. Review of recorded streams review only occasionally and often can deleted even automatically after a certain period of time.
Just imagine if millions and millions of cctv operators would be able to store or backup their cctv recordings on Tardigrade.

10 Likes
2

Thanks for the suggestion, @jammerdan.

Cctv recordings are a really great use case for storage. The use case positioning is somewhat similiar to database snapshots – whereby for enterprises, cold storage won’t cut it. Video data needs to be readily accessible via internet and often globally available - but the demand on bandwidth isn’t at CDN levels.

This is definitely a market segment that we are pursing - in addition to analogues like autonomous vehicle recordings.

6 Likes
3

Excellent suggestion!

We will also need a bunch of actors going around, making ruckus in round-robin fashion, causing the owners to download their recordings constantly!

5 Likes
4

Have you heard of Comma.ai? Self hosted Driverless learning. Going to make a separate thread about it.

1 Like
5

@jammerdan Not recently, but worth taking a look again. It does seem like a pretty good fit.

1 Like
6

This seems to indeed be a great fit.

I’ve been thinking about this a bit more and then I realized it might be really interesting to contact Wyze specifically. Their usage seems to be ideal, with lots of storage needs for short video clips, which are relatively large files, that are only downloaded by the customer who owns the camera. They’re also expanding into other IoT areas, which no doubt require a lot of storage as well.

After listening to the podcast on use cases it seems like a perfect fit. And since they are a price competitor who specialize in using commodity hardware with superior smart software to deliver a better experience than the competition, I think they would be really interested in a smart way to reduce storage costs.

They currently use AWS as a backend as mentioned on their site, so chances are they could switch to the S3 compatible gateway without too much effort.

Edit: An example of how they like to be clever with software. They recently released firmware you can load onto their security cams to use them as a webcam to counter the webcam shortage the current word wide situation is causing. They seem to be agile and willing to experiment.

2 Likes
7

Great suggestion.
It’s true that if you get the support from the right cctv/nvr market participants, this would guarantee a huge share of the market.

It would require a thorough analysis to understand the market, identify the participants and products to address them. But I think Tardigrade could be really a great fit. And this is a growing market as more and more businesses and home users will be using cctvs.

@super3 I am not a cctv expert and stumbled across this idea only recently. But I would be happy to provide some more links or thoughts in case you are interested.

2 Likes
8

It would be great if we could find a community member that already has use for this.

9

I think Storj should not wait any longer. There is no doubt that there is a market for cctv cloud storage.
Storj must take action not wait.

1 Like
10

@super3
I did a bit more research and maybe entering that market is easier than it seems.
But to write more I need to know if Tardigrade can act as FTP server?

I have seen with V2 there was the Filezilla integration. But I don’t know if this was Filezilla specific.

Is there a general way, Tardigrade can act like a regular FTP server? Or can such a bridge be build? I see that a Filezilla integration for V3 is on the way: Easy to use software for v3 like FileZilla?
Could this to be build to act as FTP server generally?

2 Likes
11

Why the focus on FTP? You never explained this part. Anything can be built. You can just translate FTP protocol commands to storjlib commands.
It can also be hacked together, for example using the S3 gateway, s3fs-fuse and a simple FTP server.

12

@super3

Yeah this is what my research has discovered. It seems that for most cctv cameras FTP is the total standard for uploading stuff to somewhere and therefore already built into many/most of them natively.
You can check out here:

Also in NVR software:

So by providing a FTP gateway, it seems all those cctvs should be able to instantly connect and upload their stuff onto Tardigrade network out of the box.

Only 4 things required from Storj:

  1. A FTP gateway/bridge or whatever it is called.
  2. A really foolproof easy sign up and setup procedure for Tardigrade (satellite selection, bucket creation etc.)
  3. Competitive pricing
  4. Some marketing and advertising on boards like: https://www.cctvforum.com/ or https://www.ipcamtalk.com/ or Blue Iris - Index page

And the money flows in.

3 Likes
13

For an MVP can you find out if any of these support an S3 endpoint?

2 Likes
14

Natively? As far as I understand they don’t.
I looked around and also checked what others are offering. And it is more like this How do images get from my camera to the Sensr.net site?:

We use FTP, a standard way to move files around the internet. Most cameras support the FTP push of images. When you click the new camera link, we’ll generate an FTP user name and password for your camera. You add this information to your camera and tell it to FTP images to us.

2 Likes
15

I have one more for you here: https://reolink.com/how-to-set-up-ip-camera-ftp-servers/

It describes how easily FTP can be setup provided that it has been implemented in the device, here the device is a Reolink cam. It seems that not much more than the FTP server is required. So if Storj can provide Tardigrade as FTP server those cctvs could use it instantly.

16

Below is my complete guide for setting up a Reolink camera to backup to FTP hosted on Tardigrade. The setup was conducted on a fresh Ubuntu 20.04 LTS Server installation. I generally recommend using tmux or byobu to allow one to open multiple shell windows. This guide uses the following components:

Prerequisites

Create and fund a Tardigrade.io account. Create a project and an API Key as shown on the Tardigrade documentation site.

Uplink S3 Gateway and AWS CLI Tool

Install Uplink S3 Gateway.

$ curl -L https://github.com/storj/gateway/releases/latest/download/gateway_linux_amd64.zip -O && unzip gateway_linux_amd64.zip
$ chmod 755 gateway
$ sudo mv gateway /usr/local/bin/gateway

Also install the aws CLI tool.

$ sudo apt install awscli

Complete the S3 Gateway setup with the satellite, API key, and encryption password.

$ gateway setup
Select your satellite:
        [1] us-central-1.tardigrade.io
        [2] europe-west-1.tardigrade.io
        [3] asia-east-1.tardigrade.io
Enter number or satellite address as "<nodeid>@<address>:<port>" [1]: 1
Enter your API key: <api key>
Enter your encryption passphrase:
Enter your encryption passphrase again:

With your permission, Tardigrade can automatically collect anonymized analytics information from your S3 Gateway and send it to Storj Labs (makers of Tardigrade) to help improve the quality and per
formance of our products.
Do you agree to submit such information to Storj Labs (y/N): y

Your S3 Gateway is configured and ready to use!

Some things to try next:

* See https://documentation.tardigrade.io/api-reference/s3-gateway for some example commands

Run the gateway

$ gateway run
2020-05-06T02:00:50.481Z        INFO    Configuration loaded    {"Location": "/home/ubuntu/.local/share/storj/gateway/config.yaml"}
2020-05-06T02:00:50.489Z        INFO    Starting Storj S3-compatible gateway!


2020-05-06T02:00:50.490Z        INFO    Endpoint: 127.0.0.1:7777

2020-05-06T02:00:50.490Z        INFO    Access key: <access key>

2020-05-06T02:00:50.491Z        INFO    Secret key: <secret key>

Configure the aws CLI tool

$ aws configure
AWS Access Key ID [None]: <access key>
AWS Secret Access Key [None]: <secret key>
Default region name [None]:
Default output format [None]:

Create a new bucket in the Tardigrade project for the FTP data.

$ aws s3 --endpoint=http://localhost:7777/ mb s3://ftp
make_bucket: ftp

vsFTPd

Install vsFTPd.

$ sudo apt install vsftpd

Create a vsftpd.conf file, replacing <local IP address> with your local IP address.

$ sudo su -
$ mv /etc/vsftpd.conf /etc/vsftpd.conf.bak
$ cat > /etc/vsftpd.conf << EOF
anonymous_enable=NO
local_enable=YES
write_enable=YES
local_umask=022
dirmessage_enable=YES
xferlog_enable=YES
connect_from_port_20=YES
xferlog_std_format=YES
chroot_local_user=YES
listen=YES
pam_service_name=vsftpd
tcp_wrappers=YES
user_sub_token=\$USER
local_root=/srv/ftp
pasv_min_port=40000
pasv_max_port=50000
pasv_address=<local IP address>
userlist_file=/etc/vsftpd.userlist
userlist_enable=YES
userlist_deny=NO
EOF
$ exit

Add the local FTP user to the vsftpd.userlist file and start the vsftpd service.

$ echo "ubuntu" | sudo tee -a /etc/vsftpd.userlist
ubuntu
$ sudo systemctl enable vsftpd
$ sudo systemctl start vsftpd

Create a local folder for the ubuntu user in the FTP root.

$ sudo mkdir /srv/ftp/ubuntu
$ sudo chown ubuntu:ubuntu /srv/ftp/ubuntu

s3fs

s3fs is used to create a local mount point for the vsFTPd user folder hosted on Tardigrade and exposed via the Uplink S3 Gateway.

Install s3fs.

$ sudo apt install s3fs

Create a credentials file for s3fs with the S3 Access key and Secret key as defined in the Tardigrade gateway tool.

$ echo <access key>:<secret key> > ${HOME}/.passwd-s3fs
$ chmod 600 ${HOME}/.passwd-s3fs

Mount the S3 bucket to the local /srv/ftp/ubuntu directory created during the vsFTPd installation.

$ s3fs ftp /srv/ftp/ubuntu -o passwd_file=${HOME}/.passwd-s3fs \
  -o url=http://localhost:7777/ -o use_path_request_style

Reolink POE Camera FTP Setup

I have a few Reolink POE RLC-420-5MP turret security cameras that I generally access via RTSP but they also allow for FTP connectivity. In the Web UI of the camera, I’ve set the FTP server settings and set the mode to passive (i.e. PASV). For this test, I’m taking a still image every 30 seconds for 24/7 upload.

reolink-ftp
reolink-ftp1607Ă—1513 100 KB

With these settings, I can confirm that the files are showing on the local mountpoint as well as the S3 bucket.

$ tree -r /srv/ftp/ubuntu
.
└── 2020
    └── 05
        └── 05
            ├── Front_Door_01_20200505221130.jpg
            ├── Front_Door_01_20200505221100.jpg
            ├── Front_Door_01_20200505221041.jpg
            └── Front_Door_01_20200505221038.jpg

3 directories, 4 files
$ aws s3 --endpoint=http://localhost:7777/ ls s3://ftp/2020/05/05/
2020-05-06 04:11:34     333027 Front_Door_01_20200505221130.jpg
2020-05-06 04:10:42     262144 Front_Door_01_20200505221038.jpg
2020-05-06 04:10:37          0
2020-05-06 04:11:04     333836 Front_Door_01_20200505221100.jpg
2020-05-06 04:10:45     330283 Front_Door_01_20200505221041.jpg

Hopefully this helps shed some light on how one could leverage FTP to connect CCTV to Tardigrade. I’ll keep this running for a while to test out the stability.

References

Along with links scattered throughout this post, I found the below article instrumental in getting the above setup working for me.

  1. Cloud Academy S3 FTP: Build a Reliable and Inexpensive FTP Server Using Amazon’s S3
13 Likes
17

That’s pretty amazing. Please ping @super3!

2 Likes
18

Since these camers use unencrypted FTP connections, are they typically connecting to an online ftp-server directly? That is quite the security concern imho…

Great tutorial but quite some work to get a local ftp-server running. I think it would be great if the satellites could offer a ftp server you can connect to directly. Maybe by combining the api_key and bucket_name as username (or is there a length limitation?) and the encryption key as password?

I know, never send your encryption key anywhere… especially not over an unencrypted ftp connection (but ultimately it is not different from using any other cloud ftp-server where you send your password over ftp).

This would of course put more load on the satellites as they would be responsible for encrypting and uploading the files to tardigrade. Probably even have to pay for the bandwith this generates. But maybe that service could be offered for an additional fee?

That was just an idea I had seeing that tutorial because there might be people that don’t like to bother setting up a local ftp server or creating a single point of failure in the local LAN. A high availability ftp server directly linking to tardigrade might be more appealing in some scenarios.

1 Like
19

Well that surely is just a demo and nowhere near to a final product. A final concept has to be somewhat similar to the Filezilla integration with V2.

Preferrably even easier without the hassle to register, select satellite, create bucket and stuff on the Tardigrade site.
My vision would be basically to enter the address of the bridge as FTP server, put in your credentials and you are done.

20

Most cameras that support FTP will only be capable of basic unencrypted connections anyways, in which case I’d rather have a local FTP server not exposed to the Internet.

If I were to architect a CCTV system and leverage cloud storage, I probably wouldn’t use FTP, and instead a local server with NVR software that uses Storj on the backend for storing clips, likely leveraging the same s3fs connector.

1 Like