Hello,
I currently store +2TB of data on Pinata and I am considering migration to Storj (once i get access to the beta).
I checked the docs and there’s no mention of how to pin CID’s already on the IPFS network.
Ideally I would be doing this: ipfs pin add CID -r but I want to make sure that’s possible or not with Storj.
Thanks
@gregorym We haven’t enabled pinning existing CIDs as it involves DHT search on the network, which requires more effort to make it stable and efficient, especially with many CIDs. We are still evaluating the potential use cases.
How many CIDs do you have in your +2TB of data? If it is a significant number and a one-time activity, I’d recommend just uploading the files to the Storj pinning service. If you uploaded them on Pinata with the default chunker settings, you’d get the same CID when uploading them to Storj.
Would this work for you? If not, let me know what your primary concern is.
I have hundreds of IPFS folders so it’s not ideal but I can work around it.
Just to confirm, can I also up I also upload folders with Storj?
Yes, we support uploading folders as described here: HTTP APIs for IPFS | IPFS Docs
The
addcommand not only allows adding files, but also uploading directories and complex hierarchies.
This happens as follows: Every part in the multipart request is a directory or a file to be added to IPFS.
Directory parts have a special content type
application/x-directory. These parts do not carry any data. The part headers look as follows:
Content-Disposition: form-data; name="file"; filename="folderName"
Content-Type: application/x-directory
File parts carry the file payload after the following headers:
Abspath: /absolute/path/to/file.txt
Content-Disposition: form-data; name="file"; filename="folderName%2Ffile.txt"
Content-Type: application/octet-stream
...contents...
The above file includes its path in the “folderName/file.txt” hierarchy and IPFS will therefore be able to add it inside “folderName”. The parts declaring the directories are optional when they have files inside and will be inferred from the filenames. In any case, a depth-first traversal of the directory tree is recommended to order the different parts making the request.
It’s not as straightforward as executing a single curl command. I can prepare a CLI tool to make that easier.
Could you give me some numbers of your largest folder? The number of files in it and the total size?
In terms of file number, it usually between 1k up to 10k files. In terms of size, up to 25gb.
Wow, this is large! Uploading a folder to IPFS requires the entire content to be in a single HTTP request body. Not impossible, yet it’s large…
Out of curiosity, what is the need for such large folders? Is it bulk NFT minting or is there another reason?
Yes it’s large 
We upload NFT collections on IPFS and some of our users have 10k gifs/mp4s which results in many Gb
What was the exact method you used to upload such a folder to Pinata?
Thanks! What would work best for you: a similar JavaScript example to upload a folder to the Storj pinning service, or a CLI tool?
A Javascript example.
@gregorym The example code you provided above also works well with the Storj IPFS pinning service. It just needs a small change with regards to the authentication method.
Here is a working example for the Storj IPFS pinning service:
// The 'got' module gives a promised-based HTTP client.
const got = require('got');
// The 'fs' built-in module provides us access to the file system.
const fs = require('fs');
// The 'recursive-fs' module provides async recursive file system operations.
const rfs = require("recursive-fs");
// The 'base-path-converter' module trims file paths from a base path.
const basePathConverter = require("base-path-converter");
// The 'form-data' built-in module helps us submit forms and file uploads
// to other web applications.
const FormData = require('form-data');
/**
* Uploads a folder from `folderpath` and pins it to the Storj IPFS pinning service.
* @param {string} username your username for the Storj IPFS pinning service
* @param {string} password your password for the Storj IPFS pinning service
* @param {string} folderpath the path to the folder
*/
async function pinFolderToIPFS(username, password, folderpath) {
try {
// The HTTP upload endpoint of the Storj IPFS pinning service
const url = `https://www.storj-ipfs.com/api/v0/add`;
// Create a form with the folder and its files to upload
let data = new FormData();
const { dirs, files } = await rfs.read(folderpath);
for (const file of files) {
data.append(`file`, fs.createReadStream(file), {
filepath: basePathConverter(folderpath, file),
});
}
// Execute the Upload request to the Storj IPFS pinning service
const response = await got.post(url, {
username: username,
password: password,
headers: {
"Content-Type": `multipart/form-data; boundary=${data._boundary}`,
},
body: data
}).on('uploadProgress', progress => {
console.log(progress);
});
console.log(response.body);
} catch (error) {
console.log(error);
}
};
/**
* The main entry point for the script that checks the command line arguments and
* calls pinFolderToIPFS.
*
* To simplify the example, we don't do any fancy command line parsing. Just three
* positional arguments for username, password, and folder path.
*/
async function main() {
const args = process.argv.slice(2)
if (args.length !== 3) {
console.error(`usage: ${process.argv[0]} ${process.argv[1]} <username> <password> <folderpath>`)
process.exit(1)
}
const [username, password, folderpath] = args
const result = await pinFolderToIPFS(username, password, folderpath)
console.log(result)
}
/**
* Don't forget to actually call the main function!
* We can't `await` things at the top level, so this adds
* a .catch() to grab any errors and print them to the console.
*/
main()
.catch(err => {
console.error(err)
process.exit(1)
})Thank you!
Can I also get the folder cid? I need to be able to do this https://ipfs.io/ipfs/QmXLDiS4ZG8XUNmKELr3GtHT7waWcN2eXVCgcz1DifQwjE/1.json
Also, what’s the data retention policy? The landing page says 30 days but what if I need longer retention?
Yes. The response body will print a line for each added file and folder, its CID, and size. The last line is about the top folder. It will include its CID too.
The 30 days retention policy is supposed to give enough time to explore the product and eventually integrate with it. If you need longer retention, you should talk to a salesperson. Simply reply to the email that provided you with the beta credentials.
Hello, I know this question has been marked as solved. But I have something to add as far potential use cases go. At least, I created this account for your forums to ask about a similar use case.
I’m looking for a pinning service that will allow me to pin an existing IPNS hash. Reason being is for the same reason that axiom, “not your keys, not your crypto” exists.
I’d like to retain control of my IPFS node key, periodically start up my locally controlled node, locally stored keys, publish updated content in my directory published with IPNS, call an API on pinning service(such as storj) to mirror those newly added files, or mirror modified files. Then shut off my local node and know my IPNS signed content will stay online without giving up control of my IPFS keys.
Thanks for reading!