S3 sync does not work

kubotor · November 24, 2021, 12:24am

It just won’t skip the files already present in my storj bucket. It copies them every time over the network.

I’ve used: aws s3 --endpoint=https://gateway.eu1.storjshare.io sync --size-only --delete /mnt/foo s3://bar/bar/

No matter if I use --size-only or --delete. I will transfer the file every time.

Alexey · November 24, 2021, 7:51am

Yes, it’s not preserve the modification time, so the object will be always new.

I would recommend to use rclone sync instead. The rclone preserve the modification time unlike AWS CLI.
If you would choose the s3 protocol again, then you would need to use rclone --no-update-modtime sync.

kubotor · November 27, 2021, 12:00pm

Great, it is working! Thank you.
But I used S3 (again) over rclone, because it eats 1:1 data on upload.

Now unchanged files are skipped.

ALFRED123 · May 30, 2022, 5:43pm

This script synchronizes local directory with storj directory with uplink
It does not add, remove, or update any file in the local directory
It does not remove or update any file in the remote directory .
It just copies new files from local directory to remote storj directory
It just check filename and filepath, not file size or file date creation

#example
python3 ./synchronize_storj.py --local_dir /dir6565/dir773 --remote_dir sj://bucket_845/dir747/dir334 --mode_test_or_real real --show_new_files_yes_or_not not

github.com

financebot/synchronize_storj_with_uplink/blob/93e39e70b7953493d85d3dba8b2abe180d5a2eb7/synchronize_storj.py

import os
import subprocess
import codecs  
import re
import argparse
import time


#PART1. INTRODUCTION
print('\n\n')



#tutorial. example. usage.
example1='python3 /home/user6565/synchronize_storj.py --local_dir /home/user6565/local_directory_67565  --remote_dir sj://remote_bucket_5643/directory_5665 --mode_test_or_real real --show_new_files_yes_or_not yes'



#define argument
parser = argparse.ArgumentParser(description='synchronize local directory with storj directory')

This file has been truncated. show original