Skip to main content

Lyve Cloud Documentation

Using Rclone

Use rclone to connect with Lyve Cloud and manage your files from the command line, or mount the cloud storage as a drive.

Prerequisites

  • You will need the access key and secret key for each account you’ll be using to connect with Rclone. For more information, see Creating service accounts

Note

Consult your organization's policies and the EULA policies of the software before downloading 3rd-party applications.

Connecting to Lyve Cloud from Linux
Installing Rclone

  1. Download rclone for Linux, then extract the rclone binary to your desired location.

  2. To use rclone, open a terminal window and navigate to the directory where you saved the executable.

    Configuring rclone to connect to Lyve Cloud
    Procedure. To configure a remote connection with rclone:

    1. Run rclone config to setup and select n for a new remote.

      No remotes found - make a new one       
n) New remote        
s) Set configuration password        
q) Quit config       
n/s/q> n

    2. Enter a name for the configuration.

      name> <Name>

    3. Select s3 storage.

      Type the storage to configure.
Choose a number from below, or type in your own value        
1 /Fichier 
 \(fichier)  
2 /Akamai NetStorage
  \(netstorage) 
3 / Alias for an existing remote 
 \ (alias)        
4 / Amazon Drive 
 \ (amazon cloud drive)        
5 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, Digital Ocean, Dreamhost, IBM COS,  Lyve Cloud, Minio,  RackCorp, SeaweedFS, and Tencent COS  
 \ (s3)        
6 / Backblaze B2   
 \ (b2)  [snip]        
46 / seafilehttp Connection 
  \ (seafile)
 Storage> 5

    4. Choose Lyve Cloud as the storage provider.

      Choose the S3 provider.
Choose a number from below, or type in your own value
Press Enter for the default ("")
  1 / Amazon Web Services (AWS) S3
   \ (AWS)
 2 / Alibaba Cloud Object Storage System (OSS) formerly Aliyun
   \ (Alibaba)
 3 / Ceph Object Storage
   \ (Ceph)
 4 / Digital Ocean Spaces
   \ (DigitalOcean)
 5 / Dreamhost DreamObjects
   \ (Dreamhost)
 6 / IBM COS S3
   \ (IBMCOS)
 7 / Seagate Lyve Cloud
   \ (LyveCloud)
 8 / Minio Object Storage
   \ (Minio)
 9 / Netease Object Storage (NOS)
   \ (Netease)
10 / RackCorp Object Storage
   \ (RackCorp)
11 / Scaleway Object Storage
   \ (Scaleway)
12 / SeaweedFS S3
   \ (SeaweedFS)
13 / StackPath Object Storage
   \ (StackPath)
14 / Storj (S3 Compatible Gateway)
   \ (Storj)
15 / Tencent Cloud Object Storage (COS)
   \ (TencentCOS)
16 / Wasabi Object Storage
   \ (Wasabi)
17 / Any other S3 compatible provider
   \ (Other)          
Provider>7

    5. Enter false to enter your credentials.

      Get AWS credentials from the runtime (environment variables or EC2/ECS meta data if no env vars). 
Only applies if access_key_id and secret_access_key is blank.Enter a boolean value (true or false). 
Please Enter for the default ("false").Choose a number from below, or type in your own value     
   1 / Enter AWS credentials in the next step     
    \ "false"  
   2 / Get AWS credentials from the environment (env vars or IAM) 
     \ "true"
   env_auth>false

    6. Enter your access key and secret key.

      AWS Access Key ID.
Leave blank for anonymous access or runtime credentials.
Enter a string value. Press Enter for the default ("") 
    access_key_id> <access key>
AWS Secret Access Key (password)
Leave blank for anonymous access or runtime credentials.
Enter a string value. Press Enter for the default ("")
     secret_access_key> <secret key>

    7. Leave the region blank.

      Region to connect to.
Leave blank if you are using an S3 clone and you don't have a region.
Enter a string value. Press Enter for the default ("")
Choose a number from below, or type in your own value
1 / Use this if unsure. 
  | Will use v4 signatures and an empty region.
  \ ()
2 / Use this only if v4 signatures don't work. 
  | E.g. pre Jewel/v10 CEPH.
  \ "other-v2-signature"
region> <>

    8. Specify the endpoint for Lyve Cloud. For more information about endpoints, see S3 API endpoints

      • Currently, Lyve Cloud supports only region-specific S3 endpoints. To access buckets created in different regions in the S3 client, add an endpoint connection for each of the regions.

      Endpoint for S3 API.
Required when using an S3 clone.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
 1 / Seagate Lyve Cloud US East 1 (Virginia)
   \ (s3.us-east-1.lyvecloud.seagate.com)
 2 / Seagate Lyve Cloud US West 1 (California)
   \ (s3.us-west-1.lyvecloud.seagate.com)
 3 / Seagate Lyve Cloud AP Southeast 1 (Singapore)
   \ (s3.ap-southeast-1.lyvecloud.seagate.com)
endpoint> 1

    9. Press Enter to skip the location constraint as there is no location constraint.

      Location constraint - must be set to match the Region.
Leave blank if not sure. Used when creating buckets only.
Enter a string value. 
Press Enter for the default ("")location constraint>

    10. Choose default ACL (private).

      Canned ACL used when creating and or storing or copying objects.
This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too.
Note that this ACL is applied when server-side copying objects as S3
It doesn't copy the ACL from the source but rather writes a fresh one.
Enter a string value. Press Enter for the default ("")
Choose a number from below, or type in your own value   
1 / Owner gets FULL_CONTROL. 
  | No one else has access rights (default). 
   \(private)   
2 / Owner gets FULL_CONTROL. 
  | The ALLUsers group gets READ access. 
  \(public-read)    
3 /Owner gets FULL_CONTROL. 
  | The ALLUsers group gets READ and WRITE access.   
[snip]
acl>1

    11. Select n to save the default advanced configuration.

      Edit advanced config? (y/n)
y) Yes
n) No (default)
y/n>n

    12. Review the displayed configuration and accept to save the remote and then quit. The config file should look like this:

      NAME]
    type = s3
    Provider = LyveCloud
    env_auth = false
    access_key_id = xxx
    secret_access_key = yyy
    region = us-west-1
    endpoint = s3.us-east-1.lyvecloud.seagate.com
    acl = private

    13. Click y to confirm the configuration.

      y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d>y

    14. Type q to quit the configuration, else select any of the following to edit, delete, rename, copy, Set configuration password.

      Current remotes:

Name                 Type
====                 ====
ashrcl               s3

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q>q
      How to configure Rclone

      Mounting Lyve Cloud as a drive

      Prior to mounting Lyve Cloud as a drive, connect and test the connection by viewing the contents of one or more buckets using the  rclone ls command.

      . To mount Lyve Cloud as a drive, use this command where:

      • remote is the name of the remote server

      • path/to/files is the exact path to the bucket(s)

      • path/to/local/mount is the local directory:

        rclone mount remote:path/to/files /path/to/local/mount

      For more information, see rclone’s mount command documentation.

      Managing data

      Here are several of the more commonly-needed commands for viewing and managing your data from the command line. See the rclone docs, including information on global flags, for additional information.

      Viewing information about your buckets and directories

      There are three list commands with easily readable output available: ls, lsd, and lsl.

      1. To list all data in a certain bucket, where remote is the name of the remote and path is the name of the bucket:

        rclone ls remote:path [flags] 

      2. To list the directories in a certain remote and see the total directory size, modification time, and number of objects in the directories

        rclone lsd remote:path [flags] 

        Or

        rclone lsd remote: [flags]

      3. To list all objects in a certain remote and see modification time, size and path where path is the remote path beginning with the bucket name. Any of the filtering options can be applied to this command.

        rclone lsl remote:path [flags] 

      4. Learn more about ls, lsd, and lsl.

        How to use Rclone List commands?

        Uploading data to a bucket

        Note

        The object name can contain any of these special characters like @, #, *, $, %, &amp;amp;, !, ?, , , ;, ’, ”, |, +, =, &amp;lt;, &amp;gt;, ^, (, ), {, }, [, ] and alphanumeric characters like 0-9, a-z, A-Z . However, using any of these characters can cause issues due to limiting factors of S3 client SDK .

        1. To upload data into a bucket, use either of these commands:

          rclone copy C:/path/to/filename remote:path [flags] 
rclone copy filename remote:path 

        2. You can also use rclone copy to copy a file or directory to a new location and rename the directory at the same time. Neither of these commands deletes the file from the source, and neither of these commands will copy unchanged files.

        3. Learn more about copy and copyto.

          How to use Rclone Copy-to and Copy-sync commands?

          Downloading data to local storage

          This is the same as copying, but the source path is something in the remote or in a bucket, while the destination path is on your local storage.

          rclone copy remote:path C:/path/to/filename [flags] 
          Delete bucket data

          • To delete files in a certain path from a certain bucket:

            rclone delete remote:path [flags]

          • You can use flags to delete only files with certain characteristics:

            For example, to delete files that are over 100MB:

            rclone --min-size 100MB delete remote:path 

            To delete only a specific file:

            rclone deletefile remote:path [flags]

          Learn more about rclone delete and deletefile.

          Creating a new folder
          . To create a new folder

          • For example, to create a file named blue in the current location:

            rclone mkdir blue

          Create folders in other paths, or with other permissions, by setting the proper flags. Learn more about mkdir.

          Delete a remote from Lyve Cloud

          Note

          rclone will also disconnect whenever you shut down your computer.

          1. To delete a remote from Lyve Cloud, delete the remote’s name using this command, changing REMOTE_NAME to the name of the remote to disconnect:

            rclone config delete REMOTE_NAME [flags] \

          2. Learn more about config delete.

            Deleting a folder

            • To delete an empty folder.

              For example, to delete a file named blue in the current location:

              rclone rmdir blue 

            • Add flags to delete folders in locations other than the current directory.

              For example, to delete an empty folder named blue that contains other empty folders:

              rclone rmdirs blue 

            The folders must be empty for rmdiror rmdirsto work.

            Learn more about rmdirand rmdirs.

            Copying data

            1. First, connect rclone to Lyve Cloud. For step by step instructions, see, Configuring rclone to connect to Lyve Cloud. Once the configuration is complete, the rclone.config file must be updated as:

              [REMOTE NAME]type = s3provider = Otherenv_auth = falseaccess_key_id = XXXXXXXXXXsecret_access_key = YYYYYYYYYYYYYYYYYYYYYYYYYYYendpoint = https://s3.us-east-1.lyvecloud.seagate.comacl = privateregion = us-east-1

            2. Start copying data from the existing cloud provider buckets to the buckets created in Lyve Cloud.

            3. To copy data

              $ rclone copy  SOURCE REMOTE:[SOURCE BUCKET] <TARGET REMOTE>[TARGET BUCKET>/<PREFIX>]

              1. To copy all the data, including prefixes, from the source bucket to the target bucket.

                $ rclone copy  SRT:[ SB ]   TRT:[TB]

              2. To copy all objects with a prefix to the target.

                $ rclone copy  SRT:[ SB ]/mypath1  TRT:[TB]/mypath1
              Deleting a remote from Lyve Cloud

              To delete a remote from Lyve Cloud, delete the remote’s name using this command, changing REMOTE_NAME to the name of the remote to disconnect:

              rclone config delete REMOTE_NAME [flags] 

              Learn more about config delete.

              Note

              rclone will also disconnect whenever you shut down your computer.

              Migrating Data
              Procedure. To migrate data

              1. Set the Source and the Target as remote. For more information, see Configuring rclone to connect to Lyve Cloud.

              2. Sync the Source and Target remote using rclone sync command.

                rclone sync <source remote name>:path <target remote name>:path

              3. Once the source and the target are synced, all the data from the source is copied, removed or migrated to the target remotely.

              4. You can use the following flags in the command to check the status of the sync/copy/migration.

                • --progress Displays the real-time transfer progress.

                • --interactive: Enables interactive mode and displays interactive for every action taken.

                For more information on RClone, see https://rclone.org/s3/

                How to use Rclone Delete and Purge commands?

                In this section: