How to use Nextcloud with rclone¶
rclone is a tool that enables you to synchronize contents between your machine and a remote storage. It is kind of an rsync but for remote storages. Although we will demonstrate here how to use it with Nextcloud, it can be used with many different remote storages (Dropbox, Google Drive, Amazon S3, etc)
1. Installing rclone¶
All applications in the AI4OS Dashboard are packed in a Docker image and have rclone installed by default. If you want to create a Docker containing your own application, you should install rclone in the container to be able to access the data stored remotely. When developing an application with the AI4OS Modules Template, the Dockerfile already includes installation of rclone.
To install rclone on a Docker container you should add the following code to your Dockerfile:
RUN curl https://rclone.org/install.sh | bash
To install it locally on your machine:
$ curl https://rclone.org/install.sh | sudo bash
2. Configuring rclone¶
… in a Dashboard deployment¶
Before creating the deployment you must do the following:
Sync Nextcloud from the profile section. This will create your rclone credentials.
Select Nextcloud when configuring the deployment. This will set set up rclone configuration for you via setting environment variables in your deployment.
When the deployment is created, you should have your env variables available to work with the latest RCLONE:
$ printenv | grep RCLONE_CONFIG_RSHARE_
RCLONE_CONFIG_RSHARE_VENDOR=nextcloud
RCLONE_CONFIG_RSHARE_PASS=<YOUR-PASSWORD>
RCLONE_CONFIG_RSHARE_URL=https://share.services.ai4os.eu/remote.php/dav/files/<YOUR-USER>
RCLONE_CONFIG_RSHARE_TYPE=webdav
RCLONE_CONFIG_RSHARE_USER=<YOUR-USER>
ㅤㅤ A note on older RCLONE versions
In RCLONE versions =< 1.62.2
, the RCLONE environment variables had the following shape (issue).
RCLONE_CONFIG_RSHARE_VENDOR=nextcloud
RCLONE_CONFIG_RSHARE_PASS=<YOUR-PASSWORD>
RCLONE_CONFIG_RSHARE_URL=https://share.services.ai4os.eu/remote.php/webdav/
RCLONE_CONFIG_RSHARE_TYPE=webdav
RCLONE_CONFIG_RSHARE_USER=<YOUR-USER>
So in your module uses an old version of RCLONE, you can either:
Update to a newer RCLONE version,
Still use and old version but make sure to update your
.bashrc
with these variables.
… in your local machine¶
First, you need to generate your RCLONE credentials. For this, log into the AI4OS Nextcloud, go to (1) Settings (top right corner) ➜ (2) Security ➜ (3) Devices & sessions. Set a name for your application (typically in the docs we will use rshare
) and click on Create new app password. This will generate your <user>
and <password>
credentials. Your username should start with EGI_Checkin-...
.
Then run rclone config
command, these are the answers you should provide:
$ rclone config
# choose "n" for "New remote"
# choose name for AI4OS Nextcloud --> rshare
# choose "Type of Storage" --> Webdav
# provide AI4OS Nextcloud URL for webdav access --> ttps://share.services.ai4os.eu/remote.php/dav/files/<YOUR-USER>
# choose Vendor --> Nextcloud
# specify "user" --> (see `<user>` in "Configuring rclone" above).
# password --> y (Yes type in my own password)
# specify "password" --> (see `<password>` in "Configuring rclone" above).
# bearer token --> ""
# Edit advanced config? --> n (No)
# Remote config --> y (Yes this is OK)
# Current remotes --> q (Quit config)
This will create an configuration file in $HOME/.config/rclone/rclone.conf
.:
[rshare]
type = webdav
url = https://share.services.ai4os.eu/remote.php/dav/files/<YOUR-USER>
vendor = nextcloud
user = <YOUR-USER>
pass = <YOUR-PASSWORD> --> this is equivalent to `rclone obscure <password>`
Security warning
For security reasons, the rclone.conf
should never be saved as part of the Docker image. If you are running rclone from inside a Docker container, you should mount rclone.conf
at runtime directly as a volume.
$ docker run -ti -v $HOSTDIR_WITH_RCLONE_CONF/rclone.conf:/$HOME/.config/rclone/rclone.conf <your-docker-image>
One can also mount the rclone.conf
file at a custom location and tell rclone where to find it:
$ docker run -ti -v $HOSTDIR_WITH_RCLONE_CONF/rclone.conf:/custom/path/to/rclone.conf <your-docker-image>
$ rclone --config /custom/path/to/rclone.conf
3. Using rclone¶
You can check that everything works fine with:
$ rclone listremotes # check you don't have two remote storages with same name
$ rclone about rshare: # should output your used space in Nextcloud.
Tip
If listremotes
is listing two remotes with the same name you probably configured the rclone twice. Most likely you ran rclone config
on a machine deployed with the Dashboard, so you have both the env
and rclone.conf
configurations. To fix this, either remove the env
variables (echo unset
command into the .bashrc
) or delete the rclone.conf
file.
You can start copying files from your remote to your local:
$ rclone copy rshare:/some/remote/path /some/local/path
Tip
Uploading to Nextcloud can be particularly slow if your dataset is composed of lots of small files. Considering zipping your folder before uploading.
$ zip -r <foldername>.zip <foldername>
$ unzip <foldername>.zip