SIDB: Multiple file upload and download

The easiest way to use the Scientific Image Data Base (SIDB) located at the 3Dgenome project server is to upload or download images one by one, using the hypertext transfer protocol (http). Thus, the image is directly uploaded through the web interface, and filed into the database automatically, with the parameters we set. Downloading filed images is also quite easy.

But some users take a lot of images in one night, and they want to file all (or many) of them into the SIDB. If the number of images is too high, uploading them one by one becomes tiresome.

For these users, an alternative exists that can make filing many images very easy. Downloading multiple files is a similar procedure, but with the steps reverted. We will explain uploading first:

It is a two-step procedure:

1. Upload the images to a temporary location in the project server using rsync, and
2. file the images into the SIDB archive, reading them from that location in the server instead of from your local disk.

Direct transfer from your local disk to the SIDB archive is not possible because extra information is needed that you must provide: the metadata to describe the images. This can be typed manually or reused from templates. Please keep reading.

Rsync for the SILS / 3Dgenome project

A special bundle of programs is available to the SILS/3Dgenome users that gathers the necessary binaries to run rsync without using an interface like DeltaCopy. It is basically the ssh_win distribution as available here plus a couple of .bat files to take care of using rsync with the necessary options, and a setup.bat file to help you with the initial configuration (ssh keys and so).

This collection of executables and libraries is intended to provided file synchronization with the 3Dgenome server, with the purpose of uploading and downloading image to the Scientific Image Data Base running there.

It should be already installed in your computer. If not, ask the 3Dgenome project leader for assistance.

Rsync installation notes

Read the instructions in the file 0-README.txt that goes along with that distribution. Run setup.bat and follow the instructions on the screen.

How to proceed after setup.bat:

In the transfer directories (d:\3DgUpload and d:\3DgDownload) you have to create subfolders for the different users using the same login name as in the remote SIDB. Mind the letter case!

Modify 3DgDownload.bat (only that file) to specify your personal remote folder there. Replace USERNAME (twice) after 3DgDownload with yours.

The .bat files expect the transfer directories to be in the disk unit D. Edit as necessary if that is not the case.

Details about these .bat files included below.

Uploading images

Step 1: Transfer the files to the server

To upload the contents of your local transfer directory just run 3DgUpload.bat. Files will be moved to the remote server, to a temporary directory where the SIDB can find them and handle them properly.

Step 2: File the images in the SIDB archive

1. Login the SIDB (http://3dgenome.uva.sara.nl/sidb with your SIDB username and password.
2. Select the menu item 'upload', then select 'Move file from an image directory' instead of 'HTML file upload'.
3. Like always, select the file type, and describe the images by typing the relevant image parameters in the appropiate text boxes. (Or use a template that you have defined previosly). 'Be careful': if you make any mistake in the image description, it will be applied to ALL the images you are filing, and it will be difficult to correct later. Describe your images carefully!.
4. Select the files you want to file in the database. Here you can make a multiple selection, by keeping the Ctrl key pressed-down. Usually, you will select all the files that you have transfered in Step 1. All the images you select must be of the same type!. Also, it is recommendable that they were taken under the same conditions, as they will have the same description parameters. Otherwise, you must modify the description later, in every image.
5. Finally, push the "upload" button, and let the SIDB do its job. The files will be transferred into the SIDB, and deleted from your temporary upload directory, so it will be empty and ready for the next time.

A detailed tutorial for step 2 can be found in Sidb Upload To The Archive.

Downloading images

You can always download a single image with the http protocol, but that is again very tedious if you have many of them.

Downloading many files simultaneously is also a two-step procedure: first you place files in your temporary transfer directory on the server, then download them from there to your local computer using rsync.

To download the exported files available in the remote server, run 3DgDownload.bat.

Basic questions

• To learn how to use the SIDB read the Sidb Tutorial.
• The project SIDB is located at http://3dgenome.uva.sara.nl/sidb

Troubleshooting

In this section we will gather instructions for troubleshooting as we gain experience with the program usage. You are welcome to edit this article and improve it based on your experience.

Contents of the batch files

The upload script basically runs:

rsync.exe  -v -rlt -z -p --remove-sent-files --chmod=ugo=rwX "/cygdrive/d/UPDIR/" "USERNAME@3dgenome.uva.sara.nl:/home/USERNAME/UPDIR/"
@pause > nul

The download script does:

rsync.exe  -v -rlt -z -p --remove-sent-files "USERNAME@3dgenome.uva.sara.nl:/home/USERNAME/DOWNDIR/USERNAME"  "/cygdrive/d/DOWNDIR/USERNAME"
@pause > nul

The files are deleted from the source after copied to the destination, to prevent duplicated transfers later. In this sense rsync is just used as an effective scp, but not real synchronization is executed (in the sense that there are not two versions of the same file to compare on both sides of the line).

The setup.bat script does:

@echo off
echo Creating transfer directories: \UPDIR and \DOWNDIR.
mkdir \UPDIR
mkdir \DOWNDIR
echo.
echo Creating \usr\bin to enable scp (in case you need it).
mkdir \usr
mkdir \usr\bin
copy ssh.exe \usr\bin\
echo.
echo ************************************************
echo Generating ssh key. Do not enter any passphrase!
echo      (Just press return to all questions).
echo ************************************************
ssh-keygen -t dsa
chmod 600 ~/.ssh/id_dsa.pub
chmod 600 ~/.ssh/id_dsa
cp ~/.ssh/id_dsa.pub .
echo.
echo ******************************************************************
echo A file named id_dsa.pub should have been copied to this directory.
echo Send it to the administrator.
echo ******************************************************************
echo.
echo (He will just append it to the server ~/.ssh/authotized_keys2,
echo you can do it yourself if you know how to).
echo When this key exchange is done, you should be able to run
echo 3DgUpload.bat without being asked for a password.
echo.
echo. Press any key to finish
pause > nul

More questions?

Contact Jose Viña.