We are working on using iRODS, as a way to increase storage for users of the HUB. Possible uses include simple staging of data for simulation, and housing searchable collections of documents. Some theory about our approach can be found in the paper here.
We were using two servers to test iRods at CCR. The first was both a data and metadata server. The second was just a data server. Now, we upgraded to iRODS 3.0 and are only using one server.
Setup for New User
Ask the administrator to create an irods account for you with a password you’d like.
File: irodsEnv (435 B , uploaded by Kyle Marcus 1 decade 2 years ago)
Modify the user name inside the file linked above, and change the file name to .irodsEnv
Create a folder name .irods in the user’s home directory and place .irodsEnv in the folder.
Create a folder named irods in your home directory. This folder will be used as the mount location for the the iRODS filesystem.
Type the command use irods in your workspace command line to be able to run icommands.
To make your password persist, type iinit and enter your irods password.
Create a bash profile file name .bash_profile with the following content:
if [ -n $IRODS_MOUNT -a -e $IRODS_MOUNT ]; then $IRODS_MOUNT fi
Then, run the command source .bash_profile
Finally, if you type ls irods from your home directory, you should see all the users’ home folders in the irods directory, including your own.
.bashrc files for users
Once icommands and irodsFs are installed into /apps the script will read similar to this:
PATH=/apps/iRODS/clients/icommands/bin:/apps/iRODS/clients/fuse/bin:$PATH irodsFs $HOME/irods 1>/dev/null 2>&1
submit and irods
Make sure preprocess_irods.sh and postprocess_irods.sh are set to executable
chmod +x preprocess_irods.sh chmod +x postprocess_irods.sh
Also when using the submit command, be sure to include the preprocess_irods.sh and postprocess_irods.sh files
submit ... -i preprocess_irods.sh -i postprocess_irods.sh ...
If you are placing the output of a job into iRods, make sure to use the -q option with submit so that there will be no limit on the file size that you can create.
modification to /san/user/vhub/u2/bin/receiveinput.sh
IRODS_INPUT_NAME="preprocess_irods.sh" if [ -x ./$IRODS_INPUT_NAME ]; then ./$IRODS_INPUT_NAME fi
modification to /san/user/vhub/u2/bin/transmitresults.sh
IRODS_OUTPUT_NAME="postprocess_irods.sh" if [ -x ./$IRODS_OUTPUT_NAME ]; then ./$IRODS_OUTPUT_NAME fi
Common Problems
Client cannot connect to server
Server may go down unexpectedly. Admin should log onto irods1.ccr.buffalo.edu, change to the iRods directory and type
irodsctl restart
It’s better to use restart than istart because sometimes the SQL server has gone down but the iRODS daemon is still running. istart will not resart the database.
Check the server log to see whether the restart succeded. The log is in iRODS/server/log and it is named by the date, e.g. rodsLog.2010.7.11
vhub cannot access the users file
The user must do the command
ichmod inherit username
ichmod -r own vhub /vhub/home/username
Idea for submitted apps to get irods files
Use file browser to get path to file.
- Parse path to get file name.
- when submit arrives, it makes a symbolic link to the file name and name the link with the file name.
- remove the symbolic link after the app runs.
ln -s irods/username/pathToFile/filename filename
- submit execs app, file “resides” in cwd.
rm filename
To use a remote command-line client:
Get a copy of the file .irodsEnv from us (Note: the irodsHost needs to be an IP address rather than a domain name) and have us set up your login name and password on the iRods server.
From your home directory on the hub:
mkdir ~/.irods mv .irodsEnv ~/.irods
Follow these instructions for installing additional clients.
At this point you will need to set your path:
PATH=home/vhub/username/iRODS/clients/icommands/bin:$PATH
It’s easiest if you put this command in your .bashrc or .profile file so the commands are available every time you open a terminal.
Some basic concepts
- collection: a directory
- resource: represents a physical data server location that can hold many collections
- zone: multiple resources can be members of the same zone
Some useful commands
- iinit: enter your password and it will get cahed so you don’t have to give your password for every command.
- ils: list files
- imkdir: create a directory (also called a collection)
imkdir username/folder
- iput: put a file into iRods
iput foo.tar username/folder
- ireg: copy a directory revursively into iRods
NOTE: If you add files to the actual physical directory, this is not reflected in iRODS.
ireg -C /data/iRODS/testIreg/ /tempZone/home/rods/newCol ils newCol /tempZone/home/rods/newCol: foo1 foo2 cd testIreg/ touch bar ils newCol /tempZone/home/rods/newCol: foo1 foo2
iRods FUSE notes
Install: Need to install libfuse-dev to get the header files in the same path as the library
edit iRODS/config/config.mk
uncomment IRODS_FS = 1
set fuseHomeDir=path/to/fuse where the path is just above include/fuse.h and lib/libfuse.a
on your vhub account this is simply fuseHomeDir=/usr
in iRODS/client/fuse do make
Add another path so that you can mount with the command irodsFs
NOTE: You cannot unmount on your vhub account, once you’ve mounted irods. Only a vhub dministrator can run fusermount.
PATH=/home/alisanee/iRods/iRODS/clients/icommands/bin:/home/alisanee/iRods/iRODS/clients/fuse/bin:$PATH
Public share directory
While using iRODS you may want to share files with different users. One way to do this is to put the files you want to share in /vhub/home/public. Any user that is on the iRODS system can read and write to this public directory.
icp foo /vhub/home/public
ADMIN NOTE: When users are created, they are automatically added to the share group. The permissions on the /vhub/home/public directory is set up so anyone in the share group is allowed r/w access.
If you want to allow users that do not have an account on the iRODS system to access the public share directory, you need to create the anonymous user. Create the user just like all the other users, but you do not have to set the password.
iadmin mkuser anonymous rodsuser
After creating the user, allow that user to read the /vhub/home/public directory.
ichmod read anonymous /vhub/home/public
ichmod inherit anonymous
After these steps, anyone that does not have an account on the irods system is able to read the files located in the /vhub/home/public directory. They are not able to write or go up the directory structure.
In order for users to login to iRODS with this anonymous user, all you have to do is use the username anonymous with any password.
The username anonymous is a special account name that is recognised in the iRODS system.
Admin’s can use the following script to automate the process of creating the necessary files needed to use the anonymous user in vhub and mount the /vhub/home/public directory on iRODS to ~/irods directory on the users filesystem on vhub.
File : irods_anonymous.sh (4 KB, uploaded by Kyle Marcus 1 decade 2 years ago)
Administrators’ To Do List For Installing on HUB
put the icommands in the path for every user
install .irods directory for every user and provide an irodsEnv file with their Hub ID as the user name
do iinit once with their user name and password. This will create a file, .irods/.irodsA which holds the user’s encrypted password, so that they won’t have to enter their password to do an icoomand.
in irods, create a home directory for the user and set the group to the user name.
irodsFs mount in an empty folder named irods
irodsFs irods
For New User
As the ‘rods’ administrator, for a new hub user:
iadmin mkuser username#vhub rodsuser create new user (use rodsadmin if they are an admin)
iadmin moduser username password somepassword set user’s password
imkdir username make home directory for new user
ichmod -V own username username let user rwx her home directory
ichmod read vhub /home/username let vhub rwx user’s home directory
ichmod write vhub /home/username
ichmod inherit username let vhub rwx the subdirectories and files the user creates
NOTE: if you are creating a rodsadmin account, then you must use the -M option with ichmod
users can then do
iput
iput foo username
Note that the irods1.ccr.buffalo.edu name is not available to a DNS server, so until that changes, the irodsEnv has to be edited to have the IP address instead.
Edit .bashrc in iRods server(done):
PATH=/data/iRODS/iRODS/clients/icommands/bin:$PATH
Install iRods on vhub@u2-grid (done)
edit .bashrc
PATH=/san/user/vhub/u2/iRODS/clients/icommands/bin:$PATH
iadmin mkuser username rodsuser
iadmin mkuser vhub rodsadmin
Rename resource
iadmin modresc oldName name newName
edit config/irods.config
$RESOURCE_NAME = 'newName';
restart the server
./irodsctl restart
Change hostname
edit config/irods.config
$DATABASE_HOST = 'new.hostname';
edit server/config/irodsHost
localhost irods.ccr.buffalo.edu 10.122.1.8
Transferring large files
When you send or receive a file to/from iRODS that’s over 32 MB, it starts opening multiple ports so it can multithread the data transfers. You need to make sure the correct ports are open or the data transfer will not work.
Open ports 20,000 to 20,199 in the firewall
For irods@irods1.ccr.buffalo.edu edit ~/iRODS/config/irods.config to say
$SVR_PORT_RANGE_START = '20000'; $SVR_PORT_RANGE_END = '20199';
and also
~/iRODS/scripts/perl/irodsctl.pl
to say
$svrPortRangeStart=20000; $svrPortRangeEnd=20199;
Then restart the server
irodsctl restart
Note: when doing the multithreaded data transfers we had a problem with the server only listening on the internal IP address.
> iput -V -f test_dir_timing_40mb_80files.tar kmarcus/test_time NOTICE: irodsHost=irods1.ccr.buffalo.edu NOTICE: irodsPort=1247 NOTICE: irodsDefResource=demoResc NOTICE: irodsHome=/vhub/home NOTICE: irodsCwd=/vhub/home NOTICE: irodsUserName=kmarcus NOTICE: irodsZone=vhub >From server: NumThreads=2, addr:10.6.1.20, port:20169, cookie=810769104 ERROR: connectToRhostPortal: connectTo Rhost 10.6.1.20 port 20169 error, status = -347000
Fix: edit ~/iRODS/server/config/irodsHost to say:
localhost irods1.ccr.buffalo.edu 10.6.1.20
This tells iRODS to prefer the interface irods1.ccr.buffalo.edu when binding.
Information for this fix was provided by Chris Smith on iRODS Chat
Uninstalling iRODS server
remove the collections (i.e. directories and files), e.g.
irm -r aneeman
turn irods off
irodsctl stop
remove directories for irods and its postgres installation.
Move a collection or file to a specific server
iphymv -r resource2Collection -R demoResc2
-r recursive -R resource name
The “mount” always looks the same, regardless of the physical location. You can move a file to any data server.
iRods Timing
