iRods for HUBs

Search

Search pages

Wiki

Tools

Version 164
by (unknown)
Version 165
by (unknown)

Deletions or items before changed

Additions or items after changed

1 -
We are working on using [https://www.irods.org iRODS], as a way to increase storage for users of the HUB. Possible uses include simple staging of data for simulation, and housing searchable collections of documents. Some theory about our approach can be found in the paper [http://hpc2.org/resources/39 here].
+
We are working on using [https://www.irods.org iRODS], as a way to increase storage for users of the HUB. Possible uses include simple staging of data for simulation, and housing searchable collections of documents. Some theory about our approach can be found in the paper [http://hpc2.org/resources/39 here].
2
3 We were using two servers to test iRods at CCR. The first was both a data and metadata server. The second was just a data server. Now, we upgraded to iRODS 3.0 and are only using one server.
4
5 == Prerequisites ==
6
7 iRods requires access to the CCR irods server (irods.ccr.buffalo.edu). By default, the hub does not allow outbound network access so user will need to be granted access via the Joomla administration interface. Each user requiring irods access must be
8
9 * a hub administrator
10
11 * a member of the '''network''' group
12
13 '''Note: It may take up to an hour for access to propogate once a user is added to the group.'''
14
15 == Clients ==
16
17 * '''iDrop''': Java client. https://code.renci.org/gf/project/irodsidrop/
18
19 == Basic concepts ==
20
21 * '''collection''': a directory
22
23 * '''resource''': represents a physical data server location that can hold many collections
24
25 * '''zone''': multiple resources can be members of the same zone
26
27 == Useful commands ==
28
29 * '''iinit''': enter your password and it will get cahed so you don't have to give your password for every command.
30
31 * '''ils''': list files
32
33 * '''imkdir''': create a directory (also called a collection). '''NOTE''': A directory will not show up in the server directory structure until a file is put in it using iRods. {{{ imkdir username/folder }}}
34
35 * '''iput''': put a file into iRods. {{{ iput foo.tar username/folder }}}
36
37 * '''ireg''': copy a directory revursively into iRods. '''NOTE''': If you add files to the actual physical directory, this is not reflected in iRODS. {{{
38 #> ireg -C /data/iRODS/testIreg/ /tempZone/home/rods/newCol
39 #> ils newCol
40 /tempZone/home/rods/newCol:
41 foo1
42 foo2
43 #> cd testIreg/
44 #> touch bar
45 #> ils newCol
46 /tempZone/home/rods/newCol:
47 foo1
48 foo2
49 }}}
50
51
52 == Current iRods Configuration ==
53
54 '''server''': irods.ccr.buffalo.edu
55
56 '''port''': 1247
57
58 '''zone''': vhub
59
60 '''resource''': vhub-ccr-buffalo
61
62 '''Web client: ''': [http://irods.ccr.buffalo.edu/irodsweb/ irods.ccr.buffalo.edu/irodsweb/]
63
64 '''Public files''': [http://irods.ccr.buffalo.edu/public/ irods.ccr.buffalo.edu/public/]
65
66 == Setup for New Users ==
67
68 Ask the administrator to [iRodsforHUBs#Administrator_Tasks_and_Commands create an irods account] for you with a password you'd like.
69
70 To create the necessary files for irods to run, download the [[File(irods_newuser.sh)]] bash script and execute it in a terminal.
71
72 To use the default user ''annonymous'' and only be able to read the ''public'' directory on irods, you can just run the script as follows:
73
74 {{{
75 > chmod +x irods_newuser.sh
76 > ./irods_newuser.sh
77 Do you want to make 'use irods' persistent? (y/n) y
78 }}}
79
80 If you want to change the settings to use a different username, zone, etc, append the ''-i'' argument to the end of the name of the script. Here is a example of changing the username to ''vhub''.
81
82 {{{
83 #> chmod +x irods_newuser.sh
84 #> ./irods_newuser.sh -i
85
86 Answer the following questions to configure iRODS
87 Press ENTER for the default setting for each question
88
89 iRODS server name [irods.ccr.buffalo.edu]:
90 iRODS port [1247]:
91 iRODS username [anonymous]: vhub
92 iRODS zone [vhub]:
93 iRODS resource name [vhub-ccr-buffalo]:
94 iRODS home dir [/vhub/home/vhub]:
95 iRODS current working dir [/vhub/home/vhub]:
96 Do you want to make 'use irods' persistent? (y/n) y
97 Enter your current iRODS password:
98
99 }}}
100
101 After the script has run, you will be able to use the [https://www.irods.org/index.php/icommands i-commands] and use files mounted in your irods directory in the'' ~/irods'' directory on your hub environment. To see if everything is working correctly you can type '''ils''' or '''ls ~/irods''' and you should see the files that are currently in your home directory in irods.
102
103 ----
104
105 If you would just like to download the ''irodsEnv'' file and modify it yourself, you can download it below. (The file below is already configured to run as the ''annonymous'' user and read the ''public'' directory).
106
107 File: [[File(irodsEnv)]]
108
109 If you decided to just download the ''irodsEnv'' file, then you need to follow the following 7 steps. If you ran the ''irods_newuser.sh'' script, then you can skip these 7 steps.
110
111 1. Modify the user name inside the file linked above, and change the file name to ''.irodsEnv''
112
113 2. Create a folder name ''.irods'' in the user's home directory and place ''.irodsEnv'' in the folder.
114
115 3. Create a folder named ''irods'' in your home directory. This folder will be used as the mount location for the the iRODS filesystem.
116
117 4. Type the command '''use irods''' in your workspace command line to be able to run icommands.
118
119 5. Create a bash profile file name ''.bash_profile'' with the following content:
120
121 {{{
122 if [ -n $IRODS_MOUNT -a -e $IRODS_MOUNT ]; then
123 $IRODS_MOUNT
124 fi
125 }}}
126
127 6. To make your password persist, type '''iinit''' and enter your irods password.
128
129 7. Then, run the command '''source .bash_profile'''
130
131 After these steps you should be able to use the [https://www.irods.org/index.php/icommands i-commands] and use files in your ''~/irods'' directory.
132
133 === .bashrc Files For Users ===
134
135 Once icommands and irodsFs are installed into ''/apps'' the script will read similar to this:
136
137 {{{
138 which mountIrodsFs >& /dev/null
139 if [ 0 = $? ]; then
140 mountIrodsFs
141 fi
142 }}}
143
144 === Setup New Groups ===
145
146 To create a group, do the following
147
148 1. Create the group
149
150 {{{
151 iadmin mkgroup testgroup
152 }}}
153
154 2. Add users to the group
155
156 {{{
157 iadmin atg testgroup kmarcus
158 iadmin atg testgroup smgallo
159 }}}
160
161 When a group is created, a home directory for the group is also created.
162
163 You can also modify a collection or file to allow a group/user to access it
164
165 {{{
166 ichmod read testgroup testgroupdir
167 ichmod write testgroup testgroupdir
168 ichmod own testgroup testgroupdir
169 }}}
170
171 [http://groups.google.com/group/irod-chat/browse_thread/thread/68cc2c715f75f3df# Google Groups question on group permissions]
172
173 == Using iRods with the Submit Command ==
174
175 Make sure ''preprocess_irods.sh'' and ''postprocess_irods.sh'' are set to '''executable'''
176
177 {{{
178 chmod +x preprocess_irods.sh
179 chmod +x postprocess_irods.sh
180 }}}
181
182 Also when using the ''submit'' command, be sure to include the ''preprocess_irods.sh'' and ''postprocess_irods.sh'' files
183
184 {{{
185 submit ... -i preprocess_irods.sh -i postprocess_irods.sh ...
186 }}}
187
188 If you are placing the output of a job into iRods, make sure to use the '''-q''' option with submit so that there will be no limit on the file size that you can create.
189
190 '''modification to /san/user/vhub/u2/bin/receiveinput.sh'''
191
192 {{{
193 IRODS_INPUT_NAME="preprocess_irods.sh"
194 if [ -x ./$IRODS_INPUT_NAME ]; then
195 ./$IRODS_INPUT_NAME
196 fi
197 }}}
198
199 '''modification to /san/user/vhub/u2/bin/transmitresults.sh'''
200
201 {{{
202 IRODS_OUTPUT_NAME="postprocess_irods.sh"
203 if [ -x ./$IRODS_OUTPUT_NAME ]; then
204 ./$IRODS_OUTPUT_NAME
205 fi
206 }}}
207
208
209 == iRods FUSE notes ==
210
211 Install: Need to install ''libfuse-dev'' to get the header files in the same path as
212 the library
213
214 edit `iRODS/config/config.mk`
215
216 {{{
217 uncomment IRODS_FS = 1
218 }}}
219
220 set ''fuseHomeDir=path/to/fuse'' where the path is just above ''include/fuse.h'' and ''lib/libfuse.a''
221
222 on your vhub account this is simply ''fuseHomeDir=/usr''
223
224
225 in iRODS/client/fuse do '''make'''
226
227 Add another path so that you can mount with the command ''irodsFs''
228
229 '''NOTE: You cannot unmount on your vhub account, once you've mounted irods.
230 Only a vhub dministrator can run fusermount.'''
231
232 {{{
233 export PATH=/apps/irods/3.1.0/icommands/bin:/apps/irods/3.1.0/fuse/bin:$PATH
234 }}}
235
236 == Making Files Publically Available ==
237
238 === Public Share Directory ===
239
240 While using iRODS you may want to share files with different users. One way to do this is to put the files you want to share in ''/vhub/home/public''. Any user that is on the iRODS system can read and write to this public directory.
241
242 ''icp'' foo /vhub/home/public
243
244 '''ADMIN NOTE:''' When users are created, they are automatically added to the ''share'' group. The permissions on the ''/vhub/home/public'' directory is set up so anyone in the ''share'' group is allowed r/w access.
245
246 If you want to allow users that do not have an account on the iRODS system to access the public share directory, you need to create the ''anonymous'' user. Create the user just like all the other users, but you do not have to set the password.
247
248 ''iadmin'' mkuser anonymous rodsuser
249
250 After creating the user, allow that user to ''read'' the ''/vhub/home/public'' directory.
251
252 ''ichmod'' read anonymous /vhub/home/public
253
254 ''ichmod'' inherit anonymous
255
256 After these steps, anyone that does not have an account on the irods system is able to read the files located in the ''/vhub/home/public'' directory. They are not able to write or go up the directory structure.
257
258 In order for users to login to iRODS with this ''anonymous'' user, all you have to do is use the username ''anonymous'' with any password.
259
260 The username ''anonymous'' is a special account name that is recognised in the iRODS system.
261
262 Admin's can use the following script to automate the process of creating the necessary files needed to use the ''anonymous'' user in vhub and mount the ''/vhub/home/public'' directory on iRODS to ''~/irods'' directory on the users filesystem on vhub.
263
264 File : [[File(irods_anonymous.sh)]]
265
266 === Public Share Web Access ===
267
268 Data placed in the public share directory is also made available via the following url: http://irods.ccr.buffalo.edu/public/
269
270
271 == Administrator Tasks and Commands ==
272
273 Put the icommands in the path for every user
274
275 install .irods directory for every user and provide an irodsEnv file with their Hub ID as the user name
276
277 Run iinit once with their user name and password. This will create the .irods/.irodsA rile which holds the user's encrypted password so that they won't have to enter their password to do an icommand.
278
279 in irods, create a home directory for the user and set the group to the user name.
280
281 Use irodsFs to mount an empty folder named irods in the user's home directory
282
283 {{{
284 irodsFs irods
285 }}}
286
287 === Create an iRods User ===
288
289 As the 'rods' administrator (irods vhub user), create a new hub user:
290
291 1. Create new user (use "rodsadmin" rather than "rodsuser" if they are an admin)
292
293 {{{
294 iadmin mkuser username#vhub rodsuser
295 }}}
296
297 2. Set user's password
298
299 {{{
300 iadmin moduser username password somepassword
301 }}}
302
303 3. Make home directory for new user
304
305 {{{
306 imkdir /vhub/home/username
307 }}}
308
309 4. Let user rwx her home directory
310
311 {{{
312 ichmod -M own username /vhub/home/username
313 }}}
314
315 5. Let the vhub user rwx user's home directory and subdirectories
316
317 {{{
318 ichmod -M read vhub /vhub/home/username
319 ichmod -M write vhub /vhub/home/username
320 ichmod -M inherit /vhub/home/username
321 }}}
322
323 '''NOTE:''' We need the '''-M''' option to operate in admin mode for changing ownership and file permissions.
324
325 For example, to create a new user "scarn"
326
327 {{{
328 iadmin mkuser scarn#vhub rodsuser
329 iadmin moduser scarn password ********
330 imkdir scarn
331 ichmod -M own scarn /vhub/home/scarn
332 ichmod -M read vhub /vhub/home/scarn
333 ichmod -M write vhub /vhub/home/scarn
334 ichmod -M inherit /vhub/home/scarn
335 }}}
336
337 Users can use '''iinit''' to initialize their password into the ~/.irods/.irodsA file
338
339 Users can then do
340
341 ''iput''
342
343 ''iput'' foo username
344
345 Edit `.bashrc` in iRods server(done):
346
347 {{{
348 PATH=/data/iRODS/iRODS/clients/icommands/bin:$PATH
349 }}}
350
351 Install iRods on vhub@u2-grid
352
353 edit `.bashrc`
354
355 {{{
356 PATH=/san/user/vhub/u2/iRODS/clients/icommands/bin:$PATH
357 }}}
358
359 ''iadmin'' mkuser username rodsuser
360
361 ''iadmin'' mkuser vhub rodsadmin
362
363 === Add Files Directly to iRods Server ===
364
365 You can place files directly inside the iRods directory structure on the sever and the register them later.
366
367 {{{
368 mv foo.txt /data/iRODS/iRODS/Vault/home/public
369 ireg -R vhub-ccr-buffalo /data/iRODS/iRODS/Vault/home/public/foo.txt /vhub/home/public/foo.txt
370 }}}
371
372 == Transferring Large Files ==
373
374 When you send or receive a file to/from iRODS that's over 32 MB, it starts opening multiple ports so it can multithread the data transfers. You need to make sure the correct ports are open or the data transfer will not work.
375
376 Open ports ''20,000'' to ''20,199'' in the firewall
377
378 For irods@irods1.ccr.buffalo.edu edit `~/iRODS/config/irods.config` to say
379
380 {{{
381 $SVR_PORT_RANGE_START = '20000';
382 $SVR_PORT_RANGE_END = '20199';
383 }}}
384
385 and also
386
387 `~/iRODS/scripts/perl/irodsctl.pl`
388
389 to say
390
391 {{{
392 $svrPortRangeStart=20000;
393 $svrPortRangeEnd=20199;
394 }}}
395
396 Then restart the server
397
398 ''irodsctl'' restart
399
400 '''Note:''' when doing the multithreaded data transfers we had a problem with the server only listening on the internal IP address.
401
402 {{{
403 > iput -V -f test_dir_timing_40mb_80files.tar kmarcus/test_time
404 NOTICE: irodsHost=irods1.ccr.buffalo.edu
405 NOTICE: irodsPort=1247
406 NOTICE: irodsDefResource=demoResc
407 NOTICE: irodsHome=/vhub/home
408 NOTICE: irodsCwd=/vhub/home
409 NOTICE: irodsUserName=kmarcus
410 NOTICE: irodsZone=vhub
411 >From server: NumThreads=2, addr:10.6.1.20, port:20169,
412 cookie=810769104
413 ERROR: connectToRhostPortal: connectTo Rhost 10.6.1.20 port 20169
414 error, status = -347000
415 }}}
416
417 '''Fix:''' edit `~/iRODS/server/config/irodsHost` to say:
418
419 {{{
420 localhost irods1.ccr.buffalo.edu 10.6.1.20
421 }}}
422
423 This tells iRODS to prefer the interface ''irods1.ccr.buffalo.edu'' when binding.
424
425 Information for this fix was provided by Chris Smith on [http://groups.google.com/group/irod-chat/browse_thread/thread/24ade1a0b1954d42/a39ad7965695e5d1?lnk=gst&q=kmarcus30#a39ad7965695e5d1 iRODS Chat]
426
427
428 === iRods Timing ===
429
430 [[Image(irods_timing.png)]]
431
432 == Creating iRods Rules ==
433
434 Documentation on iRods rules can be found [https://www.irods.org/index.php/Rules here]
435
436 iRods 3.0 has added [https://www.irods.org/index.php/Changes_and_Improvements_to_the_Rule_Language_and_the_Rule_Engine new rule syntax]
437
438 === Example iRods Rule Setup ===
439
440 For the example rule, we have a directory on irods (''/irods/home/public'') where the 'anonymous' user is able to download files. For the files in this public directory we also want to serve them on a web page. One way we came up with to do this was to add the 'apache' user to the 'irods' group and change all the permissions on the files in the ''public'' directory so that the group permissions were readable (this way the apache server running as 'apache' could read the files, but the 'irods' user could still do anything it need to do with the files). In order to do this I came up with the following rule which is located at ''server/config/reConfigs/publicGroupPermissions.re''. This rule uses the 'on' keyword as its main logic. When the $objPath (variable that is defined by the irods session) is in the ''public'' directory, it will then call a shell script to change the permissions on all the files in that ''public'' directory. Writing to the log file with msiWriteRodsLog is not necessary, but it is useful for debugging purposes.
441
442 {{{
443 publicGroupPermissions {
444 on ($objPath like "/vhub/home/public/*") {
445 msiExecCmd("publicPermissions","null","null","null","null",*Result);
446 msiWriteRodsLog("--- INSIDE publicGroupPermissions ---",*Status);
447 }
448 }
449 }}}
450
451 this rule will call the publicPermission script (''server/bin/cmd''), its permissions set to execute. This script will change the group permissions for all the files in the public directory. Here is the ''server/bin/cmd/publicPermissions'' shell script:
452
453 {{{
454 #!/bin/sh
455 chmod g+r /data/iRODS/iRODS/Vault/home/public/*
456 }}}
457
458 Then add the publicGroupPermissions rule file to the ''server/config/server.config'' file (this is where all the rule files that are going to be used are defined, they are comma seperated).
459
460 {{{
461 -- reRuleSet core
462 ++ reRuleSet core,publicGroupPermissions
463 }}}
464
465 Finally, you must specify when the rule should be called. In this case we wanted the rule to be called when someone put a file into the iRods ''public'' directory. To do this, you would use the ''iput'' command to add file. To execute the publicGroupPermissions rule when the ''iput'' command is used, the rule must be added to the ''server/config/reConfigs/core.re'' file. The place where we want to add our rule is ''acPostProcForPut'', this is the function that gets executed after ''iput'' has succeeded.
466
467 {{{
468 -- acPostProcForPut { }
469 ++ acPostProcForPut { publicGroupPermissions; }
470 }}}
471
472 After all of this is in place and all the permissions on the files are properly set, restart the server.
473
474 {{{
475 ./irodsctl restart
476 }}}
477
478 To test this rule, start placing file in the ''public'' directory and then look at the permissions in ''Vault/home/public'', you should now see that all the files are now readable by the irods group.
479
480 === Useful Notes ===
481
482 * rules ending in ''.re'' are 3.0 syntax
483 * rules must be located in: ''server/config/reConfigs/''
484 * scripts called from rules must be located at: ''server/bin/cmd/''
485 * rule config file: ''server/config/server.config''
486 * rules must be comma seperated
487 * example rules: ''clients/icommands/test/rules3.0/''
488 * iRods 3.0 api [https://irods.org/doxygen/index.html documentation]
489 * watch log files for debugging
490 * iRods: {{{tail -f server/log/rodsLog.YYYY.MM.DD}}}
491 * rules engine: {{{tail -f server/log/reLog.YYYY.MM.DD}}}
492 * The users home directory on irods: ''Vault/home/''
493 * Session variables listed in ''server/config/reConfigs/core.dvm'', more info in iRods Primer book
494
495 == iRods Clients ==
496
497 Currently we are using the old extjs [http://irods.ccr.buffalo.edu iRods web client]. This was downloaded from [https://code.renci.org/gf/project/irodsphp/ code.renci.org/gf/project/irodsphp/]. The files were extracted into the clients directory in the iRODS home directory. Then, an alias was created in Apache to point to this directory. Everything works good when the STRICT option is turned off in the iRods config. The STRICT setting determines what the rodsuser can see in a directory. If the STRICT setting is on, then a rodsuser user can only see the contents of their directory. If it is off, then the rodsuser user is able to see all the files, however they are only able to read files that they own. Note, that the STRICT setting does not effect the rodsadmin user.
498
499 To turn the setting on and off, you have to edit the file ''iRODS/server/config/reConfigs/core.re''
500
501 On: {{{ acAclPolicy {msiAclPolicy("STRICT"); } }}}
502
503 Off: {{{ acAclPolicy { } }}}
504
505 If you deceide to turn the STRICT setting on, you then need to change the permissions on ''/vhub'' and ''/vhub/home'' so that the web client can read these directories for a normal rodsuser user. Do the following to change the permissions:
506
507 {{{
508 > ichmod read public /vhub
509 > ichmod read public /vhub/home
510 }}}
511
512 Once this is complete, both a rodsuser, and a rodsadmin user will be able to log into the web client and everything should work.
513
514 ==Useful links for admins/developers==
515
516 [https://www.irods.org/index.php/Publications iRods Publications]
517
518 [http://www.morganclaypool.com/doi/abs/10.2200/S00233ED1V01Y200912ICR012?journalCode=icr iRODS Primer: Integrated Rule-Oriented Data System] Morgan & Claypool, 2010 (PDF)
519
520 [https://irods.org/doxygen/index.html iRods 3.0 api]
521
522 [https://www.irods.org/web/index.php iRods web client]
523
524
525 == Troubleshooting ==
526
527 === Client cannot connect to server ===
528
529 Server may go down unexpectedly. Admin should log onto ''irods1.ccr.buffalo.edu'', change to the iRods directory and type
530
531 ''irodsctl'' restart
532
533 It's better to use restart than istart because sometimes the SQL server has gone down but the iRODS daemon is still running. istart will not resart the database.
534
535 Check the server log to see whether the restart succeded. The log is in iRODS/server/log
536 and it is named by the date, e.g. rodsLog.2010.7.11
537
538 === vhub cannot access the users file ===
539
540 The user must do the command
541
542 ''ichmod'' inherit username
543
544 ''ichmod'' -r own vhub /vhub/home/username
545
546 == Misc Notes ==
547
548 === Changing the iRods Server ===
549
550 '''Rename resource'''
551
552 {{{
553 iadmin modresc oldName name newName
554 }}}
555
556 edit config/irods.config
557
558 {{{
559 $RESOURCE_NAME = 'newName';
560 }}}
561
562 restart the server
563
564 {{{
565 ./irodsctl restart
566 }}}
567
568 '''Change hostname'''
569
570 stop the server
571
572 {{{
573 ./irodsctl stop
574 }}}
575
576 Edit the following files, changing the host name to the new address:
577
578 * `.odbc.ini` in the home directory
579
580 * `.irods/.irodsEnv` in home directory
581
582 * `pgsql/etc/odbc.ini` in the Postgres directory
583
584 * `server/config/server.config` in iRODS distribution directory
585
586 Edit server/config/irodsHost
587
588 {{{
589 localhost new-host-name 10.122.1.8
590 }}}
591
592 Start iRods again
593
594 {{{
595 ./irodsctl start
596 }}}
597
598 Finally, modify the address of the local resource
599
600 {{{
601 iadmin modresc demoResc host new-host-name
602 }}}
603
604 === Move a collection or file to a specific server ===
605
606 {{{
607 iphymv -r resource2Collection -R demoResc2
608 }}}
609
610 ''-r'' recursive
611 ''-R'' resource name
612
613 The "mount" always looks the same, regardless of the physical location.
614 You can move a file to any data server.
615
616 === Idea for submitted apps to get irods files ===
617
618 Use file browser to get path to file.
619
620 * Parse path to get file name.
621
622 * when submit arrives, it makes a symbolic link to the file name and name the link with the file name.
623
624 * remove the symbolic link after the app runs.
625
626 {{{
627 ln -s irods/username/pathToFile/filename filename
628 }}}
629
630 * submit execs app, file "resides" in cwd.
631
632 {{{
633 rm filename
634 }}}