How to DRSLib

Present date:



 . Before running drs-tool 
  . Make sure you have disk space allocated to your data set: use ceda database management tool at http://cedadb.ceda.ac.uk/admin/ 
  . give the path, for instance : /badc/specs/data/SPECS/output/KNMI
  . give the estimated amount of data, for instance: : 15000000000000 (13.5 Teras)


 . How to run  DRSLib  from the command line      
    (go to ingest1 as badc)
     1) activate the virtualenvironment: cd  /usr/local/ingest_software/venv27; source bin/activate
     2) (venv27)[badc@ingest1 venv27]$ drs_tool "upgrade/todo" -M cp -R /badc/specs/data/SPECS/ -j /group_workspaces/jasmin/specs/dacosta/CEDA_CC/batch00_KNMI_EC-EARTH2_seaIceInit/Rec_batch00_KNMI_EC-EARTH2_seaIceInit_6hr_atmos.json -s specs --component activity=SPECS
    . "todo"  dry run, otherwise "upgrade" to copy to the archive
    . "-M cp" tells drs_tool to copy rather than move files.
    . output dir in the archive :   /badc/specs/data/SPECS (DRSLib will create the correct paths (including subdirectories version, latest, and files) from the info in the json file created  by ceda-cc
    . -j path to the json file created by ceda-cc
    . -s and --component activity tell DRSLib where to find the directory configuration for the SPECS files

 . Some tips  
    1) ceda-cc may produce dozens of .json files. It may be convenient to concatenate all those files into one file and serve that to DRSLib
       An example of this kind of concatenation is given in the script /group_workspaces/jasmin/specs/dacosta/DRS_TOOL/create_all_RecJson_files.sh
       The concatenated file will be in /group_workspaces/jasmin/specs/dacosta/CEDA_CC
    2) In order to tell DRSLib to use the concatenated file one can use a script like this one: /group_workspaces/jasmin/specs/dacosta/DRS_TOOL/drs-tool_KNMI_EC-EARTH2_seaIceInit.sh 
    3) When the archiving has been done it is useful to check if files are realy where they are expected:
       run : du -sh \ to get the volume of data stored
       run : find \ -type f | wc -l  to get the number of files strored


Related sites:
   http://esgf-drslib.readthedocs.org/en/latest/intro.html  
   https://media.readthedocs.org/pdf/esgf-drslib/latest/esgf-drslib.pdf  
   https://github.com/ESGF/esgf-drslib