wiki:Howtos/ESGFDownloadScript

HOWTO for Script Based Download from ESGF Secured OPeNDAP Service

Secured services in the ESG Federation such as OPeNDAP support both OpenID and PKI based authentication. OpenID provides a convenient means for browser based access but for script based access an alternative PKI based solution is more suited. For this, a user obtains a short term authentication token which they can pass in their client programs or scripts to access secured services. The token is actually a key pair, a private key and associated X.509 certificate. The certificate typically last a few hours before it expires. Users obtain a certificate using a token service MyProxy passing their usual username/password to obtain a certificate in response. Once obtained, it along with the private key can be used with programs like wget, to make secured calls to the service to obtain data.

The steps shown below assume a Linux environment with wget installed. With some modification they should also work with Windows and Mac.

1) Obtaining Credentials from MyProxy

Three different client programs are described here:

  1. Java MyProxyLogon Webstart application
  2. Python MyProxyClient package
  3. Bash script for use with CEDA's MyProxy Logon Web Service.

They are alternative means of performing the same task of getting credentials.

Java MyProxyLogon WebStart

Prerequisite: Java Runtime Environment version 1.5 or later installed on your host machine.

Select this  link to invoke the WebStart application. If this doesn't work the application can be  downloaded and run from a command line instead:

$ java -jar MyProxyLogon.jar

A window should appear when the program is run.

  1. Enter your usual username/password in the Username and Passphrase textboxes respectively.
  2. For the Hostname field enter, myproxy.ceda.ac.uk for CEDAs MyProxy service.
  3. Alter the Output field to read, <home directory>/.esg/credentials.pem where <home directory> is your home directory path e.g. /home/jbloggs
  4. Click on the tickbox to select Write trust roots
  5. Click the Logon button
  6. Copy CA files downloaded to the standard location for ESG:
    $ cp -r ~/.globus/certificates ~/.esg/
    

Python MyProxyClient Package

This package provides a command line script for obtaining credentials from a MyProxy server. To install,

$ sudo easy_install MyProxyClient

If you don't have easy_install installed, you can get the  bootstrap script and then run:

$ sudo python ez_setup.py

If you don't have sudo or admin access rights see below for alternative installation instructions.

Call the logon script

Give your usual CEDA / BADC username and when prompted enter your password.

$ myproxyclient logon -s myproxy.ceda.ac.uk -b -C ~/.esg/certificates -o ~/.esg/credentials.pem -l <username>

Install without admin Privileges

This assumes a user who doesn't have root or sudo access.

  1. Make a new directory
    $ mkdir myproxy-env
    $ cd myproxy-env
    
  2. Get the ez_setup.py bootstrap script (Tip: make sure your http_proxy environment variable is set up):
    $ wget http://peak.telecommunity.com/dist/ez_setup.py
    
  3. Run the script installing the virtualenv package:
    $ export PYTHONPATH=. && python ez_setup.py -d . virtualenv
    
  4. Make a new Virtual environment:
    $ ./virtualenv --no-site-packages .
    
  5. Install the MyProxyClient:
    $ ./bin/easy_install MyProxyClient
    
  6. Nb. To run the MyProxyClient script give the local path to the script:
    $ ./bin/myproxyclient
    

Install Troubleshooting

The MyProxyClient installation may fail with compilation errors for OpenSSL. This is because the OpenSSL development package needs to be installed on your system. Install this using your Linux package manager e.g. yast2 or yum or contact your system administrator to help you.

Bash Script

At CEDA, there is a web service interface for MyProxy logon which can be called via a bash shell script.

  1. Download scripts:
    $ wget http://ndg-security.ceda.ac.uk/browser/trunk/MyProxyWebService/myproxy/ws/client/myproxy-ws-get-trustroots.sh
    $ wget http://ndg-security.ceda.ac.uk/browser/trunk/MyProxyWebService/myproxy/ws/client/myproxy-ws-logon.sh
    
  2. Set up trust roots (directory containing CA certificates to enable your client to correctly verify the identity of the MyProxy and data download services):
    $ myproxy-ws-get-trustroots.sh -U https://myproxy.ceda.ac.uk/get-trustroots -b -c ~/.esg/certificates
    
  3. Call logon script to get a new credential giving your username and entering your password when prompted:
    $ myproxy-ws-logon.sh -U https://myproxy.ceda.ac.uk/logon -c ~/.esg/certificates -l <username> -o ~/.esg/credentials.pem
    

2) WGet Script

  1. Download the script
    $ wget http://ndg-security.ceda.ac.uk/browser/trunk/esg_wget_script/esg-download.sh?format=txt
    
  2. Add execute permissions:
    $ chmod 755 ./esg-download.sh
    
  3. Ensure you have credentials (following the steps in 1) above).
  4. Run the script giving the URL for the data you wish to download:
    $ /esg-download.sh <download URL>
    

Nb. The script has many different options which can be set via command line switches or environment variables. Use the help option to check:

$ /esg-download.sh -h