irfpy.ica.pipeline¶
Functions to syncronize data from the ICA pipeline and ICA related spice kernels The functions here support the new key exchange based access method to the pipeline. For key exchange contact Leif Kalla (leif@irf.se).
Code author: Martin Wieser
Module: irfpy.ica.pipeline
You need to have our ssh key public registered with Leif Kalla (leif@irf.se)!
Usage examples:
updateproc('~/icadata/','20150901')
print the progress to stdout:
updateproc('~/icadata/', '2015', privatekey='myicakey', progress=print)
Syncronize latest SPICE kernels:
Get an updated metakernel with associated kernel files:
updatemetakernel('~/icadata/','ros_kernels_3dtool.txt', privatekey='myicakey', progress=print)
|
Fetches all kernel files referenced by local file metakernel using using using public key authentication from data.irf.se. |
|
Fetches the specified metakernel and associated kernel files from the ica pipeline using using public key authentication from data.irf.se. |
|
Fetches level1 proc files from data using public key authentication from data.irf.se. |
|
Fetches special files from data using public key authentication from data.irf.se. |
|
Fetches aux files from data using public key authentication from data.irf.se. |
|
Fetches bestc (level2) files from data using public key authentication from data.irf.se. |
|
Fetches command log files from data using public key authentication from data.irf.se. |
|
Fetches mag files from data using public key authentication from data.irf.se. |
|
Fetches cops files from data using public key authentication from data.irf.se. |
- irfpy.ica.pipeline.updatekernels(datarootpath, metakernel='ros_kernels_3dtool.txt', privatekey='~/.ssh/icapipelinekey', progress=None)[source]¶
Fetches all kernel files referenced by local file metakernel using using using public key authentication from data.irf.se. The private key file is specified by privatekey.
PARAMETERS
- metakernel:
Filename of local metakernel
- datarootpath:
The kernels are copied to the local path datarootpath/kernels/xxx mirroring the directory structure on data.irf.se.
- progress:
Function with one parameter, usially used to print the progress of the sync process
- privatekey:
Path to the private key file, default ‘~/.ssh/icapipelinekey’
LIMITATIONS
It is assumed that a key exchange has happened previously for user icapy@data.irf.se. Contact Leif Kalla (leif@irf.se) for key exchange.
- Authenticate using specific key::
- updatekernels(‘~/icadata/’,’ros_kernels_3dtool.txt’,
‘~/.ssh/ica_pipeline_key’, progress=print)
- or short by making use of defaults::
updatekernels(‘~/icadata/’, progress=print)
Note that there are some restrictions regarding the metakernel format for this to work. It is requried that the following is on one line:
PATH_VALUES = ( ‘/irf/data/comet/rosetta/spice/kernels’ )
and that there is only one PATH_SYMBOL called $KERNELS. See ros_kernels_3dtool.txt for an example.
The metakernel ‘ros_kernels_3dtool.txt’ automatically generated by the pipeline fulfills these requiremetns automatically.
- irfpy.ica.pipeline.updatemetakernel(datarootpath, metakernel='ros_kernels_3dtool.txt', privatekey='~/.ssh/icapipelinekey', progress=None)[source]¶
Fetches the specified metakernel and associated kernel files from the ica pipeline using using public key authentication from data.irf.se. The private key file is specified by privatekey.
- metakernel:
Filename of local metakernel
- datarootpath:
The kernels are copied to the local path datarootpath/kernels/xxx mirroring the directory structure on data.irf.se.
- progress:
Function with one parameter, usially used to print the progress of the sync process
- privatekey:
Path to the private key file, default ‘~/.ssh/icapipelinekey’
- Limitations:
It is assumed that a key exchange has happened previously for user icapy@data.irf.se. Contact Leif Kalla (leif@irf.se) for key exchange.
The kernels are copied to the local path datarootpath/kernels/xxx.
- Authenticate using specific key::
- updatemetakernel(‘~/icadata/’, ‘ros_kernels_3dtool.txt’,
‘~/.ssh/ica_pipeline_key’)
- or short by making use of defaults::
updatemetakernel(‘~/icadata/’, progress=print)
Path info to the individual kernel files in the downloaded metakernel is then modified to point to datarootpath/kernels// mirroring the spice directory structure on data.irf.se.
The metakernel ‘ros_kernels_3dtool.txt’ automatically generated by the pipeline fulfills these requiremetns automatically.
- irfpy.ica.pipeline.updatelevel1(datarootpath, mask='proc', privatekey='~/.ssh/icapipelinekey', progress=None, dataformat='mat', onlynewfiles=False, **kwargs)[source]¶
Fetches level1 proc files from data using public key authentication from data.irf.se. The private key file is specified by privatekey.
Consider using updateproc(), updateaux(), updatespecial(), etc. instead.
- datarootpath:
The data is copied to the local directory datarootpath.
- mask:
mask defines what files are fetched: ‘proc’ : fetch all proc files, all existing dates, default ‘proc201502’ : fetch all proc files from february 2015 ‘special20141227T00’ : fetch special from 2014-12-27 00h
- privatekey:
Path to the private key file, default ‘~/.ssh/icapipelinekey’
- progress:
Function with one parameter usually used to print the progress
- dataformat (str):
‘mat’ or ‘h5’. if equal to ‘h5’ then the hdf5 data tree is read, if equal to ‘mat’ then the matlab data tree is read.
- onlynewfiles (bool):
If set to True, only files not existing locally will be fetched. Files that exist locally but in an outdated version are not updated. This is useful for slow links. Note this may result in inconsistent data files. Default is False.
- Limitations:
It is assumed that a key exchange has happened previously for user icapy@data.irf.se. Contact Leif Kalla (leif@irf.se) for key exchange.
- Fetch all proc from 2015 using the key icakey::
updatelevel1(‘/home/myuser/icadataroot/’, mask=’proc2015’, privatekey=’icakey’ progress=p)
progress is a function with one string parameter that is called for each line rsync returns. Example:
def p(line): print(line)
- irfpy.ica.pipeline.updateraw(datarootpath, mask='', privatekey='~/.ssh/icapipelinekey', progress=None, dataformat='mat', onlynewfiles=False, **kwargs)[source]¶
Fetches raw files (level0) from data using public key authentication from data.irf.se. The private key file is specified by privatekey.
- datarootpath:
The data is copied to the local directory datarootpath.
- mask:
mask defines what files are fetched: ‘’ : fetch all level0 files, all existing dates, default ‘201502’ : fetch all level0 files from february 2015 ‘20141227’ : fetch level0 files from 2014-12-27 ‘20141227T00’ : fetch level0 files from 2014-12-27 00h
- progress:
Function with one parameter usually used to print the progress
- privatekey:
Path to the private key file, default ‘~/.ssh/icapipelinekey’
- dataformat (str):
‘mat’ or ‘h5’. if equal to ‘h5’ then the hdf5 data tree is read, if equal to ‘mat’ then the matlab data tree is read.
- onlynewfiles (bool):
If set to True, only files not existing locally will be fetched. Files that exist locally but in an outdated version are not updated. This is useful for slow links. Note this may result in inconsistent data files. Default is False.
- Limitations:
It is assumed that a key exchange has happened previously for user icapy@data.irf.se. Contact Leif Kalla (leif@irf.se) for key exchange.
- Fetch all raw files from 2015 using the key icakey::
- updateraw(‘/home/myuser/icadataroot/’, mask=’2015’,
privatekey=’icakey’ progress=p)
progress is a function with one string parameter that is called for each line rsync returns. Example:
def p(line): print(line)
- irfpy.ica.pipeline.updatelevel0(datarootpath, mask='', privatekey='~/.ssh/icapipelinekey', progress=None, dataformat='mat', onlynewfiles=False, **kwargs)¶
Fetches raw files (level0) from data using public key authentication from data.irf.se. The private key file is specified by privatekey.
- datarootpath:
The data is copied to the local directory datarootpath.
- mask:
mask defines what files are fetched: ‘’ : fetch all level0 files, all existing dates, default ‘201502’ : fetch all level0 files from february 2015 ‘20141227’ : fetch level0 files from 2014-12-27 ‘20141227T00’ : fetch level0 files from 2014-12-27 00h
- progress:
Function with one parameter usually used to print the progress
- privatekey:
Path to the private key file, default ‘~/.ssh/icapipelinekey’
- dataformat (str):
‘mat’ or ‘h5’. if equal to ‘h5’ then the hdf5 data tree is read, if equal to ‘mat’ then the matlab data tree is read.
- onlynewfiles (bool):
If set to True, only files not existing locally will be fetched. Files that exist locally but in an outdated version are not updated. This is useful for slow links. Note this may result in inconsistent data files. Default is False.
- Limitations:
It is assumed that a key exchange has happened previously for user icapy@data.irf.se. Contact Leif Kalla (leif@irf.se) for key exchange.
- Fetch all raw files from 2015 using the key icakey::
- updateraw(‘/home/myuser/icadataroot/’, mask=’2015’,
privatekey=’icakey’ progress=p)
progress is a function with one string parameter that is called for each line rsync returns. Example:
def p(line): print(line)
- irfpy.ica.pipeline.updatecmd(datarootpath, mask='', privatekey='~/.ssh/icapipelinekey', progress=None, dataformat='mat', onlynewfiles=False, **kwargs)[source]¶
Fetches command log files from data using public key authentication from data.irf.se. The private key file is specified by privatekey.
- datarootpath:
The data is copied to the local directory datarootpath. Note that by default the cmd files are part of the level0 data tree.
- mask:
mask defines what files are fetched: mask defines what files are fetched: ‘’ : fetch all cmd files, all existing dates, default ‘201502’ : fetch all cmd files from february 2015 ‘20141227’ : fetch cmd files from 2014-12-27 ‘20141227T00’ : fetch cmd files from 2014-12-27
- progress:
Function with one parameter usually used to print the progress
- privatekey:
Path to the private key file, default ‘~/.ssh/icapipelinekey’
- dataformat (str):
‘mat’ or ‘h5’. if equal to ‘h5’ then the hdf5 data tree is read, if equal to ‘mat’ then the matlab data tree is read.
- onlynewfiles (bool):
If set to True, only files not existing locally will be fetched. Files that exist locally but in an outdated version are not updated. This is useful for slow links. Note this may result in inconsistent data files. Default is False.
- Limitations:
It is assumed that a key exchange has happened previously for user icapy@data.irf.se. Contact Leif Kalla (leif@irf.se) for key exchange.
- Fetch all raw files from 2015 using the key icakey::
- updatecmd(‘/home/myuser/icadataroot/’, mask=’2015’,
privatekey=’icakey’ progress=p)
progress is a function with one string parameter that is called for each line rsync returns. Example:
def p(line): print(line)
- irfpy.ica.pipeline.updateproc(datarootpath, mask='', privatekey='~/.ssh/icapipelinekey', progress=None, dataformat='mat', onlynewfiles=False, **kwargs)[source]¶
Fetches level1 proc files from data using public key authentication from data.irf.se. The private key file is specified by privatekey.
- datarootpath:
The data is copied to the local directory datarootpath.
- mask:
mask defines what files are fetched: ‘’ : fetch all proc files, all existing dates, default ‘201502’ : fetch all proc files from february 2015 ‘20141227T00’ : fetch special from 2014-12-27 00h
- privatekey:
Path to the private key file, default ‘~/.ssh/icapipelinekey’
- progress:
Function with one parameter usually used to print the progress
- dataformat (str):
‘mat’ or ‘h5’. if equal to ‘h5’ then the hdf5 data tree is read, if equal to ‘mat’ then the matlab data tree is read.
- onlynewfiles (bool):
If set to True, only files not existing locally will be fetched. Files that exist locally but in an outdated version are not updated. This is useful for slow links. Note this may result in inconsistent data files. Default is False.
- Limitations:
It is assumed that a key exchange has happened previously for user icapy@data.irf.se. Contact Leif Kalla (leif@irf.se) for key exchange.
- Fetch all proc from 2015 using the key icakey::
- updateproc(‘/home/myuser/icadataroot/’, mask=’2015’,
privatekey=’icakey’,progress=p)
progress is a function with one string parameter that is called for each line rsync returns. Example:
def p(line): print(line)
- irfpy.ica.pipeline.updatespecial(datarootpath, mask='', privatekey='~/.ssh/icapipelinekey', progress=None, dataformat=None, onlynewfiles=False, **kwargs)[source]¶
Fetches special files from data using public key authentication from data.irf.se. The private key file is specified by privatekey.
- datarootpath:
The data is copied to the local directory datarootpath.
- privatekey:
Path to the private key file, default ‘~/.ssh/icapipelinekey’
- progress:
Function with one parameter usually used to print the progress
- onlynewfiles (bool):
If set to True, only files not existing locally will be fetched. Files that exist locally but in an outdated version are not updated. This is useful for slow links. Note this may result in inconsistent data files. Default is False.
- Limitations:
It is assumed that a key exchange has happened previously for user icapy@data.irf.se. Contact Leif Kalla (leif@irf.se) for key exchange.
- Fetch all special from 2015 using the key icakey::
- updatespecial(‘/home/myuser/icadataroot/’,
privatekey=’icakey’,progress=p)
progress is a function with one string parameter that is called for each line rsync returns. Example:
def p(line): print(line)
- irfpy.ica.pipeline.updateaux(datarootpath, mask='', privatekey='~/.ssh/icapipelinekey', progress=None, dataformat='mat', onlynewfiles=False, **kwargs)[source]¶
Fetches aux files from data using public key authentication from data.irf.se. The private key file is specified by privatekey.
- datarootpath:
The data is copied to the local directory datarootpath.
- mask:
mask defines what files are fetched: ‘20141227’ : fetch aux files from 2014-12-27 ‘201412’ : fetch aux files from 2014-12-01 to 2014-12-31
- privatekey:
Path to the private key file, default ‘~/.ssh/icapipelinekey’
- progress:
Function with one parameter usually used to print the progress
- dataformat (str):
‘mat’ or ‘h5’. if equal to ‘h5’ then the hdf5 data tree is read, if equal to ‘mat’ then the matlab data tree is read.
- onlynewfiles (bool):
If set to True, only files not existing locally will be fetched. Files that exist locally but in an outdated version are not updated. This is useful for slow links. Note this may result in inconsistent data files. Default is False.
- Limitations:
It is assumed that a key exchange has happened previously for user icapy@data.irf.se. Contact Leif Kalla (leif@irf.se) for key exchange.
- Fetch all aux from 2015 using the key icakey::
- updatelevel1(‘/home/myuser/icadataroot/’, mask=’2015’,
privatekey=’icakey’ progress=p)
progress is a function with one string parameter that is called for each line rsync returns. Example:
def p(line): print(line)
- irfpy.ica.pipeline.updatelap(datarootpath, mask='', privatekey='~/.ssh/icapipelinekey', progress=None, dataformat='mat', onlynewfiles=False, **kwargs)[source]¶
Fetches lap files from data using public key authentication from data.irf.se. The private key file is specified by privatekey.
- datarootpath:
The data is copied to the local directory datarootpath.
- mask:
mask defines what files are fetched: ‘20141227’ : fetch lap files from 2014-12-27
- privatekey:
Path to the private key file, default ‘~/.ssh/icapipelinekey’
- progress:
Function with one parameter usually used to print the progress
- dataformat (str):
‘mat’ or ‘h5’. if equal to ‘h5’ then the hdf5 data tree is read, if equal to ‘mat’ then the matlab data tree is read.
- onlynewfiles (bool):
If set to True, only files not existing locally will be fetched. Files that exist locally but in an outdated version are not updated. This is useful for slow links. Note this may result in inconsistent data files. Default is False.
- Limitations:
It is assumed that a key exchange has happened previously for user icapy@data.irf.se. Contact Leif Kalla (leif@irf.se) for key exchange.
- Fetch all lap from 2015 using the key icakey::
- updatelevel1(‘/home/myuser/icadataroot/’, mask=’2015’,
privatekey=’icakey’ progress=p)
progress is a function with one string parameter that is called for each line rsync returns. Example:
def p(line): print(line)
- irfpy.ica.pipeline.updatecops(datarootpath, mask='', privatekey='~/.ssh/icapipelinekey', progress=None, dataformat='mat', onlynewfiles=False, **kwargs)[source]¶
Fetches cops files from data using public key authentication from data.irf.se. The private key file is specified by privatekey.
- datarootpath:
The data is copied to the local directory datarootpath.
- mask:
mask defines what files are fetched: ‘20141227’ : fetch cops files from 2014-12-27
- privatekey:
Path to the private key file, default ‘~/.ssh/icapipelinekey’
- progress:
Function with one parameter usually used to print the progress
- dataformat (str):
‘mat’ or ‘h5’. if equal to ‘h5’ then the hdf5 data tree is read, if equal to ‘mat’ then the matlab data tree is read.
- onlynewfiles (bool):
If set to True, only files not existing locally will be fetched. Files that exist locally but in an outdated version are not updated. This is useful for slow links. Note this may result in inconsistent data files. Default is False.
- Limitations:
It is assumed that a key exchange has happened previously for user icapy@data.irf.se. Contact Leif Kalla (leif@irf.se) for key exchange.
- Fetch all cops from 2015 using the key icakey::
- updatecops(‘/home/myuser/icadataroot/’, mask=’2015’,
privatekey=’icakey’ progress=p)
progress is a function with one string parameter that is called for each line rsync returns. Example:
def p(line): print(line)
- irfpy.ica.pipeline.updatemag(datarootpath, mask='', privatekey='~/.ssh/icapipelinekey', progress=None, dataformat='mat', onlynewfiles=False, **kwargs)[source]¶
Fetches mag files from data using public key authentication from data.irf.se. The private key file is specified by privatekey.
- datarootpath:
The data is copied to the local directory datarootpath.
- mask:
mask defines what files are fetched: ‘20141227’ : fetch mag files from 2014-12-27
- privatekey:
Path to the private key file, default ‘~/.ssh/icapipelinekey’
- progress:
Function with one parameter usually used to print the progress
- dataformat (str):
‘mat’ or ‘h5’. if equal to ‘h5’ then the hdf5 data tree is read, if equal to ‘mat’ then the matlab data tree is read.
- onlynewfiles (bool):
If set to True, only files not existing locally will be fetched. Files that exist locally but in an outdated version are not updated. This is useful for slow links. Note this may result in inconsistent data files. Default is False.
- Limitations:
It is assumed that a key exchange has happened previously for user icapy@data.irf.se. Contact Leif Kalla (leif@irf.se) for key exchange.
- Fetch all cops from 2015 using the key icakey::
- updatemag(‘/home/myuser/icadataroot/’, mask=’2015’,
privatekey=’icakey’ progress=p)
progress is a function with one string parameter that is called for each line rsync returns. Example:
def p(line): print(line)
- irfpy.ica.pipeline.updatebestc(datarootpath, mask='', privatekey='~/.ssh/icapipelinekey', progress=None, dataformat='mat', onlynewfiles=False, **kwargs)[source]¶
Fetches bestc (level2) files from data using public key authentication from data.irf.se. The private key file is specified by privatekey.
- datarootpath:
The data is copied to the local directory datarootpath.
- mask:
mask defines what files are fetched: ‘20141227’ : fetch bestc files from 2014-12-27
- privatekey:
Path to the private key file, default ‘~/.ssh/icapipelinekey’
- progress:
Function with one parameter usually used to print the progress
- dataformat (str):
‘mat’ or ‘h5’. if equal to ‘h5’ then the hdf5 data tree is read, if equal to ‘mat’ then the matlab data tree is read.
- onlynewfiles (bool):
If set to True, only files not existing locally will be fetched. Files that exist locally but in an outdated version are not updated. This is useful for slow links. Note this may result in inconsistent data files. Default is False.
- Limitations:
It is assumed that a key exchange has happened previously for user icapy@data.irf.se. Contact Leif Kalla (leif@irf.se) for key exchange.
- Fetch all bestc from 2015 using the key icakey::
- updatebestc(‘/home/myuser/icadataroot/’, mask=’2015’,
privatekey=’icakey’ progress=p)
progress is a function with one string parameter that is called for each line rsync returns. Example:
def p(line): print(line)
- irfpy.ica.pipeline.updateflux(datarootpath, mask='', privatekey='~/.ssh/icapipelinekey', progress=None, dataformat='mat', onlynewfiles=False, **kwargs)[source]¶
Fetches flux (level3) files from data using public key authentication from data.irf.se. The private key file is specified by privatekey.
- datarootpath:
The data is copied to the local directory datarootpath.
- mask:
mask defines what files are fetched:
‘20141227’ : fetch flux files from 2014-12-27
- privatekey:
Path to the private key file, default ‘~/.ssh/icapipelinekey’
- progress:
Function with one parameter usually used to print the progress
- dataformat (str):
‘mat’ or ‘h5’. if ‘h5’ then the hdf5 data tree is read if ‘mat’ then the matlab data tree is read.
- onlynewfiles (bool):
If set to True, only files not existing locally will be fetched. Files that exist locally but in an outdated version are not updated. This is useful for slow links. Note this may result in inconsistent data files. Default is False.
- Limitations:
It is assumed that a key exchange has happened previously for user icapy@data.irf.se. Contact Leif Kalla (leif@irf.se) for key exchange.
- Fetch all flux from 2015 using the key icakey::
- updatebestc(‘/home/myuser/icadataroot/’, mask=’2015’,
privatekey=’icakey’ progress=p)
progress is a function with one string parameter that is called for each line rsync returns. Example:
def p(line): print(line)