# irfpy.ica.io¶

Read data from a local copy of the ICA data pipeline.

Author: Martin Wieser

Module: irfpy.ica.io

Functions to load transparently ICA pipeline data from the lcoal filesystem. Functions all return a dictionary with all loaded variables.

mat=readraw('~/icadata','20150211','20150213',variables=['time_interval','E'])

Summary:

The most modern and environmentally friendly way to access ICA data:

For more advanced ways to access the data use these low level functions:

Read any version of .mat or .h5 file directly without doing the data type conversions of the load or read functions:

• irfpy.ica.io.savemat

• irfpy.ica.io.saveh5

irfpy.ica.io.readraw(datarootpath, from_day=None, to_day='', variables=None, verbose=False, flat=False, dataformat='mat', **kwargs)[source]

Loads one hour or more worth of level0 data analog to scipy.io.loadmat(‘xyz.mat’) but with assembling all hourly files corresponding to the time interval specified.

AUTHOR: Gabriella Stenberg Wieser

PARAMETERS

datarootpath (string):

local path to the ‘root’ of the data tree This data tree should contain the file(s) in subdirectories level0/xxxx/yyyy/mm/dd/ as it is done in the pipeline.

from_day (string or datetime.datetime):

string is of the form ‘20150115T1011’

to_day (string or datetime.datetime):

optional, string is of the form ‘20150116T1112’. If given data to the end of the given intervall is loaded, otherwise one day.

variables (list of strings):

a list of variable names to load. Default is to load all variables. If the loaded variables include ‘time_instances’ then these are automatically converted to python datetime object

flat (boolean):

If flat is set to True, the load function will if no files were found also attempt to load files which are at the root of the data tree (without the yyyy/mm/dd/ subdirectories)

OUTPUT:

returns a dictionary with all loaded variables

EXAMPLES:

mat=readraw('~/icadata','20150211')

irfpy.ica.io.readproc(datarootpath, from_day=None, to_day='', variables=None, verbose=False, flat=False, includeerrors=False, partialelevationscan=False, dataformat='mat', filters=None, time_instances='time_instances')[source]

Loads one day or more worth of proc*.mat data analog to scipy.io.loadmat(‘xyz.mat’) but with assembling all hourly files corresponding to that day.

PARAMETERS

datarootpath (string):

local path to the ‘root’ of a data directory tree This data tree should contain the file(s) in subdirectories level1/xxx/yyyy/mm/dd/ as it is done in the pipeline.

from_day (string or datetime.datetime):

string is of the form ‘20150115T1011’

to_day (string or datetime.datetime):

optional, string is of the form ‘20150116T1112’. If given data to the end of the given intervall is loaded, otherwise one day.

variables (dict):

a list of variable names to load. Default is to load all variables. If the loaded variables include ‘time_instances’ then these are automatically converted to python datetime object

flat (boolean):

If flat is set to True, the load function will if no files were found also attempt to load files which are at the root of the data tree (without the yyyy/mm/dd/ subdirectories)

partialelevationscan (boolean):

Optional parameter, default False. If set to False the loading function will only load complete elevation scans. If set to True, the neartest time_instances will be loaded independent of where an elevation scan starts.

verbose (boolean):

Optional parameter, if set to True the loading function will print the names of the files it loads for debugging purposes

includeerrors (boolean), OUTDATED:

Optional parameter, if set to True the loading function will also load data with decoder or formatting errors This parameter is outdated. Use filters keyword instead, e.g. filters={‘time_error_flag’:0}

dataformat (str):

‘mat’ or ‘h5’. if h5 then the hdf5 data tree is read ifmat then the matlab data tree is read.

OUTPUT

Returns a dictionary with all loaded variables. The dictionary is empty if no matching data could be loaded

Use like this:

mat=readproc('~/icadata','20150211')

irfpy.ica.io.readaux(datarootpath, from_day=None, to_day='', variables=None, verbose=False, flat=False, includeerrors=False, partialelevationscan=False, dataformat='mat', filters=None, time_instances='time_instances')[source]

Loads one day or more worth of aux*.mat data analog to scipy.io.loadmat(‘xyz.mat’) but with assembling all hourly files corresponding to that day.

PARAMETERS

datarootpath (string):

local path to the ‘root’ of a data directory tree This data tree should contain the file(s) in subdirectories aux/xxxx/yyyy/mm/dd/ as it is done in the pipeline.

from_day (string or datetime.datetime):

string is of the form ‘20150115T1011’

to_day (string or datetime.datetime):

optional, string is of the form ‘20150116T1112’. If given data to the end of the given intervall is loaded, otherwise one day.

variables (dict):

a list of variable names to load. Default is to load all variables. If the loaded variables include ‘time_instances’ then these are automatically converted to python datetime object

flat (boolean):

If flat is set to True, the load function will if no files were found also attempt to load files which are at the root of the data tree (without the yyyy/mm/dd/ subdirectories)

partialelevationscan (boolean):

Optional parameter, default False. If set to False the loading function will only load complete elevation scans. If set to True, the neartest time_instances will be loaded independent of where an elevation scan starts.

verbose (boolean):

Optional parameter, if set to True the loading function will print the names of the files it loads for debugging purposes

includeerrors (boolean), OUTDATED:

Optional parameter, if set to True the loading function will also load data with decoder or formatting erroes This parameter is outdated. Use filters keyword instead, e.g. filters={‘time_error_flag’:0}

dataformat (str):

‘mat’ or ‘h5’. if h5 then the hdf5 data tree is read ifmat then the matlab data tree is read.

OUTPUT

Returns a dictionary with all loaded variables. The dictionary is empty if no matching data could be loaded. If bs_xxx variables are wanted and they are not in the aux files loaded, then an automatic fallback to xaux files is done.

Use like this:

mat=readaux('~/icadata','20150211')


Loads one day or more worth of special*.mat data analog to scipy.io.loadmat(‘xyz.mat’) but with assembling all hourly files corresponding to that day.

PARAMETERS

datarootpath (string):

local path to the ‘root’ of a data directory tree This data tree should contain the file(s) in subdirectories special/gen??? as it is done in the pipeline.

generation (int):

generation number of the requested files. If None, variables are loaded from the newest generation.

variables (dict):

a list of variable names to load. Default is to load all variables. The variables are collected from .npz and .mat files in this order of priority.

verbose (boolean):

Optional parameter, if set to True the loading function will print the names of the files it loads for debugging purposes

OUTPUT

Returns a dictionary with all loaded variables. The dictionary is empty if no matching data could be loaded

Use like this:

mat=readspecial('~/icadata')

irfpy.ica.io.readlap(datarootpath, from_day=None, to_day='', variables=None, verbose=False, flat=False, dataformat='mat', filters=None, time_instances='asw_time')[source]

Loads one day or more worth of lap*.mat data analog to scipy.io.loadmat(‘xyz.mat’) but with assembling all hourly files corresponding to that day.

PARAMETERS

datarootpath (string):

local path to the ‘root’ of a data directory tree This data tree should contain the file(s) in subdirectories lap/xxxx/yyyy/mm/dd/ as it is done in the pipeline.

from_day (string or datetime.datetime):

string is of the form ‘20150115T1011’

to_day (string or datetime.datetime):

optional, string is of the form ‘20150116T1112’. If given data to the end of the given intervall is loaded, otherwise one day.

variables (dict):

a list of variable names to load. Default is to load all variables. If the loaded variables include ‘time_instances’ then these are automatically converted to python datetime object

flat (boolean):

If flat is set to True, the load function will if no files were found also attempt to load files which are at the root of the data tree (without the yyyy/mm/dd/ subdirectories)

verbose (boolean):

Optional parameter, if set to True the loading function will print the names of the files it loads for debugging purposes

dataformat (str):

‘mat’ or ‘h5’. if h5 then the hdf5 data tree is read ifmat then the matlab data tree is read.

OUTPUT

Returns a dictionary with all loaded variables. The dictionary is empty if no matching data could be loaded

Use like this:

mat=readlap('~/icadata','20150211')

irfpy.ica.io.readcops(datarootpath, from_day=None, to_day='', variables=None, verbose=False, flat=False, dataformat='mat', filters=None, time_instances='cops_time')[source]

Loads one day or more worth of cops*.mat data analog to scipy.io.loadmat(‘xyz.mat’) but with assembling all hourly files corresponding to that day.

PARAMETERS

datarootpath (string):

local path to the ‘root’ of a data directory tree This data tree should contain the file(s) in subdirectories cops/xxxx/yyyy/mm/dd/ as it is done in the pipeline.

from_day (string or datetime.datetime):

string is of the form ‘20150115T1011’

to_day (string or datetime.datetime):

optional, string is of the form ‘20150116T1112’. If given data to the end of the given intervall is loaded, otherwise one day.

variables (dict):

a list of variable names to load. Default is to load all variables. If the loaded variables include ‘time_instances’ then these are automatically converted to python datetime object

flat (boolean):

If flat is set to True, the load function will if no files were found also attempt to load files which are at the root of the data tree (without the yyyy/mm/dd/ subdirectories)

verbose (boolean):

Optional parameter, if set to True the loading function will print the names of the files it loads for debugging purposes

dataformat (str):

‘mat’ or ‘h5’. if h5 then the hdf5 data tree is read ifmat then the matlab data tree is read.

OUTPUT

Returns a dictionary with all loaded variables. The dictionary is empty if no matching data could be loaded

Use like this:

mat=readcops('~/icadata','20150211')

irfpy.ica.io.readmag(datarootpath, from_day=None, to_day='', variables=None, verbose=False, flat=False, dataformat='mat', filters=None, time_instances='mag_time')[source]

Loads one day or more worth of mag*.mat data analog to scipy.io.loadmat(‘xyz.mat’) but with assembling all hourly files corresponding to that day.

PARAMETERS

datarootpath (string):

local path to the ‘root’ of a data directory tree This data tree should contain the file(s) in subdirectories mag/xxxx/yyyy/mm/dd/ as it is done in the pipeline.

from_day (string or datetime.datetime):

string is of the form ‘20150115T1011’

to_day (string or datetime.datetime):

optional, string is of the form ‘20150116T1112’. If given data to the end of the given intervall is loaded, otherwise one day.

variables (dict):

a list of variable names to load. Default is to load all variables. If the loaded variables include ‘time_instances’ then these are automatically converted to python datetime object

flat (boolean):

If flat is set to True, the load function will if no files were found also attempt to load files which are at the root of the data tree (without the yyyy/mm/dd/ subdirectories)

verbose (boolean):

Optional parameter, if set to True the loading function will print the names of the files it loads for debugging purposes

dataformat (str):

‘mat’ or ‘h5’. if h5 then the hdf5 data tree is read ifmat then the matlab data tree is read.

OUTPUT

Returns a dictionary with all loaded variables. The dictionary is empty if no matching data could be loaded

Use like this:

mat=readmag('~/icadata','20150211')

irfpy.ica.io.readbestc(datarootpath, from_day=None, to_day='', variables=None, verbose=False, flat=False, partialelevationscan=False, dataformat='mat', filters=None, time_instances='time_instances')[source]

Loads one day or more worth of bestc*.mat data analog to scipy.io.loadmat(‘xyz.mat’) but with assembling all hourly files corresponding to that day.

PARAMETERS

datarootpath (string):

local path to the ‘root’ of a data directory tree This data tree should contain the file(s) in subdirectories level2/xxxx/yyyy/mm/dd/ as it is done in the pipeline.

from_day (string or datetime.datetime):

string is of the form ‘20150115T1011’

to_day (string or datetime.datetime):

optional, string is of the form ‘20150116T1112’. If given data to the end of the given intervall is loaded, otherwise one day.

variables (dict):

a list of variable names to load. Default is to load all variables. If the loaded variables include ‘time_instances’ then these are automatically converted to python datetime object

flat (boolean):

If flat is set to True, the load function will if no files were found also attempt to load files which are at the root of the data tree (without the yyyy/mm/dd/ subdirectories)

partialelevationscan (boolean):

Optional parameter, default False. If set to False the loading function will only load complete elevation scans. If set to True, the neartest time_instances will be loaded independent of where an elevation scan starts.

verbose (boolean):

Optional parameter, if set to True the loading function will print the names of the files it loads for debugging purposes

dataformat (str):

‘mat’ or ‘h5’. if h5 then the hdf5 data tree is read ifmat then the matlab data tree is read.

OUTPUT

Returns a dictionary with all loaded variables. The dictionary is empty if no matching data could be loaded

Use like this:

mat=readbestc('~/icadata','20150211')

irfpy.ica.io.readflux(datarootpath, from_day=None, to_day='', variables=None, verbose=False, flat=False, partialelevationscan=False, dataformat='mat', filters=None, time_instances='time_instances')[source]

Loads one day or more worth of flux*.mat data analog to scipy.io.loadmat(‘xyz.mat’) but with assembling all hourly files corresponding to that day.

PARAMETERS

datarootpath (string):

local path to the ‘root’ of a data directory tree This data tree should contain the file(s) in subdirectories level3/xxxx/yyyy/mm/dd/ as it is done in the pipeline.

from_day (string or datetime.datetime):

string is of the form ‘20150115T1011’

to_day (string or datetime.datetime):

optional, string is of the form ‘20150116T1112’. If given data to the end of the given intervall is loaded, otherwise one day.

variables (dict):

a list of variable names to load. Default is to load all variables. If the loaded variables include ‘time_instances’ then these are automatically converted to python datetime object

flat (boolean):

If flat is set to True, the load function will if no files were found also attempt to load files which are at the root of the data tree (without the yyyy/mm/dd/ subdirectories)

partialelevationscan (boolean):

Optional parameter, default False. If set to False the loading function will only load complete elevation scans. If set to True, the neartest time_instances will be loaded independent of where an elevation scan starts.

verbose (boolean):

Optional parameter, if set to True the loading function will print the names of the files it loads for debugging purposes

dataformat (str):

‘mat’ or ‘h5’. if h5 then the hdf5 data tree is read ifmat then the matlab data tree is read.

OUTPUT

Returns a dictionary with all loaded variables. The dictionary is empty if no matching data could be loaded

Use like this:

mat=readflux('~/icadata','20150211')

irfpy.ica.io.conditional_gc_collect()[source]

Calls gc.collect() if the available memory is less than 30%.

Loads the dictionary “variable_names” from a v7 or v7.3 mat file. if the dictionary is not specified, then everything is loaded.:

filename = 'test.mat'
print (m.keys())


loads the dictionary “variable_names” from a h5 file.

irfpy.ica.io.savemat(file, data)[source]

Save the dictionary data in v7.3 mat format to file analog to scipy.io.savemat:

a = np.range(10)
b = np.range(33)
c = np.range(22) ** 2

filename = 'test.mat'

m = dict()
m['a'] = a
m['b'] = b
m['c'] = c

irfpy.ica.io.savemat(filename,m)

irfpy.ica.io.saveh5(file, data)[source]

Save the dictionary data in hdf5 format using deepdish

irfpy.ica.io.minonedim(a)[source]

return an array with at least one dimension based on a

irfpy.ica.io.flatstring(version)[source]

Tries hard to make a string out of the argument.

irfpy.ica.io.flatstringlist(list_of_lists)[source]

Tries hard to make a list of strings out of the argument.

irfpy.ica.io.expectedversion(proc=None, aux=None, bestc=None, flux=None, lap=None, cops=None, mag=None, mip=None, verbose=True, **kwargs)[source]

Defines the data versions that the user would like to have. If the existing version is different, then at the first occurence there will be a warning printed explaining what the difference is between the required version and the present version. If no version is specified then all versions are allowed.

All parameters are strings of the form “xx.y” or floating numbers. e.g. proc=’2.3’ means that for proc, version 2.3 is requested.

The version found in the datafiles is printed once. If verbose is set to false, this is suppressed.

Usage:

expectedversion(proc='2.3',lap='0.3')

irfpy.ica.io.loadlevelN(datarootpath, fileprefix='bestc', from_day=None, to_day='', variables=None, verbose=False, flat=False, partialelevationscan=False, dataformat='mat', branch=None, time_instances='time_instances', filters=None, varshapes=None)[source]

Loads one day or more worth of level1 data analog to scipy.io.loadmat(‘xyz.mat’) but with assembling all hourly files corresponding to that day.

PARAMETERS

datarootpath (string):

local path to the ‘root’ of the level1 data tree This data tree should contain the file(s) in subdirectories yyyy/mm/dd/ as it is done in the pipeline.

fileprefix (string):

from_day (string or datetime.datetime):

string is of the form ‘20150115T1011’

to_day (string or datetime.datetime):

optional, string is of the form ‘20150116T1112’. If given data to the end of the given intervall is loaded, otherwise one day.

variables (dict):

a list of variable names to load. Default is to load all variables. If the loaded variables include ‘time_instances’ then these are automatically converted to python datetime object

flat (boolean):

If flat is set to True, the load function will if no files were found also attempt to load files which are at the root of the data tree (without the yyyy/mm/dd/ subdirectories)

partialelevationscan (boolean):

Optional parameter, default False. If set to False the loading function will only load complete elevation scans. If set to True, the neartest time_instances will be loaded independent of where an elevation scan starts.

verbose (boolean):

Optional parameter, if set to True the loading function will print the names of the files it loads for debugging purposes.

dataformat (string):

One of: ‘mat’(default) or ‘h5’.

time_instances (str):

Name of the time_instances variable (default: ‘time_instances’)

branch (str):

Directory below datarootpath the data is stored, e.g. ‘level1’

filters (dict):

Filter list

varshapes (dict):

data structure describing the dimensionality and the special data types of the variables to be read. Use predefined defaults, e.g. default_varshapes1 for level1 variable shapes.

OUTPUT

Returns a dictionary with all loaded variables. The dictionary is empty if no matching data could be loaded

Use like this:

matfile=irfpy.ica.io.loadlevel?(icapath,'bestc','20150211')
variables=['time_instances','E'])
variables=['time_instances','E'])

irfpy.ica.io.loadlevel2(datarootpath, mattype='bestc', from_day=None, to_day='', variables=None, verbose=False, flat=False, partialelevationscan=False, dataformat='mat', filters=None)[source]

Loads one day or more worth of level1 data analog to scipy.io.loadmat(‘xyz.mat’) but with assembling all hourly files corresponding to that day.

PARAMETERS

datarootpath (string):

local path to the ‘root’ of the level1 data tree This data tree should contain the file(s) in subdirectories yyyy/mm/dd/ as it is done in the pipeline.

mattype (string):

one of ‘bestc’

from_day (string or datetime.datetime):

string is of the form ‘20150115T1011’

to_day (string or datetime.datetime):

optional, string is of the form ‘20150116T1112’. If given data to the end of the given intervall is loaded, otherwise one day.

variables (dict):

a list of variable names to load. Default is to load all variables. If the loaded variables include ‘time_instances’ then these are automatically converted to python datetime object

flat (boolean):

If flat is set to True, the load function will if no files were found also attempt to load files which are at the root of the data tree (without the yyyy/mm/dd/ subdirectories)

partialelevationscan (boolean):

Optional parameter, default False. If set to False the loading function will only load complete elevation scans. If set to True, the neartest time_instances will be loaded independent of where an elevation scan starts.

verbose (boolean):

Optional parameter, if set to True the loading function will print the names of the files it loads for debugging purposes.

dataformat (string):

One of: ‘mat’(default) or ‘h5’.

OUTPUT

Returns a dictionary with all loaded variables. The dictionary is empty if no matching data could be loaded

Use like this:

matfile=irfpy.ica.io.loadlevel2(icapath,'bestc','20150211')
variables=['time_instances','E'])
variables=['time_instances','E'])

irfpy.ica.io.loadlevel1(datarootpath, mattype='proc', from_day=None, to_day='', variables=None, verbose=False, flat=False, partialelevationscan=False, dataformat='mat', filters=None)[source]

Loads one day or more worth of level1 data analog to scipy.io.loadmat(‘xyz.mat’) but with assembling all hourly files corresponding to that day.

PARAMETERS

datarootpath (string):

local path to the ‘root’ of the level1 data tree This data tree should contain the file(s) in subdirectories yyyy/mm/dd/ as it is done in the pipeline.

mattype (string):

one of ‘proc’, ‘special’ or ‘aux’

from_day (string or datetime.datetime):

string is of the form ‘20150115T1011’

to_day (string or datetime.datetime):

optional, string is of the form ‘20150116T1112’. If given data to the end of the given intervall is loaded, otherwise one day.

variables (dict):

a list of variable names to load. Default is to load all variables. If the loaded variables include ‘time_instances’ then these are automatically converted to python datetime object

flat (boolean):

If flat is set to True, the load function will if no files were found also attempt to load files which are at the root of the data tree (without the yyyy/mm/dd/ subdirectories)

partialelevationscan (boolean):

Optional parameter, default False. If set to False the loading function will only load complete elevation scans. If set to True, the neartest time_instances will be loaded independent of where an elevation scan starts.

verbose (boolean):

Optional parameter, if set to True the loading function will print the names of the files it loads for debugging purposes

dataformat (string):

One of: ‘mat’(default) or ‘h5’.

OUTPUT

Returns a dictionary with all loaded variables. The dictionary is empty if no matching data could be loaded

Use like this::

variables=[‘time_instances’,’E’])

‘20150211’,’20150213’, variables=[‘time_instances’,’E’])

variables=[‘sp_cso’,’time_instances’])

irfpy.ica.io.loadlevel1oktime(datarootpath, mattype, from_day, to_day='', variables=None, verbose=False, flat=False, partialelevationscan=False, dataformat='mat', filters=None)[source]

Loads one day or more worth of level1 data analog to irfpy.ica.io.loadmat(‘xyz.mat’) but with assembling all hourly files corresponding to that day. Only data with timeswhere time_error_flag is == 0 is loaded

PARAMETERS

datarootpath (string):

local path to the ‘root’ of the level1 data tree This data tree should contain the file(s) in subdirectories yyyy/mm/dd/ as it is done in the pipeline.

mattype (string):

filetype to be loaded, one of ‘proc’ or ‘aux’

from_day (string or datetime.datetime):

string is of the form ‘20150115T1011’

to_day (string or datetime.datetime):

optional, string is of the form ‘20150116T1112’. If given data to the end of the given intervall is loaded, otherwise one day.

variables (dict):

a list of variable names to load. Default is to load all variables. If the loaded variables include ‘time_instances’ then these are automatically converted to python datetime object

flat (boolean):

If flat is set to True, the load function will if no files were found also attempt to load files which are at the root of the data tree (without the yyyy/mm/dd/ subdirectories)

partialelevationscan (boolean):

Optional parameter, default False. If set to False the loading function will only load complete elevation scans. If set to True, the neartest time_instances will be loaded independent of where an elevation scan starts.

verbose (boolean):

Prints what the function does

dataformat (string):

One of: ‘mat’(default) or ‘h5’.

This function is identical to the call of loadlevel1 with a filters parameter:

OUTPUT

Returns a dictionary with all loaded variables

Limitation: If this function is used to load an aux file, the aux file

must be version 1.9 or larger. This function can not be used to load special files.

irfpy.ica.io.loadlevel1okall(datarootpath, mattype='proc', from_day=None, to_day='', variables=None, verbose=False, flat=False, partialelevationscan=False, dataformat='mat', filters=None)[source]

Loads one day or more worth of level1 data analog to irfpy.ica.io.loadmat(‘xyz.mat’) but with assembling all hourly files corresponding to that day. Only data with times where error_flags is == 0 is loaded

PARAMETERS

datarootpath (string):

local path to the ‘root’ of the level1 data tree This data tree should contain the file(s) in subdirectories yyyy/mm/dd/ as it is done in the pipeline.

mattype (string):

filetype to be loaded, one of ‘proc’ or ‘aux’

from_day (string or datetime.datetime):

string is of the form ‘20150115T1011’

to_day (string or datetime.datetime):

optional, string is of the form ‘20150116T1112’. If given data to the end of the given intervall is loaded, otherwise one day.

variables (dict):

a list of variable names to load. Default is to load all variables. If the loaded variables include ‘time_instances’ then these are automatically converted to python datetime object

flat (boolean):

If flat is set to True, the load function will if no files were found also attempt to load files which are at the root of the data tree (without the yyyy/mm/dd/ subdirectories)

partialelevationscan (boolean):

Optional parameter, default False. If set to False the loading function will only load complete elevation scans. If set to True, the neartest time_instances will be loaded independent of where an elevation scan starts.

verbose (boolean):

Prints what the function does

dataformat (string):

One of: ‘mat’(default) or ‘h5’.

This function is identical to the call of loadlevel1 with a filters parameter:

OUTPUT

Returns a dictionary with all loaded variables

irfpy.ica.io.loadlevel1filter(datarootpath, mattype='proc', from_day=None, to_day='', variables=None, verbose=False, flat=False, partialelevationscan=False, dataformat='mat', time_error_flag=None, decoder_error_flag=None, fmt_error_flag=None, edf_error_flag=None, error_flags=None, mode=None, sw_version=None, cur_pacc=None)[source]

Loads one day or more worth of level1 data analog to irfpy.ica.io.loadmat(‘xyz.mat’) but with assembling all hourly files corresponding to that day. Only data with times where the filter criteria apply are loaded

PARAMETERS

datarootpath (string):

local path to the ‘root’ of the level1 data tree This data tree should contain the file(s) in subdirectories yyyy/mm/dd/ as it is done in the pipeline.

mattype (string):

filetype to be loaded, one of ‘proc’ or ‘aux’

from_day (string or datetime.datetime):

string is of the form ‘20150115T1011’

to_day (string or datetime.datetime):

optional, string is of the form ‘20150116T1112’. If given data to the end of the given intervall is loaded, otherwise one day.

variables (dict):

a list of variable names to load. Default is to load all variables. If the loaded variables include ‘time_instances’ then these are automatically converted to python datetime object

time_error_flag (int) : 0 decoder_error_flag (int) : 0 fmt_error_flag (int) : 0 edf_error_flag (int) : 0 error_flags (int) : 0 mode (int) : 0 sw_version (int) : 6 cur_pacc (int) : 6

flat (boolean):

If flat is set to True, the load function will if no files were found also attempt to load files which are at the root of the data tree (without the yyyy/mm/dd/ subdirectories)

partialelevationscan (boolean):

Optional parameter, default False. If set to False the loading function will only load complete elevation scans. If set to True, the neartest time_instances will be loaded independent of where an elevation scan starts.

verbose (boolean):

Prints what the function does

dataformat (string):

One of: ‘mat’(default) or ‘h5’.

OUTPUT

Returns a dictionary with all loaded variables.

irfpy.ica.io.loadlevel0(datarootpath, from_day, to_day='', variables=None, verbose=False, flat=False, dataformat='mat', **kwargs)[source]

Loads one hour or more worth of level0 data analog to scipy.io.loadmat(‘xyz.mat’) but with assembling all hourly files corresponding to the time interval specified.

AUTHOR: Gabriella Stenberg Wieser

PARAMETERS

datarootpath (string):

local path to the ‘root’ of the level0 data tree This data tree should contain the file(s) in subdirectories yyyy/mm/dd/ as it is done in the pipeline.

from_day (string or datetime.datetime):

string is of the form ‘20150115T1011’

to_day (string or datetime.datetime):

optional, string is of the form ‘20150116T1112’. If given data to the end of the given intervall is loaded, otherwise one day.

variables (list of strings):

a list of variable names to load. Default is to load all variables. If the loaded variables include ‘time_instances’ then these are automatically converted to python datetime object

flat (boolean):

If flat is set to True, the load function will if no files were found also attempt to load files which are at the root of the data tree (without the yyyy/mm/dd/ subdirectories)

dataformat (string):

One of: ‘mat’(default) or ‘h5’.

OUTPUT

returns a dictionary with all loaded variables

EXAMPLES:

matfile=ica.io.loadlevel0(icapath,'20150211')