irfpy.ica.io¶
Read data from a local copy of the ICA data pipeline.
Author: Martin Wieser
Module: irfpy.ica.io
Functions to load transparently ICA pipeline data from the lcoal filesystem. Functions all return a dictionary with all loaded variables.
Example load data:
mat=readraw('~/icadata','20150211','20150213',variables=['time_interval','E'])
mat=readproc('~/icadata','20150211','20150213',variables=['time_instances','E'])
mat=readlap('~/icadata','20150211',variables=['lap_time','lap_potential'])
- Summary:
The most modern and environmentally friendly way to access ICA data:
irfpy.ica.io.readraw
irfpy.ica.io.readproc
irfpy.ica.io.readaux
irfpy.ica.io.readspecial
irfpy.ica.io.readlap
irfpy.ica.io.readcops
irfpy.ica.io.readbestc
irfpy.ica.io.readflux
For more advanced ways to access the data use these low level functions:
irfpy.ica.io.loadlevel0
irfpy.ica.io.loadlevel1
irfpy.ica.io.loadlevel1oktime
irfpy.ica.io.loadlevel1okall
irfpy.ica.io.loadlevel1filter
irfpy.ica.io.loadlevel2
irfpy.ica.io.loadlevel3
Read any version of .mat or .h5 file directly without doing the data type conversions of the load or read functions:
irfpy.ica.io.loadmat
irfpy.ica.io.savemat
irfpy.ica.io.loadh5
irfpy.ica.io.saveh5
- irfpy.ica.io.readraw(datarootpath, from_day=None, to_day='', variables=None, verbose=False, flat=False, dataformat='mat', **kwargs)[source]¶
Loads one hour or more worth of level0 data analog to scipy.io.loadmat(‘xyz.mat’) but with assembling all hourly files corresponding to the time interval specified.
AUTHOR: Gabriella Stenberg Wieser
PARAMETERS
- datarootpath (string):
local path to the ‘root’ of the data tree This data tree should contain the file(s) in subdirectories level0/xxxx/yyyy/mm/dd/ as it is done in the pipeline.
- from_day (string or datetime.datetime):
string is of the form ‘20150115T1011’
- to_day (string or datetime.datetime):
optional, string is of the form ‘20150116T1112’. If given data to the end of the given intervall is loaded, otherwise one day.
- variables (list of strings):
a list of variable names to load. Default is to load all variables. If the loaded variables include ‘time_instances’ then these are automatically converted to python datetime object
- flat (boolean):
If flat is set to True, the load function will if no files were found also attempt to load files which are at the root of the data tree (without the yyyy/mm/dd/ subdirectories)
OUTPUT:
returns a dictionary with all loaded variables
EXAMPLES:
mat=readraw('~/icadata','20150211') mat=readraw('~/icadata','20150211',variables=['iontime','ionspectra']) mat=readraw('~/icadata','20150211','20150213',variables=['time_interval','E'])
- irfpy.ica.io.readproc(datarootpath, from_day=None, to_day='', variables=None, verbose=False, flat=False, includeerrors=False, partialelevationscan=False, dataformat='mat', filters=None, time_instances='time_instances')[source]¶
Loads one day or more worth of proc*.mat data analog to scipy.io.loadmat(‘xyz.mat’) but with assembling all hourly files corresponding to that day.
PARAMETERS
- datarootpath (string):
local path to the ‘root’ of a data directory tree This data tree should contain the file(s) in subdirectories level1/xxx/yyyy/mm/dd/ as it is done in the pipeline.
- from_day (string or datetime.datetime):
string is of the form ‘20150115T1011’
- to_day (string or datetime.datetime):
optional, string is of the form ‘20150116T1112’. If given data to the end of the given intervall is loaded, otherwise one day.
- variables (dict):
a list of variable names to load. Default is to load all variables. If the loaded variables include ‘time_instances’ then these are automatically converted to python datetime object
- flat (boolean):
If flat is set to True, the load function will if no files were found also attempt to load files which are at the root of the data tree (without the yyyy/mm/dd/ subdirectories)
- partialelevationscan (boolean):
Optional parameter, default False. If set to False the loading function will only load complete elevation scans. If set to True, the neartest time_instances will be loaded independent of where an elevation scan starts.
- verbose (boolean):
Optional parameter, if set to True the loading function will print the names of the files it loads for debugging purposes
- includeerrors (boolean), OUTDATED:
Optional parameter, if set to True the loading function will also load data with decoder or formatting errors This parameter is outdated. Use filters keyword instead, e.g. filters={‘time_error_flag’:0}
- dataformat (str):
‘mat’ or ‘h5’. if h5 then the hdf5 data tree is read ifmat then the matlab data tree is read.
OUTPUT
Returns a dictionary with all loaded variables. The dictionary is empty if no matching data could be loaded
Use like this:
mat=readproc('~/icadata','20150211') mat=readproc('~/icadata','20150211',variables=['time_instances','E']) mat=readproc('~/icadata','20150211','20150213',variables=['time_instances','E'])
- irfpy.ica.io.readaux(datarootpath, from_day=None, to_day='', variables=None, verbose=False, flat=False, includeerrors=False, partialelevationscan=False, dataformat='mat', filters=None, time_instances='time_instances')[source]¶
Loads one day or more worth of aux*.mat data analog to scipy.io.loadmat(‘xyz.mat’) but with assembling all hourly files corresponding to that day.
PARAMETERS
- datarootpath (string):
local path to the ‘root’ of a data directory tree This data tree should contain the file(s) in subdirectories aux/xxxx/yyyy/mm/dd/ as it is done in the pipeline.
- from_day (string or datetime.datetime):
string is of the form ‘20150115T1011’
- to_day (string or datetime.datetime):
optional, string is of the form ‘20150116T1112’. If given data to the end of the given intervall is loaded, otherwise one day.
- variables (dict):
a list of variable names to load. Default is to load all variables. If the loaded variables include ‘time_instances’ then these are automatically converted to python datetime object
- flat (boolean):
If flat is set to True, the load function will if no files were found also attempt to load files which are at the root of the data tree (without the yyyy/mm/dd/ subdirectories)
- partialelevationscan (boolean):
Optional parameter, default False. If set to False the loading function will only load complete elevation scans. If set to True, the neartest time_instances will be loaded independent of where an elevation scan starts.
- verbose (boolean):
Optional parameter, if set to True the loading function will print the names of the files it loads for debugging purposes
- includeerrors (boolean), OUTDATED:
Optional parameter, if set to True the loading function will also load data with decoder or formatting erroes This parameter is outdated. Use filters keyword instead, e.g. filters={‘time_error_flag’:0}
- dataformat (str):
‘mat’ or ‘h5’. if h5 then the hdf5 data tree is read ifmat then the matlab data tree is read.
OUTPUT
Returns a dictionary with all loaded variables. The dictionary is empty if no matching data could be loaded. If bs_xxx variables are wanted and they are not in the aux files loaded, then an automatic fallback to xaux files is done.
Use like this:
mat=readaux('~/icadata','20150211') mat=readaux('~/icadata','20150211',variables=['sp_cso','time_instances']) mat=readaux('~/icadata','20150211','20150213')
- irfpy.ica.io.readspecial(datarootpath, generation=None, variables=None, verbose=False)[source]¶
Loads one day or more worth of special*.mat data analog to scipy.io.loadmat(‘xyz.mat’) but with assembling all hourly files corresponding to that day.
PARAMETERS
- datarootpath (string):
local path to the ‘root’ of a data directory tree This data tree should contain the file(s) in subdirectories special/gen??? as it is done in the pipeline.
- generation (int):
generation number of the requested files. If None, variables are loaded from the newest generation.
- variables (dict):
a list of variable names to load. Default is to load all variables. The variables are collected from .npz and .mat files in this order of priority.
- verbose (boolean):
Optional parameter, if set to True the loading function will print the names of the files it loads for debugging purposes
OUTPUT
Returns a dictionary with all loaded variables. The dictionary is empty if no matching data could be loaded
Use like this:
mat=readspecial('~/icadata') mat=readspecial('~/icadata',generation=6,verbose=True)
- irfpy.ica.io.readlap(datarootpath, from_day=None, to_day='', variables=None, verbose=False, flat=False, dataformat='mat', filters=None, time_instances='asw_time')[source]¶
Loads one day or more worth of lap*.mat data analog to scipy.io.loadmat(‘xyz.mat’) but with assembling all hourly files corresponding to that day.
PARAMETERS
- datarootpath (string):
local path to the ‘root’ of a data directory tree This data tree should contain the file(s) in subdirectories lap/xxxx/yyyy/mm/dd/ as it is done in the pipeline.
- from_day (string or datetime.datetime):
string is of the form ‘20150115T1011’
- to_day (string or datetime.datetime):
optional, string is of the form ‘20150116T1112’. If given data to the end of the given intervall is loaded, otherwise one day.
- variables (dict):
a list of variable names to load. Default is to load all variables. If the loaded variables include ‘time_instances’ then these are automatically converted to python datetime object
- flat (boolean):
If flat is set to True, the load function will if no files were found also attempt to load files which are at the root of the data tree (without the yyyy/mm/dd/ subdirectories)
- verbose (boolean):
Optional parameter, if set to True the loading function will print the names of the files it loads for debugging purposes
- dataformat (str):
‘mat’ or ‘h5’. if h5 then the hdf5 data tree is read ifmat then the matlab data tree is read.
OUTPUT
Returns a dictionary with all loaded variables. The dictionary is empty if no matching data could be loaded
Use like this:
mat=readlap('~/icadata','20150211') mat=readlap('~/icadata','20150211',variables=['lap_time','lap_potential']) mat=readlap('~/icadata','20150211','20150213')
- irfpy.ica.io.readcops(datarootpath, from_day=None, to_day='', variables=None, verbose=False, flat=False, dataformat='mat', filters=None, time_instances='cops_time')[source]¶
Loads one day or more worth of cops*.mat data analog to scipy.io.loadmat(‘xyz.mat’) but with assembling all hourly files corresponding to that day.
PARAMETERS
- datarootpath (string):
local path to the ‘root’ of a data directory tree This data tree should contain the file(s) in subdirectories cops/xxxx/yyyy/mm/dd/ as it is done in the pipeline.
- from_day (string or datetime.datetime):
string is of the form ‘20150115T1011’
- to_day (string or datetime.datetime):
optional, string is of the form ‘20150116T1112’. If given data to the end of the given intervall is loaded, otherwise one day.
- variables (dict):
a list of variable names to load. Default is to load all variables. If the loaded variables include ‘time_instances’ then these are automatically converted to python datetime object
- flat (boolean):
If flat is set to True, the load function will if no files were found also attempt to load files which are at the root of the data tree (without the yyyy/mm/dd/ subdirectories)
- verbose (boolean):
Optional parameter, if set to True the loading function will print the names of the files it loads for debugging purposes
- dataformat (str):
‘mat’ or ‘h5’. if h5 then the hdf5 data tree is read ifmat then the matlab data tree is read.
OUTPUT
Returns a dictionary with all loaded variables. The dictionary is empty if no matching data could be loaded
Use like this:
mat=readcops('~/icadata','20150211') mat=readcops('~/icadata','20150211',variables=['cops_time','cops_pressure']) mat=readcops('~/icadata','20150211','20150213')
- irfpy.ica.io.readmag(datarootpath, from_day=None, to_day='', variables=None, verbose=False, flat=False, dataformat='mat', filters=None, time_instances='mag_time')[source]¶
Loads one day or more worth of mag*.mat data analog to scipy.io.loadmat(‘xyz.mat’) but with assembling all hourly files corresponding to that day.
PARAMETERS
- datarootpath (string):
local path to the ‘root’ of a data directory tree This data tree should contain the file(s) in subdirectories mag/xxxx/yyyy/mm/dd/ as it is done in the pipeline.
- from_day (string or datetime.datetime):
string is of the form ‘20150115T1011’
- to_day (string or datetime.datetime):
optional, string is of the form ‘20150116T1112’. If given data to the end of the given intervall is loaded, otherwise one day.
- variables (dict):
a list of variable names to load. Default is to load all variables. If the loaded variables include ‘time_instances’ then these are automatically converted to python datetime object
- flat (boolean):
If flat is set to True, the load function will if no files were found also attempt to load files which are at the root of the data tree (without the yyyy/mm/dd/ subdirectories)
- verbose (boolean):
Optional parameter, if set to True the loading function will print the names of the files it loads for debugging purposes
- dataformat (str):
‘mat’ or ‘h5’. if h5 then the hdf5 data tree is read ifmat then the matlab data tree is read.
OUTPUT
Returns a dictionary with all loaded variables. The dictionary is empty if no matching data could be loaded
Use like this:
mat=readmag('~/icadata','20150211') mat=readmag('~/icadata','20150211',variables=['mag_B_cseq']) mat=readmag('~/icadata','20150211','20150213')
- irfpy.ica.io.readbestc(datarootpath, from_day=None, to_day='', variables=None, verbose=False, flat=False, partialelevationscan=False, dataformat='mat', filters=None, time_instances='time_instances')[source]¶
Loads one day or more worth of bestc*.mat data analog to scipy.io.loadmat(‘xyz.mat’) but with assembling all hourly files corresponding to that day.
PARAMETERS
- datarootpath (string):
local path to the ‘root’ of a data directory tree This data tree should contain the file(s) in subdirectories level2/xxxx/yyyy/mm/dd/ as it is done in the pipeline.
- from_day (string or datetime.datetime):
string is of the form ‘20150115T1011’
- to_day (string or datetime.datetime):
optional, string is of the form ‘20150116T1112’. If given data to the end of the given intervall is loaded, otherwise one day.
- variables (dict):
a list of variable names to load. Default is to load all variables. If the loaded variables include ‘time_instances’ then these are automatically converted to python datetime object
- flat (boolean):
If flat is set to True, the load function will if no files were found also attempt to load files which are at the root of the data tree (without the yyyy/mm/dd/ subdirectories)
- partialelevationscan (boolean):
Optional parameter, default False. If set to False the loading function will only load complete elevation scans. If set to True, the neartest time_instances will be loaded independent of where an elevation scan starts.
- verbose (boolean):
Optional parameter, if set to True the loading function will print the names of the files it loads for debugging purposes
- dataformat (str):
‘mat’ or ‘h5’. if h5 then the hdf5 data tree is read ifmat then the matlab data tree is read.
OUTPUT
Returns a dictionary with all loaded variables. The dictionary is empty if no matching data could be loaded
Use like this:
mat=readbestc('~/icadata','20150211') mat=readbestc('~/icadata','20150211',variables=['time_instances','h_counts']) mat=readbestc('~/icadata','20150211','20150213')
- irfpy.ica.io.readflux(datarootpath, from_day=None, to_day='', variables=None, verbose=False, flat=False, partialelevationscan=False, dataformat='mat', filters=None, time_instances='time_instances')[source]¶
Loads one day or more worth of flux*.mat data analog to scipy.io.loadmat(‘xyz.mat’) but with assembling all hourly files corresponding to that day.
PARAMETERS
- datarootpath (string):
local path to the ‘root’ of a data directory tree This data tree should contain the file(s) in subdirectories level3/xxxx/yyyy/mm/dd/ as it is done in the pipeline.
- from_day (string or datetime.datetime):
string is of the form ‘20150115T1011’
- to_day (string or datetime.datetime):
optional, string is of the form ‘20150116T1112’. If given data to the end of the given intervall is loaded, otherwise one day.
- variables (dict):
a list of variable names to load. Default is to load all variables. If the loaded variables include ‘time_instances’ then these are automatically converted to python datetime object
- flat (boolean):
If flat is set to True, the load function will if no files were found also attempt to load files which are at the root of the data tree (without the yyyy/mm/dd/ subdirectories)
- partialelevationscan (boolean):
Optional parameter, default False. If set to False the loading function will only load complete elevation scans. If set to True, the neartest time_instances will be loaded independent of where an elevation scan starts.
- verbose (boolean):
Optional parameter, if set to True the loading function will print the names of the files it loads for debugging purposes
- dataformat (str):
‘mat’ or ‘h5’. if h5 then the hdf5 data tree is read ifmat then the matlab data tree is read.
OUTPUT
Returns a dictionary with all loaded variables. The dictionary is empty if no matching data could be loaded
Use like this:
mat=readflux('~/icadata','20150211') mat=readflux('~/icadata','20150211',variables=['time_instances','flux']) mat=readflux('~/icadata','20150211','20150213')
- irfpy.ica.io.conditional_gc_collect()[source]¶
Calls gc.collect() if the available memory is less than 30%.
- irfpy.ica.io.loadmat(file, variable_names=None)[source]¶
Loads the dictionary “variable_names” from a v7 or v7.3 mat file. if the dictionary is not specified, then everything is loaded.:
filename = 'test.mat' m = irfpy.ica.io.loadmat(filename) print (m.keys())
- irfpy.ica.io.loadh5(file, variable_names=None)[source]¶
loads the dictionary “variable_names” from a h5 file.
- irfpy.ica.io.savemat(file, data)[source]¶
Save the dictionary data in v7.3 mat format to file analog to scipy.io.savemat:
a = np.range(10) b = np.range(33) c = np.range(22) ** 2 filename = 'test.mat' m = dict() m['a'] = a m['b'] = b m['c'] = c irfpy.ica.io.savemat(filename,m)
- irfpy.ica.io.flatstringlist(list_of_lists)[source]¶
Tries hard to make a list of strings out of the argument.
- irfpy.ica.io.expectedversion(proc=None, aux=None, bestc=None, flux=None, lap=None, cops=None, mag=None, mip=None, verbose=True, **kwargs)[source]¶
Defines the data versions that the user would like to have. If the existing version is different, then at the first occurence there will be a warning printed explaining what the difference is between the required version and the present version. If no version is specified then all versions are allowed.
All parameters are strings of the form “xx.y” or floating numbers. e.g. proc=’2.3’ means that for proc, version 2.3 is requested.
The version found in the datafiles is printed once. If verbose is set to false, this is suppressed.
Usage:
expectedversion(proc='2.3',lap='0.3')
- irfpy.ica.io.loadlevelN(datarootpath, fileprefix='bestc', from_day=None, to_day='', variables=None, verbose=False, flat=False, partialelevationscan=False, dataformat='mat', branch=None, time_instances='time_instances', filters=None, varshapes=None)[source]¶
Loads one day or more worth of level1 data analog to scipy.io.loadmat(‘xyz.mat’) but with assembling all hourly files corresponding to that day.
PARAMETERS
- datarootpath (string):
local path to the ‘root’ of the level1 data tree This data tree should contain the file(s) in subdirectories yyyy/mm/dd/ as it is done in the pipeline.
- fileprefix (string):
Data files start with this prefix. e.g. ‘bestc’
- from_day (string or datetime.datetime):
string is of the form ‘20150115T1011’
- to_day (string or datetime.datetime):
optional, string is of the form ‘20150116T1112’. If given data to the end of the given intervall is loaded, otherwise one day.
- variables (dict):
a list of variable names to load. Default is to load all variables. If the loaded variables include ‘time_instances’ then these are automatically converted to python datetime object
- flat (boolean):
If flat is set to True, the load function will if no files were found also attempt to load files which are at the root of the data tree (without the yyyy/mm/dd/ subdirectories)
- partialelevationscan (boolean):
Optional parameter, default False. If set to False the loading function will only load complete elevation scans. If set to True, the neartest time_instances will be loaded independent of where an elevation scan starts.
- verbose (boolean):
Optional parameter, if set to True the loading function will print the names of the files it loads for debugging purposes.
- dataformat (string):
One of: ‘mat’(default) or ‘h5’.
- time_instances (str):
Name of the time_instances variable (default: ‘time_instances’)
- branch (str):
Directory below datarootpath the data is stored, e.g. ‘level1’
- filters (dict):
Filter list
- varshapes (dict):
data structure describing the dimensionality and the special data types of the variables to be read. Use predefined defaults, e.g. default_varshapes1 for level1 variable shapes.
OUTPUT
Returns a dictionary with all loaded variables. The dictionary is empty if no matching data could be loaded
Use like this:
matfile=irfpy.ica.io.loadlevel?(icapath,'bestc','20150211') matfile=irfpy.ica.io.loadlevel?(icapath,'bestc','20150211', variables=['time_instances','E']) matfile=irfpy.ica.io.loadlevel?(icapath,'bestc','20150211','20150213', variables=['time_instances','E'])
- irfpy.ica.io.loadlevel2(datarootpath, mattype='bestc', from_day=None, to_day='', variables=None, verbose=False, flat=False, partialelevationscan=False, dataformat='mat', filters=None)[source]¶
Loads one day or more worth of level1 data analog to scipy.io.loadmat(‘xyz.mat’) but with assembling all hourly files corresponding to that day.
PARAMETERS
- datarootpath (string):
local path to the ‘root’ of the level1 data tree This data tree should contain the file(s) in subdirectories yyyy/mm/dd/ as it is done in the pipeline.
- mattype (string):
one of ‘bestc’
- from_day (string or datetime.datetime):
string is of the form ‘20150115T1011’
- to_day (string or datetime.datetime):
optional, string is of the form ‘20150116T1112’. If given data to the end of the given intervall is loaded, otherwise one day.
- variables (dict):
a list of variable names to load. Default is to load all variables. If the loaded variables include ‘time_instances’ then these are automatically converted to python datetime object
- flat (boolean):
If flat is set to True, the load function will if no files were found also attempt to load files which are at the root of the data tree (without the yyyy/mm/dd/ subdirectories)
- partialelevationscan (boolean):
Optional parameter, default False. If set to False the loading function will only load complete elevation scans. If set to True, the neartest time_instances will be loaded independent of where an elevation scan starts.
- verbose (boolean):
Optional parameter, if set to True the loading function will print the names of the files it loads for debugging purposes.
- dataformat (string):
One of: ‘mat’(default) or ‘h5’.
OUTPUT
Returns a dictionary with all loaded variables. The dictionary is empty if no matching data could be loaded
Use like this:
matfile=irfpy.ica.io.loadlevel2(icapath,'bestc','20150211') matfile=irfpy.ica.io.loadlevel2(icapath,'bestc','20150211', variables=['time_instances','E']) matfile=irfpy.ica.io.loadlevel2(icapath,'bestc','20150211','20150213', variables=['time_instances','E'])
- irfpy.ica.io.loadlevel1(datarootpath, mattype='proc', from_day=None, to_day='', variables=None, verbose=False, flat=False, partialelevationscan=False, dataformat='mat', filters=None)[source]¶
Loads one day or more worth of level1 data analog to scipy.io.loadmat(‘xyz.mat’) but with assembling all hourly files corresponding to that day.
Consider using one of the readproc(), readaux(), read…() functions instead.
PARAMETERS
- datarootpath (string):
local path to the ‘root’ of the level1 data tree This data tree should contain the file(s) in subdirectories yyyy/mm/dd/ as it is done in the pipeline.
- mattype (string):
one of ‘proc’, ‘special’ or ‘aux’
- from_day (string or datetime.datetime):
string is of the form ‘20150115T1011’
- to_day (string or datetime.datetime):
optional, string is of the form ‘20150116T1112’. If given data to the end of the given intervall is loaded, otherwise one day.
- variables (dict):
a list of variable names to load. Default is to load all variables. If the loaded variables include ‘time_instances’ then these are automatically converted to python datetime object
- flat (boolean):
If flat is set to True, the load function will if no files were found also attempt to load files which are at the root of the data tree (without the yyyy/mm/dd/ subdirectories)
- partialelevationscan (boolean):
Optional parameter, default False. If set to False the loading function will only load complete elevation scans. If set to True, the neartest time_instances will be loaded independent of where an elevation scan starts.
- verbose (boolean):
Optional parameter, if set to True the loading function will print the names of the files it loads for debugging purposes
- dataformat (string):
One of: ‘mat’(default) or ‘h5’.
OUTPUT
Returns a dictionary with all loaded variables. The dictionary is empty if no matching data could be loaded
- Use like this::
icapath = ‘/home/user/icadata’ matfile=irfpy.ica.io.loadlevel1(icapath,’proc’,’20150211’) matfile=irfpy.ica.io.loadlevel1(icapath,’proc’,’20150211’,
variables=[‘time_instances’,’E’])
- matfile=irfpy.ica.io.loadlevel1(icapath,’proc’,
‘20150211’,’20150213’, variables=[‘time_instances’,’E’])
- matfile=irfpy.ica.io.loadlevel1(icapath, ‘aux’, theinterval,
variables=[‘sp_cso’,’time_instances’])
- irfpy.ica.io.loadlevel1oktime(datarootpath, mattype, from_day, to_day='', variables=None, verbose=False, flat=False, partialelevationscan=False, dataformat='mat', filters=None)[source]¶
Loads one day or more worth of level1 data analog to irfpy.ica.io.loadmat(‘xyz.mat’) but with assembling all hourly files corresponding to that day. Only data with timeswhere time_error_flag is == 0 is loaded
Consider using one of the readproc(), readaux(), read…() functions instead.
PARAMETERS
- datarootpath (string):
local path to the ‘root’ of the level1 data tree This data tree should contain the file(s) in subdirectories yyyy/mm/dd/ as it is done in the pipeline.
- mattype (string):
filetype to be loaded, one of ‘proc’ or ‘aux’
- from_day (string or datetime.datetime):
string is of the form ‘20150115T1011’
- to_day (string or datetime.datetime):
optional, string is of the form ‘20150116T1112’. If given data to the end of the given intervall is loaded, otherwise one day.
- variables (dict):
a list of variable names to load. Default is to load all variables. If the loaded variables include ‘time_instances’ then these are automatically converted to python datetime object
- flat (boolean):
If flat is set to True, the load function will if no files were found also attempt to load files which are at the root of the data tree (without the yyyy/mm/dd/ subdirectories)
- partialelevationscan (boolean):
Optional parameter, default False. If set to False the loading function will only load complete elevation scans. If set to True, the neartest time_instances will be loaded independent of where an elevation scan starts.
- verbose (boolean):
Prints what the function does
- dataformat (string):
One of: ‘mat’(default) or ‘h5’.
- This function is identical to the call of loadlevel1 with a filters parameter:
loadlevel1(datarootpath, …. , filters={‘time_error_flag’:0})
OUTPUT
Returns a dictionary with all loaded variables
- Limitation: If this function is used to load an aux file, the aux file
must be version 1.9 or larger. This function can not be used to load special files.
- irfpy.ica.io.loadlevel1okall(datarootpath, mattype='proc', from_day=None, to_day='', variables=None, verbose=False, flat=False, partialelevationscan=False, dataformat='mat', filters=None)[source]¶
Loads one day or more worth of level1 data analog to irfpy.ica.io.loadmat(‘xyz.mat’) but with assembling all hourly files corresponding to that day. Only data with times where error_flags is == 0 is loaded
Consider using one of the readproc(), readaux(), read…() functions instead.
PARAMETERS
- datarootpath (string):
local path to the ‘root’ of the level1 data tree This data tree should contain the file(s) in subdirectories yyyy/mm/dd/ as it is done in the pipeline.
- mattype (string):
filetype to be loaded, one of ‘proc’ or ‘aux’
- from_day (string or datetime.datetime):
string is of the form ‘20150115T1011’
- to_day (string or datetime.datetime):
optional, string is of the form ‘20150116T1112’. If given data to the end of the given intervall is loaded, otherwise one day.
- variables (dict):
a list of variable names to load. Default is to load all variables. If the loaded variables include ‘time_instances’ then these are automatically converted to python datetime object
- flat (boolean):
If flat is set to True, the load function will if no files were found also attempt to load files which are at the root of the data tree (without the yyyy/mm/dd/ subdirectories)
- partialelevationscan (boolean):
Optional parameter, default False. If set to False the loading function will only load complete elevation scans. If set to True, the neartest time_instances will be loaded independent of where an elevation scan starts.
- verbose (boolean):
Prints what the function does
- dataformat (string):
One of: ‘mat’(default) or ‘h5’.
- This function is identical to the call of loadlevel1 with a filters parameter:
loadlevel1(datarootpath, …. , filters={‘error_flags’:0})
OUTPUT
Returns a dictionary with all loaded variables
- irfpy.ica.io.loadlevel1filter(datarootpath, mattype='proc', from_day=None, to_day='', variables=None, verbose=False, flat=False, partialelevationscan=False, dataformat='mat', time_error_flag=None, decoder_error_flag=None, fmt_error_flag=None, edf_error_flag=None, error_flags=None, mode=None, sw_version=None, cur_pacc=None)[source]¶
Loads one day or more worth of level1 data analog to irfpy.ica.io.loadmat(‘xyz.mat’) but with assembling all hourly files corresponding to that day. Only data with times where the filter criteria apply are loaded
PARAMETERS
- datarootpath (string):
local path to the ‘root’ of the level1 data tree This data tree should contain the file(s) in subdirectories yyyy/mm/dd/ as it is done in the pipeline.
- mattype (string):
filetype to be loaded, one of ‘proc’ or ‘aux’
- from_day (string or datetime.datetime):
string is of the form ‘20150115T1011’
- to_day (string or datetime.datetime):
optional, string is of the form ‘20150116T1112’. If given data to the end of the given intervall is loaded, otherwise one day.
- variables (dict):
a list of variable names to load. Default is to load all variables. If the loaded variables include ‘time_instances’ then these are automatically converted to python datetime object
- filters select the data loaded:
time_error_flag (int) : 0 decoder_error_flag (int) : 0 fmt_error_flag (int) : 0 edf_error_flag (int) : 0 error_flags (int) : 0 mode (int) : 0 sw_version (int) : 6 cur_pacc (int) : 6
- flat (boolean):
If flat is set to True, the load function will if no files were found also attempt to load files which are at the root of the data tree (without the yyyy/mm/dd/ subdirectories)
- partialelevationscan (boolean):
Optional parameter, default False. If set to False the loading function will only load complete elevation scans. If set to True, the neartest time_instances will be loaded independent of where an elevation scan starts.
- verbose (boolean):
Prints what the function does
- dataformat (string):
One of: ‘mat’(default) or ‘h5’.
OUTPUT
Returns a dictionary with all loaded variables.
- irfpy.ica.io.loadlevel0(datarootpath, from_day, to_day='', variables=None, verbose=False, flat=False, dataformat='mat', **kwargs)[source]¶
Loads one hour or more worth of level0 data analog to scipy.io.loadmat(‘xyz.mat’) but with assembling all hourly files corresponding to the time interval specified.
Consider using the readraw() function instead.
AUTHOR: Gabriella Stenberg Wieser
PARAMETERS
- datarootpath (string):
local path to the ‘root’ of the level0 data tree This data tree should contain the file(s) in subdirectories yyyy/mm/dd/ as it is done in the pipeline.
- from_day (string or datetime.datetime):
string is of the form ‘20150115T1011’
- to_day (string or datetime.datetime):
optional, string is of the form ‘20150116T1112’. If given data to the end of the given intervall is loaded, otherwise one day.
- variables (list of strings):
a list of variable names to load. Default is to load all variables. If the loaded variables include ‘time_instances’ then these are automatically converted to python datetime object
- flat (boolean):
If flat is set to True, the load function will if no files were found also attempt to load files which are at the root of the data tree (without the yyyy/mm/dd/ subdirectories)
- dataformat (string):
One of: ‘mat’(default) or ‘h5’.
OUTPUT
returns a dictionary with all loaded variables
EXAMPLES:
matfile=ica.io.loadlevel0(icapath,'20150211') matfile=ica.io.loadlevel0(icapath,'20150211', variables=['iontime','ionspectra']) matfile=ica.io.loadlevel0(icapath,'20150211','20150213', variables=['time_interval','E'])