irfpy.mima.imaextra

IMA extra data file access

See IMA Extra dataset to start using the IMA extra data.

This module provides the access to the MEX/IMA extra dataset. The database location should be specified by [mima] section imaddncbase entry of .irfpyrc.

>>> mimaddncpath = irfpy.util.irfpyrc.Rc().get('mima', 'imaddncbase')
>>> mimaextrapath = os.path.join(mimaddncpath, 'IMAEXTRA')
>>> mimaparampath = os.path.join(mimaddncpath, 'IMAPARAM')

The simplest way of getting the file is to specify the file name to getImaExtraFile().

>>> fe = getImaExtraFile('imaextra20040240408.nc.gz')
>>> fp = getImaParamFile('imaparam20070121932.nc.gz')

You may also use the time directly.

>>> fe = getImaExtraFile(t=datetime.datetime(2004, 1, 25, 5, 0))
>>> fp = getImaParamFile(t=datetime.datetime(2007, 1, 13, 20, 0))

More on module

Each IMAEXTRA file corresponds to ImaExtraFile. This can be instanced manually, but users are recommended to use ImaExtraFileFactory, which provides accessor and cache to ImaExtraFile object.

>>> fact = ImaExtraFileFactory.createFactory()
>>> iex = fact.getImaExtraFile(os.path.join(mimaextrapath, 'imaextra20040240408.nc.gz'))
>>> print(len(iex.var['Time']))
34

>>> fact2 = ImaParamFileFactory.createFactory()
>>> ipr = fact2.getImaParamFile(os.path.join(mimaparampath, 'imaparam20070121932.nc.gz'))
>>> print(len(ipr.var['Time']))
38

The dataset include many files (thus, ImaExtraFile objects). The ImaExtraDatabase provides a simple database of the dataset. This class has functionality converting the time to filename (get_filename()).

>>> db = ImaExtraDatabase.createDatabase()
>>> fn = db.get_filename(datetime.datetime(2004, 1, 25, 5, 0))
>>> print(os.path.basename(fn))    # The filename
imaextra20040240408.nc.gz

>>> db2 = ImaParamDatabase.createDatabase()
>>> fn = db2.get_filename(datetime.datetime(2007, 1, 13, 20, 0))
>>> print(os.path.basename(fn))
imaparam20070121932.nc.gz

class irfpy.mima.imaextra.DataCenterImaParam

Bases: irfpy.util.datacenter.BaseDataCenter

ImaParam data’s data center

The ImaParam object is produced from this data center.

>>> from irfpy.mima import imaextra
>>> dc = imaextra.DataCenterImaParam()
>>> import datetime
>>> t0 = datetime.datetime(2011, 6, 18, 6, 30)
>>> tobs, iparam = dc.nearest(t0)
>>> print(tobs)
2011-06-18 06:30:00.236000
>>> print(iparam.ndat)
1
>>> print(iparam.data['Density_P'])
[1.1828649]

Initializer.

Parameters

• cache_size – Size of the ring cache.

• name – The name of the

• copy – Boolean if the returned data is to be deep-copied (True) or reference (False). It is good to return the data after the copy, since then the data is always original. Returning reference is possibly faster, while there are side effect that the post-processing will destroy the original data. Therefore, it is recommended to set True always. The copy value can be overwritten by each method as necessity.

search_files()

Search the data files, returning a list of data file.

This method searches the data files under the base_folder. This method should return a list / tuple of the data file name (usually a full path).

This method is called only once when __init__() was called.

Returns

A list / tuple of the data file. It should be full path (or relative path from the current path), and sorted from earlier data to later data.

approximate_starttime(filename)

Start time should be guessed for each file.

A guessed start time should be returned. It is OK if it is very approximate, but the orders of the guessed-start and the exact-start should be identical. This method must be very fast, because it is called for all the files in the data base (i.e. all the files retuned by search_files() method).

A practical suggestion for implementation is to guess the time from the filename.

Parameters

filename – A string, filename.

Returns

An approximate, guessed start time of the file

Return type

datetime.datetime

read_file(filename)

The file is read, and return the contents as a tuple with size 2, (tlist, dlist).

This method is an abstract method, meaning that the developer of the data center should implement it. See SampleDataCenter for more details.

The implementation of this method should follow:

• Returned value is a tuple with a size of 2. - The first element is a tuple/list specifying the time (with each element as datetime.datetime object) - The second element is a tuple/list specifying the data, with any format. - The length of both two elements should be the same.

If the given filename is corrupted or empty, a two empty tuple would be returned (i.e., return (), ()). In this case, return None for the exact_starttime() method.

Parameters

filename – File name

Returns

The contents of the data file

Return type

tuple

irfpy.mima.imaextra.isdb()

class irfpy.mima.imaextra.ImaExtra

Bases: irfpy.imacommon.imaextra2.ImaExtraCommon

MEX/IMA extra data for a specific time range.

This class is for the IMA extra data for a specific time range. It is generally produced by getImaExtra() or by iter_imaextra() iterator. See respective functions how to get the data.

Once produced, the following methods will give you mass-separated count spectra.

• getHpSpec() for proton spectra

• getHeavySpec() for heavy spectra

• getRestRm() for rest matrix

• getHGSpec() for proton ghost

• getobstime() for observation time

data_keys = ['AngTableN', 'ETableN', 'Noise', 'HGhostSpec', 'FracO2', 'HeavySpec', 'HpSpec', 'Pacc', 'RestRm', 'Time']

getHGspec()

Return the proton ghost spectra. Counts. Order is AEPT (irfpy standard)

Returns

Proton spectra, in the shape of (A16, E96, P16, Time). Counts.

class irfpy.mima.imaextra.ImaExtraFile(filename, gunzip=True)

Bases: irfpy.imacommon.imaextra2.ImaExtraFileCommon

An IMA extra file for MEX.

IMA extra file class. Usually, you want to instance via ImaDdNcFileFactory then caching will be effective.

Member of dim contains the dimension information and var for variables.

Open the file and read the data.

classmethod get_sample_filename()

… fn = ImaExtraFile.get_sample_filename() … im = ImaExtraFile(fn) … print len(im.dim[‘TimeLength’]) 17

… print len(im.dim[‘Time’]) 17

… t0 = im.var[‘Time’][0] # t0 will be masked array. … print t0.compressed().data 2009136192402367

Getting time.

Getting time of the observations can be done via ImaExtraFile.getobstime() method.

… t = im.getobstime() … print len(t) 17

… print t[0] 2009-05-17 19:24:02.367000

getImaExtra()

Obtain the ImaExtra object in the data file

Returns

Data

Return type

ImaExtra

class irfpy.mima.imaextra.ImaParam

Bases: irfpy.imacommon.imaextra2.ImaParamCommon

The object stores data of IMA parameter for MEX.

… t0 = datetime.datetime(2010, 7, 11, 16, 12) … file0 = getImaParamFile(t=t0) … ip0 = file0.getImaParam() … print ip0.ndat 32

… t1 = datetime.datetime(2010, 7, 13, 10, 25) … file1 = getImaParamFile(t=t1) … ip1 = file1.getImaParam() … print ip1.ndat 38

… ip01 = ip0 + ip1 … print ip01.data[‘Velocity_P’].shape (70, 3)

… ip10 = ip1 + ip0 … print ip01.data[‘Velocity_O’].shape (70, 3)

… print (ip01.data[‘Temperature_P’] == ip10.data[‘Temperature_P’]).all() True

… print ip01.obstime == ip10.obstime True

If you specify the same dataset for adding, duplication check works.

… ip11 = ip1 + ip1 # If you add two identical dataset … print ip11.data[‘ScanFlag’].shape # the result is the same as before. (38,) … print ip1.obstime == ip11.obstime True

… t0 = datetime.datetime(2010, 7, 11, 16, 12) … file0 = getImaParamFile(t=t0) … ip0 = file0.getImaParam() … print ip0.ndat 32

… t1 = datetime.datetime(2010, 7, 11, 17) … t2 = datetime.datetime(2010, 7, 11, 17, 15) … ip0.trim(t1, t2) … print ip0.ndat 5

data_keys = ['ScanFlag', 'Quality_P', 'Density_P', 'Velocity_P', 'Temperature_P', 'EnergyThermal_P', 'Quality_O', 'Density_O', 'Velocity_O', 'Temperature_O', 'EnergyThermal_O', 'Quality_O2', 'Density_O2', 'Velocity_O2', 'Temperature_O2', 'EnergyThermal_O2']

class irfpy.mima.imaextra.ImaParamFile(filename, gunzip=True)

Bases: irfpy.imacommon.imaextra2.ImaParamFileCommon

An IMA param file for VEX.

IMA param file class. Usually, you want to instance via ImaDdNcFileFactory then caching will be effective.

Member of dim contains the dimension information and var for variables.

Open the file and read the data.

getImaParam()

Obtain the ImaParam object

class irfpy.mima.imaextra.ImaExtraFileFactory(*args, **kwds)

Bases: irfpy.imacommon.imaextra2.ImaDdNcFileFactory

classmethod createFactory()

Return the “master” factory.

getImaExtraFile(filename, gunzip=True)

class irfpy.mima.imaextra.ImaParamFileFactory(*args, **kwds)

Bases: irfpy.imacommon.imaextra2.ImaDdNcFileFactory

classmethod createFactory()

Return the “master” factory.

getImaParamFile(filename, gunzip=True)

class irfpy.mima.imaextra.ImaExtraDatabase(database=None)

Bases: irfpy.imacommon.imaextra2.ImaDdNcDatabase

… db = ImaExtraDatabase.createDatabase() … t = datetime.datetime(2004, 1, 25, 5, 0) … print os.path.basename(db.get_filename(t)) imaextra20040240408.nc.gz

Create the database.

Use createDatabase() class method for creating database.

Parameters

• database – Database directory path. If None, the [vima]-imaextrabase entry from irfpyrc is read.

• prefix_name – The filename should be imaXXXXX20103091457.nc.gz. The prefix, imaXXXXX should be specified to identify the start of the time expression.

• ddncfactory – Factory object that create the object of the file contents. It should be an object of the class that extends ImaDdNcFileFactory. For example, ImaExtraFileFactory can be given.

• verbose – The keyword is outdated. Do nothing. Use logging to display a log with the name of irfpy.imacommon.imaextra2.ImaDdNcDatabase.

classmethod createDatabase(refresh=False)

Create the database.

class irfpy.mima.imaextra.ImaParamDatabase(database=None)

Bases: irfpy.imacommon.imaextra2.ImaDdNcDatabase

Create the database.

Use createDatabase() class method for creating database.

Parameters

• database – Database directory path. If None, the [vima]-imaextrabase entry from irfpyrc is read.

• prefix_name – The filename should be imaXXXXX20103091457.nc.gz. The prefix, imaXXXXX should be specified to identify the start of the time expression.

• ddncfactory – Factory object that create the object of the file contents. It should be an object of the class that extends ImaDdNcFileFactory. For example, ImaExtraFileFactory can be given.

• verbose – The keyword is outdated. Do nothing. Use logging to display a log with the name of irfpy.imacommon.imaextra2.ImaDdNcDatabase.

classmethod createDatabase(refresh=False)

Create the database.

irfpy.mima.imaextra.getImaExtraFile(filename=None, t=None, gunzip=True)

An easy accessor to ImaExtraFile getter.

With a support of cache (ImaExtraFileFactory), an optimized ImaExtraFile getter.

irfpy.mima.imaextra.getImaParamFile(filename=None, t=None, gunzip=True)

An easy accessor to ImaParamFile getter.

With a support of cache (ImaParamFileFactory), an optimized ImaParamFile getter.

irfpy.mima.imaextra.getImaParam(t0, t1)

Return the ImaParam object.

Parameters

• t0 – Time to start

• t1 – Time to end

irfpy.mima.imaextra.getImaExtra(t0, t1)

Get the IMA extra data in a form of ImaExtra object

Parameters

• t0 – Time to start

• t1 – Time to end

Returns

IMA extra object

Return type

ImaExtra

>>> imaExtra = getImaExtra(datetime.datetime(2012, 10, 5, 6), datetime.datetime(2012, 10, 6, 8))
>>> t = imaExtra.obstime
>>> print(len(t))
167
>>> hp = imaExtra.getHpSpec()
>>> print(hp.shape)
(16, 96, 16, 167)
>>> print(hp.max())
494.02887
>>> op = imaExtra.getHeavySpec()
>>> print(op.shape)
(16, 96, 16, 167)
>>> print(op.max())
5126.1104

class irfpy.mima.imaextra.iterparam(t0, t1)

Bases: object

Iterate the IMA parameter file.

Iterated is a list of (t, dat) were t is the time of the observation start, and dat is a dictionary that contains the data.

The key of dat is ImaParam.data_keys. The dat is not a ImaParam object, but rather, ImaParam.data data.

Parameters

• t0 – Time to start. Iterated data’s start time is always later than or equal to t0.

• t1 – Time to stop. Iterated data’s stop time is always earlier than or equal to t1.

Returns

A list, (t, dat). t is time, and dat is a dictionary containing the IMAPARAM data. Its key is ImaParam.data_keys.

Return type

list of (datetime.datetime, dict)

If you want to iterate data from 2011-11-05T05:15 for 10 minutes, you can do as follows:

>>> t0 = datetime.datetime(2010, 11, 7, 5, 15)
>>> t1 = datetime.datetime(2010, 11, 7, 5, 25)

>>> for t, mom in iterparam(t0, t1):
...     print(t, mom['Density_P'])
2010-11-07 05:17:53.821000 0.61079055
2010-11-07 05:21:05.789000 0.39960635
2010-11-07 05:24:17.851000 0.21043392

data_keys = ['ScanFlag', 'Quality_P', 'Density_P', 'Velocity_P', 'Temperature_P', 'EnergyThermal_P', 'Quality_O', 'Density_O', 'Velocity_O', 'Temperature_O', 'EnergyThermal_O', 'Quality_O2', 'Density_O2', 'Velocity_O2', 'Temperature_O2', 'EnergyThermal_O2']

Key of the data

class irfpy.mima.imaextra.iter_imaextra(t0, t1)

Bases: object

Iterate the IMA extra data.

Iterate the IMA extra data, as a tuple of (time, data). Here data is an object of ImaExtra.

>>> t0 = datetime.datetime(2010, 11, 7, 5, 15)
>>> t1 = datetime.datetime(2010, 11, 7, 5, 25)

>>> for t, d in iter_imaextra(t0, t1):
...     print(t, d.getHeavySpec().shape, d.getHpSpec().max())
2010-11-07 05:17:53.821000 (16, 96, 16, 1) 42.640816
2010-11-07 05:21:05.789000 (16, 96, 16, 1) 63.204727
2010-11-07 05:24:17.852000 (16, 96, 16, 1) 57.680725

data_keys = ['AngTableN', 'ETableN', 'Noise', 'HGhostSpec', 'FracO2', 'HeavySpec', 'HpSpec', 'Pacc', 'RestRm', 'Time']