irfpy.vima.imaextra

IMA extra data file access.

See also a tutorial IMA Extra dataset. A demo script to plot E-t diagram in vima_demo/vima_dd_simpple.py.

IMA extra is a calibrated database that Andrei Fedorov at IRAP prepared. This module provides the access to the VEX/IMA extra dataset. The database location should be specified by .irfpyrc configuration system.

[vima]
imaddncbase = /path/to/the/imaextra/database

IMA extra consists of four kinds of dataset.

  • IMAPARAM (supported by irfpy)

  • IMAEXTRA (supported by irfpy)

  • IMAANG (not supported by irfpy yet)

  • IMAEXTRALIGHT (not supported by irfpy yet)

The supported datasets are explained in the following.

IMAPARAM dataset

Parameter (moment) values that IRAP produced. It is not the official version.

Warning

There is no responsibility at IRF and irfpy develop team, as well as ASPERA-4 PI group, for the accuracy of the daset. Dataset should be used by the user’s responsibility.

It is strongly recommended to contact to the ASPERA-4 PI team for the use of the dataset.

There are eleven data in the IMA parameter database.

  • ScanFlag: ?

  • Quality_P: ?

  • Density_P: Density H+

  • Velocity_P: Velocity H+

  • Temperature_P: Temperature H+

  • EnergyThermal_P: Thermal Energy H+

  • Quality_O: ?

  • Density_O: Density O+

  • Velocity_O: Velocity O+

  • Temperature_O: Temperature O+

  • EnergyThermal_O: Thermal Energy O+

The simplest way of getting data is using getImaParam() function.

>>> import datetime
>>> t0 = datetime.datetime(2008, 1, 27, 3, 0, 0)
>>> t1 = datetime.datetime(2008, 1, 27, 3, 10, 0)
>>> ima_parameters = getImaParam(t0, t1)

obstime is a list of the observation time.

>>> print(ima_parameters.obstime)
[datetime.datetime(2008, 1, 27, 3, 0, 18, 779000), datetime.datetime(2008, 1, 27, 3, 3, 30, 875000), datetime.datetime(2008, 1, 27, 3, 6, 42, 875000), datetime.datetime(2008, 1, 27, 3, 9, 54, 812000)]

data is a dictionary of the dataset.

>>> print(ima_parameters.data['Density_P'])     
[0.00756063 0.00653219 0.00567522 0.01819916]

IMAEXTRA dataset

IMA extra dataset is a separated counts to Proton and Oxygen. Example of data access follows.

>>> from irfpy.vima import imaextra
>>> import datetime
>>> t0 = datetime.datetime(2011, 11, 5, 6)
>>> t1 = datetime.datetime(2011, 11, 5, 7)
>>> extdata = imaextra.getImaExtra(t0, t1)

You can know the observation start time as

>>> obstime = extdata.getobstime()    # Returns an array
>>> print(obstime[0])
2011-11-05 06:01:48.605000

You can get the proton count spectrum as

>>> proton = extdata.getHpSpec()
>>> print(proton.shape)     # The shape should be (A16, E96, P16, Time)
(16, 96, 16, 19)
>>> print(proton.sum())     # Total counts that owes to Proton is
11609479.0

Also, the oxygen (heavy) count spectrum as

>>> heavy = extdata.getHeavySpec()
>>> print(heavy.shape)     # The shape should be (A16, E96, P16, Time)
(16, 96, 16, 19)
>>> print(heavy.max())     # Maximum counts in the bin that owes to Oxygen (heavy) is
3348.076

The count that could not be assigned either of proton or oxygen (rest matrix) is

>>> rest = extdata.getRestRm()
>>> print(rest.shape)    # Rest matrix is ordered as (M32, A16, E96, P16, Time)
(32, 16, 96, 16, 19)
>>> print(rest.sum())    # Total counts of rest matrix
799203.06

Direct access to the IMA extra data

You can access the data directly via the attribute data; while this is not recommended. Rather, use the access functions as getHpSpec() or getHeavySpec().

Nevertheless, the data is a dictironary, with key and entries as follows

  • ‘AngTableN’: Angular table index. (N,)

  • ‘ETableN’: Energy table index. 0 (ver-1) or 1 (ver-3). (N,)

  • ‘FracO2’: O2+ fraction (TBC). (N, P16, E96)

  • ‘HeavySpec’: Heavy ion spectrum. (N, P16, E96, A16)

  • ‘HpSpec’: Proton ion spectrum. (N, P16, E96, A16)

  • ‘Pacc’: Post acceleration (N,)

  • ‘RestRm’: The rest spectrum. (N, P16, E96, A16, M32)

  • ‘Time’: Time. (N,)

Here N is the number of data.

The order in the raw matrix is the “inverted way” as MAEPT order (MAEPT is the standard order of array for irfpy).

Warning

Again, be careful using the “RAW” data, since the order of the axis of the array is different from the irfpy-standard.

Therefore, again and again, it is recommended to use the accessing methods, like ImaExtra.getHpSpec() or similar, since these have the standard MAEPT order.

The following text is for developers

>>> vimaddncpath = irfpy.util.irfpyrc.Rc().get('vima', 'imaddncbase')
>>> vimaextrapath = os.path.join(vimaddncpath, 'IMAEXTRA')
>>> vimaparampath = os.path.join(vimaddncpath, 'IMAPARAM')

getImaExtraFile() and getImaParamFile() will open the data file.

>>> fe = getImaExtraFile('imaextra20123640214.nc.gz')
>>> fp = getImaParamFile('imaparam20100630434.nc.gz')

You may also use the time directly.

>>> fe = getImaExtraFile(t=datetime.datetime(2007, 9, 10, 3, 0))
>>> fp = getImaParamFile(t=datetime.datetime(2008, 1, 27, 3, 0))

Each IMAEXTRA file corresponds to ImaExtraFile. This can be instanced manually, but use ImaExtraFileFactory, which provides accessor and cache to ImaExtraFile object.

>>> fact = ImaExtraFileFactory.createFactory()
>>> iex = fact.getImaExtraFile(os.path.join(vimaextrapath, 'imaextra20123640214.nc.gz'))
>>> print(len(iex.var['Time']))
39
>>> fact2 = ImaParamFileFactory.createFactory()
>>> ipr = fact2.getImaParamFile(os.path.join(vimaparampath, 'imaparam20100630434.nc.gz'))
>>> print(len(ipr.var['Time']))
38

The dataset include many files (thus, ImaExtraFile objects). The ImaExtraDatabase provides a simple database of the dataset. This class has functionality converting the time to filename (get_filename()).

>>> db = ImaExtraDatabase.createDatabase()
>>> fn = db.get_filename(datetime.datetime(2007, 9, 10, 3, 0))
>>> print(os.path.basename(fn))    # The filename
imaextra20072520101.nc.gz
>>> db2 = ImaParamDatabase.createDatabase()
>>> fn = db2.get_filename(datetime.datetime(2008, 1, 27, 3, 0))
>>> print(os.path.basename(fn))
imaparam20080260247.nc.gz
irfpy.vima.imaextra.isdb()[source]
class irfpy.vima.imaextra.ImaExtraFile(filename, gunzip=True)[source]

Bases: irfpy.imacommon.imaextra2.ImaExtraFileCommon

An IMA extra file for VEX.

IMA extra file class. Usually, you want to instance via ImaDdNcFileFactory then caching will be effective.

Member of dim contains the dimension information and var for variables.

Open the file and read the data.

getImaExtra()[source]

Obtain the ImaExtra object in the data file

Returns

Data

Return type

ImaExtra

class irfpy.vima.imaextra.ImaParam[source]

Bases: irfpy.imacommon.imaextra2.ImaParamCommon

The object stores data of IMA parameter

>>> t0 = datetime.datetime(2010, 7, 10, 4, 30)
>>> file0 = getImaParamFile(t=t0)
>>> ip0 = file0.getImaParam()
>>> print(ip0.ndat)
32

>>> t1 = datetime.datetime(2010, 7, 13, 8, 40)
>>> file1 = getImaParamFile(t=t1)
>>> ip1 = file1.getImaParam()
>>> print(ip1.ndat)
37

>>> ip01 = ip0 + ip1
>>> print(ip01.data['Velocity_P'].shape)
(69, 3)

>>> ip10 = ip1 + ip0
>>> print(ip01.data['Velocity_O'].shape)
(69, 3)

>>> print((ip01.data['Temperature_P'] == ip10.data['Temperature_P']).all())
True

>>> print(ip01.obstime == ip10.obstime)
True

>>> # specifying the same dataset for adding, duplication check works.
>>> ip11 = ip1 + ip1   # If you add two identical dataset
>>> print(ip11.data['ScanFlag'].shape)  # the result is the same as before.
(37,)

>>> print(ip1.obstime == ip11.obstime)
True
data_keys = ['ScanFlag', 'Quality_P', 'Density_P', 'Velocity_P', 'Temperature_P', 'EnergyThermal_P', 'Quality_O', 'Density_O', 'Velocity_O', 'Temperature_O', 'EnergyThermal_O']
class irfpy.vima.imaextra.ImaExtra[source]

Bases: irfpy.imacommon.imaextra2.ImaExtraCommon

MEX/IMA extra data for a specific time range.

This class is for the IMA extra data for a specific time range. It is generally produced by getImaExtra() or by iter_imaextra() iterator. See respective functions how to get the data.

  • getHpSpec() for proton spectra

  • getHeavySpec() for heavy spectra

  • getRestRm() for rest matrix

  • getobstime() for observation time

data_keys = ['AngTableN', 'ETableN', 'FracO2', 'HeavySpec', 'HpSpec', 'Pacc', 'RestRm', 'Time']
class irfpy.vima.imaextra.ImaParamFile(filename, gunzip=True)[source]

Bases: irfpy.imacommon.imaextra2.ImaParamFileCommon

An IMA param file for VEX.

IMA param file class. Usually, you want to instance via ImaDdNcFileFactory then caching will be effective.

Member of dim contains the dimension information and var for variables.

Open the file and read the data.

getImaParam()[source]

Obtain the ImaParam object

class irfpy.vima.imaextra.ImaExtraFileFactory(*args, **kwds)[source]

Bases: irfpy.imacommon.imaextra2.ImaDdNcFileFactory

classmethod createFactory()[source]

Return the “master” factory.

getImaExtraFile(filename, gunzip=True)[source]
class irfpy.vima.imaextra.ImaParamFileFactory(*args, **kwds)[source]

Bases: irfpy.imacommon.imaextra2.ImaDdNcFileFactory

classmethod createFactory()[source]

Return the “master” factory.

getImaParamFile(filename, gunzip=True)[source]
class irfpy.vima.imaextra.ImaExtraDatabase(database=None)[source]

Bases: irfpy.imacommon.imaextra2.ImaDdNcDatabase

>>> db = ImaExtraDatabase.createDatabase()
>>> t = datetime.datetime(2007, 9, 10, 3, 0)
>>> print(os.path.basename(db.get_filename(t)))
imaextra20072520101.nc.gz

Create the database.

Use createDatabase() class method for creating database.

Parameters

  • database – Database directory path. If None, the [vima]-imaextrabase entry from irfpyrc is read.

  • prefix_name – The filename should be imaXXXXX20103091457.nc.gz. The prefix, imaXXXXX should be specified to identify the start of the time expression.

  • ddncfactory – Factory object that create the object of the file contents. It should be an object of the class that extends ImaDdNcFileFactory. For example, ImaExtraFileFactory can be given.

  • verbose – The keyword is outdated. Do nothing. Use logging to display a log with the name of irfpy.imacommon.imaextra2.ImaDdNcDatabase.

classmethod createDatabase(refresh=False)[source]

Create the database.

class irfpy.vima.imaextra.ImaParamDatabase(database=None)[source]

Bases: irfpy.imacommon.imaextra2.ImaDdNcDatabase

Create the database.

Use createDatabase() class method for creating database.

Parameters

  • database – Database directory path. If None, the [vima]-imaextrabase entry from irfpyrc is read.

  • prefix_name – The filename should be imaXXXXX20103091457.nc.gz. The prefix, imaXXXXX should be specified to identify the start of the time expression.

  • ddncfactory – Factory object that create the object of the file contents. It should be an object of the class that extends ImaDdNcFileFactory. For example, ImaExtraFileFactory can be given.

  • verbose – The keyword is outdated. Do nothing. Use logging to display a log with the name of irfpy.imacommon.imaextra2.ImaDdNcDatabase.

classmethod createDatabase(refresh=False)[source]

Create the database.

irfpy.vima.imaextra.getImaExtraFile(filename=None, t=None, gunzip=True)[source]

An easy accessor to ImaExtraFile getter.

With a support of cache (ImaExtraFileFactory), an optimized ImaExtraFile getter.

irfpy.vima.imaextra.getImaParamFile(filename=None, t=None, gunzip=True)[source]

Get the IMA parameter in a form of ImaParam object.

It is a user function, but slightly low-level. High level functions are to be made

Parameters

  • t0 – Time to start

  • t1 – Time to end

Returns

IMA parameter object

Return type

ImaParam

>>> imaParam = getImaParam(datetime.datetime(2011, 11, 5, 6), datetime.datetime(2011, 11, 5, 8))
irfpy.vima.imaextra.getImaParam(t0, t1)[source]

Return the ImaParam object.

Parameters

  • t0 – Time to start

  • t1 – Time to end

Returns

IMA parameter object

Return type

ImaParam

irfpy.vima.imaextra.getImaExtra(t0, t1)[source]

Get the IMA extra data in a form of TBD

Parameters

  • t0 – Time to start

  • t1 – Time to end

Returns

IMA extra object

Return type

ImaExtra

>>> imaExtra = getImaExtra(datetime.datetime(2011, 11, 5, 6), datetime.datetime(2011, 11, 6, 8))
>>> t = imaExtra.obstime
>>> print(len(t))
123
>>> hp = imaExtra.getHpSpec()
>>> print(hp.shape)
(16, 96, 16, 123)
>>> print(hp.max())
25832.723
>>> op = imaExtra.getHeavySpec()
>>> print(op.shape)
(16, 96, 16, 123)
>>> print(op.max())
9019.518
class irfpy.vima.imaextra.iterparam(t0, t1)[source]

Bases: object

Iterate the IMA parameter file.

Iterated is a list of (t, dat) were t is the time of the observation start, and dat is a dictionary that contains the data.

The key of dat is ImaParam.data_keys. The dat is not a ImaParam object, but rather, ImaParam.data data.

Parameters

  • t0 – Time to start. Iterated data’s start time is always later than or equal to t0.

  • t1 – Time to stop. Iterated data’s stop time is always earlier than or equal to t1.

Returns

A list, (t, dat). t is time, and dat is a dictionary containing the IMAPARAM data. Its key is ImaParam.data_keys.

Return type

list of (datetime.datetime, dict)

If you want to iterate data from 2011-11-05T05:15 for 10 minutes, you can do as follows:

>>> t0 = datetime.datetime(2011, 11, 5, 5, 15)
>>> t1 = datetime.datetime(2011, 11, 5, 5, 25)

>>> for t, mom in iterparam(t0, t1):
...     print(t, mom['Density_P'])
2011-11-05 05:17:00.200000 9.732157
2011-11-05 05:20:12.262000 11.949523
2011-11-05 05:23:24.262000 10.680418
data_keys = ['ScanFlag', 'Quality_P', 'Density_P', 'Velocity_P', 'Temperature_P', 'EnergyThermal_P', 'Quality_O', 'Density_O', 'Velocity_O', 'Temperature_O', 'EnergyThermal_O']

Key of the data

class irfpy.vima.imaextra.iter_imaextra(t0, t1)[source]

Bases: object

Iterate the IMA extra data.

Iterate the IMA extra data, as a tuple of (time, data). Here data is an object of ImaExtra.

>>> t0 = datetime.datetime(2011, 11, 5, 5, 15)
>>> t1 = datetime.datetime(2011, 11, 5, 5, 25)

>>> for t, ext in iter_imaextra(t0, t1):
...     print(t, ext.getHpSpec().shape, ext.getHeavySpec().max())
2011-11-05 05:17:00.200000 (16, 96, 16, 1) 7.0941877
2011-11-05 05:20:12.262000 (16, 96, 16, 1) 1.4826224
2011-11-05 05:23:24.262000 (16, 96, 16, 1) 0.0
data_keys = ['AngTableN', 'ETableN', 'FracO2', 'HeavySpec', 'HpSpec', 'Pacc', 'RestRm', 'Time']
class irfpy.vima.imaextra.iterextra(*args, **kwds)[source]

Bases: irfpy.vima.imaextra._iterextra

Iterate the IMA extra data.

Deprecated since version v4.3.10: (TBD)

Use iter_imaextra.

Iterated is a list of (t, dat) where t is the time of the observation start, and dat is a dictionary that contains the data.

The key of the dat is data_keys. It is not the ImaExtra object.

Parameters

  • t0 – Time to start. Iterated data’s start time is always later than or equal to t0.

  • t1 – Time to stop. Iterated data’s start time is always earlier than or equal to t1.

Returns

A list, (t, dat). t is time, and dat is a dictionary containing the IMAEXTRA data. Its key is ImaExtra.data_keys.

Return type

list of (datetime.datetime, dict)

>>> t0 = datetime.datetime(2011, 11, 5, 7)
>>> t1 = datetime.datetime(2011, 11, 5, 8)
>>> for t, dat in iterextra(t0, t1):    # Iterate over the time t0 and t1.
...     print(t, dat['RestRm'].sum())   # Output time and the total counts in 'RestRm' data.  
2011-11-05 07:02:37.043000 70305.94
2011-11-05 07:05:49.138000 105089.24
2011-11-05 07:09:01.107000 49731.812
...
2011-11-05 07:53:49.543000 3901.1763
2011-11-05 07:57:01.513000 6311.3394