IMA Extra dataset

This section describes how to use the IMA extra dataset by irfpy.

Preparation

Note

This is one time setup. Go to Reading the data for those who have prepared the dataset.

IMA extra dataset is prepared by Andrei Fedorov (IRAP). The dataset is huge, but should be downloaded locally. Contact to Futaana for download.

Warning

IRF, irfpy develop team, and ASPERA-3/-4 PI team do not guarantee anything about IMA extra dataset. The use of the data should be conducted by user’s responsibility.

It is strongly recommended to contact to the PI team (futaana@irf.se).

The path structure should be preserved as is.

% ls /Volumes/scidata/data/venus/DD/NC/
IMAANG          IMAEXTRA        IMAEXTRALIGHT   IMAPARAM

The dataset path should be specified in .irfpyrc file, for example,

[vima]
imaddncbase = /Volumes/scidata/data/venus/DD/NC/

[mima]
imaddncbase = /Volumes/scidata/data/mars/DD/NC/

Reading the data

IMA parameters (MEX)

The simplest way to access the parameter (mement) data is using the irfpy data center for ImaParam (irfpy.mima.imaextra.DataCenterImaParam).

Example follows.

All the data access starts from

>>> from irfpy.mima import imaextra
>>> dc = imaextra.DataCenterImaParam()

The object dc here handles the ImaParam dataset.

>>> print(dc)
irfpy DataCenter (Name: 'Unnamed DataCenter'): 18974 data files combined

To access the data, several methods of DataCenter (dc) may be used. For example, the nearest method will return the data nearest to the gien time.

>>> import datetime
>>> obstime, param = dc.nearest(datetime.datetime(2011, 3, 10, 5, 30))
>>> print(obstime)
2011-03-10 05:30:46.967000
>>> print(param)
<irfpy.mima.imaextra.ImaParam object at 0x1180ba160>

As seen, the returned data is a tuple with two elements: The first for the actual (start) time of the observation, and the second for the data.

The data is an object of irfpy.mima.imaextra.ImaParam. You can access the data as follows.

>>> param.data['Velocity_P']
array([[-355.2511  ,   31.835934,   16.740742]], dtype=float32)

As seen the data is a dictionary. The key of the dictorinary is obtained from data_keys attirbute.

>>> print(param.data_keys)
['ScanFlag', 'Quality_P', 'Density_P', 'Velocity_P', 'Temperature_P', 'EnergyThermal_P', 'Quality_O', 'Density_O', 'Velocity_O', 'Temperature_O', 'EnergyThermal_O', 'Quality_O2', 'Density_O2', 'Velocity_O2', 'Temperature_O2', 'EnergyThermal_O2']

There is also the way to control the data, for example, iterate the data over the specific time range.

Just an example code block is shown.

>>> t0 = datetime.datetime(2011, 3, 10, 5)
>>> t1 = datetime.datetime(2011, 3, 10, 6)
>>> for obstime, param in dc.iter(t0, t1):
...     print(obstime, param.data['Density_P'])
2011-03-10 05:01:58.779000 [0.85120714]
2011-03-10 05:05:10.749000 [0.63146347]
...
2011-03-10 05:56:23.187000 [1.1041325]
2011-03-10 05:59:35.279000 [0.38089478]

For more information about the data center, please refer to https://irfpy.irf.se/projects/util/tutorial/tutorial_datacenter.html#how-to-read-the-data

IMA parameters (VEX)

The simplest way of reading paramter (moment) file is using irfpy.vima.imaextra.getImaParam() and irfpy.mima.imaextra.getImaParam() functions.

Note

DataCenter approach may be implemented in the future.

Example follows.

Suppose you want to get the moment data between

>>> import datetime
>>> t0 = datetime.datetime(2010, 10, 5)
>>> t1 = datetime.datetime(2010, 10, 10)

Then, you call the irfpy.mima.imaextra.getImaParam() function. You will get the data as irfpy.mima.imaextra.ImaParam object.

>>> import irfpy.mima.imaextra
>>> parameters = irfpy.mima.imaextra.getImaParam(t0, t1)

The data in paramters can be refered by data_keys member.

>>> print(parameters.data_keys)
['ScanFlag', 'Quality_P', 'Density_P', 'Velocity_P', 'Temperature_P', ...]

To access the data, you can use ndat, obstime and data

>>> print(parameters.ndat)
468
>>> print(parameters.obstime[0:5])
[datetime.datetime(2010, 10, 5, 5, 41, 37, 812000),
 datetime.datetime(2010, 10, 5, 5, 44, 49, 812000),
 datetime.datetime(2010, 10, 5, 5, 48, 1, 907000),
 datetime.datetime(2010, 10, 5, 5, 51, 13, 875000),
 datetime.datetime(2010, 10, 5, 5, 54, 25, 970000)]
>>> print(parameters.data['Density_P'][0:5])
[ 0.99981284  2.05224562  1.81596529  1.52338517  1.5351398 ]

Same for VEX.

Iteration

Sometimes, you may want to iterate over some period. In such cases, you can consider the following example using irfpy.mima.imaextra.iterparam() or irfpy.vima.imaextra.iterparam().

>>> import irfpy.vima.imaextra
>>> for t, d in irfpy.vima.imaextra.iterparam(t0, t1):
...     print(t, d)
2010-10-05 05:30:37.944000 {'ScanFlag': 1, 'Quality_P': 0.9888113, 'Density_P': 4.522062, ...}
2010-10-05 05:33:49.944000 {'ScanFlag': 1, 'Quality_P': 0.96211284, 'Density_P': 4.9183373, ...}
...
2010-10-09 18:35:36.845000 {'ScanFlag': 1, 'Quality_P': 0.9074254, 'Density_P': 0.8174821, ...}

IMA extra dataset

The IMA extra dataset can be read as follows.

Note

So far only VEX is implemented.

Prepare the time and the module.

>>> from irfpy.vima import imaextra
>>> import datetime
>>> t0 = datetime.datetime(2011, 11, 5, 5)
>>> t1 = datetime.datetime(2011, 11, 5, 6)

Then, you can get the data via irfpy.vima.imaextra.getImaExtra() function. The obtained data is an object of irfpy.vima.imaextra.ImaExtra.

>>> extdata = imaextra.getImaExtra(t0, t1)

You can access the observation times and counts as follows.

>>> tlist = extdata.getobstime()
>>> print(tlist[:5])
[datetime.datetime(2011, 11, 5, 5, 17, 0, 200000),
datetime.datetime(2011, 11, 5, 5, 20, 12, 262000),
datetime.datetime(2011, 11, 5, 5, 23, 24, 262000),
datetime.datetime(2011, 11, 5, 5, 26, 36, 230000),
datetime.datetime(2011, 11, 5, 5, 29, 48, 325000)]

>>> hp = extdata.getHpSpec()
>>> print(hp.shape)   # The shape of the proton spectrum
(16, 96, 16, 14)

The obtained counts are ordered in a standard way (MAEPT order, namely, (A16, E96, P16, T14) in this case. Note that the mass is already selected for proton so there is no corresponding axis.

Heavy ion counts are

>>> op = extdata.getHeavySpec()
>>> print(op.max())    # Maximum counts
259.09006

and the “rest” matrix, namely, the counts that are not assigned to either of the speces, will remain

>>> rest = extdata.getRestRm()
>>> print(rest.shape)    # Rest matrix shape should be (M32, A16, E96, P16, T)
(32, 16, 96, 16, 14)

Iteration

Deprecated since version v4.3.10: Iteration using iterextra <irfpy.vima.imaextra.iterextra() for VEX is now deprecated. Recommend to use iter_imaextra <irfpy.vima.imaextra.iter_imaextra() for VEX.

Recipe of changing the code

The following is the old-fashioned way of iteration.

>>> from irfpy.vima.imaextra import iterextra
>>> for t, d in iterextra(t0, t1):
...     hp = d['HpSpec']     # Here data ``d`` is a dictionary containing only the data
...     print(hp.shape)

The obtained hp is the non-standard order of the array.

The new iteration scheme returns irfpy.vima.imaextra.ImaExtra object.

>>> for t, dnew in iter_imaextra(t0, t1):
...     hpnew = dnew.getHpSpec()     # Here, data ``d`` is ImaExtra object.

hpnew is the standard order, namely, (A16, E96, P16).

Iteration of IMA extra data is done using irfpy.vima.imaextra.iter_imaextra().

>>> for t, d in iter_imaextra(t0, t1):
...     h = d.getHpSpec()     # Here, data ``d`` is ImaExtra object.
...     o = d.getHeavySpec()     # Here, data ``d`` is ImaExtra object.
...     r = d.getRestRm()

Here, you can get the IMA extra data in a form of irfpy.vima.ImaExtra, with which you can control the data more flexible way.

Other dataset

Not yet fully implemented.

For users who want to contribute, let Futaana know.