irfpy.util.tdict

This module provides an implementation of time series of data

Code author: Yoshifumi Futaana

It provides an implementation of time series data handling. This is similar to a dictionary with a key of datetime.datetime object, but provides a rahter dedicated interface.

TimeDict()

Implementation of the timeseries database.

Note

If you want to use the time-float pair, you may also consider using irfpy.util.timeseries.ScalarSeries class.

If you want to use the time-(x, y, z) pair, you may also consider using irfpy.util.timeseries.VectorSeries class.

If you want to use the time-string pair, you may also consider using irfpy.util.timeseriesdb module, which uses similar algorithm to this module. The difference is that the pyana.util.timeseriesdb module provide “bi-direction” key, which means that both time->string and time<-string reference can be possible.

Todo

Write a use case.

exception irfpy.util.tdict.DataNotInDbError(value)

Bases: IrfpyError

exception irfpy.util.tdict.DuplicatedError(value)

Bases: IrfpyError

class irfpy.util.tdict.TimeDict

Bases: object

Implementation of the timeseries database.

logger = <Logger irfpy.util.tdict.TimeDict (DEBUG)>

append(starttime, data, dup='raise')

Append the data into the database together with the start time.

Parameters:

• starttime – Start time (datetime.datetime object)

• data – Data

• dup – If the start time is duplicated, what action is taken? ‘raise’ will raise the exception (default). ‘ignore’ will ignore the given data, and old data is kept. ‘update’ will update the data, and old data is disregarded.

>>> td = TimeDict()
>>> td.append(datetime.datetime(2010, 3, 10, 15), {"a": 30, "b": 70})
>>> td.append(datetime.datetime(2010, 3, 11, 15), {"a": 20, "b": 90})
>>> td.append(datetime.datetime(2010, 3, 12, 15), {"a": 30, "b": 70})

getobstime()

Return the list of observation time in the database.

>>> td = TimeDict()
>>> td.append(datetime.datetime(2010, 3, 10, 15), {"a": 30, "b": 70})
>>> td.append(datetime.datetime(2010, 3, 11, 15), {"a": 20, "b": 90})
>>> td.append(datetime.datetime(2010, 3, 12, 15), {"a": 30, "b": 70})
>>> print(td.getobstime())
[datetime.datetime(2010, 3, 10, 15, 0), datetime.datetime(2010, 3, 11, 15, 0), datetime.datetime(2010, 3, 12, 15, 0)]

keys()

Return the list of observation time in the database.

Returns:

List of the observation time.

This is just an alias to getobstime().

t0()

Return the start time of the first data.

>>> td = TimeDict()
>>> td.append(datetime.datetime(2010, 3, 10, 15), {"a": 30, "b": 70})
>>> td.append(datetime.datetime(2010, 3, 11, 15), {"a": 20, "b": 90})
>>> td.append(datetime.datetime(2010, 3, 12, 15), {"a": 30, "b": 70})

>>> print(td.t0())
2010-03-10 15:00:00

t1()

Return the start time of the last data

Note that this is the start time of the last data. The end time of the last data is not defined (so that the user should handle this in their script).

>>> td = TimeDict()
>>> td.append(datetime.datetime(2010, 3, 10, 15), {"a": 30, "b": 70})
>>> td.append(datetime.datetime(2010, 3, 11, 15), {"a": 20, "b": 90})
>>> td.append(datetime.datetime(2010, 3, 12, 15), {"a": 30, "b": 70})

>>> print(td.t1())
2010-03-12 15:00:00

get(t)

Return the data of time t and the start time.

Return the data of time t and the start time. If the specified time is before the DB start time, DataNotInDbError is raised.

Parameters:

t – Time (datetime.datetime)

Returns:

(t0, data)

2004-01-05T15:30:00 contains in orb0001.

>>> td = TimeDict()
>>> td.append(datetime.datetime(2010, 3, 10, 15), ["a", 30, "b", 70])
>>> td.append(datetime.datetime(2010, 3, 11, 15), ["a", 20, "b", 90])
>>> td.append(datetime.datetime(2010, 3, 12, 15), ["a", 30, "b", 70])
>>> td.append(datetime.datetime(2010, 3, 13, 15), ["a", 20, "b", 90, "c", 50])

>>> t0 = datetime.datetime(2010, 3, 11, 23, 0)
>>> print(td.get(t0))
(datetime.datetime(2010, 3, 11, 15, 0), ['a', 20, 'b', 90])

>>> t1 = datetime.datetime(2010, 3, 13, 0)
>>> print(td.get(t1))
(datetime.datetime(2010, 3, 12, 15, 0), ['a', 30, 'b', 70])

2004-01-05T00:00:00 is before the database.

>>> t2 = datetime.datetime(2004, 1, 5, 0)
>>> print(td.get(t2))    
Traceback (most recent call last):
tdict.DataNotInDbError: 'This is the first file in the DB (file0).'

2030-01-01T00:00:00 is, in a common sense, not included in this data base, but it returns the last data. Thus, it is a user’s responsibility to validate the end time of the data.

>>> t3 = datetime.datetime(2030, 1, 1)
>>> print(td.get(t3))
(datetime.datetime(2010, 3, 13, 15, 0), ['a', 20, 'b', 90, 'c', 50])

getbetween(tmin=None, tmax=None, strict=False)

Return the tuple the start time tuple and data tuple.

Parameters:

• tmin – Start. (datetime.datetime)

• tmax – End. (datetime.datetime) Inclusive.

• strict – If True, the strict mode is used. By default False. See clipped() function for what is the strict mode.

Returns:

((ti), (di)) where (ti) is the tuple of time and (di) is the tuple of data.

>>> db = TimeDict()
>>> db.append(datetime.datetime(2004, 1, 5, 15, 30), "A")
>>> db.append(datetime.datetime(2004, 1, 5, 18, 30), "AB")
>>> db.append(datetime.datetime(2004, 1, 6, 0, 30), "ABC")
>>> db.append(datetime.datetime(2004, 1, 6, 9, 30), "D")
>>> db.append(datetime.datetime(2004, 1, 6, 18, 30), "DE")
>>> db.append(datetime.datetime(2004, 1, 7, 3, 30), "DEF")
>>> db.append(datetime.datetime(2004, 1, 12, 11, 30), "D")
>>> db.append(datetime.datetime(2004, 1, 12, 21, 30), "GH")
>>> print(db.getbetween(datetime.datetime(2004, 1, 6), datetime.datetime(2004, 1, 7)))
((datetime.datetime(2004, 1, 5, 18, 30), datetime.datetime(2004, 1, 6, 0, 30), datetime.datetime(2004, 1, 6, 9, 30), datetime.datetime(2004, 1, 6, 18, 30)), ('AB', 'ABC', 'D', 'DE'))
>>> print(db.getbetween(datetime.datetime(2004, 1, 6), datetime.datetime(2004, 1, 7), strict=True))
((datetime.datetime(2004, 1, 6, 0, 30), datetime.datetime(2004, 1, 6, 9, 30), datetime.datetime(2004, 1, 6, 18, 30)), ('ABC', 'D', 'DE'))
>>> print(db.getbetween(datetime.datetime(2004, 1, 1), datetime.datetime(2004, 1, 5, 18, 30)))
((datetime.datetime(2004, 1, 5, 15, 30), datetime.datetime(2004, 1, 5, 18, 30)), ('A', 'AB'))

>>> t_all, d_all = db.getbetween()
>>> print(len(t_all), len(d_all))
8 8

clipped(tstart, tend, strict=False)

Return the clipped object.

Parameters:

• tstart – Start. (datetime.datetime)

• tsend – End. (datetime.datetime) Inclusive.

• strict – Enable/Disable strict mode. By default, False.

Returns:

TimeDict object that is clipped by tmin and tmax.

A newly generated, clipped TimeDict is returned.

Strict mode The strict mode considers the time in the original TimeDict as a point. The default (non-strict) mode considers the time in the original TimeDict as a start time of the interval.

>>> db = TimeDict()
>>> db.append(datetime.datetime(2004, 1, 5, 15, 30), "A")
>>> db.append(datetime.datetime(2004, 1, 5, 18, 30), "AB")
>>> db.append(datetime.datetime(2004, 1, 6, 0, 30), "ABC")
>>> db.append(datetime.datetime(2004, 1, 6, 9, 30), "D")
>>> db.append(datetime.datetime(2004, 1, 6, 18, 30), "DE")
>>> db.append(datetime.datetime(2004, 1, 7, 3, 30), "DEF")
>>> db.append(datetime.datetime(2004, 1, 12, 11, 30), "D")
>>> db.append(datetime.datetime(2004, 1, 12, 21, 30), "GH")

Let’s try to clip the data.

>>> db_clipped = db.clipped(datetime.datetime(2004, 1, 6), datetime.datetime(2004, 1, 7))
>>> print(len(db_clipped))
4
>>> print(db_clipped[0])
(datetime.datetime(2004, 1, 5, 18, 30), 'AB')
>>> print(db_clipped[-1])
(datetime.datetime(2004, 1, 6, 18, 30), 'DE')

Note that the first element is the data in Jan 5th, not Jan 6th. This feature is because the data of Jan 6th 0:00:00 may be contained in the dataset starting from Jan 5th 18:30.

>>> db_clipped2 = db.clipped(datetime.datetime(2012, 10, 5), datetime.datetime(2012, 11, 5), strict=True)
>>> print(len(db_clipped2))
0
>>> db_clipped2 = db.clipped(datetime.datetime(2012, 10, 5), datetime.datetime(2012, 11, 5), strict=False)
>>> print(len(db_clipped2))
1
>>> db_clipped2 = db.clipped(datetime.datetime(1912, 10, 5), datetime.datetime(1912, 11, 5), strict=True)
>>> print(len(db_clipped2))
0
>>> db_clipped2 = db.clipped(datetime.datetime(1912, 10, 5), datetime.datetime(1912, 11, 5), strict=False)
>>> print(len(db_clipped2))
0

clear()

irfpy.util.tdict.doctests()