Tutorial 06: MetaData

Exploring metadata collected and saved during data collections.

from lisc.io import SCDB, load_object

Metadata

Whenever you collect data with LISC, meta data is collected about the API requests and databases accessed.

Here we will explore the metadata collected during our previous investigations.

# Reload the counts object
counts = load_object('tutorial_counts', SCDB('lisc_db'))

Metadata Object

Metadata information is collected into a custom MetaData object.

If you are collecting data using the LISC object, such as the Counts or Words object, this collection information is attached and saved to the object as the meta_data attribute.

# Check the date on which the collection happened
print(counts.meta_data.date)

2023-10-29_00:27:28

# Check the information about the database from which data was collected
counts.meta_data.db_info

{'dbname': 'pubmed', 'menuname': 'PubMed', 'description': 'PubMed bibliographic record', 'dbbuild': 'Build-2023.10.29.00.10', 'count': '36388316', 'lastupdate': '2023/10/29 00:10'}

For the next part, we’ll reload the Words object, which also has stored meta data.

# Reload the words object
words = load_object('tutorial_words', SCDB('lisc_db'))

Requester MetaData

The collected metadata also includes information from the Requester object, which is used to launch URL requests.

The Requester object tracks information including when URL requests were launched and the number of requests made.

# Check meta data from the requester object
words.meta_data.requester

{'n_requests': 9, 'wait_time': 0.3333333333333333, 'start_time': '00:27:17 Sunday 29 October 2023', 'end_time': '00:27:22 Sunday 29 October 2023', 'logging': None}

The End!

That is the end of the main tutorial for LISC!

There are some additional stand alone examples on the examples page.

If you have any further questions, or find any problems with the code, please get in touch through the Github issues page.