qcodes_loop.data.data_set

DataSet class and factory functions.

class qcodes_loop.data.data_set.DataSet(location=None, arrays=None, formatter=None, io=None, write_period=5)

A container for one complete measurement loop.

May contain many individual arrays with potentially different sizes and dimensionalities.

Normally a DataSet should not be instantiated directly, but through new_data or load_data.

Parameters:

• location (Union[str,bool]) – A location in the io manager, or False for an only-in-memory temporary DataSet. Note that the full path to or physical location of the data is a combination of io + location. the default DiskIO sets the base directory, which this location is a relative path inside.

• io (Optional[io_manager]) – base physical location of the DataSet. Default DataSet.default_io is initially DiskIO('.') which says the root data directory is the current working directory, ie where you started the python session.

• arrays (Optional[List[qcodes.data.data_array.DataArray]) – arrays to add to the DataSet. Can be added later with self.add_array(array).

• formatter (Optional[Formatter]) – sets the file format/structure to write (and read) with. Default DataSet.default_formatter which is initially GNUPlotFormat().

• write_period (Optional[float]) – Only if mode=LOCAL, seconds between saves to disk. If not LOCAL, the DataServer handles this and generally writes more often. Use None to disable writing from calls to self.store. Default 5.

add_array(data_array)

Add one DataArray to this DataSet, and mark it as part of this DataSet.

Note: DO NOT just set data_set.arrays[id] = data_array, because this will not check if we are overwriting another array, nor set the reference back to this DataSet, nor that the array_id in the array matches how you’re storing it here.

Parameters:

data_array (DataArray) – the new array to add

Raises:

ValueError – if there is already an array with this id here.

add_metadata(new_metadata)

Update DataSet.metadata with additional data.

Parameters:

new_metadata (dict) – new data to be deep updated into the existing metadata

background_functions: dict[str, Callable[..., Any]] = {}

The value fn is a callable accepting no arguments, and key is a name to identify the function and help you attach and remove it.

In DataSet.complete we call each of these periodically, in the order that they were attached.

Note that because this is a class attribute, the functions will apply to every DataSet. If you want specific functions for one DataSet you can override this with an instance attribute.

complete(delay=1.5)

Periodically sync the DataSet and display percent complete status.

Also, each period, execute functions stored in (class attribute) self.background_functions. If a function fails, we log its traceback and continue on. If any one function fails twice in a row, it gets removed.

Parameters:

delay (float) – seconds between iterations. Default 1.5

default_parameter_array(paramname='amplitude')

Return default parameter array

Parameters:

paramname (str) – Name to match to parameter name. Defaults to ‘amplitude’

Returns:

array corresponding to the default parameter

Return type:

DataArray

See also

default_parameter_name

default_parameter_name(paramname: str | None = None) → str | None

Return name of default parameter for plotting

The default parameter is determined by looking into metdata[‘default_parameter_name’]. If this variable is not present, then the closest match to the argument paramname is tried.

Parameters:

paramname – Name to match to parameter name

Returns:

Name of the default parameter, or None if no parameter is found

delegate_attr_dicts: ClassVar[list[str]] = ['arrays']

A list of names (strings) of dictionaries which are (or will be) attributes of self, whose keys should be treated as attributes of self.

finalize(filename=None, write_metadata=True)

Mark the DataSet complete and write any remaining modifications.

Also closes the data file(s), if the Formatter we’re using supports that.

Parameters:

• filename (Optional[str]) – The file name (minus extension) to write to. The location of the file is the usual one.

• write_metadata (bool) – Whether to save a snapshot. For e.g. dumping raw data inside a loop, a snapshot is not wanted.

fraction_complete()

Get the fraction of this DataSet which has data in it.

Returns:

the average of all measured (not setpoint) arrays’

fraction_complete() values, independent of the individual array sizes. If there are no measured arrays, returns zero.

Return type:

float

classmethod from_xarray(xarray_dataset: xr.Dataset) → DataSet

Convert the dataset to an xarray DataSet

get_array_metadata(array_id)

Get the metadata for a single contained DataArray.

Parameters:

array_id (str) – the array to get metadata for.

Returns:

metadata for this array.

Return type:

dict

get_changes(synced_indices)

Find changes since the last sync of this DataSet.

Parameters:

synced_indices (dict) – {array_id: synced_index} where synced_index is the last flat index which has already been synced, for any (usually all) arrays in the DataSet.

Returns:

keys are array_id for each array with changes,

values are dicts as returned by DataArray.get_changes and required as kwargs to DataArray.apply_changes. Note that not all arrays in synced_indices need be present in the return, only those with changes.

Return type:

Dict[dict]

read()

Read the whole DataSet from storage, overwriting the local data.

read_metadata()

Read the metadata from storage, overwriting the local data.

remove_array(array_id)

Remove an array from a dataset

Throws an exception when the array specified is refereced by other arrays in the dataset.

Parameters:

array_id (str) – array_id of array to be removed

save_metadata()

Evaluate and save the DataSet’s metadata.

snapshot(update=False)

JSON state of the DataSet.

store(loop_indices, ids_values)

Insert data into one or more of our DataArrays.

Parameters:

• loop_indices (tuple) – the indices within whatever loops we are inside. May have fewer dimensions than some of the arrays we are inserting into, if the corresponding value makes up the remaining dimensionality.

• values (Dict[Union[float, Sequence]]) – a dict whose keys are array_ids, and values are single numbers or entire slices to insert into that array.

sync()

Synchronize this DataSet with the DataServer or storage.

If this DataSet is on the server, asks the server for changes. If not, reads the entire DataSet from disk.

Returns:

True if this DataSet is live on the server

Return type:

bool

to_xarray() → xr.Dataset

Convert the dataset to an xarray Dataset

write(write_metadata=False, only_complete=True, filename=None)

Writes updates to the DataSet to storage. N.B. it is recommended to call data_set.finalize() when a DataSet is no longer expected to change to ensure files get closed

Parameters:

• write_metadata (bool) – write the metadata to disk

• only_complete (bool) – passed on to the match_save_range inside self.formatter.write. Used to ensure that all new data gets saved even when some columns are strange.

• filename (Optional[str]) – The filename (minus extension) to use. The file gets saved in the usual location.

write_copy(path=None, io_manager=None, location=None)

Write a new complete copy of this DataSet to storage.

Parameters:

• path (Optional[str]) – An absolute path on this system to write to. If you specify this, you may not include either io_manager or location.

• io_manager (Optional[io_manager]) – A new io_manager to use with either the DataSet’s same or a new location.

• location (Optional[str]) – A new location to write to, using either this DataSet’s same or a new io_manager.

qcodes_loop.data.data_set.dataset_to_xarray_dictionary(data_set: DataSet, include_metadata: bool = True) → dict[str, Any]

Convert QcodesDataSet to dictionary.

Parameters:

• data_set – The data to convert.

• include_data – If True then include the ndarray field.

• include_metadata – If True then include the metadata.

Returns:

Dictionary containing the serialized data.

qcodes_loop.data.data_set.load_data(location=None, formatter=None, io=None)

Load an existing DataSet.

Parameters:

• location (Optional[str]) – the location to load from. Default is the current live DataSet. Note that the full path to or physical location of the data is a combination of io + location. the default DiskIO sets the base directory, which this location is a relative path inside.

• formatter (Optional[Formatter]) – sets the file format/structure to read with. Default DataSet.default_formatter which is initially GNUPlotFormat().

• io (Optional[io_manager]) – base physical location of the DataSet. Default DataSet.default_io is initially DiskIO('.') which says the root data directory is the current working directory, ie where you started the python session.

Returns:

A new DataSet object loaded with pre-existing data.

qcodes_loop.data.data_set.new_data(location=None, loc_record=None, name=None, overwrite=False, io=None, **kwargs)

Create a new DataSet.

Parameters:

• location (Optional[Union[str,Callable, Bool]]) –

  If you provide a string, it must be an unused location in the io manager. Can also be:

  • a Callable location provider with one required parameter (the io manager), and one optional (record dict), which returns a location string when called

  • False - denotes an only-in-memory temporary DataSet.

  Note that the full path to or physical location of the data is a combination of io + location. the default DiskIO sets the base directory, which this location is a relative path inside. Default DataSet.location_provider which is initially FormatLocation()

• loc_record (Optional[dict]) – If location is a callable, this will be passed to it as record

• name (Optional[str]) – overrides the name key in the loc_record.

• overwrite (bool) – Are we allowed to overwrite an existing location? Default False.

• io (Optional[io_manager]) – base physical location of the DataSet. Default DataSet.default_io is initially DiskIO('.') which says the root data directory is the current working directory, ie where you started the python session.

• arrays (Optional[List[qcodes.data.data_array.DataArray]) – arrays to add to the DataSet. Can be added later with self.add_array(array).

• formatter (Optional[Formatter]) – sets the file format/structure to write (and read) with. Default DataSet.default_formatter which is initially GNUPlotFormat().

• write_period (Optional[float]) – seconds between saves to disk.

Returns:

A new DataSet object ready for storing new data in.

qcodes_loop.data.data_set.qcodes_dataset_to_xarray_dataset(data_set: DataSet) → xr.Dataset

Convert QCoDeS gridded dataset to xarray dataset

qcodes_loop.data.data_set.xarray_dataset_to_qcodes_dataset(xarray_data_set: xr.Dataset) → DataSet

Convert QCoDeS gridded dataset to xarray dataset

qcodes_loop.data.data_set.xarray_dictionary_to_dataset(xarray_dictionary: dict[str, Any]) → DataSet

Convert xarray dictionary to Qcodes DataSet.

Parameters:

xarray_dictionary – data to convert

Returns:

QCoDeS dataSet with converted data.