This page was generated from docs/examples/logging/logging_example.ipynb. Interactive online version: Binder badge.

Logging

(back to the overview offline,online)

read on nbviewer

TL;DR

  • There is a QCoDeS logging module: qcodes.logger

  • Call logger.start_all_logging at the start of every script/session to make sure all log messages get stored to the .qcodes/logs folder in your home directory, or set logger.start_logging_on_import to True in your qcodesrc.json configuration.

  • For debugging purposes you can log messages of an individual instrument (and see the VISA dialog).

  • You can obtain all logging messages in a pandas.DataFrame for further analysis.

Contents

Introduction

Python offers an excellent logging framework that is widely used in QCoDeS. As a user, you might be interested in reading the log messages that are generated by QCoDeS for debugging and also profiling purposes.

All log records(i.e., messages and metadata) are created using a python logging.logger object. In QCoDeS there is a logger for every module with the same name as the module itself, i.e., you can use logging.getLogger('qcodes.instrument.base') for getting the logger of the QCoDeS Instrument. From the dots in the names of the loggers a hierarchy is defined, at the top of which is the so called root logger that catches all messages. The log records created in a logger are processed by the handlers that have been attached to this specific logger. The handlers can, for example, be used to output the log messages into different locations, like the console or a log file.

Every logger, handler and log record has an associated level. The default levels are: CRITICAL, ERROR, WARNING, INFO and DEBUG. The levels determine how messages are passed: a logger passes only records that exceeds its minimal passing level to its handlers and the handlers in turn only output those messages that exceed their minimal passing level.

For a more detailed description of Python logging refer to the official manual (This is all well summed up in this chart)

Instrument communication loggers

Not all the used loggers in QCoDeS follow the previously presented rule of naming the logger after the module. For instruments there are the following loggers: - qcodes.instrument.base.com: This logger logs all the communication of the Instrument and its descendants with the actual physical instruments, e.g., the messages generated by the write and query calls. - qcodes.instrument.base.com.visa: This is a sublogger that logs all the communication related messages of the VisaInstrument and its descendants.

Set up logging in QCoDeS

At the beginning of every experiment script file you should include the following lines:

[1]:
from qcodes.logger import start_all_logging
start_all_logging()
Logging hadn't been started.
Activating auto-logging. Current session state plus future input saved.
Filename       : /home/runner/.qcodes/logs/command_history.log
Mode           : append
Output logging : True
Raw input log  : False
Timestamping   : True
State          : active
Qcodes Logfile : /home/runner/.qcodes/logs/220701-19435-qcodes.log

The call to start_all_logging does two things: 1. Create handlers to output logging messages from the root logger to the console and to a log file. 2. Enable logging of the IPython commands into a separate log file.

Alternatively we can configure qcodes to automatically start logging on every import of qcodes, by running the following code once. (This will persist the current configuration in ~\qcodesrc.json)

[2]:
from qcodes import config
config.logger.start_logging_on_import = 'always'
config.save_to_home()

QCoDeS default handlers

The effect of the console handler can be seen in the two printed lines with a red background below the previous code cell. Every line corresponds to a log record. The fields of the records are separated by ¦ and have the following meaning: 0. time stamp 1. logger name, here the module name 2. record log level 3. name of the function that created the log message 4. line at which the message has been issued 5. actual log message

The same format is used for the file handler. By default QCoDeS logs to the file <user directory>/.qcodes/logs/qcodes.log. To avoid endlessly long files, the files roll over at midnight and get the date appended to their name where qcodes.log without date always gives you the log messages of the same day.

To configure the levels you want to log/handle, you need to edit the qcodesrc.json file in you home directory. The default values are:

"logger":{
    "console_level": "WARNING",
    "file_level": "INFO",
    "logger_levels":{
        "pyvisa": "INFO"
    }

While console_level and file_level describe the levels of the two default handlers described previously, logger_levels is a dictionary that can be used to limit the messages passed on by module level loggers. This means with these defaults, the pyvisa module will not log messages on a DEBUG level. Therefore setting the console_level to DEBUG will not show the pyvisa debug messages (you certainly don’t want to see them as they log every single character that is passed to an instrument with an individual message). On the other hand setting the console level to warning will still suppress pyvisa info messages.

IPython command history

The output above that follows these logging messages comes from IPython. It tells us that all issued IPython commands will be logged to a file. The file lives in the same log directory as the python logs and is called command_history.log. It will not roll over.

To change the command history logging directory you will need to call start_command_history_logger with the new path as an argument.

Temporarily elevating the logging level

Sometimes you might wish to catch more than that meets to the eye. To this end, one can temporarily elevate the logging level to catch all debug messages by using a context manager. This can simply be done via:

[3]:
import logging
import qcodes.logger as logger
log = logging.getLogger('example_logger')

log.debug('This message will not be visible as the logging level is set to `DEBUG`')

with logger.console_level(logging.DEBUG):
    log.debug('This message is visible as the logging level is temporarily elevated to `DEBUG`')

log.debug('The level is back to what it used to be')
2022-07-01 15:01:59,592 ¦ example_logger ¦ DEBUG ¦ 4053955214 ¦ <cell line: 7> ¦ 8 ¦ This message is visible as the logging level is temporarily elevated to `DEBUG`

For other handlers than the console handler, you can do the same thing using the handler_level context manager.

Filtering log messages by instrument

A very common use case for logging in QCoDeS is to analyze the communication with a VISA instrument. For this purpose the logger module provides a context manager to filter the logging messages by instrument.

Generally one could of course simply attach a handler to the logger of qcodes.instrument_drivers.Vendor.Type but this would not show the messages that are generated by the visa communication in qcodes.instrument.visa. Additionally with the approach implemented here one can not only filter messages for a given type of instrument but also of a specific instance.

To demonstrate this consider the following mock up of the AMI430 Magnet controller that consists of individual VISA instruments for each axis, here with the names x, y and z:

[4]:
from qcodes.instrument.ip_to_visa import AMI430_VISA
from qcodes.instrument_drivers.american_magnetics.AMI430 import AMI430_3D, AMI430Warning

import qcodes.instrument.sims as sims

visalib = sims.__file__.replace('__init__.py', 'AMI430.yaml@sim')

mag_x = AMI430_VISA('x', address='GPIB::1::INSTR', visalib=visalib,
                        terminator='\n', port=1)
mag_y = AMI430_VISA('y', address='GPIB::2::INSTR', visalib=visalib,
                        terminator='\n', port=1)
mag_z = AMI430_VISA('z', address='GPIB::3::INSTR', visalib=visalib,
                        terminator='\n', port=1)

import numpy as np
field_limit = [
    lambda x, y, z: x == 0 and y == 0 and z < 3,
    lambda x, y, z: np.linalg.norm([x, y, z]) < 2
]

driver = AMI430_3D("AMI430_3D", mag_x, mag_y, mag_z, field_limit)

Connected to: QCoDeS AMI430_simulation (serial:1337, firmware:0.0.01) in 0.04s
Connected to: QCoDeS AMI430_simulation (serial:1337, firmware:0.0.01) in 0.00s
Connected to: QCoDeS AMI430_simulation (serial:1337, firmware:0.0.01) in 0.00s

In the above messages you can see the prefix [<instrument_name>(<instrument_type>)]. To analyze the visa communication one can use filter_instrument:

[5]:
driver.cartesian((0, 0, 0))
with logger.console_level('DEBUG'):
    with logger.filter_instrument(mag_x):
        driver.cartesian((0, 0, 1))
2022-07-01 15:01:59,672 ¦ qcodes.instrument.instrument_base.com.visa ¦ DEBUG ¦ visa ¦ ask_raw ¦ 224 ¦ [x(AMI430_VISA)] Querying: STATE?
2022-07-01 15:01:59,673 ¦ qcodes.instrument.instrument_base.com.visa ¦ DEBUG ¦ visa ¦ ask_raw ¦ 226 ¦ [x(AMI430_VISA)] Response: 2
2022-07-01 15:01:59,674 ¦ qcodes.instrument.instrument_base.com.visa ¦ DEBUG ¦ visa ¦ ask_raw ¦ 224 ¦ [x(AMI430_VISA)] Querying: FIELD:MAG?
2022-07-01 15:01:59,674 ¦ qcodes.instrument.instrument_base.com.visa ¦ DEBUG ¦ visa ¦ ask_raw ¦ 226 ¦ [x(AMI430_VISA)] Response: 0.0
2022-07-01 15:01:59,675 ¦ qcodes.instrument.instrument_base.com.visa ¦ DEBUG ¦ visa ¦ ask_raw ¦ 224 ¦ [x(AMI430_VISA)] Querying: FIELD:MAG?
2022-07-01 15:01:59,676 ¦ qcodes.instrument.instrument_base.com.visa ¦ DEBUG ¦ visa ¦ ask_raw ¦ 226 ¦ [x(AMI430_VISA)] Response: 0.0

The output on the console shows as expected only messages from the x instrument. For multiple instruments pass a sequence of instruments to filter_instrument we write:

[6]:
driver.cartesian((0, 0, 0))
with logger.console_level('DEBUG'):
    with logger.filter_instrument((mag_x, mag_y)):
        driver.cartesian((0, 0, 1))
2022-07-01 15:01:59,689 ¦ qcodes.instrument.instrument_base.com.visa ¦ DEBUG ¦ visa ¦ ask_raw ¦ 224 ¦ [x(AMI430_VISA)] Querying: STATE?
2022-07-01 15:01:59,690 ¦ qcodes.instrument.instrument_base.com.visa ¦ DEBUG ¦ visa ¦ ask_raw ¦ 226 ¦ [x(AMI430_VISA)] Response: 2
2022-07-01 15:01:59,690 ¦ qcodes.instrument.instrument_base.com.visa ¦ DEBUG ¦ visa ¦ ask_raw ¦ 224 ¦ [y(AMI430_VISA)] Querying: STATE?
2022-07-01 15:01:59,691 ¦ qcodes.instrument.instrument_base.com.visa ¦ DEBUG ¦ visa ¦ ask_raw ¦ 226 ¦ [y(AMI430_VISA)] Response: 2
2022-07-01 15:01:59,692 ¦ qcodes.instrument.instrument_base.com.visa ¦ DEBUG ¦ visa ¦ ask_raw ¦ 224 ¦ [x(AMI430_VISA)] Querying: FIELD:MAG?
2022-07-01 15:01:59,692 ¦ qcodes.instrument.instrument_base.com.visa ¦ DEBUG ¦ visa ¦ ask_raw ¦ 226 ¦ [x(AMI430_VISA)] Response: 0.0
2022-07-01 15:01:59,693 ¦ qcodes.instrument.instrument_base.com.visa ¦ DEBUG ¦ visa ¦ ask_raw ¦ 224 ¦ [y(AMI430_VISA)] Querying: FIELD:MAG?
2022-07-01 15:01:59,693 ¦ qcodes.instrument.instrument_base.com.visa ¦ DEBUG ¦ visa ¦ ask_raw ¦ 226 ¦ [y(AMI430_VISA)] Response: 0.0
2022-07-01 15:01:59,694 ¦ qcodes.instrument.instrument_base.com.visa ¦ DEBUG ¦ visa ¦ ask_raw ¦ 224 ¦ [x(AMI430_VISA)] Querying: FIELD:MAG?
2022-07-01 15:01:59,695 ¦ qcodes.instrument.instrument_base.com.visa ¦ DEBUG ¦ visa ¦ ask_raw ¦ 226 ¦ [x(AMI430_VISA)] Response: 0.0
2022-07-01 15:01:59,695 ¦ qcodes.instrument.instrument_base.com.visa ¦ DEBUG ¦ visa ¦ ask_raw ¦ 224 ¦ [y(AMI430_VISA)] Querying: FIELD:MAG?
2022-07-01 15:01:59,696 ¦ qcodes.instrument.instrument_base.com.visa ¦ DEBUG ¦ visa ¦ ask_raw ¦ 226 ¦ [y(AMI430_VISA)] Response: 0.0

Capturing Pandas.DataFrame

To process the logs, especially with the timestamps, the pandas module is highly suited. With logger.log_to_dataframe or logger.logfile_to_dataframe you can convert a log(file) to a pandas DataFrame. See this (online, offline) notebook for an example.

You can also use a context manager to capture the logs directly into a DataFrame.

[7]:
from qcodes.logger import capture_dataframe
[8]:
with logger.console_level(logging.WARN):
    driver.cartesian((0, 0, 0))
    with capture_dataframe(level='DEBUG') as (handler, get_dataframe):
        driver.cartesian((0, 0, 1))
        df = get_dataframe()
        driver.cartesian((0, 0, 2))
        df2 = get_dataframe() # this is the cummulative log
[9]:
df
[9]:
asctime name levelname module funcName lineno message
0 2022-07-01 15:01:59,719 qcodes.instrument.instrument_base DEBUG AMI430 _adjust_child_instruments 919 [AMI430_3D(AMI430_3D)] Checking whether fields...
1 2022-07-01 15:01:59,720 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 224 [x(AMI430_VISA)] Querying: STATE?
2 2022-07-01 15:01:59,720 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 226 [x(AMI430_VISA)] Response: 2
3 2022-07-01 15:01:59,720 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 224 [y(AMI430_VISA)] Querying: STATE?
4 2022-07-01 15:01:59,720 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 226 [y(AMI430_VISA)] Response: 2
5 2022-07-01 15:01:59,720 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 224 [z(AMI430_VISA)] Querying: STATE?
6 2022-07-01 15:01:59,720 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 226 [z(AMI430_VISA)] Response: 2
7 2022-07-01 15:01:59,720 qcodes.instrument.instrument_base DEBUG AMI430 _adjust_child_instruments 935 [AMI430_3D(AMI430_3D)] Field values OK, procee...
8 2022-07-01 15:01:59,720 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 224 [x(AMI430_VISA)] Querying: FIELD:MAG?
9 2022-07-01 15:01:59,720 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 226 [x(AMI430_VISA)] Response: 0.0
10 2022-07-01 15:01:59,720 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 224 [y(AMI430_VISA)] Querying: FIELD:MAG?
11 2022-07-01 15:01:59,720 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 226 [y(AMI430_VISA)] Response: 0.0
12 2022-07-01 15:01:59,721 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 224 [z(AMI430_VISA)] Querying: FIELD:MAG?
13 2022-07-01 15:01:59,721 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 226 [z(AMI430_VISA)] Response: 0.0
14 2022-07-01 15:01:59,721 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 224 [x(AMI430_VISA)] Querying: FIELD:MAG?
15 2022-07-01 15:01:59,721 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 226 [x(AMI430_VISA)] Response: 0.0
16 2022-07-01 15:01:59,721 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 224 [y(AMI430_VISA)] Querying: FIELD:MAG?
17 2022-07-01 15:01:59,721 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 226 [y(AMI430_VISA)] Response: 0.0
18 2022-07-01 15:01:59,721 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 224 [z(AMI430_VISA)] Querying: FIELD:MAG?
19 2022-07-01 15:01:59,721 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 226 [z(AMI430_VISA)] Response: 0.0
20 2022-07-01 15:01:59,721 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 224 [z(AMI430_VISA)] Querying: COIL?
21 2022-07-01 15:01:59,722 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 226 [z(AMI430_VISA)] Response: 2.0
22 2022-07-01 15:01:59,722 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 224 [z(AMI430_VISA)] Querying: CURR:LIMIT?
23 2022-07-01 15:01:59,722 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 226 [z(AMI430_VISA)] Response: 80
24 2022-07-01 15:01:59,722 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 224 [z(AMI430_VISA)] Querying: QU?
25 2022-07-01 15:01:59,722 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 226 [z(AMI430_VISA)] Response: 0
26 2022-07-01 15:01:59,722 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 224 [z(AMI430_VISA)] Querying: PERS?
27 2022-07-01 15:01:59,722 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 226 [z(AMI430_VISA)] Response: 0
28 2022-07-01 15:01:59,722 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 224 [z(AMI430_VISA)] Querying: STATE?
29 2022-07-01 15:01:59,722 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 226 [z(AMI430_VISA)] Response: 2
30 2022-07-01 15:01:59,722 qcodes.instrument.instrument_base.com.visa DEBUG visa write_raw 210 [z(AMI430_VISA)] Writing: PAUSE
31 2022-07-01 15:01:59,722 qcodes.instrument.instrument_base.com.visa DEBUG visa write_raw 210 [z(AMI430_VISA)] Writing: CONF:FIELD:TARG 1.0
32 2022-07-01 15:01:59,723 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 224 [z(AMI430_VISA)] Querying: PS:INST?
33 2022-07-01 15:01:59,723 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 226 [z(AMI430_VISA)] Response: 0
34 2022-07-01 15:01:59,723 qcodes.instrument.instrument_base.com.visa DEBUG visa write_raw 210 [z(AMI430_VISA)] Writing: RAMP
35 2022-07-01 15:01:59,723 qcodes.instrument.instrument_base DEBUG AMI430 set_field 359 [z(AMI430_VISA)] Starting blocking ramp of z t...
36 2022-07-01 15:01:59,723 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 224 [z(AMI430_VISA)] Querying: STATE?
37 2022-07-01 15:01:59,723 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 226 [z(AMI430_VISA)] Response: 2
38 2022-07-01 15:01:59,723 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 224 [z(AMI430_VISA)] Querying: STATE?
39 2022-07-01 15:01:59,723 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 226 [z(AMI430_VISA)] Response: 2
40 2022-07-01 15:01:59,723 qcodes.instrument.instrument_base DEBUG AMI430 set_field 361 [z(AMI430_VISA)] Finished blocking ramp
[10]:
df2
[10]:
asctime name levelname module funcName lineno message
0 2022-07-01 15:01:59,719 qcodes.instrument.instrument_base DEBUG AMI430 _adjust_child_instruments 919 [AMI430_3D(AMI430_3D)] Checking whether fields...
1 2022-07-01 15:01:59,720 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 224 [x(AMI430_VISA)] Querying: STATE?
2 2022-07-01 15:01:59,720 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 226 [x(AMI430_VISA)] Response: 2
3 2022-07-01 15:01:59,720 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 224 [y(AMI430_VISA)] Querying: STATE?
4 2022-07-01 15:01:59,720 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 226 [y(AMI430_VISA)] Response: 2
... ... ... ... ... ... ... ...
77 2022-07-01 15:01:59,728 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 224 [z(AMI430_VISA)] Querying: STATE?
78 2022-07-01 15:01:59,728 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 226 [z(AMI430_VISA)] Response: 2
79 2022-07-01 15:01:59,728 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 224 [z(AMI430_VISA)] Querying: STATE?
80 2022-07-01 15:01:59,728 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 226 [z(AMI430_VISA)] Response: 2
81 2022-07-01 15:01:59,728 qcodes.instrument.instrument_base DEBUG AMI430 set_field 361 [z(AMI430_VISA)] Finished blocking ramp

82 rows × 7 columns

You can of course combine the context managers:

[11]:
with logger.console_level(logging.WARN):
    driver.cartesian((0, 0, 0))
    with capture_dataframe(level='DEBUG') as (handler, get_dataframe):
        with logger.filter_instrument(mag_x, handler=handler):
            driver.cartesian((0, 0, 1))
            df = get_dataframe()
df
[11]:
asctime name levelname module funcName lineno message
0 2022-07-01 15:01:59,776 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 224 [x(AMI430_VISA)] Querying: STATE?
1 2022-07-01 15:01:59,776 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 226 [x(AMI430_VISA)] Response: 2
2 2022-07-01 15:01:59,776 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 224 [x(AMI430_VISA)] Querying: FIELD:MAG?
3 2022-07-01 15:01:59,776 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 226 [x(AMI430_VISA)] Response: 0.0
4 2022-07-01 15:01:59,777 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 224 [x(AMI430_VISA)] Querying: FIELD:MAG?
5 2022-07-01 15:01:59,777 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 226 [x(AMI430_VISA)] Response: 0.0

For an analysis of the timestamps please also refer to the log analysis example notebook.

[ ]: