Representing data in Python

Better Code, Better Science: Chapter 3, bonus section

This is a possible section from the open-source living textbook Better Code, Better Science, which is being released in sections on Substack. The entire book can be accessed here and the Github repository is here. This material is released under CC-BY-NC-ND.

This week I am posting a number of bonus sections, which are sections that were written after the earlier chapters had been posted.

It’s often useful to bundle together multiple data objects into a single object that can be passed into or returned from a function. A prime example is a results bundle - that is, a group of data objects of varying types that we want to pass together as output from an analysis. For example, results from a linear regression analysis might include the model design matrix, parameter estimates, residuals, a goodness of fit value, and statistical results. It’s common to return these as a dictionary, mapping each result type to a value. However, this is not an optimal way to structure output, since there is no schema that specifies the names and types of the components and no way to easily validate that the structure is correct. Another example is configuration objects that contain multiple config variables. In these kinds of cases it is common to take advantage of Python’s dataclass feature. Dataclasses are real classes that are designed to make it easy to generate objects that are primarily meant as containers for a group of data variables.

Here is an example of a simple dataclass representing a measurement:

from dataclasses import dataclass, field

@dataclass
class Measurement:
    subject_id: str
    value: float
    unit: str = "mV"                               # a default
    tags: list[str] = field(default_factory=list)  # mutable default, done right

    def is_outlier(self, threshold: float) -> bool:  # still an ordinary class
        return abs(self.value) > threshold

A dataclass is a real Python class; what the @dataclass decorator does is to automatically set up several methods: __init__() to initialize the specified variables, __eq__() for equality testing, and __repr__() for printing. Because it’s a real class, we can also add our own methods, like the is_outlier() method seen here. Here is an example of the dataclass in action:

>>> m1 = Measurement("S01", 0.42, tags=["clean"])
>>> m2 = Measurement("S01", 0.42, tags=["clean"])
>>> print("repr: ", m1) 
repr: Measurement(subject_id='S01', value=0.42, unit='mV', tags=['clean'])
>>> print("value equality: ", m1 == m2) 
value equality: True
>>> print("method works: ", m1.is_outlier(0.3))
method works: True

We can also easily serialize this object into a dictionary using dataclasses.asdict:

>>> from dataclasses import asdict
>>> asdict(m1)
{'subject_id': 'S01', 'value': 0.42, 'unit': 'mV', 'tags': ['clean']}

Robust data representation using Pydantic

While the dataclass includes type annotations, it doesn’t actually validate the input, which can allow non-sensical values:

>>> sketchy = Measurement("S02", value="not a number") 
>>> print("unenforced annotation: ", repr(sketchy.value))
unenforced annotation:  'not a number'

If we wanted to validate the dataclass, we could add a __post_init__() method that runs after initialization and checks the values of the variables. However, there is a much easier way to validate a dataclass using the Pydantic package. Pydantic provides a powerful BaseModel that has several very useful features that help make it more robust than a standard dataclass. Here is an example of the earlier class as a Pydantic model:

from pydantic import BaseModel, Field
from typing import List

class Measurement(BaseModel):
    subject_id: str
    value: float = Field(ge=-1000, le=1000)
    unit: str = "mV"
    tags: List[str] = []

We can then create instances of the class, where we will see that an exception is raised if we specify a value outside of the allowable range:

>>> # Valid
>>> m1 = Measurement(subject_id="A001", value=50.0)
>>> print(m1.value)
50.0
>>> m2 = Measurement(subject_id="A001", value=1500.0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    m2 = Measurement(subject_id="A001", value=1500.0)
  File ".venv/lib/python3.13/site-packages/pydantic/main.py", line 250, in __init__
    validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
pydantic_core._pydantic_core.ValidationError: 1 validation error for Measurement
value
  Input should be less than or equal to 1000 [type=less_than_equal, input_value=1500.0, input_type=float]

Another useful feature of Pydantic is type coercion, in which inputs are coerced (if possible) to match the type specification of the class. For example, the value field is specified as a floating point variable, but if we pass in a string representation of a floating point value, it will be coerced into a proper floating point value:

>>> m1 = Measurement(subject_id="A001", value="50.4")
>>> m1.value
50.4
>>> type(m1.value)
<class 'float'>

Anyone who is interested in using dataclasses should consider using Pydantic classes instead, as they provide a much more robust implementation of the same functionality without a lot of overhead or added complexity. Reasons to stick with standard dataclasses over Pydantic classes are performance (dataclasses are faster due to the lack of validation) and simplicity (Pydantic does require some additional knowledge to understand the code).