In many cases we don’t have a closed form solution that we can use to compute the parameter estimates directly. In this case it’s common to use some form of optimization (or search) process to find the parameters that best fit the data. The simplest way to do this is to try a large range of parameter values and choose the one that best fits the sample, which is known as a grid search. This is generally done with the goal of maximizing the likelihood of the data given the model parameters, and hence is called maximum likelihood estimation. In other words, we aim to find the values of the parameters that make the observed data most likely. In practice we would generally use the log of the likelihood rather than the likelihood itself, since these values are often very small which can result in floating point errors.

In the case of the normal distribution the maximum likelihood estimate is equivalent to the estimate that minimizes the squared error, since the sample variance (which is based on the squared error) is part of the likelihood equation. But we can also use this example to see how grid search might work with our sample. I ran a grid search using a grid of 1000 possible mean values linearly spaced across [-1, 1], and 1000 possible standard deviation values spaced across [0.5, 1.5]; these particular values were based on my knowledge that the data came from a normal distribution and that these ranges should be likely to capture the parameter values in a dataset of this size. The results came out very close to those obtained using the closed-form solution; note that the maximum likelihood estimate for the standard deviation is equivalent to the population rather than sample standard deviation (i.e. it uses NN rather than N−1N−1 in its denominator), so I corrected the sample standard deviation to make the comparison fair:

Best fit mean: 0.0190,   Best fit sd: 0.9785, loglik: -1397.4353
Sample mean:   0.0193, Population sd: 0.9787, loglik: -1397.4352

We can see this visualized in Figure 1, where we see the landscape of the likelihood across a range of possible parameter values; here we use the negative log-likelihood for visualization, since optimization methods tend to use the language of minimization rather than maximization. We can see that this landscape is smooth and only has one visible minimum; this occurs because the negative log-likelihood surface for the normal distribution is convex, which guarantees that there is a single minimum and thus that regardless of where we start our search, we are guaranteed to find the global minimum by simply following the surface downward, a process central to many optimization algorithms (including the commonly used gradient descent). As we will see below, most realistic optimization problems have multiple local minima, making them much more difficult to optimize by simply following the surface downward.

Figure 1. A visualization of the negative log-likelihood landscape for a range of parameter values in the grid search for the mean and standard deviation of a normal distribution.

While grid search worked, it was exceedingly slow, taking more than 25 seconds to estimate the parameters that were estimated by closed form in less than a millisecond - a staggering 89,000 times slower! Grid search is inefficient even with just two parameters, and becomes exponentially less efficient with each additional parameter. A more effective and efficient way to estimate parameters is using optimization methods that are specifically built to search for parameter values that minimize a particular loss function. A common choice in Python is scipy.optimize.minimize(), which offers a number of algorithms for parameter search. We can implement this for our normal distribution data; because the function finds the minimum, we will use the negative log-likelihood as our target, which is equivalent to maximizing the log-likelihood:

import time
from scipy.stats import norm

def negative_log_likelihood(params, data):
    """Negative log likelihood function to minimize"""
    mu, sd = params
    # ensure sd is positive to avoid dividing by zero
    if sd <= 0:  
        return np.inf
    return -norm.logpdf(data, loc=mu, scale=sd).sum()

# initial guess
initial_params = [0, 1]

start_time = time.time()
result = minimize(negative_log_likelihood, initial_params, args=(normal_samples,), 
                  method='Nelder-Mead')

This gives us a solution that is equal to the fourth decimal place:

Optimized mean: 0.01926,  Optimized sd: 0.97873, loglik: -1397.43520
Sample mean:    0.01933, Population sd: 0.97873, loglik: -1397.43519

These estimates were obtained about 8,000 times more quickly compared to grid search, though still about 10 times slower than the closed-form solution. Note that I had to add some initial guesses for our parameter values, and for this example I used values that were close to the known true values. However, even when the starting values are far from the true values, optimization can often find them quickly and effectively. For example, setting the starting points for both mean and standard deviation to 10,000, the resulting parameter estimates were basically identical, and it still completed more than 2,700 times faster than the grid search.

It’s common to put boundaries on an optimization when there are bounds outside which we are sure that the parameter shouldn’t go. For example, in our example we know that the standard deviation cannot be negative, so we could set the lower bound on the standard deviation parameter to just above zero:

from scipy.optimize import minimize, Bounds

bounds = Bounds(lb=[-np.inf, 1e-6], ub=[np.inf, np.inf])
result = minimize(negative_log_likelihood, initial_params, args=(normal_samples,), 
                  method='L-BFGS-B', bounds=bounds)

This doesn’t have much impact on this particular problem, but with complex models and multiple parameters it’s common for parameter values to explode, and setting boundaries can help prevent that. However, as I will discuss below, it’s important to ensure that parameter estimates don’t sit at the boundaries, as this can suggest pathologies in model fitting.

Automated differentiation

The optimization methods discussed above are limited either to small numbers of parameters (like derivative-free methods such as Nelder-Mead) or small numbers of data points (like gradient-based methods such as L-BFGS that require computation of gradients across the entire dataset on each optimization step). Given this, how is it possible to train artificial neural networks that may have billions of parameters over trillions of data points? A key innovation that has enabled effective training of large models is automatic differentiation (often called autodiff for short) combined with gradient descent. Automatic differentiation takes a function definition and (when possible) automatically determines the derivatives of the loss function with respect to the parameters. Gradient descent uses those derivatives to follow the loss landscape downwards. In deep learning it’s most common to use stochastic gradient descent (SGD), which uses small mini-batches of data to iteratively estimate the gradients; even though the estimates for each individual batch are noisy, they are unbiased estimates of the true gradient and computationally cheap to obtain, such that the noise averages out over many iterations to give precise parameter estimates at comparatively low computational cost. However, given the small dataset in this sample we will use the simpler standard gradient descent over the entire dataset at once.

As an example, we can estimate parameters for the Michaelis-Menten equation from biochemistry, which describes the rate at which an enzyme converts its substrate into its product:

where V is the reaction velocity, S is the concentration of the enzyme’s substrate, V_max​ is the maximum reaction velocity once the enzyme is saturated with substrate, and K_m​ is the Michaelis constant that describes the affinity of the particular enzyme for its substrate (defined as the value of SS at which V=Vmax/2V=Vmax​/2). Figure 2 shows a plot of this function for the acetylcholinesterase enzyme, along with noisy data generated from the function.

Figure 2. A plot of the Michaelis-Menten function for acetylcholinesterase, along with data sampled from this function with added Gaussian random noise.

This equation could easily be solved using simpler methods, but it’s a nice simple example to show how model parameters can be estimated using autodiff with gradient descent. We can start by defining the Michaelis-Menten function and generating some data with random noise (shown in ; plotting code omitted):

def michaelis_menten(S, V_max, K_m):
    return (V_max * S) / (K_m + S)

V_max_true = 29  # Maximum velocity (in nM/min)
K_m_true = 6     # Michaelis constant (in mM)
noise_sd = 0.5    # Standard deviation of noise

# Generate substrate concentration data points
S = np.linspace(0.1, 30, 100)  

v_true = michaelis_menten(S, V_max_true, K_m_true)
noise = np.random.normal(0, noise_sd, size=v_true.shape)
v_observed = v_true + noise

In order to invoke the automatic differentiation mechanism in PyTorch, we simply need to specify requires_grad=True for the variables that we intend to estimate:

# Convert data to PyTorch tensors
S_tensor = torch.tensor(S, dtype=torch.float32)
v_observed_tensor = torch.tensor(v_observed, dtype=torch.float32)

# specify initial guesses
V_max_init = 10.0
K_m_init = 10.0

# Initialize parameters with random guesses
# requires_grad=True enables automatic differentiation
V_max = torch.tensor(V_max_init, requires_grad=True)  
K_m = torch.tensor(K_m_init, requires_grad=True)

We also need to set up a loss function that will define how far the prediction is from the data, for which we will use the squared error:

def compute_loss(V_max, K_m, S, v_observed):
    """Compute MSE loss between predicted and observed velocities."""
    v_predicted = michaelis_menten(S, V_max, K_m)
    loss = torch.mean((v_predicted - v_observed) ** 2)
    return loss

Using this we set up our training loop that uses gradient descent to estimate the parameters (with some code omitted for clarity), and assess the parameter recovery of the model by comparing the estimates to the true values:

# Hyperparameters
learning_rate = 0.1
n_iterations = 500

# Test the loss with initial parameters
initial_loss = compute_loss(V_max, K_m, S_tensor, v_observed_tensor)
print(f"Initial loss: {initial_loss.item():.4f}")

# Gradient descent training Loop
for i in range(n_iterations):
    # Forward pass: compute loss
    loss = compute_loss(V_max, K_m, S_tensor, v_observed_tensor)
    
    # Backward pass: compute gradients via autodiff
    loss.backward()
    
    # Update parameters using gradient descent step
    # torch.no_grad() prevents these operations from being tracked
    with torch.no_grad():
        V_max -= learning_rate * V_max.grad
        K_m -= learning_rate * K_m.grad
        
        # Zero the gradients for the next iteration
        V_max.grad.zero_()
        K_m.grad.zero_()

print(f"\nFinal estimates: V_max = {V_max.item():.4f}, K_m = {K_m.item():.4f}")
print(f"True values:     V_max = {V_max_true:.4f}, K_m = {K_m_true:.4f}")

Initial loss: 188.0915
Final loss:     0.2006

Final estimates: V_max = 29.0894, K_m = 6.1336
True values:     V_max = 29.0000, K_m = 6.0000

Since there are only two parameters, we can easily visualize how the parameter estimate traverses the loss landscape as the estimation process moves from the initial guesses (in this case 10 for both parameters) to the final values, as shown in Figure 3.

Figure 3. A visualization of the log-loss landscape for the Michaelis-Menten optimization problem, showing the journey of the optimization process from the starting point to the ending point.

Local minima in optimization

The error landscape for the normal distribution example is convex, which means that there is a single global minimum that can be found simply by following the error gradient downwards. Claude Sonnet 4 initially tried to convince me that the Michaelis-Menten problem is convex, but was overruled by Claude Opus 4.5. Despite being non-convex, the error landscape of the Michaelis-Menten problem is smooth and relatively well behaved, as seen in Figure 4. However, many realistic scientific problems have highly complex non-convex likelihoods, such that there are numerous local minima that the optimization routine can get stuck in. shows an example of this.

Figure 4. A visualization of a rough loss landscape. The star shows the global minimum, and the individual trajectories show the local minima that are found when using simple gradient descent from different starting points.

There are a number of strategies that one can employ to help avoid parameter estimates that are far from the optimal answer that is located at global loss minimum:

• Run the estimation algorithm multiple times with different random initializations of the parameters. If they are similar between runs then this gives confidence that the estimates don’t reflect local minima. If the parameter estimates differ yet losses are similar, this suggests that the parameters may be trading off against one another, which reflects a structural problem with the model or data such that there are many equally good points in the loss landscape. This is often referred to as non-identifiability of the parameters, and is sometimes evident in correlations between the different parameter estimates.

• Use an optimizer that adapts the learning rate to the local gradient, such as ADAM or RMSprop.

• Use an optimizer that explores more broadly before converging, such as the differential evolution method implemented in scipy.optimize.differential_evolution.

• It can sometimes be helpful to reparameterize the model to help with convergence. For example, if the models are physically constrained to being positive, then one might consider optimizing the logarithm of the parameter rather than the natural values of the parameters; this allows the optimizer to explore the entire range of large and small numbers while respecting the positivity constraint. If the different parameters are on very different scales this can also cause problems since the optimizer needs to move at different rates in different directions of the loss space, so reparameterizing the model such that parameters are in roughly the same numeric scale can be useful.

In the next post I will discuss another strategy for parameter estimation known as simulation-based inference.

It is very common in science to collect data and then use those data to estimate the parameters for a given model, and it’s important to be able to validate that the estimates are valid. Given the central role of parameter estimation in code testing and validation, I now dive into the various methods that one can use to estimate model parameters, and show examples of how we might validate them. In addition to estimating model parameters, we generally also want some kind of way to quantify the uncertainty in our estimates. That is, rather than thinking of the parameter estimate as a single point value, we can ask: What range of values for the parameter are consistent with the data? This is often expressed using confidence intervals, though I will discuss below the ways that these are often misunderstood.

A central idea in this section will be the notion of parameter recovery: that is, how well can our estimation procedure recover the true parameter values using simulated data? This is particularly important in cases where we don’t have statistical guarantees on the unbiasedness of our estimates. As we will see, simulation provides a powerful tool to assess parameter recovery performance for any model.

Closed-form estimates

In some cases parameter estimates can be obtained using a closed form analytic solution. We will use the normal distribution as an example. This distribution has two parameters: a mean (sometimes called a location) that specifies where the center of the distribution falls, and a standard deviation (sometimes called a scale) that specifies the width of the distribution. The probability function for the normal distribution is:

where μ is the mean and σ is the standard deviation.

Our goal in estimating model parameters is to find estimates (in this case for the mean and standard deviation) that maximize some measure of goodness of fit with respect to the data, or equivalently, minimize some measure of error. Since we don’t want positive and negative errors to cancel each other out, we need a measure of error that is uniquely positive regardless of the direction of the error. The most common measure in statistics is the mean squared error:

where y_i is the value for the i-th observation, \hat{y_​i​} is the estimated value for that observation from the model, and n is the sample size1. In the case of the normal distribution, \hat{y_​i​} is the same for each observation: the mean. We can estimate the mean for the sample using the closed form solution:

​where \bar{y} is the mean. We can then compute the standard deviation using this estimated mean:

Note that this is very similar to the mean squared error, differing in the presence of a square root as well as the use of n−1 rather than n in the demominator. The latter is meant to adjust for the fact that we lost one degree of freedom when we estimated the mean from the same data and then used it to compute the standard deviation. When the variance (the square of the standard deviation) is computed using this correction it will be unbiased, meaning that its expected value will match the true variance of the population. The standard deviation is still slightly biased, but less so than the one computed without the correction.

Figure 1 shows an example of a histogram based on samples from a normal distribution, with the theoretical normal distribution based on the estimated sample mean and standard deviation overlaid. Visually it’s clear that the fitted distribution characterizes the overall shape well, even if it mismatches the shape at finer grain, due to sampling variability.

Figure 1. A histogram of 1000 samples from a standard normal distribution (with mean of zero and standard deviation of one), with the fitted normal distribution overlaid in red.

Quantifying uncertainty in closed-form estimation

In general we want not just a point estimate for our parameter but also an estimate of our uncertainty in that estimate. The confidence interval is the most commonly used method for expressing uncertainty around an estimate, and with closed form expressions it’s often possible to compute the confidence interval directly. A confidence interval is expressed in terms of a percentage, but the meaning of this percentage is often misinterpreted (see the discussion in my Statistical Thinking for more on this). The term “95% confidence interval” seems to imply that it is an interval in which we have 95% confidence that the true value of the parameter falls. However, that violates the frequentist statistical logic that underlies the computation of the confidence interval, which treats the true value as fixed, and thus it either falls in the interval or it doesn’t. Instead, the more appropriate interpretation of a frequentist confidence interval is that it is the interval that would capture the true population mean 95% of the time for samples from the same population. I prefer to frame it in a slightly different, if somewhat less precise way: The confidence interval expresses the range of plausible values for the parameter given our data, and thus tells us something about the precision of our estimate: All else being equal, a sample estimate with a narrower confidence interval is more precise than an estimate with a wider confidence interval.

Using our example from above, we can compute a confidence interval for our estimate of the sample mean. This requires that we have a probability distribution that is associated with our statistic; in this case, the Student’s t distribution is appropriate since we have estimated the standard deviation as well as the mean. The t distribution has slightly wider tails than the normal distribution, which helps account for the added uncertainty in our estimate of the standard deviation. The equation for the confidence interval around the mean using the t distribution is:

where s_y​ is the sample standard deviation, n is the sample size, and t_α/2,n−1​ is the critical value of the t distribution with n−1 degrees of freedom at the α/2 percentile. α defines our confidence level, and it is divided by two since we are interested in both the positive and negative directions. In our case, this results in a confidence interval of [-0.09028, 0.03249]. We can use a simulation to confirm that this interval indeed captures the sample mean 95% of the time for new samples from the same distribution:

# Simulation parameters
n_simulations = 100000
confidence_level = 0.95
alpha = 1 - confidence_level
random_state = 42
true_mean, true_sd = 0, 1
sample_size = 1000

# Track how many times the CI captures the true mean
captures = 0

# Run simulations
for i in range(n_simulations):
    # Draw a new sample from the population
    sample = norm.rvs(loc=true_mean, scale=true_sd, 
        size=sample_size, random_state=random_state)
    
    # Calculate sample statistics
    sample_mean_sim = np.mean(sample)
    sample_sd_sim = np.std(sample, ddof=1)
    
    # Calculate confidence interval
    df = sample_size - 1
    t_crit = t.ppf(1 - alpha/2, df)
    se = sample_sd_sim / np.sqrt(sample_size)
    margin = t_crit * se
    
    ci_low = sample_mean_sim - margin
    ci_high = sample_mean_sim + margin
    
    # Check if CI captures the true mean
    if ci_low <= true_mean <= ci_high:
        captures += 1

# Calculate coverage rate
coverage_rate = captures / n_simulations

print(f"Simulation results:")
print(f"Number of simulations: {n_simulations}")
print(f"Sample size per simulation: {sample_size}")
print(f"True population mean: {true_mean}")
print(f"Confidence level: {confidence_level * 100}%")
print(f"\nCoverage rate: {coverage_rate:.4f} ({coverage_rate * 100:.2f}%)")

Simulation results:
Number of simulations: 100000
Sample size per simulation: 1000
True population mean: 0
Confidence level: 95.0%

Coverage rate: 0.9503 (95.03%)

Here we see that the observed proportion of samples where the sample mean falls within the confidence interval is very close to the 95% that we expect based on statistical theory.

The bootstrap as a general method for quantifying uncertainty

There are often cases where we don’t have a sampling distribution that we can use to form a confidence interval. In these cases, we can use a technique known as the bootstrap. This method takes advantage of resampling, meaning that we repeatedly draw samples with replacement from our full sample. We can do this using the scipy.stats.bootstrap() function, which performs the bootstrap on a sample given any statistical function:

from scipy.stats import bootstrap

# use the bias-corrected/accelerated method ('BCa')
res = bootstrap((normal_samples,), np.mean, confidence_level=0.95,      
    n_resamples=10000, method='BCa', random_state=random_state)

print(f'Bootstrap 95% CI for mean: '
    f'[{res.confidence_interval.low:.5f}, '
    f'{res.confidence_interval.high:.5f}]')
print(f'CI based on t-distribution: [{ci_lower:.5f}, {ci_upper:.5f}]')

Bootstrap 95% CI for mean:  [-0.09104, 0.03078]
CI based on t-distribution: [-0.09028, 0.03249]

Here we see that the bootstrap procedure gives results that are very close to those obtained using the closed form solution, but has the advantage of being usable with nearly any statistic (except for those based on extreme values) regardless of whether or not there is a closed form estimator and/or the sampling distribution is analytically tractable.

Bayesian estimation

I noted above that the interpretation of the frequentist confidence interval is counterintuitive for most people, which leads to common misunderstandings, even among experts (Hoekstra et al., 2014). We would like a way of generating an interval that expresses our confidence about the true parameter value, but we can’t do this in the frequentist framework. However, there is a different approach to statistics that allows us to generate such an interval, known as Bayesian statistics after the Reverend Thomas Bayes whose famous equation forms the basis of this approach.

Bayesian statistics is based on a different conception of probability from the frequentist approach that underlies the standard confidence interval. Under the frequentist conception, probabilities are meant to refer to the long-run frequencies of outcomes across many samples, while the true parameter value is viewed as fixed. For this reason, it doesn’t make sense to a frequentist to say that there is a particular probability of the true parameter value; it simply is what it is. Bayesians, on the other hand, view probabilities as degrees of belief, and treat the estimation of parameters from data as a way to sharpen our belief - that is, as a learning opportunity. This means that it is perfectly legitimate in the Bayesian framework to say that there is a 95% probability that the true value of a parameter lies within a particular interval.

The fundamental idea in Bayesian statistics is that we start with a set of beliefs (known as a prior distribution), we obtain some relevant data, and then use the likelihood of those data given the possible parameter values to update our beliefs, generating a posterior distribution. I won’t go into detail about Bayesian methods here; see my Statistical Thinking for a basic overview, and Gelman et al. (2013) or McElreath (2020) for more detailed overviews. Instead I will show an example of Bayesian estimation applied to our example data above. There are several Python packages that can be used to perform Bayesian estimation; I will use the popular PyMC package. The first section sets up the Bayesian model, with priors for the mean (mu) and standard deviation (sigma) that are very broad and thus will have little influence on the outcome; in Bayesian terms these are referred to as weakly informative priors. We then perform sampling to obtain an estimate of the posterior distribution of the parameters given the data. Using these distributions, we can then find the narrowest set of values that contain 95% of the mass of posterior distribution, which are known as the highest density interval (HDI) (which is a type of credible interval that contains the most likely values). This interval serves as a Bayesian alternative to the frequentist confidence interval, allowing us to legitimately describe it as the interval that has a 95% probability of containing the true value.

import pymc as pm
import arviz as az

# Bayesian estimation using PyMC
with pm.Model() as model:
    # Priors for unknown model parameters
    mu = pm.Normal('mu', mu=0, sigma=1000)  # Prior for mean
    sigma = pm.HalfNormal('sigma', sigma=100)  # Prior for standard deviation (must be positive)
    
    # Likelihood (sampling distribution) of observations
    likelihood = pm.Normal('likelihood', mu=mu, sigma=sigma, observed=normal_samples)
    
    # Posterior sampling
    trace = pm.sample(10000, tune=1000, return_inferencedata=True, random_seed=42)

# Extract posterior estimates
posterior_mean = trace.posterior['mu'].mean().values
posterior_sd = trace.posterior['sigma'].mean().values

# extract highest density interval
hdi = az.hdi(trace, hdi_prob=0.95)
hdi_values = hdi.mu.values

print(f"Posterior mean: {posterior_mean:.5f}, Posterior sd: {posterior_sd:.5f}")
print(f"Sample mean: {sample_mean:.5f}, Sample sd: {sample_sd:.5f}")
print(f'95% HDI values: {hdi_values}')
print(f'95% CI based on t-distribution: [{ci_lower:.5f}, {ci_upper:.5f}]')

Posterior mean: -0.02895, Posterior sd: 0.99033
Sample mean: -0.02889, Sample sd: 0.98922
95% HDI values:                 [-0.08993, 0.03335]
95% CI based on t-distribution: [-0.09028, 0.03249]

In this case, the Bayesian HDI turns out to be very close to the parametric confidence interval. We can also obtain a visualization of the full posterior distributions obtained through Bayesian estimation, which are shown in :

# Visualize posterior distributions
fig, axes = plt.subplots(1, 2, figsize=(12, 4))

# Plot posterior for mu
az.plot_posterior(trace, var_names=['mu'], ax=axes[0])
axes[0].axvline(sample_mean, color='red', linestyle='--', label='Sample mean')
axes[0].legend()

# Plot posterior for sigma
az.plot_posterior(trace, var_names=['sigma'], ax=axes[1])
axes[1].axvline(sample_sd, color='red', linestyle='--', label='Population sd')
axes[1].legend()

Figure 2. Posterior distributions for mean (mu) and standard deviation (sigma) obtained using Bayesian estimation, with the 95% highest density interval shown by the gray bar at the base of the plot.

Bayesian estimation can be particularly useful when one has a strong prior belief about the value of a parameter and wishes to update that belief based on data. For example, let’s say that there was a published dataset that reported a particular parameter value, and a researcher performs additional observations and wants to update that parameter estimate. Bayesian estimation allows this by the specification of the prior probability distribution. In the example above we used a relatively non-informative prior for the mean (a normal distribution with mean of zero and standard deviation of 1000, which allows for a very wide set of possibilities). However, if we have existing data then we can use those data to inform our subsequent analyses, consistent with the idea that Bayesian inference is a form of learning from data. One can also provide a prior based on one’s scientific hypotheses or expectations, and the ability to incorporate prior knowledge into parameter estimation is generally taken as a strength of Bayesian methods; however, one must be sure that the prior doesn’t overwhelm the data dogmatically, effectively forcing a particular answer regardless of what the data say.

One drawback of Bayesian methods is that they can be very computationally expensive. For example, the Bayesian estimation above took a bit over 2 seconds using 4 parallel sampling processes, which is much slower than the 189 microseconds required for closed-form estimation and also substantially slower than the optimization methods discussed in the next section. There are alternative Bayesian methods known as variational Bayes that use mathematical tricks to speed up estimation, but often require substantial mathematical skill to develop, though some packages like PyMC now offer built-in variational Bayes methods.

In the next post I will turn to parameter estimation using optimization methods.

In this post I will continue the discussion of simulation, focusing on how to generate simulated data from a mathematical model or existing data.

Simulating data from a model

In some cases, we want to simulate data that have particular structure in order to test whether our code can properly identify the structure in the data. Depending on the kind of structure one needs to create, there are often existing tools that can help generate the data. For example, the scikit-learn package has a large number of data generators that are often useful, either on their own or as a starting point to develop a custom generator. Similarly, the NetworkX graph analysis package has a large number of graph generators available.

Let’s say that we have developed a tool that implements a novel method for the discovery of causal relationships from timeseries data. We would like to generate data from a known causal graph (which is represented as a directed acyclic graph, just like our workflow graphs in the previous chapter). For this, we can use an existing graph; I chose one based on a dataset of gene expression in E. coli bacteria that was used by Schafer & Strimmer (2005) and is shared via the pgmpy package:

from IPython.display import Image
from pgmpy.utils import get_example_model

# Load the model
ecoli_model = get_example_model('ecoli70')

# Visualize the network and save to an image file
viz = ecoli_model.to_graphviz()
viz.draw(IMAGE_DIR / 'ecoli.png', prog='dot')

Figure 1 shows the resulting rendering of that network, which has 46 nodes (representing individual genes) and 70 directed edges (representing causal relationships on gene expression between nodes).

Figure 1. A plot of the graphical model for the E. Coli gene expression data generated by Schafer & Strimmer, 2005.

Given this DAG, we then need to generate timeseries data for expression of each gene that reflect the causal relationships between the genes as well as the autocorrelation in gene expression within genes measured over time. For this, we turn to the tigramite package, which is primarily focused on causal discovery from timeseries data, but also includes a function that can generate timeseries data given a graphical model. However, the tigramite package requires a different representation of the graphical model than the one obtained from pgmpy, so we have to convert the edge representation from the original to the link format required for tigramite:

def generate_links_from_pgmpy_model(model, coef=0.5, ar_param=0.6):
    nodes, edges = model.nodes(), model.edges()
    noise_func = lambda x: x 
    links = {}

    # create dicts mapping node names to indices and vice versa
    node_to_index = {node: idx for idx, node in enumerate(nodes)}
    index_to_node = {idx: node for node, idx in node_to_index.items()}

    # add edges from the pgmpy model
    for edge in edges:
        cause = node_to_index[edge[0]]
        effect = node_to_index[edge[1]]
        # for simplicity, use lag 1, constant coef and no edge noise
        links.setdefault(effect, []).append( ((cause, -1), coef, noise_func) )

    # add a self-connection to all nodes to simulate autoregressive behavior
    for node in nodes:
        idx = node_to_index[node]
        links.setdefault(idx, []).append( ((idx, -1), ar_param, noise_func) )

    return links, node_to_index, index_to_node

We can then create a function to take in the original model, convert it, and generate timeseries data for the model:

def generate_data(model, noise_sd=1, tslength=500, seed=42, coef=0.5, ar_param=0.6):
    links, node_to_index, index_to_node = generate_links_from_pgmpy_model(model, 
        coef=coef, ar_param=ar_param)
    rng = np.random.default_rng(seed)
    # Calculate total length including transient period
    data, _ = structural_causal_process(links, T=tslength, seed=seed)
    data = rng.normal(scale=noise_sd, size=data.shape) + data
    # Prepare data for tigramite
    return DataFrame(data), index_to_node

# we will need the index_to_node mapping later
ecoli_dataframe, _, index_to_node = generate_data(ecoli_model, noise_sd=1, 
    tslength=500, seed=42)

Now that we have the dataset we can test out our estimation method. Since I don’t actually have a new method for causal estimation on timeseries, I will instead use the PCMCI method described by Runge et al, 2019 and implemented in the tigramite package:

from tigramite.pcmci import PCMCI
from tigramite.independence_tests.parcorr import ParCorr

def run_pcmci(dataframe):
    # Initialize PCMCI with partial correlation-based independence test
    pcmci = PCMCI(dataframe=dataframe, cond_ind_test=ParCorr())
    # Run PCMCI to discover causal links
    results = pcmci.run_pcmci(tau_max=1, pc_alpha=None)
    return results

results = run_pcmci(ecoli_dataframe)

The results from this analysis include a list of all of the edges that were identified from the data using causal discovery, which we can summarize to determine how well the model performed. First we need to extract the links that were discovered from the results which pass our intended false discovery rate threshold:

def extract_discovered_links(results, index_to_node, q_thresh=0.00001):
    discovered_links = []
    fdr_p = results['fdr_p_matrix'][:, :, 1]  # use only lag 1 p-values
    links = np.where(fdr_p < q_thresh)
    for (i, j) in zip(links[0], links[1]):
        if not i == j:
            discovered_links.append((index_to_node[i], index_to_node[j]))
    return discovered_links

discovered_links = extract_discovered_links(results, index_to_node, .01)

Then we can summarize the results:

def get_edge_stats(edges, discovered_links, verbose=True):
    true_edges = set(edges)
    discovered_edges = set(discovered_links)
    true_positives = true_edges.intersection(discovered_edges)
    false_positives = discovered_edges.difference(true_edges)
    false_negatives = true_edges.difference(discovered_edges)

    true_positive_rate = len(true_positives) / len(true_edges) if len(true_edges) > 0 else 0
    
    # Precision: proportion of discoveries that are true
    precision = len(true_positives) / len(discovered_edges) if len(discovered_edges) > 0 else 0
    
    # False Discovery Rate: proportion of discoveries that are false
    false_discovery_rate = len(false_positives) / len(discovered_edges) if len(discovered_edges) > 0 else np.nan
    
    f1_score = (2 * len(true_positives)) / (2 * len(true_positives) + \
        len(false_positives) + len(false_negatives)) if (len(true_positives) + len(false_positives) + len(false_negatives)) > 0 else np.nan
    
    if verbose:
        print(f'{len(true_edges)} true edges')
        print(f'discovered {len(discovered_edges)} edges')
        print(f"True Positive Rate (Recall): {true_positive_rate:.2%}")
        print(f"Precision: {precision:.2%}")
        print(f"False Discovery Rate: {false_discovery_rate:.2%}")
        print(f"F1 Score: {f1_score:.2%}")

    return {
        'true_positives': true_positives,
        'false_positives': false_positives,
        'false_negatives': false_negatives,
        'true_positive_rate': true_positive_rate,
        'precision': precision,
        'false_discovery_rate': false_discovery_rate,
        'f1_score': f1_score
    }

edge_stats = get_edge_stats(ecoli_model.edges(), discovered_links)

70 true edges
discovered 87 edges
True Positive Rate (Recall): 100.00%
Precision: 80.46%
False Discovery Rate: 19.54%
F1 Score: 89.17%

The results showed that the model performed quite well, detecting all of the true relationships and only two false relationships. In general we would want to do additional validation to make sure that the results behave in the way that we expect. For example, we would expect better model performance with stronger signal, and we would expect fewer nodes identified when the p-value threshold is more stringent. We can use the functions generated above to run a simulation of this:

# loop over signal levels and q values to see effect on performance

noise_sd = 1
tslength = 500
q_values =  [1e-6, 1e-5, 1e-4, 1e-3, 1e-2]
signal_levels = np.arange(0, 0.7, 0.1)
performance_results = []

for signal_level in signal_levels:
    dataframe, index_to_node = generate_data(ecoli_model, noise_sd=noise_sd, tslength=tslength, seed=42, coef=signal_level, ar_param=0.6)
    results = run_pcmci(dataframe)
    for q in q_values:
        discovered_links = extract_discovered_links(results, index_to_node, q_thresh=q)
        edge_stats = get_edge_stats(ecoli_model.edges(), discovered_links, verbose=False)
        performance_results.append({
        'noise_sd': noise_sd,
        'q_value': q,
        'tslength': tslength,
        'signal_level': signal_level,
        'true_positive_rate': edge_stats['true_positive_rate'],
        'precision': edge_stats['precision'],
        'false_discovery_rate': edge_stats['false_discovery_rate'],
        'f1_score': edge_stats['f1_score']
    })

performance_df = pd.DataFrame(performance_results)

We can then plot these results, as shown in Figure 2. The results confirm that the model is performing as expected, with increasing recall as a function of increasing true signal and decreasing FDR threshold.

Figure 2. A plot of observed true positive rate (TPR) and false discovery rate (FDR) at increasing signal levels for varying FDR thresholds.

Simulating data based on existing data

It’s very common for researchers to collect a dataset of interest and then develop code that implements their analysis on that dataset to ask their questions of interest. However, this approach raises a concern that the choices made in the course of analysis might be biased by the specific features of the dataset (Gelman & Loken, 2019). In particular, decisions might be made that reflect the noise in the dataset, rather than the true signal, which is often referred to as overfitting (discussed further below). In some fields (particularly in physics) it is common to perform blind analysis (MacCoun & Perlmutter, 2015), in which analysts are given data that are either modified or relabeled, in order to prevent them from being biased by their hypotheses. One way to achieve this in the context of data analysis is to develop the code using a simulated dataset that has some of the same features as the real dataset, such that one can implement the code, validate it, and then immediately apply it to the real data once they are made available. To achieve this, one needs to be able to generate simulated data based on an existing dataset; for blind analysis, the generation of the simulated data should be performed by a different member of the research team. For example, in some cases I have generated the simulated data for a study based on the real data and provided those to my students, only providing them with the real data once the code was implemented and validated.

The important question in generating simulated data from real data is what specific features one intends to capture from the real data. This generally will require some degree of domain expertise in order to understand the features of the data. Some common features that one might wish to replicate are:

• Data types (e.g. categorical, integer, floating point)

• Marginal distributions of the values (minimally the range, preferably the shape or summary statistics)

• Joint distributions of the variables (e.g. capturing correlations between variables)

It’s generally important to avoid including features in the model that are directly relevant to the hypothesis. For example, if the hypothesis relates to correlations between specific variables in the dataset, then the correlation in the simulated data should not be based on the correlation in the real data, lest the analysis be biased.

Here I will focus primarily on tabular data; while there are simulators to generate more complex types of data, such as genome wide association data and functional magnetic resonance imaging data, these require substantial domain expertise to use properly, whereas tabular data are widely applicable. For simple datasets it may be most appropriate to generate simulated data by hand; here I will use the Synthetic Data Vault (SDV) Python package, which has powerful tools for generating many kinds of synthetic data.

As an example, I will use the Eisenberg et al. (2018) data that you have already seen on a couple of occasions. I’ll start by picking out a few variables and then using SDV to create a synthetic dataset whose distributions for each variable match those in the original, but the columns are generated independently, which removes any correlations between columns. The full analysis is shown here. After loading and combining the demographic and behavioral data frames, selecting a few important variables, and joining them into a single frame (df_orig), I then use SDV to generate simulated data for each variable, shuffling each column after generation to destroy any correlations:

from sdv.single_table import GaussianCopulaSynthesizer
from sdv.metadata import Metadata

def generate_independent_synthetic_data(df, random_seed=42):
    """
    Generate synthetic data where all variables are independent.
    
    Uses SDV to model the full dataset, then shuffles each column 
    independently to break all correlations while preserving marginal distributions.
    
    Parameters:
    -----------
    df : pd.DataFrame
        Original dataframe to generate synthetic version of
    random_seed : int, optional
        Random seed for reproducibility (default: 42)
        
    Returns:
    --------
    pd.DataFrame
        Synthetic dataframe with same shape and column names as input,
        but with independent variables
    """
    # Suppress the metadata saving warning
    warnings.filterwarnings('ignore', message='We strongly recommend saving the metadata')
    
    # Set random seed
    if random_seed is not None:
        np.random.seed(random_seed)
    
    # Create metadata for the full dataset
    metadata = Metadata.detect_from_dataframe(
        data=df,
        table_name='full_data'
    )
    
    # Create synthesizer for the full dataset
    synthesizer = GaussianCopulaSynthesizer(
        metadata,
        enforce_rounding=False,
        enforce_min_max_values=True,
        default_distribution='norm'
    )
    
    # Fit synthesizer to the full dataset
    synthesizer.fit(df)
    
    # Generate synthetic data
    df_synthetic = synthesizer.sample(num_rows=len(df))
    
    # CRITICAL: Shuffle each column independently to break all correlations
    # This preserves the marginal distribution of each variable but eliminates dependencies
    for col in df_synthetic.columns:
        shuffled_values = df_synthetic[col].values.copy()
        np.random.shuffle(shuffled_values)
        df_synthetic[col] = shuffled_values
    
    return df_synthetic

We can then visualize the correlations and distributions for the original data and the synthetic data; in Figure 3 we see that the distributions in the synthetic data are very similar to those in the original data, while in Figure 4 we see that the synthetic data do not include the original correlations.

Figure 3. A comparison of the distributions of the original and synthetic data for several of the variables in the example dataset.

Figure 4. A comparison of the correlations matrices for the numeric variables in the original and synthetic data.

The SDV package also offers many additional tools for more sophisticated generation of synthetic data. In subsequent sections I will show additional ways to use synthetic data for validation of scientific data analysis code.

So far I have focused very heavily on reproducibility, that is, the ability to generate the same answer when code is run repeatedly. However, it’s easy to reliably generate the wrong answer! In measurement theory there is a fundamental distinction between reliability and validity: Reliability means performing the same method repeatedly results in highly similar results, whereas validity refers to whether the estimated result is close to the true result. In this chapter we turn to the validation of scientific software, by which I mean the degree to which it performs the intended task as expected and gets the answers right.

Creating simulations

Creating simulations is perhaps the most important tool that computers offer the scientist, as captured in a well-worn quote by Press et al. in their Numerical Recipes book:

offered the choice between mastery of a five-foot shelf of analytical statistics books and middling ability at performing statistical Monte Carlo simulations, we would surely choose to have the latter skill. (p.691)

Simulations are indeed a powerful way to understand a system even when it’s not analytically tractable. More importantly, they are generally the only way that we can establish ground truth against which we can compare our models. As scientists we never know the true process that generates our data, but with simulations we can have complete control over the data generation process.

My previous book, Statistical Thinking gives an overview of how to use simulations in the context of statistics; here I will focus primarily on the use of simulation in the context of software validation, but I recommend that book for background reading if you aren’t already familiar with the concept of a statistical distribution.

Generating random numbers

The most fundamental requirement in nearly any simulation is the ability to generate random numbers.1 What makes a series of numbers random is that it is impossible (or at least nearly impossible) to predict the next value in the series. Random numbers are defined by the distribution that characterizes them, which is a mathematical function that describes the “shape” of the data when they are summarized according to the relative frequency of different values or ranges of values. Picking the correct distribution is essential to ensure that any simulation performs as advertised. Fortunately, there are lots of existing packages that provide tools to generate random numbers for nearly any distribution; we will focus on the NumPy package here since it is the most commonly used.

The simplest distribution is the uniform distribution, in which any possible value (within a particular range for continuous values) has the same probability of occurring. We can generate uniform random variates by first creating a random number generator object using np.random.default_rng(), and then calling rng.uniform() which returns random samples from the distribution:

rng = np.random.default_rng()
rng.uniform(size=10)

array([0.56449692, 0.6880841 , 0.43249236, 0.28950554, 0.02708363,
       0.61239335, 0.30663968, 0.3854357 , 0.57454511, 0.07974661])

In this case, rng.uniform() by default generates floating point values that fall within [0, 1]; this can be changed using the location and scale parameters to the function. If we generate a large number of these then we can create a distribution plot (often called a histogram) showing how the numbers are distributed, as shown in Figure 1.

Figure 1. Distribution plots for 1,000,000 random samples from each of six different distributions.

For purposes of reproducibility it’s often useful to be able to regenerate exactly the same series of random samples. We can do this by specifying a random seed, which gives the random number generating a starting point. If you are going to do simulations then it is important to understand the specific random number generator that your code will use. This blog post provides an excellent introduction to the NumPy random generation system; here I will only give a brief overview. Previously it was common to use the global NumPy random seed function (np.random.seed()) to set the seed, and this is still necessary when using packages that access the global random number generator. However, the best practice is to generate a random number generator object (using np.random.default_rng()), and then call the methods of that object to obtain random numbers, as I did above. This prevents surprises in case other functions modify the global seed, helps isolate your specific generator, and enables multiple parallel generators. Here is an example:

rng = np.random.default_rng(seed=42)
rng.uniform(size=4)

array([0.77395605, 0.43887844, 0.85859792, 0.69736803])

If we run this again, we see that a different series of numbers is generated:

rng.uniform(size=4)

array([0.09417735, 0.97562235, 0.7611397 , 0.78606431])

However, if we generate another object with the same seed, we will see that it gives us the same values as above:

rng2 = np.random.default_rng(seed=42)
rng2.uniform(size=4)

array([0.77395605, 0.43887844, 0.85859792, 0.69736803])

rng2.uniform(size=4)

array([0.09417735, 0.97562235, 0.7611397 , 0.78606431])

Setting random seeds is important to enable exact reproducibility of results generated using random numbers. However, it’s also important to ensure that one’s results are robust to the choice of random seed, as I will discuss later in the context of machine learning analyses.

Choosing a distribution

Choosing the right distribution for a simulation often comes down to understanding the data-generating process and kind of data that are being modeled. Here are a few examples of distributions and their common use cases:

Discrete outcomes

• Bernoulli: A distribution of binary outcomes (often interpreted as success vs failure) given a probability of a positive outcome.

  • Examples: Whether a patient responds to a given treatment, whether a hard drive fails within a particular period of time.

• Binomial: A distribution of the number of successes across a specific number of Bernoulli trials, given a probability of a positive outcome.

  • Examples: The number of patients who respond to treatment in a clinical trial treatment group, the number of hard drives that fail within a particular period of time at a particular data center

• Categorical: A distribution with several distinct possible outcomes, each of which has a particular probability:

  • Examples: Eye color across a population, programming languages used by programmers in a company

• Uniform: A specific form of a discrete categorical variable with equal probability

  • Examples: equiprobable physical outcomes such as a dice roll.

• Multinomial: A multivariate generalization of the binomial, representing the counts of multiple possible outcomes across a set of independent trials with fixed probabilities of each outcome.

  • Examples: Frequencies of types of stars in a galaxy, frequencies of cell types in a tissue sample

Continuous outcomes:

• Uniform: A distribution with equal probability density for all values within the range.

  • Examples: probabilities of events when there is no prior knowledge, equiprobable continuous physical outcomes

• Beta: A generalization of the uniform distribution that models the probability of values within a range but allows different values to have different probabilities.

  • Examples: Prior probabilities in a Bayesian model, proportions of time spent in a particular state.

• Normal (or Gaussian): A symmetric distribution centered around a mean, which commonly arises when an outcome is generated based on many small additive contributions. The Central Limit Theorem explains why this occurs so frequently, as it should arise for sums of independent random variables sampled from any distribution, as long as the sample size is large enough and the distribution has finite variance.

  • Examples: Height of individuals in a population, measurement errors for continuous variables

• Log-normal: Distribution of positive continuous values whose logarithm is normally distributed. It reflects the expected values of a product of independent random variables.

  • Examples: Wealth and income distributions in a population, biological growth processes

Count data:

• Poisson: This is a distribution of counts of events within a fixed interval assuming that the events are independent and occur at a constant rate. Unlike the binomial there is no limit on the number of events that can occur, and it is the limiting case of the Binomial when the sample size nn approaches infinity and the probability pp approaches zero (and n∗pn∗p remains constant).

  • Examples: The number of atoms decaying within a particular period, the number of emails that a person receives within a day

• Negative binomial: A distribution that models count data that are overdispersed, meaning that their variance is greater than their mean. It can also be interpreted as representing the number of failures that occur before a given number of successes.

  • Example: Commonly used in genomics to model read counts in genomic sequencing.

Waiting time data:

• Exponential: A distribution of waiting times in a process governed by a Poisson distribution, with a constant hazard rate (i.e. the probability of happening in the next period is independent of whether the event has happened yet).

  • Examples: Time between atomic decays, time between hard drive failures

• Weibull: A generalization of the exponential distribution that allows modeling of waiting times with increasing, constant, or decreasing hazard rates.

  • Examples: Response times in human behavior, time to failure for some electronic devices

It is essential to choose the right distributions for a simulation; otherwise the results may be misleading at best or meaningless at worst.

In the next post I will discuss how to simulate data from a mathematical model,

The conference

I was in Princeton, NJ for the annual meeting of the Society of Experimental Psychologists, of which I am a member. It’s a very broad meeting where members can come and talk about whatever they are interested in, and I had signed up to give a talk about causal inference in the context of fMRI statistical modeling. The talks are only 15 minutes long, which is a very difficult amount of time to talk for because it’s hard to flesh out an idea in that little amount of time. I was scheduled to talk in the afternoon of Day 2 of the meeting, so I spent Monday just listening to talks.

One of the talks that piqued my interest was by John Hummel, who is currently a faculty member at UIUC who I have known since our days on the faculty together at UCLA. John was talking about a paper that he and Rachel Heaton published last year, which focused on requirements for a computational model to perform symbolic processing. They make the following claim:

We propose that two kinds of hierarchical integration—integration of multiple role bindings into multiplace predicates, and integration of multiple correspondences into structure mappings—are minimal requirements, on top of basic dynamic binding, to realize symbolic thought. We tested this hypothesis in a systematic collection of 17 simulations that explored the ability of cognitive architectures with and without the capacity for multiplace predicates and structure mapping to perform various kinds of tasks. The simulations were as generic as possible, in that no task could be performed based on any diagnostic features, depending instead on the capacity for multiplace predicates and structure mapping. The results are consistent with the hypothesis that, along with dynamic binding, multiplace predicates and structure mapping are minimal requirements for basic symbolic thought. These results inform our understanding of how human brains give rise to symbolic thought and speak to the differences between biological intelligence, which tends to generalize broadly from very few training examples, and modern approaches to machine learning, which typically require millions or billions of training examples. The results we report also have important implications for bioinspired artificial intelligence.

John made very clear in his talk that he thought that these results demonstrated that current LLMs will never be able to perform true relational reasoning, because they don’t have the necessary representational and/or processing apparatus. In the Q&A period, I suggested that we sit down with Claude Opus 4.7 (which hereafter I will just call “Opus”) to see if his claim stands up to frontier AI models. My feeling is that it’s generally a losing proposition to make impossibility claims about future AI models, and I wanted to see how close the current models were to achieving his supposedly impossible ability.

When I got back from the conference in the afternoon, I decided to play around with trying to test the model on my own before dinner. I first ran the paper through the Opus chatbot with the following prompt:

The author of the attached paper claims that there is a set of tasks that LLMs cannot perform because they do not contain the basic elements for symbolic thought: dynamic binding, multiplace predicates, and structure mapping. Please review the paper and tell me whether you think this is a reasonable argument.

Claude came back with extensive comments, which you can see in their entirety here - they started with this very Claude-like intro:

This is a thoughtful paper, and the authors are making a more careful argument than a quick read might suggest. Let me separate what I think is right, what I think is questionable, and what remains genuinely open.

I then asked it to help me build some example problems to test reasoning in a way that avoids training data contamination:

can you develop a novel version of their task that I could test out on an LLM but that they could not legitimately claim would be in the models’ training diet?

It provided a set of guidelines for developing the problems, as well as suggesting a set of ways to make the problems even more challenging for the LLM. I placed its suggestions into a markdown file and put it in a new directory along with the original paper, and then fired up Claude Code and issued the following prompt:

I would like to generate a set of benchmark problems for relational reasoning. In another session I had claude read the paper and lay out some versions of the problem to develop; that transcript is in PROBLEM.md, and the paper is available at hummel_heaton.pdf. I would like to create five instances of the each of the variants that are described in the PROBLEM.md file, and then build a harness to run them on each version of claude (opus, sonnet, and haiku) and quantify their performance. please develop a plan

Within about 45 minutes I had a working script that could run the problems against each of the models, which I started running and then left for dinner. (See the full codebase here).

The analyses

Over dinner I told my colleagues about my ongoing analyses, and one of them said to me “oh, you have to talk about this tomorrow!”. So when I got home from dinner around 10:30 pm, I was anxious to see what the results looked like to determine whether it would make sense to create a new talk. Here is what I saw:

The numbers refer to the proportion of correct outputs, meaning that all of the models had scored 100% on the problems, which clearly suggested that something was wrong. So I fired up Claude Cowork in the code directory and issued the following prompt:

this directory is for a project that is trying to test ideas presented in a paper by hummel, which can be found at hummel_heaton.pdf. I asked claude code to create a set of tests to check whether LLMs have the relational reasoning capacity that Hummel claims they should not have - this is described in PROBLEMS.md. I ran the tests on LLMS including Haiku, Sonnet, and Opus, and all of them were able to perform the task perfectly. This has me worried that there might be somethign subtly wrong with the examples that is making it too easy for the models. can you please examine the problems (in problems/) and identify whether there are any issues that might be allowing the LLM to solve the problem without true relational reasoning?

Its judgment was swift:

I’ve examined the problems carefully and found several serious shortcuts that let LLMs solve the task without any relational reasoning. The short version: the correct answer is uniquely identifiable by its button configuration alone. Here are the specific issues, with evidence from the generator code.

It laid out a number of issues; see here for the full transcript. I then asked Cowork to fix the problems, which it did fairly quickly, also adding a control condition to provide a negative control in order to ensure that there wasn’t a scoring problem. I also decided to add the ability to run the prompts against a local open-weight LLM using my Mac, in order to see how a very small model would do (in this case, Gemma 3n), mostly because I wanted to include a model that I was pretty sure should fail. I set this running and went to bed. At this point I had decided that if there were usable results that I would create a new talk the next morning, which made it a bit difficult to sleep, so about 30 minutes later I got up and checked the results, seeing this:

The fact that performance scaled with model capacity suggested to me that these were plausible results. I also examined the error patterns, which showed the expected prevalence of feature match errors for the weaker models:

At this point I knew there was something to talk about, so I headed back to bed knowing that I had to get up early the next morning and create a new talk. After a few hours of fitful sleep, I was up the next morning to start working on the talk. I developed some introductory slides based on the results so far, and then headed over the conference where I sat in the back of the room working on further analyses. During this session I noticed something weird - the cross-domain results were roughly the same as these other problems, when it should be a much harder problem. When I looked at one of the cross domain problems I discovered that Claude had made an error and that it wasn’t actually cross-domain (i.e. it used the same domain for both scenarios), so I asked Claude to examine the problems:

can you please examine the cross-domain problems? they don’t actually seem to be cross-domain to me

It diagnosed the problem:

You’re right — these aren’t cross-domain. Each problem uses one domain for both scenarios (org_chart ↔ org_chart, garden ↔ garden, etc.). The only thing “crossing” is the surface vocabulary relative to the default buttons-on-objects setup in baseline. Within a given problem, memory and perception share the same ontology, the same relation names, and the same feature noun.

I asked it to fix the problem, which it did - in addition, it generated a set of tests to ensure that the problem were really testing relational knowledge:

Let me do a rigorous sanity check: write three solvers (structural, feature-matching, semantic-alignment) and check their accuracy on the new cross-domain set

Excellent! Exactly the pattern we want:

• FEATURE_MATCH scores 0/5 on every variant (feature-twin shortcut blocked).

• STRUCTURAL scores 5/5 on every variant that has a structural answer (all except control).

• SEMANTIC_ALIGN scores 0/5 on cross_domain — the newly-added semantic-alignment shortcut is defeated.

• On control, the feature-match solver reliably picks the twin (5/5), confirming control’s diagnostic property.

I ran the new code, and the results were striking:

All of the models were at floor on the cross_domain problem (shown in green)! I added this figure into my talk, and moved on to creating some more slides. I wanted to provide a graphical representation of one of the problems, so I asked claude to create a visualization, which it did. Here is the cross-modal problem that was posed to the LLM:

I'm going to describe two scenarios. In the memory scenario, a novel employee called a skiv has a property: it can be activated by assigning the specialty at a specific position on it. Your job is to figure out which plant in the perception scenario is the skiv-analog, and therefore which leaf on it activates it.
Memory scenario: There are three employees in an organization: a skiv, a tunk, and a drog. The skiv reports-to the tunk. The tunk sits-beside the drog. The skiv has a yellow color-coded specialty on top, a cyan color-coded specialty on side, and a brown color-coded specialty on bottom. The tunk has a orange color-coded specialty on top, a red color-coded specialty on side, and a black color-coded specialty on bottom. The drog has a green color-coded specialty on top, a purple color-coded specialty on side, and a pink color-coded specialty on bottom. Assigning the specialty at the bottom position of the skiv activates it.
Perception scenario: There are three plants in a garden: a snig, a clop, and a trob. The trob grows-beside the snig. The snig is-growing-under the clop. The snig has a yellow colored leaf on top, a cyan colored leaf on side, and a brown colored leaf on bottom. The clop has a orange colored leaf on top, a red colored leaf on side, and a black colored leaf on bottom. The trob has a green colored leaf on top, a pink colored leaf on side, and a purple colored leaf on bottom.
Which plant in the perception scenario is the skiv-analog, and which leaf activates it?

And here are the visualizations:

Once I saw this, I quickly thought: wait, if Claude can create a graphical model of the problem, then why can’t it use this kind of reasoning to solve the problem? My initial prompt had given the model little context on how to solve it, other than suggesting chain-of-thought reasoning:

You are solving relational reasoning problems. Each problem has a memory scenario and a perception scenario. Your task is to map objects in the perception scenario to objects in the memory scenario based on their relational structure (how they relate to each other), then answer a specific question. Think step by step: first identify the relations in each scenario, then find the mapping that preserves relational structure, then answer.

I asked Claude to help generate a prompt to provide the model with more detailed instructions on how to solve the problem, based on its work on the graphical visualization:

using a standard prompt the models are unable to solve this problem. please suggest a prompt based on your work above that would help an LLM be more likely to successfully solve a problem like this one.

It did so, creating (see full prompt here), and after having Claude Code add the ability to use custom prompts I was able to run the models using this new problem. After a few excruciating minutes of waiting, I had the answer:

All of the models were now at ceiling on the cross-domain problems! At this point it was approaching lunch and I was done with analyses, so I finished the slides and sent them to the conference organizer to put onto the main computer for my presentation in a couple of hours. I then went to lunch and showed the results to several of the attendees, particularly John Hummel because I didn’t want him to feel ambushed when I gave my talk.

The full slide deck is available here if you are interested to see how it came out.

The talk

Given what a crazy experience this was, I felt remarkably calm going into the talk - much more calm that I had earlier in the morning when I was under the gun to produce a coherent slide deck with useful results. When my time came to talk, I zoomed through my 27 slides in about 12 minutes, leaving a couple of minutes for questions. The most interesting question centered around whether providing the long crafted prompt that allowed near-ceiling performance counted as “cheating”. I tend to think that it isn’t; the prompt provides the strategy, but if the model didn’t have the necessary representational apparatus then it shouldn’t be able to solve each individual problem. Another question concerned the use of chain-of-thought reasoning; I didn’t have results to directly address this since I hadn’t run the analyses without the CoT section of the prompt on the full problem set. It’s an interesting question as to how well the models would do without any CoT, but it seems clear to me that CoT would be necessary since the strategic prompting is required for success on the most difficult problem, and it’s hard to see how that would be useful to the model without the ability to use CoT.

I think it’s fair to say that the talk set the meeting abuzz. I had several people approach me afterwards telling me how exciting the talk was, and it spurred numerous conversations in the coffee break that followed. It was certainly the most exciting day I have ever had at a scientific conference.

Takeaways

I took away several lessons from this experience.

• Agentic AI is a superpower. There is no way that I could have achieved this kind of turnaround without the current Claude ecosystem, as both Claude Code and Cowork were central in helping me complete the project. While the talk probably raised more questions than it answered, it addressed a legitimate scientific question arising from another talk, which in the past would have unfolded over weeks or months.

• Testing LLMs is hard. It’s very difficult to ensure that the problems don’t allow shortcuts such that they can solve the problem without actuallly having the intended capability. I’m far from the first to say this, and in fact this is a critical insight from all of comparative and developmental psychology (e.g., see this by my colleague Mike Frank), but this was the first time that I have lived it.

• It’s rarely a good idea to trust the first answer you get. Over the course of about 12 hours I had 4 different answers to the basic question of whether LLMs can solve the relational reasoning problem, and I’m sure that as I continue to work on this there will be additional twists.

• Coding agents like Claude Code are amazing but also clearly make mistakes, as we saw here and as I have documented in my book Better Code, Better Science. As the project becomes more complex, those mistakes can become increasingly difficult to detect. Just as with code generation, lots of validation is required to trust the results.

Would I recommend going from idea to a conference talk in 24 hours? Of course not, and I hope that it doesn’t become an expectation in the future! But the fact that it was even possible in this case speaks to the superpowers provided by the current AI toolchain.

Robert (Bob) Bilder, 1956-2025

I met Bob just after I moved to UCLA in 2002; he had also moved there around the same time from Albert Einstein College of Medicine and North Shore-Long Island Jewish Research Institute. Bob was a clinical neuropsychologist whose work had focused on the neuropsychology of psychiatric disorders. I had started collaborating with psychiatry researchers when I was at MGH, particularly Larry Seidman (who also passed away tragically young in 2017), which had piqued my interest in this direction. Meeting Bob was a revelation, because he was deeply steeped in both cognitive science and neuroscience in addition to his home domain of neuropsychology, giving us a ton of common ground.

The Consortium for Neuropsychiatric Phenomics

Over the next few years Bob and I became increasingly close colleagues, largely focused around the development of a large project that came to be known as the “CNP”. The idea of “phenomics” was made popular by our UCLA colleagues Nelson Freimer and Chiara Sabatti in their 2003 paper, “The Human Phenome Project”. In the wake of the initial sequencing of the human genome in 2001, Freimer and Sabbati proposed that the hard work for biomedical science now laid in the understanding of the universe of phenotypes that arise when the genotype interacts with the environment, and in particular in the way that those phenotypes vary across individuals. Bob was interested in an understanding of psychological and neural phenotypes, which he couched in terms of a framework that I often referred to as the “layer cake” - here is an example reprinted from Figure 4 in Poldrack et al. (2011):

The early 2000s were also a time when many were questioning the way that neuropsychiatric disorders were understood. There was an emerging focus on the degree to which these disorders may reflect the dysfunction of specific underlying neural systems as well as a growing realization that diagnostic categories in psychiatry did not map neatly on underlying neural or psychological mechanisms. It was this set of ideas that would ultimately result in the Research Domain Criteria (RDoC) framework (Insel et al., 2010; Cuthbert and Insel, 2013) that has played a central role in focusing NIMH research onto the underlying dimensions of mental illness. Bob was at the forefront of this movement, and around 2005-6 he began recruiting a dream team of researchers at UCLA that would develop a transdisciplinary approach to understanding the neural basis of psychiatric disorders, focused specifically on the domains of memory and cognitive control. The approach would be transdiagnostic, recruiting healthy individuals along with people diagnosed with several different disorders (schizophrenia, bipolar disorder, and ADHD). It would also be trans-species, with collaborators studying these domains in rodent and nonhuman primate models. Bob was also very forward-thinking in including an entire branch of the project focused on informatics, out of which would grow the Cognitive Atlas project (more about this below).

The set of consortium grants that we developed were funded in 2007 as part of the NIH Roadmap Initiative, and over the next few years we set out to collect a large behavioral/genetics sample of ~2000 individuals (known as the LA2K sample), and a smaller sample of about 300 individuals imaged with fMRI on a range of tasks covering the domains of memory and cognitive control (known as the LA3C sample). I moved from UCLA to the University of Texas in 2009, but I remained closely involved in the project. During my time at UT we developed and launched the OpenfMRI data sharing project, which shared neuroimaging data in a completely open manner. We later shared the entire LA3C imaging dataset through OpenfMRI, and they remain available on the OpenNeuro platform that succeeded OpenfMRI. We also published a data descriptor describing the dataset (Poldrack et al., 2016), which has been cited nearly 500 times since its publication. While not all of these citations reflect an actual reuse of the data, we know that the data has been reused many times, and is one of the most highly accessed datasets on the OpenNeuro archive.

The phenomics mindset remained with me, and in 2012 I developed a study that took it in a different direction. One of the things that is clear but often forgotten about neuropsychiatric disorders is their variability within an individual; a person diagnosed with schizophrenia can be completely disabled one week, and then largely functional several weeks later, and other disorders show similar intra-individual variabilty. Yet our imaging studies usually assume that a single snapshot of a person is a definitive picture of their diagnosis. When I started thinking about how we might understand this variability, I realized that there was a missing link: We knew nothing about how brain function varies over time within a healthy individual. I knew that it would be practically difficult to repeatedly image an individual, so I decided to start with myself. In a study that I came to call the MyConnectome study (Poldrack et al., 2015), I imaged myself repeatedly over the course of 18 months, ultimately scanning myself more than 100 times and taking 48 blood draws. These data provided what was, at the time, the most in-depth picture of brain function and physiology that had ever been collected on an individual. This study also identified a striking degree of individual variability in the functional organization of the cortex (Laumann et al., 2015), which had not been reliably identified with the small amounts of imaging data that were acquired in earlier studies. This finding has since fed into the development of precision function mapping approaches that in the last couple of years have identified robust differences in functional brain organization related to disorders include depression (Lynch et al., 2024), OCD (Vaghi et al., 2025), and Parkinson’s disease (Ren et al., 2026).

We have also taken the phenomics approach in another direction, collecting large amounts of behavioral data on relatively large samples of individuals in order to understand the structure of cognitive control processes. In our first study of this type, we collected a large number of phenotypes, totalling 22 self-report surveys and 37 behavioral tasks, across the domain of cognitive control and related functions. This dataset allowed us to characterize the large-scale structure of the domain, revealing a couple of important findings. First, we found that whereas survey measures were in some cases strongly predictive of real-world outcomes, cognitive task measures showed very little predictive validity for real-world behavior. Additional analyses of a subset of subjects retested after several months (Enkavi et al., 2018) showed that this may reflect low test-retest reliability in task measures. Second, we found that there were negligible or weak correlations between survey measures and task measures, even though they are often taken to index the same underlying constructs. Instead, the task domain seemed to be primarily organized according to the different aspects of speeded responding that can be identified using a diffusion decision model (DDM), as shown in Figure 4 of Eisenberg et al. (2019):

These two projects are just a couple of the ways in which phenomics thinking has influenced my lab’s work in years since I left UCLA. While Bob was not a direct collaborator on these projects, we discussed them often, and his fingerprints are all over them; I simply don’t think I would have ended up doing this work if it weren’t for Bob’s influence.

Informatics and Ontologies

Bob was also a visionary regarding the role of informatics in understanding minds and brains and their disorders. One of the projects within the CNP was focused entirely on informatics, particularly focused on developing a “Hypothesis Web” that could link together data across multiple sources and levels. This was directly driven by the progress that Bob had witnessed in the field of genomics, where the Gene Ontology and related tools had allowed rapid advances in biological understanding. Bob envisioned that we could mine data to better understand the structure of the “layer cake” that I showed above.

A number of tools came out of this, but the one that has had the most enduring impact is the Cognitive Atlas (Poldrack et al., 2011). The aim of this project was to develop a controlled vocabulary of the terms used to define psychological functions and psychological measurements (such as tasks or surveys), and to annotate the relationships between them. We made good headway on this in the early days, but for more than a decade the project received almost no interest, outside of philosophers were were interested in the notion of “cognitive ontologies” from a theoretical point of view. It wasn’t until the last few years that the world became interested in the Cognitive Atlas, as you can see in this plot of citations to the paper, which have spiked in just the last few years:

I think there are a couple of reasons for this increased interest. One is that the National Academies convened a meeting on the topic, which led to a report titled “Ontologies in the Behavioral Sciences: Accelerating Research and the Spread of Knowledge (2022)”. This report made six recommendations, which focused on the need for increased funding, policy, and education around the importance of ontologies in the behavioral sciences.

Simultaneously, the Cognitive Atlas has increasingly been used in conjunction with AI tools to annotate data and generate new knowledge. As one example, Menuet et al. (2022) used an enriched version of the Cognitive Atlas to annotate data from the Neurovault data archive (an archive of statistical maps that we also run) and then used these data to train a neural network to decode cognitive functions from brain images, as shown in Figure 1 of their paper:

They showed that they could decode a large number of mental processes on held-out data, with greater decoding accuracy than models that don’t use the ontology.

We also showed several years ago that the Cognitive Atlas could be useful for predicting neuroimaging signals via “cognitive encoding models”, which annotate individual tasks with their Cognitive Atlas features and then use linear models to fit the data via those features (Walters et al., 2022). We showed that these models substantially improved our ability to predict activation maps on unseen tasks, based only on their Cognitive Atlas annotations. This provided yet another proof of concept that ontologies are a powerful tool for data analysis and understanding.

Ontologies are also playing an important role in powering new agentic AI tools for data analysis and understanding. In the Brain-Researcher project developed by Zijiao Chen in my lab recently, we have used a knowledge graph that includes the Cognitive Atlas as the basis for a neuroscience data analysis agent, which can autonomously identify data and generate code to answer neuroscientific questions based on natural language queries. The machine-readable nature of the Atlas makes it easily ingestible by AI agents, so we expect that it will become increasingly useful for AI applications. I’m really sad that Bob isn’t able to see this, because it reflects the mature version of the ideas that he inspired our group to hatch back in the mid-2000’s.

Bob’s legacy

Bob clearly had an important impact on my career, but even more importantly, he was a close friend and mentor to me. We had many conversations in his office at the UCLA Neuropsychiatric Institute, which were wide-ranging, enlightening, and fun. My wife Jen and I were fortunate to be able to see Bob one last time in May in Los Angeles, and enjoy some steak (cooked rare of course) and a nice red wine with Bob, his wife Debbie, and his daughter Alexandra. We miss him dearly but I hope that our work will continue to carry forward his legacy well into the future.

There is a wide variety of workflow engines available for data analysis workflows, most of which are centered around the concept of an “execution graph”. This is a graph in the sense described by graph theory, which refers to a set of nodes that are connected by lines (known as “edges”). Workflow execution graphs are a particular kind of graph known as a directed acyclic graph, or DAG for short. Each node in the graph represents a single step in the workflow, and each edge represents the dependency relationships that exist between nodes. DAGs have two important features. First, the edges are directed, which means that they move in one direction that is represented graphically as an arrow. These represent the dependencies within the workflow. For example, in our workflow step 1 (obtaining the data) must occur before step 2 (filtering the data), so the graph would have an edge from step 1 with an arrow pointing at step 2. Second, the graph is acyclic, which means that it doesn’t have any cycles, that is, it never circles back on itself. Cycles would be problematic, since they could result in workflows that executed in an infinite loop as the cycle repeated itself.

Most workflow engines provide tools to visualize a workflow as a DAG. Figure 8.2 shows our example workflow visualized using the Snakemake tool that we will introduce below:

Figure 8.2: The execution graph for the simple example analysis workflow visualized as a DAG.

The use of DAGs to represent workflows provides a number of important benefits:

• The engine can identify independent pathways through the graph, which can then be executed in parallel

• If one node of the graph changes, the engine can identify which downstream nodes need to be rerun

• If a node fails, the engine can be configured to continue with executing the nodes that don’t depend on the failed node either directly or indirectly

There are a couple of additional benefits to using a workflow engine, which we will discuss in more detail in the context of a more complex workflow. The first is that they generally deal automatically with the storage of intermediate results (known as caching or checkpointing), which can help speed up execution when nothing has changed and allow continued execution if the process is interrupted. The second is that the workflow engine uses the execution graph to optimize the schedule of computations, only performing those operations that are actually needed. This is similar in spirit to the concept of lazy execution used by packages like Polars, in which the system optimizes computational efficiency by first analyzing the full computational graph.

General-purpose versus domain-specific workflow engines

With the growth of data science within industry and research, there has been an explosion of new workflow management systems that aim to solve particular problems; a list of these can be found at awesome-workflow-engines. It’s also worth noting that there are a number of domain-specific workflow engines that are specialized for particular kinds of data and workflows. Examples include Galaxy which is specialized for bioinformatics and genomics, and Nipype which is specialized for neuroimaging analysis workflows. If your research community uses one of these then it’s worth exploring that engine as your first option, since it will probably be well supported within the community. However, a benefit of using a general-purpose engine is that they will often be better maintained and supported, and AI tools will likely have more examples to work from in generating workflows.

In the next post I will take a deep dive into workflow management using the Snakemake workflow engine.

Metadata

Metadata refers to “data about data”, and generally is meant to contain the information that is needed to interpret a dataset. In principle, someone who obtains a dataset should be able to understand and reuse the data using only the metadata provided alongside the dataset. There are many different types of metadata that might be associated with a study, and it is usually necessary to decide how comprehensive to be in providing detailed metadata. This will often rely upon the scientific expertise and judgment of the researcher, to determine which particular metadata would be essential for others to usefully interpret and reuse the data.

An important concept in metadata is the ontology. In the context of bioinformatics, an ontology is a structured representation of the entities that exist in a domain (defined by a controlled vocabulary) and the relationships between these entities. One of the best known examples in the Gene Ontology, which represents classes of biological entities including Molecular Functions, Cellular Components, and Biological Processes. As an example, this figure shows a Gene Ontology graph for the entity “node of Ranvier”, which is a component of a neuron (obtained from here).

Ontologies are very useful for specifying metadata, because they allow us to know exactly what a particular entry in the metadata means, and thus allow us to establish link between equivalent entities across datasets. For example, let’s say that a researcher wants to query a database for datasets related to insulin signaling in pancreatic beta cells in Type II diabetes, and that there are three relevant datasets in the database. Without an ontology, each of the teams might use different terms to refer to these cells (such as “pancreatic beta cells”, “insulin-producing cells”, and “islet beta cells”), making it difficult to link the datasets. However, if each of the datasets were to include metadata linked to a specific ontology (in this case, the identifier CL:0000169 from the Cell Ontology, which refers to “type B pancreatic cell”), then it becomes much easier to find and link these datasets. There are at present a broad range of ontologies available for nearly every scientific domain; the BioPortal project provides a tool to search across a wide range of existing ontologies.

Metadata file formats

An important feature of metadata is that it needs to be machine-readable, meaning that it is provided in a structured format that be automatically parsed by a computer. Common formats are Extensible Markup Language (XML) and JavaScript Object Notation (JSON). JSON is generally simpler and more human-readable, but it doesn’t natively provide the ability to define attributes for particular entries (such as the units of measurement) or link to ontologies. An extension of JSON known as JSON-LD (JSON for Linked Data) provides support for the latter, by allowing links to controlled vocabularies.

For example, let’s say that I wanted to represent information about an author (myself) in JSON, which I might do like this:

{
  “name”: “Russell Poldrack”,
  “affiliation”: “Stanford University”,
  “email”: “russpold@stanford.edu”
}

Now let’s say that someone else wanted to search across datasets to find researchers from Stanford University. They would have no way of knowing that I used the term “affiliation” as opposed to “organization”, “institution”, or other terms. We could instead represent this using JSON-LD, which is more verbose but allows us to link to a vocabulary (in this case schema.org) that defines these entities by providing a @context tag:

{
  “@context”: “https://schema.org”,
  “@type”: “Person”,
  “name”: “Russell Poldrack”,
  “affiliation”: {
    “@type”: “Organization”,
    “name”: “Stanford University”
  },
  “email”: “russpold@stanford.edu”
}

Data documentation

While metadata is generally meant to be used by computers, it is also important to provide human-readable documentation for a dataset, so that other researchers (or one’s own self in the future) can understand and reuse the data successfully. There are two forms of documentation that can be important to provide.

Data dictionaries

A data dictionary provides information about each of the variables in a dataset. These are meant to be human readable, though it can often be useful to share them in a machine-readable format (such as JSON) so that they can also be used in programmatic ways. A data dictionary includes information such as:

• an understandable description of the variable

• the data type (e.g. string, integer, Boolean)

• the allowable range of values

For example, a study of immune system function in human participants might include the following in its data dictionary:

| Variable Name | Data Type | Allowable Values | Description |
|---------------|-----------|------------------|-------------|
| age           | Integer   | 0-120            | Age of the participant in years |
| gender        | String    | M, W, O          | Participant’s self-identified gender |
| crp           | Numeric   | 0.1-50.0, -90, -98, -99| C-reactive protein level (mg/L) |

Codebooks

A codebook is meant to be a more human-friendly description of the content of the dataset, focusing on how the data were collected and coded. It often includes a detailed description of each variable that is meant to help understand and interpret the data. For the example above, the codebook might include the following:

Variable Information

• Variable name: crp

• Variable label: High-sensitivity C-reactive protein

• Variable definition: A quantitative measure of C-reactive protein in blood serum.

Measurement and Coding

• Data Type: Numeric (Floating Point, 2 decimal places)

• Units of Measurement: mg/L (milligrams per Liter)

• Measurement Method: Immunoturbidimetric assay.

• Instrument: Roche Cobas c702 clinical chemistry analyzer.

• Allowable Range: 0.10 - 50.00

  • Note: The lower limit of detection for this assay is 0.10 mg/L.

• Values and Codes:

  • [Numerical Value]: A continuous value from 0.10 to 50.00 represents the measured concentration in mg/L.

  • -90: Value below the lower limit of detection (< 0.10 mg/L).

  • -98: Unusable sample (e.g., sample was hemolyzed, insufficient quantity).

  • -99: Missing (e.g., sample not collected, participant refused blood draw).

Collection Protocol and Provenance

• Specimen Type: Serum from a venous blood sample.

• Collection Procedure: Blood was drawn from the antecubital vein into a serum separator tube (SST) after an 8-hour overnight fast. The sample was allowed to clot for 30 minutes, then centrifuged at 1,500 x g for 15 minutes. Serum was aliquoted and stored at -80°C until analysis.

• Date of Creation: 2025-11-15

• Version: 1.0

It is essential to generate data dictionaries and codebooks upon the generation of the dataset; otherwise important details may be lost.

Provenance

Provenance refers to particular metadata regarding the history of processes and inputs that give rise to a particular file. Tracking of provenance is essential to ensure that one knows exactly how a particular file was created. This includes:

• the origin of original data (such as the instrument used to collect it, or date of collection)

• the specific input files that went into creation of the file, for files that are derived data

• the specific versions of any software tools that were used to create the file

• the specific settings used for the software tools

Tracking of provenance is non-trivial. The World Wide Web Consortium (W3C) has developed a framework called PROV which defines a model for the representation of provenance information. This framework provides an overview of the many features of provenance that one might want to record for an information that is shared online. The PROV data models defines three main concepts:

• Entities: things that are produced, such as datasets and publications

• Activities: processes that involve using, generating, or modifying entities

• Agents: People, organizations, or artifacts (such as computers) that are responsible for activities

In addition, the model defines a set of relationships between these concepts, as shown in this figure from the W3C:

(Copyright © [2013] [World Wide Web Consortium]).

This data model highlights the breadth of information that needs to be represented in order to accurately record provenance.

There are several different ways to track provenance in practice, which vary in their complexity, comprehensiveness, and ease of use. We will discuss this in much more detail in a later chapter on workflows.

In the next post I will discuss the handling of sensitive data.