Changelog

Here we list noteworthy changes to the Engine and Experiments APIs. Some minor versions are skipped here as they did not include noteworthy changes. For full release version histories see Engine release history and Experiments release history

---

Engine (since initial alpha release in September, 2022):

0.8.4

• Add decision tree and random forest (with leaf probability tuning) models.
• Add error rate objective function and measure function.
• Add weighted importance sampling objective function and measure function.

0.8.1

• Fixed a bug where the datasize term in the confidence interval calculations was incorrect in some edge cases. Now, the datasize is always based off of the length of the vector of estimator samples (from the zhat function).
• Separate out zhat functions (from objectives.py) into their own module zhat_funcs.py.

0.8.0

• Create a class to hold metadata. Metadata are stored in dataset's via a DataSet.meta attribute.
• Add support for alternate RL rewards. Arbitrary number of RL rewards are supported and can be referenced in constraint strings via "J_pi_new_[i]", where i is the alternate reward number, starting at 1. Alternate rewards are stored in the Episode object as a single multidimensional array under the Episode.alt_rewards attribute.

0.7.8

• Add support for scikit-learn models; adds a base class and a linear regressor template.

0.7.7

• Add method for extracting value of upper bounds on constraint functions after having run the safety test.

0.7.6

• Allow sensitive attributes in RL datasets.
• Better logging for gradient descent.
• Add option to plot running average of primary objective function and Lagrangian in gradient descent plots.
• Fixes bug where mini-batches were incorrectly calculated when constraints included conditional columns.
• Add PyTorch facial recognition example.

0.7.5

• PyTorch forward/backward passes made more efficient.

0.7.4

• Add support for Tensorflow models.
• Add option to batch safety data for the safety test.

0.7.3

• RL environment specified as object in hyperparam dict

0.7.1

• Add support for PyTorch models.
• Create example demonstrating PyTorch CNN on MNIST.
• Pandas no longer used internally. It is currently only for data loading for CSV data files.
• Supervised learning features can be list of arrays or a single NumPy array.
• Add option to mini-batch in gradient descent.

0.6.3

• Add housing prices example - no tutorial for it yet.
• Add accuracy as a measure function for binary and multi-class classification.

0.6.2

• Remove "include_intercept_term" parameter in dataset, so user no longer can specify if they want a column of ones added in a supervised dataset. The intercept is now handled entirely by the model.

0.6.1

• Add multi-class classification.

0.6.0 (initial alpha)

---

Experiments (since initial alpha release):

0.2.1

• Include decision tree and random forest scikit-learn baselines.

0.2.0

• Record each g value (the value of the constraint function) for each constraint separately in trial output files.
• In three plots, each constraint gets its own row with its own g plot.
• Start testing examples.
• Baselines are now objects, no longer hardcoded. User can easily provide custom baseline.

0.1.1

• Add parameters to three plots to allow for more customization.

0.1.0

• Examples are now class-based and can all be run from a single master script.
• Get basic headless example working.

0.0.13

• Add (uniform) random classifier baseline.

0.0.12

• Add more customization to the three plots.

0.0.11

• Add ability to mini-batch performance and g evaluation of QSA.
• Add ability to mini-batch when running baselines.
• An initial attempt to modularize baselines.
• Fix bug where parse tree was not resetting before g evaluation.

0.0.10

• Add support for mini-batching.
• RL environment specified as object in hyperparam dict.
• Fairlearn made optional import.
• Option to choose log scale for vertical axis on the performance plot.
• Fix bug where parse tree base node dict was not being reset before calculating g on ground truth dataset for prob(violated constraint) plot.
• Add option to provide batch_epoch_dict to specify the batch size and number of epochs to use for each data_frac.

0.0.9

• Update to reflect changes to binary classification in the Engine version>=0.6.1.
• Remove "include_intercept_term" functionality to conform with changes to the Engine version>=0.6.2
• Add linear regression baseline.

0.0.8

• Turn off parallel processing for generating the episodes for running experiments.
• Remove "include_intercept_term" functionality to conform with changes to the Engine version>=0.6.2.
• Add linear regression baseline.

0.0.7 (initial alpha release)