ImageNetData¶
-
class
pycalib.benchmark.ImageNetData(run_dir, clf_output_dir, classifier_names, cal_methods, cal_method_names, use_logits=False, n_splits=10, test_size=0.9, train_size=None, random_state=None)[source]¶ Bases:
pycalib.benchmark.BenchmarkModel evaluation using the benchmark vision dataset ImageNet.
Implements a data generation method returning a new evaluation data set for each scoring round. Pre-trained models are available at 1.
- Parameters
run_dir (str) – Directory to run benchmarking in and save output and logs to.
clf_output_dir (str) – Directory containing calibration data obtained from ImageNet classification.
classifier_names (list) – Names of classifiers to be calibrated. Classification results on ImageNet must be contained in data_dir.
cal_methods (list) – Calibration methods to benchmark.
cal_method_names (list) – Names of calibration methods.
use_logits (bool, default=False) – Should the calibration methods be used with classification probabilities or logits.
n_splits (int, default=10) – Number of splits for cross validation.
test_size (float, default=0.9) – Size of test set.
train_size (float, default=None) – Size of calibration set.
random_state (int, RandomState instance or None, optional (default=None)) – If int, random_state is the seed used by the random number generator; If RandomState instance, random_state is the random number generator; If None, the random number generator is the RandomState instance used by np.random.
References
Methods Summary
classify_val_data(file, clf_name[, …])Classify the ImageNet evaluation data set with a given model.
data_gen()Returns the full dataset or a generator of datasets.
plot(out_file, results_file, score, methods)Plot results from benchmark experiments as an error bar plot.
run([n_jobs])Train all models, evaluate on test data and save the results.
Methods Documentation
-
static
classify_val_data(file, clf_name, n_classes=1000, file_ending='JPEG', data_folder='data/val', output_folder='clf_output')[source]¶ Classify the ImageNet evaluation data set with a given model.
- Parameters
file (str) – Directory containing ImageNet evaluation data. Folder ‘data_folder’ under file is searched for images. Output from the model is saved in the given directory.
clf_name (str) – Name of classifier (CNN architecture) to classify data with. Must be one of pretrainedmodels.model_names.
n_classes (int, default=1000) – Number of classes of pretrained model on ImageNet.
file_ending (str, default="JPEG") – File suffix of images to classify.
data_folder (str, default='val') – Folder where validation images are contained. Images must be contained in folder named after their class.
output_folder (str, default='clf_output') – Folder where output is stored.
-
data_gen()[source]¶ Returns the full dataset or a generator of datasets.
- Returns
- Return type
X, y giving uncalibrated predictions and corresponding classes.
-
static
plot(out_file, results_file, score, methods, classifiers='all', width=5.0, height=2.5)¶ Plot results from benchmark experiments as an error bar plot.
- Parameters
out_file (str) – File location for the output plot.
results_file (str) – The location of the csv files containing experiment results.
score (str) – Type of score to plot.
methods (list) – Calibration methods to plot.
classifiers (list or "all") – List of classifiers for which to show results.
width (float, default=5.) – Width of the plot.
height (float, default=2.5) – Height of the plot.