Postprocessing#

Postprocessing extends the autrainer postprocessing functionality to include curriculum-specific postprocessing such as aggregating over scoring or pacing functions.

Summarization#

class aucurriculum.postprocessing.SummarizeCurriculum(results_dir, experiment_id, summary_dir='summary', training_dir='training', training_type=None, max_runs_plot=None)[source]#

Summarize the results of a grid search including curricula.

Parameters:

  • results_dir (str) – The directory where the results are stored.

  • experiment_id (str) – The ID of the grid search experiment.

  • summary_dir (str) – The directory where the the grid search summary will be stored. Defaults to “summary”.

  • training_dir (str) – The directory of the training results of the experiment. Defaults to “training”.

  • clear_old_outputs – Whether to clear existing summary outputs. Defaults to True.

  • training_type (Optional[str]) – The type of training in [“epoch”, “step”]. If None, it will be inferred from the training results. Defaults to None.

  • max_runs_plot (Optional[int]) – The maximum number of best runs to plot. If None, all runs will be plotted. Defaults to None.

summarize()[source]#

Summarize the pacing functions of the curricula.

Return type:

None

Aggregation#

class aucurriculum.postprocessing.AggregateCurriculum(results_dir, experiment_id, aggregate_list, aggregate_prefix='agg', training_dir='training', max_runs_plot=None, aggregate_name=None, aggregated_dict=None, plot_params=None)[source]#

Aggregate the results of a grid search over one or more parameters.

If loggers have been used for the grid search, the aggregated results will be logged to the same loggers.

Parameters:

  • results_dir (str) – The directory where the results are stored.

  • experiment_id (str) – The ID of the grid search experiment.

  • aggregate_list (List[str]) – The list of parameters to aggregate over.

  • aggregate_prefix (str) – The prefix for the aggregated experiment ID. Defaults to “agg”.

  • training_dir (str) – The directory of the training results of the experiment. Defaults to “training”.

  • max_runs_plot (Optional[int]) – The maximum number of best runs to plot. If None, all runs will be plotted. Defaults to None.

  • aggregate_name (Optional[str]) – The name of the aggregated experiment. If None, it will be generated from the aggregate_list. Defaults to None.

  • aggregated_dict (Optional[dict]) – A dictionary mapping the aggregated experiment names to the runs to aggregate. If None, the runs will be aggregated based on the aggregate_list. Defaults to None.

  • plot_params (Optional[dict]) – Additional parameters for plotting. Defaults to None.

Grouping#

class aucurriculum.postprocessing.GroupCurriculum(results_dir, groupings, max_runs=None, plot_params=None)[source]#

Group runs of one or more grid search experiments based on the specified groupings.

Parameters:

  • results_dir (str) – The directory where the results are stored.

  • groupings (Union[ListConfig[DictConfig], List[Dict]]) – A list of experiments to create containing one or more runs to group.

  • max_runs_plot – The maximum number of best runs to plot. If None, all runs will be plotted. Defaults to None.

  • plot_params (Optional[dict]) – Additional parameters for plotting. Defaults to None.

On this page