Simulation testing the robustness of stock assessment models to error: some results from the ICES strategic initiative on stock assessment methods

The World Conference on Stock Assessment Methods (July 2013) included a workshop on testing assessment methods through simulations. The exercise was made up of two steps applied to datasets from 14 representative fish stocks from around the world. Step 1 involved applying stock assessments to datasets with varying degrees of effort dedicated to optimizing fit. Step 2 was applied to a subset of the stocks and involved characteristics of given model fits being used to generate pseudo-data with error. These pseudo-data were then provided to assessment modellers and fits to the pseudo-data provided consistency checks within (self-tests) and among (cross-tests) assessment models. Although trends in biomass were often similar across models, the scaling of absolute biomass was not consistent across models. Similar types of models tended to perform similarly (e.g. age based or production models). Self-testing and cross-testing of models are a useful diagnostic approach, and suggested that estimates in the most recent years of time-series were the least robust. Results from the simulation exercise provide a basis for guidance on future large-scale simulation experiments and demonstrate the need for strategic investments in the evaluation and development of stock assessment methods.