An approach to ranking and selection for multiple performance measures
33
Citation
12
Reference
20
Related Paper
Citation Trend
Abstract:
We develop a ranking and selection procedure for making multiple comparisons of systems that have multiple performance measures. The procedure combines multiple attribute utility theory with ranking and selection to select the best configuration from a set of K configurations using the indifference zone approach. We demonstrate our procedure on a simulation model of a large project that has six performance measures.We consider the problem of comparing a small number of stochastic systems via computer simulation when the basis for comparison is the expected value of some system performance measure. To solve this problem we develop two-stage sampling procedures that provide confidence intervals for the difference between the expected performance of each system and the best of the others. These confidence intervals are valid under mild conditions, and the procedures allow the experimenter to specify the desired precision in advance. Special cases of our results include standard indifference-zone selection procedures. The paper includes guidelines for experiment design and an illustrative example.
Basis (linear algebra)
Value (mathematics)
Cite
Citations (98)
Article Free Access Share on Sensitivity analysis in ranking and selection for multiple performance measures Authors: Douglas J. Morrice MSIS Department, The University of Texas at Austin, Austin, TX MSIS Department, The University of Texas at Austin, Austin, TXView Profile , John Butler Department of Accounting and MIS, The Ohio State University, Columbus, OH Department of Accounting and MIS, The Ohio State University, Columbus, OHView Profile , Peter Mullarkey Maxager Technology, 103 Copperleaf Road, Austin, TX Maxager Technology, 103 Copperleaf Road, Austin, TXView Profile , Srinagesh Gavirneni Schlumberger, 8311 North RR 620, Austin, TX Schlumberger, 8311 North RR 620, Austin, TXView Profile Authors Info & Claims WSC '99: Proceedings of the 31st conference on Winter simulation: Simulation---a bridge to the future - Volume 1December 1999 Pages 618–624https://doi.org/10.1145/324138.324445Published:01 December 1999Publication History 4citation325DownloadsMetricsTotal Citations4Total Downloads325Last 12 Months12Last 6 weeks0 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteeReaderPDF
Cite
Citations (12)
Abstract In this paper we discuss a modification of the Dudewicz-Dalal procedure for the problem of selecting the population with the largest mean from k normal populations with unknown variances. We derive some inequalities and use them to lower-bound the probability of correct selection. These bounds are applied to the determination of the second-stage sample size which is required in order to achieve a prescribed probability of correct selection. We discuss the resulting procedure and compare it to that of Dudewicz and Dalai (1975). Keywords: normal meansunknown variancesprobability of correct selection
Cite
Citations (513)
From the Publisher:
This second edition of Modeling and Analysis includes a chapter on Simulation in Manufacturing Systems and examples. The text is designed for a one-term or two-quarter course in simulation offered in departments of industrial engineering,business,computer science and operations research.
Cite
Citations (9,195)
"Design and Analysis of Experiments for Statistical Selection, Screening and Multiple Comparisons." Technometrics, 38(3), pp. 289–290
Statistical Analysis
Multiple comparisons problem
Cite
Citations (536)
Managers of large industrial projects often measure performance by multiple attributes. For example, our paper is motivated by the simulation of a large industrial project called a land seismic survey, in which project performance is based on duration, cost, and resource utilization. To address these types of problems, we develop a ranking and selection procedure for making comparisons of systems (e.g., project configurations) that have multiple performance measures. The procedure combines multiple attribute utility theory with statistical ranking and selection to select the best configuration from a set of possible configurations using the indifference-zone approach. We apply our procedure to results generated by the simulator for a land seismic survey that has six performance measures, and describe a particular type of sensitivity analysis that can be used as a robustness check.
Robustness
Cite
Citations (220)
We present a state-of-the-art review of screening, selection, and multiple comparison procedures that are used to compare system designs via computer simulation. We describe methods for three broad classes of problems: screening a large number of system designs, selecting the best system, and comparing all systems to a standard (either known or unknown). We concentrate primarily on recent methods that we would be likely to use in practice. Where possible, we unify the screening, selection, and multiple comparison perspectives.
Cite
Citations (53)
Standard “indifference-zone” procedures that allocate computer resources to infer the best of a finite set of simulated systems are designed with a statistically conservative, least favorable configuration assumption consider the probability of correct selection (but not the opportunity cost) and assume that the cost of simulating each system is the same. Recent Bayesian work considers opportunity cost and shows that an average case analysis may be less conservative but assumes a known output variance, an assumption that typically is violated in simulation. This paper presents new two-stage and sequential selection procedures that integrate attractive features of both lines of research. They are derived assuming that the simulation output is normally distributed with unknown mean and variance that may differ for each system. We permit the reduction of either opportunity cost loss or the probability of incorrect selection and allow for different replication costs for each system. The generality of our formulation comes at the expense of difficulty in obtaining exact closed-form solutions. We therefore derive a bound for the expected loss associated potentially incorrect selections, then asymptotically minimize that bound. Theoretical and empirical results indicate that our approach compares favorably with indifference-zone procedures.
Generality
Variance reduction
Replication
Asymptotically optimal algorithm
Cite
Citations (375)
In this paper, we address the problem of finding the simulated system with the best (maximum or minimum) expected performance when the number of alternatives is finite, but large enough that ranking-and-selection (R&S) procedures may require too much computation to be practical. Our approach is to use the data provided by the first stage of sampling in an R&S procedure to screen out alternatives that are not competitive, and thereby avoid the (typically much larger) second-stage sample for these systems. Our procedures represent a compromise between standard R&S procedures—which are easy to implement, but can be computationally inefficient—and fully sequential procedures—which can be statistically efficient, but are more difficult to implement and depend on more restrictive assumptions. We present a general theory for constructing combined screening and indifference-zone selection procedures, several specific procedures and a portion of an extensive empirical evaluation.
Sample (material)
Cite
Citations (364)
Article Free AccessSelecting the best system: a decision-theoretic approach Author: Stephen E. Chick Department of Industrial and Operations Engineering, The University of Michigan, 1205 Beal Avenue, Ann Arbor, Michigan Department of Industrial and Operations Engineering, The University of Michigan, 1205 Beal Avenue, Ann Arbor, MichiganView Profile Authors Info & Claims WSC '97: Proceedings of the 29th conference on Winter simulationDecember 1997Pages 326–333https://doi.org/10.1145/268437.268499Published:01 December 1997Publication History 30citation249DownloadsMetricsTotal Citations30Total Downloads249Last 12 Months15Last 6 weeks1 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteeReaderPDF
Cite
Citations (47)