*Organizational Change *Program Evaluation Monitoring Organizational Behavior [3660]. Human. ACT-R associative strength cognitive architectures cognitive modeling cognitive skill acquisition Common Lisp computational linguistics dips DMAP EPIC evaluation expertise extreme expertise GOMS GOMS/Analysis methods/Empirical studies/User models/Cognitive models/Methods for analysis or assessment/Prototyping/Protocol analysis/Theory in HCI interaction design JSON leaps Measures of Semantic Relatedness natural language processing performance plateaus Rational analysis Semantic similarity measures soar Space Fortress Tetris tutorial VCR
2017 |
Gray, Wayne D; Lindstedt, John K Plateaus, Dips, and Leaps: Where to Look for Inventions and Discoveries during Skilled Performance Journal Article Cognitive Science, 41 (7), pp. 1838-1870, 2017. Abstract | Links | BibTeX | Tags: BreakOut, changepoint detection, digit span, dips, expertise, extreme expertise, leaps, performance, plateaus, Space Fortress @article{gray17csj-pdl, title = {Plateaus, Dips, and Leaps: Where to Look for Inventions and Discoveries during Skilled Performance}, author = {Wayne D. Gray and John K. Lindstedt}, url = {http://homepages.rpi.edu/~grayw/pubs/papers/2017/gray17csj-pdl.pdf}, doi = {10.1111/cogs.12412}, year = {2017}, date = {2017-10-18}, journal = {Cognitive Science}, volume = {41}, number = {7}, pages = {1838-1870}, abstract = {The framework of plateaus, dips, and leaps shines light on periods when individuals may be inventing new methods of skilled performance. We begin with a review of the role performance plateaus have played in (a) experimental psychology, (b) human--computer interaction, and (c) cognitive science. We then reanalyze two classic studies of individual performance to show plateaus and dips which resulted in performance leaps. For a third study, we show how the statistical methods of Changepoint Analysis plus a few simple heuristics may direct our focus to periods of performance change for individuals. For the researcher, dips become the marker of exploration where performance suffers as new methods are invented and tested. Leaps mark the implementation of a successful new method and an incremental jump above the path plotted by smooth and steady log--log performance increments. The methods developed during these dips and leaps are the key to surpassing one's teachers and acquiring extreme expertise.}, keywords = {BreakOut, changepoint detection, digit span, dips, expertise, extreme expertise, leaps, performance, plateaus, Space Fortress}, pubstate = {published}, tppubtype = {article} } The framework of plateaus, dips, and leaps shines light on periods when individuals may be inventing new methods of skilled performance. We begin with a review of the role performance plateaus have played in (a) experimental psychology, (b) human--computer interaction, and (c) cognitive science. We then reanalyze two classic studies of individual performance to show plateaus and dips which resulted in performance leaps. For a third study, we show how the statistical methods of Changepoint Analysis plus a few simple heuristics may direct our focus to periods of performance change for individuals. For the researcher, dips become the marker of exploration where performance suffers as new methods are invented and tested. Leaps mark the implementation of a successful new method and an incremental jump above the path plotted by smooth and steady log--log performance increments. The methods developed during these dips and leaps are the key to surpassing one's teachers and acquiring extreme expertise. |
Gray, Wayne D Plateaus and Asymptotes: Spurious and Real Limits in Human Performance Journal Article Current Directions in Psychological Science, 26 (1), pp. 59-67, 2017. Abstract | Links | BibTeX | Tags: asymptotes, cognitive skill acquisition, expertise, memory, performance, plateaus, spurious limits @article{gray17cdps, title = {Plateaus and Asymptotes: Spurious and Real Limits in Human Performance}, author = {Gray, Wayne D.}, url = {http://homepages.rpi.edu/~grayw/pubs/papers/2017/gray17cdps.pdf}, doi = {10.1177/0963721416672904}, year = {2017}, date = {2017-02-15}, journal = {Current Directions in Psychological Science}, volume = {26}, number = {1}, pages = {59-67}, abstract = {One hundred twenty years ago, the emergent field of experimental psychology debated whether plateaus of performance during training were real or not. Sixty years ago, the battle was over whether learning asymptoted or not. Thirty years ago, the research community was seized with concerns over stable plateaus at suboptimal performance levels among experts. Applied researchers viewed this as a systems problem and referred to it as the paradox of the active user. Basic researchers diagnosed this as a training problem and embraced deliberate practice. The concepts of plateaus and asymptotes and the distinction between the two are important as the questions asked and the means of overcoming one or the other differ. These questions have meaning as we inquire about the nature of performance limits in skilled behavior and the distinction between brain capacity and brain efficiency. This article brings phenomena that are hiding in the open to the attention of the research community in the hope that delineating the distinction between plateaus and asymptotes will help clarify the distinction between real versus ``spurious limits'' and advance theoretical debates regarding learning and performance.}, keywords = {asymptotes, cognitive skill acquisition, expertise, memory, performance, plateaus, spurious limits}, pubstate = {published}, tppubtype = {article} } One hundred twenty years ago, the emergent field of experimental psychology debated whether plateaus of performance during training were real or not. Sixty years ago, the battle was over whether learning asymptoted or not. Thirty years ago, the research community was seized with concerns over stable plateaus at suboptimal performance levels among experts. Applied researchers viewed this as a systems problem and referred to it as the paradox of the active user. Basic researchers diagnosed this as a training problem and embraced deliberate practice. The concepts of plateaus and asymptotes and the distinction between the two are important as the questions asked and the means of overcoming one or the other differ. These questions have meaning as we inquire about the nature of performance limits in skilled behavior and the distinction between brain capacity and brain efficiency. This article brings phenomena that are hiding in the open to the attention of the research community in the hope that delineating the distinction between plateaus and asymptotes will help clarify the distinction between real versus ``spurious limits'' and advance theoretical debates regarding learning and performance. |
2016 |
Destefano, Marc ; Gray, Wayne D Proceedings of the 38th Annual Conference of the Cognitive Science Society, Cognitive Science Society, Austin, TX, 2016. Abstract | Links | BibTeX | Tags: changepoint analysis, dips, expertise, leaps, method invention, performance, plateaus, SAX, skill acquisition, Space Fortress, strategy discovery @conference{marc16csc, title = {Where Should Researchers Look for Strategy Discoveries during the Acquisition of Complex Task Performance? The Case of Space Fortress}, author = {Destefano, Marc and Gray, Wayne D.}, editor = {Papafragou, A. and Grodner, D. and Mirman, D. and Trueswell, J. C.}, url = {http://homepages.rpi.edu/~grayw/pubs/papers/2016/marc16csc.pdf}, year = {2016}, date = {2016-08-05}, booktitle = {Proceedings of the 38th Annual Conference of the Cognitive Science Society}, publisher = {Cognitive Science Society}, address = {Austin, TX}, abstract = {In complex task domains, such as games, students may exceed their teachers. Such tasks afford diverse means to tradeoff one type of performance for another, combining task elements in novel ways to yield method variations and strategy discoveries that, if mastered, might produce large or small leaps in performance. For the researcher interested in the development of extreme expertise in the wild, the problem posed by such tasks is ``where to look'' to capture the explorations, trials, errors, and successes that eventually lead to the invention of superior performance. In this paper, we present several successful discoveries of methods for superior performance. For these discoveries we used Symbolic Aggregate Approximation as our method of identifying changepoints within score progressions in the venerable game of Space Fortress. By decomposing performance at these changepoints, we find previously unknown strategies that even the designers of the task had not anticipated.}, keywords = {changepoint analysis, dips, expertise, leaps, method invention, performance, plateaus, SAX, skill acquisition, Space Fortress, strategy discovery}, pubstate = {published}, tppubtype = {conference} } In complex task domains, such as games, students may exceed their teachers. Such tasks afford diverse means to tradeoff one type of performance for another, combining task elements in novel ways to yield method variations and strategy discoveries that, if mastered, might produce large or small leaps in performance. For the researcher interested in the development of extreme expertise in the wild, the problem posed by such tasks is ``where to look'' to capture the explorations, trials, errors, and successes that eventually lead to the invention of superior performance. In this paper, we present several successful discoveries of methods for superior performance. For these discoveries we used Symbolic Aggregate Approximation as our method of identifying changepoints within score progressions in the venerable game of Space Fortress. By decomposing performance at these changepoints, we find previously unknown strategies that even the designers of the task had not anticipated. |
2008 |
Veksler, Bella Z; Gray, Wayne D Counting sheep is a good way to get to sleep, but the occasional aardvark will wake you up: How a salient event improves performance Incollection Sloutsky, Vladimir ; Love, Brad ; McRae, Ken (Ed.): 30th Annual Meeting of the Cognitive Science Society, pp. 637-642, Cognitive Science Society, Austin, TX, 2008. Abstract | BibTeX | Tags: alertness, boredom, oddball paradigm, performance, stimulus salience, sustained attention, vigilance @incollection{bella08csc, title = {Counting sheep is a good way to get to sleep, but the occasional aardvark will wake you up: How a salient event improves performance}, author = { Bella Z. Veksler and Wayne D. Gray}, editor = {Sloutsky, Vladimir and Love, Brad and McRae, Ken}, year = {2008}, date = {2008-01-01}, booktitle = {30th Annual Meeting of the Cognitive Science Society}, pages = {637-642}, publisher = {Cognitive Science Society}, address = {Austin, TX}, abstract = {Sustaining attention is a problem many of us face in various settings from the workplace to school. Sustaining attention is often crucial for good performance. Vigilance decrement refers to a decline in task performance as time on task increases. The current study induces a vigilance decrement across 2000 trials of a monotonous task. In this study each trial is composed of one target and one distracter presented to the participant and the participant's goal is to determine if the target is above or below the distracter. In the control condition, both stimuli are always presented in black font. For the experimental condition, on trial 1200 and 1900 the target is presented in red font. There are no other differences between conditions. This minor, task irrelevant change suffices to reduce the vigilance decrement relative to the control condition on all subsequent trials. We dub this phenomenon the Aardvark Effect.}, keywords = {alertness, boredom, oddball paradigm, performance, stimulus salience, sustained attention, vigilance}, pubstate = {published}, tppubtype = {incollection} } Sustaining attention is a problem many of us face in various settings from the workplace to school. Sustaining attention is often crucial for good performance. Vigilance decrement refers to a decline in task performance as time on task increases. The current study induces a vigilance decrement across 2000 trials of a monotonous task. In this study each trial is composed of one target and one distracter presented to the participant and the participant's goal is to determine if the target is above or below the distracter. In the control condition, both stimuli are always presented in black font. For the experimental condition, on trial 1200 and 1900 the target is presented in red font. There are no other differences between conditions. This minor, task irrelevant change suffices to reduce the vigilance decrement relative to the control condition on all subsequent trials. We dub this phenomenon the Aardvark Effect. |