*Organizational Change *Program Evaluation Monitoring Organizational Behavior [3660]. Human. ACT-R associative strength cognitive architectures cognitive modeling cognitive skill acquisition Common Lisp computational linguistics dips DMAP EPIC evaluation expertise extreme expertise GOMS GOMS/Analysis methods/Empirical studies/User models/Cognitive models/Methods for analysis or assessment/Prototyping/Protocol analysis/Theory in HCI interaction design JSON leaps Measures of Semantic Relatedness natural language processing performance plateaus Rational analysis Semantic similarity measures soar Space Fortress Tetris tutorial VCR
2017 |
Gray, Wayne D; Lindstedt, John K Plateaus, Dips, and Leaps: Where to Look for Inventions and Discoveries during Skilled Performance Journal Article Cognitive Science, 41 (7), pp. 1838-1870, 2017. Abstract | Links | BibTeX | Tags: BreakOut, changepoint detection, digit span, dips, expertise, extreme expertise, leaps, performance, plateaus, Space Fortress @article{gray17csj-pdl, title = {Plateaus, Dips, and Leaps: Where to Look for Inventions and Discoveries during Skilled Performance}, author = {Wayne D. Gray and John K. Lindstedt}, url = {http://homepages.rpi.edu/~grayw/pubs/papers/2017/gray17csj-pdl.pdf}, doi = {10.1111/cogs.12412}, year = {2017}, date = {2017-10-18}, journal = {Cognitive Science}, volume = {41}, number = {7}, pages = {1838-1870}, abstract = {The framework of plateaus, dips, and leaps shines light on periods when individuals may be inventing new methods of skilled performance. We begin with a review of the role performance plateaus have played in (a) experimental psychology, (b) human--computer interaction, and (c) cognitive science. We then reanalyze two classic studies of individual performance to show plateaus and dips which resulted in performance leaps. For a third study, we show how the statistical methods of Changepoint Analysis plus a few simple heuristics may direct our focus to periods of performance change for individuals. For the researcher, dips become the marker of exploration where performance suffers as new methods are invented and tested. Leaps mark the implementation of a successful new method and an incremental jump above the path plotted by smooth and steady log--log performance increments. The methods developed during these dips and leaps are the key to surpassing one's teachers and acquiring extreme expertise.}, keywords = {BreakOut, changepoint detection, digit span, dips, expertise, extreme expertise, leaps, performance, plateaus, Space Fortress}, pubstate = {published}, tppubtype = {article} } The framework of plateaus, dips, and leaps shines light on periods when individuals may be inventing new methods of skilled performance. We begin with a review of the role performance plateaus have played in (a) experimental psychology, (b) human--computer interaction, and (c) cognitive science. We then reanalyze two classic studies of individual performance to show plateaus and dips which resulted in performance leaps. For a third study, we show how the statistical methods of Changepoint Analysis plus a few simple heuristics may direct our focus to periods of performance change for individuals. For the researcher, dips become the marker of exploration where performance suffers as new methods are invented and tested. Leaps mark the implementation of a successful new method and an incremental jump above the path plotted by smooth and steady log--log performance increments. The methods developed during these dips and leaps are the key to surpassing one's teachers and acquiring extreme expertise. |
Sibert, Catherine ; Gray, Wayne D; Lindstedt, John K Interrogating Feature Learning Models to Discover Insights Into the Development of Human Expertise in a Real-Time, Dynamic Decision-Making Task Journal Article Topics in Cognitive Science, 9 (2), pp. 374–394, 2017. Abstract | Links | BibTeX | Tags: Cognitive skill, cognitive skill acquisition, Cross-entropy reinforcement learning, expertise, Experts, Machine learning, Methods, Perceptual learning, Strategies, Tetris @article{sibert17topiCS.gxp, title = {Interrogating Feature Learning Models to Discover Insights Into the Development of Human Expertise in a Real-Time, Dynamic Decision-Making Task}, author = {Sibert, Catherine and Gray, Wayne D. and Lindstedt, John K.}, url = {http://homepages.rpi.edu/~grayw/pubs/papers/2017/sibert17topics.gxp.pdf}, doi = {10.1111/tops.12225}, year = {2017}, date = {2017-04-15}, journal = {Topics in Cognitive Science}, volume = {9}, number = {2}, pages = {374--394}, abstract = {Tetris provides a difficult, dynamic task environment within which some people are novices and others, after years of work and practice, become extreme experts. Here we study two core skills; namely, (a) choosing the goal or objective function that will maximize performance and (b) a feature-based analysis of the current game board to determine where to place the currently falling zoid (i.e., Tetris piece) so as to maximize the goal. In Study 1, we build cross-entropy reinforcement learning (CERL) models (Szita & Lorincz, 2006) to determine whether different goals result in different feature weights. Two of these optimization strategies quickly rise to performance plateaus, whereas two others continue toward higher but more jagged (i.e., variable) heights. In Study 2, we compare the zoid placement decisions made by our best CERL models with those made by 67 human players. Across 370,131 human game episodes, two CERL models picked the same zoid placements as our lowest scoring human for 43% of the placements and as our three best scoring experts for 65% of the placements. Our findings suggest that people focus on maximizing points, not number of lines cleared or number of levels reached. They also show that goal choice influences the choice of zoid placements for CERLs and suggest that the same is true of humans. Tetris has a repetitive task structure that makes Tetris more tractable and more like a traditional experimental psychology paradigm than many more complex games or tasks. Hence, although complex, Tetris is not overwhelmingly complex and presents a right-sized challenge to cognitive theories, especially those of integrated cognitive systems.}, keywords = {Cognitive skill, cognitive skill acquisition, Cross-entropy reinforcement learning, expertise, Experts, Machine learning, Methods, Perceptual learning, Strategies, Tetris}, pubstate = {published}, tppubtype = {article} } Tetris provides a difficult, dynamic task environment within which some people are novices and others, after years of work and practice, become extreme experts. Here we study two core skills; namely, (a) choosing the goal or objective function that will maximize performance and (b) a feature-based analysis of the current game board to determine where to place the currently falling zoid (i.e., Tetris piece) so as to maximize the goal. In Study 1, we build cross-entropy reinforcement learning (CERL) models (Szita & Lorincz, 2006) to determine whether different goals result in different feature weights. Two of these optimization strategies quickly rise to performance plateaus, whereas two others continue toward higher but more jagged (i.e., variable) heights. In Study 2, we compare the zoid placement decisions made by our best CERL models with those made by 67 human players. Across 370,131 human game episodes, two CERL models picked the same zoid placements as our lowest scoring human for 43% of the placements and as our three best scoring experts for 65% of the placements. Our findings suggest that people focus on maximizing points, not number of lines cleared or number of levels reached. They also show that goal choice influences the choice of zoid placements for CERLs and suggest that the same is true of humans. Tetris has a repetitive task structure that makes Tetris more tractable and more like a traditional experimental psychology paradigm than many more complex games or tasks. Hence, although complex, Tetris is not overwhelmingly complex and presents a right-sized challenge to cognitive theories, especially those of integrated cognitive systems. |
Gray, Wayne D Plateaus and Asymptotes: Spurious and Real Limits in Human Performance Journal Article Current Directions in Psychological Science, 26 (1), pp. 59-67, 2017. Abstract | Links | BibTeX | Tags: asymptotes, cognitive skill acquisition, expertise, memory, performance, plateaus, spurious limits @article{gray17cdps, title = {Plateaus and Asymptotes: Spurious and Real Limits in Human Performance}, author = {Gray, Wayne D.}, url = {http://homepages.rpi.edu/~grayw/pubs/papers/2017/gray17cdps.pdf}, doi = {10.1177/0963721416672904}, year = {2017}, date = {2017-02-15}, journal = {Current Directions in Psychological Science}, volume = {26}, number = {1}, pages = {59-67}, abstract = {One hundred twenty years ago, the emergent field of experimental psychology debated whether plateaus of performance during training were real or not. Sixty years ago, the battle was over whether learning asymptoted or not. Thirty years ago, the research community was seized with concerns over stable plateaus at suboptimal performance levels among experts. Applied researchers viewed this as a systems problem and referred to it as the paradox of the active user. Basic researchers diagnosed this as a training problem and embraced deliberate practice. The concepts of plateaus and asymptotes and the distinction between the two are important as the questions asked and the means of overcoming one or the other differ. These questions have meaning as we inquire about the nature of performance limits in skilled behavior and the distinction between brain capacity and brain efficiency. This article brings phenomena that are hiding in the open to the attention of the research community in the hope that delineating the distinction between plateaus and asymptotes will help clarify the distinction between real versus ``spurious limits'' and advance theoretical debates regarding learning and performance.}, keywords = {asymptotes, cognitive skill acquisition, expertise, memory, performance, plateaus, spurious limits}, pubstate = {published}, tppubtype = {article} } One hundred twenty years ago, the emergent field of experimental psychology debated whether plateaus of performance during training were real or not. Sixty years ago, the battle was over whether learning asymptoted or not. Thirty years ago, the research community was seized with concerns over stable plateaus at suboptimal performance levels among experts. Applied researchers viewed this as a systems problem and referred to it as the paradox of the active user. Basic researchers diagnosed this as a training problem and embraced deliberate practice. The concepts of plateaus and asymptotes and the distinction between the two are important as the questions asked and the means of overcoming one or the other differ. These questions have meaning as we inquire about the nature of performance limits in skilled behavior and the distinction between brain capacity and brain efficiency. This article brings phenomena that are hiding in the open to the attention of the research community in the hope that delineating the distinction between plateaus and asymptotes will help clarify the distinction between real versus ``spurious limits'' and advance theoretical debates regarding learning and performance. |
2016 |
Destefano, Marc ; Gray, Wayne D Proceedings of the 38th Annual Conference of the Cognitive Science Society, Cognitive Science Society, Austin, TX, 2016. Abstract | Links | BibTeX | Tags: changepoint analysis, dips, expertise, leaps, method invention, performance, plateaus, SAX, skill acquisition, Space Fortress, strategy discovery @conference{marc16csc, title = {Where Should Researchers Look for Strategy Discoveries during the Acquisition of Complex Task Performance? The Case of Space Fortress}, author = {Destefano, Marc and Gray, Wayne D.}, editor = {Papafragou, A. and Grodner, D. and Mirman, D. and Trueswell, J. C.}, url = {http://homepages.rpi.edu/~grayw/pubs/papers/2016/marc16csc.pdf}, year = {2016}, date = {2016-08-05}, booktitle = {Proceedings of the 38th Annual Conference of the Cognitive Science Society}, publisher = {Cognitive Science Society}, address = {Austin, TX}, abstract = {In complex task domains, such as games, students may exceed their teachers. Such tasks afford diverse means to tradeoff one type of performance for another, combining task elements in novel ways to yield method variations and strategy discoveries that, if mastered, might produce large or small leaps in performance. For the researcher interested in the development of extreme expertise in the wild, the problem posed by such tasks is ``where to look'' to capture the explorations, trials, errors, and successes that eventually lead to the invention of superior performance. In this paper, we present several successful discoveries of methods for superior performance. For these discoveries we used Symbolic Aggregate Approximation as our method of identifying changepoints within score progressions in the venerable game of Space Fortress. By decomposing performance at these changepoints, we find previously unknown strategies that even the designers of the task had not anticipated.}, keywords = {changepoint analysis, dips, expertise, leaps, method invention, performance, plateaus, SAX, skill acquisition, Space Fortress, strategy discovery}, pubstate = {published}, tppubtype = {conference} } In complex task domains, such as games, students may exceed their teachers. Such tasks afford diverse means to tradeoff one type of performance for another, combining task elements in novel ways to yield method variations and strategy discoveries that, if mastered, might produce large or small leaps in performance. For the researcher interested in the development of extreme expertise in the wild, the problem posed by such tasks is ``where to look'' to capture the explorations, trials, errors, and successes that eventually lead to the invention of superior performance. In this paper, we present several successful discoveries of methods for superior performance. For these discoveries we used Symbolic Aggregate Approximation as our method of identifying changepoints within score progressions in the venerable game of Space Fortress. By decomposing performance at these changepoints, we find previously unknown strategies that even the designers of the task had not anticipated. |
2011 |
Destefano, Marc; Lindstedt, John K; Gray, Wayne D Use of complementary actions decreases with expertise Incollection Carlson, Laura ; H"olscher, Christoph ; Shipley, Thomas (Ed.): Proceedings of the 33rd Annual Conference of the Cognitive Science Society, pp. 2709-2014, Cognitive Science Society, Austin, TX, 2011. Abstract | BibTeX | Tags: complementary action, embodied cognition, epistemic action, expertise, games, pragmatic action, soft constraints hypothesis, Tetris @incollection{marc11csc, title = {Use of complementary actions decreases with expertise}, author = { Marc Destefano and John K. Lindstedt and Wayne D. Gray}, editor = {Carlson, Laura and H"olscher, Christoph and Shipley, Thomas}, year = {2011}, date = {2011-01-01}, booktitle = {Proceedings of the 33rd Annual Conference of the Cognitive Science Society}, pages = {2709-2014}, publisher = {Cognitive Science Society}, address = {Austin, TX}, abstract = {Evidence that the use of complementary (or epistemic) actions increases with expertise in the fast-paced interactive video game of Tetris has been previously reported (Kirsh, 1995; Kirsh & Maglio, 1994; Maglio & Kirsh, 1996). However, the range of expertise considered was small and classifying such actions can be difficult. We sample across a wide range of Tetris expertise and define complementary actions across multiple criterion of varying strictness. Contrary to prior work, our data suggest that complementary actions decrease with expertise, regardless of the criteria used. These findings cast into doubt the accepted wisdom on the role of complementary actions in expertise.}, keywords = {complementary action, embodied cognition, epistemic action, expertise, games, pragmatic action, soft constraints hypothesis, Tetris}, pubstate = {published}, tppubtype = {incollection} } Evidence that the use of complementary (or epistemic) actions increases with expertise in the fast-paced interactive video game of Tetris has been previously reported (Kirsh, 1995; Kirsh & Maglio, 1994; Maglio & Kirsh, 1996). However, the range of expertise considered was small and classifying such actions can be difficult. We sample across a wide range of Tetris expertise and define complementary actions across multiple criterion of varying strictness. Contrary to prior work, our data suggest that complementary actions decrease with expertise, regardless of the criteria used. These findings cast into doubt the accepted wisdom on the role of complementary actions in expertise. |