| Research
> Configural Memory with Network Reinforcement Learning (CMNRL) |
Researchers |
|
|
|
| Abstract |
Configural Memory with Network Reinforcement Learning (CMNRL, pronounced Sea Mineral) is a categorization-based cognitive architecture and an autonomous agent. It is an unsupervised incremental neural network with two main components. The first component, configural memory, is similar to the configural approaches of Gluck & Bower (1988) and Heydemann (1995). Configural approaches have been used to model a wide variety of psychological data (e.g. Pearce, 1994). The second component of CMNRL, Network Reinforcement Learning (NRL) extends traditional reinforcement learning (Sutton & Barto, 1998) by allowing for simultaneous updates of multiple state-action pairs. Just as configural memory, reinforcement learning has been affirmed as a psychologically and biologically plausible mechanism (e.g. Holroyd & Coles, 2002).
|
| Description |
Categorization and Automaticity
The primary focus of CMRNL is rational action through categorization. It is surprising that some of the most prominent cognitive architectures (ACT-R, SOAR, etc.) have no inherent mechanism of categorization. The ability to divide the world into categories is central to cognition (e.g. Rosch, Mervis, Gray, Johnson, & Boyes-Braem, 1976), and is found in both human and lower animals.
Connectionist architectures are more concerned with categorization than production systems. However, traditional neural networks have two shortcomings – supervised learning and pre-specified topology. CMNRL, on the other hand, does not require any human supervision, nor does it need to be pre-wired for a given task. With mere specification of sensory and motor nodes, CMNRL is ready to start exploring its world. This out-of-the-box automaticity separates CMNRL from most other cognitive architectures.
Configural Memory
Configural memory supposes that sensory nodes can be combined into configurations. Suppose the agent has three sensory nodes: one that is activated by square objects, one by red objects, and one by large objects. In this case there may be a configural node that is activated by red square objects, one that is activated by red large objects, one by large square objects, and one by red large square objects. The problem with creating all possible configural nodes is that the number of possible configurations explodes with a growing number of sensory nodes. This problem is resolved in IAK approach to configural memory (Heydemann, 1995) with probabilistic sampling of configurations. In accordance with this approach, CMNRL grows configural nodes based on statistical co-occurrence of features in the environment. The connections from parent sensory nodes to child configural nodes increase in Hebbian fashion with every co-occurrence of parent nodes, and decay with time. Together, these two configural learning mechanisms – Hebbian learning and connection decay, are called Incremental Chunking.
Network Reinforcement Learning
The basic idea behind reinforcement learning is that every state-action pair has a utility value that gets updated with a reinforcement value supplied by the environment (pleasure/pain). Given that multiple nodes are active at the same time in a configural memory network (upon seeing a white square object, the {square}, {white}, and {white square} nodes will all be activated), there are multiple winning state-action pairs after every chosen action. NRL updates the utility values for all state-action pairs SxAy, where Sx is one of the active sensory/configural nodes, and Ay is the active action node.
Future Directions
Current work in testing and advancing CMNRL is focused on fitting human/animal data from prominent psychology paradigms. In the near future we will be examining CMRNL using text comprehension and game domains.
References
Gluck, M. A., & Bower, G.H. (1988). From conditioning to category learning: An adaptive network model. Journal of Experimental Psychology: General, 117, 227-247.
Heydemann, M. (1995). A connectionist model for classification learning – the IAK model. In Proceedings of the seventeenth annual conference of the Cognitive Science Society, 293-297. Mahwah, New Jersey: Lawrence Erlbaum Associates.
Holroyd, C. B., & Coles, M. G. (2002). The neural basis of human error processing: reinforcement learning, dopamine, and the error-related negativity. Psychological Review, 109(4), 679-709.
Pearce, J. M. (1994). Similarity and discrimination: a selective review and a connectionist model. Psychological Review, vol. 101(4), 587–607.
Rosch, E., Mervis, C. B., Gray, W. D., Johnson, D. M., Boyes-Braem, P. (1976). Basic objects in natural categories. Cognitive Psychology, 8, 382-439.
Sutton, R. S., & Barto, A. G. (1998). Reinforcement Learning: An Introduction. MIT Press, Cambridge, Mass.
|
|
