Difficulty in Cessation of Undesired Habits: Goal-Based Reduced Successor Representation and Reward Prediction Errors

Difficulty in cessation of drinking, smoking, or gambling, even with strong intention, has been widely recognized. Reasons for this, and whether there are reasons common to substance and non-substance reward, remain elusive. We present a computational model of common potential mechanisms underlying the difficulty in resisting habitual behavior to obtain reward. Consider that a person has long been regularly taking a series of actions leading to a purchase of alcohol, cigarette, or betting ticket without any hesitation. Referring to the recently suggested representation of states by their successors in human reinforcement learning as well as the dimension reduction in state representations in the brain, we assumed that the person has acquired a rigid representation of states along the series of habitual actions by the discounted future occupancy of the final successor state, namely, the rewarded goal state, under the established non-resistant policy. Then, we show that if the person takes a different policy to resist temptation of habitual behavior, negative reward prediction error (RPE) is generated when s/he makes "No-Go" decisions whereas no RPE occurs upon "Go" decisions, and a large positive RPE is generated upon eventually reaching the goal, given that the state representation acquired under the non-resistant policy is so rigid that it does not easily change. In the cases where the states are instead represented in the punctate manner or by the discounted future occupancies of all the states (i.e., by the genuine successor representation), negative and positive RPEs are generated upon "No-Go" and "Go" decisions, respectively, whereas no or little RPE occurs at the goal. We suggest that these RPEs, especially the large positive RPE generated upon goal reaching in the case with the goal-based reduced successor representation, might underlie the difficulty in cessation of undesired habitual or addictive behavior to obtain substance and non-substance reward.