When it's fun. Seriously, that's all. If playing this game with these people is rewarding, then you either do it more to enjoy it some more, or perhaps you say "we've done it" and stop happy. During play, when you're talking, listening, and enjoying both, then you're being rewarded. When you can look back at a session or game experience in general and feel good about it socially and creatively, then you were rewarded.
'Why it is or was fun ranges widely. My take is that play is maximally rewarding when a shared Creative Agenda is reliably present.
Not all role-playing is rewarding. Sometimes people play because they hope it will be rewarding some day; others gave up that hope and play for totally different reasons like merely hanging out (Zilchplay); and horrifyingly, some play in the grip of twisted or subverted hope (Dysfunction).
The term Reward system does not necessarily indicate any formal mechanics at work. It refers to a particular group's established way of recognizing their own fun in play, whether it's using rules-based or mechanics-based methods or not. Some group's reward system is found only in moments of appreciative hilarity during play and fond recollections afterwards. Others' may be found in applying specific textual rules which tie directly into the emerging plot. It depends greatly on the game, but primarily on what the people are actually doing.
Color and Reward
These two features (see Color) are perhaps the most global principles of role-playing. You start with all Color and no Reward yet; you end with or without genuine Reward. What happens in between is all about whether the Reward occurs and in what way.
... are not quite the same thing. These Techniques serve to focus and reinforce a particular application of the Creative Agenda, or perhaps some aspect of play which is less global or constant but which is still relevant to it.
Character improvement is probably the most common, historically
Character damage and Character death, when they are not aversive or "punishment" mechanics
Bonus mechanics of various kinds
Opportunities for complication or closure
Some games include incentive mechanisms which can ultimately become empty, e.g., "refer to your character's motivation and gain a Story Point! Spend a Story Point to get a re-roll!" These are effectively intended reward mechanics which fail to generate genuine reward.