|The Big Model|
You're role-playing, with some other people. Some character sheets are on the table, maybe some dice and all that stuff. Now consider what is happening fictionally - characters are doing stuff, right? I can watch you for half an hour, and then I can say "Rog the Swordsman and Jenny the Pirate burned the bad guy's ship." Before that half hour, they hadn't; after that half hour, they had.
System is how that fictional event came to be. A good way to look at System is that without it, characters in their settings would merely sit there frozen - once you put System into action, the Shared Imagined Space acquires time, and the fictional situations in play can change.
The core concept which makes System functional is the Lumpley Principle, meaning, it (what happened) only came to be because one or more people spoke together and listened to one another, eventually or continuously producing "what happens next" so everyone can move the fictional events forward.
Don't get me wrong: I'm not talking about negotiating to agree, which is tedious and horrible. I'm talking about genuine System, such that the means of agreeing are functional, ongoing, and effective.
Nuts and bolts
What we call the game mechanics are inside the System, utilized to impose constraints to announcements or to add features under certain conditions. In groups of interlocking parts, these are best understood as Techniques. The Rules are nothing more than the group's accepted Techniques in some officialized format.
This goes double for Rules, which despite the way we talk about them are actually utterly powerless - what has power, or presence or functionality if you prefer, is when we the people at that table agree to be using that particular rule.
The means by which imaginary events are established during play, including character creation, resolution of imaginary events, reward procedures, and more. It may be considered to introduce fictional time into the Shared Imagined Space. See also the Lumpley Principle.
Periodic Refresher (Vincent Baker)
From Applied Theory by Mark J. Young: Mr. Edwards has said that system within a game is the equivalent of time. To understand this, you have to understand something about time: it is the medium for change. Without time, nothing changes. In the game, the system determines what happens, what changes; without it, nothing changes. Thus the system determines and controls change, and therefore is effectively time for the imagined world.