Simply, role-playing which is not fun. Most Forge discussions presume that un-fun role-playing is worse than no role-playing.
Loss of temper. Anyone might have a bad moment and be grumpy, or even speak out angrily, once in a while. The sign of dysfunction is repeated loss of temper about the same things. Storming out of the room - as opposed to "let's talk it over" or "let's take a break" - might be part of this sign.
Wrangling over some logistic component of the imagined situation, especially in terms of whether something that's been announced to happen "could" have happened that way. Ocasional blips or revisions of communication are ordinary; what's dysfunctional is when such discussions (i) arise whenever anything involving game mechanics occurs or might occur; and (ii) take up the majority of play time.
Social divisions regarding who can and cannot contribute significantly to the in-play events. These divisions typically show up in terms of age, sibling interactions, real or potential or imagined romantic interactions, and various geek status indicators.
Checking out. People wander around and do other stuff during play, paying little attention to what is occurring in the in-game events, requiring updates when they tune in again or sometimes not even bothering.
Chris Chinn, 2009 The Roots of the Big Problems