Simply, role-playing which is not fun. You can tell because someone, or more than one, is pissed off, crying, sullen, ominously noncommittal, very bored, or emotionally exhausted from any combination of the foregoing. Extra points for endlessly processing over the game, either during (alleged) play or in complicated ways between sessions. Hey, let's not forget "fuck you" retaliatory behavior during play as the sessions proceed, and frequent absenteeism. Even better? Pretend that everything's fine!
I say that un-fun role-playing is worse than no role-playing. Others seem to disagree, but I am happier than they are.
Loss of temper. Anyone might have a bad moment and be grumpy, or even speak out angrily, once in a while. The sign of dysfunction is repeated loss of temper about the same things. Storming out of the room - as opposed to "let's talk it over" or "let's take a break" - might be part of this sign.
Wrangling over some logistic component of the imagined situation, especially in terms of whether something that's been announced to happen "could" have happened that way. Ocasional blips or revisions of communication are ordinary; what's dysfunctional is when such discussions (i) arise whenever anything involving game mechanics occurs or might occur; and (ii) take up the majority of play time.
Social divisions regarding who can and cannot contribute significantly to the in-play events. These divisions typically show up in terms of age, sibling interactions, real or potential or imagined romantic interactions, and various geek status indicators.
Checking out. People wander around and do other stuff during play, paying little attention to what is occurring in the in-game events, requiring updates when they tune in again or sometimes not even bothering.
A taxonomy of horrors
A failure of Reward. It typically arises from a toxic Social Contract or violations against a decent one, Agenda Clash, and the Murk. It often looks like it's composed of rules disputes, but typically, such disputes are symptoms of one or more these larger problems.
Chris Chinn, 2009 The Roots of the Big Problems