How many cross-dressers have said those words? I've said them myself – and meant them. Meanwhile, significant others and the rest of the world look on, see us getting off, and think "yeah, right". But cross-dressers are not wrong and are not lying. It's just that the language we're all (CDs, SOs, everyone) using is inadequate.
Rather than "sex" and "gender", the crucial (and missing) word, I think, is sexuality. By sexuality I don't mean who you're attracted to (whether you're straight, gay, bi, or whatever); nor what turns you on as part of a sexual encounter (morphology, nakedness, latex, whipped cream, baked beans). These may be part of someone's sexuality, but they don't encompass it. Instead, I'm talking about another aspect: how you feel about yourself, your body, your desire; how you see yourself as a sexual being; and how you express it all.
For male cross-dressers that aspect might be exemplified along these lines: wanting to be the pretty woman in the room, the confident and sexy woman, the one that turns heads, the revamped Olivia Newton-John in Grease (never mind how implausible that thought); to feel that particular female sexual energy, not in a sexual act, but within oneself, as bliss.
To a certain extent a male cross-dresser can do that on his own. He can get dressed up, stand in front of a mirror, give himself the thumbs up (in female Fonz style), turn his own head (a good dose of self-deception is quite helpful there). Hence also, I'd guess, the practice by some CDs (not me in this case) of posting provocative photos online: so as to be the sexy woman in the virtual room, turning the virtual heads, responding flirtatiously to salacious (but anonymous) comments.
Of course, when a man's sexual feelings are heightened he may want to do something about it – and for a man by himself (with or without computer interaction) there is an obvious solution. But that, I think, is a consequence of his feelings, not the impetus for them. To put it more bluntly: he's not dressing in order to jerk off; he jerks off because his sexual feelings have been so amplified that he wants or needs to, and because he can. At the start he very likely had no such intentions at all; he just wanted to get dressed up.
So why wear women's clothes? After all, any man can dress to feel good about himself: perhaps powerfully masculine in a smart suit or uniform; cool and confident in t-shirt, jeans and a leather jacket (the Fonz proper); even sexy and gorgeous (handsome) for a night out. Yes, but those feelings are rather different from the ones we're after. I'd also suggest that they're not even male as such, but butch – the adjunct of femme – which brings me back to my blog theme.
Notice my use of the words "female" and "woman" in the third and fourth paragraphs above. That was deliberate; I wanted to put things in more familiar (for cross-dressers) and universal terms. But I'll say again now that "female" and "woman" seem inappropriate; that these are femme (not female) feelings; this particular sexual energy belongs to femmes (not women in general); that it's the femme who turns heads in this particular way.
In the end, for me, it probably is about sex after all, because it's about sexuality. But it's about gender too, because this facet of sexuality is inseparable from gender expression. And the taxonomy that connects these aspects of sex and gender in the most apposite way in this precise context is — wait for it — femme.
(This stuff becomes so much easier when you think about it in femme/butch terms and put woman/man to one side.)