gosh, I'm the opposite. Sex, I could potentially forgive sex - for almost exactly the reasons you gave, Chaeli! It's just physical contact, if the other partner doesn't offer physical fulfillment/affection generally, then the person might look elsewhere to feel 'loved'.
Emotional.... god that's awful. If it's just sex then you can pretend like they were imagining you during the act, that they would have preferred to be sleeping with you but that for whatever reason you weren't available, whereas emotional betrayal is like they're laughing at you behind your back, like they have secret 'in' jokes, like their love is worth more than what you had with your partner.
Either way in my mind, a prolonged, emotionally involving affair is way worse than a one-off.
I think that I have this view for two reasons. The first is because I've sought someone at the lowest point in my worst relationship - I made out with someone else, simply because my bf was denying me sexual intimacy and I can't describe to you how unattractive and unloved I felt. This other person came along and made me feel sexy and wanted and loved, and it had been so long since I felt like that that I couldn't help myself. Granted, my relationship was already on the rocks, granted, I broke up with him following that incident, but the fact remains I felt absolutely awful, even though it was just a kiss, so it didn't evolve into anything more.
I've also been cheated on, and the thing that horrified me the most, was the thought of them together, laughing at how stupid I was to not realise what they were doing.
A drunken pash or a one night stand is one thing, but emotional intimacy is another. I pride myself on being the most easily approachable, most open to explanation and discussion person you could meet, so I think I would find it extremely offensive if they offered me the whole 'you were unapproachable' excuse. In my case that's pretty much impossible, haha, so it would make me furious if they told me that I think!
anyway, sorry for rambling on!