Cowell, 49, recently spoke to TV Guide to deny Paula Abdul's allegations that he whispers in her ear to disorient her right before American Idol's live show comes back from commercial to cause her to appear even more fruity.

He does admit that he's "guilty!" of whispering to her.

"I've done it from Day 1. That's part of the relationship I've had with Paula [and] I've looked upon it in a kind of a fun way," Cowell says. "It's never done with any maliciousness."

He also says that Abdul, 46, probably didn't mean to come off as so dramatic about his behavior when she was talking to Barbara Walters.

This is the same interview that included Abdul's accusations that the Idol producers put her in danger when allowing suicidal Abdul stalker Paula Goodspeed to audition for AI in January of 2006.

Simon says that Abdul's comments weren't "as barbed as they may have appeared" and "she's never really had an issue with me about it. If I thought I've gone too far I've apologized. And if she said to me, 'I don't want you to do it again,' I wouldn't do it."

Here's Simon taking a break from NOT psychologically torturing his co-worker in the Caribbean. Nothing quite like enjoying the beautiful azure water with a coffin nail firmly in mouth.