Richard, you're absolutely right: Khamenei and Rafsanjani have said some inflammatory things in their day. (Although, as Matt Yglesias notes, you're cherry-picking with that Rafsanjani quote; in context he may have been talking about nuclear deterrence.) Of course, you could fish around and find Khamenei saying non-crazy things too: He's expressly ruled out a first strike against Israel, for instance, and issued a fatwa against using nuclear weapons. Yet those statements are always disregarded, while others are seen as revealing the "true face" of Iran.
So maybe the better option is to look at Iran's actual past behavior rather than mining for quotes. Ray Takeyh's new book, Hidden Iran--which earned a favorable review from Vali Nasr in this magazine--makes a strong case that, historically, Iran has acted quite pragmatically in its foreign affairs. That's why Khamenei and Rafsanjani have the reputation they do, even though everyone agrees that they're loathsome people. Of course, The New Republic tried to get around this point by running a cover story on how Ahmadinejad was totally different and an honest-to-Allah psychopath. But as Juan Cole pointed out, this would be far more troubling if Ahmadinejad actually ran the show.
That's hardly a "cavalier" attitude. Justin Logan of the Cato Institute recently wrote what strikes me as the canonical case against attacking Iran, and it's perfectly hard-headed. No one wants Iran to get nukes--seeing as how India and Pakistan have come close to incinerating each other on occasion, it's scary even when secular regimes get the bomb. But attacking Iran will, at best, merely set their nuclear program back a few years and further entrench the ruling regime in Tehran (and that's if a strike manages to disable all of Iran's nuclear facilities--no small feat). At worst, it will lead to a horrific regional war and make future Iranian nuclear attacks even more likely.
On the other hand, Iran's past behavior--behavior, not various quotes culled from MEMRI--strongly suggests that it can be deterred and even reasoned with. (Indeed, the BBC just reported that Iran offered to cut off all funding for Hezbollah in 2003 in exchange for closer ties with the United States, but Dick Cheney nixed the offer.) So if diplomacy can't convince Iran to abandon its nuclear ambitions--something the Bush administration has never seriously tried--then deterrence is the least-bad option left. Disagree if you want, but don't call it "cavalier". Also note that Iran seems years away from a bomb, giving the United States ample time to at least try diplomacy.
You've also mentioned that Israel can't afford to be as sanguine about Iran's nukes as the United States. Well, true, U.S. interests on this score may not line up perfectly with Israeli interests--for starters, the United States has more to lose from any potential blowback in Iraq, in the event of a military strike. That's worth hashing out, but again, I don't think it's cavalier (or dismissive of Israeli concerns) to do so.