Another thanks to Armchair Generalist for the pointer to this analysis of Libya's abandonment of its nuclear program. The conclusion: the invasion of Iraq had, at most, a marginal effect on this decision.
Libya has been increasingly isolated since the collapse of the USSR, its chief patron, and the Lockerbie bombing, Qaddafi's biggest miscalculation. Like Castro, Qaddafi has been focused on survival since the Soviet Union evaporated. Unlike North Korea, Libya never had a large enough military to maintain the threat to its neighbors, even as its tanks, artillery, aircraft, and economy decayed. Unfortunately, also like North Korea, Libya did business with the A.Q. Khan network. The Libyan regime abandoned its nuclear program long before it progressed to where its Iranian, North Korean, and (of course) Pakistani counterparts are today.
Ironically for an age when Americans dread the combination of nuclear weapons and terrorism, it was terrorism that undid Libya's nuclear program. Libya was already a pariah before Pan Am 103 exploded over Scotland. Libya's invasion of Chad turned into a farce; Qaddafi's anti-Western posturing led to a humiliating series of economic sanctions and tangles with the US military; his arrogance isolated him from other secular regimes in the region; his attempts to dress himself in the garb of a virtuous Islamic ruler fooled no one, including Al Qaeda, which began plotting against his regime. Libya's sponsorship of the Lockerbie bombing took the enmity against Qaddafi to a new level, ending any chance that Europe could prove slightly more tolerant of Libya than the United States would.
Qaddafi understood how tenuous his own position was, since his blunderings increased the possibility of a military coup against him. He also realized how little the nuclear weapons program gained him. With the intense scrutiny of multiple Western intelligence agencies turned on Libya, its leaders understood how little chance they had of perfecting nuclear technology, acquiring the necessary materials, building the bombs, and delivering them to any targets. The delivery problem was especially acute: Libyan planes would likely be intercepted by the US Sixth Fleet long before they reached any targets in Europe; Libyan efforts to smuggle a "suitcase bomb" into a foreign country would likely be detected and stopped by European, Israeli, or American spies. The nuclear weapons program had turned into a far greater risk than a potential benefit, with the United States and its allies waiting for a reason to end the Libyan threat once and for all, and Qaddafi's own government equally likely to turn on him if he bungled again.
Therefore, Qaddafi began a fifteen-year dialogue with his bitter enemies, the Americans and Europeans, to get out of the trap of his own creation. Libya's admission in court that it had been responsible for the Lockerbie atrocity, and its agreement to pay restitution to the victims' families, was a far more important landmark in the recent history of Libya than the invasion of Iraq. Libya was already straining to prove to the West that it could be trusted. While in public, one set of Libyan representatives were making amends to the Lockerbie families, in private, another group of Libyan representatives were working with American and European officials to expose, document, and dismantle their nuclear program. (It's important to note the roles that Nelson Mandela and Kofi Annan played in these negotiations. Aside from keeping the negotiations moving, Mandela and Annan helped Qaddafi save face, by claiming he was negotiating with the UN, not directly capitulating to the Americans and Europeans.)
The architects of the Iraq invasion—Feith, Wolfowitz, Perle, and others—believed that the US attack would have a powerful "demonstration effect" on other radical, anti-Western regimes. Their attempts to take credit for Libya's abandonment of nuclear weapons aside, the real Libyan story belies the "demonstration effect" theory. The end of Libya's nuclear ambition wasn't a sudden decision, but the result of a decade and a half of patient, diligent diplomacy. The United States did not rely on coercion alone, but instead followed a recipe of inducements and punishments, brewing under a lid of secrecy and intelligence-gathering. The United States did not have to act unilaterally; instead, the US worked in concert with European and Middle Eastern governments to seal off all decisions for Qaddafi but the one the American leaders wanted him to take. Once Qaddafi agreed, rather than be left wondering what to do next, the United States handed much of the remaining work to a Dutch court, UN diplomats, NATO military and intelligence experts, and other representatives of established and trusted organizations. In the end, the United States did not have to pay the political, economic, and military costs of the Iraq invasion and occupation, but could point to a genuine victory at very low cost against a once-troublesome foe.