By Mariam Karouny

(Reuters) - With a confident smile, the leader of Lebanon's Hezbollah movement warned in a recent interview that allies of Syrian President Bashar al-Assad would strike back if Israeli attacks inside Syria continued."This attack shows that Israel has crossed the red line in the security war with Hezbollah, which means the rules have changed," said a senior security source close to the group.

The attack could have repercussions. It has put Hezbollah under pressure to respond, sources close to the group said, and also put a ceasefire between Israel and Syria at risk. The group's leadership has yet to comment.

Hezbollah considers the Jewish state its main enemy. But its fighters in the Syrian province of Quneitra, where the Israeli attack took place, have turned a blind eye to the presence of Israeli soldiers across the border there.

Israel has struck Syria several times during the present conflict, hitting weapons deliveries to Hezbollah, but the group has never acknowledged those attacks. This time the importance of those killed, in addition to Nasrallah's warning, make the latest raid difficult to ignore.

"It was like an unannounced agreement: 'You ignore us and we ignore you'. Attacks should not rise to full provocation. This attack is a full provocation of Hezbollah," said a security source contacted by Reuters.

"If the group does not respond it means it is stuck in Syria's mud and has lost its deterrent ability."

A Lebanese official close to Hezbollah said: "We should expect retaliation from Hezbollah, but it will be done coldly."

Its options are limited. It could strike from its stronghold in Lebanon, triggering all-out war with Israel. It could attack targets in Israel, but risk a wider war between Israel and Syria, or hit Israeli interests abroad. But all come at a price.

An Israeli defense official told Reuters that a response from Hezbollah was expected, but in the form of limited attacks unlikely to lead to all-out war.

ISRAELI STRIKE

The Hezbollah members were killed in an Israeli strike near the border with Israel, where a ceasefire is in effect between the Jewish state and Syria.

The frontier between Israel and Syria has been administered by the United Nations since 1974, a year after the last war between them. The area has remained quiet, with both sides avoiding provocation.

Hezbollah's commitment to the truce was never made public because it was an agreement between two countries, but that has now changed.

"It has collapsed now," a security source said.

Hezbollah was set up by Iran in the 1980s to fight Israel in Lebanon. It controls large parts of Lebanon, mainly near the southern border with Israel, but its influence has grown beyond Lebanon.

While the Lebanese government distanced itself from the war in Syria, Hezbollah sent fighters to help Assad.

Hezbollah officials privately say that while they have lost fighters in Syria, the impact of the war has been positive, allowing the group to expand its arsenal and train thousands of fighters to operate in unfamiliar territory.

"Our mujahideen are fighting in a foreign country, in lands and geography that we never fought in before that are completely different from the conditions of south Lebanon," a senior Hezbollah official said.

"We are fighting as an army and with allies. We are used to fighting in small groups, so these are major developments in our fighting skills."

POLITICAL DIVISIONS

But with political and sectarian divisions at home, it will be difficult to win backing for any attacks from Lebanon which could trigger war with Israel.

"The rules of the game are to respond outside Lebanon unless the Israelis bring war to Lebanon," a source close to Hezbollah told Reuters, explaining that it wants to avoid all-out war.

Lebanon has not recovered from its 2006 war with Israel and sending fighters to Syria has stretched Hezbollah's capabilities. In addition, the Israeli strike came at a sensitive time for the group.

Hezbollah is trying to contain the damage caused by one of its operatives who confessed to spying for Israel in a case that shattered the group's aura of impenetrability.

The suspected spy is believed to have leaked Hezbollah's plans to avenge the assassination of Imad Moughniyah.

Moughniyah, whose son was killed by Israel last week, was implicated in the 1983 bombings of the U.S. Embassy and U.S. and French barracks in Beirut, and attacks on Israeli targets. Hezbollah accuses Israel of killing him, which Israel denies.

"The group tried to retaliate but was not successful for several reasons. Some of the operations were leaked to the Israelis and others were not possible for technical reasons," said one of the security sources.

Despite that, the group has little choice but to respond.

Lebanese analyst Nabil Boumonsef said Hezbollah could not start a war with Israel. Instead, the response would be a targeted attack that stopped short of full confrontation.

(Additional reporting by Laila Bassam in Beirut and Dan Williams in Jerusalem; editing by Giles Elgood)