Imam Humayun Islam was forced to flee his community's place of worship at Millars 2, off Southmill Road, and seek refuge in an Indian restaurant when he was chased by the drunken yobs.
The group of about five men, aged in their 20s and 30s, first struck on Monday last week while worshippers were praying for the fourth time, just before 9pm.
The imam said: "They came inside and started throwing our shoes out. They were heavily drunk."
The cleric said that when he remonstrated with the intruders and asked them to leave, he was threatened with a hammer and called a "suicide bomber".
A punch was thrown and the gang started breaking glass in the mosque door, before running off as police arrived.
After the officers left, the imam and his fellow Muslims began clearing up the damage and were horrified to hear the attackers return, this time wielding bottles.
Several worshippers fled and the imam was chased. He managed to avoid harm only by giving the gang the slip and hiding in the takeaway.
On Tuesday, he told the Observer: "It was a serious racist attack."
He said that it was the most frightening of a series of problems for the town's Muslims amid a growing atmosphere of Islamophobia.
His view was echoed by a mosque member, who did not wish to be identified but said: "This attack should be reported because the distress, fear and isolation it has caused us should be known by the public. We are decent and law-abiding members of society and we do not deserve to be treated like this."
The Muslim community which uses the Herts and Essex Islamic Cultural Centre, including some 50 local members and up to 150 from a wider area who attend prayers on Fridays, is locked in a dispute with East Herts District Council over the future of the mosque.
In April, trustees of the building, which was bought two years ago with charitable donations from worshippers, were given four months to quit because they don't have the correct planning permission.
Community members said they felt "picked-on and let down" by the council, with one saying: "The mosque is a very important part of our lives and we only have a few weeks left until we may no longer have a place of worship near our homes."
However, a council spokeswoman said: "We want to work with them to try to find a solution."
The Muslims had to leave their last base, in the former library in Adderley Road, after it was redeveloped.