Vandals also scrawled "F*** Islam" on a wall outside the mosque in Thornlie, Perth, which is adjacent to the Australian Islamic College. Hundreds of local Muslims had gathered there for evening prayers.
No one was injured in the attack.
Thornlie Mosque Imam Yahya Adel Ibrahim told CNN worshipers had run outside when they heard one of the cars explode.
"Most people were in dismay and alarm, (asking), 'why us really? Don't they know that there's children inside?' This is a place of prayer," he said.
"The fear and apprehension -- the neighborhood is quite traumatized by it. This is a residential area, there are homes ten feet across the road."
"This, undoubtedly, is a criminal act of hate," he wrote.
Australians condemn mosque attack
A spokeswoman for West Australian police said three people were seen running down an alleyway next to the college after the attack.
Speaking at a press conference on Wednesday, Australian Prime Minister Malcolm Turnbull told reporters he "cannot condemn strongly enough any attacks of that kind."
Ibrahim said there had been an outpouring of support from the Perth community in the past 24 hours, from people of all faiths and ages.
"(After the attack) the imam continued to lead the prayers and everyone stayed behind," he said. "It was not just an act of defiance but an act of therapy and self help... I'm sure the mosque will have many more people there today"
Hundreds of people condemned the attacks on social media, describing them as "disgusting" and cowardly.