But the process of holding people responsible for some of the devastation is just beginning.
Three people were arrested Tuesday, Xiao Bo-ren, the Tainan government's legal affairs secretary, told CNN. All are former executives of the real estate development company that built Tainan's 17-story Weiguan Golden Dragon tower, which collapsed when the magnitude-6.4 tremor hit early Saturday.
Lin Ming-hui, former chairman of the now defunct Weiguan company, and two other former executives -- Chang Kui-an and Cheng Chin-kui -- face charges of professional negligence resulting in death.
The ruins of the toppled high-rise are the most visible evidence of the massive quake, which killed at least 40 people in the normally quiet city of 1.9 million people, according to the Tainan City Disaster Response Center. Four days later, more than 100 people remained missing.
Heavy machinery began drilling around the site Tuesday, starting a new phase of removing debris layer by layer, exposing a mass of tangled wires, broken glass and shards of cement. The hope is that this will let authorities get to any survivors more easily, though many onlookers feared it could kill them by causing concrete slabs to collapse.
"If there are still people breathing in there and they are harmed now, after surviving for days, that would be the greatest tragedy," said Mao Jiecheng, who had rushed from Taipei to volunteer after the quake.
Lin Jianfang is there, too, camping out around the clock with three generations of relatives. The rescue effort is heart-wrenchingly personal for them since Lin's brother, a 41-year-old engineer, was asleep in his apartment when the quake struck and hasn't been heard from since.
Clutching his wife's hand and coughing as pulverized building debris wafted through the air, Lin said, "We refuse to give up hope."
Restaurants, hotels, volunteers pitch in
They're hoping for a miracle like the discovery of an 8-year-old girl and her aunt almost 72 hours after they were buried, as reported by Taiwan's SETTV. Military doctor Li Biqing told reporters that the girl was in "normal condition" despite her ordeal.
Tainan's fire department said that about 103 people -- dead or alive -- were believed to be trapped in the Golden Dragon tower's ruins. Some are students who attended nearby Kun Shan University.
A woman named Li -- she wouldn't give her full name -- has been urgently waiting these last few days for news of her 20-year-old nephew.
"We absolutely will keep waiting -- we cannot give up hope, we won't let our hearts go," Li said tearfully. "We never thought something this terrible could happen to our beloved Tainan, and in the main city area -- what are the chances?"
The Golden Dragon tower is located in Tainan's most densely populated neighborhood, according to city government statistics. That area has been both devastated and transformed by the collapse, not only by the swarms of rescue workers but by the businesses and residents who have converged to do what they can.
A pub, for instance, has been repurposed as an emergency response center. Volunteers have set up tents next door at a colorful temple -- a bright contrast to the grim, gray debris -- to distribute water, blankets, clothing and hot food for rescuers and relatives like Li and Lin. Restaurants have stayed open to offer food and shelter as well, while hotels are giving free accommodation to people who have lost their homes.
Emergency medical and military vehicles dot Yongda Road, where the building once stood, in parallel with bright paper lanterns hung up to celebrate the usually joyous Chinese New Year holiday.
It's the biggest annual celebration here, a time for family, friends, laughter and feasts, but there is no festive spirit in the air this year.
"We are too somber to celebrate the holiday," said Jerrie Wu, a volunteer who has been handing out snacks and hot drinks to those working on the scene.
'So many other buildings ... still standing'
Condolences have poured in from around the world, including from U.S. senators, Pope Francis and celebrities like tennis great Venus Williams.
"We all join our Taiwanese-American community here in Los Angeles to mourn this tragic loss of life," Los Angeles Mayor Eric Garcetti said. "When we witness horrific incidents like this, we are reminded of the immense destruction a major earthquake can cause."
Amid the sadness, there's also anger and uncertainty about what happened with the Golden Dragon tower, especially given how most of the rest of Tainan escaped relatively unscathed.
Central News Agency reported that "jerry-building" -- cheap or flimsy construction -- "is believed to be one of the causes of the collapse."
Images that have surfaced of tin cans believed to be used in the construction of the tower have caused city residents to speculate about whether the building company cut corners when the high-rise went up.
"There are so many other older buildings in Tainan that are still standing. Why was it only this building that was completely destroyed?" asked Wang Xingyou, a city cab driver.
Shaking his head woefully, he sighed deeply.
"This is not a happy new year for Tainan."
CNN's Sophia Yan reported from Tainan, and CNN's Greg Botelho reported and wrote from Atlanta.