In his testimony, Kagoike said he believes there was some sort of political intervention in the transaction because the process started moving quicker once he began asking for help.

He later clarified what he meant at a news conference with reporters, saying that he believes people working for Abe -- specifically bureaucrats in the Ministry of Finance, which sold him the land -- helped in the sale. He did not name specific people.

"I don't think there is direct influence from Prime Minister Abe," Kagoike said, according to the real-time translator supplied by The Foreign Correspondents' Club of Japan, which hosted the news conference.

The discount was fair, Kagoike said, though he stressed it was the federal government's decision to reduce the price.

"(It) was not out of the bounds of reality. It took a lot of money to take out the household waste in the land and replace it with good soil," he said.

Prime Minister Abe denies any connection to the land sale, and says he'll step down if it's proven otherwise.

"I have no understanding about the details of what this school is doing," he said in a session of parliament in February. "I only took a brief look at their brochure."

Kagoike said he's withdrawn the application to build the school at the advice of his lawyer despite the fact that construction is almost finished.

"It was a heartbreaking moment," he said. "But after that nothing has improved ... (the) government acted like I am the only bad guy. I'm starting to find it strange."

It's the largest decline since December 2012, when Abe's second Cabinet took office, according to the newspaper.

The Moritomo Gakuen group declined to speak with CNN. Representatives for the Abes did not give specific answers to CNN's inquiries, instead pointing to public comments the Prime Minister has made.

The honorary principal

Abe's wife, Akie, was a prominent supporter of the planned school, serving as "honorary principal" until her abrupt resignation last month.

Her picture and title were listed on the school's website but have since taken down.

Abe told parliament in February his wife resigned due to "various incidents," without going into more detail.

"She thought being honorary principal could result in giving trouble to children and parents," he told lawmakers.

But Kagoike told reporters and politicians the Abes have been supporters of his for a while. He claims Akie Abe donated 1 million yen (about $8,900) on her husband's behalf to Moritomo Gakuen in 2015 -- something the Prime Minister has denied himself and through his spokesman.

The donation isn't necessarily illegal, but if verified it would mean Abe misled lawmakers in a session of parliament.

"I have no idea why he (Kagoike) has said this," Chief Cabinet Secretary Yoshihide Suga said.

During his testimony, Kagoike said Akie Abe gave him the donation when she came to one of his schools to give a speech on September 5, 2015. Akie Abe asked her aide to leave the room during a meeting and then gave him an envelope with the money inside, Kagoike said in his testimony.

"(The donation) was a great honor for us, so we remember it very vividly," he said.

The only piece of hard evidence Kagoike offered was a copy of a fax, but it only proves he communicated with Akie Abe's aide. The document says Kagoike's request couldn't be met but the aide said they would let him know if they could help.

Back-and-forth

Akie Abe took to Facebook on March 23 to deny the allegations.

"I have never given a donation of 1 million yen to Mr. Kagoike, nor received a lecture fee. As for this point, I have been exchanging e-mails with Mrs. Kagoike many times since February, but she has not mentioned that there was a donation or that I received a lecture fee. I clearly communicated to her that I had no memory to that effect," she said. " I don't ask my secretary to leave when I visit for speeches and lectures. I would not do such a thing. I have confirmed with my two secretaries that I have not done such a thing."

The Prime Minister's ruling Liberal Democrat Party jumped on this as an opportunity to discredit Kagoike's testimony. If Akie Abe and Kagoike were alone and the cash was exchanged, there's no way Kagoike could conclusively prove his account about the donation, Prime Minister Abe's allies said.

Suga again denied allegations Thursday that either of the Abes made a donation to Moritomo Gakuen after Kagoike's testimony. He said the government will not ask Akie Abe about it again and reiterated his belief the land deal was done in a fair and legal manner.

Shinzo Abe said Friday it was "extremely regrettable" Kagoike made what the Prime Minister says were false claims about a closed-door conversation that is difficult to prove.

Kagoike said he's now the victim of a "malicious and libelous" smear campaign.

"There's a lot of hysteria around this situation, we're being targeted by people," he told reporters Thursday.

'My way of thinking'

The scandal first broke in February after a local assembly member sued the government for not disclosing the details of the sale.

The Prime Minister was fielding questions about the school by the middle of February, when he denied any connection to the transaction. But he told lawmakers his wife said Kagoike is a passionate educator who shares his "way of thinking very much."

Kagoike said the Abes support his school's philosophy during his Thursday testimony.

"I wonder why the people who has been supportive to my school suddenly turned their back and left me alone," he said. "I won't be cast off like the tail of lizard."

This has caused concern among the Japanese public, as officials at Kagoike's Moritomo Gakuen have been accused of using derogatory language when referring to Chinese and Korean people in letters home to parents.

A note on one school's website blamed "delinquent parents" of "former students coming from C country and K country" for making the letter public.

"Our school stands firm against such actions by those extreme minority from C country and K country living in Japan," Moritomo Gakuen said on its website.

"C Country" and "K country" likely refer to China and South Korea.

The school group eventually took down the note and issued an apology on its website for their "misleading expressions."

Moritomo Gakuen also requires its students to recite Imperial Rescript of Education, a relic of Japan's imperial era education system that deifies the Emperor. Today, reciting the rescript is seen by some in the Japanese public as glorifying the history of Japanese aggression.

Kagoike said Thursday rescript is "something that is useful for the moral education of children." He said he does not believe it is linked to militarism, and that his organization is not hateful.

The accusations, however, strike a chord with Abe's opponents, who believe the Prime Minister himself has been too cavalier when it comes to Japan's history

The defense minister's connection to Moritomo Gakuen

Tomomi Inada, who is believed to be Abe successor-in-waiting, has also found herself embroiled in the scandal. She denies any connection to the land sale. Inada did not answer CNN's questions about whether she supported the group's educational policies.

Inada told opposition lawmakers she did not have a relationship with Kagoike or his company when she was asked about the scandal in parliament last week.

But Japanese media uncovered court records showing Inada, a lawyer, represented Moritomo Gakuen during a trial in 2004.

Inada later confirmed she attended the trial in place of her husband, another lawyer, when their firm was representing the group.

Shuji Akiba, an official at the Defense Ministry, told CNN this was the only case Inada worked on for Moritomo Gakuen. The contract between the law firm and the Moritomo Gakuen ended in 2009.

Inada apologized for the mistake, but opposition lawmakers say she should resign for making a false statement while in a parliament session.

Inada has acknowledged her error and wants to now focus on her duties as defense minister, Akiba said.

Inada hasn't been in touch with Kagoike in 10 years, according to Akiba.

CORRECTION: An earlier version of this story misspelled Moritomo Gakuen