The decision was made by the heads of the organizers, including the International Olympic Committee and the Tokyo metropolitan government, during a remote meeting that was held just four months before the rescheduled Summer Games are set to open in the Japanese capital.
"It is very unfortunate, but seeing the current state of infection and what should be done to avoid causing strains on our medical system, it cannot be helped," Seiko Hashimoto, president of the committee, told a press conference after the meeting.
IOC chief Thomas Bach said the top priority of the organizers is to ensure the safety of all participants of the games and people in Japan.
"We have to take decisions which may need sacrifice from everybody and we have to ask for understanding," Bach said in the meeting.
Toshiro Muto, the CEO of the Japanese organizing committee, said it will refund overseas fans who bought roughly 600,000 Olympic tickets and 30,000 Paralympic tickets.
The meeting was attended by Bach, Hashimoto, International Paralympic Committee chief Andrew Parsons, Tokyo Government, Yuriko Koike and Japan's Olympic minister Tamayo Marukawa.
The number of recorded coronavirus cases in Japan is relatively small compared to some other countries, but many people have voiced skepticism over whether the organizers can prevent the virus from spreading during the Olympics and Paralympics, which will involve some 15,000 athletes from around the world.
Before the meeting, the Japanese government had already concluded that allowing overseas spectators for the games is not feasible, given that the threat from the virus is far from over and fears that foreigners could bring more contagious variants detected in many countries.