But concerns over the impact on coronavirus infections kept most opposed to holding the event this year, although the 58% in opposition is about 20 percentage points lower than earlier opinion polls.
If the Games were to go ahead as scheduled, 91% of respondents said spectators should be kept to a minimum or not allowed at all, the Yomiuri poll showed.
A survey by global consultancy Kekst CNC published on Wednesday showed similar rates of majority opposition in Japan, at 56%, as well as in Britain and Germany, at 55% and 52% respectively.
In France and Sweden, more people opposed than approved, while in the United States, respondents were split at one-third between those who agreed and disagreed that the Games should go ahead, according to the survey.
A Reuters poll published last month showed nearly two-thirds of Japanese companies also oppose holding the Games as planned, swinging from the previous survey showing most in favour.
Japan has so far confirmed 431,250 coronavirus cases and 7,931 deaths as of Monday.