The referendum was approved by the Styria State Parliament in its final sitting before a summer break after an emergency request by the Austrian Communist Party (KPÖ).
They gathered 11,031 valid signatures petitioning against pressing ahead with the bid, easily more than the 10,000 required to force a vote under local rules.
A final date for the referendum has not yet been proposed, but late September is considered the most likely time frame.
A bid from Graz, Austria's second largest city, emerged as surprise challenger earlier this year after a failed referendum in Innsbruck last year.
A feasibility study conducted by the University of Applied Sciences Campus 02, Joanneum Research and Graz University of Technology last week concluded that a Winter Olympics in the region is "possible without great risk".
Figures involved in the bid initially claimed that there would be no need for a referendum.
They then argued that, even if one was held, laws in Graz and Styria stipulate that referendum results are not binding.
The rules, however, are different for a state-wide referendum and it is hard to see the bid continuing without popular support.
The KPÖ have vowed to collect more signatures and continue its campaign against the bid.
"[The signatures] have apparently caused a change of opinion," the Party's Council member Claudia Klimt-Weithaler told ORF.
Graz Mayor Siegfried Nagl admitted to being disappointed about the decision, but claimed to be confident in public support.
"I have not only found supporters politically at Federal, state and city level, everybody knows that - I fight for something, others want to be against something," he said.
"I hope that the Styrians decide with vision, the result is binding for me.
"The referendum is likely to take place in September.
"This is because at the beginning of October, the International Olympic Committee (IOC) will make a pre-selection of the Olympic candidates."
A referendum over an Olympic bid has not been successful since one in Oslo in 2013 in support of a 2022 Winter attempt which collapsed anyway the following year.
Calgary in Canada, Sapporo in Japan, Erzurum in Turkey, Stockholm in Sweden and either Milan, Turin or Cortina d'Ampezzo in Italy are other cities still bidding for the 2026 Games.
Calgary could face a referendum at the end of this year while a bid from Sion in Switzerland was scuppered last month after a failed vote.
A final decision on who will host the Games is due to be made by the IOC next year.