Shopping centre owner Hammerson, owner of Birmingham's Bullring, is no longer urging its shareholders to vote in favour of its acquiring rival Intu.
It comes months after it announced an all-share offer for Intu, owner of the Trafford Centre in Manchester.
The proposed £3.4bn takeover would have created the UK's biggest property company, worth £21bn.
Hammerson said: "The proposed Intu acquisition is no longer in the best interests of shareholders."
In early morning trading in London, shares in Hammerson were up by 3.57%, but those of Intu were down by 7%.
Among the reasons Hammerson gave for withdrawing its recommendation was that the stock market's view of the retail property market had "deteriorated" since the turn of the year.
"This perception has been intensified by market concerns over the extended period of time that it would take to complete the transaction and realise longer-term returns from the Intu acquisition," it said.
The combined group would have been led by Hammerson chief executive David Atkins and chaired by its chairman, David Tyler.
Hammerson's offer for Intu will lapse if its shareholders do not approve the purchase when they meet.
Shopping centre giants
- 22 major shopping centres
- These include Brent Cross, Birmingham Central and Cabot Circus
- 15 retail parks
- 20 shopping centres in UK and Spain
- Owns the Trafford Centre, Metrocentre and Lakeside
- 400 million customer visits a year
The development comes less than a week after French shopping centre operator Klepierre abandoned a £5bn bid for Hammerson.
Hammerson also owns the Bicester Village designer outlet and London's Brent Cross shopping centre, while Intu also owns the Lakeside shopping centre in Essex.
Analysis: Dominic O'Connell, business presenter, Today programme
The boardroom of Hammerson, the property giant that owns some of Britain's best-known shopping centres, must have been an uncomfortable place in recent weeks.
The company's leadership – chief executive David Atkins and chairman David Tyler – had thrown their weight behind a controversial £3.5bn takeover of a big rival, Intu.
Shareholders, however, had other ideas.
They were wary of doubling down on UK retail, all the more so as several High Street chains have collapsed or had to shrink drastically in recent months.
The directors of Hammerson have been fretting about the strength of this opposition and what to do next – press ahead, and risk investor anger, or face the public humiliation of ditching the bid.
This morning, they chose the latter course of action, saying the retail market had changed significantly in recent months and shareholder opposition had hardened.
The Square Mile tends to be unforgiving of chief executives who lead companies into unusual situations, and while Mr Atkins will receive credit for scrapping the takeover, he can afford no further slip-ups.
Michael Hewson, chief market analyst at CMC Markets, said: "With Hammerson's share price already down on the back of concern about future income, due to the difficult retail environment, it can't have escaped the notice of bigger institutional shareholders that the risks to UK retail property were starting to become much more of a concern.
"Retail businesses are already struggling with higher business rates as well as declining footfall so today's news… is quite a sensible move, particularly since some bigger shareholders were expressing disquiet about the deal."