Demolition of the buildings—by new owners Campagna Construction Corporation—began in 1965, and was started on a weekend so that officials wouldn’t be able to stop it. Their destruction (along with that of Penn Station) played a significant role in advancing landmarks legislation in New York City. In 1962 the Landmarks Preservation Commission had already recognized the structures as landmark buildings—but there was no legislation to back up the Commission’s authority.

The New York Times ran a scathing editorial called “Rape of the Brokaw Mansion” which “decried the ‘weekend stealth…[of] the despoilers’ who demolished these buildings and noted that if the city did not pass pending landmarks legislation there would be no landmarks left to save.” In 1965, TIME ran an article called “The Gargoyle Snatcher,” which shared an anecdote about Ivan Karp, founder of the Anonymous Arts Recovery Society. He allegedly offered to buy two copper finials perched on the roof of the mansion, but “was told by the wreckers that removing them with care was too dangerous and would slow up the job of razing the building. Said the sympathetic foreman, ‘Sure it’s a shame, but something should have been done about it before we got the job.’”