Residents say the massive crane used to carry huge loads to the top of the site made a horrible boom when it crashed to the street below.
Building resident Evan Epstein said, "All of a sudden you hear this loud kind of earthquake sounding like buh bump buh bump buh bump buh bump. It sounded like you know just all the balconies were just kinda coming down one by one or something."
Rescue crews searched for possible victims on the ground and in the mangled mess of the residential high-rise.
Meanwhile safety experts secured the area and workers searched for answers.
One worker said, "It seems the turntable snapped off which is unusual. The only thing I can think of is the amount of stress in jumping the machine. It seems like it might not have been balanced properly. Just my assessment of it."
It is the second deadly crane accident in just over two months.
In March a crane toppled, crushing a residential building, and killing seven people.
And several other construction accidents have put pressure on the city.
The mayor pledged then to improve safety measures in a city undergoing a construction boom.
Mayor Michael Bloomberg said, "The question we have to make sure is do we have the appropriate level of inspection"
On Friday, the mayor called the accident tragic and unacceptable.
In the meantime, crews will have to figure out how to safely remove the crane and secure the damaged buildings.