You're underestimating the weight of the floors in these kinds of buildings. Each floor supports not only the one above it but the ones above those. When the plane hit the building, it could have survived that impact, but the key issue was the fire.
The fire weakened more and more supports in the building, and being an office building full of mostly air and combustible materials, the fire spread and fueled by jet fuel, was VERY hot. When the supports on enough floors got weakened, things started to buckle. But when one floor starts to fall, it puts a tremendous amount more stress on the ones below it. The force of a higher floor hitting a lower floor plus having other floors on top of it is too much for the supports below it.
Think of it as impulse-momentum. A heavier object traveling with some kind of velocity has a very high momentum. To stop this object and prevent the collapse, one has to exert a huge force against it since it has high inertia due to its mass. If the floors are already weakened due to fire, they can't stop the floor above it and like an inelastic collision, they stick together and fall together.
The mass then is greater and the floor below it buckles with it.
As you can see this pancakes down until even the strongly supported floors buckle. Those floors are intact, but the mass above them with its momentum is too much to bear and the collapse continues all the way down.