The meteors that land on the Earth seem to come from about 5-6 distinct families defined by their particular element and isotopic abundances, so this suggests that perhaps 5-6 major bodies were originally involved in producing most of the known meteorite falls. Also, we know that meteorites come in 'stony' and 'iron/nickel varieties, so that the original bodies must have been large enough for the material to 'differentiate' so that lighter elements formed a 'stony' crust surrounding 'iron/nickel cores. Also, the crystalline structure of mateoritic material often shows 'Weidenstatten figures' which are caused by high-pressures; possibly high-speed impacts. Also, the range of sizes of meteoritic crystals suggests than many of them are on the large-side which also points to a slow cooling process like the kind you would find inside a rather large parent body.
If you were to add up the mass of all the known asteroids larger than a few kilometers, the total mass would be about 5 percent of the mass of the Moon, so if this were divided into 5-6 equal-sized parent bodies as suggested by the meteoric evidence, these bodies would be perhaps 1000 kilometers across. We know that, when the solar system was young, there were many such 'planetessimals' in the solar system because we can see impact craters on distant moons that require very large bodies to produce them. We think that some may have been trapped in the region of the asteroid belt, and then suffered collisions. because the material in the asteroid belt consists of differentiated material, it is unlikely that it arose from matter that never got the chance to build-up a planet. Differentiation requires a strong, local gravitational field. Slow cooling and large crystal patterns also require a large body. This is why the evidence points to the collision of a few large bodies, rather to material that never could get assembled into a planet, as the origin of the asteroid belt.