Upon hearing objects collide, humans can estimate physical attributes such as material and mass. Although the physics of sound generation is well established, the inverse problem that listeners solve - of inferring physical parameters from sound - remains poorly understood. Classical accounts posit the use of acoustic cues that correlate with physical variables, but do not explain how humans might distinguish multiple concurrent physical causes. To study this problem, we built a probabilistic generative model of impact sounds, combining theoretical acoustics with statistics of object resonances measured from hundreds of everyday objects, and used it to synthesize and manipulate experimental stimuli. Humans accurately judged object properties from collision sounds. However, when both of the colliding objects varied, performance was impaired if the distribution of object resonances deviated from those measured in real-world objects. The results suggest that listeners use internal physical models to separate the acoustic contributions of objects in the world.