yep, depends on the size...
most comets' nuclei are pretty small and would probably just result in a new crater on the moon's surface.
there are some large asteroids which, if they hit the moon hard enough [meaning: fast enough] could probably destroy it. Ceres might be an example...
on the other hand, most asteroids aren't that big and don't operate in orbits that bring them near the earth or the moon.
however, as they orbit the sun, asteroids' orbits can be perturbed by the more massive planets or even gentle collisions with each other, and their orbits might be changed enough to bring them into the inner solar system and close to the earth and moon.
current science shows that some very large objects have hit the earth and done some major damage. it's very likely that the results of the impact in mexico changed the earth's climate long enough to wipe out many plant and animal species, including virtually all of the dinosaurs. the evidence is pretty solid on that.
if a big ENOUGH asteroid hit the moon, parts of the resulting debris might do the same to us, and at the very least, cause one amazingly intense meteor shower as the parts fell throught atmosphere and [hopefully] burned up before hitting the ground....
scientists are currently running several sky-scanning programs to determine if any objects are likely to even get within a few diameters of earth.
the good news is: space is VERY big and planets are relatively small in that vastness, so the odds of a big, bad hit to earth is slim. the moon, much smaller than the earth, is even less likely to be in the cross-hairs of a runaway asteroid.
if we're lucky, humans might even develop some technology which could steer even the remotely dangerous objects onto paths that would preserve our safety. problem is: it takes one heck of a lot of energy to change an orbit like that, and a lot of energy to get the machine that'll do it close enough to the asteroid to give it a push!
Astronomy magazine recent issues have touched on this.