The other thing is that writing music down is an efficient way to communicate - to make sure we are all on the same page (so to speak). It could be argued that music without notation is more effective. In any case the performance is what counts - good musicians make good music.
in Western culture, written language is considered more valid than spoken language (which is why I am writing this down ;) - but seriously, we place higher value on a signed contract than someone saying "my word is my bond". Unfortunately, this has flowed over to music. How may times have you heard the expression " have you got the music for..." meaning of course the sheet music. But I think there is an underlying message here that the sheet music IS the music. This is wrong - the music is the performance. So it all gets down to how we use the notation. I think it should be more of a guide.
A final point is that we have only had audio recording for about 100 years. The reason that our musical knowledge and appreciation of the last 6-700 years is intact is because the music was notated. Otherwise how every piece of Western art music went would be conjecture, until the 20th Century when the phonograph was available.