Most neurology fellowships are not competitive, and fellowships are mostly in "name-brand" places as they can better offer higher load of patients within each subspecialty for training purposes.
More prestigious programs have more neurologists and more subspecialists, and among those, more research-oriented neurologists. For those who are more research minded, it would then be important to match into a more "prestigious" program as they would then have access to these researchers and find a mentor with whom they can also jump start their research career. Certainly, residents coming from these programs will then have letters of recommendation from these researchers and more easily transition to a more "prestigious" fellowship program.
I should add that for majority of students who are not interested in research, prestige would then matter less. It would be more important to find a program where all subspecialities are represented where you can gain adequate exposure to each field and make an informed choice on fellowship. Beyond that, location would be as important, so are the people at the program (making sure they are collegial and not "toxic").
I trained in both medium and large programs, and I would prefer medium-sized programs for residency training as you develop closer relations with every attendings, while in larger programs (>8 residents per year), your exposure to many attendings are very limited. Larger programs are better for fellowship training, as there would be sub-subspecialists who may have panel of patients with very rare diseases for training purposes.
Just my 2 cents.