One thing I find really dumb about the old fanservice excuse is that back in the Itagaki days, DOA actually had very little fanservice. For some reason he and/or TK/TN decided to largely promote the game with fanservice, yet almost nothing in the game actually offered that. You had very few, tame outfits (like a basic swimsuit or something that just shows off some leg or cleavage) and then a brief CG scene or two that shows off some conveniently censored nudity.
Basically the game didn't really even have any notable fanservice, yet it was promoted largely on that and created a bad image. People shunned it for something that wasn't even there. Nowadays though, there actually is a ton fanservice in the game with the inclusion of gravure videos, loads of smutty swimsuits and tons of other ecchi costumes, even including destructible outfits now.
Aside from that, it seems as though a lot of people don't like the triangle system or the idea of holds. Maybe the game's stronger than usual reliance of mind games caused by the triangle system puts people off (I've personally spoken with someone who hates it because of this). There's also the fact that there may be people out there who have nothing against the game, but simply don't want to bother trying to learn it since they likely won't have anyone to play it with due to its lack of popularity.
I assume that DOA4 also may have dealt some lasting damage. Everyone hated that game. Even DOA players hated it. And for a very long time it was all we had. During that seven year period, anyone who decided to give DOA a chance had to play DOA4 and were turned off by it. Many of those people are likely to keep that negative experience in mind when they think of DOA.