This line really becomes apparent in modern day horror games. Take for instance how a lot of RPGs of the old White Wolf lines tend to go. They started out with an idea of horror but soon it turned into Dark Super Heroes or Dark Modern Day Fantasy (especially stuff like Changeling). They lost a lot of that early personal horror idea that was present in first edition Vampire and a some other works they did around that time. Now it doesn't mean you are playing it wrong by any means. If you and your group are having fun then you are playing it right and that's what counts.
But you need to acknowledge when it stops being horror.
What really tends to break that feel is when the PCs start having various powers to battle back at the darkness with. You become something more than human that can stand to toe to toe with the bad guys which isn't so scary all of a sudden. This is, as I said earlier, more akin to Dark Super Heroes. Playing Blade and the Nightstalkers or something.
The best single piece of advice I can give for running a modern day horror game is simple; keep the PCs human. If they become a vampire by being bitten or a werewolf or whatever then it should be game over for that character. Add them to the NPC list for the GM to use for the rest of the campaign. Place them in that vulnerable spot of being very mortal and unshakably human. They may gain knowledge, a few skills and maybe even an item or two. But never let them become anything like the things they are trying to fight.
If they get magic then make it a ritual requiring long term effort to make it work. Kult is a great source of inspiration for this as well as Unknown Armies (just stay away from the other abilities). Make them costly to get and even more costly to use. So they become a tool of last ditch desperation.
Keep the special items down to a minimum also. Think of things that may allow them to detect a very specific kind of creature but it can't give you anything exact. An amulet that makes it harder for a vampire to hurt them... but not impossible. Or something that requires an activation of the users blood to work, or some other sort of sacrifice. Keep them to a minimum and very specific so they don't become a catch all for everything they run into.
There is my advice about how to keep the game on the horror side of the line.