I’ve generally stayed clear of Call of Cthulhu.
And it’s not (completely) for the fact that it appropriates the foundational works of the genre. The much reviled/studied/enjoyed Cthulhu Mythos by H. P. Lovecraft is inarguably the basis for the mythology of the RPG system, but enough people already talk about its more problematic aspects.
Like with a lot of RPG systems, its the mechanics which are generally my turn-off. Sure, subject matter is important, but presentation (Feel) is more so. You can find tons of other RPGs which share settings and ideas, but mechanics and Feel are more unique to each system.
The main reason why I do not prefer Call of Cthulhu comes down to the way it treats the monsters of the Cthulhu Mythos. This feeling is not out of respect for the work so much as respect for the Feel.
This feeling comes out of the main artwork which usually surrounds Call of Cthulhu and other games like it.
It’s more about how Call of Cthulhu takes concepts such as cosmic helplessness and fear and (not all the time, but most of the time) boils it down to “hit it with whiffle bats until it dies.”
Call of Cthulhu does have a sanity mechanic, and this creates danger to parts of the character other than hit points, but as the saying goes:
“If it has Hit Points we can kill it.”
And the creatures in Call of Cthulhu sure have those.
So what does this mean? That the creatures should more closely reflect the creatures in the Lovecraft stories? If we did that, then characters would spend the game slowly getting more and more terrified before coming to a sudden and unavoidable end, usually involving insanity at something most RPG characters would barely care about.
“Creatures older than our civilization once ruled this place? Well yea, I assumed as much!”
“I’m descended from cannibal apes? Cool! Do I get anything for that?”
“The Great Old One has emerged from the sea in front of us? Ok, I steer the boat into-” oh wait… that one already happened, didn’t it?
One way to make Eldritch monsters more engaging would be to remove the comprehensible aspect completely. That doesn’t mean what it sounds like though: Human comprehension and RPG character comprehension are different things.
For us, an incomprehensible monster is something closer to the creatures from the Lovecraft stories; they shift and contort, confound the eye, and have a terrifying effect on the mind.
RPG characters however, are different. What is comprehensible to them are things which they can interact with, usually using skills, stats, and attacks. Take those away, and suddenly the character is confronted with something else… something beyond their understanding.
Perhaps a monster which can hurt the character, but they cannot hurt it, no matter what they try. Maybe it steals memories, eroding skills and experiences. For a more destructive and deadly feel, take a page from The Tearable RPG, and tear a piece off your character sheet which covers at least a letter of written words with one hand!
These beings are supposed to be beyond comprehension, interacting with our minds and bodies in ways we are not equipped to understand. Just as a Dimensional Shambler or Hound of Tindalos would be horrifying to encounter to us, a creature which targets very specific parts of a character would be horrifying to them.
That way, we can have our eldritch horror cake and eat it alive too.