Fairly close to my heart is devising role-playing game monsters. What expertise do I have in this subject? In the 70s and 80s I made up a lot of monsters that were published in White Dwarf and Dragon magazines, as well as for my own campaign. I designed several monsters that are in the original Fiend Folio. The Princes of Elemental Evil are particularly well-known and even have their own entry in Wikipedia (archomental). I'm also relying in this piece on a panel discussion I attended at GenCon 2015 early on a Sunday morning with, among others, Wolfgang Bauer and Jeff Grubb up front.
Now I'm talking about monsters primarily for tabletop RPGs because there's a big difference between tabletop RPGs and video games. In video games, you have the boss mentality: boss monsters, really big, bad-ass, lone monsters that are very, very dangerous. I have never thought in those boss terms as I'll explain. I've always used a large number of monsters in a big climax led by some powerful leader. But the leader is not individually nearly as powerful as the character group. It's just that with all the other monsters around both the monsters and the leader collectively become very dangerous.
The big difference is that in tabletop RPGs, unlike video games, if you die you don't have a save game to go back to. Bosses are designed with the idea that there's a save game to go back to. They are designed to kill you several times before you succeed. You can't play tabletop RPGs that way, even today with all the easy healing, because if you die you’re dead (more or less). So in video games the purpose of any monster can be to kill the characters the first several times, whereas in tabletop the purpose is to scare the snot out of the players by threatening their characters in some way, but not by actually killing the characters. Death may happen occasionally (just to keep everyone "honest"), but it can't happen frequently, or you're not going to have much of a campaign.
So video game bosses tend to be much tougher in relation to the adventuring party or individual than the monsters you meet at a climax in a tabletop RPG. This is a fundamental difference. Video gamers would be disappointed if almost every time they had a climax they win the first time. They'd feel cheated, that it was too easy. It's a matter of expectations is much as a game functionality.
Of course, there are many ways tabletop RPGs are unlike computer RPGs and many of those are because of "save games" or lack of same. When you're making up monsters I think you should focus on the element of surprise, not just on making them super tough. Some game designers, including R. Knizia and S. Miyamoto ("We want to entertain people by surprising them ...") espouse this view. Likely Miyamoto would say that a major objective in any game is to surprise players, so perhaps the most effective way to design RPG monsters is to surprise the players, and many of my suggestions derive from surprise. A specific surprise is only going to work once, but that's one reason why so many people keep making up new monsters, to provide new surprises.
So what do we look at? Here's a list, then I'll discuss each one:
• The Unknown
• One Unusual Characteristic (kind of a loop)
• Two Types of Monsters Cooperating
• Characteristics from two types combined into one
• “Worse things than killing you”
• Really Smart Enemies
• Time Pressure
• Relentless Hordes
Fear of the Unknown is the first one. A major reason to make up new monsters is to surprise the players with something they don't already know. The players will probably feel it's more fair and perhaps more true to life if they can derive some of the characteristics of the unknown monsters from past experience or from appearance. "It looks like a giant, it may be about as tough as a giant." "With those big teeth, I bet it bites HARD."
Sometimes it'll be just one unusual characteristic. This may work particularly well if you take a well-known monster and give it a single surprising quirk. The obvious that comes to mind is regeneration. Regeneration is very powerful and should be used sparingly, but if you have an ordinary monster that regenerates, it will surprise the heck out of players, especially when a monster gets back up off the dungeon floor.
A single characteristic can be a focus of an unknown monster as well. Some refs won't want to go to extremes such as flying orcs or regenerating orcs, on the other hand, we don't mind the flying monkeys from the Wizard of Oz. I once made up a group of several kinds of lightning spitting monsters roughly analogous to military tanks (in my mind), although the players never realized that. They were big and they looked dangerous, and they were even without the lightning. The catlike ones were faster, sluglike ones really hard to kill, and so on. But it was lightning that set them apart and scared the players, in many encounters with them.
You can take two types of monsters and have them cooperate. Keep in mind the truism, there's hardly anything original under the sun; but combinations of things can provide new experiences, and that can surprise. We see this kind of cooperation whenever a monster type is said to normally employ a different monster type as guards. Of course, powerful monsters may enslave entire groups of weaker monsters and those weaker monsters can nonetheless provide good interference when our heroes show up.
We can also take the characteristics from two monster types and combine them into one. There's the classic owl bear, chimera, gryphon, dragon turtle, and so forth. You can take normally unintelligent monsters and provide them with human intelligence or normally intelligent monsters that aren't intelligent now. Some combinations may not be very believable, and I like believability in games and try to avoid them, but in this the age of TV and movie silliness not too many people care. The standards have changed over the past half century, so you can do things that would've been laughed out of the building, so to speak, 50-60 years ago, which now most people shrug at and accept.
Another way to make monsters interesting is misdirection. Play on the expectations of the players: change the appearance of the monster, pretend to be another monster, change stats (although it's easy to overdo that so I try to avoid just changing the stats of an existing monster).
There are worse things than killing you. Monsters don't have to kill to be frightening. They can turn your bones to rubber. The rust monster eats equipment. Permanent level drain, even temporary can be bad. Characters can be captured - slavers are monsters too. Theft - lots of monsters that nick your items such as leprechauns. There lots of things you can think of that are not death but will frighten the players. Threaten their characters' well-being, their possessions.
Foreshadowing is something you can do with any monster. It helps foster fear of the unknown. You can provide clues signaling danger - tracks, even something as simple as noises. Maybe the players will find something in writing that indicates that some intelligent monster is around - somewhere.
Really smart enemies. Face it, classic movie enemies are often stupid. This is why the Evil Overlord list of vows exists, and if you haven't read the Evil Overlord list I strongly recommend that you do so. http://www.eviloverlord.com/lists/overlord.html . Relatively dim monsters can be cunning: the great boxer Muhammed Ali was often said to be a dim brained man, but he was a cunning boxer. Consider though, you have to put your brain into the monster preparation. If you're not trying to be smart, how can the monsters be smart?
Time pressure is the classic videogame way to make monsters more dangerous. There's just not enough time to do all the characters want to do. But you can do this in tabletop games as well. Time stress leads to mistakes. “Watch out, it’s going to blow up!” or the enemy has diverted water into a room that's filling up with you trapped in it, or there's a fire spreading or the monster itself has some time limit associated with it. There are all kinds of ways to implement time pressure even if you're playing strictly on a turn basis. You know there only so many turns before something happens, you're still under pressure.
Positioning is another thing you can do with any monster. The classic is that you have a balcony that protects otherwise vulnerable archers because they're up there and you're down here on the floor or on the ground. Simple barricades, very low ceilings with/for short monsters: you're going after Duergar and they've kept their ceilings low so that humans have to bend down and are much less effective in every way, especially in a fight. Burrows of monsters can be hard to move around in. Water barriers can make a big difference. You can think of lots of ways to do this, but you have to think of it to make it happen.
You can have societies or factions or groups where the group as a whole may be more effective than the sum of its individual parts. I've often found that a group of monsters, even if individually weak, is more effective than one powerful monster, especially if they're subordinate to a leader that organizes them, a commander or "mastermind."
The last one is relentless hordes. Sheer numbers can be terrifying even if the monsters are individually weak. The Dungeons & Dragons 4th edition "minions" rule is quite brilliantly simple. Any damage kills a minion, but you can have lots of them and they're easy to keep track of (tabletop) because either they're doing fine or they're dead. Relentless hordes are the opposite of the videogame boss syndrome where an often-lone monster is super tough, but try it, you may find it interesting.
I have talked about the Do's and not the Don'ts, now let's look at the other side. The general principle is, give the players a chance, don't spring something on them. Don't rely on them having to die to find out something. (Some people have given a name to that particular characteristic but I don't recall what it is.) You don't ever want to force the players to die to learn something. I'm thinking in terms of a large set of players of many different attitudes, and trying not to really piss off any of the subgroups.
So, no "invulnerable to everything but X," though that's not so bad IF players know about it ahead of time. For example, we know about iron golem invulnerabilities in the older versions of D&D, which is to say virtually nothing hurts but +3 or greater weapons, and so we have time to prepare or avoid. We don't always manage to do that, but we've got the chance. At least that's what counts.
Another is sudden, unwarned-about death as in Monty Python and the Holy Grail, where the carnivorous bunny kills in one attack. (Yes, "Tim" warned them, but it was not a believable monster so the warning had no effect.)
Another no-no: take an innocuous looking thing and make it a super monster, which turns out to be (again) like the Carnivorous Rabbit from Monty Python's Holy Grail. You may think that's funny, but serious players won't think that's funny when they're the victims. (As with everything else, "it depends".)
The golden rule applies. In fact, both golden rules, the general Golden Rule and the golden rule of RPGs. The general Golden Rule, "do unto others as you would have them do unto you" is pretty good. Ask yourself how you'd feel if such-and-such happened. The RPG golden rule is, "what's good for the players is good for the monsters and vice versa," that is, if the monsters can do something wild or drastic shouldn’t the players also have a chance to do it? And if the players can do something, shouldn't the monsters be able to as well? Think about it. Try to be at least halfway sensible and always put yourself in the shoes of the players and ask yourself how you would react if this happened to you.
Someone sufficiently steeped in the myriads of RPG rules published since 1974 could probably write a book (with many examples) about monster design. But this is enough to provide a guide for the inexperienced.