Some of the writing up I have done (work in progress, but feel free to comment):
1. General
(1) The main objective of game design is to create a game that is fun. That means it should be fun to the people playing the game, and not primarily to you, the mod. The best way to ensure that players have fun is to give them agency, i.e. to give the players the feeling that their actions really influence the way the game is going. This is a key ingredient and it will be referenced throughout this guide. There are other parts to a fun game - creative roles, interesting interactions, good flavor - but agency is the most important.
2. Conception
(1) Have a core idea. It is hard to get going on the design process without an idea of what to put in. Having an idea of what to do keeps you focused early on. The core idea can be many things - a unique mechanic, a theme, showing off an unusual role - just make sure it helps you come of with the first couple of roles. It is not a problem if at a later stage in the design process you realize that your idea won't work out as intended - by that time you have already built some of the setup and hopefully can take it from there.
(2) Don't give players bad roles. You may think that a VT without a vote would be an interesting challenge, but it is not motivating. The problem is you take away agency from the player. Try to avoid this in role design. If you need a nerf or an anti-town role, there are other options that do not reduce agency (like Macho or Hated). A plain Traitor without any knowledge or powers is a bad idea for similar reasons. You can still take agency away, but give your players something to make up for it.
(3) Opt for player choice whenever possible. A JOAT that can pick a power every Night is more fun than one that can only use a specific power in a specific Night. This is again because player choice increases agency. Therefore, an Innocent Child that can pick when to be revealed is better then one that is simply revealed when the game starts. Of course adding more choice makes the role more powerful, and sometimes you will still need to go with the weaker and less interactive version for balance reasons.
(4) Avoid random events. Leaving anything up to RNG removes agency. Consider a role that is Doctor 50% or the time and Vigilante 50% of the time and doesn't know which. That poses an interesting challenge, you may think, but it carries the risk of being frustrating. Roles that only work a fraction of the time based on a coin lip are similarly problematic; a player not knowing whether their power worked hinders their ability to figure out what is going on and thus reduces agency.
(5) Think about the rules of your game. Determining length of days, plurality lynch, whether mafia has daychat and what happens in stalemate situations often come as an afterthought. I would encourage you to consider these things early on. Some roles are going to be more or less powerful depending on those parameters, and considering them a part of your setup concept helps bring your setup together.
(6) It should not be possible to completely out scum. Even if some investigative role has an incriminating result on scum, they should have some way to credibly argue their innocence. If they don't, they have nothing to do and the rest of town just follows the Cop, removing agency for everyone but the Cop.
(7) Map your interactions. A good game has interesting interactions between different roles. You can draw a dot for each role you have and a line for each interaction. The resulting graph should ideally be highly connected. It is still fine if it splits into multiple clusters. A shape you should avoid is the star, where there is one special role that everything else interacts with, because if that role dies early, it will leave your setup sort of bland.
2.1 Closed setups
(8) Have a way to get started. Closed setups are hard, and it's hard to find things to talk about D1. In order to get the first day running more smoothly, give your players something to talk about. That could be some public information, a unique mechanic or a role that wants to claim early on. Providing an easy entry point goes a long way to keep your players engaged.
(9) Avoid red herrings. It may sound interesting to put a Ninja role into a setup that does not have any tracking roles. Usually, it is a bad idea. A Ninja role means players reasonably assume that some form of Tracker is in the setup, and when this assumption decides the game it's not going to be fun for anyone. That doesn't mean you shouldn't surprise players, but don't lead them on.
(10) Make sure that scum can fakeclaim. At some point, there is going to be claiming in your game, and then scum should not lose because you created a bunch of highly complex roles for town that they had no chance of coming up with. Either make it clear that roles are going to be relatively simple, or give scum some way to learn about the setup, or provide sample fakeclaims. This is especially important in Role Madness games.
2.2 Semi-open setups
(11) Limit the number of possible setups. It sounds cool if you might roll a million different setups, but how often is you game going to be played anyway, realistically? Usually a two-digit number is enough, and since every setup has to be checked for balance, a lot less work.
(12) Consider the solvability of your setup. How many flips/claims does it take to determine the exact setup that is played? Does scum have sufficient knowledge to make fakeclaims that cannot immediately be countered, and how quickly will they be found out? Generally, you'd want to aim for situations where "of those 3 players, at least one must be lying" rather than 1v1 situations.
2.3 Open setups
(13) Unique roles are powerful. Everyone with unique role can claim and confirm their townieness. Keep this in mind. You can break this by putting in more than one of the same role, but consider that oftentimes, copies of the same role can be broken for other reasons.
(14) Look at possible plans. If there is coordination to be done that clearly benefits town, then this is bad. If no planning improves town's chances, that is good. The game is arguably at its most interesting if there are mutually exclusive plans and it is not clear which is the best, but that is a difficult balance to achieve. Note that the more complex your game is, the more time you should spend thinking about this. Also consider what happens if people decide to claim "with role X, I would do Y".
3. Theme
(1) Randomize fakeclaims. If you first think about which characters to include and then give fakeclaims to scum, it will most often result in them getting only minor characters, so that their pool of possible claims is limited. A way to avoid this is to start designing by making a list of characters, one per player, and then determine randomly which of those are scum. You can then go on from there.
(2) For open or semi-open games, a different way to avoid flavor outing is to give publicly known flavors to the possible power roles.
(3) Do not get carried away with theme. Your roles should focus one one or two characteristics and turn them into powers that fit together. It is more important to have roles that are interesting to play than to have roles that perfectly fit the flavor. RMMs give a bit more leeway in this, but you should still make sure that the roles you designed fit not only the flavor, but also each other.
(4) Give informative role flips. This is mostly relevant for closed games. Maybe you want to describe the ability of your character in flavor, not with the standard mafia names. Consider that this makes it harder for your players to figure out what is going on. A flip should provide some information on what that player's powers were. You might give a standard name to the power, or reveal parts of the role PM on flip, but in general it's bad to just flip "Tyrion Lannister, the Hand of the King".