When you are playtesting, how do you distinguish between "this card won't work as is" and "this isn't a set where this card will shine?"
Well if the card "won't work as is" then probably there's no set I want it in as is. Putting it in another set with more cards that make it worth playing only works when you play a high frequency of cards from that set; you might do that when you first get the set, and you might only have one expansion, so sure it's good if a card works well in its set; but lots of the time the cards are mixed in with everything else and those combos are not there. It's good if a card from Cornucopia works well with other Cornucopia cards, but it has to be good enough to get use out of when it's the only card from Cornucopia in the game, and that's certainly the intention.
Also, how do you distinguish "this card won't work as is" and "this card won't work at all?"
Generally in the end by some argument that's card-specific, although sometimes I endlessly waste time on them first.
Some cards just die because people don't like them. It could be that the card is hated, or just has no fans. The Remodel from/to deck top just didn't have fans. It seemed fine but left for something that people would like more. The attack that made people discard down to 2 then draw was hated. It's not like I should have tried to add something so fun to it that the result wasn't hated; I can do the fun thing without the thing people hate.
Some cards have tracking issues or other problems with handling them. Dark Ages had "+$2 +1 Buy, copies of cards in the trash cost $1 less this turn." You spread out the trash. It's not great spreading out the trash. Graverobber / Rogue just make you divide the trash into $3-$6 and not; Forager just makes you have a pile of one of each unique treasure. This card also had rules confusion involving some cards being cheaper and others not, and cards moving to/from the trash during your turn. Then the other part of the card was, "setup: each player puts a supply card costing up to $6 into the trash." People spent forever making this decision; it was a classic, "the less it matters, the harder it is to decide" situation. Neither part wanted more work done on it.
Unsolvable power level issues are rarer and take longer to give up on. There was a card in the main set at one point that I finally gave up on in Guilds. "+$1 per action in play" started in Intrigue, and went through many iterations to become Horn of Plenty; the premise was fixable. Madman was too strong when you could buy it, but even that was fixable.