If there are no better reasons for buying a potion, just find a way around your opponents familiar strategy. It will be faster.
Yeah, this is only true sometimes. If there is a strong trasher on the board like remake, sure, don't open potion, but even then in many cases I might pick up the potion later (especially with remake, as I can later turn it into a $5). Familiar is hated because it is swingy, but also because it is strong, so you often can't ignore it.
There are exceptions, and you should definitely look for them. Trader is pretty good at combating familiar. Jack is too fast for familiar, at least without some very strong support. But don't assume that you can always "find your way around" familiar either.
One final thing to remember is that familiar is a much better engine card than a money card, because it is a cantrip. Because it doesn't give you any money, BM+familiar is fairly weak. So if you are thinking of "finding your way around" familiar, check if its more of a money board or an engine board. The worse the engine on the board, or the stronger the money cards on the board, the more likely you are to be able to skip familiar.