It is dangerous, but losing Followers can be more dangerous. On a board without discarding or curses getting Followers can easily be a 16 point swing (2 point swing each play of Followers and any of the following swinging duchies or provinces: discarding 2 coppers, having to discard a "dead" province and letting your opponent cantrip through their Tournaments, or having to discard provinces/gold on hands like Gold/Province/Province/Tournament/Tournament). It is awfully unlikely for a silver or two to swing more than 12 points, on some boards Followers can swing over 20. Particularly with something like Haven or Scheme out, Followers can become a race for who can draw their province first and the half turn that 2 or 3 Silver buys sets you back can be crippling.
I agree, my point is that a couple of silvers will not decrease the likelihood of you missing Tournament by a huge margin, and it will probably affect that in a single reshuffle, while the boost in economy may last several shuffles.