Generally, the reasons not to buy an island are more important than the reasons to buy an island. If there's another important 4 cost card then you might just want to get that instead. If there's an important 5 cost card you might want to start double silver instead as you otherwise have to be prepared for 4 coin hands on turns three and four. If there's an option to trash you might prefer that, and so on.
Having accepted all that, you are generally looking at games where removing an estate or two from your deck is going to be particularly good. This could be in a deck with stables for example, where you don't want to be discarding copper to draw estates, and if you keep the deck a bit smaller you are likely to be able to keep connecting cards together (such as copper and stables, or green cards with other islands later). There are plenty of other situations too. The island points also seem stronger in games with limited scoring options.