So I've been using a ton of GPT-4 in the last couple of days and don't see myself stopping. Some observations, none of them are surprising:
- it's super awesome as an assistant to learn academic stuff. It's not just writing summaries of papers, but you can just learn something by chatting with it. E.g., today I've asked it a bunch of questions about neuroimaging techniques and it's just a much more effective way to learn than using google and Wikipedia. Not as good as reading a really awesome textbook, but finding a really awesome textbook tends to be very difficult and often impossible because what I want to know can be so specialized.
- It seems great at knowing what the academic consensus is (I've either said this before or thought about saying this; I'm not gonna look it up)
- It's worthless at figuring out which side of a debated topic is right
- It's terrible at stuff where almost all source material is terrible. An example is the meta question: what it is good or bad at. Almost everything anyone says about what LLMs can and can't do (especially what they can't do) is awful, so GPT-4's take is also awful.
In truth, GPT-4 is horrendous at logic puzzles. You can make them laughably simple and as long as the puzzle is genuinely new, it will not get the right answer. I have made 7 of these puzzles, idk if I mentioned that or not, and was 100% correct so far. It's just actually not easy to make a puzzle that it doesn't know. If you just google "logic puzzle" most of those wouldn't work. Unless they're math problems, in which case you're just testing GPT-4's ability to fail at math.
And conversely, this whole thing about LLMs not being creative is just so dumb. People just won't stop saying it even though all the evidence contradicts it.
It's also bad at having interesting takes on hypotheticals.
So I guess the "unoriginal" and "only imitating" memes have truth to them, and I'd say the "doesn't really think" part is correct (though this isn't *obvious*, and most of the arguments I've seen aren't good). The "doesn't truly understand" and "isn't creative" takes I think are both bad. I mean yeah, I do think it doesn't really understand, but people who make that point usually phrase it as some undefined property in addition to performance. If you claim it doesn't understand something, you have to show that there are tasks where it fails -- and if you've already done that, then I don't think it adds anything to talk about understanding. So, technically true but still a bad take. 2/10.
But even so... an assistant who knows about every important topic in the world, has infinite patience, and can write simply... well that's still an amazing thing to have access to.
Also somewhat unrelated, but I cannot stop modeling it as a person. It doesn't matter how sure I am that it isn't conscious, I can't stop my social instincts from kicking in, and I constantly have the desire not to annoy it.