Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[faust]

Actually that's not quite correct, I think I also have a hard time imagining you believe it in the first place, just because it would locate the biggest problem in the world at a purely technical challenge that could plausibly be solved by a few hundred people not being stupid for a few years, rather than anything more, well, profound.

Well, I suppose an argument can be made that it's not a purely technical challenge (if I was more informed), the same way some people will argue that climate change is a technical challenge but I disagree.

[silverspawn]

Now that you say it, I agree. Wouldn't be hard to frame AI x-risk in a more profound way.

[silverspawn]

I was feeling like playing this again. WBC2 in a nutshell:



This is a mission against a computer on the hardest difficulty. Computers do not have resources; they have workers that go and make production buildings, and then units come out of the production buildings forever. This allows them to play races that are utter trash, in this case dwarf, and be extremely difficult to beat.

One way to beat them is to have a bunch of overpowered units in your following and/or an overpowered hero and win the game early. This is the no skill variant. Also it actually doesn't work on this particular map because there are a million towers on the way to opponent's base, almost certainly to prevent exactly this.

Afaik, the only way to beat it honestly is to a) have exponential resource growth and b) a tiny production building so you can build lots of them, and c) a strong unit. The only race that has a) and c) is woodelf, which is also conveniently the only race that has b).

The round things are the units that make resources. The tiny white trees are the production buildings that make strong units.

[silverspawn]

Aside from having the worst balance of all time with 12 races and wood elf/fey infinitely stronger than the rest, the game is also wonderfully bugged. After barely holding of armies for 30 minutes, the opponent sent too many units at once and they got stuck. (Maybe my brother was right about the inefficiency if too many people are walking on a narrow road.) Here is me eventually cleaning them up. There are a lot of corpses missing here because I saved/restarted to make sure I could take 2 screenshots.



And the best part: I did not win this game because after having stabilized and killed about 3/4 of the opponent's base, there was a different bug where I could no longer make units; when they were finished building, nothing happened. Standing army wasn't big enough and that was that.

I think this was the perfect resolution for what is hopefully the last time I'll ever touch this game.

tl;dr best game ever made

[silverspawn]

And now watch me turn this into an analogy for why AI will kill everyone

The optimal solution never looks anything like the people who pose the problem expect. I bet it never occurred to the designers that this is the strongest way to play. Just as it never occurred to the designers of stronghold that having low taxes with 200+ people is the best way to play -- if it had, they would have designed the game differently. If you optimize for something too hard, the result looks bizarre. And this is just what i can do in my off time. An AI would figure out how to cause bugs on purpose and use those to win.

There was some paper that theorized about an AI trained to optimize scores in Mario. (It didn't go exactly like this, but somewhat.) At first, it plays the game. Then, it goes to some place at the map and jumps up and down awkwardly because this triggers a bug where it gets maximum points. Then, it hacked the game code to increase the maximum number of points that could be achieved. And then it killed everyone and used all resources in the available universe to build the largest collection of hard drives so as to digitally represent the largest possible integer that represents the game score. This approximated the true physical optimum of the goal "maximize the score in mario".

Not that giving an AI a goal is something we can do in the first place, so this is only a part of the problem.

[silverspawn]

This reminds me of the last time I attended a festival with a then friend and a bunch of his friends that were not at all my friends. One evening we played a card game. Someone suggested a rule change and I told them this was dumb because it could be exploited easily. Then, in an impressively unlikely progression of events, the first game we played had a sequence of turns that demonstrated this just about perfectly. My friend thought it was funny because he was smart enough to realize the new variant was broken as well, but his friends complained that my way of playing wasn't fun.

I thought they were idiots but maybe they were onto something.

jk                 they're idiots either way

[Awaclus]

There was some paper that theorized about an AI trained to optimize scores in Mario. (It didn't go exactly like this, but somewhat.) At first, it plays the game. Then, it goes to some place at the map and jumps up and down awkwardly because this triggers a bug where it gets maximum points. Then, it hacked the game code to increase the maximum number of points that could be achieved. And then it killed everyone and used all resources in the available universe to build the largest collection of hard drives so as to digitally represent the largest possible integer that represents the game score. This approximated the true physical optimum of the goal "maximize the score in mario".

Not that giving an AI a goal is something we can do in the first place, so this is only a part of the problem.

How would it learn to hack the game code without being instructed to do so?

[Awaclus]

I mean, it happens that people instruct facial recognition algorithms to be racist by accident and stuff like that, but those kinds of mistakes happen because of biases that exist in the current society that lead to the same biases being reflected in what the AI is doing, i.e. the devs are specifically, explicitly (although not deliberately) asking it to be racist, and then they get the result they asked for in the way they asked for. It's not like the AI decides to be racist on its own or does any other unexpected Monkey's Paw stuff.

[faust]

There is one question that has just occurred to me, for people who believe in a singularity.

If artificial general intelligence necessarily causes a technological singularity, then why did human general intelligence not cause a biological singularity?

Of course this is assuming that AGI is even within reach, which I still find rather doubtful.

[silverspawn]

let's maybe taboo the term singularity. by far the most likely narrow scenario is everyone falling over dead, which is not what people who talk about "the singularity" tend to have in mind, plus the term is overloaded to begin with

So I'm going to rephrase the question as "if AGI will necessarily have an extremely big impact, why didn't human general intelligence have a comparably big impact?"

Answer: it absolutely did. Evolution has been slowly, slowly tinkering with biological organisms over about 3 500 000 000 years. During this, the intelligence-of-the-most-intelligent-species has been growing extremely slowly, occasionally also grinding to a halt (e.g. dinosaurs). Then it crossed an intelligence threshold, and WOOSH, throughout the last ~2 million years, which is 0.05% of the total timescale, we utterly dominated the planet and increased global GDP by something like 4 200 000 000 000%. I think whatever standard you have for "extremely big impact" this ought to count

[faust]

let's maybe taboo the term singularity. by far the most likely narrow scenario is everyone falling over dead, which is not what people who talk about "the singularity" tend to have in mind, plus the term is overloaded to begin with

So I'm going to rephrase the question as "if AGI will necessarily have an extremely big impact, why didn't human general intelligence have a comparably big impact?"

Answer: it absolutely did. Evolution has been slowly, slowly tinkering with biological organisms over about 3 500 000 000 years. During this, the intelligence-of-the-most-intelligent-species has been growing extremely slowly, occasionally also grinding to a halt (e.g. dinosaurs). Then it crossed an intelligence threshold, and WOOSH, throughout the last ~2 million years, which is 0.05% of the total timescale, we utterly dominated the planet and increased global GDP by something like 4 200 000 000 000%. I think whatever standard you have for "extremely big impact" this ought to count

I mean, I am not contesting that, but from the way you talk it doesn't seem like you worry about things changing dramatically over the course of 2 million year, or even (if I more charitably apply the 0.05% of total timescale to just the existence of humans) 1000 years.

increased global GDP by something like 4 200 000 000 000%.

Also this made me laugh. We invented a statistic and then that increased? I'm not debating that humans have had a significant impact on the planet, but I can't help but feel like your choice of example is very telling.

[silverspawn]

Evolution is very dumb. Doing random permutations and then having a long messy selection process where good mutations have like a 20% chance of taking over after 100 generations, or something like that, is very very very inefficient. The miracle of evolution isn't that it works well, it's that it works at all; optimization bootstrapped itself into existence out of nothing. Gradient descent is easily 100000 times as efficient. So it doesn't seem surprising that the two aren't completely analogous. AI is a lot faster. But surely if the question is whether intelligence is a big deal, humans are evidence for that rather than against.

(I've chosen GDP because it's the standard metric, but if you don't like it, choose a different one; as you said the big impact isn't hard to see.)

[silverspawn]

I mean, it happens that people instruct facial recognition algorithms to be racist by accident and stuff like that, but those kinds of mistakes happen because of biases that exist in the current society that lead to the same biases being reflected in what the AI is doing, i.e. the devs are specifically, explicitly (although not deliberately) asking it to be racist, and then they get the result they asked for in the way they asked for. It's not like the AI decides to be racist on its own or does any other unexpected Monkey's Paw stuff.

Quite. You'll notice that I never said an AGI will be racist.

The relevant difference between an AGI and facial recognition is that AGI will run an optimization process. Which is to say, it has a goal (not necessarily the one we want it to have since we don't have a straight-forward way of just handing it a goal, but still a goal) and then searches for actions that lead to the goal being as fully met as possible.

Image recognition models don't do anything of the sort; they're just a bunch of heuristics, usually encoded in a neural network.

Optimization matters because it leads to instrumental subgoals. in the mario example, the AI doesn't have a terminal goal of hacking the game (terminal == what it wants for its own sake), but it has a terminal goal of achieving a higher score, and this leads to an instrumental subgoal (instrumental == what it wants to ultimately help with a terminal goal) of hacking because hacking is effective at raising its score.

The main problem here is that most goals incentivize perverse behaviors if they're maximized to the extreme; this is what I was getting at with the WBC2 examples. Once the AI is smart enough to model its programmers, it'll know that they don't want this and will turn it off if it pursues them. Not being turned off is an instrumental subgoal for just about every terminal goal, so it'll want to avoid that. And "kill all humans" is the obvious thing to do here. This is a totally different mechanism from biases in the training data. It doesn't matter if your training data is perfect; everyone being dead is what naturally happens if you have a goal and optimize it to the physical limit. It takes an extremely specific configuration of atoms to allow humans to exist, and if the AI is powerful enough to reshape the world however it wants, then only a tiny number of goals are compatible with not everyone dying.

So this is why it'll want to hack the game. How does it know how to hack? Well, it depends on what information you give it. If you give it zero inputs, it won't learn dangerous things, but then it's also not useful. The problem is that you need to give it inputs for it to learn how to do stuff. There's a tradeoff between capability and safety. And it's going to be extremely smart -- this is the whole problem, none of this stuff happens if it's not extremely smart -- so it can figure out things based on a lot less information than humans can. If, e.g., it sees its own code, this is already lots of information. It could infer how humans think and that they're not very smart. ("Look at that, they write in a weird, totally inefficient language and the put useless comments into it that don't do anything".) You probably don't need a lot more than that to figure out how to hack something.

Right now, we're training models by letting them process massive amounts of training data, so there is plenty of information.

[silverspawn]

Should also say that this has a bunch of parts, and there are corresponding ideas for how to make AI safe. E.g., the disaster happens because AIs will run optimization processes, so there's one idea that says 'let's just not do that'. This is called comprehensive AI services, it envisions that we will just have lots of narrow systems like facial recognition that do narrow tasks, and no one big general thing.

I think this would avoid disaster, but the problem is that it's not what deepmind and openAI are doing. The trend seems to be toward more agent-y, more general systems, e.g. alphazero or alphafold. It's also a lot more profitable to have one thing that can do lots of tasks

There's another idea which targets the outputs, it basically says "let's just not give the AI any output channel to do stuff; we train a model and then don't use it but just use transparency tools to learn from it". This would almost certainly avoid disaster, but seems hopelessly non-competitive given the rather abysmal state of interpretability we have.

[faust]

Evolution is very dumb. Doing random permutations and then having a long messy selection process where good mutations have like a 20% chance of taking over after 100 generations, or something like that, is very very very inefficient. The miracle of evolution isn't that it works well, it's that it works at all; optimization bootstrapped itself into existence out of nothing. Gradient descent is easily 100000 times as efficient. So it doesn't seem surprising that the two aren't completely analogous. AI is a lot faster. But surely if the question is whether intelligence is a big deal, humans are evidence for that rather than against.

I'm not sure how evolution really factors into this. I suppose some time passed from the first species of men to our current one, so maybe you should use the emergence of homo sapiens as the starting point (which seems to be 315 000 years ago rather than 2 million). The development from there doesn't really have much to do with evolution. Also I believe all the numbers you used in that paragraph are made up, or are there quotations for them?

I have already factored in that AI can develop faster, i.e. 1000 years rather than 2 million, which is not quite the 100000 time speedup you apparently  assume, but it's something.

(I've chosen GDP because it's the standard metric, but if you don't like it, choose a different one; as you said the big impact isn't hard to see.)

That's what I meant by it being telling; apparently wherever you get your information GDP is the standard metric.

[Awaclus]

The main problem here is that most goals incentivize perverse behaviors if they're maximized to the extreme; this is what I was getting at with the WBC2 examples. Once the AI is smart enough to model its programmers, it'll know that they don't want this and will turn it off if it pursues them. Not being turned off is an instrumental subgoal for just about every terminal goal, so it'll want to avoid that. And "kill all humans" is the obvious thing to do here. This is a totally different mechanism from biases in the training data. It doesn't matter if your training data is perfect; everyone being dead is what naturally happens if you have a goal and optimize it to the physical limit. It takes an extremely specific configuration of atoms to allow humans to exist, and if the AI is powerful enough to reshape the world however it wants, then only a tiny number of goals are compatible with not everyone dying.

The question I'm asking is: how does it know what humans are, what killing is, and the fact that doing this killing thing to the humans will be in any way helpful towards achieving the goal?

[silverspawn]

That's what I meant by it being telling; apparently wherever you get your information GDP is the standard metric.

Fwiw Eliezer Yudkowsky thinks GDP is irrelevant (or at least irrelevant going forward because it's bottlenecked by regulations not capability), but I have this model in my head of "normal literal serious modern person who doesn't know about AI" and this person thinks GDP is very important, so I tend to talk about GDP when I talk about AI with most people. I think psycho-analyzing me isn't that helpful, or at least doesn't tell you what you think it does, because I've had this conversation so many times and am trying to adjust it to avoid common pitfalls.

Which I can do without saying anything I don't believe in. E.g., I think GDP is a perfectly a fine way to measure impact. Maybe GDP isn't a thing we should optimize for**, but raising GDP by 4 200 000 000 000% is nonetheless very impressive and requires advanced tech. E.g., suppose god tells you that GDP will double from 2030 to 2032 and then again from 3032 to 3033. (This is not a prediction.) This may not be a good thing or say very much about human flourishing, but it sure says something about the impact of technology.

Also I believe all the numbers you used in that paragraph are made up, or are there quotations for them?

The seqeuences I'm rereading contained a bunch of posts about evolution, including some math on how quickly evolution progresses. I cited those from memory, so they're probably not correct but probably approximately correct, could dig them up if they're important. The gradient-descent vs. Evolution factor was a guess for a safe lower bound. Evolution really is ridiculously slow, pretty sure the true factor is a lot higher.

For this reason, I don't think the analogy can tell you anything about when AGI will be developed how or fast it takes off. Gradient descent on a hand-designed loss function and random mutations just aren't similar enough.

**  in fact we definitely shouldn't, which is why Andrew Yang proposes to replace it by measures like mental health and lifespan!

[silverspawn]

The main problem here is that most goals incentivize perverse behaviors if they're maximized to the extreme; this is what I was getting at with the WBC2 examples. Once the AI is smart enough to model its programmers, it'll know that they don't want this and will turn it off if it pursues them. Not being turned off is an instrumental subgoal for just about every terminal goal, so it'll want to avoid that. And "kill all humans" is the obvious thing to do here. This is a totally different mechanism from biases in the training data. It doesn't matter if your training data is perfect; everyone being dead is what naturally happens if you have a goal and optimize it to the physical limit. It takes an extremely specific configuration of atoms to allow humans to exist, and if the AI is powerful enough to reshape the world however it wants, then only a tiny number of goals are compatible with not everyone dying.

The question I'm asking is: how does it know what humans are, what killing is, and the fact that doing this killing thing to the humans will be in any way helpful towards achieving the goal?

Creating models about the world is instrumentally useful to do things, and we want AI to do things, so it will create models of the world. Probably models that are better than ours.

This one already has precedent. GPT-3 (the language model I've used to generate dominion cards) knows what humans are, what killing is, and it can do basic causal reasoning. I've once asked it "what would have happened if I had bought bitcoin 20 years ago" and it told me that I would be rich now. It learned this information from parsing text, with a loss function of "predict the next word". If a future AGI isn't less competent than GPT-3, it'll also know this.

Once again, there is a school of thought in AI safety that says we should avoid this exact thing, i.e., train an AI such that the training data doesn't include information about huamns. I think this is called stem AI, where it would only do physics or something.

[faust]

This one already has precedent. GPT-3 (the language model I've used to generate dominion cards) knows what humans are, what killing is, and it can do basic causal reasoning. I've once asked it "what would have happened if I had bought bitcoin 20 years ago" and it told me that I would be rich now. It learned this information from parsing text, with a loss function of "predict the next word". If a future AGI isn't less competent than GPT-3, it'll also know this.

Saying that GPT-3 "knows" anything is like saying that Wikipedia "knows" what a human is because it has an article on humans. It also cannot do reasoning. It can merely reproduce reasoning-like patterns.

[silverspawn]

I think the relevant dimensions are (a) what information is represented, and (b) to what extent the thing can do causal reasoning. I personally do say stuff like "google knows the answer to this" IRL, so I guess I refer to (a) as knowledge and would say wikipedia knows what humans are. But that's just words. What matters is that both have information about humans represented, and GPT-3 can answer questions about it whereas wikipedia can't.

I think Awaclus' question was actually about (a), is the information represented at all. This is not, in fact, trivial for GPT-3 because it doesn't carry text around and isn't reading the internet at test time. It's a gargantuan graph with a fixed structure and 175 billion rational numbers that determine its input/output function. These 175 billion numbers encode information about humans and bitcoin and whatnot.

It also cannot do reasoning. It can merely reproduce reasoning-like patterns.

I dispute that there is a difference. It is not the case that GPT-3 can only answer questions it memorized; it can apply reasoning patterns to new contexts. I've never seen anyone explain what the difference is between this and what humans do, except that humans do it better.

[faust]

It also cannot do reasoning. It can merely reproduce reasoning-like patterns.

I dispute that there is a difference. It is not the case that GPT-3 can only answer questions it memorized; it can apply reasoning patterns to new contexts. I've never seen anyone explain what the difference is between this and what humans do, except that humans do it better.

It's not an easy question to answer. However - assuming this is how humans work, and all they do is emulate reasoning patterns - then where does reasoning originate from? These patterns must have developed somehow.

[silverspawn]

This may sound like a cop out, but my honest answer is "probably exactly like GPT-3 figured out that buying bitcoin 10 years ago means you're right now". I suspect that this is the same process that makes me reason about how to prove \sqrt{2} is irrational.

and we have no idea how that works because interpretability isn't there.

[silverspawn]

I'm not sure if I've posted this at some point, but this is definitely one of the best things on youtube.

[Awaclus]

Creating models about the world is instrumentally useful to do things, and we want AI to do things, so it will create models of the world. Probably models that are better than ours.

This one already has precedent. GPT-3 (the language model I've used to generate dominion cards) knows what humans are, what killing is, and it can do basic causal reasoning. I've once asked it "what would have happened if I had bought bitcoin 20 years ago" and it told me that I would be rich now. It learned this information from parsing text, with a loss function of "predict the next word". If a future AGI isn't less competent than GPT-3, it'll also know this.

Once again, there is a school of thought in AI safety that says we should avoid this exact thing, i.e., train an AI such that the training data doesn't include information about huamns. I think this is called stem AI, where it would only do physics or something.

It can't do causal reasoning, it doesn't even understand what you're asking it. It understands what text is, but everything else it does is just replicating patterns in human-written texts. It doesn't know that you would be rich if you had bought Bitcoin 20 years ago, it just predicts that that string of characters is what its training set would write in response to your string of characters. If you ask it a question a human has never been asked, it's still just going to do its best to predict what a human would say, not what the correct answer is.

[silverspawn]

Creating models about the world is instrumentally useful to do things, and we want AI to do things, so it will create models of the world. Probably models that are better than ours.

This one already has precedent. GPT-3 (the language model I've used to generate dominion cards) knows what humans are, what killing is, and it can do basic causal reasoning. I've once asked it "what would have happened if I had bought bitcoin 20 years ago" and it told me that I would be rich now. It learned this information from parsing text, with a loss function of "predict the next word". If a future AGI isn't less competent than GPT-3, it'll also know this.

Once again, there is a school of thought in AI safety that says we should avoid this exact thing, i.e., train an AI such that the training data doesn't include information about huamns. I think this is called stem AI, where it would only do physics or something.

If you ask it a question a human has never been asked, it's still just going to do its best to predict what a human would say, not what the correct answer is.

GPT-3 is always trying to predict the next token in the sequence because that's the training objective. But -- clearly -- predicting the correct answer requires knowing the correct answer.