Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[silverspawn]

I forgot "programs that don't let you choose their  installation path" on my top 10 most annoying things list

[silverspawn]

Then, in my lower-bound concretely-visualized strategy for how I would do it, the AI either proliferates or activates already-proliferated tiny diamondoid bacteria and everybody immediately falls over dead during the same 1-second period, which minimizes the tiny probability of any unforeseen disruptions that could be caused by a human responding to a visible attack via some avenue that had not left any shadow on the Internet, previously scanned parts of the physical world, or other things the AI could look at.

The cynical view suggests that, even in the event that programmers see an earlier and dumber version of the AI actively thinking things like this (which is very optimistic anyway because presently our interpretability tools are infinite far away from being able to do anything like this (I'll know; I just wrote a paper about interpretability), and also it assumes there is a stage where the AI is smart enough to plot but not smart enough to hide it), this probably won't prevent the disaster, rather the following happens

- programmers try to patch evil motivations and utterly fail
- AI gets smart enough to hide plotting
- programmers high five each other because they no longer see the AI plotting
- AI still kills everyone

An even if a miracle happens and a dumber AI tries to do something malicious and fails but kills a few people in the process, will that really make a difference? Covid killed 790 000 americans and this didn't cause bureaucracy to move faster

There are times when I hope that the people who say we have nothing to worry about because X (there is no common value for X: everyone has different reasons) are right after all, but I'm afraid I just know better.

[faust]

The optimization problem I'm trying to solve is minimize P(everyone dies). A phd looked like a reasonable step toward that, but recently I'm less sure

I don't know the answer, but I can say that I wouldn't recommend a PhD if your main motivation is extrinsic.

why not?

To make the answer more productive: the case for a PhD is roughly

- You want to do research anyway, and a phd allows you to get paid to do that
- You have to learn new things anyway, and a phd allows you to do that
- Machine Learning engineers are really well paid, so having a phd gives you the fallback plan to do earning to give

In this case, it depends a lot on what kind of research I would be doing during the phd; the answer is likely interpretability research since that's in the intersection of safety relevant and mainstream.

Well the first point is important I feel like. If that's not there, then it is going to be hard to last through a PhD. I wasn't sure in my first answer whether that was there.
To the second I would say - you definitely learn new stuff during a PhD, but not as much as during undergrad studies. A lot of time is also spent on trying to make things work.

And then it's worth keeping in mind the downsides. I think during non-COVID times, I would attend at least one overseas conference a year, that comes with a signficant CO2 footprint, and with the way academia currently works is kind of necessary for a successful PhD. You might have more conference availability reachable by train though; after all, machine learning is way bigger a field than knot theory.
The other cost is in mental health. I don't have hard data to back this up, but my impression is that overall PhD students have a significantly higher risk of depression or related sicknesses. Of course some of that is probably selection bias, but not all I believe.

[silverspawn]

I totally buy that last one, but I also think that I'm going to be unusually capable of dealing with that. Primarily because of mindfulness. The prospect of a lot of lonely work on a single topic doesn't scare me.

I was steelmanning the first point. I would do research, but it's not going to be the kind of research that I think has the most potential. In a nutshell, that's the reason for me now doubting the whole idea.

[silverspawn]

Philosophical question that was asked somewhere: could you have a universe where Libertarian Free Will is true?

The short answer is, I think, "no". Even if I gave you god tier powers to create anything you want and change the laws of physics, you can't get Libertarian free will because the concept is incoherent.

Imagine being on a chess board and looking at an enemy knight. From your perspective, the knight may be said to have free will. It doesn't matter how fully you understand the rules of chess, you can't predict where the knight is going. Chess permits several possibilities and specifies no laws about how the knight must move. It's also not random.

This of course is not free will as people want to have, the knight is just controlled by someone outside the system. But I think that's secretly exactly what's going on if people try to come up with ways that LFW could be the case. They have to say things like "well there have to be entities in the environment which are not themselves governed by the environment" (because if they are just physics, well then physics determines their actions). But then they have to obey their own laws, and now they're just like the knights on the chess board. You've just moved the part where they're lawful one level up.

So "not obeying physical laws" is a nonsequitor. They have to obey some laws, and if they do, it's either deterministic or random because there is no other known principle. At the very least, I've never seen anyone explain how the concept could be coherent.

[silverspawn]

You can of course have all sorts of  notions of free will that are compatible with determinism as some people like Daniel Dennet like to do. But at that point, the question is entirely about words.

[silverspawn]

Random mathematical tangent:

People often say that the real numbers are uncountable. I was told this at at least once at the university, probably several times. I've also read a whole bunch of textbooks without having that corrected.

And yet it's not true. You can't prove whether the real numbers are countable or not. (For those interested, "countable" means "I can specify some fixed way of listing them all one after the other, such that, when you point at any one real number, I always say "ah, that's #3523412 in the list" or something like that.)

The proof goes like this:

1. Take the natural numbers {0, 1, 2, 3, ...}
2. Take the set of all subsets of natural numbers {set of even numbers, set of odd numbers, set of only {1,2}, ...}
3. Prove that this set is uncountable
4. Prove that this set is bijective to the set of real numbers
5. Conclude that the reals are uncountable

Steps 1,2, 4,5 are fine. Step 3 is not. You can't prove that the set of subsets of natural numbers is uncountable.

The argument that purports to show this is called the diagonal lemma. In a nutshell, you take an arbitrary way of listing these subsets and then construct from that a particular subset that's not on the list.

This indeed works. You can construct such a collection. However, that collection may not be a set that exists. This is the problem.

People always point to the power set axiom, one of the axioms of set theory. They think it says that, given any set, all subsets of that set exist and there is a different set that contains them all. Wrong! The power set axiom only says "given a set X, there is a set P such that, for any set S that happens to exist and is a subset of X, that set is in P (and no other sets are in P)". it doesn't say anything about whether or how many such sets P exist. The collection you construct with the diagonal lemma may just not be a set that exists.

In the most common formalization of math, you can't prove that there are unaccountably many sets period. Not in the real numbers and not anywhere else. You can't prove that the set of subsets of real numbers is uncountable. You need second order logic to do that, and while I like that approach, most mathematicians don't think of math as formalized that way.

[faust]

This of course is not free will as people want to have, the knight is just controlled by someone outside the system. But I think that's secretly exactly what's going on if people try to come up with ways that LFW could be the case. They have to say things like "well there have to be entities in the environment which are not themselves governed by the environment" (because if they are just physics, well then physics determines their actions). But then they have to obey their own laws, and now they're just like the knights on the chess board. You've just moved the part where they're lawful one level up.

It seems to me that there is a logical leap in here; why do these higher entities have to have some laws restricting them?

[silverspawn]

That step is why I added the final sentence. You can say the phrase "an entity isn't governed by any laws" but what does that mean? If it does a thing, how is that decision made?

[faust]

Random mathematical tangent:

Oh this bring me back to the set theory course I took during my undergrad 

I think the diagonal lemma works like this (more formally):

Assume a bijective map f: N -> P(N).
Define a new map g: N -> N as follows:
g(0) = min(n \in N | not(n \in f(n)))
g(k+1) = min(n > g(k) | not(n \in f(n)))

By the axiom schema of replacement, g(N) is a set.
Assume g(N) = f(n) for some n.
Is n in g(N)? Either answer leads to a contradiction.

Really the reason this works is the axiom schema of replacement, which essentially implies that any subclass of a set is again a set.

[faust]

That step is why I added the final sentence. You can say the phrase "an entity isn't governed by any laws" but what does that mean? If it does a thing, how is that decision made?

By free will? I see that this is hard to conceptualize, but that doesn't mean that it's logically inconsistent. You say it's either deterministic or random because there are no other known principles, but I am sure an advocate for the free will theory would argue that free will is a third principle. You might say that's incoherent, but I'm not sure how it's any more incoherent than something being "random".

[silverspawn]

Random mathematical tangent:

Oh this bring me back to the set theory course I took during my undergrad 

I think the diagonal lemma works like this (more formally):

Assume a bijective map f: N -> P(N).
Define a new map g: N -> N as follows:
g(0) = min(n \in N | not(n \in f(n)))
g(k+1) = min(n > g(k) | not(n \in f(n)))

By the axiom schema of replacement, g(N) is a set.
Assume g(N) = f(n) for some n.
Is n in g(N)? Either answer leads to a contradiction.

As written, this fails on the step where g may not be defined because {n > g(k) | not(n \in f(n)} could be empty. But I suppose you can fix this by first singling out the case where {n | not(n \in f(n)} is finite; in that case, let m be the largest member of the set, then for all k > m, we have k \in f(k) and hence all subsets of {0, ..., k-1} aren't reached by g(k), g(k+1), g(k+2), ... and since there are 2^k many, they can't all be reached by g(0, ..., g(k)) either, so we have a missing set that way.

I think the part where this version fails is just that g may not exist.

[silverspawn]

That step is why I added the final sentence. You can say the phrase "an entity isn't governed by any laws" but what does that mean? If it does a thing, how is that decision made?

By free will? I see that this is hard to conceptualize, but that doesn't mean that it's logically inconsistent. You say it's either deterministic or random because there are no other known principles, but I am sure an advocate for the free will theory would argue that free will is a third principle. You might say that's incoherent, but I'm not sure how it's any more incoherent than something being "random".

I think there is a good chance that true randomness is also logically incoherent. That said, I admit it's not easy to argue that this doesn't work (although you can't formalize how it would work).

[faust]

I think the part where this version fails is just that g may not exist.

I don't remember the details but you can definitely prove that recursive definitions give valid functions, so I am not sure at which point the existence of g would be in question.

[silverspawn]

I think the part where this version fails is just that g may not exist.

you can definitely prove that recursive definitions give valid functions

I doubt it. Most likely, the proof uses the overpowered power set axiom somewhere (which assumes all of the collections-that-look-like-subsets exist), but there is no overpowered power set axiom. There is only a regular power set axiom, which doesn't tell you that any of those sets exist.

I talked about this *because* people do it wrong all the time even at advanced math courses. When I read that there are countable models of ZFC, this was a major wtf moment. Alas there is a theorem that has exactly this as a trivial corollary, so your proof has to fail at some point. Probably it's the point where you don't recall the details.

[silverspawn]

(btw your signature doesn't refer to anything related to this thread does it?)

[faust]

I think the part where this version fails is just that g may not exist.

you can definitely prove that recursive definitions give valid functions

I doubt it. Most likely, the proof uses the overpowered power set axiom somewhere (which assumes all of the collections-that-look-like-subsets exist), but there is no overpowered power set axiom. There is only a regular power set axiom, which doesn't tell you that any of those sets exist.

I talked about this *because* people do it wrong all the time even at advanced math courses. When I read that there are countable models of ZFC, this was a major wtf moment. Alas there is a theorem that has exactly this as a trivial corollary, so your proof has to fail at some point. Probably it's the point where you don't recall the details.

I'm confused why you are so adamant about this. Surely foundations of set theory have been examined very closely. You say "most mathematicians don't think of math as formalized that way" (referring to second order logic), but it was pretty much that same group of people that introduced ZFC and the concept of second-order logic.

I don't know much about model theory, but I suppose there can be a countable model of ZFC because in that model "uncountable" will mean something different from our usual understanding.

(btw your signature doesn't refer to anything related to this thread does it?)

I mean, in a general sort of way? It's a quote from the Netflix special "Inside".

[faust]

I don't think AI will kill us all in the foreseeable future though, if that's what you're asking.

[silverspawn]

no, countable means exactly the same thing. There are only countably many sets (and in particular only countably many real numbers). You can't prove that the reals are countable unless you flat out reject the first order model.

If you do have second order logic, then you can have a power set axiom that actually does what people think the power set axiom does. But then you have a totally different set of axioms with different problems. In particular, you probably know Godel's incompleteness theorem. Whenever people who don't know math bring this up, they always think it says "there are valid statements which aren't provable". This isn't what it says, and in fact Godel's completeness theorem says the opposite, i.e. that all valid statements are provable. But that's again all first order logic. If you take the second order model seriously, then you actually have the proper incompleteness theorem that really says not all valid statements are provable.

So your choice is to a) accept that all proofs about uncountable sets don't really work, and in fact the reals could be countable; or b) lose Godel's completeness theorem and have true-but-unprovable statements.

My impression from attending math courses, reading text books, talking to people, and googling  has been that first order logic is the unquestioning default and second order logic this weird thing that a few people study on the side. (But then those same people also conveniently ignore the implications most of the time.) Hence the framing of "you can't prove the reals uncountable"; I'm treating first order models as the default.

[silverspawn]

I mean, just take yourself. You clearly know the first order axioms of ZFC. I don't even know if second order models have a replacement scheme. Probably not? If anything you can probably get there with a single axiom. But I don't know because I don't know any good textbooks about second order logic

[silverspawn]

I don't think AI will kill us all in the foreseeable future though, if that's what you're asking.

Yeah, you thinking that would have been an extremely low probability event in my model. Ironically lower than AI not killing us.

[faust]

no, countable means exactly the same thing. There are only countably many sets (and in particular only countably many real numbers). You can't prove that the reals are countable unless you flat out reject the first order model.

Upon first glance, this stackexchange thread seems to contradict your statement. Obviously it's not the best source but at least it seems that this problem has been considered, and I cannot provide more as I am not an expert in the matter.

I don't think AI will kill us all in the foreseeable future though, if that's what you're asking.

Yeah, you thinking that would have been an extremely low probability event in my model. Ironically lower than AI not killing us.

I wonder why you think that; in my mind, it's not too hard to imagine a world in which I believe that.

[silverspawn]

I believe you are right! The difference between inside and outside cardinality is new to me, but I doubt these people made it up.

Unsure if this means the diagonal proof works after all but mb it does

[silverspawn]

I wonder why you think that; in my mind, it's not too hard to imagine a world in which I believe that.

I don't have a hard time imagining you believing that all along, but a very hard time imagining I had anything to do with that. Causing minds to change is always a very low probability event.

But, accidentally or not, the signature fits really well

[silverspawn]

Actually that's not quite correct, I think I also have a hard time imagining you believe it in the first place, just because it would locate the biggest problem in the world at a purely technical challenge that could plausibly be solved by a few hundred people not being stupid for a few years, rather than anything more, well, profound.