Timeline for Why would a decision making machine decide to destroy itself?
Current License: CC BY-SA 3.0
19 events
when toggle format | what | by | license | comment | |
---|---|---|---|---|---|
Dec 20, 2016 at 17:00 | comment | added | Cort Ammon | @Holger scale and nuance. A chess machine needs self reference too, to pick positions it is good at. However, the rules of chess are much cleaner, and it's easy to use tools like alpha beta pruning to simplify itsmodel of itself. A machine capable of parsing "maximize human well-being" "correctly" will be playing a nuanced game whose rules will not support such simple pruning | |
Dec 20, 2016 at 16:56 | comment | added | Cort Ammon | @benmillwood I agree with you there . I focused on what I felt was the core of the question, the destruction of The machine. The solution you propose is just as valid. The only difference is how much one believes an AI can help us | |
Dec 20, 2016 at 16:26 | comment | added | Ben Millwood | I find this unconvincing because giving an answer like "do action X, and then destroy the machine" also doesn't require any self-reference – the machine's simulated world still doesn't involve any further use of the machine. | |
Dec 20, 2016 at 16:20 | comment | added | Holger | @Cort Ammon: whether you plan a chess game or the entire humanity, is only a matter of scale. Of course, it would require lots of more computing power, but your answer focuses on the self reference, claiming that it makes a fundamental difference that could even turn to the computer’s decision to destroy itself. | |
Dec 20, 2016 at 15:18 | comment | added | Cort Ammon | @IshanSingh Yes, Infinite Descent is one of the major challenges facing this computer. Attempts to get around it get really interesting. | |
Dec 20, 2016 at 15:17 | comment | added | Cort Ammon | @Holger And when was the last time your daily business of strategic thinking tried to maximize the wellbeing of all humanity? It turns out to be harder than it might seem at first. Some goals are well suited for strategic thinking. Others are not. In philosophy, there is a large body of content dealing with utilitarianism, which is an attempt to bring human wellbeing into the strategic world. It's quite the challenge to even define the concept, much less find an answer. | |
Dec 20, 2016 at 9:45 | comment | added | Holger | Actually, that’s the daily business of all strategic thinking. You are planning for a future that is affected by your plans. That’s also how chess computers work. They are calculating lots of moves ahead, which includes their own future moves. | |
Dec 20, 2016 at 3:55 | comment | added | MathGod | Reminds me of Infinite Descent. | |
Dec 19, 2016 at 19:01 | comment | added | Cort Ammon | @Brian True, I didn't pay too much attention to that tag. The answer is still the same: the implementation details of the magic will factor into the answer. | |
Dec 19, 2016 at 18:52 | comment | added | Brian | I think we're just calling it magic, since the question has the "magic" tag. | |
Dec 19, 2016 at 18:47 | comment | added | Cort Ammon | @Brian True. I intentionally did not explore the issue of the computer somehow being able to model the infinite variations in reality with imperfect information. That one's a bit of a quagmire, and leads to the idea of hyper-Turing machines. If we call this computer a hyper-Turing machine, we'll need to understand its full capabilities, but that may open some answers in the form of "I want to live!" | |
Dec 19, 2016 at 18:43 | comment | added | Brian | That said, I think your answer is a good one. The machine certainly could be designed to simulate itself in full, I'm just saying that it wouldn't have to be. | |
Dec 19, 2016 at 18:37 | comment | added | Brian | @CortAmmon The machine has a defined range of outputs, which is presumably any instruction written in some language. Since the machine is already simulating infinite possibilities for all of reality, simulating the infinite list of possible outputs should be no problem. It would be essentially creating a mock version of itself that doesn't do any processing, but can output any instruction that the actual machine could. | |
Dec 19, 2016 at 17:27 | comment | added | Cort Ammon | @Brian Doing that implies that it can model itself to a sufficient degree, which would imply the computer could be made smaller. If it could simulate its response to its own response to itself (required to be able to answer 3 questions), the computer would need to have two smaller models of itself, nested. This approach does have a lot of potential value, but it runs into implementation challenges. If you succeed, you have to ponder whether you have made an AGI that's smarter than the entire human race combined, and maybe we just don't matter. | |
Dec 19, 2016 at 17:23 | comment | added | Brian | Since the machine is a black box, it would only have to simulate its possible outputs rather than its full self. | |
Dec 19, 2016 at 17:09 | comment | added | Dotan | I would argue that if the machine can simulate the world it's in without extra space, it derives that the simulated machines can simulate the simulated world without extra space. So that's not really a problem | |
S Dec 19, 2016 at 15:30 | history | mod moved comments to chat | |||
S Dec 19, 2016 at 15:30 | comment | added | HDE 226868♦ | Comments are not for extended discussion; this conversation has been moved to chat. | |
Dec 17, 2016 at 22:05 | history | answered | Cort Ammon | CC BY-SA 3.0 |