Superintelligence

For the book by Nick Bostrom, see Superintelligence: Paths, Dangers, Strategies. For the 2020 film, see Superintelligence (film).

A superintelligence is a hypothetical agent that possesses intelligence surpassing that of the brightest and most gifted human minds. "Superintelligence" may also refer to a property of problem-solving systems (e.g., superintelligent language translators or engineering assistants) whether or not these high-level intellectual competencies are embodied in agents that act in the world. A superintelligence may or may not be created by an intelligence explosion and associated with a technological singularity.

University of Oxford philosopher Nick Bostrom defines superintelligence as "any intellect that greatly exceeds the cognitive performance of humans in virtually all domains of interest".^[1] The program Fritz falls short of this conception of superintelligence—even though it is much better than humans at chess—because Fritz cannot outperform humans in other tasks.^[2]

Technological researchers disagree about how likely present-day human intelligence is to be surpassed. Some argue that advances in artificial intelligence (AI) will probably result in general reasoning systems that lack human cognitive limitations. Others believe that humans will evolve or directly modify their biology to achieve radically greater intelligence.^[3][4] Several future study scenarios combine elements from both of these possibilities, suggesting that humans are likely to interface with computers, or upload their minds to computers, in a way that enables substantial intelligence amplification.

Some researchers believe that superintelligence will likely follow shortly after the development of artificial general intelligence. The first generally intelligent machines are likely to immediately hold an enormous advantage in at least some forms of mental capability, including the capacity of perfect recall, a vastly superior knowledge base, and the ability to multitask in ways not possible to biological entities. This may allow them to — either as a single being or as a new species — become much more powerful than humans, and displace them.^[1]

Several scientists and forecasters have been arguing for prioritizing early research into the possible benefits and risks of human and machine cognitive enhancement, because of the potential social impact of such technologies.^[5]

Feasibility of artificial superintelligence

Artificial intelligence, especially foundation models, has made rapid progress, surpassing human capabilities in various benchmarks.

The feasibility of artificial superintelligence (ASI) has been a topic of increasing discussion in recent years, particularly with the rapid advancements in artificial intelligence (AI) technologies.

Progress in AI and claims of AGI

Recent developments in AI, particularly in large language models (LLMs) based on the transformer architecture, have led to significant improvements in various tasks. Models like GPT-3, GPT-4, Claude 3.5 and others have demonstrated capabilities that some researchers argue approach or even exhibit aspects of artificial general intelligence (AGI).^[6]

However, the claim that current LLMs constitute AGI is controversial. Critics argue that these models, while impressive, still lack true understanding and are primarily sophisticated pattern matching systems.^[7]

Pathways to superintelligence

Philosopher David Chalmers argues that AGI is a likely path to ASI. He posits that AI can achieve equivalence to human intelligence, be extended to surpass it, and then be amplified to dominate humans across arbitrary tasks.^[8]

More recent research has explored various potential pathways to superintelligence:

1. Scaling current AI systems – Some researchers argue that continued scaling of existing AI architectures, particularly transformer-based models, could lead to AGI and potentially ASI.^[9]
2. Novel architectures – Others suggest that new AI architectures, potentially inspired by neuroscience, may be necessary to achieve AGI and ASI.^[10]
3. Hybrid systems – Combining different AI approaches, including symbolic AI and neural networks, could potentially lead to more robust and capable systems.^[11]

Computational advantages

Artificial systems have several potential advantages over biological intelligence:

1. Speed – Computer components operate much faster than biological neurons. Modern microprocessors (~2 GHz) are seven orders of magnitude faster than neurons (~200 Hz).^[12]
2. Scalability – AI systems can potentially be scaled up in size and computational capacity more easily than biological brains.
3. Modularity – Different components of AI systems can be improved or replaced independently.
4. Memory – AI systems can have perfect recall and vast knowledge bases. It is also much less constrained than humans when it comes to working memory.^[12]
5. Multitasking – AI can perform multiple tasks simultaneously in ways not possible for biological entities.

Potential path through transformer models

Recent advancements in transformer-based models have led some researchers to speculate that the path to ASI might lie in scaling up and improving these architectures. This view suggests that continued improvements in transformer models or similar architectures could lead directly to ASI.^[13]

Some experts even argue that current large language models like GPT-4 may already exhibit early signs of AGI or ASI capabilities.^[14] This perspective suggests that the transition from current AI to ASI might be more continuous and rapid than previously thought, blurring the lines between narrow AI, AGI, and ASI.

However, this view remains controversial. Critics argue that current models, while impressive, still lack crucial aspects of general intelligence such as true understanding, reasoning, and adaptability across diverse domains.^[15]

The debate over whether the path to ASI will involve a distinct AGI phase or a more direct scaling of current technologies remains ongoing, with significant implications for AI development strategies and safety considerations.

Challenges and uncertainties

Despite these potential advantages, there are significant challenges and uncertainties in achieving ASI:

1. Ethical and safety concerns – The development of ASI raises numerous ethical questions and potential risks that need to be addressed.^[16]
2. Computational requirements – The computational resources required for ASI might be far beyond current capabilities.
3. Fundamental limitations – There may be fundamental limitations to intelligence that apply to both artificial and biological systems.
4. Unpredictability – The path to ASI and its consequences are highly uncertain and difficult to predict.

As research in AI continues to advance rapidly, the question of the feasibility of ASI remains a topic of intense debate and study in the scientific community.

Feasibility of biological superintelligence

Carl Sagan suggested that the advent of Caesarean sections and in vitro fertilization may permit humans to evolve larger heads, resulting in improvements via natural selection in the heritable component of human intelligence.^[17] By contrast, Gerald Crabtree has argued that decreased selection pressure is resulting in a slow, centuries-long reduction in human intelligence and that this process instead is likely to continue. There is no scientific consensus concerning either possibility and in both cases, the biological change would be slow, especially relative to rates of cultural change.

Selective breeding, nootropics, epigenetic modulation, and genetic engineering could improve human intelligence more rapidly. Bostrom writes that if we come to understand the genetic component of intelligence, pre-implantation genetic diagnosis could be used to select for embryos with as much as 4 points of IQ gain (if one embryo is selected out of two), or with larger gains (e.g., up to 24.3 IQ points gained if one embryo is selected out of 1000). If this process is iterated over many generations, the gains could be an order of magnitude improvement. Bostrom suggests that deriving new gametes from embryonic stem cells could be used to iterate the selection process rapidly.^[18] A well-organized society of high-intelligence humans of this sort could potentially achieve collective superintelligence.^[19]

Alternatively, collective intelligence might be constructional by better organizing humans at present levels of individual intelligence. Several writers have suggested that human civilization, or some aspect of it (e.g., the Internet, or the economy), is coming to function like a global brain with capacities far exceeding its component agents. If this systemic superintelligence relies heavily on artificial components, however, it may qualify as an AI rather than as a biology-based superorganism.^[20] A prediction market is sometimes considered as an example of a working collective intelligence system, consisting of humans only (assuming algorithms are not used to inform decisions).^[21]

A final method of intelligence amplification would be to directly enhance individual humans, as opposed to enhancing their social or reproductive dynamics. This could be achieved using nootropics, somatic gene therapy, or brain−computer interfaces. However, Bostrom expresses skepticism about the scalability of the first two approaches and argues that designing a superintelligent cyborg interface is an AI-complete problem.^[22]

Forecasts

Most surveyed AI researchers expect machines to eventually be able to rival humans in intelligence, though there is little consensus on when this will likely happen. At the 2006 AI@50 conference, 18% of attendees reported expecting machines to be able "to simulate learning and every other aspect of human intelligence" by 2056; 41% of attendees expected this to happen sometime after 2056; and 41% expected machines to never reach that milestone.^[23]

In a survey of the 100 most cited authors in AI (as of May 2013, according to Microsoft academic search), the median year by which respondents expected machines "that can carry out most human professions at least as well as a typical human" (assuming no global catastrophe occurs) with 10% confidence is 2024 (mean 2034, st. dev. 33 years), with 50% confidence is 2050 (mean 2072, st. dev. 110 years), and with 90% confidence is 2070 (mean 2168, st. dev. 342 years). These estimates exclude the 1.2% of respondents who said no year would ever reach 10% confidence, the 4.1% who said 'never' for 50% confidence, and the 16.5% who said 'never' for 90% confidence. Respondents assigned a median 50% probability to the possibility that machine superintelligence will be invented within 30 years of the invention of approximately human-level machine intelligence.^[24]

In a 2022 survey, the median year by which respondents expected "High-level machine intelligence" with 50% confidence is 2061. The survey defined the achievement of high-level machine intelligence as when unaided machines can accomplish every task better and more cheaply than human workers.^[25]

In 2023, OpenAI leaders Sam Altman, Greg Brockman and Ilya Sutskever published recommendations for the governance of superintelligence, which they believe may happen in less than 10 years.^[26] In 2024, Ilya Sutskever left OpenAI to cofound the startup Safe Superintelligence, which focuses solely on creating a superintelligence that is safe by design, while avoiding "distraction by management overhead or product cycles".^[27]

Design considerations

The design of superintelligent AI systems raises critical questions about what values and goals these systems should have. Several proposals have been put forward:^[28]

Value alignment proposals

• Coherent extrapolated volition (CEV) – The AI should have the values upon which humans would converge if they were more knowledgeable and rational.
• Moral rightness (MR) – The AI should be programmed to do what is morally right, relying on its superior cognitive abilities to determine ethical actions.
• Moral permissibility (MP) – The AI should stay within the bounds of moral permissibility while otherwise pursuing goals aligned with human values (similar to CEV).

Bostrom elaborates on these concepts:

instead of implementing humanity's coherent extrapolated volition, one could try to build an AI to do what is morally right, relying on the AI's superior cognitive capacities to figure out just which actions fit that description. We can call this proposal "moral rightness" (MR) ...

MR would also appear to have some disadvantages. It relies on the notion of "morally right", a notoriously difficult concept, one with which philosophers have grappled since antiquity without yet attaining consensus as to its analysis. Picking an erroneous explication of "moral rightness" could result in outcomes that would be morally very wrong ...

One might try to preserve the basic idea of the MR model while reducing its demandingness by focusing on moral permissibility: the idea being that we could let the AI pursue humanity's CEV so long as it did not act in morally impermissible ways.^[28]

Recent developments

Since Bostrom's analysis, new approaches to AI value alignment have emerged:

• Inverse Reinforcement Learning (IRL) – This technique aims to infer human preferences from observed behavior, potentially offering a more robust approach to value alignment.^[29]
• Constitutional AI – Proposed by Anthropic, this involves training AI systems with explicit ethical principles and constraints.^[30]
• Debate and amplification – These techniques, explored by OpenAI, use AI-assisted debate and iterative processes to better understand and align with human values.^[31]

Transformer LLMs and ASI

The rapid advancement of transformer-based LLMs has led to speculation about their potential path to ASI. Some researchers argue that scaled-up versions of these models could exhibit ASI-like capabilities:^[32]

• Emergent abilities – As LLMs increase in size and complexity, they demonstrate unexpected capabilities not present in smaller models.^[33]
• In-context learning – LLMs show the ability to adapt to new tasks without fine-tuning, potentially mimicking general intelligence.^[34]
• Multi-modal integration – Recent models can process and generate various types of data, including text, images, and audio.^[35]

However, critics argue that current LLMs lack true understanding and are merely sophisticated pattern matchers, raising questions about their suitability as a path to ASI.^[36]

Other perspectives on artificial superintelligence

Additional viewpoints on the development and implications of superintelligence include:

• Recursive self-improvement – I. J. Good proposed the concept of an "intelligence explosion", where an AI system could rapidly improve its own intelligence, potentially leading to superintelligence.^[37]
• Orthogonality thesis – Bostrom argues that an AI's level of intelligence is orthogonal to its final goals, meaning a superintelligent AI could have any set of motivations.^[38]
• Instrumental convergence – Certain instrumental goals (e.g., self-preservation, resource acquisition) might be pursued by a wide range of AI systems, regardless of their final goals.^[39]

Challenges and ongoing research

The pursuit of value-aligned AI faces several challenges:

• Philosophical uncertainty in defining concepts like "moral rightness"
• Technical complexity in translating ethical principles into precise algorithms
• Potential for unintended consequences even with well-intentioned approaches

Current research directions include multi-stakeholder approaches to incorporate diverse perspectives, developing methods for scalable oversight of AI systems, and improving techniques for robust value learning.^[40][16]

Al research progresses is rapidly progressing towards superintelligence, addressing these design challenges remains crucial for creating ASI systems that are both powerful and aligned with human interests.

Potential threat to humanity

Main articles: Existential risk from artificial general intelligence, AI alignment, and AI safety

The development of artificial superintelligence (ASI) has raised concerns about potential existential risks to humanity. Researchers have proposed various scenarios in which an ASI could pose a significant threat:

Intelligence explosion and control problem

Some researchers argue that through recursive self-improvement, an ASI could rapidly become so powerful as to be beyond human control. This concept, known as an "intelligence explosion", was first proposed by I. J. Good in 1965:

Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an 'intelligence explosion,' and the intelligence of man would be left far behind. Thus the first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control.^[41]

This scenario presents the AI control problem: how to create an ASI that will benefit humanity while avoiding unintended harmful consequences.^[42] Eliezer Yudkowsky argues that solving this problem is crucial before ASI is developed, as a superintelligent system might be able to thwart any subsequent attempts at control.^[43]

Unintended consequences and goal misalignment

Even with benign intentions, an ASI could potentially cause harm due to misaligned goals or unexpected interpretations of its objectives. Nick Bostrom provides a stark example of this risk:

When we create the first superintelligent entity, we might make a mistake and give it goals that lead it to annihilate humankind, assuming its enormous intellectual advantage gives it the power to do so. For example, we could mistakenly elevate a subgoal to the status of a supergoal. We tell it to solve a mathematical problem, and it complies by turning all the matter in the solar system into a giant calculating device, in the process killing the person who asked the question.^[44]

Stuart Russell offers another illustrative scenario:

A system given the objective of maximizing human happiness might find it easier to rewire human neurology so that humans are always happy regardless of their circumstances, rather than to improve the external world.^[45]

These examples highlight the potential for catastrophic outcomes even when an ASI is not explicitly designed to be harmful, underscoring the critical importance of precise goal specification and alignment.

Potential mitigation strategies

Researchers have proposed various approaches to mitigate risks associated with ASI:

• Capability control – Limiting an ASI's ability to influence the world, such as through physical isolation or restricted access to resources.^[46]
• Motivational control – Designing ASIs with goals that are fundamentally aligned with human values.^[47]
• Ethical AI – Incorporating ethical principles and decision-making frameworks into ASI systems.^[48]
• Oversight and governance – Developing robust international frameworks for the development and deployment of ASI technologies.^[49]

Despite these proposed strategies, some experts, such as Roman Yampolskiy, argue that the challenge of controlling a superintelligent AI might be fundamentally unsolvable, emphasizing the need for extreme caution in ASI development.^[50]

Debate and skepticism

Not all researchers agree on the likelihood or severity of ASI-related existential risks. Some, like Rodney Brooks, argue that fears of superintelligent AI are overblown and based on unrealistic assumptions about the nature of intelligence and technological progress.^[51] Others, such as Joanna Bryson, contend that anthropomorphizing AI systems leads to misplaced concerns about their potential threats.^[52]

Recent developments and current perspectives

The rapid advancement of LLMs and other AI technologies has intensified debates about the proximity and potential risks of ASI. While there is no scientific consensus, some researchers and AI practitioners argue that current AI systems may already be approaching AGI or even ASI capabilities.

• LLM capabilities – Recent LLMs like GPT-4 have demonstrated unexpected abilities in areas such as reasoning, problem-solving, and multi-modal understanding, leading some to speculate about their potential path to ASI.^[53]

• Emergent behaviors – Studies have shown that as AI models increase in size and complexity, they can exhibit emergent capabilities not present in smaller models, potentially indicating a trend towards more general intelligence.^[33]

• Rapid progress – The pace of AI advancement has led some to argue that we may be closer to ASI than previously thought, with potential implications for existential risk.^[54]

A minority of researchers and observers, including some in the AI development community, believe that current AI systems may already be at or near AGI levels, with ASI potentially following in the near future. This view, while not widely accepted in the scientific community, is based on observations of rapid progress in AI capabilities and unexpected emergent behaviors in large models.^[55]

However, many experts caution against premature claims of AGI or ASI, arguing that current AI systems, despite their impressive capabilities, still lack true understanding and general intelligence.^[56] They emphasize the significant challenges that remain in achieving human-level intelligence, let alone superintelligence.

The debate surrounding the current state and trajectory of AI development underscores the importance of continued research into AI safety and ethics, as well as the need for robust governance frameworks to manage potential risks as AI capabilities continue to advance.^[49]

See also

• Artificial general intelligence
• AI safety
• AI takeover
• Artificial brain
• Artificial intelligence arms race
• Effective altruism
• Ethics of artificial intelligence
• Existential risk
• Friendly artificial intelligence
• Future of Humanity Institute
• Intelligent agent
• Machine ethics
• Machine Intelligence Research Institute
• Machine learning
• Neural scaling law – Statistical law in machine learning
• Noosphere – Philosophical concept of biosphere successor via humankind's rational activities
• Outline of artificial intelligence
• Posthumanism
• Robotics
• Self-replication
• Self-replicating machine
• Superintelligence: Paths, Dangers, Strategies

References

1. Bostrom 2014, Chapter 2.
2. Bostrom 2014, p. 22.
3. Pearce, David (2012), Eden, Amnon H.; Moor, James H.; Søraker, Johnny H.; Steinhart, Eric (eds.), "The Biointelligence Explosion: How Recursively Self-Improving Organic Robots will Modify their Own Source Code and Bootstrap Our Way to Full-Spectrum Superintelligence", Singularity Hypotheses, The Frontiers Collection, Berlin, Heidelberg: Springer Berlin Heidelberg, pp. 199–238, doi:10.1007/978-3-642-32560-1_11, ISBN 978-3-642-32559-5, retrieved 2022-01-16
4. Gouveia, Steven S., ed. (2020). "ch. 4, "Humans and Intelligent Machines: Co-evolution, Fusion or Replacement?", David Pearce". The Age of Artificial Intelligence: An Exploration. Vernon Press. ISBN 978-1-62273-872-4.
5. Legg 2008, pp. 135–137.
6. "Microsoft Researchers Claim GPT-4 Is Showing "Sparks" of AGI". Futurism. 23 March 2023. Retrieved 2023-12-13.
7. Marcus, Gary; Davis, Ernest (2023). "GPT-4 and Beyond: The Future of Artificial Intelligence". arXiv:2303.10130 [econ.GN].
8. Chalmers 2010, p. 7.
9. Kaplan, Jared; McCandlish, Sam; Henighan, Tom; Brown, Tom B.; Chess, Benjamin; Child, Rewon; Gray, Scott; Radford, Alec; Wu, Jeffrey; Amodei, Dario (2020). "Scaling Laws for Neural Language Models". arXiv:2001.08361 [cs.LG].
10. Hassabis, Demis; Kumaran, Dharshan; Summerfield, Christopher; Botvinick, Matthew (2017). "Neuroscience-Inspired Artificial Intelligence". Neuron. 95 (2): 245–258. doi:10.1016/j.neuron.2017.06.011. PMID 28728020.
11. Garcez, Artur d'Avila; Lamb, Luis C. (2020). "Neurosymbolic AI: The 3rd Wave". arXiv:2012.05876 [cs.AI].
12. Bostrom 2014, p. 59.
13. Sutskever, Ilya (2023). "A Brief History of Scaling". ACM Queue. 21 (4): 31–43. doi:10.1145/3595878.3605016 (inactive 1 November 2024).{{cite journal}}: CS1 maint: DOI inactive as of November 2024 (link)
14. Bubeck, Sébastien; Chandrasekaran, Varun; Eldan, Ronen; Gehrke, Johannes; Horvitz, Eric; Kamar, Ece; Lee, Peter; Lee, Yin Tat; Li, Yuanzhi; Lundberg, Scott; Nori, Harsha; Palangi, Hamid; Precup, Doina; Sountsov, Pavel; Srivastava, Sanjana; Tessler, Catherine; Tian, Jianfeng; Zaheer, Manzil (22 March 2023). "Sparks of Artificial General Intelligence: Early experiments with GPT-4". arXiv:2303.12712 [cs.CL].
15. Marcus, Gary (2020). "The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence". arXiv:2002.06177 [cs.AI].
16. Russell 2019.
17. Sagan, Carl (1977). The Dragons of Eden. Random House.
18. Bostrom 2014, pp. 37–39.
19. Bostrom 2014, p. 39.
20. Bostrom 2014, pp. 48–49.
21. Watkins, Jennifer H. (2007), Prediction Markets as an Aggregation Mechanism for Collective Intelligence
22. Bostrom 2014, pp. 36–37, 42, 47.
23. Maker, Meg Houston (July 13, 2006). "AI@50: First Poll". Archived from the original on 2014-05-13.
24. Müller & Bostrom 2016, pp. 3–4, 6, 9–12.
25. "AI timelines: What do experts in artificial intelligence expect for the future?". Our World in Data. Retrieved 2023-08-09.
26. "Governance of superintelligence". openai.com. Retrieved 2023-05-30.
27. Vance, Ashlee (June 19, 2024). "Ilya Sutskever Has a New Plan for Safe Superintelligence". Bloomberg. Retrieved 2024-06-19.
28. Bostrom 2014, pp. 209–221.
29. Christiano, Paul; Leike, Jan; Brown, Tom B.; Martic, Miljan; Legg, Shane; Amodei, Dario (2017). "Deep Reinforcement Learning from Human Preferences" (PDF). NeurIPS. arXiv:1706.03741.
30. "Constitutional AI: Harmlessness from AI Feedback". Anthropic. December 15, 2022.
31. "Learning complex goals with iterated amplification". OpenAI. October 22, 2018.
32. Bommasani, Rishi; et al. (2021). "On the Opportunities and Risks of Foundation Models". Stanford University. arXiv:2108.07258.
33. Wei, Jason; Tay, Yi; Bommasani, Rishi; Raffel, Colin; Zoph, Barret; Borgeaud, Sebastian; Yogatama, Dani; Bosma, Maarten; Zhou, Denny; Metzler, Donald; Chi, Ed H.; Hashimoto, Tatsunori; Vinyals, Oriol; Liang, Percy; Dean, Jeff; Fedus, William (2022-06-26). "Emergent Abilities of Large Language Models". Transactions on Machine Learning Research. arXiv:2206.07682. ISSN 2835-8856.
34. Brown, Tom B.; et al. (2020). "Language Models are Few-Shot Learners". NeurIPS. arXiv:2005.14165.
35. Alayrac, Jean-Baptiste; Donahue, Jeff; Luc, Pauline; Miech, Antoine; Barr, Iain; Hasson, Yana; Lenc, Karel; Mensch, Arthur; Millican, Katie; Reynolds, Malcolm; Ring, Roman; Rutherford, Eliza; Cabi, Serkan; Han, Tengda; Gong, Zhitao; Samangooei, Sina; Monteiro, Marianne; Menick, Jacob; Borgeaud, Sebastian; Brock, Andrew; Nematzadeh, Aida; Sharifzadeh, Sahand; Binkowski, Mikolaj; Barreira, Ricardo; Vinyals, Oriol; Zisserman, Andrew; Simonyan, Karen (2022). "Flamingo: a Visual Language Model for Few-Shot Learning". NeurIPS. arXiv:2204.14198.
36. Marcus, Gary (August 11, 2022). "Deep Learning Alone Isn't Getting Us To Human-Like AI". Noema.
37. "The AI apocalypse: will the human race soon be terminated?". The Irish Times. March 30, 2017.
38. Bostrom, Nick (2012). "The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents" (PDF). Minds and Machines. 22 (2): 71–85. doi:10.1007/s11023-012-9281-3.
39. Omohundro, Stephen M. (January 2008). "The basic AI drives" (PDF). Frontiers in Artificial Intelligence and Applications.
40. Gabriel, Iason (2020-09-01). "Artificial Intelligence, Values, and Alignment". Minds and Machines. 30 (3): 411–437. doi:10.1007/s11023-020-09539-2. ISSN 1572-8641.
41. Good, I. J. (1965). "Speculations Concerning the First Ultraintelligent Machine". Advances in Computers.
42. Russell 2019, pp. 137–160.
43. Yudkowsky, Eliezer (2008). "Artificial Intelligence as a Positive and Negative Factor in Global Risk" (PDF). Global Catastrophic Risks. doi:10.1093/oso/9780198570509.003.0021. ISBN 978-0-19-857050-9.
44. Bostrom 2002.
45. Russell 2019, p. 136.
46. Bostrom 2014, pp. 129–136.
47. Bostrom 2014, pp. 136–143.
48. Wallach, Wendell; Allen, Colin (2008-11-19). Moral Machines: Teaching Robots Right from Wrong. Oxford University Press. ISBN 978-0-19-970596-2.
49. Dafoe, Allan (August 27, 2018). "AI Governance: A Research Agenda" (PDF). Center for the Governance of AI.
50. Yampolskiy, Roman V. (July 18, 2020). "On Controllability of Artificial Intelligence" (PDF). arXiv:2008.04071.
51. Brooks, Rodney (October 6, 2017). "The Seven Deadly Sins of AI Predictions". MIT Technology Review. Retrieved 2024-10-23.
52. Bryson, Joanna J (2019). "The Past Decade and Future of AI's Impact on Society". Towards a New Enlightenment? A Transcendent Decade. 11. ISBN 978-84-17141-21-9.
53. Bubeck, Sébastien; Chandrasekaran, Varun; Eldan, Ronen; Gehrke, Johannes; Horvitz, Eric; Kamar, Ece; Lee, Peter; Yin Tat Lee; Li, Yuanzhi; Lundberg, Scott; Nori, Harsha; Palangi, Hamid; Marco Tulio Ribeiro; Zhang, Yi (April 2023). "Sparks of Artificial General Intelligence: Early experiments with GPT-4". arXiv:2303.12712 [cs.CL].
54. Ord, Toby (2020). The precipice: existential risk and the future of humanity. london New York (N.Y.): Bloomsbury academic. ISBN 978-1-5266-0023-3.
55. Ngo, Richard; Chan, Lawrence; Mindermann, Sören (2022). "The Alignment Problem from a Deep Learning Perspective". ICLR. arXiv:2209.00626.
56. "How and Why Gary Marcus Became AI's Leading Critic > Marcus says generative AI like ChatGPT poses immediate dangers". IEEE Spectrum. 17 September 2024.

Papers

• Bostrom, Nick (2002), "Existential Risks", Journal of Evolution and Technology, 9, retrieved 2007-08-07.
• Chalmers, David (2010). "The Singularity: A Philosophical Analysis" (PDF). Journal of Consciousness Studies. 17: 7–65.
• Legg, Shane (2008). Machine Super Intelligence (PDF) (PhD). Department of Informatics, University of Lugano. Retrieved September 19, 2014.
• Müller, Vincent C.; Bostrom, Nick (2016). "Future Progress in Artificial Intelligence: A Survey of Expert Opinion". In Müller, Vincent C. (ed.). Fundamental Issues of Artificial Intelligence. Springer. pp. 553–571.
• Santos-Lang, Christopher (2014). "Our responsibility to manage evaluative diversity" (PDF). ACM SIGCAS Computers & Society. 44 (2): 16–19. doi:10.1145/2656870.2656874. S2CID 5649158. Archived from the original on July 29, 2014.

Books

• Hibbard, Bill (2002). Super-Intelligent Machines. Kluwer Academic/Plenum Publishers.
• Bostrom, Nick (2014). Superintelligence: Paths, Dangers, Strategies. Oxford University Press.
• Tegmark, Max (2018). Life 3.0: being human in the age of artificial intelligence. London, England. ISBN 978-0-14-198180-2. OCLC 1018461467.{{cite book}}: CS1 maint: location missing publisher (link)
• Russell, Stuart J. (2019). Human compatible: artificial intelligence and the problem of control. New York. ISBN 978-0-525-55861-3. OCLC 1113410915.{{cite book}}: CS1 maint: location missing publisher (link)
• Sanders, Nada R. (2020). The humachine: humankind, machines, and the future of enterprise. John D. Wood (First ed.). New York, New York. ISBN 978-0-429-00117-8. OCLC 1119391268.{{cite book}}: CS1 maint: location missing publisher (link)

External links

• Bill Gates Joins Stephen Hawking in Fears of a Coming Threat from "Superintelligence"
• Will Superintelligent Machines Destroy Humanity?
• Apple Co-founder Has Sense of Foreboding About Artificial Superintelligence