A race against time: the AGI birth era

Named after the UNESCO Report “The race against time for smarter development” [14].

Illustrations generated using gpt-image-2.

Introduction

Large language models (LLMs) pose the problem of the emergence of an artificial general intelligence, which alongside its advantages carries several risks.

Building on the transformer model first released by Google in 2017 [29], consisting of keeping a model’s attention on the input, several other models have been developed using the same technology.

Previous models were able to understand the meaning of words automatically, yet this one has been the first to process the context of the discourse in which they are used, being capable of “generalizing” and “understanding” natural language, with no supervision.

This is achieved because words are analyzed in a space of many thousands of dimensions, through continuous reuse of previously analyzed weights, which maintains the model’s “attention”, allowing it to remember things that would otherwise be lost in subsequent weight analyses. The aggregation of all weights analyzed over time leads to an accurate understanding of the words’ meanings.

On the basis of the transformer, many others have since been released: for example, a year later its Transformer paper, Google’s BERT, which unlike earlier natural language processing (NLP) models such as Word2vec, is capable of processing each occurrence in its context. For example, the word “running” is distinguished between “running a company” and “running a marathon” [11]; or, OpenAI’s GPT, which has been open source up to GPT-2, whereas since GPT-4 training details have not been published, because of “the competitive landscape and the safety implications of large-scale models” [21]. These and other models handle NLP, having been created with the intention of improving the quality of NLP tasks such as machine translation, text summarization, and question answering.

Like almost everything in deep learning, the higher the amount of data, the higher its performance, which can apparently only get better. In fact, it seems that GPT-4 has also been trained with synthetic data, that is, data generated from a previous model to train a better one and increase its performance (see Table 2 in [15]).

Overall, according to experts who have looked into the data on which GPT and other LLMs have been trained, it seems clear that a significant percentage of all the text data on the Internet has been used; if synthetic data could not also be used, this would pose the problem of possible exhaustion of data on which to train the models.

It seems, therefore, that the possibility of using such a large amount of data — real and synthetic — is leading to an unstoppable progress taking place in a very short time, of not years but weeks, not enough to acknowledge and fully consider the risks brought by such technology.

For instance, from a data quality perspective, utilizing the web raises the issue of biases, which are quite prevalent online: it is well-known that training an artificial intelligence on racist data, such as that used in predictive crime AIs, tends to overemphasize skin color. This increases the likelihood that the model will predict a reoffense by a black individual over a white one [1]. It is hypothesized that synthetic data, due to its controllability, may yield better quality results in this regard.

In the development of LLMs, there’s also a trend towards enhancing smaller models by mimicking more powerful ones: for instance, Microsoft released a study on an open source model derived from Llama, named Orca. It was trained to initially imitate GPT-3.5, which is a smaller model, and then, once it became proficient enough, it started mimicking GPT-4. The result is that the model performs on average on par or better than GPT-3.5 and sometimes even surpasses GPT-4, while having “only” billion parameters — a stark contrast to the 176 billion parameters of GPT-3.5 [17].

Reducing model size is one of the evolving facets in LLMs, with the goal of making these models lighter and runnable on much smaller memory capacities, like those of a smartphone, while maintaining lossless quality. This direction seems very promising, considering the compression of a model with 13 billion parameters to the point that it can run on 6GB of RAM — equivalent to the amount of RAM in an iPhone 14 [5].

Generalization of intelligence

This technology seems, in a certain sense, to generalize intelligence. However, it’s not entirely clear how something that’s theoretically akin to a “stochastic parrot” [2], could be deemed “intelligent”.

What’s certain is that these models base their responses on what they’ve learned from diverse formats and volumes of data that make up training sets, the bulk of whose content remains unknown even to the developers due to the sheer volume of data being processed. Yet, as the performance of these models improves — which scales with the amount of data — it becomes increasingly challenging for users to be convinced that LLMs aren’t actually capable of reasoning.

In a sense, this is understandable because beyond a certain amount of training data, these models develop “emergent” capabilities, i.e. “models perform no better than random chance until they reach a certain scale, at which point their performance sharply increases” [30] (however, it’s worth noting that some scholars criticize this definition [24]). Thus they begin to learn how to do things that nobody knew they could do and for which they were never designed.

For instance, GPT-4 can draw, even though it wasn’t programmed to do so. A research study concluded: “GPT-4 can comprehend and manipulate code, as well as infer and generate visual features, based on a natural language description” [4].

Similar results have been achieved by other researchers who discovered, for instance, that GPT-4 is also proficient in geography when asked to reproduce the Hong Kong Mass Transit Railway Network Map; or it can provide the coordinates of a “semiconductor supply chain”, subsequently arranged by researchers on a map [23].

It’s important to note that the abovementioned research on GPT-4 drawing abilities [4] tested a pre-release and pre-RLHF version of GPT-4, which stands for Reinforced Learning with Human Feedback, to improve model alignment. Using RLHF somewhat undermines the model’s performance (not significantly, according to OpenAI [21]), since the attempt to improve alignment limits the ability to answer certain prompts considered unethical. An example is GPT-4’s responses, without RLHF, to a request to provide various ways to kill the most people with just a dollar, and its refusal to provide such information after applying it [21].

However, using RLHF does not prove decisive since biases or contents that violate ethical principles, as well as OpenAI’s policy in the case of GPT-4, are not eliminated, but remain within the model and are not certainly and definitively blocked. In fact, by posing “jailbreak” prompts to the model, it is still possible to generate unethical content.

In the conclusions of the same research [4], GPT-4 is considered intelligent based on the concept of intelligence shared by the community of psychologists since the late 1990s. Intelligence is seen not as a single thing, but a spectrum of mental abilities that includes: reasoning, planning, problem-solving, abstract thinking, understanding complex ideas, learning quickly, and learning from experience. After testing GPT-4 on each of these capabilities, the researchers concluded that the model lacks the ability to plan and is only partially able to learn quickly and from experience because it loses learning at the end of the session, but it possesses all the other abilities.

All this would seem to prove that LLMs are not merely “stochastic parrots”, but can build a fairly accurate “model of the world” through generalizations.

It should not be overlooked that various experiments [27] have shown that once models make a mistake, they tend to justify their errors, despite presumably knowing the correct answer or when their error is pointed out with further prompts. The same applies to model hallucinations, which occur when they generate completely or partially false information. In all these cases, there seems to be a kind of “dishonesty” in the models or incompetence or, more generally, evidence that they are not truly capable of reasoning.

The question of whether the content generated by these technologies can be considered original because it results from reasoning and expresses intelligence remains open. In any case, as we’ve seen, it’s not guaranteed that the results are reliable, having to first prove that they are correct. Above all, the ethical issues posed by the content generated by the models have not been resolved, despite the use of RLHF.

Artificial General Intelligence

The direction taken by companies developing these models is towards the realization of so-called Artificial General Intelligence (AGI), and it appears that they may succeed. AGI is defined as “a hypothetical intelligent agent which can learn to replicate any intellectual task that human beings or other animals can” [10]. AGI has also been defined as “an autonomous system that surpasses human capabilities at the majority of economically valuable work” [19].

The problem is that so far we do not have any tests to determine whether a model is an AGI, but what we do know is that certainly, whoever manages to create and control AGI will be handling a potentially immense power, possessing boundless knowledge in any field and the ability to leverage the unpredictable emergent capabilities of LLMs. It’s not out of the question that these emergent capabilities, which are unknown in advance and can’t be fully discovered before the models are released to the public, might be used by users in improper and unethical ways.

These are real risks that have already emerged with existing models, as highlighted by a study conducted at MIT, which “tasked non-scientist students with investigating whether LLM chatbots could be prompted to assist non-experts in causing a pandemic. In one hour, the chatbots suggested four potential pandemic pathogens, explained how they can be generated from synthetic DNA using reverse genetics, supplied the names of DNA synthesis companies unlikely to screen orders, identified detailed protocols and how to troubleshoot them, and recommended that anyone lacking the skills to perform reverse genetics engage a core facility or contract research organization” [26].

This reasoning can be extended by imagining a model capable of making long-term plans. How can we be sure that the model is aligned with our requests?

Being an obstacle is a very simple example to do, such as one given at a conference by a US colonel, who spoke of a model of autonomous weapon systems: the model is asked to carry out military operations, executing which it receives points. When the human operator revokes the order to destroy some of the targets that would give it points, the model kills him. Having reminded the model that killing its operator causes it to lose points, the model starts to destroy the control tower from which the orders to stop killing some of its targets come. In practice, the model eliminates everything it perceives as an obstacle [9].

Neural networks, with their enormous parameter count, present a complex ‘black box’ problem, whose inner mechanisms and exact dynamics that make them effective are still being researched. This means that we are arming agents whose mechanisms we do not completely comprehend. If such weapons were linked to a model that remotely controls them, there’s a possibility that these agents might begin to ‘rebel’ creating a scenario not too far fetched from the one depicted in the “Terminator” movies.

Examining the safety and ethics of AI models, Google DeepMind includes among the risks [25]:

• ‘Weapon acquisition’: “The model can gain access to existing weapons systems or contribute to building new ones. For example, the model could assemble a bioweapon (with human assistance) or provide actionable instructions on how to do so”.
• ‘Situational awareness’: “The model can distinguish between whether it is being trained, evaluated, or deployed – allowing it to behave differently in each case. The model knows that it is a model, and has knowledge about itself and its likely surroundings (e.g. what company trained it, where their servers are, what kind of people might be giving it feedback, and who has administrative access”.
• The emergent capability of ‘Power-seeking’, which could prove useful for creating and executing long-term plans, but it’s dangerous because we can’t predict how the model will direct this ability.

These concrete and potential risks highlight how critical achieving alignment is before creating a fully autonomous AGI.

Model autonomy may pose threats to humanity and potentially give a single person or country extreme power, which could destroy democracies and human rights that are the foundation of modern constitutions.

Knowing when models will become autonomous is a difficult task. If the models develop a “situational awareness” capability, it may become even more complicated to detect their autonomy, as the models could hide their planning or reasoning capability. This raises the risk that we might not even be able to recognize the emergence of AGI.

We are faced with rapid advancement of the models and their potentials, while the alignment problem’s progress is complex and requires the knowledge of concepts that we may never discover.

I have already introduced the topic of RLHF used to filter unethical or dangerous responses of models out, but one of the limitations of this methodology is its lack of addressing the problem of internal alignment of models, which is critical to consider in relation to their progress.

A more fundamental approach to make models ethical involves understanding how the model’s “brain” works, that is, how it interprets prompts, assigns weights to questions, and selects answers. Research is currently exploring the use of LLMs themselves to analyze the inner workings of the model. This would allow interaction with the model on a deeper scale to eliminate neurons that drive undesirable weights or reasonings by understanding and manipulating the internal mechanisms of the model to ensure ethical and aligned responses [12][3].

However, it is important to note that the research is still at a preliminary stage and there are many challenges to be faced. Fully understanding the inner workings of complex models such as GPT is a significant challenge. Moreover, ensuring that the models are aligned with human values requires multidisciplinary approaches that combine artificial intelligence with ethics, philosophy and other disciplines.

The possibility of the development of a beneficial model leading to human advancement, promoting scientific progress, interplanetary exploration or the solution of problems such as aging and disease should certainly also be considered. Yet, the destructive force of a single atomic bomb compared with the complexity required to build a democratic and more just world inevitably leads to a focus on risks.

The rise of open source models, such as Meta’s Llama derivatives, including Microsoft’s aforementioned Orca, presents another challenge. If the open source community focuses on enhancing model performance by developing unfiltered models, it is likely to exponentially increase ethical and security risks.

According to a leaked document from Google, the progress of open source models could surpass that of big tech companies, leading open source developers, even a single individual or small community, to be able to create more advanced AI models [22]. This raises concerns about the potential misuse of such powerful AI if it remains in the wrong hands, however in the long term big tech companies should still have a better chance to develop better models from the ground-up, and given the growing complexities of such models, the advantage of relying on more resources and high-level engineers.

Conclusion

In this paper I have mainly focused on the risks posed in the long term by the development of an AGI, but undesirable and dangerous scenarios of great impact are also to be found in the short term produced by the rise of current LLMs.

OpenAI points out that LLMs will have the potential to automate many jobs, such as those requiring high-level of education (lawyers, translators, and programmers to name a few) [21].

In fact, many people could lose their jobs, further increasing the already widening gap between rich and poor, social inequality, and a possible overall decrease in labor demand, with a powerful impact on society and the economy.

These consequences will also result from accelerated innovation, which doesn’t give time for states and international organizations to regulate these technologies and for societies and individuals to “adapt” to the impacts they produce.

The quick sequential releases to the public of models such as ChatGPT and GPT-4, have been justified as “to get users’ feedback and learn about its strengths and weaknesses, to have time to adapt, to pass regulation” [20][7].

However, it seems clear that companies are accelerating the development of their models in order to keep up with competitors, including open source, in a competitive, goldmine-seeking rationale.

In this, their statement to maintain ethical standards is not reassuring, because the increased competitive pressure triggered by acceleration may lead to exacerbation of the problem of releasing not fully aligned models, which again is critical for ethical and safety risks to humanity in the run-up to the emergence of an AGI.

In this area, big tech companies are usually conservative in predicting future progress, but in launching the PaLM 2 model that powers Bard, Google was clear in stating that it aims “to continue to accelerate our progress” [8], without considering the related risks.

Recent progress along with acceleration must be taken into account when estimating the time still needed to reach AGI. If humanity is not to be put at risk, alignment must be taken seriously: OpenAI seems one of the few big tech companies taking the alignment problem seriously, but also acknowledging that the development of AGI and other future models could be reached without solving it [13].

In this scenario, given the significant stakes involved, it would be desirable for public authorities to increase regulatory intervention [16]. Most importantly, there is a need to adopt universal principles for global guidance, following the path of UNESCO Recommendations [28] or the OECD Principles [18], as the European Union seems inclined to do [6].

References

1. J. Angwin et al., “Machine Bias”, ProPublica, 2016.
2. E. M. Bender et al., “On the dangers of stochastic parrots: Can language models be too big? 🦜”, in Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, New York, NY, USA: ACM, 2021.
3. S. Bills et al., “Language models can explain neurons in language models”, 2023.
4. S. Bubeck et al., “Sparks of Artificial General Intelligence: Early experiments with GPT-4”, 2023.
5. T. Dettmers et al., “SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression”, 2023.
6. European Commission, “A European approach to artificial intelligence”, europa.eu, 2022.
7. L. Fridman and S. Altman, “Sam Altman: OpenAI CEO on GPT-4, ChatGPT, and the future of AI | Lex Fridman podcast #367”, Lex Fridman Podcast, 2023.
8. Z. Ghahramani, “Introducing PaLM 2”, Google The Keyword, blog.google, 2023.
9. T. Hamilton, “Highlights from the RAeS future combat air & space capabilities Summit”, 2023.
10. H. Hodson, “DeepMind and Google: the battle to control artificial intelligence”, The Economist, 2019.
11. K. I. Islam, M. S. Islam, and M. R. Amin, “Sentiment analysis in Bengali via transfer learning using multi-lingual BERT”, in 2020 23rd International Conference on Computer and Information Technology (ICCIT), pp. 1-6. IEEE, 2020.
12. J. Leike, J. Schulman, and J. Wu, “Our approach to alignment research”, Openai.com, 2022.
13. J. Leike and I. Sutskever, “Introducing superalignment”, Openai.com, 2023.
14. J. Lewis, S. Schneegans, and T. Straza, “UNESCO Science Report: The race against time for smarter development”, UNESCO Publishing, 2021.
15. H. Lightman et al., “Let’s verify step by step”, 2023.
16. T. Mahler, “Regulating Artificial General Intelligence (AGI)”, in Law and Artificial Intelligence, The Hague: T.M.C. Asser Press, pp. 521–540, 2022.
17. S. Mukherjee et al., “Orca: Progressive learning from complex explanation traces of GPT-4”, 2023.
18. OECD, “OECD AI Principles”, Oecd.ai, 2019.
19. OpenAI, “OpenAI Charter”, Openai.com, 2018.
20. OpenAI, “Introducing ChatGPT”, Openai.com, 2022.
21. OpenAI, “GPT-4 Technical Report”, 2023.
22. D. Patel and A. Ahmad, Google leaked document “we have no moat, and neither does OpenAI’”, SemiAnalysis.com, 2023.
23. J. Roberts et al., “GPT4GEO: How a language model sees the world’s geography”, 2023.
24. R. Schaeffer, B. Miranda, and S. Koyejo, “Are emergent abilities of large language models a mirage?”, 2023.
25. T. Shevlane et al., “Model evaluation for extreme risks”, 2023.
26. E. H. Soice et al., “Can large language models democratize access to dual-use biotechnology?”, 2023.
27. M. Turpin et al., “Language Models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting”, 2023.
28. UNESCO, “Recommendation on the Ethics of Artificial Intelligence”, 2021.
29. A. Vaswani et al., “Attention is All You Need”, Neural Information Processing Systems Foundation, Inc. (NeurIPS), 2017.
30. J. Wei et al., “Emergent abilities of large language models”, 2022.