Taylor Webb knew artificial intelligence had historically struggled with solving certain questions on IQ tests. Prior research had suggested that AI found abstract reasoning tasks challenging, making problems such as identifying the missing piece in a matrix nearly impossible. So when he proposed these abstract tasks to GPT3 in 2020, he expected it to struggle considerably.
Instead, the psychology postdoctoral researcher was taken aback to find GPT3 passed these tests with flying colors.
This moment inspired a research study where Webb began feeding the model more difficult problems. GPT3 easily conquered many of those tasks as well. Webb began wondering about the future of AI in research, regarding it with a mixture of curiosity, surprise and skepticism that I have seen many times before as a student researcher.
The release of generative AI tools from ChatGPT to DALL-E sent shockwaves across the scientific community with their ability to generate text, images and videos. But while some scientists are embracing generative AI, others remain skeptical of its abilities.
I felt torn about generative AI’s capacity to truly affect the scientific process due to my differing research experiences in wet and dry lab research this summer. Terms like transformers and ChatGPT were frequently used in conversation in one lab focused on computational research, while they were hardly ever mentioned in the other lab. The uncertainty that surrounds AI led me to turn to experts such as Webb to shed some light on what the future may hold for science overall.
With increased attention from the media after ChatGPT launched last year, generative AI seemed like a new development in the technology scene. However, assistant professor of computer science Violet Peng said the idea of training a model to understand and generate human language has been around for decades. The field of natural language processing emerged from this trend in research, with ChatGPT as the latest effort to develop a generative NLP model, she explained.
Aditya Grover, assistant professor of computer science, said recent advancements in generative AI are largely due to deep learning developments during the past decade. This methodology trains a model by feeding data into a network of nodes that mimic the interconnected network of neurons in a person’s brain. By incorporating new deep learning techniques into generative AI models, scientists could drastically scale up the models and the data used to train them, he said.
Peng added that the recent advent of transformers as a new deep learning model architecture shaped the field of generative AI, allowing scientists to quickly and dramatically scale up models. Grover said generative AI’s progress has surprised even those in the field.
"I remember the early days when I was working on generative models for image generation," he said. "There used to be folks who would be skeptics about generative AI ever generating something as beautiful as art pieces."
Now a person has the world of AI-generated art at their fingertips. Trained on large collections of digitized artwork and associated text, tools such as DALL-E and Midjourney can generate an image exactly as specified, leading to amazement and uncertainty among users and professionals alike.
But, as Grover pointed out, generative AI has irreversibly altered how artists view their discipline. While generative AI has added new tools to the creation process, it also raises tough questions about creativity and plagiarism that are relevant to the research sector. I became intrigued. How has generative AI altered the research process for scientists at UCLA and the greater scientific community?
UCLA researchers have found immediate benefits to incorporating AI into their work on a day-to-day basis.
Generative AI has proven useful in removing barriers to performing research. Grover shared that not all members of his team are native English speakers, making generative AI beneficial for organizing and proofreading drafts of research papers.
Additionally, Peng and Grover noted that generative AI has enhanced the process of data generation and annotation for training models, making it efficient and cost-effective. For example, Grover said AI can be trained to label sentences with the emotion associated with the event described by the sentence. Peng added that the data generated by generative AI can be of higher quality than conventional generation methods.
Similarly, student researchers have taken advantage of new generative AI tools in their approach to research. Seongbin Park, fourth-year computer science student and an undergraduate researcher in the MINT group led by Grover, mentioned how using these tools to write code greatly expedited the development process. Webb agreed that these tools could prove effective in writing code, though he cautioned that it could lead to difficult-to-find bugs.
Generative AI also inspired new projects for Grover and his research group as they set out to probe these tools for their blind spots and create novel algorithms to address these issues. They found that ChatGPT did not effectively personalize its answers to the user it was communicating with. As a result, his team developed a tool that could be combined with any large language model to generate custom responses based on the user.
For Peng, generative AI has solved lower level problems allowing her group to focus on higher level challenging problems in their research. Basic tasks, such as teaching creative language generation models the rules of grammar, can be handled by other AI models, allowing Peng to focus on advanced tasks such as defining the elements required for creative writing.
"So essentially you can view it as this model took some of our job in a way," Peng said with a laugh.
As I marveled at the range of applications for generative AI, I also questioned if it was capable of the same creativity human researchers use to make scientific breakthroughs.
Webb provided some insight into this topic. His study found that GPT3 performed at a comparable or superior level to undergraduate students on many tasks such as determining the connection between two strings of text. But a task found in developmental studies led to an unexpected reaction from the model. In the assignment, a child is asked to use tools to move gumballs in a bowl to another bowl farther away, after hearing stories about an analogous situation. GPT3 was trained on similar stories and was asked to provide a solution to the gumball problem. Its response?
"GPT3 would propose very complex, elaborate multi-step plans for how to connect all these tools together, but which were completely nonsensical and in no sense solved the task," Webb said.
Webb concluded that GPT3’s reasoning capabilities were uneven; while it could outshine college students at certain abstract reasoning tasks, it could perform worse than a 5-year-old at complex tasks that involved physical reasoning.
Though the researchers I spoke to see benefits from incorporating AI into their day-to-day research, they remain convinced that it is not capable of the creativity required to generate novel research ideas and hypotheses.
Peng described scientific creativity as being able to build upon previous knowledge and discover ideas that are wholly new. In her experience, generative AI currently struggles with the latter, limiting its potential for scientific discovery.
But both Grover and Webb believed these models could become capable of the kind of scientific creativity required for innovation. As researchers continue to feed models larger datasets, Grover believes that models could become increasingly creative.
"I don't think there's anything so special about human creativity that it can't be automated in principle," Webb said. "I believe that we will be increasingly convinced that these systems are doing genuinely creative things."
Webb said that true creativity stems from the ability to reflect on a set of ideas for an extended period of time, which can lead to unexpected connections and guide researchers to solutions. Building long-term memory into generative AI could remedy this shortcoming, he said. He also suggested improving AI’s creativity by building a mechanism for episodic memory to enable it to connect related experiences and thoughts together.
On the other hand, Grover believes that the ability to continuously problem solve toward a greater goal is integral to being a scientist. He said the first solution to a problem, which is not difficult to generate, is liable to failure. A scientist should be able to turn this failure into a launchpad for the scientific process. His words resonated with me as I remembered recently spending months improving the efficiency of an algorithm to analyze COVID-19 samples without compromising its accuracy.
"It's always this process of being able to see the bigger vision and then iterate over your ideas to improve them to achieve that vision," Grover said.
Grover added that bringing together people with complementary skills and knowledge to accomplish ambitious research projects is key for success. Building a team that can make progress together is a nuanced process, he said.
Generative AI scientists are trying to capture that interdisciplinary knowledge with the announcement of the Polymathic AI project by an international team of scientists on Oct. 9. The project is bringing together researchers from fields such as cosmology and particle physics to create a generative model for interdisciplinary research. It aims to train the model on data from a range of scientific disciplines, a task that has proven challenging for AI historically.
As scientists have leveraged AI in their research in the last year, it has fallen prey to inaccuracy with data presented and references cited incorrectly in papers using AI.
Grover said solving this issue proves particularly difficult as the behavior of these models is complex and not inherently intuitive. He added that it can be difficult to evaluate whether models are truly making progress.
"What we teach in machine learning is that there's no intelligence if all you're going to do is memorize what you saw during training," he said.
Yet Grover said generative AI forces scientists to reevaluate many benchmarks traditionally used in machine learning. As today’s AI systems can be trained on data from throughout the internet, there is a high likelihood that it may have seen a dataset used to test it during its training.
Despite these concerns, Peng, Webb and Grover believe there is no reason to limit use of tools if they can be successfully leveraged in research. Peng added that she believes that if these tools allow us to make breakthroughs to the betterment of the world, we should embrace it.
If generative AI could become capable of these feats and could actively contribute to research, I wondered if this would change the role of humans in research. AI could one day recreate a human’s ability to hypothesize and the skills necessary to validate these hypotheses which I had spent years to learn. I began to consider how the two could co-exist in the world of science and if there were skills that were uniquely human. ChatGPT could be working alongside researchers, steering them clear of dead ends in the road to success. My feelings were echoed by a remark Grover made as he reflected on the same quandary.
"Should I be replaced by ChatGPT?" he asked.
Ultimately, his response was made in jest. He remained optimistic that humans would always have a place in research, explaining that he believed the agency inherent to a human being was powerful. Grover said he envisioned the future of research as one where generative AI and humans would act as collaborators with AI acting as a research assistant to fill in the gaps.
"Can we generate new materials which have specific properties? Can we generate new drugs which are less toxic and target specific viruses or other kinds of pathogens?" he said. "Those are areas where generative AI will build new science that could be a game changer for the state of humanity. And I am a big believer that it will be possible – it will happen."