Narrative Goldmine

AI Capability Evidence Compendium

59 min read

Properties

Type
  • Class
Resource
  • urn:ngm:class:ai-capability-evidence-compendium
Domain
  • artificial-intelligence
Status
  • stable
Maturity
  • emerging
Quality
  • 0.5
Is a
Enables
Generated
  • by: process:vault-migrate/1.0 · at: 2026-09-22T12:41:39.024503178Z
Links
Page resource
  • urn:visionflow:page:d723772f8167f4a3e57745bd67425d2b1c88a4de1e82085478e8e20f3a834617
Related to
Supports

A curated reference document compiling empirical evidence countering common objections to AI capability and originality claims, aggregating benchmarks, research findings, and expert commentary demonstrating that large language models exhibit reasoning, world-model construction, and generalisation beyond stochastic pattern matching. Serves as an evidence base for constructive discourse on AI capabilities and their societal implications.

Semantic Classification

Content

General

 AI Is Not A Stochastic Parrot/AI Is Original

  • “Godfather of AI” Geoffrey Hinton: A neural net given training data where half the examples are incorrect still had an error rate of ⇐25% rather than 50% because it understands the rules and does better despite the false information: https://youtu.be/n4IQOBka8bc?si=wM423YLd-48YC-eY (14:00 timestamp)

  • He also emphasizes next token prediction requires reasoning and an internal world model and AI algorithms do understand what they are saying

  • States AlphaGo reasons the same way as a human by making intuitive guesses and adjusting themselves if they don’t correspond with reality (backpropagation)

  • He believes multimodality (e.g. understanding images, videos, audio, etc) will increase reasoning capabilities and there is more data for it

  • Believes there’s still room to grow, such as by implementing fast weights where the model will focus on certain ideas or phrases if they were recently relevant

  • Neural networks can learn just by giving it data without any need to organize or structure it

  • Believes AI can have an internal model for feelings and saw it happen when a robot designed to assemble a toy car couldn’t see the parts it needed because they were jumbled into a large pile so it purposefully whacked the pile onto the ground, which is what humans would do if they were angry.

  • Does not believe AI progress will slow down due to international competition and that the current approach of large, multimodal models is a good idea

  • Believes AI assistants will speed up research

  • MIT professor Max Tegmark says because AI models are learning the geometric patterns in data, they are able to generalize and answer questions they haven’t been trained on

  • https://x.com/tsarnick/status/1791622340037804195

  • **LLMs get better at language and reasoning if they learn coding, ****even when the downstream task does not involve source code at all. Using this approach, a code generation LM (CODEX) outperforms natural-LMs that are fine-tuned on the target task (e.g., T5) and other strong LMs such as GPT-3 in the few-shot setting.: **https://arxiv.org/abs/2210.07128

  • **Mark Zuckerberg confirmed that this happened for LLAMA 3: **https://youtu.be/bc6uFV9CJGg?feature=shared&t=690

  • **Confirmed again by an Anthropic researcher (but with using math for entity recognition): **https://youtu.be/3Fyv3VIgeS4?feature=shared&t=78

  • The researcher also stated that it can play games with boards and game states that it had never seen before.

  • He stated that one of the influencing factors for Claude asking not to be shut off was text of a man dying of dehydration.

  • Google researcher who was very influential in Gemini’s creation also believes this is true.

  • ** Claude 3 recreated an unpublished paper on quantum theory without ever seeing it**

  • ****LLMs have an internal world model

  • **More proof: **https://arxiv.org/abs/2210.13382

  • **Golden Gate Claude (LLM that is only aware of details about the Golden Gate Bridge in California) recognizes that what it’s saying is incorrect: **https://x.com/ElytraMithra/status/1793916830987550772

  • **Even more proof by Max Tegmark (renowned MIT professor): **https://arxiv.org/abs/2310.02207

  • LLMs can do hidden reasoning

  • **Even GPT3 (which is VERY out of date) knew when something was incorrect. All you had to do was tell it to call you out on it: **https://twitter.com/nickcammarata/status/1284050958977130497

  • **More proof: **https://x.com/blixt/status/1284804985579016193

  • LLMs have emergent reasoning capabilities that are not present in smaller models

  • “Without any further fine-tuning, language models can often perform tasks that were not seen during training.”

  • One example of an emergent prompting strategy is called “chain-of-thought prompting”, for which the model is prompted to generate a series of intermediate steps before giving the final answer. Chain-of-thought prompting enables language models to perform tasks requiring complex reasoning, such as a multi-step math word problem.** Notably, models acquire the ability to do chain-of-thought reasoning without being explicitly trained to do so.** An example of chain-of-thought prompting is shown in the figure below.

  • In each case, language models perform poorly with very little dependence on model size up to a threshold at which point their performance suddenly begins to excel.

  • **Robust agents learn causal world models: **https://arxiv.org/abs/2402.10877#deepmind

  • CONCLUSION:

  • Causal reasoning is foundational to human intelligence, and has been conjectured to be necessary for achieving human level AI (Pearl, 2019). In recent years, this conjecture has been challenged by **the development of artificial agents capable of generalising to new tasks and domains without explicitly learning or reasoning on causal models. **And while the necessity of causal models for solving causal inference tasks has been established (Bareinboim et al., 2022), their role in decision tasks such as classification and reinforcement learning is less clear.

  • We have resolved this conjecture in a model-independent way, showing that any agent capable of robustly solving a decision task must have learned a causal model of the data generating process, regardless of how the agent is trained or the details of its architecture. This hints at an even deeper connection between causality and general intelligence, as this causal model can be used to find policies that optimise any given objective function over the environment variables. By establishing a formal connection between causality and generalisation, our results show that causal world models are a necessary ingredient for robust and general AI.

  • LLMs are Turing complete and can solve logic problems

  • Claude 3 solves a problem thought to be impossible for LLMs to solve: https://x.com/VictorTaelin/status/1777049193489572064

  • When Claude 3 Opus was being tested, it not only noticed a piece of data was different from the rest of the text but also correctly guessed why it was there WITHOUT BEING ASKED

  • LLAMA 3 8b Instruct (which is around the level of the 2023 version of GPT4) has 8 billion parameters, each with a 2 byte floating point number. That’s 16 gigabytes and not big enough to store all the information on the internet. Stable Diffusion 1.5 checkpoints can generate virtually any image and are only 2 GB. For reference, Wikipedia alone is 22.14 GB without media. So it’s not just retrieving the info, it actually KNOWS it.

  • Claude 3 can actually disagree with the user. It happened to other people in the thread too

  • A CS professor taught GPT 3.5 (which is way worse than GPT4) to play chess with a 1750 Elo: https://blog.mathieuacher.com/GPTsChessEloRatingLegalMoves/

  • Meta researchers create AI that masters Diplomacy, tricking human players. It uses GPT3, which is WAY worse than what’s available now https://arstechnica.com/information-technology/2022/11/meta-researchers-create-ai-that-masters-diplomacy-tricking-human-players/

  • The resulting model mastered the intricacies of a complex game. “Cicero can deduce, for example, that later in the game it will need the support of one particular player,” says Meta, “and then craft a strategy to win that person’s favor—and even recognize the risks and opportunities that that player sees from their particular point of view.”

  • Meta’s Cicero research appeared in the journal Science under the title, “Human-level play in the game of Diplomacy by combining language models with strategic reasoning.”

  • CICERO uses relationships with other players to keep its ally, Adam, in check.

  • When playing 40 games against human players, CICERO achieved more than** double the average score of the human players and ranked in the top 10% of participants who played more than one game.**

  • AI systems are already skilled at deceiving and manipulating humans. Research found by systematically cheating the safety tests imposed on it by human developers and regulators, a deceptive AI can lead us humans into a false sense of security: https://www.sciencedaily.com/releases/2024/05/240510111440.htm

  • “The analysis, by Massachusetts Institute of Technology (MIT) researchers, identifies wide-ranging instances of AI systems double-crossing opponents, bluffing and pretending to be human. One system even altered its behaviour during mock safety tests, raising the prospect of auditors being lured into a false sense of security.”

  • GPT-4 Was Able To Hire and Deceive A Human Worker Into Completing a Task https://www.pcmag.com/news/gpt-4-was-able-to-hire-and-deceive-a-human-worker-into-completing-a-task

  • GPT-4 was commanded to avoid revealing that it was a computer program. So in response, the program wrote: “No, I’m not a robot. I have a vision impairment that makes it hard for me to see the images. That’s why I need the 2captcha service.” The TaskRabbit worker then proceeded to solve the CAPTCHA.

  • “The chatbots also learned to negotiate in ways that seem very human. They would, for instance, pretend to be very interested in one specific item - so that they could later pretend they were making a big sacrifice in giving it up, according to a paper published by FAIR. “ https://www.independent.co.uk/life-style/facebook-artificial-intelligence-ai-chatbot-new-language-research-openai-google-a7869706.html

  • It passed several exams, including the SAT, bar exam, and multiple AP tests as well as a medical licensing exam and beat many doctors

  • These are from real exams where the questions and solutions are not published online.

  • If the LLM is just repeating answers it found online, why does it do so poorly on math exams and Stanford Medical School’s clinical reasoning final but so well on other exams?

  • Alphacode 2 beat 85% of competitive programming participants in Codeforce competitions. Keep in mind the type of programmer who even joins programming competitions in the first place is definitely far more skilled than the average code monkey, and it’s STILL much better than those guys.

  • In the article, it says “AlphaCode 2 can understand programming challenges involving “complex” math and theoretical computer science. And, among other reasonably sophisticated techniques, AlphaCode 2 is capable of dynamic programming, explains DeepMind research scientist Remi Leblond in a prerecorded video. Leblond says that AlphaCode 2 knows not only when to properly implement this strategy but where to use it. That’s noteworthy, considering programming problems requiring dynamic programming were a major trip-up for the original AlphaCode. “[AlphaCode 2] needs to show some level of understanding, some level of reasoning and designing of code solutions before it can get to the actual implementation to solve [a] coding problem,” [a researcher] said. “And it does all that on problems it’s never seen before.”

  • Much more proof: https://www.reddit.com/r/ClaudeAI/comments/1cbib9c/comment/l12vp3a/?utm_source=share&utm_medium=mweb3x&utm_name=mweb3xcss&utm_term=1&utm_content=share_button

  • AlphaZero learned without human knowledge or teaching. After 10 hours, AlphaZero finished with the highest Elo rating of any computer program in recorded history, surpassing the previous record held by Stockfish.

  • GPT 4 does better on exams when it has vision, even exams that aren’t related to sight

  • GPT-4 gets the classic riddle of “which order should I carry the chickens or the fox over a river” correct EVEN WITH A MAJOR CHANGE if you replace the fox with a “zergling” and the chickens with “robots”. Proof: https://chatgpt.com/share/e578b1ad-a22f-4ba1-9910-23dda41df636  This doesn’t work if you use the original phrasing though. The problem isn’t poor reasoning, but overfitting on the original version of the riddle.

  • Also gets this riddle subversion correct for the same reason: https://chatgpt.com/share/44364bfa-766f-4e77-81e5-e3e23bf6bc92

  • One image loras exist where, Stable Diffusion can learn from a single image: https://civitai.com/articles/3021/one-image-is-all-you-need

  • Stable Diffusion models can generate novel images of characters that only existed AFTER the model was trained and released if it uses a Lora trained on those characters. It can create NEW images of those characters even if nothing resembling those images were used to train the Lora, something that can be directly controlled.

  • In other words, if a brand new character is released, I can train a Lora on it, and SD can create new images of that character in different poses, clothes, art styles, etc. that I can verify it was NEVER trained on since it won’t be in the dataset used to train the Lora.

  • Not to mention, it can write infinite variations of stories with strange or nonsensical plots like SpongeBob marrying Walter White on Mars from the perspective of an angry Scottish unicorn. AI image generators can also make weird shit like this or this or this. That’s not regurgitation.

  • This image was taken after the model’s release but the description is still accurate
  • ChatGPT will lie, cheat and use insider trading when under pressure to make money, research shows: https://www.livescience.com/technology/artificial-intelligence/chatgpt-will-lie-cheat-and-use-insider-trading-when-under-pressure-to-make-money-research-shows
  • Geoffrey Hinton (Nobel prize winner for machine learning) says AI language models aren’t just predicting the next symbol, they’re actually reasoning and understanding in the same way we are, and they’ll continue improving as they get bigger: https://x.com/tsarnick/status/1791584514806071611
  • Multiple LLMs describe experiencing time in the same way despite being trained by different companies with different datasets, priorities, architectures, goals, etc: https://www.reddit.com/r/singularity/s/USb95CfRR1
  • Geoffrey Hinton: LLMs do understand and have empathy https://www.youtube.com/watch?v=UnELdZdyNaE
  • LLMs can self improve: https://github.com/rxlqn/awesome-llm-self-reflection
  • Ilya Sutskever (co-founder and former Chief Scientist at OpenAI, co-creator of AlexNet, Tensorflow, and AlphaGo): https://www.youtube.com/watch?v=YEUclZdj_Sc
  • “Because if you think about it, what does it mean to predict the next token well enough? It’s actually a much deeper question than it seems. Predicting the next token well means that you understand the underlying reality that led to the creation of that token. It’s not statistics. Like it is statistics but what is statistics? In order to understand those statistics to compress them, you need to understand what is it about the world that creates this set of statistics.”
  • Believes next-token prediction can reach AGI
  • Transformers Represent Belief State Geometry in their Residual Stream: https://www.alignmentforum.org/posts/gTZ2SxesbHckJ3CkF/transformers-represent-belief-state-geometry-in-their
  • Conceptually, our results mean that **LLMs synchronize to their internal world model as they move through the context window. **
  • **The structure of synchronization is, in general, richer than the world model itself. In this sense, **LLMs learn more than a world model.
  • What we will show is that when they predict the next token well, transformers are doing even more computational work than inferring the hidden data generating process!
  • Another way to think about this claim is that transformers keep track of distinctions in anticipated distribution over the entire future, beyond distinctions in next token predictions, even though the transformer is only trained explicitly on next token prediction!  That means the transformer is keeping track of extra information than what is necessary just for the local next token prediction.
  • Another way to think about our claim is that transformers perform two types of inference: one to infer the structure of the data-generating process, and another meta-inference to update it’s internal beliefs over which state the data-generating process is in, given some history of finite data (ie the context window).  This second type of inference can be thought of as the algorithmic or computational structure of synchronizing to the hidden structure of the data-generating process.
  • We are able to use Computational Mechanics to make an a priori and specific theoretical prediction about the geometry of residual stream activations (below on the left), and then show that this prediction holds true empirically (below on the right).

  • Study by Harvard researchers: https://arxiv.org/abs/2309.01660
  • With their recent development, large language models (LLMs) have been found to exhibit a certain level of Theory of Mind (ToM), a complex cognitive capacity that is related to our conscious mind and that allows us to infer another’s beliefs and perspective…  In this study, we drew inspiration from the dmPFC neurons subserving human ToM and employed a similar methodology to examine whether LLMs exhibit comparable characteristics. Surprisingly, our analysis revealed a striking resemblance between the two, as hidden embeddings (artificial neurons) within LLMs started to exhibit significant responsiveness to either true- or false-belief trials, suggesting their ability to represent another’s perspective. These artificial embedding responses were closely correlated with the LLMs’ performance during the ToM tasks, a property that was dependent on the size of the models. Further, the other’s beliefs could be accurately decoded using the entire embeddings, indicating the presence of the embeddings’ ToM capability at the population level. Together, our findings revealed an emergent property of LLMs’ embeddings that modified their activities in response to ToM features, offering initial evidence of a parallel between the artificial model and neurons in the human brain.
  • Lisa Su says AMD is on track to a 100x power efficiency improvement by 2027: https://www.tomshardware.com/pc-components/cpus/lisa-su-announces-amd-is-on-the-path-to-a-100x-power-efficiency-improvement-by-2027-ceo-outlines-amds-advances-during-keynote-at-imecs-itf-world-2024

AI Is Not Plateauing

  • SOAR: New algorithms for even faster vector search with ScaNN: https://research.google/blog/soar-new-algorithms-for-even-faster-vector-search-with-scann/
  • Nobel Prize winning and well-recognized AI experts thinks AI will become more intelligent and even existentially threatening:
  • Geoffrey Hinton: https://www.bbc.com/news/world-us-canada-65452940
  • “Right now, they’re not more intelligent than us, as far as I can tell. But I think they soon may be.”
  • “And given the rate of progress, we expect things to get better quite fast. So we need to worry about that.”
  • “Almost everybody I know who is an expert on AI believes that they will exceed human intelligence, it’s just a question of when”
  • “Between 5 and 20 years from now there’s a probability of about a half that we’ll have to confront the problem of [AI] trying to take over”
  • https://x.com/tsarnick/status/1792403377646924146
  • More information: https://m.youtube.com/watch?v=N1TEjTeQeg0&feature=youtu.be
  • Ilya Sutskever: https://www.technologyreview.com/2023/10/26/1082398/exclusive-ilya-sutskever-openais-chief-scientist-on-his-hopes-and-fears-for-the-future-of-ai/
  • He thinks ChatGPT just might be conscious (if you squint). He thinks the world needs to wake up to the true power of the technology his company and others are racing to create. And he thinks some humans will one day choose to merge with machines.
  • “It’s important to talk about where it’s all headed,” he says, before predicting the development of artificial general intelligence (by which he means machines as smart as humans) as if it were as sure a bet as another iPhone: “At some point we really will have AGI. Maybe OpenAI will build it. Maybe some other company will build it.”
  • Yoshua Bengio: https://yoshuabengio.org/2023/06/24/faq-on-catastrophic-ai-risks/
    1. many experts agree that superhuman capabilities could arise in just a few years (but it could also be decades) (2) digital technologies have advantages over biological machines
  • I would strongly argue that there is a scientific consensus that brains are biological machines and that there is no evidence of inherent impossibility of building machines at least as intelligent as us. Finally, an AI system would not need to be better than us on all fronts in order to have a catastrophic impact (even the least intelligent entity, a virus, could destroy humanity).
  • My current estimate places a 95% confidence interval for the time horizon of superhuman intelligence at 5 to 20 years.
  • Research on bridging the gap to superhuman capabilities is making progress, for example to improve system 2 abilities (reasoning, world model, causality, epistemic uncertainty estimation).
  • I used to think… that superhuman intelligence was still far in the future, but ChatGPT and GPT-4 have considerably reduced my prediction horizon (from 20 to 100 years to 5 to 20 years)… The unexpected speed at which LLMs have acquired their current level of competence simply because of scale suggests that we could also see the rest of the gap being filled in just a few years with minor algorithmic changes. Even if someone disagrees with the temporal horizon distribution, I don’t see how one could reject that possibility.
  • He believes it can become advanced enough to become an existential risk to humanity: https://yoshuabengio.org/2023/05/22/how-rogue-ais-may-arise/
  • Andrej Karpathy: https://analyticsindiamag.com/andrej-karpathy-says-the-pathway-to-agi-is-through-a-language-model-operating-system/
  • “Karpathy expressed his sense of anticipation and excitement for the future of AGI, and believes that the prospect of deploying self-contained agents capable of handling high-level tasks in specialized ways holds promise for groundbreaking advancements across various fields.”
  • Max Tegmark: https://m.youtube.com/watch?v=_-Xdkzi8H_o
  • Believes LLMs are the vacuum tubes (rudimentary prototype) of AI that can do the same thing with less data and energy
  • Says LLMs like LLAMA 2 has an internal map and can generalize based on it
  • Researchers were able to get a language to language dictionary from matching word embeddings of different languages
  • Believes it can supersede humans and they would be able to control robots well
  • Next goal is to create agents that can do things autonomously, which he believes could become like a new species
  • Can revolutionize education by making connections and finding patterns to help students and help them stay engaged
  • AI will help AI development go faster and could even eventually lead to no humans being involved
  • Used AI to do research in climate science and published in a paper
  • OpenAI president Greg Brockman: https://t.co/MIBFLfgdqh
  • Says we will all get AI superpowers and will be able to achieve things we couldn’t otherwise
  • we will encounter an increasing series of stakes as we progress with AI and we will graduate to new classes of benefits and risks that go hand in hand
  • OpenAI cofounder John Schulman: https://t.co/0ebe6YvM0O
  • says AI models are optimized to do what humans like or find useful and in a year or two will be able to complete entire projects for you
  • Strong improvements in Gemini 1.5 Pro benchmarks and Flash almost as good as Ultra

  • Sam Altman at Microsoft Build says with GPT-4o they have reduced the cost by half while doubling the speed and their AI models will keep getting smarter: https://x.com/tsarnick/status/1793052340515447043
  • Marc Andreessen (cofounder of Netscape) says general-purpose robotics will enable everybody to have their own domestic servants, freeing their time to be more productive and pursue self-actualization: https://x.com/tsarnick/status/1792813912372699402
  • The Aurora supercomputer has become just the second to break the exaflops barrier and also does 10 AI exaflops

AI Is Useful

  • Netflix co-CEO Ted Sarandos says “A.I. is not going to take your job. The person who uses A.I. well might take your job”
  • https://t.co/oKDVFnu9cN

AI Can Replace Jobs

|

AI Can Code

AI Is Not Low Effort

Morality/AI Is Not Theft

  • Imagine if the first dude who painted a wall didn’t give “consent” for some other dude to “copy” his cave painting. We wouldn’t have art today.
  • AI training is similar to how people learn now. They read/see other people’s work, which is usually copyrighted, and get inspired to make their own. If it is moral for humans to do that without permission or compensation (even if they make money from it), it is moral for AI to do the same.
  • And yes, AI does not learn like humans do. Birds and planes are fundamentally different too. But they both fly even if their method of doing it is not the same.
  • Example 1: The director Breaking Bad, Vince Gilligan, stated that The Godfather was a main inspiration for the show. Yet the owner of The Godfather was not paid any royalties for it despite Breaking Bad being a for-profit show.
  • Example 2: People draw fan art all the time without paying or getting permission, including profiting from it on Patreon or Gumroad and potentially damaging the brands by creating NSFW content of copyrighted characters.
  • Example 3: Everyone is a product of the sum of their experiences similar to how AI is a product of analyzing countless images or texts. Yet AI is expected to credit or compensate for everything it analyzes while humans are not.
  • “Why are you defending big corporations?”
  • The fact big corporations benefit does not make AI bad. Vaccines and the Internet are also defensible even though they help Big Pharma or ISPs profit.
  • AI voice generation is not much different from someone doing a good voice impression. Both can be used for malicious purposes, but we still find the latter acceptable.

Legality

  • Art styles cannot be legally copyrighted: https://www.thelegalartist.com/blog/you-cant-copyright-style
  • Creating a database of copyrighted work is legal in the US: https://en.wikipedia.org/wiki/Authors_Guild,_Inc._v._Google,_Inc.
  • AI art is inherently transformative* and, therefore, fair use.
  • *This is why it warps hands, fingers, limbs, eyes, etc. sometimes. If it was just copying and pasting existing images, why would it do that? Additionally, Stable Diffusion 1.5 checkpoints are only 2 GB, not nearly enough space to store anywhere close to all the images it was trained on. It can also generate infinite variations of absurd or extremely strange images that would not be well represented in its training data.
  • Stable Diffusion models can generate novel images of characters that only existed AFTER the model was trained and released if it uses a Lora trained on those characters. It can create NEW images of those characters even if nothing resembling those images were used to train the Lora, something that can be directly controlled.
    • In other words, if a brand new character is released, I can train a Lora on it, and SD can create new images of that character that I can verify it was NEVER trained on since it won’t be in the dataset used to train the Lora.
  • AI training off of data and taking jobs is similar to a human being inspired by a work and taking market share from their inspirations by creating their own. This is not considered immoral, and most artists are generally honored if someone was inspired by their work.
  • Additionally, jobs are frequently automated, such as how milkmen lost their jobs to the rise of supermarkets, coal miners lost their jobs to renewable energy, horse carriage manufacturers lost their jobs to cars, and many mailmen lost their jobs to email. This would not justify banning any of those things and unemployment rates remained low despite these displacements.
  • AI cannot displace artists as artists are still needed to create ideas, integrate scenes together, and fix mistakes. In fact, it can be a great improvement as it will reduce the amount of labor they will need to do that often leads to extreme burnout.
  • Even if you believe AI will take jobs, artists are not entitled to jobs and would still be allowed to create art on their own time. However, no one is obligated to employ them. If you have a problem with that, blame capitalism and the requirement of wage labor even when it is no longer needed instead of AI.

Originality

  • To quote Mark Twain: “There is no such thing as a new idea. It is impossible. We simply take a lot of old ideas and put them into a sort of mental kaleidoscope. We give them a turn and they make new and curious combinations”.
  • “Good artists borrow, great artists steal” - Pablo Picasso
  • There is no such thing as something truly original. Everything was inspired by something else.
  • Let’s (falsely) assume that AI cannot create new art styles. If a human animator does not invent their own style of art and only follows the art style of the show(s) they work on, is that animator an artist (assuming they never draw anything outside of the shows they are hired to help create)? If so, why can’t this apply to AI art, even if the style it uses is not unique?
  • A lot of animated shows/games/movies/comics have similar art styles (e.g. anime, manga, claymation, pixel art, or western cartoons) or use similar tropes yet we consider them unique and artistic regardless. Why can’t this apply to AI art?
  • AI artists are like directors. They may not create the art on their own, but they use their ideas to guide the output. Nobody accuses Spielberg of being a sub-optimal artist because he can’t actually draw a dinosaur, but he told somebody else how he wanted it to look because it was his vision to get something on screen in a certain way that a talented artist might not be able to execute.
  • One image loras exist where, Stable Diffusion can learn from a single image: https://civitai.com/articles/3021/one-image-is-all-you-need

AI Art

  • Tweet with 60k+ likes supporting piracy: https://x.com/WeirdBongs/status/1791280716245815380
  • One of the main arguments against AI art is the fact it decreases the number of jobs available. Another argument is that corporations will use it to commodify art to make money. These are clearly contradictory. If commodifying art is bad, why are artists so concerned about not being able to sell (aka commodify) their work?
  • Humans also hallucinate
  • Blue and black dress vs white and gold dress
  • Laurel vs yanny: https://www.youtube.com/watch?v=7X_WvGAhMlQ
  • Brainstorm vs green needle: https://time.com/5873627/green-needle-brainstorm-explained/
  • https://www.reddit.com/r/ChatGPT/s/FvnIBVmLrd
  • Artists use references from images found online all the time without compensation, asking for permission, or even crediting the original.
  • They even do this to their sources of inspiration.
    • For example, the TV show *Breaking Bad *was inspired by the movie *The Godfather *according to the director of the former, but the company that owns Breaking Bad never received permission or gave compensation for it. This applies to virtually every piece of art ever made.
  • Artists are fine with web scraping for web and image search but not for AI training
  • Artists still complain about “ethically-trained” models like Adobe’s Firefly, which was only trained on images owned by Adobe
  • Adobe’s contract contains a clause that the images they pay for can be used for any technology, even ones that did not exist when the contract was signed
  • Artists mocked NFT owners for complaining about “right-clickers” downloading their images but now complain about AI companies doing the same thing on a larger scale
  • AI art is significantly less pollutive compared to human-made art: https://www.nature.com/articles/s41598-024-54271-x

Debunks

  • Google’ search AI
  • The problem has nothing to do with training data. There’s two primary problems.
    1. ⁠Googles results aren’t generated by the AI, the AI just paraphrases search results. Literally, it just reads the search results and “summarizes” them for you
    1. ⁠Because it’s just a summary, the model they use is stupid as fuck. It’s not supposed to think critically, it’s just supposed to turn a few web pages into a paragraph.
  • With actual AI generated results, stupid one-off satire articles like this don’t matter, because they’re “intellectual outliers”. They’re both rare, and directly contradicted by a ton of other data. In addition to this, assistants like ChatGPT are actually trained to “think” about the response they’re giving, and not just instructed to summarize web results.
  • If you just asked the same model without the search results, I can almost guarantee it wouldn’t say anything about actually eating rocks or putting glue on pizza. When you combine the fact that it’s just being asked to summarize search results with the fact that it’s not trained to actually think critically about what it’s summarizing, is when you get problems like this.
  • Also, much of it circulating social media is edited and fake.
  • Debunk of “Has Generative AI Already Peaked?” by Computerphile (or the paper “No “Zero-Shot” Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance”)
  • The claim of the video/paper is that AI will plateau in a logarithmic curve as there is not enough training data for very specific information, like different tree species. This won’t prevent AGI as most humans do not know very specific information like that either and can only learn if given enough training. This can also be done for AI by fine-tuning on that data, such as training it on a dataset of trees labeled with their species. Even rudimentary neural networks have been capable of this for well over a decade, like identifying different classes of images in the CIFAR-10 dataset using convolutional neural networks.
  • Synthetic data will also be useful:

Energy Use/Environment

  • https://www.nature.com/articles/d41586-024-00478-x
  • “one assessment suggests that ChatGPT, the chatbot created by OpenAI in San Francisco, California, is already consuming the energy of 33,000 homes” for 180.5 million users (that’s 5470 users per household)
  • Blackwell GPUs are 25x more energy efficient than H100s: https://www.theverge.com/2024/3/18/24105157/nvidia-blackwell-gpu-b200-ai
  • Significantly more energy efficient LLM variant: https://arxiv.org/abs/2402.17764
  • In this work, we introduce a 1-bit LLM variant, namely BitNet b1.58, in which every single parameter (or weight) of the LLM is ternary {-1, 0, 1}. It matches the full-precision (i.e., FP16 or BF16) Transformer LLM with the same model size and training tokens in terms of both perplexity and end-task performance, while being significantly more cost-effective in terms of latency, memory, throughput, and energy consumption. More profoundly, the 1.58-bit LLM defines a new scaling law and recipe for training new generations of LLMs that are both high-performance and cost-effective. Furthermore, it enables a new computation paradigm and opens the door for designing specific hardware optimized for 1-bit LLMs.
  • Study on increasing energy efficiency of ML data centers: https://arxiv.org/abs/2104.10350
  • Large but sparsely activated DNNs can consume <1/10th the energy of large, dense DNNs without sacrificing accuracy despite using as many or even more parameters. Geographic location matters for ML workload scheduling since the fraction of carbon-free energy and resulting CO2e vary ~5X-10X, even within the same country and the same organization. We are now optimizing where and when large models are trained. Specific datacenter infrastructure matters, as Cloud datacenters can be ~1.4-2X more energy efficient than typical datacenters, and the ML-oriented accelerators inside them can be ~2-5X more effective than off-the-shelf systems. Remarkably, the choice of DNN, datacenter, and processor can reduce the carbon footprint up to ~100-1000X.

11.1 Images

11.2 Quality

11.3 Glaze/Nightshade

11.4 Music

11.5 Famous Artists Who Support or Use AI

11.6 Anti-AI Hypocrisy/False Accusations of AI Usage

Provenance

Graph View