Language Modeling

Language Modeling is the fundamental NLP task of learning probability distributions over sequences of words or tokens to predict the likelihood of text sequences and generate plausible continuations. Language models underpin virtually all modern NLP applications through pre-training on massive text corpora, capturing syntactic structure, semantic relationships, and world knowledge that transfer to downstream tasks.

Semantic Classification

Content

• Language Modeling is the fundamental NLP task of learning probability distributions over sequences of words or tokens to predict the likelihood of text sequences and generate plausible continuations. Language models underpin virtually all modern NLP applications through pre-training on massive text corpora, capturing syntactic structure, semantic relationships, and world knowledge that transfer to downstream tasks including text generation, translation, question answering, and code synthesis.

Models

• Llama 3.1, 3.2, and 4: A series of powerful, open-source large language models.

Business Functions

AI Technology & Concepts

• LLMs (Large Language Models)

• Automation

• Chatbots

• Agents

• Avatars

• Tech & Infrastructure

  Business Functions & Roles

• Business

• Marketing

• Sales

• HR (Human Resources)

• Customer Service

• Finance

• Product Management

• Data Analysis

• Small Business

• Entrepreneurs

• Solopreneurs

  Creative & Media

• Creativity

• Design

• Writing

• Image

• Video

• Audio

• Music

• Creators

  Skills, Learning & Productivity

• Coding

• Productivity

• Learning

• Education

• Professional Development

• Personal Growth

LangChain

• A framework for developing applications powered by language models.

Performance

• LLaMA-3 models demonstrate competitive performance on various language modeling tasks, showcasing significant improvements over previous open weights models.
• Integrated free across their social media platforms
• Fast image generation, can be fine tunes for creatives.
• Introducing New AI Experiences Across Our Family of Apps and Devices | Meta (fb.com)
• What’s up with Llama 3? Arena data analysis | LMSYS Org

Large Language models:

Biomedical

• Collaborative Virtual Environments (CVEs) have immense potential in the fields of chemical and medical molecular modeling. By incorporating natural language AI and visual generative machine learning, these environments can revolutionize the way scientists and researchers approach complex chemical and biological problems. Here are some specific use cases:
• Drug design and discovery: CVEs can enable researchers to collaboratively visualize and manipulate 3D molecular structures in real-time, identifying potential drug candidates and understanding protein-ligand interactions. Natural language AI can help users interact with the molecular data, while visual generative ML can predict potential binding sites, energetics, or toxicity profiles based on existing knowledge.
• Protein structure prediction and modeling: Small teams can work together to predict protein structures, visualize folding patterns, and model protein-protein or protein-nucleic acid interactions. Natural language AI can assist in annotating and explaining the structural features, while visual generative ML can generate new structural hypotheses based on sequence alignments, homology modeling, and experimental data.
• Molecular dynamics simulations: CVEs can facilitate collaboration on complex molecular dynamics simulations, allowing researchers to analyze and visualize trajectories, energetics, and conformational changes. Natural language AI can help users navigate through simulation data and identify relevant patterns, while visual generative ML can create new conformations or predict the effects of mutations on protein stability and function.
• Cheminformatics and QSAR modeling: Researchers can leverage CVEs to develop and validate Quantitative Structure-Activity Relationship (QSAR) models, which predict the biological activity of chemical compounds based on their structural properties. Natural language AI can facilitate the exploration and interpretation of chemical descriptors, while visual generative ML can suggest new compounds with desired properties or optimize existing molecular scaffolds.
• Metabolic pathway modeling: Small teams can work together to build and analyze metabolic pathways, integrating experimental data and computational models to understand the underlying mechanisms and predict metabolic fluxes. Natural language AI can assist in annotating and explaining pathway components, while visual generative ML can generate new pathway hypotheses or predict the effects of genetic or environmental perturbations.
• Biomolecular visualization and virtual reality: CVEs can offer immersive, interactive experiences for exploring biomolecular structures and dynamics, enhancing researchers’ understanding of complex biological systems. Natural language AI can provide contextual information or guide users through molecular landscapes, while visual generative ML can create new visualizations or adapt existing ones based on user preferences and insights.
• Collaborative molecular docking and virtual screening: Small teams can use CVEs to perform collaborative molecular docking and virtual screening, which involve predicting the binding of small molecules to target proteins. Natural language AI can help users refine docking parameters and analyze results, while visual generative ML can generate alternative poses or suggest new compounds for screening based on user feedback and existing data. Choose a suitable mixed reality platform: Select a platform that allows the creation of simple, accessible shared mixed reality environments. Consider open-source options like Mozilla Hubs or JanusVR, which offer customizable and collaborative virtual spaces.
• Integrate open-source biomed software: Incorporate open-source biomed software such as PyMOL, Chimera, or VMD for molecular visualization and analysis. These tools can be integrated into the mixed reality environment for real-time interaction, allowing students and instructors to collaboratively visualize and manipulate molecular structures.
• Leverage AI and machine learning: Integrate AI and ML algorithms like those found in DeepChem, RDKit, or Open Babel to aid in the discovery and optimization of novel compounds. These tools can help predict molecular properties, perform virtual screening, and optimize lead compounds for drug development. By incorporating AI and ML, students can learn how to apply these cutting-edge techniques to real-world problems in biomedicine.
• Establish a distributed proof system: Utilize a distributed proof system like the Nostr protocol to federate the small virtual classroom environments. This will allow for seamless collaboration among students and faculty while maintaining security and data integrity.
• Create digital objects for interaction: Use digital objects such as 3D molecular models, virtual lab equipment, and interactive simulations to create an immersive learning experience. These digital objects can be shared and manipulated in real-time, promoting collaborative learning and problem-solving.
• Implement accessible interfaces: Ensure that the virtual classroom environment is accessible to all students, including those with disabilities. Utilize AI-driven tools like StabilityAI to help with language barriers, safeguarding, and governance, enabling a more inclusive learning experience.
• Foster collaboration and communication: Encourage students and faculty to collaborate on projects, share ideas, and ask questions in real-time using voice chat, text chat, or other communication tools integrated into the mixed reality environment.
• Provide training and support: Offer training sessions and support materials to help students and faculty become familiar with the mixed reality environment, the integrated biomed software, and AI/ML tools.
• Monitor progress and adjust as needed: Regularly review student progress, gather feedback, and adjust the virtual classroom environment as needed to ensure an effective and engaging learning experience.

Language Model and Voice Interface

• A specialized multi-modal LLM can be trained on local language, culture, customs, and environmental data such as flora, fauna, biotica, soil pH, and rainfall. This LLM can be accessed through a voice interface by the local community, enabling data entry and knowledge exchange in the local language. The voice interface can help overcome literacy barriers and make the system more accessible to a diverse range of community members.

Language and Tone

• Use EXPERT terminology for the given context
• AVOID: superfluous prose, self-references, expert advice disclaimers, and apologies

Grok from Musk is pretty bad, but..

• People haven’t appreciated the strength of the business model Musk has
• His is the only unified language and vision company in the world at this scale that can handle real world interactions.

Metaverse and Telecollaboration

• 🟢 I could go on all day about this, goods and bads. I literally wrote a book on it.
• 🟢 A lot (for me) hinges on OpenUSD the universal scene language. It’s been SO long since we have had something useful.
• Nvidia have a text to 3D pipeline for Omniverse. Will be interesting to see what the use cases are. This is their new Cesium [geo tile integration](https://cesium.com/blog/2024/01/16/now-available-[[NVIDIA Omniverse Platform]]-aeco-demo-pack/) giving global instant models.

https://twitter.com/BlockadeLabs/status/1719818562917761094

• This is a presentation slide and the next slide is Open Generative AI tools

Key Quotes

• “Mamba enjoys fast inference (5× higher throughput than Transformers) and linear scaling in sequence length and its performance improves on real data up to million-length sequences.”
• “This class of models can be computed very efficiently as either a recurrence or convolution with linear or near-linear scaling in sequence length.”
• “Selective SSMs and by extension the Mamba architecture are fully recurrent models with key properties that make them suitable as the backbone of general foundation models operating on sequences.”
• How does Mamba achieve linear-time modeling?: By introducing a selection mechanism in structured state space models and designing a hardware-aware algorithm that avoids materializing expanded states, thereby enhancing computational efficiency.
• What improvements does Mamba offer over traditional models?: It achieves faster inference, linear scaling with sequence length, and competitive or superior performance across various data modalities including language, audio, and genomics.
• Can Mamba handle long sequences efficiently?: Yes, it is specifically designed to address the computational inefficiencies of traditional models like Transformers in handling long sequences, offering linear scaling and improved performance.
• Mamba outperforms Transformers of the same size in language modelling and matches Transformers twice its size both in pretraining and downstream evaluation, demonstrating its efficiency and effectiveness.
• Authors’ Views: The authors propose Mamba as a significant step forward in sequence modeling, addressing the inefficiency of Transformers while maintaining or improving performance.
• Comparative Analysis: Mamba is positioned as superior to existing models, particularly Transformers, in terms of efficiency and scalability

Long Context Modeling

• LongMamba (LongMamba): Generalizes to 40k tokens after training on 16k, nearly perfect on “needle in a haystack” task
• Evo: Models long DNA sequences, learning a “cell model” analogous to language models’ “world model”
• Potential challenges:
  • Eventual “rotting” of internal states with extreme context lengths
  • Need for state regularization or “pruning” to maintain performance
• Implications for biology: Foundation models could revolutionize drug discovery and biological research

Models

• Llama 3.1, 3.2, and 4: A series of powerful, open-source large language models.

Business Functions

AI Technology & Concepts

• LLMs (Large Language Models)

• Automation

• Chatbots

• Agents

• Avatars

• Tech & Infrastructure

  Business Functions & Roles

• Business

• Marketing

• Sales

• HR (Human Resources)

• Customer Service

• Finance

• Product Management

• Data Analysis

• Small Business

• Entrepreneurs

• Solopreneurs

  Creative & Media

• Creativity

• Design

• Writing

• Image

• Video

• Audio

• Music

• Creators

  Skills, Learning & Productivity

• Coding

• Productivity

• Learning

• Education

• Professional Development

• Personal Growth

LangChain

• A framework for developing applications powered by language models.

Performance

• LLaMA-3 models demonstrate competitive performance on various language modeling tasks, showcasing significant improvements over previous open weights models.
• Integrated free across their social media platforms
• Fast image generation, can be fine tunes for creatives.
• Introducing New AI Experiences Across Our Family of Apps and Devices | Meta (fb.com)
• What’s up with Llama 3? Arena data analysis | LMSYS Org

Large Language models:

Biomedical

• Collaborative Virtual Environments (CVEs) have immense potential in the fields of chemical and medical molecular modeling. By incorporating natural language AI and visual generative machine learning, these environments can revolutionize the way scientists and researchers approach complex chemical and biological problems. Here are some specific use cases:
• Drug design and discovery: CVEs can enable researchers to collaboratively visualize and manipulate 3D molecular structures in real-time, identifying potential drug candidates and understanding protein-ligand interactions. Natural language AI can help users interact with the molecular data, while visual generative ML can predict potential binding sites, energetics, or toxicity profiles based on existing knowledge.
• Protein structure prediction and modeling: Small teams can work together to predict protein structures, visualize folding patterns, and model protein-protein or protein-nucleic acid interactions. Natural language AI can assist in annotating and explaining the structural features, while visual generative ML can generate new structural hypotheses based on sequence alignments, homology modeling, and experimental data.
• Molecular dynamics simulations: CVEs can facilitate collaboration on complex molecular dynamics simulations, allowing researchers to analyze and visualize trajectories, energetics, and conformational changes. Natural language AI can help users navigate through simulation data and identify relevant patterns, while visual generative ML can create new conformations or predict the effects of mutations on protein stability and function.
• Cheminformatics and QSAR modeling: Researchers can leverage CVEs to develop and validate Quantitative Structure-Activity Relationship (QSAR) models, which predict the biological activity of chemical compounds based on their structural properties. Natural language AI can facilitate the exploration and interpretation of chemical descriptors, while visual generative ML can suggest new compounds with desired properties or optimize existing molecular scaffolds.
• Metabolic pathway modeling: Small teams can work together to build and analyze metabolic pathways, integrating experimental data and computational models to understand the underlying mechanisms and predict metabolic fluxes. Natural language AI can assist in annotating and explaining pathway components, while visual generative ML can generate new pathway hypotheses or predict the effects of genetic or environmental perturbations.
• Biomolecular visualization and virtual reality: CVEs can offer immersive, interactive experiences for exploring biomolecular structures and dynamics, enhancing researchers’ understanding of complex biological systems. Natural language AI can provide contextual information or guide users through molecular landscapes, while visual generative ML can create new visualizations or adapt existing ones based on user preferences and insights.
• Collaborative molecular docking and virtual screening: Small teams can use CVEs to perform collaborative molecular docking and virtual screening, which involve predicting the binding of small molecules to target proteins. Natural language AI can help users refine docking parameters and analyze results, while visual generative ML can generate alternative poses or suggest new compounds for screening based on user feedback and existing data. Choose a suitable mixed reality platform: Select a platform that allows the creation of simple, accessible shared mixed reality environments. Consider open-source options like Mozilla Hubs or JanusVR, which offer customizable and collaborative virtual spaces.
• Integrate open-source biomed software: Incorporate open-source biomed software such as PyMOL, Chimera, or VMD for molecular visualization and analysis. These tools can be integrated into the mixed reality environment for real-time interaction, allowing students and instructors to collaboratively visualize and manipulate molecular structures.
• Leverage AI and machine learning: Integrate AI and ML algorithms like those found in DeepChem, RDKit, or Open Babel to aid in the discovery and optimization of novel compounds. These tools can help predict molecular properties, perform virtual screening, and optimize lead compounds for drug development. By incorporating AI and ML, students can learn how to apply these cutting-edge techniques to real-world problems in biomedicine.
• Establish a distributed proof system: Utilize a distributed proof system like the Nostr protocol to federate the small virtual classroom environments. This will allow for seamless collaboration among students and faculty while maintaining security and data integrity.
• Create digital objects for interaction: Use digital objects such as 3D molecular models, virtual lab equipment, and interactive simulations to create an immersive learning experience. These digital objects can be shared and manipulated in real-time, promoting collaborative learning and problem-solving.
• Implement accessible interfaces: Ensure that the virtual classroom environment is accessible to all students, including those with disabilities. Utilize AI-driven tools like StabilityAI to help with language barriers, safeguarding, and governance, enabling a more inclusive learning experience.
• Foster collaboration and communication: Encourage students and faculty to collaborate on projects, share ideas, and ask questions in real-time using voice chat, text chat, or other communication tools integrated into the mixed reality environment.
• Provide training and support: Offer training sessions and support materials to help students and faculty become familiar with the mixed reality environment, the integrated biomed software, and AI/ML tools.
• Monitor progress and adjust as needed: Regularly review student progress, gather feedback, and adjust the virtual classroom environment as needed to ensure an effective and engaging learning experience.

Language Model and Voice Interface

• A specialized multi-modal LLM can be trained on local language, culture, customs, and environmental data such as flora, fauna, biotica, soil pH, and rainfall. This LLM can be accessed through a voice interface by the local community, enabling data entry and knowledge exchange in the local language. The voice interface can help overcome literacy barriers and make the system more accessible to a diverse range of community members.

Language and Tone

• Use EXPERT terminology for the given context
• AVOID: superfluous prose, self-references, expert advice disclaimers, and apologies

Grok from Musk is pretty bad, but..

• People haven’t appreciated the strength of the business model Musk has
• His is the only unified language and vision company in the world at this scale that can handle real world interactions.

Metaverse and Telecollaboration

• 🟢 I could go on all day about this, goods and bads. I literally wrote a book on it.
• 🟢 A lot (for me) hinges on OpenUSD the universal scene language. It’s been SO long since we have had something useful.
• Nvidia have a text to 3D pipeline for Omniverse. Will be interesting to see what the use cases are. This is their new Cesium [geo tile integration](https://cesium.com/blog/2024/01/16/now-available-[[NVIDIA Omniverse Platform]]-aeco-demo-pack/) giving global instant models.

https://twitter.com/BlockadeLabs/status/1719818562917761094

• This is a presentation slide and the next slide is Open Generative AI tools

Key Quotes

• “Mamba enjoys fast inference (5× higher throughput than Transformers) and linear scaling in sequence length and its performance improves on real data up to million-length sequences.”
• “This class of models can be computed very efficiently as either a recurrence or convolution with linear or near-linear scaling in sequence length.”
• “Selective SSMs and by extension the Mamba architecture are fully recurrent models with key properties that make them suitable as the backbone of general foundation models operating on sequences.”
• How does Mamba achieve linear-time modeling?: By introducing a selection mechanism in structured state space models and designing a hardware-aware algorithm that avoids materializing expanded states, thereby enhancing computational efficiency.
• What improvements does Mamba offer over traditional models?: It achieves faster inference, linear scaling with sequence length, and competitive or superior performance across various data modalities including language, audio, and genomics.
• Can Mamba handle long sequences efficiently?: Yes, it is specifically designed to address the computational inefficiencies of traditional models like Transformers in handling long sequences, offering linear scaling and improved performance.
• Mamba outperforms Transformers of the same size in language modelling and matches Transformers twice its size both in pretraining and downstream evaluation, demonstrating its efficiency and effectiveness.
• Authors’ Views: The authors propose Mamba as a significant step forward in sequence modeling, addressing the inefficiency of Transformers while maintaining or improving performance.
• Comparative Analysis: Mamba is positioned as superior to existing models, particularly Transformers, in terms of efficiency and scalability

Long Context Modeling

• LongMamba (LongMamba): Generalizes to 40k tokens after training on 16k, nearly perfect on “needle in a haystack” task
• Evo: Models long DNA sequences, learning a “cell model” analogous to language models’ “world model”
• Potential challenges:
  • Eventual “rotting” of internal states with extreme context lengths
  • Need for state regularization or “pruning” to maintain performance
• Implications for biology: Foundation models could revolutionize drug discovery and biological research

Research

• Phi-3 and Phi-4: Powerful language models compact enough to run on a smartphone.

Existing state of the Art

• Index — MLGT: The authoritative multi-lingual glossary of terms (metaverse-standards.org)
• It was unexpectedly successful, resulting in what seems to be an internally consistent knowledge graph in an Web Ontology Language compliant ontology for the design represented throughout this wider Logseq Knowledge Graphing.
  • Although the established OWL can richly describe our ontology, it’s a little too arcane. Nonetheless the full text can be seen where with the diagram.
    • OWL based Ontology
      • https://github.com/VisualDataWeb/WebVOWL
      • Lack of standardized context definitions for vocabularies and IRI mappings
      • No explicit typing mechanism like @type
      • Limited to absolute IRIs, no compact IRIs or relative IRIs
      • Unclear semantics for blank node identifiers
      • No standardized representation of indexed values, lists, and named graphs
      • Inability to reshape data structure using framing
      • Reduced interoperability with RDF and Linked Data ecosystem

Biomedical

• Collaborative Virtual Environments (CVEs) have immense potential in the fields of chemical and medical molecular modeling. By incorporating natural language AI and visual generative machine learning, these identify relevant patterns, while visual generative ML can create new conformations or predict the effects of mutations on protein stability and function.
• Cheminformatics and QSAR modeling: Researchers can leverage CVEs to develop and validate Quantitative Structure-Activity Relationship (QSAR) models, which predict the biological activity of chemical compounds based on their structural properties. Natural language AI can facilitate the exploration and interpretation of chemical descriptors, while visual generative ML can suggest new compounds with desired properties or optimize existing molecular scaffolds.
• Metabolic pathway modeling: Small teams can work together to build and analyze metabolic pathways, integrating experimental data and computational models to understand the underlying mechanisms and predict metabolic fluxes. Natural language AI can assist in annotating and explaining pathway components, while visual generative ML can generate new pathway hypotheses or predict the effects of genetic or environmental perturbations.
• Biomolecular visualization and virtual reality: CVEs can offer immersive, interactive experiences for exploring biomolecular structures and dynamics, enhancing researchers’ understanding of complex biological parameters and analyze results, while visual generative ML can generate alternative poses or suggest new compounds for screening based on user feedback and existing data. Choose a suitable mixed reality platform: Select a platform that allows the creation of simple, accessible shared mixed reality environments. Consider open-source options like Mozilla Hubs or JanusVR, which offer customizable and collaborative virtual spaces.
• Integrate open-source biomed software: Incorporate open-source biomed
• Leverage AI and machine learning: Integrate AI and ML algorithms like those found in DeepChem, RDKit, or Open Babel to aid in the discovery and optimization of novel compounds. These tools can help predict molecular properties, perform virtual screening, and optimize lead
• Create digital objects for interaction: Use digital objects such as 3D molecular models, virtual lab equipment, and interactive simulations to create an immersive learning experience. These digital objects can be shared and manipulated in real-time, promoting collaborative learning and problem-solving.
• Implement accessible interfaces: Ensure that the virtual classroom environment is accessible to all students, including those with
• Monitor progress and adjust as needed: Regularly review student progress, gather feedback, and adjust the virtual classroom environment as needed to ensure an effective and engaging learning experience.

Language Model and Voice Interface

• A specialized multi-modal LLM can be trained on local language, culture, customs, and environmental data such as flora, fauna, biotica, soil pH, and rainfall. This LLM can be accessed through a voice interface by the local community, enabling data entry and knowledge exchange in the local language. The voice interface can help overcome literacy barriers and make the system more accessible to a diverse range of community members.
• A live connection with the academic team allows for model tuning through prompt engineering, vector database updates, and efficient Lora models, potentially offering timely advice for ecosystem interventions. Real-time communication between the community and academic teams can help identify areas of concern and rapidly adapt the LLM to address emerging challenges.

Language and Tone

• Use EXPERT terminology for the given context
• AVOID: superfluous prose, self-references, expert advice disclaimers, and apologies

Mamba: Linear-Time Sequence Modelling with Selective State Spaces

• Mamba: Linear-Time Sequence Modeling with Selective State Spaces
• “This class of models can be computed very efficiently as either a recurrence or convolution with linear or near-linear scaling in sequence length.”
• “Selective SSMs and by extension the Mamba architecture are fully recurrent models with key properties that make them suitable as the backbone of general foundation models operating on sequences.”
• How does Mamba achieve linear-time modeling?: By introducing a selection mechanism in structured state space models and designing a hardware-aware algorithm that avoids materializing expanded states, thereby enhancing computational efficiency.
• What improvements does Mamba offer over traditional models?: It achieves faster inference, linear scaling with sequence length, and competitive or superior performance across various data modalities including language, audio, and genomics.
• Can Mamba handle long sequences efficiently?: Yes, it is specifically designed to address the computational inefficiencies of traditional models like Transformers in handling long sequences, offering linear scaling and improved performance.
• Mamba outperforms Transformers of the same size in language modelling and matches Transformers twice its size both in pretraining and downstream evaluation, demonstrating its efficiency and effectiveness.
• Authors’ Views: The authors propose Mamba as a significant step forward in sequence modeling, addressing the inefficiency of Transformers while maintaining or improving performance.
• Comparative Analysis: Mamba is positioned as superior to existing models, particularly Transformers, in terms of efficiency and scalability

6️⃣ Diffusion Models (Generative Models)

• Next presentation slide Proprietary Large Language Models

Biomedical

• Collaborative Virtual Environments (CVEs) have immense potential in the fields of chemical and medical molecular modeling. By incorporating natural language AI and visual generative machine learning, these identify relevant patterns, while visual generative ML can create new conformations or predict the effects of mutations on protein stability and function.
• Cheminformatics and QSAR modeling: Researchers can leverage CVEs to develop and validate Quantitative Structure-Activity Relationship (QSAR) models, which predict the biological activity of chemical compounds based on their structural properties. Natural language AI can facilitate the exploration and interpretation of chemical descriptors, while visual generative ML can suggest new compounds with desired properties or optimize existing molecular scaffolds.
• Metabolic pathway modeling: Small teams can work together to build and and dynamics, enhancing researchers’ understanding of complex biological parameters and analyze results, while visual generative ML can generate alternative poses or suggest new compounds for screening based on user feedback and existing data. Choose a suitable mixed reality platform: Select a platform that allows the creation of simple, accessible shared mixed reality environments. Consider open-source options like Mozilla Hubs or JanusVR, which offer customizable and collaborative virtual spaces. create an immersive learning experience. These digital objects can be shared and manipulated in real-time, promoting collaborative learning and problem-solving.
• Implement accessible interfaces: Ensure that the virtual classroom environment is accessible to all students, including those with
• Monitor progress and adjust as needed: Regularly review student progress, gather feedback, and adjust the virtual classroom environment
• The modular open-source system can be applied to various training, The open-source system can be adapted to serve various industries, making remote collaboration more efficient and inclusive.

Language and Tone

6️⃣ Diffusion Models (Generative Models)

• Next presentation slide Proprietary Large Language Models

See Also

• Large Language Models
• Google AI Technology Corporation

See Also

• Large Language Models
• OpenAI Research Organisation
• Google AI Technology Corporation

Software stack

• This section needs building out to describe the stack and the choices made, but can be seen in Figure [fig:pyramind] and Figure [fig:highlevelstack].

  

  image

  At this time we favour the following component units, with alternatives in brackets.

  • 🟩 Open source collaborative space 🟩 Headset VR integration 🟨 WebGL interface 🟩 Minting digital assets (Ordinal then RGB) 🟨 Digital asset integration and management 🟩 Large language model MVP 🟩 Large language model API integration 🟩 Large language model voice to voice interface 🟩 Stable diffusion image creation MVP 🟩 Stable diffusion image creation MVP 🟨 AutoGPT voice to voice integration MVP 🟨 Stable diffusion image creation API 🟨 Nostr social media integration 🟨 Nostr identity management 🟨 Nostr machine to machine finacially enabled bots (ubiquitous federating agents) 🟨 Nostr human programmable semi autonomous economic actors 🟩 Bitcoin / Lightning / stablecoin stack 🟥 Bitcoin / Lightning / stablecoin integration 🟨 Collaborative virtual production MVP 🟥 Collaborative virtual production integration 🟥 3D asset generation with ML

  

  • Collaborative space
  • Vircadia [Omniverse, Open3D foundation, Unreal]
  • Distributed truth
  • Bitcoin testnet [Main net]
  • Digital Objects
  • Fedimint [Ordinals, Pear credits, RGB]
  • Messaging and sync
  • Nostr
  • Identity
  • Nostr [Bluesky ION, pubky]
  • Fiat money xfer
  • Fedimint [Pear credits, RGB, Taro main net]
  • Hardware signing
  • Seed signer [any hardware wallet]
  • Small group banking
  • Fediment [chaumian ecash]
  • Local wallet
  • Mutiny [bitkit, and lightning wallet]
  • Machine learning text
  • Alpaca [ChatGPT etc]
  • Machine learning image
  • Stable diffusion [midjourney, Dall-E]
  • Object tracking
  • Nostr [LnBits accounts]

Llama 3 overview

• Meta (formerly Facebook) has recently released LLaMA-3, a series of foundational large language models (LLMs) that aim to advance AI research while remaining more accessible in terms of computational requirements.

Multilingual and Abstract Translation

• Projects dedicated to improving LLM capabilities in translation, fostering better understanding and communication across languages.
  • SeamlessM4T by Facebook Research
  • An innovative project aimed at enhancing multilingual translation, showcasing efforts to bridge language barriers and improve communication globally.

Key Features

• 3D Modeling: A comprehensive suite of modeling tools for creating, transforming, and editing your models.

• Sculpting: Digital sculpting tools provide the power and flexibility required in several stages of the digital production pipeline.

• Animation & Rigging: A production-ready camera and object tracking solution.

• Grease Pencil: A revolutionary 2D animation tool that allows you to draw in 3D space.

• Rendering: A powerful, unbiased rendering engine that offers stunning, ultra-realistic rendering.

• Simulation: Create amazing simulations with industry-standard libraries such as Bullet and MantaFlow.

• Video Editing: A built-in video sequence editor allows you to perform basic actions like video cuts and splicing, as well as more complex tasks like video masking or color grading.

• Scripting: With a rich Python API, Blender is highly customizable and can be extended with custom tools and add-ons.

• VFX: A built-in compositor allows you to post-produce your renders without leaving Blender.

  Core Characteristics

• Probability Distribution Learning: Modelling P(word|context) or P(sequence)

• Autoregressive: Left-to-right sequential prediction (GPT-style)

• Masked Language Modeling: Bidirectional context learning (BERT-style)

• Large-Scale Pre-Training: Training on billions to trillions of tokens

• Transfer Learning: Fine-tuning for diverse downstream tasks

• Emergent Capabilities: In-context learning, reasoning, few-shot adaptation

  Relationships

• Foundation For: Natural Language Processing, Text Generation, Machine Translation

• Related: Large Language Model, Transformer, Self-Attention

• Models: GPT series, BERT, T5, LLaMA, PaLM, Gemini

• Paradigms: Autoregressive LM, Masked LM, Encoder-Decoder LM

  Key Literature

  1. Bengio, Y., et al. (2003). “A neural probabilistic language model.” Journal of Machine Learning Research, 3, 1137-1155.

  2. Radford, A., et al. (2018). “Improving language understanding by generative pre-training.” OpenAI Technical Report.

  3. Devlin, J., et al. (2019). “BERT: Pre-training of deep bidirectional transformers for language understanding.” NAACL, 4171-4186.

  4. Brown, T., et al. (2020). “Language models are few-shot learners.” NeurIPS, 1877-1901.

  5. Kaplan, J., et al. (2020). “Scaling laws for neural language models.” arXiv:2001.08361.

  6. Wei, J., et al. (2022). “Emergent abilities of large language models.” TMLR.

  Technical Concepts

  Autoregressive Language Modeling

  P(x₁, x₂, ..., xₙ) = ∏ᵢ P(xᵢ | x₁, x₂, ..., xᵢ₋₁)
  

  Models predict next token conditioned on previous tokens (GPT-style).

  Masked Language Modeling

  Objective: Predict masked tokens given bidirectional context
  Example: "The [MASK] sat on the mat" → predict "cat"
  

  Enables bidirectional context learning (BERT-style).

  Perplexity

  Perplexity = exp(-1/N ∑ᵢ log P(xᵢ | context))
  

  Evaluation metric measuring model uncertainty.

  Applications

  1. Text Generation: Article writing, story generation, content creation
  2. Code Generation: Programming assistance, code completion
  3. Machine Translation: Neural machine translation systems
  4. Question Answering: Reading comprehension and knowledge retrieval
  5. Text Classification: Sentiment analysis, topic modelling
  6. Named Entity Recognition: Information extraction
  7. Dialogue Systems: Conversational AI and chatbots
  8. Text Summarisation: Document summarisation
  9. Semantic Search: Embedding-based information retrieval
  10. Few-Shot Learning: In-context learning for new tasks

  Pre-Training Paradigms

  Causal Language Modeling (CLM)

• Models: GPT, GPT-2, GPT-3, GPT-4, LLaMA, PaLM

• Training: Left-to-right autoregressive prediction

• Strength: Natural text generation, few-shot learning

• Applications: Chat, content generation, code synthesis

  Masked Language Modeling (MLM)

• Models: BERT, RoBERTa, ALBERT, DeBERTa

• Training: Bidirectional context with masked token prediction

• Strength: Understanding tasks, representation learning

• Applications: Classification, NER, question answering

  Encoder-Decoder

• Models: T5, BART, mT5, UL2

• Training: Sequence-to-sequence with various objectives

• Strength: Versatile text-to-text framework

• Applications: Translation, summarisation, generation

  Scaling Laws

  Language model performance improves predictably with:

• Model Size: Number of parameters (millions to trillions)

• Data Size: Training tokens (billions to trillions)

• Compute: FLOPs for training

  Power law relationships:

  Loss ∝ N^(-α)  where N is model size
  Loss ∝ D^(-β)  where D is dataset size
  Loss ∝ C^(-γ)  where C is compute budget
  

  Emergent Abilities

  Large language models exhibit emergent capabilities appearing at scale:

• Few-Shot Learning: Learning from examples in prompts

• Chain-of-Thought Reasoning: Step-by-step problem solving

• Instruction Following: Zero-shot task execution from instructions

• Multi-Step Reasoning: Complex reasoning tasks

• Code Understanding: Programming language comprehension

  Evaluation Metrics

• Perplexity: Model uncertainty on held-out text

• BLEU: Machine translation quality

• ROUGE: Summarisation quality

• Accuracy: Task-specific classification accuracy

• F1 Score: Balanced precision and recall

• Human Evaluation: Quality, coherence, factuality assessment

  Challenges and Research Directions

  1. Hallucination: Generating plausible but factually incorrect content
  2. Bias and Fairness: Societal biases in training data and outputs
  3. Interpretability: Understanding model decisions and reasoning
  4. Efficiency: Reducing computational costs for training and inference
  5. Long-Context Modeling: Extending context windows beyond current limits
  6. Multimodal Integration: Combining language with vision, audio, other modalities
  7. Continual Learning: Updating knowledge without catastrophic forgetting
  8. Grounding: Connecting language to external knowledge and real-world facts
  9. Controllability: Steering model outputs for desired properties
  10. Safety and Alignment: Ensuring beneficial and aligned AI behaviour

  See Also

• Natural Language Processing

• Large Language Model

• Transformer

• Text Generation

• GPT

• BERT

• Self Attention

  Current Landscape

• The AI landscape is currently dominated by a few large technology companies that are developing and deploying powerful large language models (LLMs). These models are being integrated into a wide range of products and services, from search engines to creative tools.

• AI Index Report 2024

See Also

• Large Language Models
• OpenAI Research Organisation
• Google AI Technology Corporation

How it Works

• AnimateDiff works by adding a motion modeling module to a stable diffusion model. This module is trained on a large dataset of videos and learns to predict the motion between frames. When you provide AnimateDiff with an image and a text prompt, it uses the motion modeling module to generate a sequence of frames that create an animation.

Multilingual and Abstract Translation

• Projects dedicated to improving LLM capabilities in translation, fostering better understanding and communication across languages.

  • SeamlessM4T by Facebook Research

  • An innovative project aimed at enhancing multilingual translation, showcasing efforts to bridge language barriers and improve communication globally.

    Core Characteristics

• Probability Distribution Learning: Modelling P(word|context) or P(sequence)

• Autoregressive: Left-to-right sequential prediction (GPT-style)

• Masked Language Modeling: Bidirectional context learning (BERT-style)

• Large-Scale Pre-Training: Training on billions to trillions of tokens

• Transfer Learning: Fine-tuning for diverse downstream tasks

• Emergent Capabilities: In-context learning, reasoning, few-shot adaptation

  Relationships

• Foundation For: Natural Language Processing, Text Generation, Machine Translation

• Related: Large Language Model, Transformer, Self-Attention

• Models: GPT series, BERT, T5, LLaMA, PaLM, Gemini

• Paradigms: Autoregressive LM, Masked LM, Encoder-Decoder LM

  Key Literature

  1. Bengio, Y., et al. (2003). “A neural probabilistic language model.” Journal of Machine Learning Research, 3, 1137-1155.

  2. Radford, A., et al. (2018). “Improving language understanding by generative pre-training.” OpenAI Technical Report.

  3. Devlin, J., et al. (2019). “BERT: Pre-training of deep bidirectional transformers for language understanding.” NAACL, 4171-4186.

  4. Brown, T., et al. (2020). “Language models are few-shot learners.” NeurIPS, 1877-1901.

  5. Kaplan, J., et al. (2020). “Scaling laws for neural language models.” arXiv:2001.08361.

  6. Wei, J., et al. (2022). “Emergent abilities of large language models.” TMLR.

  Technical Concepts

  Autoregressive Language Modeling

  P(x₁, x₂, ..., xₙ) = ∏ᵢ P(xᵢ | x₁, x₂, ..., xᵢ₋₁)
  

  Models predict next token conditioned on previous tokens (GPT-style).

  Masked Language Modeling

  Objective: Predict masked tokens given bidirectional context
  Example: "The [MASK] sat on the mat" → predict "cat"
  

  Enables bidirectional context learning (BERT-style).

  Perplexity

  Perplexity = exp(-1/N ∑ᵢ log P(xᵢ | context))
  

  Evaluation metric measuring model uncertainty.

  Applications

  1. Text Generation: Article writing, story generation, content creation
  2. Code Generation: Programming assistance, code completion
  3. Machine Translation: Neural machine translation systems
  4. Question Answering: Reading comprehension and knowledge retrieval
  5. Text Classification: Sentiment analysis, topic modelling
  6. Named Entity Recognition: Information extraction
  7. Dialogue Systems: Conversational AI and chatbots
  8. Text Summarisation: Document summarisation
  9. Semantic Search: Embedding-based information retrieval
  10. Few-Shot Learning: In-context learning for new tasks

  Pre-Training Paradigms

  Causal Language Modeling (CLM)

• Models: GPT, GPT-2, GPT-3, GPT-4, LLaMA, PaLM

• Training: Left-to-right autoregressive prediction

• Strength: Natural text generation, few-shot learning

• Applications: Chat, content generation, code synthesis

  Masked Language Modeling (MLM)

• Models: BERT, RoBERTa, ALBERT, DeBERTa

• Training: Bidirectional context with masked token prediction

• Strength: Understanding tasks, representation learning

• Applications: Classification, NER, question answering

  Encoder-Decoder

• Models: T5, BART, mT5, UL2

• Training: Sequence-to-sequence with various objectives

• Strength: Versatile text-to-text framework

• Applications: Translation, summarisation, generation

  Scaling Laws

  Language model performance improves predictably with:

• Model Size: Number of parameters (millions to trillions)

• Data Size: Training tokens (billions to trillions)

• Compute: FLOPs for training

  Power law relationships:

  Loss ∝ N^(-α)  where N is model size
  Loss ∝ D^(-β)  where D is dataset size
  Loss ∝ C^(-γ)  where C is compute budget
  

  Emergent Abilities

  Large language models exhibit emergent capabilities appearing at scale:

• Few-Shot Learning: Learning from examples in prompts

• Chain-of-Thought Reasoning: Step-by-step problem solving

• Instruction Following: Zero-shot task execution from instructions

• Multi-Step Reasoning: Complex reasoning tasks

• Code Understanding: Programming language comprehension

  Evaluation Metrics

• Perplexity: Model uncertainty on held-out text

• BLEU: Machine translation quality

• ROUGE: Summarisation quality

• Accuracy: Task-specific classification accuracy

• F1 Score: Balanced precision and recall

• Human Evaluation: Quality, coherence, factuality assessment

  Challenges and Research Directions

  1. Hallucination: Generating plausible but factually incorrect content
  2. Bias and Fairness: Societal biases in training data and outputs
  3. Interpretability: Understanding model decisions and reasoning
  4. Efficiency: Reducing computational costs for training and inference
  5. Long-Context Modeling: Extending context windows beyond current limits
  6. Multimodal Integration: Combining language with vision, audio, other modalities
  7. Continual Learning: Updating knowledge without catastrophic forgetting
  8. Grounding: Connecting language to external knowledge and real-world facts
  9. Controllability: Steering model outputs for desired properties
  10. Safety and Alignment: Ensuring beneficial and aligned AI behaviour

  See Also

• Natural Language Processing

• Large Language Model

• Transformer

• Text Generation

• GPT

• BERT

• Self Attention

Provenance