The concept of the Metaverse and Telecollaboration has gained significant attention. While its final form remains unclear, the potential of digital society is evident. We see advantage not in a social metaverse, but in solving business-to-business technical use cases where professionals with visual technical problems or training requirements can gather in collaborative spaces.
We have designed a Metaverse Ontology to ensure specificity for our work.
Trust, Accessibility, Governance, and Safeguarding:
The Metaverse faces challenges, including poor adoption, overstated market need, and a lack of genuine digital society use cases. Privacy, Trust and Safety abuses by incumbent providers have created an opportunity for a more open internet. Emerging markets face barriers to entry due to inadequate identification, banking infrastructure, and computing power. There is an opportunity to build pervasive digital spaces with a different and more open foundation.
The Need for Modular Open-Source Solutions:
A topologically flat, inclusive, permissionless, federated, and Open Source Metaverse is essential to address these challenges. Open-source AI tooling and large language models can improve creativity, safeguarding, and governance, while breaking down language and accessibility barriers. Secure, trusted, and task-appropriate solutions can promote collaboration and innovation.
Technical Problem Definition:
The specific technical challenges and opportunities the proposed framework addresses include:
Evergreen telecollaboration around technical issues
Exchange of goods, services, and money within systems, without friction
Identity management within virtual spaces
Access to information in the extrinsic world from within the tool
Federation of instances without overhead (scaling)
Seamless access to personal information within and without the collaborative system
Ability to take advantage of supporting smart support agents (bots, etc.) throughout
Governance, trust, and safeguarding
Lean Canvas Business Model:
Problem: Existing large-scale telecollaboration solutions suffer from poor adoption, limited accessibility, and trust issues. Meanwhile, emerging markets struggle to participate in the growing digital society due to the lack of inclusive tools and infrastructure, limiting access to global talent and new pools of ideas. There is insufficient provision of global talent pipelines for highly technical workflows.
Solution: Develop a secure, accessible, and inclusive platform for specialized telecollaboration spaces that seamlessly integrate advanced AI, ML, highly scalable and proven distributed systems, and open-source principles to create a digital society that caters to diverse industries, users globally, and captures global talent and innovative ideas.
Value Proposition: Ultra low cost training spaces, accessible 24/7 through very low end hardware. Interact with highly customizable, task-appropriate, and user-friendly specialized telecollaboration spaces supported by specially trained and optimised supportive large language AI models. Multi-lingual for emerging markets, enabling access to untapped global talent and fostering the exchange of diverse ideas.
Customer Segments: Initially Universities, but this will scale to be sector specific, catering to the global training, research, biomedical, and creative industries, with a special focus on empowering users in emerging markets such as Africa and India, and connecting them with worldwide opportunities and resources.
Revenue Streams: Tiered subscription plans to accommodate various user needs and budgets, as well as tailored enterprise solutions for large-scale clients. Bespoke consulting and support trending toward software as a service at scale.
Key Metrics: Track user growth, engagement, and retention, successful collaborations across industries, the platform’s positive impact on users in emerging markets, and the effectiveness of global talent capture and idea exchange.
Unfair Advantage: The team’s extensive experience in telecollaboration research, AI, ML, and a deep understanding of the complex landscape of emerging technologies, including highly scalable and proven distributed systems, provide a unique edge in creating a game-changing platform for specialized telecollaboration spaces that are secure, trusted, and tailored to diverse user needs while enabling access to global talent and innovative ideas.
Proposed Layered Framework
Layer 1: Bitcoin, Lightning, and Nostr Protocols:
The proposed framework leverages [Bitcoin], [Lightning and Similar L2], and [Nostr protocol] protocols to provide a secure and decentralized foundation for value exchange, identity management, and communication. These technologies enable the transfer of portable ‘goods’ across digital society and promote the development and adoption of open protocols and data formats. The Nostr protocol, in particular, can link and federate mixed reality spaces, providing identity assurances and mediating data synchronization while maintaining reasonably strong cryptography. This also allows integration with the legacy web through ubiquitous web sockets. Bitcoin and associated technologies, despite their issues, have the potential to revolutionize the way digital society operates by enabling “money-like networks” which are a cornerstone of human interaction. Representations of traditional currencies can ride securely on top of these networks as stablecoins, opening up global collaborative working practices, especially for emerging markets. Streaming micropayments and machine to machine (AI to AI) interactions are crucially and under-considered in this context.
Layer 2: Modular Human-Computer Interface:
The framework proposes the development of collaborative global networks for training, research, biomedical, and creative industries using immersive and accessible environments. Engaging with ideas from diverse cultural backgrounds can enrich the overall user experience. Industry players have noted the risk and failures associated with closed systems like Meta and are embracing the “open Metaverse” narrative to de-risk their interests. To enable a truly open and interoperable Metaverse, it is crucial to develop open-source APIs, SDKs, and data standards that allow different platforms to communicate and exchange information. While the initial focus will be on building around a simpler open-source engine, the framework aims to link across standards such as Unity, [Unreal], and [NVIDIA Omniverse] as it develops. This can be accomplished using the federation layer.
LLM and Generative ML Integration:
Bots and AI Agents:
Autonomous AI Agents, bonded to, but not bounded by, each federated mixed reality instance, can be self-governing entities that operate within their federated virtual social spaces, drawing upon private Bitcoin and Lightning wallets to perform and mediate economic exchanges within the spaces. They could also trivially operate outside the virtual space, and within other spaces on the same metaverse federation. They would accomplish this by drawing on their ‘home’ GPU/TPU processors where appropriate, or else using distributed large language model (LLM) processing to accomplish tasks assigned by their instructors. They can interact with the ‘web2’ world using open-source software called auto-gpt and have constraints, such as “time to live” and limited access to funds through their Bitcoin Lightning wallets.
Resource Management and Financial Autonomy:
These AI agents have access to dedicated LLM resources within their home instances in the federated virtual social spaces. If such resources are unavailable, they can resort to using slower, distributed open-source LLMs like Horde. This flexibility ensures that the agents can continue to function and complete tasks even if faced with limited LLM interpretive resources. The AI agents have their own private Bitcoin and Lightning wallets, which enable them to manage and utilize funds independently. They can use these funds to pay for services, acquire resources, or even trade with other agents or users within the virtual social spaces.
Social Interactions and Adaptive Learning:
Within the federated virtual social spaces, AI agents can communicate and collaborate with other agents or human users. They can participate in discussions, provide assistance, or even learn from the interactions, thereby improving their capabilities over time. Language translation, governance, and safeguarding could also be developed. Safeguarding would be handled by threshold risk triggers and transmission of data in a sovereign way to all parties, allowing external action by authorities appropriate to any abuse. As AI agents interact with their environment, other agents, and users, they can learn and adapt their behaviour. This enables them to improve their performance, better understand their assigned tasks, and become more effective at achieving their goals.
Application Case Studies
Classic Use Cases:
The proposed framework can be applied to traditional collaborative scenarios, such as small teams working on product, architectural, or industrial design. For example, a team of architects could use a collaborative virtual environment (CVE) to walk through a 3D model of a building, making changes in real-time and getting immediate feedback from colleagues. Similarly, a team of product designers could use a CVE to prototype and test a new product, iterating on the design in a shared virtual space. These teams can benefit from CVEs that allow them to visualize, modify, and iterate on 3D models in real-time.
Expanding Use Cases with AI and ML:
Virtual Training and Simulation:
CVEs can facilitate skill development and training in various industries, such as healthcare, military, aviation, and emergency response. Trainees can practice procedures in a virtual environment, with natural language AI providing instructions, explanations, or feedback. Generative AI can now create entire interactive 3D environments on the fly, allowing for the rapid prototyping and deployment of complex, adaptable virtual scenarios. AI-powered avatars and non-player characters (NPCs) are also becoming more lifelike, capable of nuanced and dynamic interactions, which is particularly impactful in areas like virtual training and customer service, where realistic simulations and interactions are paramount.
Remote Teleconferencing:
In situations where face-to-face communication is not feasible, CVEs can enable remote teams to work together on shared visual tasks like planning events, brainstorming ideas, or reviewing documents. Natural language AI can transcribe and analyse spoken conversations, providing real-time translations or summaries. AI-driven tools are also streamlining content creation, allowing artists and designers to accelerate their workflows by transforming simple sketches into detailed 3D assets. Machine learning algorithms are being employed to personalize user experiences in real-time, adapting the environment and tasks to individual preferences and skill levels.
Virtual Art & Media Collaboration:
Artists, animators, and multimedia professionals can collaborate in CVEs to create and develop their projects, such as films, animations, or video games. Natural language AI can help in storyboarding, scriptwriting, or character development, while visual generative ML can generate new visuals or adapt existing assets based on user input and style preferences.
Data Visualization and Analysis:
Small teams working with large datasets can use CVEs to visually explore and analyze data in a more intuitive and engaging way. Natural language AI can help users query and interact with the data using conversational interfaces, while visual generative ML can generate new visualizations based on patterns and trends identified in the data.
Education and Virtual Classrooms:
Educators can leverage CVEs to create immersive learning experiences that engage students in collaborative activities, such as group projects, problem-solving, or scientific experiments. Natural language AI can facilitate communication, provide personalized tutoring, or assess student progress, while visual generative ML can create customized educational content based on individual needs and interests.
Virtual Labs and Scientific Research:
Researchers can use CVEs to conduct experiments, visualize complex data, or simulate real-world conditions in a controlled environment. Natural language AI can assist in interpreting results, automating lab protocols, or identifying research gaps, while visual generative ML can generate predictions or models based on existing data to support hypothesis testing and decision-making.
Biomedical:
In fields like chemical and medical molecular modeling, the integration of AI and generative ML technologies can significantly improve collaboration and innovation. Teams can work together in immersive environments to visualize complex molecular structures, benefiting from real-time AI-generated visuals and natural language processing.
Case Study: Biodiversity Monitoring and Data Exchange with Isolated Communities:
The case study presents an open-source collaboration infrastructure that leverages advanced technologies such as multi-modal large language models (LLMs), satellite communication, and cryptocurrency networks to facilitate sustainable and reliable biodiversity monitoring and data exchange in isolated communities. Key components include:
Language Model and Voice Interface
Data Collection and Storage
Live Connection and Model Tuning
Ecosystem Interventions
Incentives and Education
Monetization and Blockchain Integration
Visual Training Support Systems
Solar Infrastructure
Open-Source Collaboration
The case study also addresses risk mitigation, ethical considerations, capacity building, and local empowerment. The proposed infrastructure has the potential to transform how isolated communities interact with their environment, enabling them to make informed decisions about conservation and ecosystem management.
Overcoming Challenges and Barriers
Trust, Accessibility, and Governance:
To create a successful open-source Metaverse, it is crucial to address trust, accessibility, and governance challenges. By integrating decentralized and secure technologies such as blockchain and distributed ledger systems, a more transparent and trustworthy infrastructure can be established. Blockchain technology is being leveraged to enhance trust by providing an immutable and transparent ledger for transactions and interactions. Decentralized identifiers (DIDs) are another important innovation, providing users with a secure digital identity that they can use across different virtual environments. One promising approach to governance is the use of Decentralized Autonomous Organizations (DAOs), which provide a model for automated governance that is based on code rather than a traditional hierarchical structure.
Ensuring Safeguarding and Privacy Compliance:
Protecting user privacy and ensuring safeguarding is vital for any digital society platform. The open-source system must be developed in compliance with legislative and cultural norms while maintaining the balance between user privacy and the need for identity verification and data management. The evidence that social media is damaging youth mental health is very compelling. The Centre for Humane Technology calls social media the ‘first contact point’ with AI, explaining that new technologies often create an arms race. The underlying arms race for attention led to what they call ‘an engagement monster’ that rewrote the rules of society. These lessons should be learnt and the problems should be proactively mitigated. This proposal is not a social metaverse, and deliberately limits both numbers of participants and avatar optionality.
Managing Scalability, Performance, and Latency:
As the Metaverse continues to grow, it is crucial to ensure that the open-source system can scale effectively and maintain optimal performance. By using distributed and federated networks, the system can better manage latency and performance issues, ensuring a seamless user experience.
Promoting Open Standards and Interoperability:
For the Metaverse to truly thrive, it is essential to promote open standards and interoperability among various platforms and systems. This can be achieved by fostering collaboration between industry stakeholders, encouraging the development of open protocols, APIs, and data standards, and actively supporting the open-source community.
Future Outlook and Potential Developments
AI and Generative ML Technologies:
As AI and generative ML technologies continue to evolve, their integration into the Metaverse will further enhance user experiences and create new opportunities for innovation. The release of models like GPT-4 have already prompted debate about general AI. It seems unavoidable that this will all impact on the Metaverse and digital society. For example, generative AI can now create entire interactive 3D environments on the fly, allowing for the rapid prototyping and deployment of complex, adaptable virtual scenarios. AI-powered avatars and non-player characters (NPCs) are also becoming more lifelike, capable of nuanced and dynamic interactions.
Inclusive Digital Society:
By overcoming barriers to entry for emerging markets and less developed nations, a more inclusive digital society can be fostered. This inclusivity will empower new ideas and perspectives, leading to a richer and more diverse digital landscape.
Spatial and Augmented Reality Technologies:
The incorporation of spatial and augmented reality technologies can expand the possibilities within the Metaverse, allowing for more immersive and interactive experiences. These technologies have the potential to reshape digital society and redefine the ways in which people interact with digital environments.
Economic Empowerment AI Actors:
The creation of an open and economically empowered Metaverse, in which AI actors can mediate governance issues and participate in economic transactions, can lead to a more efficient and dynamic digital ecosystem. This integration will enable new business models and opportunities for all users, both human and AI.
Continuous Evolution and Adaptation:
As the digital landscape continues to evolve, the open-source Metaverse system must be flexible and adaptable to meet changing needs and expectations. Continuous innovation and collaboration within the industry will be crucial for the success and longevity of the Metaverse as a transformative digital society platform.
Embracing the Open-Source Metaverse Vision:
To create a truly transformative and inclusive digital society, it is essential to embrace the vision of an open-source Metaverse. By fostering collaboration, promoting open standards, and integrating advanced AI and ML technologies, the Metaverse can become a platform that serves societal and business needs.
Learning from Past Failures:
Learning from past failures and addressing challenges head-on will be critical to the successful development of an open-source Metaverse. Trust, accessibility, governance, and safeguarding issues must be thoughtfully considered and addressed to build a secure and user-friendly platform.
Unlocking New Opportunities and Use Cases:
The integration of AI, ML, and cutting-edge technologies within the Metaverse can unlock new opportunities and use cases across various industries, including education, research, biomedical, and creative fields. By building on a modular open-source system, these opportunities can be explored and realized to their full potential.
Fostering Collaboration and Inclusivity:
Creating an inclusive digital society is a key goal for the open-source Metaverse. By breaking down barriers and making the platform accessible to a wider audience, new ideas and perspectives will enrich the digital landscape and drive innovation.
Shaping the Future of Digital Society:
As the Metaverse continues to evolve and grow, it will play an increasingly important role in shaping the future of digital society. By embracing an open-source vision, overcoming challenges, and unlocking new opportunities, the Metaverse can become a powerful platform that transforms how people live, work, and interact in the digital world.
Industry Conversations:
Continued dialogue and collaboration among industry stakeholders are vital to ensuring the successful development of the open-source Metaverse. By engaging in conversations and understanding the cautious appetite for the ideas presented, the community can work together to shape the future of digital society and overcome the challenges that lie ahead.
In-Camera VFX & Telepresence:
The proposed framework can be applied to film production and virtual production workflows. By leveraging the world’s most powerful decentralized computing network (Bitcoin) and cryptographically assured endpoints, the system can enable scale and security without high cost. New tooling in the space allows for microtransactions and micropayments, radically improving creative microtask workflows. The unified digital backend is optimized for flows of money, trust, and digital objects, offering a new area for virtual production.
Software Stack
Novel VP Render Pipeline:
Putting the ML image generation on the end of a real-time tracked camera render pipeline might remove the need for detail in set building. The set designer, DP, director, etc., will be able to ideate in a headset-based metaverse of the set design, dropping very basic elements. If the interframe consistency (img2img) can deliver, the output on the VP screen can simply inherit the artistic style from the text prompts and render production quality from the basic building blocks. This “next level pre-vis” is being trailed in the Vircadia collaborative environment described in this book.