- New NVIDIA releases
- https://github.com/NVlabs/SOMA-X
- https://research.nvidia.com/labs/sil/projects/kimodo/
- https://github.com/NVlabs/GEM-X
- https://seed-viewer.bones.studio/
- https://huggingface.co/datasets/bones-studio/seed
- https://github.com/NVIDIA/soma-retargeter/
- https://github.com/NVlabs/ProtoMotions
- https://nvlabs.github.io/GEAR-SONIC/
Humans, Avatars, and Character
Avatar Generation: Creating Digital Beings from Scratch
- This section focuses on platforms and research enabling the generation of complete avatars, encompassing both visual representation and underlying technologies.
-
- REPLIKANT: An AI-assisted 3D avatar and animation platform designed for creators.
- Meta Research Paper: A research paper from Meta exploring an unspecified aspect of avatar generation.
- Heygen: A platform for generating and animating realistic avatars from text prompts and images.
- Synthesia: A leading platform for creating AI-powered videos featuring realistic avatars.
Face Generation: Crafting Realistic and Expressive Visages
- This section explores projects and techniques dedicated to the intricate art of generating digital faces, capturing nuances of expression and identity.
-
- Media2Face: A project focused on generating faces from various media inputs.
- Towards a Simultaneous and Granular Identity-Expression Control in Personalized Face Generation: Research on personalized face generation with fine-grained control over identity and expression.
- HeadStudio: A project utilizing 3D Gaussian Splatting to create animatable head avatars from text.
- Lip Sync: Achieving realistic lip synchronization is crucial for believable avatars. These resources delve into this challenge:
- VASA-1 (Microsoft Research): A project from Microsoft Research exploring lip synchronization techniques.
- wav2lips: An article exploring the use of wav2lip for deepfake audio and lip synchronization.
- Realtime lip-sync API: An API offering real-time lip synchronization capabilities.
Body Generation: Building Digital Bodies with Realistic Form
- This section examines projects that focus on generating realistic and diverse digital bodies, laying the foundation for virtual humans and characters.
-
- Volumetric Primitives (MVP): A research paper exploring the use of volumetric primitives for avatar representation.
- Gaussian Shell Maps: A research project dedicated to the use of Gaussian Shell Maps for efficient 3D human generation.
- En3D: A project focused on generating 3D human models with realistic detail.
- Character Creator 3 (Reallusion): A powerful tool for creating realistic and stylised 3D characters.
Hand Generation: Recreating the Complexity of Human Hands
-
The human hand, with its intricate movements and expressiveness, poses a unique challenge for digital recreation. This section highlights projects tackling this complexity.
- URHand: A project dedicated to the generation and animation of realistic digital hands.
Audio-Driven Avatar: Animating Avatars with the Power of Voice
-
This section explores projects that leverage audio input to drive avatar animation, creating a more immersive and responsive experience.
- audio2photoreal (Facebook Research): A project from Facebook Research focused on generating photorealistic avatars driven by audio.
Human Texture Estimation: Bringing Digital Skin to Life
-
Realistic skin texture is crucial for believable digital humans. This section explores projects and research dedicated to accurately estimating and recreating human skin textures.
- SMPLitex: A generative model and dataset for estimating 3D human texture from a single image.
- Human texture estimation from a single image: A broader exploration of the field of human texture estimation from single images.
Miscellaneous: Exploring the Wider Landscape of Digital Human Creation
-
This section encompasses a variety of projects and resources that contribute to the broader field of digital human and avatar creation.
- StableIdentity: A project exploring the insertion of individuals into new environments.
- What You See Is What You GAN: An NVIDIA research paper on rendering high-fidelity geometry in 3D GANs.
- Generating Consistent Characters (Twitter Thread): A Twitter thread by Manu.Vision discussing methods for generating consistent characters.
- Consistent Characters with Scenario (YouTube Tutorial): A YouTube tutorial on creating consistent characters within a specific scenario.
- MoMask (GitHub Repository): The official implementation of MoMask, a method for generative masked modelling of 3D human motions.
- Character Asset Creator (Reddit Post): A Reddit post discussing a Character Asset Creator for StableDiffusion.
- Mikugg Emotions Script: A script for creating emotions in Mikugg.
- Miku.gg Bot Emotions Documentation: Documentation on creating bot emotions within the Miku.gg platform.
- Consistent Character Maker Workflow (Civitai): A workflow for generating consistent characters using Stable Diffusion.
Character Consistency: Maintaining Identity Across Generations
- Achieving consistency in character appearance across multiple generations is a key challenge. This section highlights resources and techniques addressing this issue.
-
- Consistent Character Creation (OpenArt): A workflow for achieving full-character consistency using the SDXL model with a Lightning 8-step LoRA.
A Workflow for 3D Character Creation and Animation: A Step-by-Step Guide
-
This section outlines a comprehensive workflow for creating and animating 3D characters, leveraging a variety of tools and techniques.
- LinkedIn Post detailing the workflow: A LinkedIn post outlining the workflow in detail.
- Tools used in the workflow:
- Midjourney: For generating initial character concepts.
- Tripo3d.ai: For creating 3D models from images.
- AI Digimans’ PBR Texture Generation Tool: For generating realistic textures.
- Mixamo: For animating the 3D models.
- Spline.design: For further customisation and enhancement.
Animation: Breathing Life into Digital Characters
- Bringing digital characters to life requires compelling animation. This section explores projects and resources focused on achieving realistic and expressive character movement.
-
- Animatable Gaussians (GitHub Repository): Code for “Animatable Gaussians: Learning Pose-dependent Gaussian Maps for High-fidelity Human Avatar Modeling,” presented at CVPR 2024.
- 3D Gaussian Blendshapes: Exploration of the use of 3D Gaussian Blendshapes for head avatar animation.
- Eggnog: A platform for creating infinite AI videos.
- Efficient Portrait Animation (LivePortrait): A project focused on efficient portrait animation with stitching and retargeting control.
- Note by Daniele: A note on LivePortrait by Daniele.
- ComfyUI Nodes for LivePortrait (GitHub Repository): ComfyUI nodes designed for LivePortrait.
- CLARA2 (GitHub Repository): A 3D-rendered AI agent designed to present PowerPoint presentations.
- Consistent Character API (Replicate): An API for running fofr’s consistent character model.
- Joystick-Controlled Character Manipulation (Twitter): A concept for manipulating character features using a joystick.
- Volucap Authentic Digital Avatars: A company specialising in creating authentic digital avatars.
- Openpose Controlnets (Civitai Article): An article explaining how to use and generate poses with Openpose Controlnets V1.1.
- AI Modelling Agency (Creative Bloq Article): An article discussing the emergence of AI modelling agencies.
- Free VRChat Avatars & 3D Assets (VRCMods): A collection of free VRChat avatars and 3D assets.
- AniTalker: A project focused on animating talking heads.
- VLOGGER: A project related to creating virtual vloggers.
- StoryDiffusion: A project exploring consistent self-attention for long-range image and video generation.
- SoccerNet Game State Reconstruction (GitHub Repository): A project focusing on athlete tracking and identification on a minimap.
- Ukraine’s AI Avatar for Consular Affairs (Reddit Post): A discussion on Ukraine’s use of an AI avatar for consular updates.
- Animating Images with Viggle AI (YouTube Tutorial): A tutorial on animating images using Viggle AI.
- PhysAvatar (Hugging Face Paper): Research on learning the physics of dressed 3D avatars from visual observations.
- Vid2Avatar: A project focused on reconstructing 3D avatars from videos.
- Human Tracking and SLAM Capture (YouTube Video): A demonstration of human tracking and SLAM capture technology.
- ComfyUI Character Turntable with SV3D (Reddit Post): A discussion on creating character turntables using ComfyUI and SV3D.
- Animating Characters for Free (LinkedIn Post): A post highlighting methods for animating characters for free.
- Midjourney Character Reference Feature (Medium Article): An article exploring Midjourney’s Character Reference feature.
- Full-Character Consistency with SDXL (Reddit Post): A discussion on achieving full-character consistency using SDXL.
- Create Bot Emotions (Miku.gg Documentation): Documentation on creating bot emotions within the Miku.gg platform.
- Consistent Emotions on a Character with ComfyUI (Reddit User): A Reddit user’s plans to publish a method for achieving consistent emotions on a character using ComfyUI.
- BakedAvatar: A project focused on avatar creation.
- Dreamtalk (GitHub Repository): The official implementation of Dreamtalk, focusing on expressive talking head generation.
- CharTurnerBeta LoRA (Civitai): A LoRA for multi-direction consistency in Stable Diffusion character generation.
- VividTalk: A project focused on one-shot audio-driven talking head generation.
- Gaussian-Based Avatars (Hugging Face Papers):
- Relightable Gaussian Codec Avatars: Research on using Gaussian codecs for avatar representation.
- Gaussian Head Avatar: Research on creating high-fidelity head avatars using dynamic Gaussians.
- NLW Education Discord Projects (Discord Channel): A Discord channel discussing projects related to ElevenLabs and character/avatar creation.
- D-ID AI Video Mobile App: A mobile app for creating AI videos.
- GAIA (Microsoft): A project from Microsoft exploring advanced avatar technologies.
This meticulously curated collection offers a comprehensive overview of the dynamic field of digital human and avatar creation. Explore, learn, and contribute to the ongoing evolution of this exciting frontier!
Note: Some links may lead to projects or resources that are still under development or experimental. Remember to review any licensing information before using code or assets from these projects.
-