A practitioner’s phased methodology for building generative AI applications, progressing from proof-of-concept (using the best available models and rapid prototyping tools) through stakeholder validation to a robust production build, with explicit guidance on iterative evaluation, legal risk deferral, and avoidance of premature fine-tuning.
Semantic Classification
Content
DO:
-
Pick the best model available: While GPT-4 is a leader for language tasks, for broader GenAI applications, consider the leading models in each category (e.g., vision, speech, etc.). Don’t reinvent the wheel.
-
Midjourney Text-to-Image Service v6 for images
-
Suno AI for Music and Audio.
-
OpenAI Whisper for speech.
-
Build a Custom GPT on Test Playground: Beyond LLMs, experiment with customizable versions of leading models in other domains, like custom vision models on platforms like Azure, AWS, or Google Cloud.
-
Use public data or generate synthetic with LLMs: This extends to other AI types as well. Use or generate synthetic data relevant to the task—images for vision AI, sound for audio AI, etc. DON’T DO:
-
Experiment with lower performant models: This remains a standard guideline across all AI types. Always start with the best available technology to understand the potential ceiling of your application.
-
Build a polished custom app: Stay lean and focus on the core functionality of your AI application, whether it’s LLM, computer vision, or any other AI technology.
-
Fine-tune a model: In early stages, it’s more about understanding capabilities and limitations broadly. Specific tuning can come later and might involve more domain-specific models. You can start looking into tuning modules like LoRA DoRA etc and qLoRA if you understand this stuff well enough. DO:
-
**Build a simple app (e.g., Streamlit, or Vercel v0: This applies to all AI applications. Use tools that allow rapid prototyping and sharing with stakeholders, whether for LLMs, computer vision apps, or others.
-
Experiment with new user experiences: Regardless of the AI technology, consider how it changes or enhances the user experience. This might involve interactive elements, novel data visualizations, or automating previously manual tasks.
-
Develop strong product evaluation & testing: This is critical across all AI domains to ensure the application is reliable, ethical, and effective.
-
Consult legal experts You will almost certainly need to get your project signed off by a specialist AI lawyer at some point, because this defers the risk. It’s expensive. Make sure you have excellent records of everything you have done.
-
DON’T DO:
-
Build-out a full featured & integrated app: Keep the proof of concept focused and manageable, whether you’re working with natural language understanding, image recognition, or any other AI capability.
-
Spend too much time on re-usable assets: Stay agile and ready to pivot or adapt based on feedback and findings.
-
Ignore LLM risks (e.g., prompt injection, hallucinations): Similarly, be aware of and mitigate risks specific to other types of AI, such as adversarial attacks in computer vision or privacy concerns in voice AI. DO:
-
Iterate through implementation techniques: As with LLMs, try different architectures, data sets, or integration methods relevant to the specific AI type.
-
Try a cheaper model and possibly fine-tuning: Once the base functionality is proven, optimize for cost and efficiency, which may include moving to smaller, more specialized models or fine-tuning. DON’T DO:
-
Get stuck with the first implementation attempt: Be prepared to iterate and evolve as you learn more about the AI’s performance and the users’ needs.
-
Forget about data quality (incl. for RAG): High-quality, diverse, and relevant data is crucial for training any AI model effectively.
-
Adaptability: Different AI fields evolve at different rates. Stay updated with the latest in each specific domain.
-
Interdisciplinary Integration: Combining AI types (e.g., LLMs for chatbots with voice recognition) can create more sophisticated solutions.
-
Ethical and Responsible AI: Ensure ethical considerations and responsible use are central, especially as models impact different domains differently.
-
Scalability and Infrastructure: Different AI models have varying demands on infrastructure. Plan scalability from the start.
Key Question: Can Gen AI help solve my use case?