Useful papers
F o u n d a t i o n a l C o n c e p t s
1. The Annotated Transformer (2017): This paper explains the Transformer architecture, a groundbreaking model that revolutionized natural language processing (NLP). It introduced the attention mechanism, allowing the model to focus on relevant parts of the input sequence, leading to significant improvements in machine translation and other NLP tasks.
2. The First Law of Complexodynamics (2008): This paper introduces a framework for understanding complex systems, arguing that complexity increases initially but eventually reaches a peak and then decreases. This framework is relevant for understanding the evolution of AI models and the challenges of managing their complexity.
3. The Unreasonable Effectiveness of RNNs (2015): This paper explores the surprising success of Recurrent Neural Networks (RNNs) in various tasks, highlighting their ability to model sequential data and their role in natural language understanding and generation.
4. Understanding LSTM Networks (2015): This paper delves into Long Short-Term Memory (LSTM) networks, a specific type of RNN that excels at capturing long-range dependencies in sequences. It explains the internal mechanisms of LSTMs and their key advantages.
5. Recurrent Neural Network Regularization (2014): This paper focuses on techniques for preventing overfitting in RNNs, addressing a critical issue in deep learning. It explores various regularization methods, including dropout and weight decay, which enhance the generalizability of RNN models.
6. Keeping Neural Networks Simple by Minimizing the Description Length of the Weights (2014): This paper proposes a novel approach to model simplification by minimizing the description length of the weights. It explores the connection between model complexity and generalization performance, offering valuable insights for building more efficient and robust neural networks.
7. Pointer Networks (2015): This paper introduces Pointer Networks, a neural architecture specifically designed for tasks involving selecting elements from a given input set. These networks are particularly useful in tasks like machine reading comprehension and code generation.
8. ImageNet Classification with Deep CNNs (2012): This seminal paper describes AlexNet, one of the first successful deep convolutional neural networks (CNNs) for image classification. It demonstrated the power of deep learning for computer vision, marking a major turning point in the field.
9. Order Matters: Sequence to Sequence for Sets (2016): This paper explores the application of sequence-to-sequence (seq2seq) models for handling sets of data, where the order of elements doesn’t matter. It proposes strategies for incorporating set information into seq2seq models, enhancing their capabilities for various tasks.
10. GPipe: Efficient Training of Giant Neural Networks (2018): This paper introduces GPipe, a distributed training framework for scaling up neural network training on multiple GPUs. It allows researchers to train larger models more efficiently, pushing the boundaries of deep learning capabilities.
11. Deep Residual Learning for Image Recognition (2015): This paper presents the ResNet architecture, which uses residual connections to facilitate the training of very deep networks. It demonstrated the effectiveness of residual learning for overcoming the vanishing gradient problem, leading to significant improvements in image recognition performance.
12. Multi-Scale Context Aggregation by Dilated Convolutions (2015): This paper introduces dilated convolutions, a technique for expanding the receptive field of CNNs without increasing the number of parameters. It enables the network to capture information from a larger area, improving its ability to understand complex visual patterns.
13. Neural Quantum Chemistry (2017): This paper explores the application of neural networks in quantum chemistry, a field traditionally dominated by computationally expensive methods. It shows how neural networks can be used to predict molecular properties, potentially revolutionizing the field.
14. Attention Is All You Need (2017): This paper further develops the Transformer architecture, emphasizing the importance of the attention mechanism. It proposes a model that solely relies on attention, achieving state-of-the-art results in machine translation, demonstrating the power and flexibility of attention.
15. Neural Machine Translation by Jointly Learning to Align and Translate (2014): This paper describes the original seq2seq model for machine translation, introducing the concept of jointly learning alignment and translation. It set the stage for the development of more sophisticated models like the Transformer.
16. Identity Mappings in Deep Residual Networks (2016): This paper explores the effectiveness of identity mappings in ResNet, demonstrating their role in improving the performance of deep networks. It highlights the importance of understanding the role of different architectural components.
17. A Simple NN Module for Relational Reasoning (2017): This paper introduces a simple neural network module for relational reasoning, enabling the model to reason about relationships between entities. It opens up new possibilities for tackling complex reasoning tasks in NLP and other domains.
18. Variational Lossy Autoencoder (2013): This paper proposes a variational approach to building autoencoders, allowing for a more efficient and robust representation of data. It explores the connection between variational inference and deep learning, offering new insights into the theoretical foundations of these models.
19. Relational RNNs (2015): This paper explores the application of RNNs to relational data, where relationships between entities play a crucial role. It presents a novel architecture for relational RNNs, enabling them to effectively handle data with complex relationships.
20. Quantifying the Rise and Fall of Complexity in Closed Systems (2008): This paper expands on the “First Law of Complexodynamics,” providing a mathematical framework for understanding the emergence and decline of complexity in closed systems. It offers valuable insights into the dynamics of complex systems, including AI systems.
21. Neural Turing Machines (2014): This paper introduces the concept of Neural Turing Machines (NTMs), which combine neural networks with external memory modules, allowing them to store and access information like a traditional Turing machine. NTMs represent a step towards more powerful AI models with increased memory capacity and reasoning abilities.
22. Deep Speech 2: End-to-End Speech Recognition in English and Mandarin (2015): This paper presents Deep Speech 2, a deep learning-based speech recognition system that achieves state-of-the-art performance. It demonstrates the power of deep learning for tackling complex tasks like speech recognition, opening up new possibilities for human-computer interaction.
23. Scaling Laws for Neural Language Models (2020): This paper investigates the relationship between model size, training data size, and performance in large language models (LLMs). It reveals power law scaling laws, suggesting that increasing model size and data can lead to dramatic performance improvements.
24. A Tutorial Introduction to the Minimum Description Length Principle (2002): This paper provides a thorough introduction to the Minimum Description Length (MDL) principle, a fundamental principle in information theory and model selection. It explains how MDL can be used to select the best model from a set of candidates, balancing model complexity and predictive accuracy.
25. Machine Super Intelligence Dissertation (2014): This dissertation explores the potential for and dangers of developing superintelligent AI, highlighting the importance of aligning AI goals with human values. It presents a framework for understanding and managing the risks associated with advanced AI.
26. PAGE 434 onwards: Komogrov Complexity (2009): This document introduces Kolmogorov complexity, a measure of the complexity of an object. It discusses the limitations of computable complexity and the potential for using it to understand the complexity of AI systems.
27. CS231n Convolutional Neural Networks for Visual Recognition (2016): This Stanford course provides a comprehensive introduction to convolutional neural networks (CNNs), covering their architecture, training, and applications in computer vision. It’s a great resource for anyone wanting to learn more about this foundational topic in deep learning.
28. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? (2021): This paper raises critical concerns about the potential dangers of large language models (LLMs), emphasizing the need for ethical considerations and responsible development. It highlights the risks of bias, misinformation, and misuse of these powerful models.
Key LLM Papers
GPT-1 (2018): This paper introduces the first version of Generative Pre-trained Transformer (GPT), a generative model trained on a massive dataset of text. It demonstrates the ability of LLMs to generate coherent and grammatically correct text, paving the way for future advancements.
GPT-2 (2019): This paper presents a significantly larger GPT model with improved capabilities. It showcases the ability of LLMs to perform various language tasks, including text summarization, question answering, and even code generation.
GPT-3 (2020): This paper introduces GPT-3, a truly massive LLM with billions of parameters. It demonstrates impressive capabilities in diverse tasks, showcasing the emergence of general-purpose language abilities.
GPT-4 (2023): This paper introduces the latest iteration of GPT, featuring multi-modal capabilities and advanced reasoning abilities. It further pushes the boundaries of what LLMs can achieve, demonstrating impressive performance in a wide range of tasks.
Llama-2 (2023): This paper introduces Llama-2, a large language model designed with a focus on efficiency and accessibility. It offers a more resource-friendly alternative to other LLMs, making it more accessible for research and development.
Tools (2023): This paper introduces the “Tools” paradigm for LLMs, allowing them to interact with external tools and resources. It enables LLMs to perform more complex tasks by leveraging the power of external tools, expanding their capabilities significantly.
Gemini-Pro-1.5 (2023): This paper introduces Gemini-Pro-1.5, a large language model developed by Google. It showcases impressive capabilities in various tasks, including code generation, creative writing, and reasoning. It’s a strong contender in the race for developing advanced LLMs.
Ng’s Agentic Patterns Series
This series of papers explores the concept of agentic patterns in AI, focusing on the development of agents capable of independent learning and goal-directed behavior. It delves into the principles and design of such agents, offering valuable insights into the future of AI and its potential impact on society.
- Building recommenders, recommenders-team/recommenders: Best Practices on Recommendation Systems (github.com)
- ML Blocks | Home is machine learning vision lego