-
When can transformers reason with abstract symbols?
Paper • 2310.09753 • Published • 3 -
In-Context Pretraining: Language Modeling Beyond Document Boundaries
Paper • 2310.10638 • Published • 30 -
Reward-Augmented Decoding: Efficient Controlled Text Generation With a Unidirectional Reward Model
Paper • 2310.09520 • Published • 11 -
Connecting Large Language Models with Evolutionary Algorithms Yields Powerful Prompt Optimizers
Paper • 2309.08532 • Published • 53
Lu Yu
VoladorLuYu
AI & ML interests
Neuro-Symbolic, Large Language Models, Graph Machine Learning
Recent Activity
updated
a collection
5 days ago
LLM+Self-Play RL
upvoted
a
paper
8 days ago
SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios
updated
a collection
8 days ago
AutoAgent
Organizations
None yet
AutoAgent
-
MusicAgent: An AI Agent for Music Understanding and Generation with Large Language Models
Paper • 2310.11954 • Published • 25 -
Training Chain-of-Thought via Latent-Variable Inference
Paper • 2312.02179 • Published • 10 -
Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception
Paper • 2401.16158 • Published • 20 -
A Human-Inspired Reading Agent with Gist Memory of Very Long Contexts
Paper • 2402.09727 • Published • 38
Foundation Machine Learning
Efficient LLM
-
Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads
Paper • 2401.10774 • Published • 59 -
APAR: LLMs Can Do Auto-Parallel Auto-Regressive Decoding
Paper • 2401.06761 • Published • 1 -
Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache
Paper • 2401.02669 • Published • 16 -
MambaByte: Token-free Selective State Space Model
Paper • 2401.13660 • Published • 60
Data-efficient LLMs
dataset pruning for advancing the capabilities of LLMs
-
Effective pruning of web-scale datasets based on complexity of concept clusters
Paper • 2401.04578 • Published -
How to Train Data-Efficient LLMs
Paper • 2402.09668 • Published • 42 -
A Survey on Data Selection for LLM Instruction Tuning
Paper • 2402.05123 • Published • 3 -
LESS: Selecting Influential Data for Targeted Instruction Tuning
Paper • 2402.04333 • Published • 3
LLM+Fusion
-
LLM Augmented LLMs: Expanding Capabilities through Composition
Paper • 2401.02412 • Published • 38 -
Generative Representational Instruction Tuning
Paper • 2402.09906 • Published • 54 -
Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes
Paper • 2305.02301 • Published • 5 -
Evolutionary Optimization of Model Merging Recipes
Paper • 2403.13187 • Published • 58
LLM Reports
-
Nemotron-4 15B Technical Report
Paper • 2402.16819 • Published • 46 -
InternLM2 Technical Report
Paper • 2403.17297 • Published • 34 -
Chinese Tiny LLM: Pretraining a Chinese-Centric Large Language Model
Paper • 2404.04167 • Published • 13 -
MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases
Paper • 2402.14905 • Published • 134
synthetic code generation
-
Design2Code: How Far Are We From Automating Front-End Engineering?
Paper • 2403.03163 • Published • 98 -
Wukong: Towards a Scaling Law for Large-Scale Recommendation
Paper • 2403.02545 • Published • 17 -
StarCoder: may the source be with you!
Paper • 2305.06161 • Published • 31 -
Exploring Parameter-Efficient Fine-Tuning Techniques for Code Generation with Large Language Models
Paper • 2308.10462 • Published • 2
Diffusion Models
-
ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment
Paper • 2403.05135 • Published • 45 -
Understanding Diffusion Objectives as the ELBO with Simple Data Augmentation
Paper • 2303.00848 • Published -
Scalable Diffusion Models with Transformers
Paper • 2212.09748 • Published • 18 -
High-Resolution Image Synthesis with Latent Diffusion Models
Paper • 2112.10752 • Published • 15
LLM+Architecture
-
MoA: Mixture-of-Attention for Subject-Context Disentanglement in Personalized Image Generation
Paper • 2404.11565 • Published • 15 -
Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models
Paper • 2406.06563 • Published • 20 -
DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence
Paper • 2406.11931 • Published • 67
LLM+Self-Play RL
-
Training Language Models to Self-Correct via Reinforcement Learning
Paper • 2409.12917 • Published • 140 -
Recursive Introspection: Teaching Language Model Agents How to Self-Improve
Paper • 2407.18219 • Published • 3 -
Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problems
Paper • 2408.16293 • Published • 27 -
Selective Self-Rehearsal: A Fine-Tuning Approach to Improve Generalization in Large Language Models
Paper • 2409.04787 • Published • 1
Generative Multiple Modality
-
Random Field Augmentations for Self-Supervised Representation Learning
Paper • 2311.03629 • Published • 9 -
TEAL: Tokenize and Embed ALL for Multi-modal Large Language Models
Paper • 2311.04589 • Published • 21 -
GENOME: GenerativE Neuro-symbOlic visual reasoning by growing and reusing ModulEs
Paper • 2311.04901 • Published • 9 -
Q-Instruct: Improving Low-level Visual Abilities for Multi-modality Foundation Models
Paper • 2311.06783 • Published • 28
Super Alignment
-
Trusted Source Alignment in Large Language Models
Paper • 2311.06697 • Published • 12 -
Diffusion Model Alignment Using Direct Preference Optimization
Paper • 2311.12908 • Published • 49 -
SuperHF: Supervised Iterative Learning from Human Feedback
Paper • 2310.16763 • Published • 1 -
Enhancing Diffusion Models with Text-Encoder Reinforcement Learning
Paper • 2311.15657 • Published • 2
Graph Foundation Multimodal Models
-
Multimodal Graph Learning for Generative Tasks
Paper • 2310.07478 • Published • 1 -
Rethinking Patch Dependence for Masked Autoencoders
Paper • 2401.14391 • Published • 26 -
Graph Mamba: Towards Learning on Graphs with State Space Models
Paper • 2402.08678 • Published • 17 -
GraphWiz: An Instruction-Following Language Model for Graph Problems
Paper • 2402.16029 • Published • 3
Symbolic LLM Reasoning
-
CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution
Paper • 2401.03065 • Published • 11 -
DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence
Paper • 2401.14196 • Published • 68 -
WaveCoder: Widespread And Versatile Enhanced Instruction Tuning with Refined Data Generation
Paper • 2312.14187 • Published • 49 -
On the Effectiveness of Large Language Models in Domain-Specific Code Generation
Paper • 2312.01639 • Published • 2
LLM4Recsys
Understanding LLM
-
A Language Model's Guide Through Latent Space
Paper • 2402.14433 • Published • 1 -
The Hidden Space of Transformer Language Adapters
Paper • 2402.13137 • Published -
Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Models
Paper • 2402.16438 • Published -
AtP*: An efficient and scalable method for localizing LLM behaviour to components
Paper • 2403.00745 • Published • 14
LLM+TextGen
Large language models for text generation
LLM+Math
-
MathScale: Scaling Instruction Tuning for Mathematical Reasoning
Paper • 2403.02884 • Published • 17 -
DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
Paper • 2402.03300 • Published • 138 -
Improving Small Language Models' Mathematical Reasoning via Mix Thoughts Distillation
Paper • 2401.11864 • Published • 2 -
Common 7B Language Models Already Possess Strong Math Capabilities
Paper • 2403.04706 • Published • 18
LLM+Diffusion
-
Aligning Diffusion Models by Optimizing Human Utility
Paper • 2404.04465 • Published • 15 -
ByteEdit: Boost, Comply and Accelerate Generative Image Editing
Paper • 2404.04860 • Published • 25 -
TokenCompose: Grounding Diffusion with Token-level Supervision
Paper • 2312.03626 • Published • 5 -
Adding Conditional Control to Text-to-Image Diffusion Models
Paper • 2302.05543 • Published • 58
Data Synthesis
-
Best Practices and Lessons Learned on Synthetic Data for Language Models
Paper • 2404.07503 • Published • 31 -
Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences
Paper • 2404.03715 • Published • 62 -
Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing
Paper • 2406.08464 • Published • 71 -
Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models
Paper • 2402.13064 • Published • 50
Research on LLM
-
When can transformers reason with abstract symbols?
Paper • 2310.09753 • Published • 3 -
In-Context Pretraining: Language Modeling Beyond Document Boundaries
Paper • 2310.10638 • Published • 30 -
Reward-Augmented Decoding: Efficient Controlled Text Generation With a Unidirectional Reward Model
Paper • 2310.09520 • Published • 11 -
Connecting Large Language Models with Evolutionary Algorithms Yields Powerful Prompt Optimizers
Paper • 2309.08532 • Published • 53
Generative Multiple Modality
-
Random Field Augmentations for Self-Supervised Representation Learning
Paper • 2311.03629 • Published • 9 -
TEAL: Tokenize and Embed ALL for Multi-modal Large Language Models
Paper • 2311.04589 • Published • 21 -
GENOME: GenerativE Neuro-symbOlic visual reasoning by growing and reusing ModulEs
Paper • 2311.04901 • Published • 9 -
Q-Instruct: Improving Low-level Visual Abilities for Multi-modality Foundation Models
Paper • 2311.06783 • Published • 28
AutoAgent
-
MusicAgent: An AI Agent for Music Understanding and Generation with Large Language Models
Paper • 2310.11954 • Published • 25 -
Training Chain-of-Thought via Latent-Variable Inference
Paper • 2312.02179 • Published • 10 -
Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception
Paper • 2401.16158 • Published • 20 -
A Human-Inspired Reading Agent with Gist Memory of Very Long Contexts
Paper • 2402.09727 • Published • 38
Super Alignment
-
Trusted Source Alignment in Large Language Models
Paper • 2311.06697 • Published • 12 -
Diffusion Model Alignment Using Direct Preference Optimization
Paper • 2311.12908 • Published • 49 -
SuperHF: Supervised Iterative Learning from Human Feedback
Paper • 2310.16763 • Published • 1 -
Enhancing Diffusion Models with Text-Encoder Reinforcement Learning
Paper • 2311.15657 • Published • 2
Foundation Machine Learning
Graph Foundation Multimodal Models
-
Multimodal Graph Learning for Generative Tasks
Paper • 2310.07478 • Published • 1 -
Rethinking Patch Dependence for Masked Autoencoders
Paper • 2401.14391 • Published • 26 -
Graph Mamba: Towards Learning on Graphs with State Space Models
Paper • 2402.08678 • Published • 17 -
GraphWiz: An Instruction-Following Language Model for Graph Problems
Paper • 2402.16029 • Published • 3
Efficient LLM
-
Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads
Paper • 2401.10774 • Published • 59 -
APAR: LLMs Can Do Auto-Parallel Auto-Regressive Decoding
Paper • 2401.06761 • Published • 1 -
Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache
Paper • 2401.02669 • Published • 16 -
MambaByte: Token-free Selective State Space Model
Paper • 2401.13660 • Published • 60
Symbolic LLM Reasoning
-
CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution
Paper • 2401.03065 • Published • 11 -
DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence
Paper • 2401.14196 • Published • 68 -
WaveCoder: Widespread And Versatile Enhanced Instruction Tuning with Refined Data Generation
Paper • 2312.14187 • Published • 49 -
On the Effectiveness of Large Language Models in Domain-Specific Code Generation
Paper • 2312.01639 • Published • 2
Data-efficient LLMs
dataset pruning for advancing the capabilities of LLMs
-
Effective pruning of web-scale datasets based on complexity of concept clusters
Paper • 2401.04578 • Published -
How to Train Data-Efficient LLMs
Paper • 2402.09668 • Published • 42 -
A Survey on Data Selection for LLM Instruction Tuning
Paper • 2402.05123 • Published • 3 -
LESS: Selecting Influential Data for Targeted Instruction Tuning
Paper • 2402.04333 • Published • 3
LLM4Recsys
LLM+Fusion
-
LLM Augmented LLMs: Expanding Capabilities through Composition
Paper • 2401.02412 • Published • 38 -
Generative Representational Instruction Tuning
Paper • 2402.09906 • Published • 54 -
Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes
Paper • 2305.02301 • Published • 5 -
Evolutionary Optimization of Model Merging Recipes
Paper • 2403.13187 • Published • 58
Understanding LLM
-
A Language Model's Guide Through Latent Space
Paper • 2402.14433 • Published • 1 -
The Hidden Space of Transformer Language Adapters
Paper • 2402.13137 • Published -
Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Models
Paper • 2402.16438 • Published -
AtP*: An efficient and scalable method for localizing LLM behaviour to components
Paper • 2403.00745 • Published • 14
LLM Reports
-
Nemotron-4 15B Technical Report
Paper • 2402.16819 • Published • 46 -
InternLM2 Technical Report
Paper • 2403.17297 • Published • 34 -
Chinese Tiny LLM: Pretraining a Chinese-Centric Large Language Model
Paper • 2404.04167 • Published • 13 -
MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases
Paper • 2402.14905 • Published • 134
LLM+TextGen
Large language models for text generation
synthetic code generation
-
Design2Code: How Far Are We From Automating Front-End Engineering?
Paper • 2403.03163 • Published • 98 -
Wukong: Towards a Scaling Law for Large-Scale Recommendation
Paper • 2403.02545 • Published • 17 -
StarCoder: may the source be with you!
Paper • 2305.06161 • Published • 31 -
Exploring Parameter-Efficient Fine-Tuning Techniques for Code Generation with Large Language Models
Paper • 2308.10462 • Published • 2
LLM+Math
-
MathScale: Scaling Instruction Tuning for Mathematical Reasoning
Paper • 2403.02884 • Published • 17 -
DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
Paper • 2402.03300 • Published • 138 -
Improving Small Language Models' Mathematical Reasoning via Mix Thoughts Distillation
Paper • 2401.11864 • Published • 2 -
Common 7B Language Models Already Possess Strong Math Capabilities
Paper • 2403.04706 • Published • 18
Diffusion Models
-
ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment
Paper • 2403.05135 • Published • 45 -
Understanding Diffusion Objectives as the ELBO with Simple Data Augmentation
Paper • 2303.00848 • Published -
Scalable Diffusion Models with Transformers
Paper • 2212.09748 • Published • 18 -
High-Resolution Image Synthesis with Latent Diffusion Models
Paper • 2112.10752 • Published • 15
LLM+Diffusion
-
Aligning Diffusion Models by Optimizing Human Utility
Paper • 2404.04465 • Published • 15 -
ByteEdit: Boost, Comply and Accelerate Generative Image Editing
Paper • 2404.04860 • Published • 25 -
TokenCompose: Grounding Diffusion with Token-level Supervision
Paper • 2312.03626 • Published • 5 -
Adding Conditional Control to Text-to-Image Diffusion Models
Paper • 2302.05543 • Published • 58
LLM+Architecture
-
MoA: Mixture-of-Attention for Subject-Context Disentanglement in Personalized Image Generation
Paper • 2404.11565 • Published • 15 -
Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models
Paper • 2406.06563 • Published • 20 -
DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence
Paper • 2406.11931 • Published • 67
Data Synthesis
-
Best Practices and Lessons Learned on Synthetic Data for Language Models
Paper • 2404.07503 • Published • 31 -
Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences
Paper • 2404.03715 • Published • 62 -
Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing
Paper • 2406.08464 • Published • 71 -
Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models
Paper • 2402.13064 • Published • 50
LLM+Self-Play RL
-
Training Language Models to Self-Correct via Reinforcement Learning
Paper • 2409.12917 • Published • 140 -
Recursive Introspection: Teaching Language Model Agents How to Self-Improve
Paper • 2407.18219 • Published • 3 -
Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problems
Paper • 2408.16293 • Published • 27 -
Selective Self-Rehearsal: A Fine-Tuning Approach to Improve Generalization in Large Language Models
Paper • 2409.04787 • Published • 1