Code Generation

Code generation is one of the most mature and impactful applications of large language models. The ability to generate syntactically and semantically correct code from natural language descriptions, to complete partially written functions, to translate code between programming languages, and to explain or debug existing code represents a genuine productivity transformation for software developers.

The foundation is straightforward: code is text, and large language models trained on text learn to model code as a special case. GitHub reports that approximately 35-40% of public repositories are code, meaning a significant fraction of any large language model's training data is code in various programming languages. Models like Codex (the model behind GitHub Copilot), Code Llama, DeepSeek Coder, StarCoder, and Gemini Code demonstrate that models trained on code data develop sophisticated code synthesis capabilities - understanding APIs, common patterns, debugging strategies, and even test writing.

Fill-in-the-Middle (FIM) training extends standard next-token prediction for code: the model is trained to predict the middle of a code snippet given both the prefix (code before the cursor) and suffix (code after the cursor). This enables IDE-style code completion that fills gaps in partially-written functions, constrained by the surrounding context - a more useful capability than pure left-to-right generation for interactive use.

Instruction-tuned code models (trained on pairs of natural language descriptions and their code implementations) achieve strong performance on benchmarks like HumanEval (Python function generation from docstrings) and MBPP (mostly basic programming problems). State-of-the-art models correctly solve 70-90% of HumanEval problems, though benchmark contamination concerns affect some evaluations. Harder benchmarks (SWE-Bench, which requires resolving real GitHub issues in complex codebases) show larger gaps, with top systems resolving 25-50% of issues.

Agentic code generation extends beyond single function synthesis. OpenAI's Devin, Anthropic's Claude Code, and similar systems use tool-calling architectures that allow the model to read files, write files, run tests, execute code, and iterate based on results - performing multi-step software engineering tasks rather than single-shot code generation. These agentic systems can scaffold new projects, implement multi-file features, and resolve bugs in large codebases with minimal human intervention.

Key limitations: code generation models hallucinate non-existent API endpoints, introduce subtle security vulnerabilities (insecure cryptography, SQL injection, unchecked user input), produce code that works on their training data distribution but fails for edge cases, and struggle with novel or proprietary APIs not in their training data.