Encodes HTML, XML, Markdown, and formatting syntax.
What It Does
Symbol.Markup neurons encode structural formatting symbols that specify how content should be rendered or organized: HTML tags (`<div>`, `<strong>`), Markdown syntax (** for bold, # for headers), XML structures, and document formatting conventions. They fire on the structural container signals that separate document organization from semantic content.
How It Behaves
Despite being one of the smallest Symbol sub-types, Markup neurons show a disproportionately strong late-layer concentration. This is counterintuitive: formatting syntax looks like a surface-level token pattern, so you might expect early-layer firing. The late-layer concentration suggests that markup decisions are made after semantic content is determined — the model decides what to say before deciding how to format it. Markup neurons are particularly important in models deployed for document generation tasks.
Research Example
In Mistral 7B, Symbol.Markup neurons fire when the model encounters a user prompt formatted in Markdown and determine that the response should also be Markdown-formatted. They encode the stylistic context switch, not the content. When these neurons are suppressed, models generate semantically correct but structurally inconsistent responses — appropriate content, wrong format.