Version: @tanstack/highlight 0.0.9
An apostrophe inside JSX text is treated as a string opener. Everything up to the next ' — including the element's closing tag — is consumed as one string token.
Repro
import { createHighlighter } from '@tanstack/highlight/core'
import { tsx } from '@tanstack/highlight/languages/tsx'
const h = createHighlighter({ languages: [tsx] })
console.log(h.tokenize(`<Description>We'll never share your email.</Description>`, { lang: 'tsx' }).tokens)Actual:
["<", ["Description","tag"], ">", ["We","type"], ["'ll never share your email.</Description>", "string"]]
The 'll never share your email.</Description> run is classed string, so the prose renders string-colored and the closing </Description> tag never gets its tag class.
Expected: JSX text content is inert — We'll never share your email. stays uncolored and </Description> tokenizes as a closing tag.
Context: found while migrating dotui.org's docs from shiki, character-diffing both engines over our full corpus (~733k chars of tsx). Contractions in UI copy (We'll, can't, don't) make this common in real docs.
Version:
@tanstack/highlight0.0.9An apostrophe inside JSX text is treated as a string opener. Everything up to the next
'— including the element's closing tag — is consumed as onestringtoken.Repro
Actual:
The
'll never share your email.</Description>run is classedstring, so the prose renders string-colored and the closing</Description>tag never gets itstagclass.Expected: JSX text content is inert —
We'll never share your email.stays uncolored and</Description>tokenizes as a closing tag.Context: found while migrating dotui.org's docs from shiki, character-diffing both engines over our full corpus (~733k chars of tsx). Contractions in UI copy (
We'll,can't,don't) make this common in real docs.