namumark

#what it is

namumark is the markup language of namu.wiki, the biggest wiki in korea. there's no spec for it. there's a help page, and there's whatever the site actually renders.

i wanted a library that turns namumark into a proper syntax tree, renders that tree back to html, and writes it back out as namumark. on top of it i built a small wiki: documents, revisions with diffs, per-document access rules, discussion threads and an editor with a live preview.

#two ways in, one tree out

the tricky part is that the library has two inputs. one is namumark text, which is what an editor types. the other is html as namu.wiki serves it, because that's what you actually get when you look at a real article. both have to end up as the same tree.

'''굵게''' [[나무위키|링크]][* 각주]

becomes a paragraph with a style node (bold), a link node (internal, target 나무위키, label 링크) and a footnote node, whichever side it came from.

the html side is where most of the work went. the markup is built for the browser, not for reading back:

• images are lazy-loaded. the src is an svg placeholder and the real url sits in data-src. the placeholder carries its own width and height, which belong to the svg and not the image, so early on images came out the wrong size.
• footnotes are a link at the reference and a list item at the bottom of the page. the importer does a first pass that collects every footnote body, then a second pass that turns each reference into a real footnote node. the rendered footnote list gets dropped, because the renderer makes it again.
• the table of contents can be nested inside other wrappers. a shallow check missed it and pages showed two tables of contents.
• class names are minified junk like bLnKoY3i. an early fallback carried them through, and 6 of 100 test articles leaked them into my output.
• dark mode colours live in a separate dark-style attribute. they became their own fields on the tree instead of a string i'd re-parse later.

on the text side, tables were the worst. in namumark, a cell that spans three rows still leaves empty filler cells |||| in the rows below. the parser has to track which positions are taken and drop the fillers, or every row after a rowspan shifts right.

#the rule i held to

the easy way out of every importer gap is to keep the raw html chunk in the tree and paste it into the page. i didn't allow that. the early versions had a "preserved element" fallback, and one of the last library changes replaced those fallbacks with typed nodes. if the importer doesn't understand something, that's a bug in the importer, not a reason to store a blob.

that rule made one thing harder. a fix can touch three layers at once: the importer, the renderer, and trees that are already stored in the database. a parser fix does nothing for a document imported yesterday. so some fixes also had a render-time repair for old trees. one example: links that pointed at absolute namu.wiki urls got rewritten to internal links when rendering, which fixed 8,711 stored documents without re-importing any of them. when stored trees really were wrong, i bumped an importer version number so they'd be fetched and parsed again.

#how i tested it

the test corpus is 100 random real namu.wiki articles, about 9.7 mb of html. the corpus test parses each one, renders it, and writes it back out as namumark. across those 100 pages the tree holds 12,379 paragraphs, 610 tables, 885 footnotes and 13,153 links. every page parses with zero diagnostics and zero raw-html nodes.

round-trip fidelity is less perfect, and i measured it while writing this. html to tree to namumark to tree keeps the heading structure in all 100 articles. comparing the namumark text before and after one more parse, about 90% of lines come back identical. where they differ, the first change is usually in a table cell with styled content or in an image link. no article comes back byte-identical yet.

#the wiki around it

the app is next.js with postgres. documents store the tree as json next to each revision, history shows a unified diff, and access control resolves per document, then per namespace, then a global default. there's a wysiwyg editor as well as a raw namumark tab. a background worker brought in articles from namu.wiki through the same html importer, saved images as compressed webp, and kept a link to the source page's history for attribution under its cc by-nc-sa licence.

#where it is now

the work happened in an 11-day burst in march, and it's been paused since. the library is in decent shape against real pages. what's left is the round trip: getting namumark out to come back byte-identical, starting with styled table cells.