Can TF-IDF pick the right tags for a blog post?

No. On this blog's own 88 hand-tagged posts, TF-IDF's top five terms contained the human topic tag 10.0% of the time at best, and the textbook log-scaled variant managed 5.1%. Throwing IDF away entirely did better, at 11.8%. And 27.6% of the tags people chose never appear in the post text at all, so no keyword extractor could ever produce them.

The same TF-IDF vectors were good at a different job. A nearest-centroid classifier picked the post's category correctly for 74 of 88 posts, 84.1%, against a majority-class baseline of 28.4%. Automatic tagging from keywords fails; automatic filing into a handful of known categories works.

Hardware: Apple M3, 16 GB, macOS 26.4.1, Node v23.5.0. Everything is hand written in Node with no packages. The corpus is every post committed to this blog's repository at commit fa52ea2, with other agents benchmarking on the same machine at the time, so the timings are indicative and the accuracy numbers are exact. One thing about those labels: two days before this snapshot, topic tags used by only one post were removed from the whole blog, because Ghost turned each of them into a near-empty tag page. The 127 tags measured here are still the ones people chose, but they are the shared ones.

The short answer

• TF-IDF's top 5 terms recovered 10.0% of hand-picked topic tags on 88 real blog posts (stemmed match, raw term frequency, stopwords removed). The sublinear 1 + log(tf) weighting that most tutorials recommend scored 5.1%.
• IDF works against tagging. Plain term counts with no IDF recovered 11.8% at top 5 and 21.6% at top 10. A tag is by definition a word many posts share, and IDF is designed to push shared words down.
• 27.6% of human tags (35 of 127) never appear in their post's text, even after stemming. performance was absent from 14 of the 19 posts tagged with it. That caps any extractor at 72.4% recall (calculated).
• RAKE recovered no tag at all as an exact phrase in its top 10. It picks long, specific phrases such as "absolute percentages belong", which are not tags.
• Choosing among the existing tag list instead of extracting keywords recovered 52.0% of tags with two suggestions per post. The same TF-IDF vectors put posts in the right category 84.1% of the time.

How was this measured?

The labels already existed. Every post on this blog carries a tags: line written by hand: the first tag is the category, the rest are topics. That makes the blog a small labelled corpus with no judgement calls of mine in it.

Code blocks and tables were removed because a CMS plugin reading the post would see prose, and because code is full of identifiers that would win on term frequency for the wrong reasons. Text is lowercased, hyphens split words, and tokens of one character or only digits are dropped.

Each extractor returned its top k terms for k = 3, 5 and 10. A term counts as a hit under three matching rules, from strictest to loosest:

• exact — the term equals the tag (api-design becomes api design).
• stemmed — equal after a light suffix stripper, so webhooks matches webhook and caching matches cache.
• contains — the stemmed tag appears as a token run inside the term, so the bigram postgres queue hits both postgres and queue.

Precision@k is the share of the k terms that hit a tag. Recall@k is the share of the post's topic tags that were hit. Both are averaged over the 78 posts that have topic tags.

How often does TF-IDF pick the human tags?

Rarely, and the most-recommended settings did worst.

IDF is the smoothed form ln((1+N)/(1+df)) + 1. Precision stays under 5% everywhere because a post has one or two topic tags and the extractor returns five terms, so most of any top 5 is bound to miss. Recall is the column that matters.

Counted per tag rather than per post, plain term frequency put 30 of the 127 tags in a top 10 (23.6%). Log-scaled TF-IDF put 11 there (8.7%).

Why does IDF make tagging worse?

I expected IDF to help, since it is the part that separates TF-IDF from word counting. Measured, it was the part that hurt.

A tag exists to group posts. postgres, docker and http are useful tags precisely because they recur, and IDF penalises exactly that recurrence. Log-scaling term frequency makes it worse: once counts are flattened, IDF dominates the score and the top terms become the rarest words in the corpus. Log-scaled TF-IDF with bigrams returned o200k, cl100k and vietnamese for the post on token costs. For the Docker overhead post it returned fmod, syscall and seccomp, and neither docker nor performance made the list. Those are good keywords. They describe that one post well. But a tag page for seccomp would hold one article, the kind of thin page this blog removed tags to avoid.

Dropping IDF fails the other way. The terms that reached a top 5 most often under plain counting were ms (23 posts), index (12), bytes (11), and table, run and rows (8 each). These are the frequent technical words of a benchmarking blog, and none of them is a topic. That is the failure every "auto-tag" plugin shows: it returns the words the author used most, not the subject the author was writing about.

How many tags could no extractor ever find?

35 of 127, or 27.6%, counting a tag as present if its stemmed words appear anywhere in the body. Under exact spelling, 39 (30.7%).

These are the most editorial tags. A post that measures milliseconds for 1,500 words is about performance without once using the word. devops and i18n are words for a subject that nobody writes in the middle of a sentence. Human tags are a judgement about what the post is, and extraction can only return what the post says.

Removing those 35 from the denominator leaves 92 recoverable tags. Plain term frequency found 18.5% of them in a top 5 and 32.6% in a top 10; log TF-IDF found 6.5% in a top 5. The failure is not only the missing words.

Does RAKE or adding bigrams help?

RAKE did not help. It scores phrases by word co-occurrence, and on prose that favours long, unusual runs. Its top three for the search post were "absolute percentages belong", "brute force cosine" and "high pitched whine". Only under the loosest matching rule, where a tag word may sit inside a phrase, did it reach 3.2% recall at top 5.

Bigrams mostly hurt, and that was the result I could not predict. With raw TF-IDF and stemmed matching, adding bigrams took recall@10 from 15.0% to 13.7%. Phrases like skip locked and exact hashing crowd out the single words the tags actually are. Bigrams only helped the log-scaled variant under the contains rule (5.1% to 6.2%), because a bigram that contains a tag word gets a second chance to match. For the category classifier they changed nothing, 74 of 88 either way, and doubled the leave-one-out run time from 675 ms to 1,392 ms.

Can TF-IDF at least pick the category?

Yes, surprisingly well. Each post is an L2-normalised TF-IDF vector (log tf, stopwords removed). Each category's centroid is the sum of its posts' vectors, and a post goes to the centroid with the highest cosine. Under leave-one-out, the held-out post is removed from its centroid and IDF is refitted without it, 88 times.

Accuracy 84.1%, against 28.4% for always answering "Infrastructure". Keeping stopwords gave 83.0%; raw tf gave 84.1%.

11 of the 14 errors are Infrastructure and Integration swapped, and several are arguably the labels' fault rather than the model's. postgres-job-queue-skip-locked is filed under Infrastructure; how-fast-is-a-postgres-queue under Integration. The classifier sent each to the other's category. If two posts on the same Postgres queue can get different labels, the boundary is fuzzy for people too.

The other three errors went to Games: gzip-or-brotli-for-apis, structuredclone-vs-json and parsing-huge-json. All three are throughput benchmarks, and so are most of the Games posts here (object pools, broadphase collision, fixed timesteps). My reading, not something I measured, is that the vocabulary of timing a loop outweighs the vocabulary of the subject.

The category names themselves are useless as keywords. The category's own word appears in the body of 31 of 88 posts, and in the top 10 terms of none.

What should automatic tagging actually do?

Stop generating tags and choose from the ones you have. I ranked the existing tag vocabulary for each post by how often each tag's stemmed words occur in the body, building the vocabulary from the other posts only so a one-off tag cannot suggest itself.

Two suggestions recovered 52.0% of tags; five free TF-IDF terms recovered 10.0%. Every suggestion is also a tag page that already exists. It still needs a human. node was suggested 41 times and right 10; postgres was right 15 times out of 20. It cannot suggest performance in the posts that never say the word. Treat it as a pre-filled dropdown, not an answer.

The same pattern shows up in retrieval: exact word matching is strong on some questions and blind on others, as measured in Why does my search miss the obvious answer?. How you tokenise matters here too, and normalising text for search covers the stemming and folding decisions. For the other classic corpus job, spotting near-copies before they become two tags' worth of duplicate posts, see How do you find near-duplicate documents cheaply?.

How fast is it?

Speed is not the obstacle. Tokenising all 88 posts, building document frequencies over unigrams and bigrams (34,020 terms) and taking every post's top 10 took a median of 48.8 ms over 21 runs. Tagging one new post against those frequencies took a median of 0.518 ms over 440 runs. The complete leave-one-out category evaluation took 675 ms. These ran with other benchmarks on the same machine, so read them as orders of magnitude.

Check it yourself

Save as tagcheck.cjs and run node tagcheck.cjs path/to/posts. It needs no dependencies or network and ran in 1.02 s here. It reads any folder of markdown files with a tags: Category, topic, topic line, and prints the unrecoverable-tag share, precision and recall for three weightings, and the leave-one-out category result. On this blog's posts at commit fa52ea2 it reproduces the headline numbers exactly.

// tagcheck.cjs -- can TF-IDF pick a post's tags? Measured against the human tags.
// node tagcheck.cjs <folder-of-markdown-posts>     (no deps, no network)
// Each post needs frontmatter with `tags: Category, topic, topic`.
// Frontmatter, fenced code blocks and table rows are stripped before analysis.
'use strict';
const fs = require('node:fs'), path = require('node:path');
const dir = process.argv[2] || 'posts';

const STOP = new Set(`a about above after again against all almost also am an and any are aren as at be because been before being below between both but by can cannot could couldn did didn do does doesn doing don down during each either else even ever every few for from further get gets got had hadn has hasn have haven having he her here hers herself him himself his how however i if in into is isn it its itself just let lets like ll may me might more most much must my myself need needs no nor not now of off often on once one only or other others our ours ourselves out over own per quite rather re really s same say says see seen shall she should shouldn since so some something still such t than that the their theirs them themselves then there these they this those though through thus to too two under until up upon us use used uses using ve very via was wasn way we were weren what when where whether which while who whom why will with within without won would wouldn yet you your yours yourself yourselves first second new make makes made take takes many`.split(/\s+/));

function parse(file) {
  const raw = fs.readFileSync(file, 'utf8');
  const m = raw.match(/^---\n([\s\S]*?)\n---\n?/);
  const tags = ((m ? m[1] : '').match(/^tags:\s*(.*)$/m) || [, ''])[1].split(',').map((t) => t.trim()).filter(Boolean);
  const out = []; let inCode = false;
  for (const line of (m ? raw.slice(m[0].length) : raw).split('\n')) {
    if (/^\s*(```|~~~)/.test(line)) { inCode = !inCode; continue; }
    if (!inCode && !/^\s*\|/.test(line)) out.push(line);
  }
  const text = out.join('\n').replace(/<!--[\s\S]*?-->/g, ' ').replace(/!?\[([^\]]*)\]\([^)]*\)/g, '$1')
    .replace(/https?:\/\/\S+/g, ' ').replace(/`/g, '').toLowerCase();
  return { name: path.basename(file, '.md'), category: tags[0], topics: tags.slice(1), text };
}
const tokens = (s) => (s.replace(/-/g, ' ').match(/[a-z0-9]+/g) || []).filter((w) => w.length > 1 && !/^\d+$/.test(w));
function stem(w) { // light suffix stripper: caches/caching/cache -> cach
  if (w.length <= 3) return w;
  if (w.endsWith('ies') && w.length > 4) w = w.slice(0, -3) + 'y';
  else if (w.endsWith('sses')) w = w.slice(0, -2);
  else if (w.endsWith('s') && !/(ss|us|is)$/.test(w)) w = w.slice(0, -1);
  if (w.endsWith('ing') && w.length > 5) w = w.slice(0, -3);
  else if (w.endsWith('ed') && w.length > 4) w = w.slice(0, -2);
  if (w.endsWith('e') && w.length > 3) w = w.slice(0, -1);
  return w;
}
const counts = (a) => { const m = new Map(); for (const t of a) m.set(t, (m.get(t) || 0) + 1); return m; };
const pct = (x) => (100 * x).toFixed(1) + '%';

const docs = fs.readdirSync(dir).filter((f) => f.endsWith('.md') && f !== '_template.md').sort().map((f) => parse(path.join(dir, f)));
const N = docs.length;
const tc = docs.map((d) => counts(tokens(d.text).filter((w) => !STOP.has(w)))); // unigram term counts, stopwords removed
const df = new Map(); for (const c of tc) for (const t of c.keys()) df.set(t, (df.get(t) || 0) + 1);
const idf = (t, d = df, n = N) => Math.log((1 + n) / (1 + (d.get(t) || 0))) + 1;
const nTopics = docs.reduce((a, d) => a + d.topics.length, 0);
console.log(`${N} posts, ${nTopics} topic tags on ${docs.filter((d) => d.topics.length).length} posts`);

// 1. Upper bound: topic tags whose (stemmed) words never appear in the post body
let absent = 0;
for (const d of docs) {
  const ts = tokens(d.text).map(stem);
  for (const g of d.topics) {
    const gs = tokens(g.toLowerCase()).map(stem);
    let found = false; for (let j = 0; j + gs.length <= ts.length && !found; j++) found = gs.every((x, q) => ts[j + q] === x);
    if (!found) absent++;
  }
}
console.log(`tags that never appear in their post: ${absent}/${nTopics} = ${pct(absent / nTopics)}`);

// 2. Tag recovery: top-k terms, matched to topic tags after stemming; macro-averaged over tagged posts
const top = (c, weight, k) => [...c].map(([t, f]) => [t, weight(t, f)]).sort((a, b) => b[1] - a[1] || (a[0] < b[0] ? -1 : 1)).slice(0, k).map((x) => x[0]);
for (const [label, weight] of [['TF-IDF, raw tf', (t, f) => f * idf(t)], ['TF-IDF, log tf', (t, f) => (1 + Math.log(f)) * idf(t)], ['TF only, no IDF', (t, f) => f]]) {
  const row = [];
  for (const k of [3, 5, 10]) {
    let R = 0, P = 0, n = 0;
    docs.forEach((d, i) => {
      if (!d.topics.length) return;
      const terms = top(tc[i], weight, k).map(stem), tags = d.topics.map((g) => tokens(g.toLowerCase()).map(stem).join(' '));
      R += tags.filter((g) => terms.includes(g)).length / tags.length; P += terms.filter((t) => tags.includes(t)).length / terms.length; n++;
    });
    row.push(`P@${k} ${pct(P / n)}  R@${k} ${pct(R / n)}`);
  }
  console.log(`${label.padEnd(16)} ${row.join('   ')}`);
}

// 3. Category: nearest centroid on L2-normalised log-tf TF-IDF, leave-one-out (IDF refit per fold)
const cats = [...new Set(docs.map((d) => d.category))].sort();
const conf = Object.fromEntries(cats.map((a) => [a, Object.fromEntries(cats.map((b) => [b, 0]))]));
let correct = 0;
for (let h = 0; h < N; h++) {
  const vec = (c) => { const v = new Map(); let s = 0; for (const [t, f] of c) { const d = (df.get(t) || 0) - (tc[h].has(t) ? 1 : 0); if (d <= 0) continue; const w = (1 + Math.log(f)) * (Math.log(N / (1 + d)) + 1); v.set(t, w); s += w * w; } s = Math.sqrt(s) || 1; for (const [t, w] of v) v.set(t, w / s); return v; };
  const cent = Object.fromEntries(cats.map((a) => [a, new Map()]));
  for (let i = 0; i < N; i++) if (i !== h) for (const [t, w] of vec(tc[i])) cent[docs[i].category].set(t, (cent[docs[i].category].get(t) || 0) + w);
  const q = vec(tc[h]); let best = null, bs = -Infinity;
  for (const a of cats) { let dot = 0, nn = 0; for (const w of cent[a].values()) nn += w * w; for (const [t, w] of q) dot += w * (cent[a].get(t) || 0); if (dot / Math.sqrt(nn) > bs) { bs = dot / Math.sqrt(nn); best = a; } }
  conf[docs[h].category][best]++; if (best === docs[h].category) correct++;
}
const maj = Math.max(...cats.map((c) => docs.filter((d) => d.category === c).length));
console.log(`category, nearest centroid LOO: ${correct}/${N} = ${pct(correct / N)}   (majority-class baseline ${pct(maj / N)})`);
console.log('rows = human, cols = predicted: ' + cats.join(' | '));
for (const a of cats) console.log(`  ${a.padEnd(15)} ${cats.map((b) => String(conf[a][b]).padStart(3)).join(' ')}`);

Output on the 88 committed posts:

88 posts, 127 topic tags on 78 posts
tags that never appear in their post: 35/127 = 27.6%
TF-IDF, raw tf   P@3 3.4%  R@3 5.8%   P@5 3.6%  R@5 10.0%   P@10 2.9%  R@10 15.0%
TF-IDF, log tf   P@3 2.6%  R@3 5.1%   P@5 1.5%  R@5 5.1%   P@10 1.7%  R@10 8.8%
TF only, no IDF  P@3 4.3%  R@3 7.1%   P@5 4.4%  R@5 11.8%   P@10 4.0%  R@10 21.6%
category, nearest centroid LOO: 74/88 = 84.1%   (majority-class baseline 28.4%)
rows = human, cols = predicted: AI & Data | Games | Infrastructure | Integration
  AI & Data        22   1   0   0
  Games             0  15   0   0
  Infrastructure    0   2  17   6
  Integration       0   0   5  20

The script matches whole unigrams after stemming, so a two-word tag such as api-design can never match in it; that is the "stem" column of the first table. The RAKE, bigram and tag-vocabulary numbers come from a longer benchmark over the same corpus and the same tokeniser.