# arXiv topic map — dataset

Version 2026.09.20 · format 5 · built 2026-09-20 · source dump 20 Sep 2026 02:08:33 UTC

Clusters of arXiv preprints by content similarity of title and abstract: sentence
embeddings, UMAP, HDBSCAN, then a local LLM names each cluster. Method and rationale:
`docs/adr/0007-clustering-method.md` in the ArxivNavigator repository.

## Files

| File | Contents |
|---|---|
| `map.json` | Cluster nodes (id, label, size, 2D centroid, top terms, `heat`) and edges above the similarity threshold. The map layer — everything a client needs to draw the map. |
| `clusters/<id>.json` | Papers of one cluster, newest first, each with its own 2D position, fetched lazily. |
| `timeseries.json` | Papers per cluster per month of first submission. |
| `assignments.csv.gz` | `paper_id,cluster` for every clustered paper. |
| `categories.json` | The arXiv category co-occurrence graph over the same papers: per category the paper count, monthly activity and newest papers, plus the edges between categories sharing papers. |

`heat` is how a cluster is publishing now against its own history: papers in the last
complete 12 months divided by what 12 months usually hold for it. 1 is business as usual,
2 is twice the usual rate, 0.5 is half. It is `null` where a cluster is too small or too
young to say — see the `heat` block for the exact parameters.

A paper may carry `new_claims` — claims it makes that no earlier paper in its cluster had
made, each with the abstract sentence it came from — plus `claims`, how many it made in
total, and `prior_papers`, how many earlier papers it was compared against. **A paper
without them is unscored, which is not the same as scored zero.** Novelty is computed only for the clusters the pipeline
was asked for, because extracting claims from the whole corpus is months of compute; the
`novelty` block lists which clusters were done and the parameters used.

`cluster: -1` means the paper sat in no dense region. A large noise share is expected
of density-based clustering and is reported honestly in `map.json`.

## Licence

This dataset is licensed **CC-BY-4.0**. The code that produced it is MIT.
It contains arXiv metadata (identifiers, titles, categories, dates) only — no full
text is redistributed.