# Research Digest
Offline-first arXiv digest. Fetching, scoring, and summarization run in your browser — papers are summarized extractively (key sentences selected, not generated) by a small DistilBERT model on your own device. No API keys, no cloud, no account.
# Features
- Installable PWA — works offline after the first visit.
- Client-side everything — corpus, interests, and saves live in IndexedDB; inference runs via transformers.js.
- Direct from your browser — arXiv metadata via DataCite with an OpenAlex fallback (arXiv’s own API sends no CORS header), deduped against your corpus.
- Live search — query arXiv on demand.
- Saves shape future fetches — a deterministic, recency-decayed keyword loop gated behind a minimum-saves threshold. No ML.
- Seed corpus — an optional server-side pipeline keeps
seed-corpus.jsonfresh so a new install isn’t empty.
# Run
Container (tiny Go file server on scratch, non-root), published on version tags:
docker run -d -p 8080:8080 ghcr.io/usr-wwelsh/research-digest:latest
Runs fine with --read-only --cap-drop ALL too. Optionally bind-mount a seed-corpus.json at /srv/seed-corpus.json.
# Develop
git clone https://github.com/usr-wwelsh/research-digest.git && cd research-digest
./scripts/fetch_vendor_assets.sh # onnxruntime-web WASM
(cd app && npm ci && npm run build:vendor) # app/vendor/transformers.min.js
(cd server && ROOT=.. go run .) # http://localhost:8080
Tests:
python3 -m venv venv && ./venv/bin/pip install -r requirements-dev.txt
./venv/bin/python -m pytest
(cd app && npm test)
(cd server && go test ./...)
# Seed pipeline
./run.sh runs fetch → summarize → embed → relate → export_seed and writes seed-corpus.json. ./run.sh --offline skips the fetch. Needs a venv: python3 -m venv venv && ./venv/bin/pip install -r requirements.txt. Run it from cron to keep the seed fresh. The first run downloads ~270MB (DistilBERT) of model weights to .hf_cache/.
# Configuration
config.json— interests for the seed pipeline only (arXiv query + keywords per interest). Syntax: category taxonomy.- In-app Settings — your interests, stored on-device. Seeded once from
config.json, then yours to edit.
{
"interests": {
"Efficient ML / Edge AI": {
"query": "cat:cs.LG OR cat:cs.CV OR cat:cs.CL",
"keywords": ["efficient", "edge", "quantization", "distillation"]
}
},
"settings": { "papers_per_interest": 25, "recent_days": 7, "fetch_multiplier": 3 }
}
# Layout
app/ PWA: plain ES modules, no bundler (db, sources, models.worker, feedback, fetch-orchestrator)
server/ Go static file server used by the container
*.py seed pipeline (fetch, summarize, embed, relate, export_seed)
sw.js service worker (app shell cache; API data is network-only)
# Requirements
- Client: any evergreen browser (Service Workers, IndexedDB, Web Workers, WebAssembly).
- Seed pipeline: Python 3.9+,
requests,numpy,torch(CPU),transformers. - Dev: Node.js (vendor build, client tests), Go (server tests).
# License
MIT — see LICENSE.