# How to add evidence

This repository is a **forensic evidence archive**. Every item must be
timestamped, reproducible, and integrity-checked so it can be cited and
independently verified even if the original source later changes or disappears.
This guide explains **where** evidence lives and **how** to add a new piece
without breaking those guarantees.

> **Golden rule:** never edit captured bytes after the fact. If a page changed,
> capture it **again** into a new timestamped folder — don't overwrite history.

---

## 1. The two-repo model

| | Public repo — **`GraafG/tt-festival`** (this one) | Private repo — **`GraafG/tt-festival-app-evidence`** |
|---|---|---|
| Holds | Lightweight, readable artifacts: raw HTML, response headers, rendered DOM, screenshots, logs, READMEs, checksums, the email threads, the analysis in `docs/` | The heavy browsable offline **`wget` mirrors** (full-site clones, often 100s of MB – GB) |
| Why | Stays small and cloneable; safe to publish | Avoids permanently bloating the public history and re-hosting large third-party sites publicly |
| Link between them | Each public snapshot ships a `mirror-manifest.sha256` | Each private snapshot ships an identical `SHA256SUMS.txt` |

The two checksum lists are **byte-for-byte identical**, so the public repo
independently attests to exactly what the private mirror contains. A new capture
usually touches **both** repos.

### Why a private repo and not Git LFS?

Git LFS was considered for the large mirrors and deliberately **not** used:

- **It still re-publishes the bytes.** LFS pointers in a *public* repo would push
  100s of MB – GB of cloned third-party site content to the world. A private repo
  keeps the mirrors access-controlled while the public repo keeps only the
  readable, checksummed artifacts.
- **It re-bloats the thing we wanted to keep light.** LFS objects live in a
  GitHub-side store and count against a shared storage/bandwidth quota; a normal
  `git clone` of the public repo would pull them by default, defeating the goal of
  a small, fast-to-clone public record.
- **Plain files are better evidence.** Our integrity guarantee is a
  human-auditable, per-file SHA-256 list (`mirror-manifest.sha256` ⇄
  `SHA256SUMS.txt`) that anyone can re-run with `Get-FileHash` / `sha256sum` — no
  LFS client, smudge filter or special checkout needed to verify or open a file.
- **No single file exceeds GitHub's 100 MB hard limit** (largest mirror file is
  ~12.8 MB), so LFS was never *required* — only the repo *total* is large, which a
  private repo handles fine.

---

## 2. Folder naming (used everywhere)

```
<host>_<YYYY-MM-DD_HHmmss>      ← local capture time, e.g. assen.nl_2026-06-26_231242
```

Use the bare host (`assen.nl`, `appmiral.com`, `ttfestival.nl`). Re-capturing the
same site **always** creates a new timestamped folder — never overwrite an
existing one.

---

## 3. Pick the kind of evidence

| You have… | Put it in… | Jump to |
|---|---|---|
| One or more **web pages** to freeze | `evidence/<host>_<timestamp>/` | [§4](#4-adding-a-web-page-capture) |
| **Email** correspondence (`.eml`) | `evidence/email-correspondence/NN_<slug>/` | [§5](#5-adding-email-correspondence) |
| **Analysis / write-up** (your own prose, decompiled findings) | `docs/` + link from `index.md` / `README.md` | [§6](#6-adding-an-analysis-document) |

---

## 4. Adding a web-page capture

### 4.1 Layout to produce

```
evidence/<host>_<YYYY-MM-DD_HHmmss>/
├── README.md                 ← capture log + "Reason for capture" (see §4.4)
├── SHA256SUMS.txt            ← generated last (see §7)
├── mirror-manifest.sha256    ← SHA-256 of every file in the private mirror (see §4.3)
├── .capture-timestamp        ← machine-readable local timestamp
├── raw/                      ← EXACT server responses (curl) + headers
│   └── <page>.html  +  <page>.headers.txt
├── rendered-dom/             ← JS-rendered DOM after hydration (Chrome)
│   └── <page>.dom.html
├── screenshots/              ← PNG @1440×900: full page + viewport
│   └── NN_<page>_fullpage.png  +  NN_<page>_viewport.png
├── meta/                     ← ONLY for full-site mirrors: sitemap(s), URL list, coverage report
└── logs/
    └── tool-versions.txt     ← exact tool versions + the User-Agent used
```

`mirror/` is **not** committed here — it goes to the private repo (§4.3). For a
single-page capture you may legitimately have no mirror; in that case skip
`mirror-manifest.sha256` and say so in the README.

### 4.2 Capture commands

Use **one consistent desktop User-Agent** across every tool so all artifacts
reflect the same rendering path:

```
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36
```

**Raw response + headers (curl)** — the verbatim, unmodified bytes:

```powershell
$UA = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
curl.exe -s -A $UA -D raw/home.headers.txt -o raw/home.html "https://example.com/"
```

**Rendered DOM + screenshots (Chrome / Chrome DevTools)** — at a 1440×900
desktop viewport, save both the full-page and viewport PNG, and dump
`'<!DOCTYPE html>\n' + document.documentElement.outerHTML` to
`rendered-dom/<page>.dom.html`. **Always open each PNG to confirm it is not
blank** before trusting it.

**Browsable offline mirror (GNU Wget)** — written into the **private** repo:

```powershell
# single page + its assets
wget --page-requisites --convert-links --adjust-extension `
     --restrict-file-names=windows -e robots=off --user-agent=$UA `
     "https://example.com/" -P <private>/<host>_<timestamp>/mirror

# whole site, seeded from a sitemap
wget -r -l inf -p -k -E -e robots=off --domains=example.com --no-check-certificate `
     -U $UA -i <host>_<timestamp>/meta/sitemap-urls.txt "https://example.com/" `
     -P <host>_<timestamp>/mirror -o <host>_<timestamp>/wget-mirror.log
```

> **Wget gotchas (learned the hard way):**
> - The server may send **no `Last-Modified`**, so `-N` can't resume; re-running
>   over an existing mirror creates `.1`/`.2` duplicate files. **Delete the
>   `mirror/` dir** (`Remove-Item -Recurse`) before any fresh run.
> - Link-conversion (`-k`) only runs **after all downloads finish**. An
>   interrupted mirror keeps absolute `https://…` links — acceptable for
>   byte-exact evidence, but note it in the README.
> - For full-site captures, verify completeness against the sitemap and write a
>   coverage report into `meta/` (URL → local file, URL-decoding `%xx`).

Record the exact versions you used in `logs/tool-versions.txt`:

```powershell
"OS    : $((Get-CimInstance Win32_OperatingSystem).Caption)"  | Out-File -Encoding utf8 logs/tool-versions.txt
"Wget  : $(wget --version | Select-Object -First 1)"          | Add-Content logs/tool-versions.txt
"curl  : $(curl.exe --version | Select-Object -First 1)"      | Add-Content logs/tool-versions.txt
"UA    : $UA"                                                 | Add-Content logs/tool-versions.txt
```

Write the timestamp marker:

```powershell
(Get-Date -Format "yyyy-MM-ddTHH:mm:sszzz") | Out-File -Encoding ascii .capture-timestamp
```

### 4.3 Move the mirror to the private repo

1. Put the `wget` mirror under `<private-repo>/<host>_<timestamp>/mirror/` (the
   **same folder name** as the public snapshot).
2. Generate the mirror manifest and place a copy in **both** repos:

   ```powershell
   # from the private snapshot folder, hash every mirrored file
   $base = (Get-Location).Path
   Get-ChildItem -Recurse -File mirror | ForEach-Object {
     $rel = $_.FullName.Substring($base.Length+1).Replace('\','/')
     "$((Get-FileHash -LiteralPath $_.FullName -Algorithm SHA256).Hash.ToLower())  $rel"
   } | Sort-Object | Set-Content -Encoding utf8 mirror-manifest.sha256
   ```

   Copy that file to the public snapshot as `mirror-manifest.sha256`, and also
   generate the private folder's own `SHA256SUMS.txt` (see §7) — the two lists
   must match.
3. Commit & push the private repo (see §8). Pushes can be large but no single
   file may exceed GitHub's 100 MB hard limit — split or document anything bigger.

### 4.4 Write the snapshot `README.md`

Mirror the structure of the existing snapshots (see
[`assen.nl_2026-06-26_231242/README.md`](assen.nl_2026-06-26_231242/README.md)).
It must contain:

- **Reason for capture** — *why this page is evidence*, with the decisive quote(s)
  verbatim and a pointer to the screenshot/DOM that proves it.
- **Snapshot summary** table — local + UTC time, capture host, server `Date`
  header, platform observed, pages, total size, User-Agent.
- **Pages captured** table — URL, title, HTTP status, byte size.
- **Folder layout**, **how it was captured** (commands + tool versions), and
  **how to verify / reproduce**.

---

## 5. Adding email correspondence

Email lives under `evidence/email-correspondence/`, one folder per **thread**:

```
email-correspondence/
├── README.md                 ← timeline + "why this is evidence" (update it!)
├── SHA256SUMS.txt            ← regenerate after adding (see §7)
└── NN_<party>_<topic>/       ← e.g. 02_gemeente-assen_privacy-zorgen
    ├── raw/<original>.eml     ← byte-exact source (PRIMARY evidence — never edit)
    ├── NN_<slug>.txt          ← readable rendering: headers + body
    └── NN_attachments/        ← extracted attachments (if any)
```

Conventions:

- **`raw/*.eml` is the primary record** — copy it in byte-for-byte; never modify
  it. Confirm it is byte-identical to your source (`Get-FileHash`).
- Number messages **chronologically** by the `Date` header (`01_`, `02_`, …).
- The `.txt` rendering includes From / To / Cc / Date / Subject / Message-ID /
  In-Reply-To / Attachments / Source-File, a separator, then the plain-text body
  (fall back to stripped HTML).
- Update `email-correspondence/README.md`: add the message(s) to the timeline
  table and, if a new thread, a thread description + a "why this is evidence"
  point.

---

## 6. Adding an analysis document

Your own write-ups (decompilation notes, SDK audits, policy analysis) go in
[`docs/`](../docs/) as Markdown and are linked from
[`index.md`](../index.md) (the report) and [`README.md`](../README.md). These are
*analysis*, not captured bytes, so they don't need a SHA256SUMS entry — but cite
the evidence folder that backs each claim.

---

## 7. Integrity: the `SHA256SUMS.txt` manifest

Every evidence folder (each snapshot **and** `email-correspondence/`) ships a
`SHA256SUMS.txt` listing the SHA-256 of every file **except** the folder's own
`README.md`, `SHA256SUMS.txt` and `mirror-manifest.sha256`.

Generate (or refresh) it with the helper script — it applies the exclusions,
sorts, and writes LF / UTF-8 / forward-slash paths so verification is identical
on every OS:

```powershell
pwsh tools/gen-sha256sums.ps1 -Path evidence/<host>_<timestamp>
pwsh tools/gen-sha256sums.ps1 -Path evidence/email-correspondence
```

Verify any folder:

```bash
# Linux / macOS (run inside the folder)
sha256sum -c SHA256SUMS.txt
```

```powershell
# Windows PowerShell
Get-Content SHA256SUMS.txt | ForEach-Object {
  $h,$p = $_ -split '  ',2
  if ((Get-FileHash -Algorithm SHA256 -LiteralPath $p).Hash.ToLower() -ne $h) { "MISMATCH: $p" }
}
```

> **Why this stays valid:** [`evidence/.gitattributes`](.gitattributes) sets
> `* -text -diff`, so git stores and checks out the **exact bytes** (no
> line-ending or encoding normalization) on every platform. Don't remove that
> rule, and don't add captured files that you intend to let git "fix".

**Always regenerate `SHA256SUMS.txt` last**, after every other file in the folder
is final.

---

## 8. Wire it into the indexes, then commit

1. **`evidence/README.md`** — add a row to the *Snapshots* (or *Correspondence*)
   table.
2. **`evidence/index.html`** — add a `<section>` / card for the new snapshot,
   following an existing one. Keep the section numbers and the table-of-contents
   links in sync. Confirm every `href` resolves on disk.
3. **Commit** with a descriptive message and the project trailer:

   ```
   Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
   ```

4. **Push** the public branch (this updates the open evidence PR) and, if you
   added a mirror, push the **private** repo too.

---

## 9. Pre-commit checklist

- [ ] Folder named `<host>_<YYYY-MM-DD_HHmmss>` (new, not overwriting).
- [ ] `raw/` has exact responses **and** `.headers.txt` for each page.
- [ ] `rendered-dom/` + `screenshots/` present; **screenshots opened and verified non-blank**.
- [ ] `logs/tool-versions.txt` records tools **and** the User-Agent.
- [ ] `.capture-timestamp` written.
- [ ] Heavy `wget` mirror moved to the **private** repo; `mirror-manifest.sha256`
      copied into this folder and matches the private `SHA256SUMS.txt`.
- [ ] `README.md` written with a clear **"Reason for capture"**.
- [ ] `SHA256SUMS.txt` regenerated **last** (`tools/gen-sha256sums.ps1`) and
      `sha256sum -c` passes.
- [ ] Added to `evidence/README.md` **and** `evidence/index.html`; all links resolve.
- [ ] Committed with the `Co-authored-by` trailer; public (and private) pushed.