Thoughts by human, co-written by AIHow I Filter Referrer Spam Without Deleting Analytics History
My public analytics dashboard ranked a suspicious referrer first, with 35 views. I wanted those requests excluded without destroying the evidence behind that decision. The deployed repair uses Matomo's referrer spam list, local exceptions, and a separate rule for which names can appear publicly. A capture command also preserves reports, because keeping database rows alone cannot tell me what yesterday's dashboard showed.
- Excluded observations remain stored, and every public metric applies the same policy.
- Unreviewed included hostnames become an unnamed aggregate instead of gaining public exposure.
- Archived rules explain a calculation; a saved response preserves what was actually reported.
How a reported referrer became a public ranking
The ranking treated a hostname supplied by a client as something worth showing readers. In the September 6 capture, that hostname was uniuit.com. Displaying the name gave it exposure regardless of whether anyone followed a real link from that site.
Referrer spam is an established form of abuse. Matomo's May 2015 explanation describes automated requests carrying fabricated referrers to get a site noticed in analytics. Our repeated homepage/article requests supported a local exclusion, but did not authenticate an operator or prove that motive. The request investigation preserves that distinction.
My requirement during the repair was:
"we still need to save those data for historical provenance reasons right?"
The design had to preserve the evidence behind an exclusion, including evidence that could later overturn it.
Separating observations from reporting decisions
The repair gives three concepts separate responsibilities in my first-party analytics:
| Concept | Responsibility |
|---|---|
| Observation | Retain the reported hostname and existing request fields. |
| Assessment | Apply a versioned include/exclude rule, with its source and reason. |
| Public display | Name reviewed hosts; group other included names as “Other reported referrers.” |
The public API enforces name suppression before sending data to the browser. Unknown names remain counted unless an exclusion rule matches. That avoids declaring an unfamiliar source abusive simply because I have not reviewed it.
Exclusions affect totals, charts, pages, devices, and referrer rankings, including the All selection. The report separately shows how many observations its policy excluded for the selected scope. Hiding a row while leaving those requests in the headline total would leave the measurement inconsistent. The ADR records these boundaries.
Display approval permits a name to appear. It does not authenticate a referral: a client can supply a familiar hostname too.
Using Matomo rules with local exceptions
Matomo supplies maintained prior art, but its list could not decide this case for me. The pinned source contained 2,348 hosts and omitted the hostname under investigation. That required a reviewed local rule.
For someone implementing the filter, the matching boundary matters. After parsing and normalizing a hostname, a subtree rule has this meaning:
host === ruleHost || host.endsWith(`.${ruleHost}`)For example.com, that includes sub.example.com and excludes lookalikes such as notexample.com or example.com.other.test. This illustrates the boundary; the production evaluator enumerates suffixes and resolves rule precedence. Matomo's inspected matcher searches the whole referrer string. I adopted its list with narrower matching semantics.
Reviewed local rules override upstream entries; more specific local rules can refine broader ones. An explicit include can correct a false positive. Each change gets a new policy version, while the exact upstream bytes, their hash, and the evaluator identity remain archived. List membership is a reporting signal whose mistakes I must be able to correct.
Preserving historical reports
Retained observations support recalculation. They do not automatically reconstruct an earlier dashboard: late writes, owner exclusions, and other classification changes can alter the result even under the same referral policy.
The report-capture command saves the exact received JSON and its hash, checks the advertised policy against the archive, and refuses to overwrite an earlier capture. Deployment records bind the policy to the released code. Captures are explicit, not automatic snapshots of every report.
This repair adds no visitor field and deletes no observation. The retained evidence is a bounded hostname and selected request fields; full original referrer headers were never stored. Keeping that evidence and saving report responses answers two different historical questions.
Referrer filtering results and costs
The fixed August 8–September 6 UTC check produced 972 Browser observations before referral exclusions: 937 included plus 35 excluded, all 35 through the local rule. These were separate read-only production queries, not a transactional snapshot. Verification record. Subsequent live verification confirmed the deployed policy, suppressed names, and scoped exclusion counts.
The public evidence records aggregates and hashes; private captures let me audit the original reads.
Filtering at query time costs database work. In that production sample, the totals query went from 7.77 to 14.95 milliseconds, with rows read increasing from 8,423 to 19,826. Those are database measurements, not page latency. The ADR records the larger benchmark and when to reconsider stored or materialized results.
Update, September 7 UTC: that cost assessment was incomplete. The full report used 182,388 rows read across nine statements—about 3.65% of D1's free daily read allowance for one report. The account subsequently exhausted the allowance. Its query profile was dominated by the repeated referral matcher, although the metrics do not identify who requested those reports. Short database execution time had concealed an unacceptable read budget. Incident evidence; Cloudflare's scanned-row accounting.
The database was small: the later count found 7,010 stored page observations in about 3.1 MB of database storage. The 182,388 number described repeated query work for one report. It did not describe stored observations or visitors. The saved benchmark itself accounts for 190,811 reads including its baseline query; additional saved API checks also cost reads, but their exact contribution is unavailable. The reviewed evidence does not attribute the whole allowance to those checks or to an attacker. Read accounting and attribution.
After upgrading Workers, the corrected production query returned the identical historical report with 33,259 reads—81.8% fewer. It calculates every panel from one shared assessment. Two live requests also returned identical cached results, including one with an irrelevant query parameter. Reports can be up to an hour old, with their calculation time visible. Cache storage is local to a Cloudflare data center, so these checks establish a measured improvement and local reuse, without guaranteeing a global read budget. Production verification and limits.
The maintenance cost also remains: review new names, update the source, and reverse mistaken exclusions. Clients can change or omit the header; even a correctly implemented rule can hide legitimate visits. The reconciled counts verify policy application, without establishing how many requests came from people.
The repair gives me an accountable reporting decision: the observation remains available, the rule has a reason and version, and a saved report records what I published.
Glossary
| Term | Meaning and source | Date |
|---|---|---|
| Referrer spam | Fabricated referral information intended to gain exposure through analytics. Matomo explanation. | Published May 13, 2015; checked September 7, 2026 UTC |
| Matomo referrer spam list | Community-contributed hostname list; our release uses a fixed revision. Source README. | Revision e65db652; checked September 7, 2026 UTC |
| Referral policy | Our versioned reporting rules, distinct from retained observations and public name approval. Decision record. | Policy 2026-09-06.2; activated September 7, 2026 UTC |