Block lists: words that hide items and keep them from downloading (#47)

Each person has a list for every feed they read and one per feed. An item whose title or text
holds one of the words, matched as whole words so "ai" does not hide everything that "said"
anything, is hidden from them and, since the scanner now keeps each subscriber's filters
separate, is fetched only if someone else still wants it.

Whole-word matching is not something LIKE can do on both SQLite and Postgres, so the matches are
worked out in Rust into a `hidden` table whenever a list changes, someone subscribes, or a scan
brings in new items, and the queries only look that table up. Both new tables are tables rather
than columns because create_missing adds tables but never columns. Hidden counts as read for
the reaper and for "others still want this file", since whoever it is hidden from is as done
with it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-09-25 13:13:03 +00:00
parent 868dd673f3
commit bdda9b3d2e
8 changed files with 383 additions and 55 deletions

View File

@@ -305,6 +305,25 @@ pub fn matches_keywords(keywords: &[String], haystacks: &[&str]) -> bool {
})
}
/// Whether any of these words or phrases appears, as whole words, in the haystacks. Whole words,
/// unlike `matches_keywords`, because a block hides things: "ai" should not hide everything that
/// "said" something. Case and punctuation are ignored, so "A.I." is not caught by "ai", but
/// "Trump's" is by "trump".
pub fn blocked(words: &[String], haystacks: &[&str]) -> bool {
// Letters and digits only, each run of anything else one space, padded so a phrase matches
// at either end: " new york " is in " the new york times ", " york " is not in " yorkshire ".
let norm = |s: &str| {
let mut out = String::from(" ");
for w in s.split(|c: char| !c.is_alphanumeric()).filter(|w| !w.is_empty()) {
out.push_str(&w.to_lowercase());
out.push(' ');
}
out
};
let hay = norm(&haystacks.join(" "));
words.iter().map(|w| norm(w)).any(|w| w.len() > 1 && hay.contains(&w))
}
#[cfg(test)]
mod tests {
use super::*;
@@ -404,4 +423,16 @@ mod tests {
assert!(!matches_keywords(&kws, &["Just a dive"]), "half a keyword is not a match");
assert!(matches_keywords(&[], &["anything"]), "no keywords means take everything");
}
#[test]
fn blocking_matches_whole_words_and_phrases() {
let words = vec!["AI".to_string(), "new york".to_string()];
assert!(blocked(&words, &["What AI means now"]));
assert!(blocked(&words, &["<p>ai, again</p>"]), "markup and punctuation are not words");
assert!(!blocked(&words, &["He said so", "Portrait"]), "not inside another word");
assert!(blocked(&words, &["Live from", "New York."]), "a phrase spans runs of space");
assert!(!blocked(&words, &["New Yorkshire"]));
assert!(!blocked(&[" ".to_string()], &["anything"]), "a blank word blocks nothing");
assert!(!blocked(&[], &["anything"]));
}
}