Block lists: words that hide items and keep them from downloading (#47)
Each person has a list for every feed they read and one per feed. An item whose title or text holds one of the words, matched as whole words so "ai" does not hide everything that "said" anything, is hidden from them and, since the scanner now keeps each subscriber's filters separate, is fetched only if someone else still wants it. Whole-word matching is not something LIKE can do on both SQLite and Postgres, so the matches are worked out in Rust into a `hidden` table whenever a list changes, someone subscribes, or a scan brings in new items, and the queries only look that table up. Both new tables are tables rather than columns because create_missing adds tables but never columns. Hidden counts as read for the reaper and for "others still want this file", since whoever it is hidden from is as done with it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -305,6 +305,25 @@ pub fn matches_keywords(keywords: &[String], haystacks: &[&str]) -> bool {
|
||||
})
|
||||
}
|
||||
|
||||
/// Whether any of these words or phrases appears, as whole words, in the haystacks. Whole words,
|
||||
/// unlike `matches_keywords`, because a block hides things: "ai" should not hide everything that
|
||||
/// "said" something. Case and punctuation are ignored, so "A.I." is not caught by "ai", but
|
||||
/// "Trump's" is by "trump".
|
||||
pub fn blocked(words: &[String], haystacks: &[&str]) -> bool {
|
||||
// Letters and digits only, each run of anything else one space, padded so a phrase matches
|
||||
// at either end: " new york " is in " the new york times ", " york " is not in " yorkshire ".
|
||||
let norm = |s: &str| {
|
||||
let mut out = String::from(" ");
|
||||
for w in s.split(|c: char| !c.is_alphanumeric()).filter(|w| !w.is_empty()) {
|
||||
out.push_str(&w.to_lowercase());
|
||||
out.push(' ');
|
||||
}
|
||||
out
|
||||
};
|
||||
let hay = norm(&haystacks.join(" "));
|
||||
words.iter().map(|w| norm(w)).any(|w| w.len() > 1 && hay.contains(&w))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
@@ -404,4 +423,16 @@ mod tests {
|
||||
assert!(!matches_keywords(&kws, &["Just a dive"]), "half a keyword is not a match");
|
||||
assert!(matches_keywords(&[], &["anything"]), "no keywords means take everything");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn blocking_matches_whole_words_and_phrases() {
|
||||
let words = vec!["AI".to_string(), "new york".to_string()];
|
||||
assert!(blocked(&words, &["What AI means now"]));
|
||||
assert!(blocked(&words, &["<p>ai, again</p>"]), "markup and punctuation are not words");
|
||||
assert!(!blocked(&words, &["He said so", "Portrait"]), "not inside another word");
|
||||
assert!(blocked(&words, &["Live from", "New York."]), "a phrase spans runs of space");
|
||||
assert!(!blocked(&words, &["New Yorkshire"]));
|
||||
assert!(!blocked(&[" ".to_string()], &["anything"]), "a blank word blocks nothing");
|
||||
assert!(!blocked(&[], &["anything"]));
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user