A web crawler for open-data portals fetches many duplicate URLs before expensive parsing. What lightweight structure can help dedupe early in the ingest path?
Select an answer to reveal the explanation.
Short Explanation
A Bloom filter is like a quick doorman who says we already saw that guest without digging through the whole guest book. You skip obvious repeats before the expensive kitchen work of parsing.
Full Explanation
Bloom filters provide space-efficient probabilistic membership tests, useful for early deduplication in ingest and streaming paths. They can reject likely-seen URLs before costly download or parse stages, accepting a small false-positive risk. That efficiency pattern fits high-volume civic crawl and ingest pipelines.