A multi-agent pipeline runs dozens of tasks per day. Each sub-agent writes task-specific data to a shared memory store. After several weeks, the store has grown to hundreds of gigabytes of stale task data, causing slow lookups and increased costs. What is the recommended approach to address this?
Select an answer to reveal the explanation.
Short Explanation
Imagine a whiteboard where every sub-agent writes notes and nobody erases them; it fills up and today's notes get buried. You need TTL-based expiration on task-scoped memory entries so they're automatically pruned after the task lifecycle completes, keeping the store bounded and lookups fast.
Full Explanation
Task-scoped memory is designed for the duration of a single task — once the task completes, that data has no operational value. The recommended approach is to attach a TTL (time-to-live) to task-scoped entries at write time, keyed to the expected task duration or a post-completion window. The store's expiration mechanism then automatically prunes entries without manual intervention. This keeps store size bounded and lookup performance consistent.
The approach that migrates the shared memory store to a higher-capacity database tier is incorrect because scaling up storage capacity is a cost-scaling response, not an architectural fix. The root problem is unbounded data accumulation — more capacity just defers the problem and increases cost without addressing the lack of lifecycle management.
The approach that compresses all stored data to reduce footprint without deleting any records is incorrect because compression reduces storage size but does not remove stale data. Lookup performance suffers not just from data size but from the volume of index entries — compressing old records still leaves them in the store, requiring the index to scan them.
The approach that moves completed task data to a cold archive and keeps the hot store only for active tasks, manually triggering archival once a week, describes a viable pattern (hot/cold tiering) but the manual weekly trigger is the disqualifier. Manual archival is error-prone, inconsistent, and does not scale to dozens of tasks per day. It also does not constitute automatic pruning — it just moves data to a different tier, which still requires eventual deletion. Automated TTL expiration is simpler, lower-maintenance, and aligned with how agentic frameworks recommend managing task-scoped state.