Community guidelines
Be specific and constructive. No vendor spam — promoting your own product belongs in a listing. Anyone can read; posting needs a free account.
Our AI group wants raw data, cleaned data, embeddings, checkpoints and every experiment retained because it may be useful later. Storage growth is now faster than compute growth. How are people setting retention without deleting the one dataset somebody needs next year??
Classify by reproducibility. If cleaned data and embeddings can be recreated from raw data plus code, they do not all need expensive long term storage. Keep manifests, versions and checksums so recreation is real, not theoretical.
We use tiers. Active training data stays hot, old experiments move to S3 Glacier, and temporary artifacts expire automatically unless the project owner extends them.
The team argues egress and recompute may cost more than storage. They aren't always wrong.
Correct, so price the full lifecycle. Storage, requests, retrieval, transfer, recompute and compliance. Add access frequency from real logs. The answer will differ by dataset. A retention policy is better when it has an exception process than when everyone quietly disables deletion.