That's usually down to the cache. The cached version will carry on showing unless you clear caches. However, at some point the cache will expire or someone will access a non-cached page and then...
Posted10 hours 5 min ago by Martin Price | System Horizons Ltd (yorkshirepudding) on:
Yep. Just add the 'Global: Random' Sort Criteria.