Traditionally, algorithms for counting distinct items in a stream of data would store all the items. A new algorithm, called CVM, uses randomization to estimate the number of distinct items with minimal memory usage. The trick is to keep track of items by recording them and then randomly deleting some. The probability of an item staying on the list is related to the number of rounds it survives. With this method, the researchers were able to accurately estimate the number of distinct words in Hamlet.

  • a1studmuffin@aussie.zone
    link
    fedilink
    English
    arrow-up
    2
    ·
    7 months ago

    Especially considering it scales by adjusting accuracy! That makes it very adaptable so it could be used everywhere from microcontrollers through to Google data servers.