50

Counting Distinct Users with HyperLogLog

If you’ve been following along at home and keeping up with the newer statistics developments, you might have heard about this new state of the art cardinality estimation algorithm called HyperLogLog.

This technique is now available for PostgreSQL in the extension postgresql-hll available at https://github.com/citusdata/postgresql-hll and is packaged for multiple operating systems such as Debian and RHEL, through the PostgreSQL community packaging e forts and resources.

HyperLogLog

HyperLogLog is a very special hash value. It aggregates enough information into a single scalar value to compute a distinct value with some precision loss.

Say we are counting unique visitors. With HyperLogLog we can maintain a single value per day, and then union those values together to obtain unique weekly or monthly visitor counts!

Here’s an example in SQL of the magic provided by the hll extension:

1select to_char(date, 'YYYY/MM') as month,

Chapter 50 Counting Distinct Users with HyperLogLog j 414

3 from daily_uniques

4group by month;

While we are keeping daily aggregates on disk, we can use the HyperLogLog maths to union them together and compute an approximation of the monthly unique count from the same dataset!

(1 row)

So by keeping only a small amount of data per day, typically 1280 bytes, it is then possible to compute monthly unique counts from that, without having to scan a whole month of records again.

Installing postgresql-hll

It’s as simple as create extension hll;, once the OS package is installed on your system. The extension provides a new datatype named hll and we can use \dx+ hll to discover what kind of support comes with it. Here’s an edited version of the output of the \dx+ hll command, where some lines have been l- tered out of the 71 SQL objects:

Objects in extension "hll" Object description

══════════════════════════════════════════════════════════════════════════════

cast from bigint to hll_hashval cast from bytea to hll

cast from hll to hll

cast from integer to hll_hashval function hll(hll,integer,boolean) function hll_add(hll,hll_hashval) function hll_add_agg(hll_hashval)

function hll_add_agg(hll_hashval,integer) function hll_add_agg(hll_hashval,integer,integer)

function hll_add_agg(hll_hashval,integer,integer,bigint) function hll_add_agg(hll_hashval,integer,integer,bigint,integer) function hll_add_rev(hll_hashval,hll)

...

function hll_cardinality(hll) function hll_empty()

function hll_eq(hll,hll)

...

function hll_hash_any(anyelement,integer) function hll_hash_bigint(bigint,integer) function hll_hash_boolean(boolean,integer) function hll_hash_bytea(bytea,integer)

Chapter 50 Counting Distinct Users with HyperLogLog j 415

function hll_hash_integer(integer,integer) function hll_hash_smallint(smallint,integer) function hll_hash_text(text,integer)

...

operator #(NONE,hll) operator <>(hll,hll)

operator <>(hll_hashval,hll_hashval) operator =(hll,hll)

operator =(hll_hashval,hll_hashval) operator ||(hll,hll)

operator ||(hll,hll_hashval) operator ||(hll_hashval,hll) type hll

type hll_hashval

From that output we learn the list of hll operators, such as the interesting # operator, a unary operator that works on an hll value. More about this one later…

Counting Unique Tweet Visitors

As an example use case for the HyperLogLog data type, we are going to count unique visitors to our tweets, using the application we introduced in Data Manipulation and Concurrency Control.

The two main operations around an hll data type consists of the following:

• Build a hash from an input value, such as an IP address.

• Update the already known hll value with the hash.

The main idea behind hll is to keep a single hll value per granularity, here per tweet message and per day. This means that each time we have a new visit on a tweet, we want to UPDATE our hll set to count that visitor.

As we have seen in the previous chapter, concurrency is a deal breaker for UPDATE heavy scenarios where the same row is solicited over and over again. So we are going to work in two steps again here, rst doing an INSERT per visit and then arranging a background process to transform those visits into an UPDATE to the single hll aggregate per tweet and date.

Here’s the visitor table where we can insert every single visit:

1create table tweet.visitor

2(

8unique(messageid, datetime, ipaddr)

9);

It’s a pretty simple structure, and is meant to register our online activity.

We can generate some tweet visits easily with a program such as the following. Again, I’m using Common Lisp to implement a very simple COPY-based loading program.

1(defparameter *connspec* '("appdev" "dim" nil "localhost"))

2 (defparameter *visitor-table* "tweet.visitor")

3(defparameter *visitor-columns* '("messageid" "ipaddr" "datetime"))

7(let ((count 0)

8(copier (open-db-writer connspec *visitor-table* *visitor-columns*)))

9(unwind-protect

17;; and return the number of rows copied

18count)))

The script is written so as to target a smallish range of IP addresses and range of dates in order to generate collisions and see our unique visitors count as being more than one.

1;;;

2;;; select '192.168.0.0'::ip4::bigint; == 3232235520

3;;;

4 (defparameter *ip-range-start* 3232235520)

5(defparameter *ip-range-size* (expt 2 16))

6

7(defun generate-ipaddress (&optional

10"Generate N random IP addresses, as strings."

11(int-to-ip (+ range-start (random range-size))))

12

13(defun generate-timestamp ()

14"Generate a random timestamp between now and a month ago."

15(local-time:timestamp- (local-time:now) (random #. (* 24 60 31)) :minute))

Chapter 50 Counting Distinct Users with HyperLogLog j 417

When generating data with those function, we pick the subnet in 192.168.0.0/16 and a span of a month of data. Here’s how to interactively generate 100 000 visits from the Common Lisp REPL, measuring the time that takes:

1 CL-USER> (time (shakes::insert-visistors 3 100000))

2(SHAKES::INSERT-VISISTORS 3 100000)

3took 7,513,209 microseconds (7.513209 seconds) to run.

4 244,590 microseconds (0.244590 seconds, 3.26%) of which was spent in GC.

5During that period, and with 4 available CPU cores,

65,242,334 microseconds (5.242334 seconds) were spent in user mode

7 314,728 microseconds (0.314728 seconds) were spent in system mode

8691,153,296 bytes of memory allocated.

9 770 minor page faults, 0 major page faults, 0 swaps. 10 100000

Thanks to using the COPY streaming protocol, we can mix generating the numbers and communicating with the PostgreSQL server, and have our hundred thousand visits be generated in the database in less than 8s on my laptop. That’s certainly fast enough for interactive discovery of a data model. It’s quite easy with PostgreSQL to just try it and see.

We can check the result of inserting 100000 visits to the messageid 3 with the following query:

1select messageid,

2datetime::date as date,

3count(*) as count,

4count(distinct ipaddr) as uniques,

7where messageid = 3

8 group by messageid, date

9 order by messageid, date

10 limit 10;

We have a precise count of all the visitors to the message, and we can see that even with a 16-bits range of IP addresses we already have several visits from the same IP addresses.

messageid │ date │ count │ uniques │ duplicates

═══════════╪════════════╪═══════╪═════════╪════════════