46

Denormalizing Tags with intarray

Handling user-de ned tags can be challenging in SQL when it comes to allowing advanced user queries. To illustrate the point here, we’re going to index and search for Last.fm tracks that are tagged as blu and rhythm and blu .

Using teh Last.fm dataset from the Million Song Dataset project provides a data set that we can reuse that is full of tracks and their user tags.

Advanced Tag Indexing

PostgreSQL comes with plenty of interesting datatypes, and one of them is known as the arrays type. PostgreSQL also provides a very rich set of extensions, some of them found under the contrib package; one of them is intarray. Let me quote the most interesting part of the documentation for that extension:

The @@ and ~~ operators test whether an array satis es a query, which is expressed as a value of a specialized data type query_int. A query consists of integer values that are checked against the elements of the array, possibly combined using the operators & (AND), | (OR), and ! (NOT). Parentheses can be used as needed. For example, the query 1&(2|3) matches arrays that contain 1 and also contain either 2 or 3.

1 create extension intarray;

Chapter 46 Denormalizing Tags with intarray j 388

Searches

Now we are ready for the real magic. Let’s nd all the tracks we have that have been tagged as both blu and rhythm and blu :

1 select array_agg(rowid)

2from tags

3where tag = 'blues' or tag = 'rhythm and blues';

That query gives the following result, which might not seem very interesting at rst:

array_agg

═══════════

{3,739} (1 row)

The intarray PostgreSQL extension implements a special kind of query string, named query_int. It looks like '(1880&179879)' and it supports the three logic operators not, and, and or, that you can combine in your queries.

As we want our tag search queries to be dynamically provided by our users, we are going to build the query_int string from the tags table itself:

1select format('(%s)',

4from tags

5where tag = 'blues' or tag = 'rhythm and blues';

This query uses the format PostgreSQL function to build a string for us, here puting our intermediate result inside parentheses. The intermediate result is obtained with string_agg which aggregates text values together, using a separator in between them. Usually the separator would be a comma or a semicolon. Here we are preparing a query_int string, and we’re going to search for all the tracks that have been tagged both blu and rhythm and blu , so we’re using the and operator, written &:

query

═════════

3 & 739 (1 row)

That query here allows us to easily inject as many tags as we want to, so that it’s easy to use it as a template from within an application where the user is going to provide the tags list. The intarray extension’s query format also accepts other

Chapter 46 Denormalizing Tags with intarray j 389

operators (or and not) as we saw before, so if you want to o fer those to your users you would need to tweak the query_int building part of the SQL.

Now, how many tracks have been tagged with both the blu and the rhythm and blu tags, you might be asking:

1with t(query) as (

2select format('(%s)',

5from tags

6where tag = 'blues' or tag = 'rhythm and blues'

7)

8select count(*)

9from track_tags join t on tags @@ query;

As you can see we use the query template from above in a common table expression and then inject it in the nal SQL query as join restriction over the track_tags table.

count

═══════

2278 (1 row)

We have 2278 tracks tagged with both the blu and rhythm and blu tags.

Now of course you might want to fetch some track meta-data, but here the only one we have is the track hash id:

1with t(query) as (

2select format('(%s)',

5from tags

6where tag = 'blues' or tag = 'rhythm and blues'

7)

8select track.tid,

10|| case when length(track.artist) > 26 then '…' else '' end

11as artist,

12left(track.title, 26)

13|| case when length(track.title) > 26 then '…' else '' end

14as title

15from track_tags tt

16join tids on tt.tid = tids.rowid

17join t on tt.tags @@ t.query

18join lastfm.track on tids.tid = track.tid

19order by artist;

That gives us the following result: