Chapter 22 PostgreSQL Data Types j 187

set, having this processing feature set right in PostgreSQL makes sense in several use cases:

• It makes sense when the SQL logic or ltering you want to implement depends on the result of the processing (e.g. grouping by week).

•When you have several applications using the same logic, it’s of en easier to share a SQL query than to set up a distributed service API o fering the same result in XML or JSON (a data format you then have to parse).

•When you want to reduce your run-time dependencies, it’s a good idea to understand how much each architecture layer is able to support in your implementation.

Network Address Types

PostgreSQL includes support for both cidr, inet, and macaddr data types. Again, those types are bundled with indexing support and advanced functions and operator support.

The PostgreSQL documentation chapters entitled Network Address Types and Network Address Functions and Operators cover network address types.

Web servers logs are a classic source of data to process where we nd network address types and The Honeynet Project has some free samples for us to play with. This time we’re using the Scan 34 entry. Here’s how to load the sample data set, once cleaned into a proper CSV le:

1begin;

2

3drop table if exists access_log;

4

5create table access_log

6(

9request text,

Chapter 22 PostgreSQL Data Types j 188

The script used to cleanse the original data into a CSV that PostgreSQL is happy about implements a pretty simple transformation from

211.141.115.145 - - [13/Mar/2005:04:10:18 -0500] "GET / HTTP/1.1" 403 2898 "-" "Mozill

into

"211.141.115.145";"2005-05-13 04:10:18 -0500";"GET / HTTP/1.1";"403"

Being mostly interested into network address types, the transformation from the Apache access log format to CSV is lossy here, we keep only some of the elds we might be interested into.

One of the things that’s possible to implement thanks to the PostgreSQL inet data type is an analysis of /24 networks that are to be found in the logs.

To enable that analysis, we can use the set_masklen() function which allows us to transforms an IP address into an arbitrary CIDR network address:

1select distinct on (ip)

2ip,

3set_masklen(ip, 24) as inet_24,

4 set_masklen(ip::cidr, 24) as cidr_24 5 from access_log

6limit 10;

And we can see that if we keep the data type as inet, we still get the full IP address with the /24 network notation added. To have the .0/24 notation we need to be using cidr:

ip │ inet_24 │ cidr_24

═══════════════╪══════════════════╪═════════════════

4.35.221.243 │ 4.35.221.243/24 │ 4.35.221.0/24 4.152.207.126 │ 4.152.207.126/24 │ 4.152.207.0/24 4.152.207.238 │ 4.152.207.238/24 │ 4.152.207.0/24

(10 rows)

Of course, note that you could be analyzing other networks than /24:

1select distinct on (ip)

2ip,

Chapter 22 PostgreSQL Data Types j 189

5 from access_log

6limit 10;

This computes for us the proper starting ip addresses for our CIDR notation for us, of course. Af er all, what’s the point of using proper data types if not for advanced processing?

ip │ cidr_27 │ cidr_28

═══════════════╪══════════════════╪══════════════════

4.35.221.243 │ 4.35.221.224/27 │ 4.35.221.240/28 4.152.207.126 │ 4.152.207.96/27 │ 4.152.207.112/28 4.152.207.238 │ 4.152.207.224/27 │ 4.152.207.224/28

(10 rows)

Equipped with this set_masklen() function, it’s now easy to analyze our access logs using arbitrary CIDR network de nitions.

1select set_masklen(ip::cidr, 24) as network,

2count(*) as requests,

5group by network

6 having array_length(array_agg(distinct ip), 1) > 1

7order by requests desc, ipcount desc;

In our case, we get the following result:

network │ requests │ ipcount

══════════════════╪══════════╪═════════

(12 rows)

Chapter 22 PostgreSQL Data Types j 190

Ranges

Range types are a unique feature of PostgreSQL, managing two dimensions of data in a single column, and allowing advanced processing. The main example is the daterange data type, which stores as a single value a lower and an upper bound of the range as a single value. This allows PostgreSQL to implement a concurrent safe check against overlapping ranges, as we’re going to see in the next example.

As usual, read the PostgreSQL documentation chapters with the titles Range Types and Range Functions and Operators for complete information.

The International Monetary Fund publishes exchange rate archives by month for lots of currencies. An exchange rate is relevant from its publication until the next rate is published, which makes a very good use case for our PostgreSQL range types.

The following SQL script is the main part of the ELT script that has been used for this book. Only missing from this book’s pages is the transformation script that pivots the available tsv le into the more interesting format we use here:

1begin;

2

3create schema if not exists raw;

4

5 -- Must be run as a Super User in your database instance

6-- create extension if not exists btree_gist;

7

8drop table if exists raw.rates, rates;

9

10create table raw.rates

11(

12currency text,

19create table rates

20(

21currency text,

22validity daterange,

Chapter 22 PostgreSQL Data Types j 192

2from rates

5limit 10;

We can see that the validity is a range of dates, and the standard output for this

Having this data set with the exclusion constraint means that we know we have at most a single rate available at any point in time, which allows an application needing the rate for a speci c time to write the following query:

\index{Operators!@}

1 select rate

2from rates

3where currency = 'Euro'

4and validity @> date '2017-05-18';

The operator @> reads contains, and PostgreSQL uses the exclusion constraint’s index to solve that query e ciently:

rate

══════════

1.240740 (1 row)