Chapter 8 Indexing Strategy j 77
faster than a Bloom index, but it can require many B-tree indexes to support all possible queries where one needs only a single Bloom index. Note however that Bloom indexes only support equality queries, whereas B-tree indexes can also perform inequality and range searches.
The Bloom lter index is implemented as a PostgreSQL extension starting in PostgreSQL 9.6, and so to be able to use this access method it’s necessary to rst create extension bloom.
Both Bloom indexes and BRIN indexes are mostly useful when covering mutliple columns. In the case of Bloom indexes, they are useful when the queries themselves are referencing most or all of those columns in equality comparisons.
Advanced Indexing
The PostgreSQL documentation about indexes covers everything you need to know, in details, including:
•Multicolumn indexes
•Indexes and ORDER BY
•Combining multiple indexes
•Unique indexes
•Indexes on expressions
•Partial indexes
•Partial unique indexes
•Index-only scans
There is of course even more, so consider reading this PostgreSQL chapter in its entirety, as the content isn’t repeated in this book, but you will need it to make informed decisions about your indexing strategy.
Adding Indexes
Deciding which indexes to add is central to your indexing strate . |
Not every |
query needs to be that fast, and the requirements are mostly user de |
ned. That |
Chapter 8 Indexing Strategy j 78
said, a general system-wide analysis can be achieved thanks to the PostgreSQL extension pg_stat_statements.
Once this PostgreSQL extension is installed and deployed — this needs a PostgreSQL restart, because it needs to be registered in shared_preload_libraries
— then it’s possible to have a list of the most common queries in terms of number of times the query is executed, and the cumulative time it took to execute the query.
You can begin your indexing needs analysis by listing every query that averages out to more than 10 milliseconds, or some other sensible threshold for your application. The only way to understand where time is spent in a query is by using the EXPLAIN command and reviewing the query plan. From the documentation of the command:
PostgreSQL devises a query plan for each query it receives. Choosing the right plan to match the query structure and the properties of the data is absolutely critical for good performance, so the system includes a complex planner that tries to choose good plans. You can use the EXPLAIN command to see what query plan the planner creates for any query. Plan-reading is an art that requires some experience to master, but this section attempts to cover the basics.
Here’s a very rough guide to using explain for xing query performances:
•use the spelling below when using explain to understand run time characteristics of your queries:
1 explain (analyze, verbose, buffers)
2<query here>;
•In particular when you’re new to reading query plans, use visual tools such as https://explain.depesz.com and PostgreSQL Explain Visualizer, or the one included in pgAdmin.
•First check for row count di ferences in between the estimated and the effective numbers.
Good statistics are critical to the PostgreSQL query planner, and the collected statistics need to be reasonnably up to date. When there’s a huge di ference in between estimated and e fective row counts (several orders of magnitude, a thousand times o f or more), check to see if tables are analyzed frequently enough by the Autovacuum Daemon, then check if you
Chapter 8 Indexing Strategy j 79
should adjust your statistics target.
•Finally, check for time spent doing sequential scans of your data, with a filter step, as that’s the part that a proper index might be able to optimize.
Remember Amdahl’s law when optimizing any system: if some step takes 10% of the run time, then the best optimization you can reach from dealing with this step is 10% less, and usually that’s by removing the step entirely.
This very rough guide doesn’t take into account costly functions and expressions which may be indexed thanks to index on expressions, nor ordering clauses that might be derived directly from a supporting index.
Query optimisation is a large topic that is not covered in this book, and proper indexing is only a part of it. What this book covers is all the SQL capabilities that you can use to retrieve exactly the result set needed by your application.
The vast majority of slow queries found in the wild are still queries that return way too many rows to the application, straining the network and the servers memory. Returning millions of rows to an application that then displays a summary in a web browser is far too common.
The rst rule of optimization in SQL, as is true for code in general, is to answer the following question:
Do I really need to do any of that?
The very best query optimization technique consists of not having to execute the query at all. Which is why in the next chapter we learn all the SQL functionality that will allow you to execute a single query rather than looping over the result set of a rst query only to run an extra query for each row retrieved.
Chapter 8 Indexing Strategy j 80
Figure 8.1: Advanced Django