Beginning Python - From Novice To Professional (2005)

290 C H A P T E R 1 3 ■ D A T A B A S E S U P P O R T

Table 13-7. DB API Constructors and Special Values

Downloading and Installing pysqlite

As mentioned previously, there are many SQL database engines available, with corresponding Python modules. Most of these database engines are meant to be run as server programs, and require administrator privileges even to install them. In order to lower the threshold for playing around with the Python DB API, I’ve chosen to use a tiny database engine called SQLite, which doesn’t have to be run as a standalone server, and which can work directly on local files, instead of with some centralized database storage mechanism.

You can download pysqlite from the official Web page, http://pysqlite.org. For Linux systems with package manager systems, chances are you can get pysqlite and SQLite directly from the package manager. The Windows binaries for pysqlite actually include the database engine itself (that is, SQLite), so all you have to do is to download the pysqlite installer corresponding to your Python version, run it, and you’re all set.

If you’re not using Windows, and your operating system does not have a package manager where you can find pysqlite and SQLite, you have to get the source packages for pysqlite and SQLite and compile them yourself.

You can get the sources for SQLite from the SQLite Web page, http://sqlite.org. (Make sure you get one of the source packages where automatic code generation has already been performed.) Compiling SQLite is basically a matter of following the instructions in the included README file. When subsequently compiling pysqlite, you need to make sure that the compilation process can access the SQLite libraries and include files. If you’ve installed SQLite in some standard location, it may well be that the setup script in the SQLite distribution can find it on its own. In that case, you simply need to execute the following commands:

python setup.py build python setup.py install

You could simply use the latter command, which will perform the build automatically. If this gives you heaps of error messages, chances are the installation script didn’t find the required files. Make sure you know where the include files and libraries are installed, and supply them explicitly to the install script. Let’s say I compiled SQLite in place in a directory called /home/mlh/sqlite/current; then the header files could be found in /home/mlh/sqlite/

current/src and the library in /home/mlh/sqlite/current/build/lib. In order to let the installation process use these paths, edit the setup script, setup.py. In this file you’ll want to set the variables include_dirs and library_dirs:

include_dirs = ['/home/mlh/sqlite/current/src'] library_dirs = ['/home/mlh/sqlite/current/build/lib']

After rebinding these variables, the install procedure described earlier should work without errors.

Getting Started

Once you’ve got pysqlite installed, you can import it as a module, under the name sqlite. You can then create a connection directly to a database file—which will be created if it does not exist—by supplying a file name (which can be a relative or absolute path to the file):

>>>import sqlite

>>>conn = sqlite.connect('somedatabase.db')

You can then get a cursor from this connection:

>>>curs = conn.cursor()

This cursor can then be used to execute SQL queries. Once you’re done, if you’ve made any changes, make sure you commit them, so they’re actually saved to the file:

>>> conn.commit()

You can (and should) commit each time you’ve modified the database, not just when you’re ready to close it. When you are ready to close it, just use the close method:

>>> conn.close()

An Example Database Application

As an example program, I’ll construct a little nutrient database, based on data from the USDA Nutrient Data Laboratory (http://www.nal.usda.gov/fnic/foodcomp). On their Web page, under Food Composition Products, follow the link to the USDA National Nutrient Database for Standard Reference. There you should find lots of different data files in plain text (ASCII) format, just the way we like it. Follow the “Download” link, and download the zip file referenced by the ASCII

Listing 13-1. Importing Data into the Database (importdata.py)

import sqlite

def convert(value):

if value.startswith('~'): return value.strip('~')

if not value: value = '0'

return float(value)

conn = sqlite.connect('food.db') curs = conn.cursor()

field_count = 10

markers = ', '.join(['%s']*field_count)

query = 'INSERT INTO food VALUES (%s)' % markers

for line in open('ABBREV.txt'): fields = line.split('^')

vals = [convert(f) for f in fields[:field_count]] curs.execute(query, vals)

conn.commit()

conn.close()

When you run this program (with ABBREV.txt in the same directory), it will create a new file called food.db, containing all the data of the database.

I’d really like to encourage you to play around with this example, using other inputs, adding print statements, and the like.

294 C H A P T E R 1 3 ■ D A T A B A S E S U P P O R T

Searching and Dealing with Results

Using the database is really simple. Again, you create a connection and get a cursor from that connection. Execute the SQL query with the execute method and extract the results with, for example, the fetchall method. Listing 13-2 shows a tiny program that takes an SQL SELECT condition as a command-line argument and prints out the returned rows in a record format. You could try it out with a command line like the following:

$ python food_query.py "kcal <= 100 AND fiber >= 10 ORDER BY sugar"

You may notice a problem when you run this. The first row, raw orange peel, seems to have no sugar at all. That’s because the field is missing in the data file. You could improve the import script to detect this condition, and insert None instead of a real value, to indicate missing data. Then you could use a condition such as "kcal <= 100 AND fiber >= 10 AND sugar ORDER BY sugar", requiring the sugar field to have real data in any returned rows. As it happens, this strategy will work with the current database, as well, where this condition will discard rows where the sugar level is zero.

■Caution You might want to try a condition that searches for a specific food item, using an ID, such as 08323 for Cocoa Pebbles. The problem is that SQLite handles its values in a rather nonstandard fashion. Internally, all values are, in fact, strings, and some conversion and checking goes on between the database and the Python API. Usually, this works just fine, but this is an example of where you might run into trouble. If you supply the value 08323, it will be interpreted as the number 8323, and subsequently converted into the string "8323"—an ID that doesn’t exist. One might have expected an error message here, rather than this surprising and rather unhelpful behavior, but if you are careful, and use the string "08323" in the first place, you’ll be fine.

Listing 13-2. Food Database Query Program (food_query.py)

import sqlite, sys

conn = sqlite.connect('food.db') curs = conn.cursor()

query = 'SELECT * FROM food WHERE %s' % sys.argv[1] print query

curs.execute(query)

names = [f[0] for f in curs.description] for row in curs.fetchall():

for pair in zip(names, row): print '%s: %s' % pair

print

A Quick Summary

This chapter has given a rather brief introduction to making Python programs interact with relational databases. It’s brief because, if you master Python and SQL, the coupling between the two, in the form of the Python DB API, is quite easy to master. Here are some of the concepts covered in this chapter:

The Python DB API. A simple, standardized interface that database wrapper modules should conform to, to make it easier to write programs that will work with several different databases.

Connections. A connection object represents the communication link with the SQL database. From it, you can get individual cursors, using the cursor method. You also use the connection object to commit or roll back transactions. After you’re done with the database, the connection can be closed.

Cursors. A cursor is used to execute queries and to examine the results. Resulting rows can be retrieved one by one, or many (or all) at once.

Types and Special Values. The DB API specifies the names of a set of constructors and special values. The constructors deal with date and time objects, as well as binary data objects. The special values represent the types of the relational database, such as STRING,

NUMBER, and DATETIME.

SQLite. A small, embedded SQL database, whose Python wrapper is called pysqlite. It’s fast and simple to use, and does not require a separate server to be set up.

New Functions in This Chapter

a. The parameters to the connect function are database dependent.

What Now?

Persistence and database handling is an important part of many, if not most, big program systems. Another component shared by a great number of such systems is dealt with in the next chapter . . .

298 C H A P T E R 1 4 ■ N E T W O R K P R O G R A M M I N G

modules (such as those that deal with various forms of data encoding for network transport) may be seen as network-related. I’ve been fairly restrictive in my selection of modules here.

socket

A basic component in network programming is the socket. A socket is basically an “information channel” with a program on both ends. The programs may be on different computers (connected through a network) and may send information to each other through the socket. Most network programming in Python hides the basic workings of the socket module, and doesn’t interact with the sockets directly.

Sockets come in two varieties: server sockets and client sockets. After you create a server socket, you tell it to wait for connections. It will then listen at a certain network address (a combination of an IP address and a port number) until a client socket connects. The two can then communicate.

Dealing with client sockets is usually quite a bit easier than dealing with the server side, because the server has to be ready to deal with clients whenever they connect, and it must deal with multiple connections, while the client simply connects, does its thing, and disconnects. Later in this chapter, I deal with server programming through the SocketServer class family and the Twisted framework.

A socket is an instance of the socket class from the socket module. It is instantiated with up to three parameters: an address family (defaulting to socket.AF_INET), whether it’s a stream (socket.SOCK_STREAM, the default) or a datagram (socket.SOCK_DGRAM) socket, and a protocol (defaulting to 0, which should be okay). For a plain-vanilla socket, you don’t really have to supply any arguments.

A server socket uses its bind method followed by a call to listen to listen to a given address. A client socket can then connect to the server by using its connect method with the same address as used in bind. (On the server side, you can, for example, get the name of the current machine using the function socket.gethostname.) In this case, an address is just a tuple of the form (host, port), where host is a host name (such as www.example.com) and port is a port number (an integer). The listen method takes a single argument, which is the length of its backlog (the number of connections allowed to queue up, waiting for acceptance, before connections start being disallowed).

Once a server socket is listening, it can start accepting clients. This is done using the accept method. This method will block (wait) until a client connects, and then it will return a tuple of the form (client, address), where client is a client socket and address is an address, as explained earlier. The server can then deal with the client as it sees fit, and then start waiting for new connections, with another call to accept. This is usually done in an infinite loop.

■Note This form of server programming is called blocking or synchronous network programming. Later, you’ll see examples of nonblocking or asynchronous network programming, as well as using threads to be able to deal with several clients at once.

For transmitting data, sockets have two methods, send and recv (for “receive”). You can call send with a string argument to send data, and recv with a desired (maximum) number of bytes to receive data. If you’re not sure what number to use, 1024 is as good a choice as any.

Listings 14-1 and 14-2 show an example client/server pair that is about as simple as it gets. If you run them on the same machine (starting the server first), the server should print out a message about getting a connection, and the client should then print out a message it received from the server. You can run several clients while the server is still running. By replacing the call to gethostname in the client with the actual host name of the machine where the server is running, you can have the two programs connect across a network from one machine to another.

■Note The port numbers you use are normally restricted. In a Linux or UNIX system, you need administrator privileges in order to use a port below 1024. These low-numbered ports are used for standard services, such as port 80 for your Web server (if you have one). If you stop a server with Ctrl-C, you might need to wait for a bit before using the same port number again.

Listing 14-1. A Minimal Server

import socket

s = socket.socket()

host = socket.gethostname() port = 1234

s.bind((host, port))

s.listen(5) while True:

c, addr = s.accept()

print 'Got connection from', addr c.send('Thank you for connecting') c.close()

Listing 14-2. A Minimal Client

import socket

s = socket.socket()

host = socket.gethostname() port = 1234

s.connect((host, port)) print s.recv(1024)