Beginning Python (2005)

Network Programming

For example, if you issue the command /nick leonardr to an IRC server, you’re attempting to change your nickname from its current value to leonardr. Your attempt might or might not succeed, depending on whether or not there’s already a leonardr on the IRC server.

Our server will support the following three commands, taken from IRC and simplified:

/nick [nickname]: As described above, this attempts to change your nickname. If the nickname is valid and not already taken, your nickname will be changed and the change will be announced to the room. Otherwise, you’ll get a private error message.

/quit [farewell message]: This command disconnects the user from the chat server. Your farewell message, if any, will be broadcast to the room.

/names: This retrieves the nicknames of the users in the chat room as a space-separated string.

The Python Chat Server Protocol

Having decided on a feature set and a design, we must now define an application-specific protocol for our Python Chat Server. This protocol will be similar to SMTP, HTTP, and the IRC protocol in that it will run atop TCP/IP to provide the structure for a specific type of application. However, it will be much simpler than any of those protocols.

The mirror server also defined a protocol, though it was so simple it may have escaped notice. The mirror server protocol consists of three simple rules:

1.Send lines of text to the server.

2.Every time you send a newline, the server will send you back that line of text, reversed, with a newline at the end.

3.Send a blank line to terminate the connection.

The protocol for the Python Chat Server will be a little more complex than that, but by the standards of protocol design it’s still a fairly simple protocol. The following description is more or less the information that would go into an RFC for this protocol. If we were actually writing an RFC, we would go into a lot more detail and provide a formal definition of the protocol; that’s not as necessary here, because the protocol definition will be immediately followed by an implementation in Python.

Of course, if we did write an RFC for this, it wouldn’t be accepted. The IRC protocol already has an RFC, and it’s a much more useful protocol than this example one.

Our Hypothetical Protocol in Action

One good way to figure out the problems involved in defining a protocol is to write a sample session to see what the client and server need to say to each other. Here’s a sample session of the Python Chat Server. In the following transcript, a user nicknamed leonardr connects to a chat room in which a

shady character nicknamed pnorton is already lurking. The diagram shows what leonardr might send to the server, what the server would send to him in response, and what it would send to the other client (pnorton) as a result of leonardr’s input.

341

TEAM LinG

Chapter 16

Initial Connection

After establishing a connection between the client and server, the first stage of the protocol is to get a nickname for the client. A client can’t be allowed into a chat room without a nickname because that would be confusing to the other users. Therefore, the server will ask each new client: “Who are you?” and expect a nickname in response, terminated by a newline. If what’s sent is an invalid nickname or the nickname of a user already in the chat room, the server will send an error message and terminate the connection. Otherwise, the server will welcome the client to the chat room and broadcast an announcement to all other users that someone has joined the chat.

Chat Text

After a client is admitted into the chat room, any line of text they send will be broadcast to every user in the room, unless it’s a server command. When a line of chat is broadcast, it will be prefaced with the nickname of the user who sent it, enclosed in angle brackets (e.g., “<leonardr> Hello, all.”). This will prevent confusion about who said what, and visually distinguish chat messages from system messages.

Server Commands

If the client sends a recognized server command, the command is executed and a private system message may be sent to that client. If the execution of the command changes the state of the chat room (for instance, a user changes his nickname or quits), all users will receive a system message notifying them of

Network Programming

the change (e.g., “leonardr is now known as leonard”). An unrecognized server command will result in an error message for the user who sent it.

General Guidelines

For the sake of convenience and readability, the chat protocol is designed to have a line-based and human-readable format. This makes the chat application usable even without a special client (although we will write a special client to make chatting a little easier). Many TCP/IP protocols work in similar ways, but it’s not a requirement. Some protocols send only binary data, to save bandwidth or because they encrypt data before transmitting it.

Here’s the server code, in PythonChatServer.py. Like MultithreadedMirrorServer, its actual server class is a ThreadingTCPServer. It keeps a persistent map of users’ nicknames that point to the wfile members. That lets the server send those users data. This is how one user’s input can be broadcast to everyone in the chat room:

#!/usr/bin/python import SocketServer import re

import socket

class ClientError(Exception):

“An exception thrown because the client gave bad input to the server.” pass

class PythonChatServer(SocketServer.ThreadingTCPServer): “The server class.”

def __init__(self, server_address, RequestHandlerClass):

“””Set up an initially empty mapping between a user’s nickname and the file-like object used to send data to that user.””” SocketServer.ThreadingTCPServer.__init__(self, server_address,

RequestHandlerClass)

self.users = {}

class RequestHandler(SocketServer.StreamRequestHandler): “””Handles the life cycle of a user’s connection to the chat server: connecting, chatting, running server commands, and disconnecting.”””

NICKNAME = re.compile(‘^[A-Za-z0-9_-]+$’) #Regex for a valid nickname

def handle(self):

“””Handles a connection: gets the user’s nickname, then processes input from the user until they quit or drop the connection.”””

self.nickname = None

self.privateMessage(‘Who are you?’) nickname = self._readline()

done = False try:

self.nickCommand(nickname)

343

TEAM LinG

Chapter 16

self.privateMessage(‘Hello %s, welcome to the Python Chat Server.’\ % nickname)

self.broadcast(‘%s has joined the chat.’ % nickname, False) except ClientError, error:

self.privateMessage(error.args[0]) done = True

except socket.error: done = True

#Now they’re logged in; let them chat. while not done:

try:

done = self.processInput() except ClientError, error:

self.privateMessage(str(error)) except socket.error, e:

done = True

def finish(self):

“Automatically called when handle() is done.” if self.nickname:

#The user successfully connected before disconnecting. #Broadcast that they’re quitting to everyone else. message = ‘%s has quit.’ % self.nickname

if hasattr(self, ‘partingWords’):

message = ‘%s has quit: %s’ % (self.nickname, self.partingWords)

self.broadcast(message, False)

#Remove the user from the list so we don’t keep trying to #send them messages.

if self.server.users.get(self.nickname): del(self.server.users[self.nickname])

self.request.shutdown(2)

self.request.close()

def processInput(self):

“””Reads a line from the socket input and either runs it as a command, or broadcasts it as chat text.”””

done = False

l = self._readline()

command, arg = self._parseCommand(l) if command:

done = command(arg) else:

l = ‘<%s> %s\n’ % (self.nickname, l) self.broadcast(l)

return done

Each server command is implemented as a method. The _parseCommand method, defined later, takes a line that looks like “/nick” and calls the corresponding method (in this case, nickCommand):

Network Programming

#Below are implementations of the server commands.

def nickCommand(self, nickname):

“Attempts to change a user’s nickname.” if not nickname:

raise ClientError, ‘No nickname provided.’ if not self.NICKNAME.match(nickname):

raise ClientError, ‘Invalid nickname: %s’ % nickname if nickname == self.nickname:

raise ClientError, ‘You are already known as %s.’ % nickname if self.server.users.get(nickname, None):

raise ClientError, ‘There\’s already a user named “%s” here.’ %

nickname

oldNickname = None if self.nickname:

oldNickname = self.nickname del(self.server.users[self.nickname])

self.server.users[nickname] = self.wfile self.nickname = nickname

if oldNickname:

self.broadcast(‘%s is now known as %s’ % (oldNickname, self.nickname))

def quitCommand(self, partingWords):

“””Tells the other users that this user has quit, then makes sure the handler will close this connection.”””

if partingWords:

self.partingWords = partingWords

#Returning True makes sure the user will be disconnected. return True

def namesCommand(self, ignored):

“Returns a list of the users in this chat room.” self.privateMessage(‘, ‘.join(self.server.users.keys()))

# Below are helper methods.

def broadcast(self, message, includeThisUser=True):

“””Send a message to every connected user, possibly exempting the user who’s the cause of the message.”””

message = self._ensureNewline(message)

for user, output in self.server.users.items(): if includeThisUser or user != self.nickname:

output.write(message)

def privateMessage(self, message):

“Send a private message to this user.” self.wfile.write(self._ensureNewline(message))

def _readline(self):

“Reads a line, removing any whitespace.” return self.rfile.readline().strip()

345

TEAM LinG

Chapter 16

def _ensureNewline(self, s):

“Makes sure a string ends in a newline.” if s and s[-1] != ‘\n’:

s += ‘\r\n’ return s

def _parseCommand(self, input):

“””Try to parse a string as a command to the server. If it’s an implemented command, run the corresponding method.””” commandMethod, arg = None, None

if input and input[0] == ‘/’: if len(input) < 2:

raise ClientError, ‘Invalid command: “%s”’ % input commandAndArg = input[1:].split(‘ ‘, 1)

if len(commandAndArg) == 2: command, arg = commandAndArg

else:

command, = commandAndArg

commandMethod = getattr(self, command + ‘Command’, None) if not commandMethod:

raise ClientError, ‘No such command: “%s”’ % command return commandMethod, arg

if __name__ == ‘__main__’: import sys

if len(sys.argv) < 3:

print ‘Usage: %s [hostname] [port number]’ % sys.argv[0] sys.exit(1)

hostname = sys.argv[1] port = int(sys.argv[2])

PythonChatServer((hostname, port), RequestHandler).serve_forever()

The Python Chat Client

As with the mirror server, this chat server defines a simple, human-readable protocol. It’s possible to use the chat server through telnet, but most people would prefer to use a custom client.

Here’s PythonChatClient.py, a simple text-based client for the Python Chat Server. It has a few niceties that are missing when you connect with telnet. First, it handles the authentication stage on its own: If you run it on a Unixlike system, you won’t even have to specify a nickname, because it will use your account name as a default. Immediately after connecting, the Python Chat Client runs the /names command and presents the user with a list of everyone in the chat room.

After connecting, this client acts more or less like a telnet client would. It spawns a separate thread to handle user input from the keyboard even as it reads the server’s output from the network:

#!/usr/bin/python import socket import select

import sys import os

from threading import Thread

Network Programming

class ChatClient:

def __init__(self, host, port, nickname):

self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) self.socket.connect((host, port))

self.input = self.socket.makefile(‘rb’, 0) self.output = self.socket.makefile(‘wb’, 0)

#Send the given nickname to the server. authenticationDemand = self.input.readline()

if not authenticationDemand.startswith(“Who are you?”):

raise Exception, “This doesn’t seem to be a Python Chat Server.” self.output.write(nickname + ‘\r\n’)

response = self.input.readline().strip() if not response.startswith(“Hello”):

raise Exception, response print response

#Start out by printing out the list of members. self.output.write(‘/names\r\n’)

print “Currently in the chat room:”, self.input.readline().strip()

self.run()

def run(self):

“””Start a separate thread to gather the input from the keyboard even as we wait for messages to come over the network. This makes it possible for the user to simultaneously send and receive chat text.”””

propagateStandardInput = self.PropagateStandardInput(self.output) propagateStandardInput.start()

#Read from the network and print everything received to standard #output. Once data stops coming in from the network, it means #we’ve disconnected.

inputText = True while inputText:

inputText = self.input.readline() if inputText:

print inputText.strip() propagateStandardInput.done = True

class PropagateStandardInput(Thread):

“””A class that mirrors standard input to the chat server until it’s told to stop.”””

def __init__(self, output):

“””Make this thread a daemon thread, so that if the Python interpreter needs to quit it won’t be held up waiting for this thread to die.”””

Thread.__init__(self) self.setDaemon(True) self.output = output self.done = False

347

TEAM LinG

Chapter 16

def run(self):

“Echo standard input to the chat server until told to stop.” while not self.done:

inputText = sys.stdin.readline().strip() if inputText:

self.output.write(inputText + ‘\r\n’)

if __name__ == ‘__main__’: import sys

#See if the user has an OS-provided ‘username’ we can use as a default #chat nickname. If not, they have to specify a nickname.

try:

import pwd

defaultNickname = pwd.getpwuid(os.getuid())[0] except ImportError:

defaultNickname = None

if len(sys.argv) < 3 or not defaultNickname and len(sys.argv) < 4: print ‘Usage: %s [hostname] [port number] [username]’ % sys.argv[0] sys.exit(1)

hostname = sys.argv[1] port = int(sys.argv[2])

if len(sys.argv) > 3: nickname = sys.argv[3]

else:

#We must be on a system with usernames, or we would have #exited earlier.

nickname = defaultNickname

ChatClient(hostname, port, nickname)

A more advanced chat client might have a GUI that put incoming text in a separate window from the text the user types, to keep input from being visually confused with output. As it is, in a busy chat room, you might be interrupted by an incoming message while you’re typing, and lose your place.

Single-Threaded Multitasking with select

The reason PythonChatClient spawns a separate thread to gather user input is that a call to sys. stdin.readline won’t return until the user enters a chat message or server command. A naïve chat client might call sys.stdin.readline and wait for the user to type something in, but while it was waiting the other users would keep chatting and the socket connection from the server would fill up with a large backlog of chat. No chat messages would be displayed until the user pressed the Enter key (causing sys.stdin.readline to return), at which time the whole backlog would come pouring onto the screen. Trying to read from the socket connection would cause the opposite problem: The user would be unable to enter any chat text until someone else in the chat room said something. Using two threads avoids these problems: One thread can keep an eye on standard input while the other keeps an eye on the socket connection.

Network Programming

However, it’s possible to implement the chat client without using threads. (After all, telnet works more or less the same way as PythonChatClient, and the telnet program is older than the idea of threads.) The secret is to just peek at standard input and the socket connection — not trying to read from them, just seeing if there’s anything to read. You do this by using the select function, provided by Python’s select module.

select takes three lists of lists, and each second-level list contains file-type objects: one for objects you read (like sys.stdin), one for objects to which you write (like sys.stdout), and one for objects to which you write errors (like sys.stdout). By default, a call to select will block (wait for input) but only until at least one of the file-type objects you passed in is ready to be used. It will then return three lists of lists, which contain a subset of the objects you passed in: only the ones that are ready and have some data for the program to pay attention to. You might think of select as acting sort of like Python’s built-in filter function, filtering out the objects that aren’t ready for use. By using select, you can avoid the trap of calling read on a file-type object that doesn’t have any data to read.

Here’s a subclass of ChatClient that uses a loop over select to check whether standard input or the server input have unread data:

class SelectBasedChatClient(ChatClient):

def run(self):

“””In a tight loop, see whether the user has entered any input or whether there’s any from the network. Keep doing this until the network connection returns EOF.”””

socketClosed = False while not socketClosed:

toRead, ignore, ignore = select.select([self.input, sys.stdin], [], [])

#We’re not disconnected yet. for input in toRead:

if input == self.input:

inputText = self.input.readline() if inputText:

print inputText.strip() else:

#The attempt to read failed. The socket is closed. socketClosed = True

elif input == sys.stdin:

input = sys.stdin.readline().strip() if input:

self.output.write(input + ‘\r\n’)

We must pass in three lists to select, but we pass in empty lists of output files and error files. All we care about are the two sources of input (from the keyboard and the network), as those are the ones that might block and cause problems when we try to read them.

In one sense, this code is more difficult to understand than the original ChatClient, because it uses a trick to rapidly switch between doing two things, instead of just doing both things at once. In another sense, it’s less complex than the original ChatClient because it’s less code and it doesn’t involve multithreading, which can be difficult to debug.

349

TEAM LinG

Chapter 16

It’s possible to use select to write servers without forking or threading, but I don’t recommend writing such code yourself. The Twisted framework (described in the section “The Twisted Framework,” later in this chapter) provides a select-based server framework that will take care of the details for you, just as the classes in SocketServer take care of the details of forking and threading.

Other Topics

Many aspects of network programming are not covered in this chapter. The most obvious omission (the technologies and philosophies that drive the World Wide Web) will be taken up Chapter 21. The following sections outline some other topics in networking that are especially interesting or important from the perspective of a Python programmer.

Miscellaneous Considerations for Protocol Design

The best way to learn about protocol design is to study existing, successful protocols. Protocols are usually well documented, and you can learn a lot by using them and reading RFCs. Here are some common design considerations for protocol design not covered earlier in this chapter.

Trusted Servers

The Python Chat Server is used by one client to broadcast information to all other clients. Sometimes, however, the role of a server is to mediate between its clients. To this end, the clients are willing to trust the server with information they wouldn’t trust to another client.

This happens often on web sites that bring people together, such as auction sites and online payment systems. It’s also implemented at the protocol level in many online games, in which the server acts as referee.

Consider a game in which players chase each other around a map. If one player knew another’s location on the map, that player would gain an unfair advantage. At the same time, if players were allowed to keep their locations secret, they could cheat by teleporting to another part of the map whenever a pursuer got too close. Players give up the ability to cheat in exchange for a promise that other players won’t be allowed to cheat either. A trusted server creates a level playing field.

Terse Protocols

Information that can be pieced together by a client is typically not put into the protocol. It would be wasteful for a server that ran chess games to transfer a representation of the entire board to both players after every successful move. It would suffice to send “Your move was accepted.” to the player who made the move, and describe the move to the other player. State-based protocols usually transmit the changes in state, rather than send the whole state every time it changes.

The protocol for the Python Chat Server sends status messages in complete English sentences. This makes the code easier to understand and the application easier to use through telnet. The client behavior depends on those status messages: For instance, PythonChatClient expects the string “Who are you?” as soon as it connects to the server. Doing a protocol this way makes it difficult for the server to customize the status messages, or for the client to translate them into other languages. Many protocols define numeric codes or short abbreviations for status messages and commands, and explain their meanings in the protocols’ RFC or other definition document.