Beginning Python (2005)

Web Applications and Web Services

>>>import xmlrpclib

>>>server = xmlrpclib.ServerProxy(“http://localhost:8001/”)

>>>server.system.listMethods()

[‘bittywiki.delete’, ‘bittywiki.getPage’, ‘bittywiki.save’, ‘system.listMethods’, ‘system.methodHelp’, ‘system.methodSignature’]

>>>server.system.methodHelp(“bittywiki.save”) ‘Saves a page of the wiki.’

>>>server.system.methodSignature(“bittywiki.save”) ‘signatures not supported’

XML-RPC introspection isn’t meant as a substitute for a human-readable API document. For one thing, it’s hard to get people excited about using your API if they must use XML-RPC method calls to even see what it is. However, the introspection API does make it a lot easier to experiment with an XML-RPC web service from an interactive Python shell.

A Python-centric reference to the XML-RPC introspection API is at www.python.org/doc/lib/ serverproxy-objects.html.

WSDL

Many SOAP-based web services define their interface in a WSDL file. WSDL is basically a machineparseable version of the human-readable API document shown earlier in this section.

Recall that XML-RPC defines a set of rules for transforming a few basic data structures into XML documents and back into data structures. WSDL allows such rules to be constructed on the fly. It’s more or less a programming-language-agnostic schema for describing functions: their names, the data types of their arguments, and the data types of their return values. Although WSDL is associated with SOAP, it’s possible to use SOAP without using WSDL (in fact, we did just that throughout this chapter’s section on SOAP).

A WSDL file is an XML document (of course!), which defines the following aspects of your web service inside its definitions element:

Any custom data types defined by your web service. These go into complexType elements of a types list.

The formats of the messages sent and received by your web service; that is, the signatures and return values of the functions your web service defines. These are defined in a series of message elements, and may make reference to any custom data types you defined earlier.

The names of the functions your web service provides, along with the input and output messages expected by each. This is in the portType element, which contains an operation element for each of the web service’s functions.

A binding of your web service’s functions to a specific protocol — that is, HTTP. For simple SOAP applications, this section is an exercise in redundancy: You end up just listing all of your functions again. It exists because SOAP is protocol-independent; you need to explicitly state that you’re exposing your methods over HTTP. This goes in the binding element.

Finally, the URL to your web service. This is defined in the service element.

531

TEAM LinG

Chapter 21

Here’s BittyWiki.wsdl, a WSDL file for the SOAP API exposed by BittyWiki:

<?xml version=”1.0”?> <definitions name=”BittyWiki”

targetNamespace=”urn:BittyWiki”

xmlns:xsd=”http://www.w3.org/2001/XMLSchema”

xmlns:soap=”http://schemas.xmlsoap.org/wsdl/soap/”

xmlns=”http://schemas.xmlsoap.org/wsdl/”>

<!--Descriptions of the functions exposed by the BittyWiki API. The definitions of the functions reference message elements which will be defined afterwards.-->

<portType name=”BittyWikiPortType”> <operation name=”getPage”>

<input message=”sendPageName”/> <output message=”getPageText”/>

</operation>

<operation name=”save”>

<input message=”sendPageNameAndText”/> <output message=”getStatusMessage”/>

</operation>

<operation name=”delete”>

<input message=”sendPageName”/> <output message=”getStatusMessage”/>

</operation>

</portType>

The WSDL parser now knows which functions are exposed by BittyWiki, but nothing about the signatures or return types of those functions. Those come next:

<!--Descriptions of the method signatures used by the BittyWiki API.

For instance, this first one is for a method where you send in a page name. This method signature is common to getPage() and delete().-->

<message name=”sendPageName”>

<part name=”pageName” type=”xsd:string”/> </message>

<message name=”sendPageNameAndText”>

<part name=”pageName” type=”xsd:string”/> <part name=”pageText” type=”xsd:string”/>

</message>

<!--Descriptions of the possible return values obtained from the BittyWiki API. The first one is for a return value that contains a wiki page’s markup: that is, the return value of getPage().--> <message name=”getPageText”>

<part name=”pageText” type=”xsd:string”/> </message>

<message name=”getStatusMessage”>

<part name=”message” type=”xsd:string”/>

</message>

Web Applications and Web Services

A rather redundant section follows, as the four SOAP functions are bound to SOAP-over-HTTP:

<!--A binding of the BittyWiki API functions (previously defined only in the abstract) to the specific “SOAP-over-HTTP” protocol.--> <binding type=”BittyWikiPortType” name=”BittyWikiSOAPBinding”>

<soap:binding style=”rpc” transport=”http://schemas.xmlsoap.org/soap/http” /> <operation name=”getPage”>

<input><soap:body use=”literal” namespace=”urn:BittyWiki” /></input> <output><soap:body use=”literal” namespace=”urn:BittyWiki” /></output> </operation>

<operation name=”save”>

<input><soap:body use=”literal” namespace=”urn:BittyWiki” /></input> <output><soap:body use=”literal” namespace=”urn:BittyWiki” /></output> </operation>

<operation name=”delete”>

<input><soap:body use=”literal” namespace=”urn:BittyWiki” /></input> <output><soap:body use=”literal” namespace=”urn:BittyWiki” /></output> </operation>

</binding>

Finally, the code to let WSDL know where to find the BittyWiki web service:

<!--A link to the BittyWiki web service on the web. It uses the BittyWiki API defined in BittyWikiPortType, as realized by its SOAP-over-HTTP binding, BittyWikiSOAPBinding.-->

<service name=”BittyWiki”>

<port name=”BittyWikiPort” binding=”BittyWikiSOAPBinding”>

<soap:address location=”http://localhost:8002/”/> </port>

</service>

</definitions>

The BittyWiki API doesn’t define any custom data types, so there’s no types element in its WSDL file. If you want to see a types element that has some complexTypes, look at the WSDL file for the Google Web APIs.

WSDL is pretty complicated: That WSDL file is bigger than the Python script implementing the web service it describes. WSDL files are usually generated from the corresponding web service source code, so that humans don’t have to specify them. It’s not possible to do this from Python code because a big part of WSDL is defining the data types, and Python functions don’t have predefined data types. Both the SOAPpy and ZSI libraries can parse WSDL (in fact, they share a WSDL library: wstools), but there’s not much in the way of Python-specific resources for generating WSDL.

The following looks more or less like the previous example of BittyWiki manipulation through direct SOAP calls:

533

TEAM LinG

Chapter 21

>>>import SOAPpy

>>>proxy = SOAPpy.WSDL.Proxy(open(“BittyWiki.wsdl”))

>>>proxy.getPage(“SOAPViaWSDL”)

<Fault SOAP-ENV:Server: Method urn:BittyWiki:getPage failed.: __main__.NoSuchPage SOAPViaWSDL>

Traceback (most recent call last):

...

SOAPpy.Types.faultType: <Fault SOAP-ENV:Server: Method urn:BittyWiki:getPage failed.: __main__.NoSuchPage SOAPViaWSDL>

>>>proxy.save(“SOAPViaWSDL”, “This page created through SOAP via WSDL.”) ‘Page saved.’

>>>proxy.getPage(“SOAPViaWSDL”)

‘This page created through SOAP via WSDL.’

The main difference here is that going through WSDL will stop you from calling web service methods that don’t exist:

>>>proxy.noSuchMethod() Traceback (most recent call last):

...

AttributeError: noSuchMethod

>>>server = SOAPpy.SOAPProxy(“http://localhost:8002/”, “urn:BittyWiki”)

>>>server.noSuchMethod()

<Fault SOAP-ENV:Client: No method urn:BittyWiki:noSuchMethod found: exceptions.AttributeError BittyWikiAPI instance has no attribute ‘noSuchMethod’> Traceback (most recent call last):

...

SOAPpy.Types.faultType: <Fault SOAP-ENV:Client: No method urn:BittyWiki:noSuchMethod found: exceptions.AttributeError BittyWikiAPI instance has no attribute ‘noSuchMethod’>

Both attempts to call noSuchMethod raised an exception, but going through WSDL meant the problem was caught on the local machine instead of the server. This ability is a lot more interesting to a compiled language: WSDL makes it possible to apply the same compile-time checks to web service calls as to local function calls.

Choosing a Web Service Standard

This chapter described three standards for web services, each with a different philosophy, each with advantages and drawbacks. REST aims to get the most out of the facilities provided by HTTP, but it lacks a standard encoding for even simple data types. XML-RPC provides that encoding, but it’s verbose and only deals with simple data types and compositions of simple data types. SOAP offers the structured data types of XML-RPC with the flexibility of REST, but its added complexity makes hard cases more difficult to understand than if they’d just been implemented with REST.

Industry trends favor REST and SOAP over XML-RPC. SOAP has the backing of large software companies such as IBM and Microsoft; REST has the backing of independent web service users and developers.

Web Applications and Web Services

That’s because APIs based around REST and XML-RPC are generally easier to learn and use. Whenever web services expose the same API through different protocols, the simplest one generally wins. For instance, Amazon exposes a SOAP API in addition to the REST API covered in this chapter, but about 80 percent of its users choose REST over SOAP.

Which should you choose? Well, if you were a big fan of large software companies like IBM and Microsoft, you probably wouldn’t be using Python in the first place. You would be using Java or .NET: two strongly typed languages with good SOAP tool support. In most cases, the extra functionality of SOAP isn’t needed, and Python’s support for SOAP isn’t consummate with the added complexity, so why choose it unnecessarily?

You should start off by planning to expose a well-designed REST or XML-RPC API. If, during the design or implementation stage, you start running into problems with your choice, look into using SOAP. Unless you’re doing heavy-duty automatic business process software, or interfacing with a statically typed language like Java or .NET, you’ll probably be able to see the REST or XML-RPC API through to the end. Your users will thank you for the simpler interface.

My ideal web service would have a RESTful interface in which each resource could accept POST data in the format defined by XML-RPC (or some simple subset of SOAP). The web service could then be designed along REST principles, but some variant of xmlrpclib or SOAPpy could be used to marshal and unmarshal the data without requiring the creation of custom parsers.

Whatever you choose, please try to keep web services in mind from the moment you begin the design: A web service is just a web application for robots. If you want your application to inspire creativity and not just meet a predefined need, you must give up some of the control to your users.

Web Service Etiquette

A web service may have users who skirt the rules, or administrators who feel the users are ungrateful for the service they’re being provided. In the interests of harmony, here are a few basic pieces of advice for managing the social aspect of web services.

For Consumers of Web Services

If you write a robot to consume someone else’s web services, it’s important to play by the rules. In particular, don’t try to evade any limitations such as license keys or daily limits on your access to the API. Access to a web service is a privilege, not a right. It’s better to run out of API calls and have to complete a task later than you planned than to have your access taken away altogether.

For Producers of Web Services

If you’re planning to expose your web application through a web service, you need to consider the flip side of these issues. If your audience is already scripting your application, you’ve got a leg up because you don’t have to guess what people might do with it. Before you design your web services, poll your robot-writing users to see what parts of your application they’re using. Make your web services available on terms that allow users to move over to the new system, or they’ll have no incentive to switch.

535

TEAM LinG

Chapter 21

As producer of a public web service, you might feel like the burden of etiquette falls completely on your users. After all, you’re providing a service to them and not expecting anything in return. Nonetheless, it’s important to make your terms of use palatable because the people writing the robots have the final advantage: So long as you provide a web application with the same functionality as the web service, determined users can always write a robot to use the web application however they want. There’s no foolproof way you can distinguish between a robot that uses your site and the web browser a human might use to use your site. They’re both pieces of software running on someone’s computer, making an HTTP request. All the HTTP headers, including the User-Agent and the authentication headers, can be forged by a robot.

That said, if a particular robot is causing you trouble, you can solve the problem with the same tools you’d use against a troublesome human user. In particular, the Apache module mod_throttle is useful for dealing with users who are taking more than their fair share of resources.

Download mod_throttle at www.snert.com/Software/mod_throttle/. Note that as of this writing, it’s only available for Apache 1.3.

Using Web Applications as Web Services

It’s possible to write scripts that consume web applications as though they were web services. After all, that’s how the idea of web services got started in the first place. Some sites still haven’t gotten the web services religion, or might have web services that don’t expose the functionality you need. To write the robot you have in mind, you’d have to go through the application.

This chapter doesn’t cover how to write such scripts, but the general principles are similar to web services; and if this topic interests you, you’ll eventually find yourself doing it. When you do, don’t do anything that violates the site’s terms of service. In addition, don’t access the site more than a human user would. If you can, run your script in off hours so you don’t add to the load on the system. Finally, ask the site administrators for a web service interface so you can work against a more stable interface that uses less bandwidth.

A Sampling of Publicly Available

Web Services

Many popular web applications are also available as web services. The two most popular are Amazon Web Services and the Google Web API (both covered in this chapter). However, most web services are made available by smaller companies as differentiators, or as an extension of noncorporate or volunteer services. Some of the most popular Web APIs fall into this category: the Technorati web API, which acts as a near-real-time search for weblogs, and the API to the social bookmarking site del.icio.us.

Most web services are made available under official or informal restrictions. Often, users are asked to sign up for a license key, and each user is allowed only a certain number of API calls per unit time (as mentioned earlier, the Google and Amazon APIs work this way). Some web services, such as eBay’s, are available only by paid subscription.

Web Applications and Web Services

Knowledge of public web services is a useful thing to have in your programmer’s toolkit. Some are mere toys, but others can be very useful if you’ve got an idea but not the data set you would need to realize it. You can also follow their examples when exposing your web applications as web services. The following table describes some of the more interesting specimens, as well as some sites that aggregate all manner of public web services.

537

TEAM LinG

Chapter 21

Summar y

Web applications are powerful and popular; with Python, they’re also easy to write. The REST architecture made the web usable and successful: Employing it when designing your application gives you a head start. Web applications are designed for humans; a web service is just a web application designed for use by software scripts instead. Expose REST and XML-RPC web services for simplicity and easy adoption, SOAP for heavy-duty tasks or when interfacing with Java or .NET applications. Make use of the web services provided by others: They’re opening up their data sets and algorithms for your benefit.

Exercises

1.What’s a RESTful way to change BittyWiki so that it supports hosting more than one Wiki?

2.Write a web application interface to WishListBargainFinder.py. (That is, a web application that delegates to the Amazon Web Services.)

3.The BittyWiki API makes the raw wiki markup of a page available when you GET a page (REST interface) or call the getPage method (XML-RPC and SOAP interfaces). Suppose your web service users start wanting to get BittyWiki pages rendered as HTML fragments so that they don’t have to write their own transformations from BittyWiki markup to HTML. What’s a RESTful way to solve the problem? What’s a solution that’s more in keeping with the XML-RPC or SOAP philosophies?

4.The wiki search-and-replace spider looks up every new WikiWord it encounters to see whether it corresponds to a page of the wiki. If it finds a page by that name, that page is processed. Otherwise, nothing happens and the spider has wasted a web service request. How could the web service API be changed so that the spider could avoid those extra web service requests for nonexistent pages?

5.Suppose that, to prevent vandalism, you change BittyWiki so that pages can’t be deleted. Unfortunately, this breaks the wiki search-and-replace spider, which sometimes deletes a page before recreating it with a new name. What’s a solution that meets both your needs and the needs of the spider’s users?

22

Integrating Java with Python

Java is an object-oriented programming language. Java programs are compiled from source code into byte codes. The Java runtime engine, called a Java virtual machine, or JVM, runs the compiled byte codes. Sound familiar? At an abstract level at least, Java and Python are very similar. Like Java, Python programs are compiled into byte codes, although this can be done at runtime.

Despite these similarities, there are some differences between the languages:

With Python, you can run scripts directly from the source code. Compiling is optional. If you don’t compile your Python code in advance, the python command will take care of this for you.

Java syntax is based on C and C++, two very popular programming languages. This makes it easy for developers using C++ to migrate to Java. Consequently, Java is considered a more serious and businesslike language than Python.

Python syntax is very simple and easy to learn, but the syntax has diverged far from C.

With its simple syntax and built-in support for lists, dictionaries, and tuples, you’ll find Python code much easier to write than Java code. Generally, Python programs require a lot less code than the corresponding Java code.

Java has an advantage over Python in terms of standard APIs, though. The base Java language includes a mature database API, an API for parsing XML documents, an API for remote communication, and even an API to access LDAP directory servers. You can do all of this in Python, but Python lacks the richness, and standardization, of the many Java APIs. This becomes more apparent when you write enterprise applications in Python. Java’s enterprise APIs, called J2EE, enable Java to be a player in the enterprise market. Python, unfortunately, has been relegated to a minimal role in the enterprise market.

When writing enterprise applications, you’ll likely need to write them in Java. Even though Python can work well in this space, as shown in Chapters 20 and 21, Java controls the mind share for the enterprise. Luckily, you can get the best of both worlds with Jython, an implementation of Python in Java.

TEAM LinG

Chapter 22

Jython enables you to execute Python code from within a Java virtual machine — that is, from within any Java application. This chapter covers that topic, including the following:

Reasons for scripting within Java applications

Comparing Jython with the regular C-based Python implementations

Installing Jython

Running Python scripts from Jython

Calling Java code from Python scripts

Extending Java classes with Python classes

Writing J2EE Servlets in Python

Embedding the Jython interpreter in your Java applications

Note that you’ll want to have some familiarity with both Java and Python to be able to integrate Python and Java.

Scripting within Java Applications

Most software developers consider Java to be a large systems programming language, a serious language for serious tasks. Python, in this way of thinking, comes from the realm of scripting languages such as Perl and Tcl. As such, many developers typically don’t respect Python because scripting languages are, of course, created for people who cannot program. You know this isn’t true, but the split between programming and scripting languages remains, even though Python gracefully bridges this gap.

Despite this lack of respect, scripting languages have proven to be very productive and are widely deployed as critical parts of companies small and large (and huge and gigantic, too). You can generally accomplish a lot more in less time with less code using a scripting language than you can with a system programming language like Java.

With Java applications, scripting comes in handy for a number of reasons, including the following:

The scripting language can act as a macro extension language. Much like Visual Basic for Applications (VBA) enables you to script extensions to Microsoft Office, you can enable users to extend your own Java applications using Jython. Complex text editors such as jEdit (www.jedit.org) enable you to write scripts in this fashion.

Use Jython to speed the development of Java applications. As a high-level scripting language, you can take advantage of the productivity features of Python when compared to the complexity of Java.

Explore and debug running systems. Using the interactive capabilities of Jython, you can explore a running Java application. You can execute code as needed, all interactively. You already take this for granted in Python, but it’s something that Java just doesn’t have.

You can script unit tests much faster than writing them in Java. Many organizations feel uncomfortable about introducing scripting languages, especially open-source ones. Using scripts for