8.5Groovy and NoSQL databases

One of the most interesting trends in software development in the past few years15 has been the growth of alternative, non-relational databases. The generic term NoSQL (which the majority of the community interpret as “Not Only SQL”) refers to a range of schema-less databases that are not based on relational approaches.

The subject of NoSQL databases is already large and rapidly growing, and it’s well beyond the scope of this book. But many of the databases have a Java API, and some of them also have Groovy wrappers that simplify them.

One of the most interesting is MongoDB,16 whose Java API is rather awkward but is dramatically improved through a Groovy wrapper called GMongo. The GMongo project, whose GitHub repository is located at https://github.com/poiati/gmongo, is the product of Paulo Poiati and is the subject of this section.

MongoDB is a document-oriented database that stores its data in binary JSON (BSON) format. This makes it perfect for storing data downloaded from RESTful web services, which often produce JSON data on request.

8.5.1Populating Groovy vampires

This example came about because I was wandering in a bookstore recently and noticed that while there was only one bookshelf labeled “Computer,” there were three others labeled “Teen Paranormal Romance.” Rather than lament the decline of Western Civilization I chose to take this as evidence that I needed to add Groovy vampires to my book.

Consider the web service provided by the movie review site Rotten Tomatoes, http://developer.rottentomatoes.com. If you register for an API key, you can make HTTP GET requests that search for movies, cast members, and more. The data is returned in JSON form. The base URL for the API is located at http://api.rottentomatoes

.com/api/public/v1.0. All requests start with that URL.

For example, searching for information about the movie Blazing Saddles17 is done by accessing http://api.rottentomatoes.com/api/public/v1.0/movies.json?q=Blazing %20Saddles&apiKey=... (supply the API key in the URL). The result is a JSON object that looks like the following listing.

Listing 8.15 A portion of the JSON object representing the movie Blazing Saddles

{

"total": 1, "movies": [

{

"id": "13581",

"title": "Blazing Saddles",

15Other than the rise of dynamic languages on the JVM, of course.

16See www.mongodb.org/ for downloads and documentation.

17That’s not a vampire movie, obviously, but the urge to save Mongo in MongoDB is irresistible. “Mongo only pawn in game of life” is a brilliant line and arguably the peak of the Alex Karras oeuvre.

www.it-ebooks.info

"year": 1974, "mpaa_rating": "R", "runtime": 93, "release_dates": {

"theater": "1974-02-07", "dvd": "1997-08-27"

}, "ratings": {

"critics_rating": "Certified Fresh", "critics_score": 89, "audience_rating": "Upright", "audience_score": 89

},

"synopsis": "",

..., "abridged_cast": [

{

"name": "Cleavon Little", "id": "162693977", "characters": [

"Bart"

]

},

{

"name": "Gene Wilder", "id": "162658425", "characters": [

"Jim the Waco Kid"

]

},

...

], "alternate_ids": {

"imdb": "0071230"

},

...

}

In addition to the data shown, the JSON object also has links to the complete cast list, reviews, and more. Another reason to use a database like MongoDB for this data is that not every field appears in each movie. For example, some movies contain a critic’s score and some do not. This fits with the whole idea of a schema-less database based on JSON.

First, to populate the MongoDB I’ll use an instance of the com.gmongo.GMongo class. This class wraps the Java API directly. In fact, if you look at the class in GMongo.groovy, you’ll see that it consists of

class GMongo {

@Delegate Mongo mongo

// ... Constructors and other methods ...

}

www.it-ebooks.info

There follow various constructors and simple patch methods. The @Delegate annotation from Groovy is an Abstract Syntax Tree (AST) transformation18 that exposes the methods in the com.mongodb.Mongo class, which comes from the Java API, through GMongo. The AST transformation means you don’t need to write all the delegate methods by hand.

Initializing a database is as simple as

GMongo mongo = new GMongo() def db = mongo.getDB('movies') db.vampireMovies.drop()

MongoDB uses movies as the name of the database, and collections inside it, like vampireMovies, are properties of the database. The drop method clears the collection.

Searching Rotten Tomatoes consists of building a GET request with the proper parameters. In this case, the following code searches for vampire movies:

String key = new File('mjg/rotten_tomatoes_apiKey.txt').text

String base = "http://api.rottentomatoes.com/api/public/v1.0/movies.json?" String qs = [apiKey:key, q:'vampire'].collect { it }.join('&')

String url = "$base$qs"

The API key is stored in an external file. Building the query string starts with a map of parameters, which is transformed into a map of strings of the form “key=value” and then joined with an ampersand. The full URL is then the base URL with an appended query string. Getting the movies and saving them into the database is almost trivial:

def vampMovies = new JsonSlurper().parseText(url.toURL().text) db.vampireMovies << vampMovies.movies

The JsonSlurper receives text data in JSON form from the URL and converts it to JSON objects. Saving the results into the database is as simple as appending the whole collection.

The API has a limit of 30 results per page. The search results include a property called next that points to the next available page, assuming there is one. The script therefore needs to loop that many times to retrieve the available data:

def next = vampMovies?.links?.next while (next) {

println next

vampMovies = slurper.parseText("$next&apiKey=$key".toURL().text) db.vampireMovies << vampMovies.movies

next = vampMovies?.links?.next

}

That’s all there is to it. Using a relational database would require mapping the movie structure to relational tables, which would be a bit of a challenge. Because MongoDB uses BSON as its native format, even a collection of JSON objects can be added with no work at all.

18Discussed in chapter 4 on integration and in appendix B, “Groovy by Feature,” and used in many other places in this book.

www.it-ebooks.info

Figure 8.5 A portion of the vampire movies database, using the MonjaDB plugin for Eclipse

There’s an Eclipse plugin, called MonjaDB, which connects to MongoDB databases. Figure 8.5 shows a portion of the vampireMovies database.

8.5.2Querying and mapping MongoDB data

Now that the data is in the database I need to be able to search it and examine the results. This can be done in a trivial fashion, using the find method, or the data can be mapped to Groovy objects for later processing.

The find method on the collection returns all JSON objects satisfying a particular condition. If all I want is to see how many elements are in the collection, the following suffices:

println db.vampireMovies.find().count()

With no arguments, the find method returns the entire collection. The count method then returns the total number.

Mapping JSON to Groovy brings home the difference between a strongly typed language, like Groovy, and a weakly typed language, like JSON. The JSON data shown is a mix of strings, dates, integers, and enumerated values, but the JSON object has no embedded type information. Mapping this to a set of Groovy objects takes some work.

For example, the following listing shows a Movie class that holds the data in the JSON object.

Listing 8.16 Movie.groovy, which wraps the JSON data

@ToString(includeNames=true) class Movie {

long id String title int year

MPAARating mpaaRating int runtime

String criticsConsensus

www.it-ebooks.info

Map releaseDates = [:] Map<String, Rating> ratings = [:] String synopsis

Map posters = [:]

List<CastMember> abridgedCast = [] Map links = [:]

}

The Movie class has attributes for each contained element, with the data types specified. It contains maps for the release dates, posters, ratings, and additional links, and a list for the abridged cast. A CastMember is just a POGO:

class CastMember { String name long id

List<String> characters = []

}

A Rating holds a string and an integer:

class Rating { String rating int score

}

Just to keep things interesting, the MPAA rating is a Java enum, though it could just as easily have been implemented in Groovy:

public enum MPAARating {

G, PG, PG_13, R, X, NC_17, Unrated

}

Converting a JSON movie to a Movie instance is done through a static method in the Movie class. A portion of the fromJSON method is shown in the next listing.

Listing 8.17 A portion of the method that converts JSON movies to Movie instances

static Movie fromJSON(data) { Movie m = new Movie() m.id = data.id.toLong() m.title = data.title

m.year = data.year.toInteger() switch (data.mpaa_rating) {

case 'PG-13' : m.mpaaRating = MPAARating.PG_13; break case 'NC-17' : m.mpaaRating = MPAARating.NC_17; break default :

m.mpaaRating = MPAARating.valueOf(data.mpaa_rating)

}

m.runtime = data.runtime

m.criticsConsensus = data.critics_consensus ?: ''

The complete listing can be found in the book source code but isn’t fundamentally different from what’s being shown here.

www.it-ebooks.info

A test to prove the conversion is working is shown in the following listing.

Listing 8.18 A JUnit test to verify the JSON conversion

Lessons learned (NoSQL19)

1NoSQL databases like MongoDB, Neo4J, and Redis are becoming quite common for specific use cases.

2Most NoSQL databases make a Java-based API available, which can be called directly from Groovy.

3Often a Groovy library will be available that wraps the Java API and simplifies it. Here, GMongo is used as an example.

Once the mapping works, finding all vampire movies that have a critic’s consensus is as simple as the following script:19

GMongo mongo = new GMongo() def db = mongo.getDB('movies')

db.vampireMovies.find([critics_consensus : ~/.*/]).each { movie -> println Movie.fromJSON(movie)

}

19 NoSQL version of Worst SQL Joke Ever Told: DBA walks into a NoSQL bar; can’t find a table, so he leaves.

www.it-ebooks.info