- •Credits
- •About the Authors
- •About the Reviewers
- •www.PacktPub.com
- •Table of Contents
- •Preface
- •Introduction
- •Installing Groovy on Windows
- •Installing Groovy on Linux and OS X
- •Executing Groovy code from the command line
- •Using Groovy as a command-line text file editor
- •Running Groovy with invokedynamic support
- •Building Groovy from source
- •Managing multiple Groovy installations on Linux
- •Using groovysh to try out Groovy commands
- •Starting groovyConsole to execute Groovy snippets
- •Configuring Groovy in Eclipse
- •Configuring Groovy in IntelliJ IDEA
- •Introduction
- •Using Java classes from Groovy
- •Embedding Groovy into Java
- •Compiling Groovy code
- •Generating documentation for Groovy code
- •Introduction
- •Searching strings with regular expressions
- •Writing less verbose Java Beans with Groovy Beans
- •Inheriting constructors in Groovy classes
- •Defining code as data in Groovy
- •Defining data structures as code in Groovy
- •Implementing multiple inheritance in Groovy
- •Defining type-checking rules for dynamic code
- •Adding automatic logging to Groovy classes
- •Introduction
- •Reading from a file
- •Reading a text file line by line
- •Processing every word in a text file
- •Writing to a file
- •Replacing tabs with spaces in a text file
- •Deleting a file or directory
- •Walking through a directory recursively
- •Searching for files
- •Changing file attributes on Windows
- •Reading data from a ZIP file
- •Reading an Excel file
- •Extracting data from a PDF
- •Introduction
- •Reading XML using XmlSlurper
- •Reading XML using XmlParser
- •Reading XML content with namespaces
- •Searching in XML with GPath
- •Searching in XML with XPath
- •Constructing XML content
- •Modifying XML content
- •Sorting XML nodes
- •Serializing Groovy Beans to XML
- •Introduction
- •Parsing JSON messages with JsonSlurper
- •Constructing JSON messages with JsonBuilder
- •Modifying JSON messages
- •Validating JSON messages
- •Converting JSON message to XML
- •Converting JSON message to Groovy Bean
- •Using JSON to configure your scripts
- •Introduction
- •Creating a database table
- •Connecting to an SQL database
- •Modifying data in an SQL database
- •Calling a stored procedure
- •Reading BLOB/CLOB from a database
- •Building a simple ORM framework
- •Using Groovy to access Redis
- •Using Groovy to access MongoDB
- •Using Groovy to access Apache Cassandra
- •Introduction
- •Downloading content from the Internet
- •Executing an HTTP GET request
- •Executing an HTTP POST request
- •Constructing and modifying complex URLs
- •Issuing a REST request and parsing a response
- •Issuing a SOAP request and parsing a response
- •Consuming RSS and Atom feeds
- •Using basic authentication for web service security
- •Using OAuth for web service security
- •Introduction
- •Querying methods and properties
- •Dynamically extending classes with new methods
- •Overriding methods dynamically
- •Adding performance logging to methods
- •Adding transparent imports to a script
- •DSL for executing commands over SSH
- •DSL for generating reports from logfiles
- •Introduction
- •Processing collections concurrently
- •Downloading files concurrently
- •Splitting a large task into smaller parallel jobs
- •Running tasks in parallel and asynchronously
- •Using actors to build message-based concurrency
- •Using STM to atomically update fields
- •Using dataflow variables for lazy evaluation
- •Index
Chapter 9
See also
ff A more sophisticated implementation of the previous DSL is put into practice by the Groovy SSH DSL project available at https://github.com/aestasit/groovy- ssh-dsl. It supports SCP operations, tunneling, key-based authentication, and many other useful features.
ff Specifics of the SSH implementation are not covered in this recipe since most of the functionality is delegated to the JSch library, which can be found at
http://www.jcraft.com/jsch/.
DSL for generating reports from logfiles
In this recipe, we will give another DSL example for constructing a simple configuration language for the analysis of logfiles, and the generation of reports based on the content of such logfiles. The technique used in this recipe is similar to the one used in the recipe DSL for executing commands over SSH.
Getting ready
Let's consider having the following performance log data:
execution of getCustomerName took 244ms execution of getCustomerName took 144ms execution of getAccountNumber took 44ms execution of getCustomerName took 244ms execution of getCustomerName took 24ms execution of getAccountNumber took 112ms execution of getCustomerName took 200ms execution of getCustomerName took 22ms
...
The goal is to calculate the average and total times spent on each method. Of course, we could have written a very simple script to reach the same result, but our purpose is to create a DSL that will allow parsing any arbitrary logfile format and extract both grouped
and aggregated numeric information from it. A reasonable DSL may look like the following code snippet:
format '^execution of (\\w+) took (\\d+)ms$' column 1, 'methodName'
column 2, 'duration'
source('PerformanceData2012') { localFile 'log1.log' localFile 'log2.log'
}
327
www.it-ebooks.info
Metaprogramming and DSLs in Groovy
report('Duration') { avg 'duration'
sum 'duration' groupBy 'methodName'
}
We will try to define the language exactly like this example.
The first expression defines a log line format; then we define a regular expression group mapping to column names, which are used later to refer to log data inside the report definition. The report definition contains a list of calculated values (average of duration and sum of duration) and a column to group report data by. Another important component of the
DSL is the definition of the data source.
How to do it...
To define our internal DSL, we first need to define its building blocks, that is, the data structures that compose our mini language:
1.The first step is to define the report data structure: class Report {
def name
def sumColumns = [] as Set def avgColumns = [] as Set def groupByColumns = [] as Set
Report(String name) { this.name = name
}
void sum(String columnName) { sumColumns << columnName
}
void avg(String columnName) { avgColumns << columnName
}
void groupBy(String columnName) { groupByColumns << columnName
}
}
328
www.it-ebooks.info
Chapter 9
2.We also define the data source structure: class Source {
def name
def files = [] as Set
Source(String name) { this.name = name
}
void localFile(File file) { if (file) {
files << file.absoluteFile.canonicalFile
}
}
void localFile(String file) { localFile(new File(file))
}
}
3.Then we compose a common configuration object, which hold sources, reports, format, and column mapping data:
class Configuration {
def format
private final columnNames = [:] private final columnIndexes = [:] private final sources = [:] private final reports = [:]
private static int sourceCounter = 0 private static int reportCounter = 0
void format(String format) { this.format = format
}
void column(int group, String name) { columnNames[group] = name columnIndexes[name] = group
}
329
www.it-ebooks.info
Metaprogramming and DSLs in Groovy
void source(Closure cl) {
def generatedName = "source${sourceCounter++}" source(generatedName, cl)
}
void source(String name, Closure cl) { Source source = new Source(name) cl.delegate = source
cl.resolveStrategy = Closure.DELEGATE_FIRST cl()
sources[name] = source
}
void report(Closure cl) {
def generatedName = "report${reportCounter++}" report(generatedName, cl)
}
void report(String name, Closure cl) { Report report = new Report(name) cl.delegate = report
cl.resolveStrategy = Closure.DELEGATE_FIRST cl()
reports[name] = report
}
}
4.The final step is to define the engine class that will glue together the configuration creation and actual report generation:
class LogReportDslEngine {
void process(Closure cl) {
Configuration config = new Configuration() cl.delegate = config
cl.resolveStrategy = Closure.DELEGATE_FIRST cl()
config.sources.values().each { Source source -> config.reports.values().each { Report report ->
// Collect report data. def reportData = [:]
source.files.each { File sourceFile ->
330
www.it-ebooks.info
Chapter 9
sourceFile.eachLine { String line ->
// Match the data line.
if (line =~ config.format) {
def fields = (line =~ config.format)[0]
// Map column names
def fieldMap = fields.collect {}
//Generate group key, for which
//to aggregate the data.
def group = report.groupByColumns
.collect { fields[config.columnIndexes[it]]
}.join(', ')
//Create empty group record
//if it does not exist. reportData[group] =
reportData[group] ?: emptyRecord
//Calculate report values for given key. def g = reportData[group] report.avgColumns.each { String column ->
def fieldIndex = config.columnIndexes[column]
g['avg'][column] = g['avg'][column] ?: 0 g['avg'][column] +=
fields[fieldIndex].toDouble()
}
report.sumColumns.each { String column -> def fieldIndex =
config.columnIndexes[column] g['sum'][column] = g['sum'][column] ?: 0 g['sum'][column] +=
fields[fieldIndex].toDouble()
}
g['count'] += 1
}
}
}
331
www.it-ebooks.info
Metaprogramming and DSLs in Groovy
// Produce report output.
def reportName = "${source.name}_${report.name}" def reportFile = new File("${reportName}.report") reportFile.text = ''
reportData.each { key, data -> reportFile <<
"Report for $key\n" reportFile <<
" Total records: ${data['count']}\n" data['avg'].each { column, value ->
reportFile <<
" Average of ${column} is " + "${value / data['count']}\n"
}
data['sum'].each { column, value -> reportFile <<
"Sum of ${column} is ${value}\n"
}
}
}
}
}
def getEmptyRecord() {
[count: 0, avg: [:], sum: [:]]
}
}
5.At this point, you are ready to use the DSL internally from your Groovy code: def engine = new LogReportDslEngine()
engine.process {
format '^execution of (\\w+) took (\\d+)ms$'
column 1, 'methodName' column 2, 'duration'
source('PerformanceData2012') { localFile 'log1.log'
332
www.it-ebooks.info
Chapter 9
localFile 'log2.log'
}
source('PerformanceData2013') { localFile 'log3.log' localFile 'log4.log'
}
report('Duration') { avg 'duration' sum 'duration'
groupBy 'methodName'
}
}
6.The previous script will produce two report files (one for each data source), named
PerformanceData2012_Duration.report and PerformanceData2013_ Duration.report. The report will look approximately like the following example:
Report for getCustomerName
Total records: 12
Average of duration is 179.0
Sum of duration is 2148.0
Report for getAccountNumber
Total records: 4
Average of duration is 64.0
Sum of duration is 256.0
How it works...
The Source and Report classes defined previously are simple structures holding information needed to build the reports; therefore, we will not spend any time on them.
The Configuration class is a bit more involved because it makes use of closure delegates (similar to the DSL for executing commands over SSH recipe).
The Configuration object is also constructed through a closure delegate inside the process method of the LogReportDslEngine class. After the configuration closure is executed, we get back a fully constructed data structure, which we are ready to use for further processing.
333
www.it-ebooks.info
Metaprogramming and DSLs in Groovy
The code executed after we have a configuration object does the following:
ff ff ff ff
ff
Loops through all data sources, and for each of them
Loops through all report definitions, and for each of them Goes through all source files and reads every line
For each line, it tries to match it against configured format expression and then fills in internal report data array
When the file processing is done, the collected data is printed into a report file
There's more...
Obviously, this DSL implementation is rather primitive and can be extended with many more features such as:
ff DSL validation rules (for example, groupBy columns cannot appear in an aggregated function)
ff More grouping functions (for example, min and max)
ff More data source types (for example, URL, FTP, and JDBC)
ff More column types (for example, date, time, Boolean, and IP address)
334
www.it-ebooks.info