4.6 Data migration

Real production database systems are seldom isolated systems without interfaces to other database systems. Interfaces for both reading the data from outside world or writing the data to other systems might be required either in the initial setup of a system, on regular basis with sparse intervals or on an almost continuous basis.

A set of typical data migration options are illustrated in Figure 4-8.

B B

Log or trigger-based continuous replication

Figure 4-8 Typical data migration options with external database systems

For data transfer between separate databases, IBM solidDB provides the following capabilities:

CDC replication provided as part of the Universal Cache product provides a subscription-based mechanism to enable initial data synchronization of any table configured to be part of a subscription and provide a mechanism for continuous near real-time synchronization of table data. This mechanism is available with all databases supported by CDC, such as DB2, IDS, Oracle, Sybase, and SQL Server.

IBM solidDB has a set of simple ASCII-based export and import utilities to create ascii files on data content of individual tables and load them to a solidDB database. The ASCII files can be used in interfacing with other database servers.

96 IBM solidDB: Delivering Data with Extreme Speed

In certain cases with specific requirements for timeliness and transactionality of the data interface, the most practical approach is to design an interface application for reading the data from one database and writing it to the other. The interface application can be based on several mechanisms, such as the following mechanisms:

–Running regular selects on the source database and write operations on the target database

–Waiting for triggers to fire on the source database upon write operations, receive notifications and write the operations on the target database.

–Using an API to access database transaction log and write the log entries to another database. In IBM solidDB this mechanism is possible through read operations in virtual table SYS_LOG.

Additional mechanisms exist for data transfers between IBM solidDB instances, such as copying the entire database file either by backup or a HotStandby netcopy operation or by using IBM solidDB Advanced replication for subscription-based data transfer between databases. However, discussion of these mechanisms are outside the scope of the chapter.

Choosing the right migration mechanism depends on several criteria:

One-time operation versus a regular or continuous basis operation

For one-time operations, the simplicity of implementation is crucial for avoiding costs. Performance and recoverability have usually limited importance in those cases. Some clients have decided to use CDC only for initial data load, without the need for continuous synchronization. Transferring individual tables through ASCII-based import and export tools is simple, but the complexity and vulnerability to manual errors increases when the number of tables increases. Implementing a specialized application is usually too costly for one-time operations.

For continuous operations, that need to migrate data on a near real-time basis, the only practical options are CDC-based replication or specialized interface application based on triggers or Log API.

Data volumes and performance criteria

For high data volumes or situations where the data must be quickly transferred between databases, CDC-based solutions are too slow because configuring them to work on parallel threads is not possible. It is possible to run parallel ASCII-based import and export tasks. Also, implementing a specialized application can enable optimization for exact requirements.

Chapter 4. Deploying solidDB and Universal Cache 97

Need for recoverability during data transfer

Often having relaxed requirements for the data durability during data transfer is possible. This is particularly true in the case of initial data loads, when the entire task can be re-executed from the beginning. The relaxed requirements, such as running the target system without transactional logging, can help in speeding up all import mechanisms.

Need for transactionality

In some cases, transactionality for migrated data is a must requirement. Usually, this means that the data arriving at the target system is expected to follow the transaction boundaries as they were when the data was originally written to the source system. Generally, this requirement is impossible to follow with table-based mechanism such as import or export. CDC-based replication and tailored interface applications running on triggers or Log API can enable transactional data migration.

Simplicity of implementation and deployment

The simplest mechanisms are ASCII-based export and import tools and CDC-based replication, assuming CDC is already installed.

Need to process data migration

In cases where table structures in source and target databases are not completely similar, some data processing is required for converting the data to fit the table structure of the target database. This task often requires implementing a specialized application to some level of the architecture or the clever use of views in the target database. The following list describes typical ways for doing the conversion:

–Define a set of views in the source database to match the table structure of the target database. Export the data in views and run a regular import.

–Write a specialized application that runs selects in the source database and populates the target database accordingly.

–Create a set of working tables in the target database that follow the table definitions in source database and transfer the data in the most applicable method. Implement a specialized application that reads the working tables to populate the target tables. This option is logically similar to the previous option, but the application can be implemented in procedure language and can be made fully transactional.

98 IBM solidDB: Delivering Data with Extreme Speed