com.univocity.api.engine
Interface DataIntegrationEngine


public interface DataIntegrationEngine

The DataIntegrationEngine is the central component of uniVocity. With it you can define data mappings between entities of different data stores, execute them, and control their execution.

A DataIntegrationEngine depends on some essential configurations in an EngineConfiguration object and can only be created via a Univocity instance.

A database with a set of tables to store metadata is required for non-trivial mappings. This configuration resides in a MetadataSettings object. In case this is not configured in EngineConfiguration.getMetadataSettings(), the DataIntegrationEngine will be started with an in-memory database. Note, however, that all metadata stored in this in-memory database is lost once the DataIntegrationEngine is shut down (using Univocity.shutdown(String)).

The DataIntegrationEngine manages an internal scope (EngineScope) for variables, constants, FunctionCalls, and DatasetProducers.

IMPORTANT: instances of DataIntegrationEngine are not thread safe. If multiple threads share the same instance, synchronization must be managed externally.

Author:
uniVocity Software Pty Ltd - dev@univocity.com
See Also:
Univocity, EngineConfiguration, MetadataSettings, FunctionCall, DatasetProducer, EngineScope

Method Summary
 void addDataset(String name, Dataset dataset)
          Adds a new Dataset to this engine.
 DatasetProducerSetup addDatasetProducer(EngineScope scope, DatasetProducer producer)
          Adds and configures a DatasetProducer for reading information from an entity accessible through this engine, and producing different datasets as a result.
 void addExpression(EngineScope scope, String name, String expression)
          Associates a variable name to an expression.
<F extends FunctionCall<?,?>>
void
addFunction(EngineScope scope, String name, F function)
          Adds a custom function implementation to this engine.
 void addFunctions(Object... objectsWithFunctions)
          Creates new functions based on the methods annotated with FunctionWrapper.
 void addInterceptor(EngineLifecycleInterceptor interceptor)
          Adds a callback object that intercepts life cycle events produced within this engine.
<F extends Map<?,?>>
void
addMap(String name, F mapFunction)
          Creates a function backed by an implementation of Map.
 QuerySetup addQuery(EngineScope scope, String queryName)
          Configures a query to be executed against a data store as a function.
 void addRowReader(EngineScope scope, String name, Class<? extends RowReader> reader)
          Associates a RowReader implementation class to a name and a scope.
 void addRowReader(String name, RowReader reader)
          Associates a RowReader instance to name.
 void disableUpdateOnAllRecords(String dataEntityName)
          Disables data modifications to all records of this data entity that were already mapped.
 void disableUpdateOnRecords(String dataEntityName, Dataset dataset)
          Disables data modifications on a given set of records of a data entity that were already mapped.
 void enableUpdateOnAllRecords(String dataEntityName)
          Re-enables data modifications on all records of a data entity.
 void enableUpdateOnRecords(String dataEntityName, Dataset dataset)
          Re-enables data modifications on a given set records of a data entity.
 void executeCycle()
          Executes a data mapping cycle with all mappings configured in this engine (i.e.
 void executeCycle(DataIncrement increment)
          Executes a data mapping cycle against a DataIncrement object with all mappings configured in this engine (i.e.
 void executeCycle(DataIncrement increment, String... destinationEntities)
          Executes a data mapping cycle against a DataIncrement object and a selection of destination entities.
 void executeCycle(String... destinationEntities)
          Executes a data mapping cycle against the selected destination entities.
 void executeCycle(Transactions transactionConfig)
          Executes a data mapping cycle with all mappings configured in this engine (i.e.
 void executeCycle(Transactions transactionConfig, DataIncrement increment)
          Executes a data mapping cycle against a DataIncrement object with all mappings configured in this engine (i.e.
 void executeCycle(Transactions transactionConfig, DataIncrement increment, String... destinationEntities)
          Executes a data mapping cycle against a DataIncrement object and a selection of destination entities.
 void executeCycle(Transactions transactionConfig, String... destinationEntities)
          Executes a data mapping cycle against the selected destination entities.
 Export exportEntities(String dataStoreName, String... entityNames)
          Configures an export operation from a given list of data entities.
 Entity getEntity(String entityName)
          Get an accessible data entity which provides direct access to the underlying data store.
 DataStoreMapping getMapping(String sourceDataStore, String destinationDataStore)
          Obtains the mapping configuration object between entities of two data stores.
 String getName()
          The name of the data integration engine.
 DataStoreMapping map(String sourceDataStore, String destinationDataStore)
          Creates a mapping configuration object between entities of two data stores.
 Object readVariable(String name)
          Reads the value of a variable or constant registered to this engine
 void removeInterceptor(EngineLifecycleInterceptor interceptor)
          Removes a EngineLifecycleInterceptor instance from this engine, so it stops being notified of updates in this engine's internal state.
 void removeMapping(String sourceDataStore, String destinationDataStore)
          Removes the mapping between two data stores.
 void setConstant(String name, Object value)
          Adds a constant value that can accessed from any scope.
 void setMappingSequence(String... sequenceOfDestinationEntities)
          Use this method to define the correct sequence of mappings that have to be executed in order to correctly process the default data mapping cycle (started with executeCycle() or executeCycle(DataIncrement)).
 void setPersistentVariable(String name, Object value)
          Sets or adds then initializes a variable in the persistent scope (EngineScope.PERSISTENT).
 void setVariable(String name, Object value)
          Sets or adds then initializes a variable in the current scope.
 

Method Detail

getName

String getName()
The name of the data integration engine. This name is unique and Univocity will always return the same engine instance for a given name.

Returns:
the data integration engine name.

map

DataStoreMapping map(String sourceDataStore,
                     String destinationDataStore)
Creates a mapping configuration object between entities of two data stores. This is the first step in defining data mappings. The DataStoreMapping instance is unique between two data stores and calling this method multiple times with the same data store names will cause and exception.

Parameters:
sourceDataStore - the name of the source data store.
destinationDataStore - the name of the destination data store. Source and destination data stores can be the same.
Returns:
a new DataStoreMapping instance.

getMapping

DataStoreMapping getMapping(String sourceDataStore,
                            String destinationDataStore)
Obtains the mapping configuration object between entities of two data stores.

Parameters:
sourceDataStore - the name of the source data store.
destinationDataStore - the name of the destination data store. Source and destination data stores can be the same.
Returns:
the existing DataStoreMapping instance associated with the given data store names, or null if no such mapping between them exists.

removeMapping

void removeMapping(String sourceDataStore,
                   String destinationDataStore)
Removes the mapping between two data stores. All configuration settings defined in the underlying DataStoreMapping will be lost.

Parameters:
sourceDataStore - the name of the source data store.
destinationDataStore - the name of the destination data store. Source and destination data stores can be the same.

addInterceptor

void addInterceptor(EngineLifecycleInterceptor interceptor)
Adds a callback object that intercepts life cycle events produced within this engine.

Parameters:
interceptor - the callback object that will be notified of this engine's life cycle events.

removeInterceptor

void removeInterceptor(EngineLifecycleInterceptor interceptor)
Removes a EngineLifecycleInterceptor instance from this engine, so it stops being notified of updates in this engine's internal state.

Parameters:
interceptor - the callback object to be removed

addExpression

void addExpression(EngineScope scope,
                   String name,
                   String expression)
Associates a variable name to an expression. When the variable is accessed, the expression will be evaluated and the result will be reused while the scope is active. These values are lost when the scope is deactivated. For example, an expression variable associated with the EngineScope.CYCLE scope won't be accessible if a there is no mapping cycle execution in progress. Attempting to read a variable whose scope is not active will yield an IllegalStateException.

Acceptable expressions conform to the following format:

Example: assume the concat function concatenates any number of arguments into a String, and the variables b = 1 and e = 2:

"a, $b, concat( d, $e, 'f')" The expression result will be an object array with: ["a", 1, "d2f"].

Parameters:
scope - the scope of this expression variable
name - the name of the expression variable
expression - the expression to be executed when the variable name is read for the first time in its scope.

addFunction

<F extends FunctionCall<?,?>> void addFunction(EngineScope scope,
                                               String name,
                                               F function)
Adds a custom function implementation to this engine. The result of a function call will be reused within a given scope. For example, if the function "currentTime('yyyy-mm-dd')" is associated with the EngineScope.CYCLE scope, the value returned by this particular invocation will be reused until a mapping cycle ends.

Note: functions are accessible from any scope.

Type Parameters:
F - the function call type
Parameters:
scope - the scope of values returned by the given function. The engine will retain and reuse the result of each function call into the given scope.
name - the name of the given function. This name can be used in expressions, followed by the function arguments.
function - the actual implementation of the function.

addMap

<F extends Map<?,?>> void addMap(String name,
                                 F mapFunction)
Creates a function backed by an implementation of Map. Functions based on a map accept only one parameter as an argument, which is used as the key to retrieve a value from the map. This map can be modified externally at any time.

Examples:



 Map<String, String> languages = new HashMap<String, String>();
 locales.put("en_US", "American English");
 locales.put("en_GB", "British English");
 locales.put("en_AU", "Australian English");

 //creates a function named "languages" with values given in the languages map
 engine.addMap("languages", languages);

 //executes "languages(en_GB)" to obtain the description and maps it to the "language_description" field in the destination entity.
 mapping.value().copy("{languages(en_GB)}").to("language_description");

 //For each value extracted from the source "lang_code", calls the "languages" function
 //to obtain the description then writes to "language_description" in the destination
 mapping.value().copy("lang_code").to("language_description").readingWith("languages");
 

Type Parameters:
F - the map type
Parameters:
name - the name of the function
mapFunction - the implementation of Map that will be used to store and fetch key-value pairs.

addQuery

QuerySetup addQuery(EngineScope scope,
                    String queryName)
Configures a query to be executed against a data store as a function. The query will be accessible from any EntityMappings of this engine. The result of a query-based function call will be reused within a given scope. For example if query named as "selectLocaleId" is associated with the EngineScope.CYCLE scope, the value returned by multiple invocations of "selectLocaleId('en_US')" will be reused until the end of a data mapping cycle.

Note: The query-based function will be accessible from any scope.

Parameters:
scope - the scope of values returned by the query. The engine will retain and reuse the result of each query call within the given scope.
queryName - the name of the given query. This name can be used in expressions, followed by the query arguments.
Returns:
an object used to properly configure the query as a function.

setVariable

void setVariable(String name,
                 Object value)
Sets or adds then initializes a variable in the current scope. To read the value of a variable in expressions, prepend it with $. For example: "$en_US" reads the value of the variable "en_US".

The scope of the new variable depends on the current active scope (see EngineScope).

Parameters:
name - the name of the new variable
value - the value of the variable to be set/created in the current scope.

setPersistentVariable

void setPersistentVariable(String name,
                           Object value)
Sets or adds then initializes a variable in the persistent scope (EngineScope.PERSISTENT). If no ScopeStorageProvider is defined in EngineConfiguration.getPersistentScopeStorageProvider(), the variable will be added/set using EngineScope.APPLICATION.

To read the value of a variable in an expression, prepend it with $. For example: "$en_US" reads the value of the variable "en_US".

Parameters:
name - the name of the new variable added to the EngineScope.PERSISTENT scope
value - the value of the variable to be set/created in the persistent scope.

setConstant

void setConstant(String name,
                 Object value)
Adds a constant value that can accessed from any scope. To read the value of a constant in expressions, prepend it with $. For example: "$en_US" reads the value of the constant "en_US".

Constants cannot have their value modified.

Parameters:
name - the name of the new constant
value - the value of the constant

readVariable

Object readVariable(String name)
Reads the value of a variable or constant registered to this engine

Parameters:
name - the name of the variable
Returns:
the value of the variable

addFunctions

void addFunctions(Object... objectsWithFunctions)
Creates new functions based on the methods annotated with FunctionWrapper. Object instances provided by the user are required, as the object might have an internal state that should be updated with each function call.

Parameters:
objectsWithFunctions - a sequence of objects whose classes define one or more methods annotated by with FunctionWrapper.

addRowReader

void addRowReader(EngineScope scope,
                  String name,
                  Class<? extends RowReader> reader)
Associates a RowReader implementation class to a name and a scope. When required, a row reader instance will be instantiated using the class's default constructor. This instance will be reused within the given scope and destroyed when the scope is deactivated.

Parameters:
scope - the row reader scope that determines when its instances will be destroyed.
name - the name of the given row reader.
reader - the implementation of a RowReader to be instantiated when required.

addRowReader

void addRowReader(String name,
                  RowReader reader)
Associates a RowReader instance to name. As this is a user-provided instance, it is the user's responsibility to determine how its internal state, if any, should be managed.

Parameters:
name - the name of the given row reader
reader - an instance of a row reader, managed by the user.

addDataset

void addDataset(String name,
                Dataset dataset)
Adds a new Dataset to this engine. The given dataset can contain data and be used as a source or, if the dataset implements ModifiableDataset, as the destination entity in entity mappings. The user can create his/her own implementation of this interface.

As a convenience, many useful dataset implementations can be obtained through a DatasetFactory.

Parameters:
name - the name of the dataset
dataset - the dataset implementation

addDatasetProducer

DatasetProducerSetup addDatasetProducer(EngineScope scope,
                                        DatasetProducer producer)
Adds and configures a DatasetProducer for reading information from an entity accessible through this engine, and producing different datasets as a result. The resulting datasets can then be used in entity mappings. The dataset producer is associated with a scope, therefore the datasets will be kept while the scope is active.

Parameters:
scope - the scope of the datasets produced by the given producer
producer - the object responsible for producing datasets.
Returns:
a DatasetProducerSetup object for configuring the inputs and outputs of the given dataset producer.

disableUpdateOnRecords

void disableUpdateOnRecords(String dataEntityName,
                            Dataset dataset)
Disables data modifications on a given set of records of a data entity that were already mapped.
This will create a flag in uniVocity metadata tables for the given records, which prevents data modifications on all rows already mapped.

Note: This only takes effect if the entity mapping configuration uses uniVocity metadata information (i.e. PersistenceSetup.usingMetadata()).

Parameters:
dataEntityName - the name of the data entity to have some records disabled for updates.
dataset - the dataset containing the identifiers of a mapped destination entity.

disableUpdateOnAllRecords

void disableUpdateOnAllRecords(String dataEntityName)
Disables data modifications to all records of this data entity that were already mapped. New records introduced after another mapping cycle will be enabled for updates.
This will create a flag in uniVocity metadata tables for the given entity, which prevents data modifications on all rows already mapped.

Note: This only takes effect if the entity mapping configuration uses uniVocity metadata information (i.e. PersistenceSetup.usingMetadata()).

Parameters:
dataEntityName - the name of the data entity to have all records already mapped disabled for updates.

enableUpdateOnRecords

void enableUpdateOnRecords(String dataEntityName,
                           Dataset dataset)
Re-enables data modifications on a given set records of a data entity. This reverts the metadata changes made with disableUpdateOnRecords(String, Dataset) or disableUpdateOnAllRecords(String) for the given records.

Parameters:
dataEntityName - the name of the data entity to have some records enabled for updates.
dataset - the dataset containing the identifiers of a mapped destination entity.

enableUpdateOnAllRecords

void enableUpdateOnAllRecords(String dataEntityName)
Re-enables data modifications on all records of a data entity. This will any metadata changes made with disableUpdateOnRecords(String, Dataset) or disableUpdateOnAllRecords(String).

Parameters:
dataEntityName - the name of the data entity to have all of its records enabled for data updates.

setMappingSequence

void setMappingSequence(String... sequenceOfDestinationEntities)
Use this method to define the correct sequence of mappings that have to be executed in order to correctly process the default data mapping cycle (started with executeCycle() or executeCycle(DataIncrement)).

By default, the mappings are executed in the order they were configured. However, entity mappings generated automatically (using DataStoreMapping.autodetectMappings()) won't have the correct mapping sequence.

This might be required to ensure data exclusions and updates occur in the proper sequence (which is usually the reverse order of insertion) and that references to other entities are correctly populated.

If there are duplicate names from different data stores, these names must be written in the format dataStoreName.entityName.

Parameters:
sequenceOfDestinationEntities - the sequence of destination fields to be mapped. Not all destination entities need to be declared here. The ones that appear in the sequence will be executed first, in the given order. Omitted data entities will have their mappings executed after the give sequence of mappings took place.

executeCycle

void executeCycle()
Executes a data mapping cycle with all mappings configured in this engine (i.e. via map(String, String)). A transactional operation for all mappings in this cycle will be created.


executeCycle

void executeCycle(DataIncrement increment)
Executes a data mapping cycle against a DataIncrement object with all mappings configured in this engine (i.e. via map(String, String)).

The data increment is used in place of one or more source data entities.

Parameters:
increment - The increment with data changes for one or more source entities that should be applied to the destination using the configured mappings.

executeCycle

void executeCycle(String... destinationEntities)
Executes a data mapping cycle against the selected destination entities. If there are duplicate names from different data stores, these names must be written in the format dataStoreName.entityName. The mappings will be executed in the same order the given entities are declared.

Parameters:
destinationEntities - the sequence of destination entities to receive data from the mappings.

executeCycle

void executeCycle(DataIncrement increment,
                  String... destinationEntities)
Executes a data mapping cycle against a DataIncrement object and a selection of destination entities. If there are duplicate names from different data stores, these names must be written in the format dataStoreName.entityName. The mappings will be executed in the same order the given entities are declared.

The data increment is used in place of one or more source data entities.

Parameters:
increment - the object with additional data to be used in place of one or more source data entities in this cycle.
destinationEntities - the sequence of destination entities to receive data from the mappings.

executeCycle

void executeCycle(Transactions transactionConfig)
Executes a data mapping cycle with all mappings configured in this engine (i.e. via map(String, String)).

Parameters:
transactionConfig - the configuration that defines how transactions should be created while executing mappings in this cycle.

executeCycle

void executeCycle(Transactions transactionConfig,
                  DataIncrement increment)
Executes a data mapping cycle against a DataIncrement object with all mappings configured in this engine (i.e. via map(String, String)).

The data increment is used in place of one or more source data entities.

Parameters:
transactionConfig - the configuration that defines how transactions should be created while executing mappings in this cycle.
increment - The increment with data changes for one or more source entities that should be applied to the destination using the configured mappings.

executeCycle

void executeCycle(Transactions transactionConfig,
                  String... destinationEntities)
Executes a data mapping cycle against the selected destination entities. If there are duplicate names from different data stores, these names must be written in the format dataStoreName.entityName. The mappings will be executed in the same order the given entities are declared.

Parameters:
transactionConfig - the configuration that defines how transactions should be created while executing mappings in this cycle.
destinationEntities - the sequence of destination entities to receive data from the mappings.

executeCycle

void executeCycle(Transactions transactionConfig,
                  DataIncrement increment,
                  String... destinationEntities)
Executes a data mapping cycle against a DataIncrement object and a selection of destination entities. If there are duplicate names from different data stores, these names must be written in the format dataStoreName.entityName. The mappings will be executed in the same order the given entities are declared.

The data increment is used in place of one or more source data entities.

Parameters:
transactionConfig - the configuration that defines how transactions should be created while executing mappings in this cycle.
increment - the object with additional data to be used in place of one or more source data entities in this cycle.
destinationEntities - the sequence of destination entities to receive data from the mappings.

getEntity

Entity getEntity(String entityName)
Get an accessible data entity which provides direct access to the underlying data store.

Parameters:
entityName - the name of a data entity managed by this DataIntegrationEngine
Returns:
an Entity for the given entity name.

exportEntities

Export exportEntities(String dataStoreName,
                      String... entityNames)
Configures an export operation from a given list of data entities.

Parameters:
dataStoreName - name of the data store whose entities will be exported
entityNames - names of the entities to export from the given data store. If no names are provided, all entities in the data store will be exported.
Returns:
an Export configuration object to determine what information to export and to where.


Copyright © 2015 uniVocity Software Pty Ltd. All rights reserved.