Overview

Processors allow you to perform initial verifications and data-cleansing for columns as they are retrieved from the file using the CSVTable.getValue(<rowIndex>, <columnIndex>|<columnName>) method. There are two types of processors available:

Column Value Processor - pass a Groovy Closure, per column, which contain logic to apply when data is retrieved from a column
Extension Processor - Extensions are written in Java, exported as a jar to extend Andi scripting verifications. Allows you to leverage Andi's standard verifications, add-on verifications jar or create your own jar with your pre-existing tried and true verifications. Refer to Create your own Extension

For complete function information including available parameters, execute the following in an Andi Integrated Scripting environment.

// The details of the andi:included extensions 
println extensionRegistry.getExtensionDetails("andi:included")

Column Value Processor

The map passed will contain the key which is the columnName that will trigger the processor and the Groovy Closure is the value. The closure begins and ends with a bracket and will have a pre-defined list pf parameters separated from the code by the -> characters. A simple closure for a processor will follow a basic pattern that allows the value to be verified and possibly cleansed.

{ rowIndex, columnIndex, table, columnValue ->
   value = columnValue
   if ( value is valid ) {
      value = <transformedValue>
   } else {
     throw new VerificationException( <columnName>, rowIndex, columnIndex, value )
   }
  
   return value
}

NOTE: that the column names you use are not important but the order of the columns must match the example above.

Column Value Processor Example

To automatically verify/cleanse a column(s), you add one or more processors as described below. A simple example: first_name and last_name columns can quickly be converted to upper case. Note that the rowIndex, columnIndex, table, value values will be passed to you for each invocation.

  
processors = ['first_name' : { rowIndex, columnIndex, table, value -> 
                if ( value != null ) { 
                   value = value.toUpperCase() 
                } else { 
                   throw new VerificationException( table.getBatchRunId(), rowIndex, 
                               columnIndex, "The first_name column is null" ) 
                }

                return value 
             }, 
            'last_name' : { rowIndex, columnIndex, table, value -> 
                if ( value != null ) { 
                   value = value.toUpperCase() 
                } else { 
                   throw new VerificationException( table.getBatchRunId(), rowIndex, 
                             columnIndex, "The last_name column is null" ) 
                }

                return value 
             } 
          ]
// Add the processors to the table 
table = new CSVTable( "c:/fileView/name/name.csv", processors )
// Enable all VerificationException thrown to be logged to allow you to resolve issues 
table.setVerificationMode( "log", "d:/andi/verification/verificationErrors.txt" ) 
table.eachRow() { rowIndex, table -> context.testExecutionCancelled()
   // Accesses the first_name and last_name column which will also invoke the appropriate
   // process. 
   println table.createDelimitedRow( rowIndex, ['first_name','last_name'] ) 
}

Extension Processor

New verifications and cleansing functions can be easily added by adding functions to an existing extension or creating a new extension to extend Andi.

To add a function call to your script, you create an ExtensionParam() instance to register a specific extension and function for a column. In addition, multiple extensions can be "daisy-chained" together to use for one column by including them in a Groovy List.

ExtensionParam Constructors

ExtensionParam( String extensionId, String functionName )

ExtensionParam( String extensionId, String functionName, List<String> parameters )

ExtensionParam( String extensionId, String functionName, List<String> parameters,

String verificationModeOverride)

NOTE: verificationModeOverride:

FAIL - fail if a VerificationException is thrown
IGNORE - continue processing
LOG - log the verification exception and continue processing

The following example shows how functions can be performed to provide common data cleansing operations.

 processors = 
[ 
    'first_name' : new ExtensionParam( "andi:included", "upperCase", ["required"] ),
    // Daisy chain two or more extensions using a list
    'last_name' : [
                new ExtensionParam( "andi:included", "trim", ["required"] ), 
    	        new ExtensionParam( "andi:included", "lastName" )
              ]
]	

table = new CSVTable( "c:/fileView/name/name.csv", processors )
table.setVerificationMode( "log", "d:/andi/verification/verificationErrors.txt" )

table.eachRow() { rowIndex, table ->
	context.testExecutionCancelled()
        // Dynamically create a new delimited row
	println table.createDelimitedRow( rowIndex, ['first_name','last_name'] )
}

Column Value Processors

Overview

Column Value Processor

Column Value Processor Example

Extension Processor