This document provides an overview and comparison of Informix's streaming technologies: Change Data Capture (CDC), Smart Triggers, Asynchronous Triggers, and V-II Socket Streaming. CDC processes database transaction logs to capture all changes and send them to clients. Smart Triggers use selective triggers and filtering to capture specific data changes. Asynchronous Triggers use post-commit triggers to route data to user-defined routines. V-II Socket Streaming sends triggered data to MQTT brokers but is not officially supported. The document also includes code examples and diagrams demonstrating how these technologies integrate with applications.

1. Informix Change Data
Streaming
External Triggers and Change Data Replication
Brian Hughes

2. • Streaming Basics
• Informix streaming technologies (overview)
• Introducing a new streaming client for Change Data Capture
• If you liked Smart Triggers – you might like this
Agenda

3. What is Informix Change Data Streaming?
• Streaming database changes(known as change streams or change data capture) is
one or more mechanisms where data that is manipulated in a database generates
events (triggers) which can be captured by either the database or an external
application and processed
• Easy to introduce into existing solutions
• All applications still connect to the database (no topology changes)
• Efficient means to push data to other parts of your infrastructure
• Filtering/preprocessing done in the data
• Only needed rows/changes can be subscribed to
• No database polling

4. Informix Streaming technologies
• Smart Triggers
• Asynchronous Triggers
• V-II Socket Streaming (unsupported, prototype)
• CDC (change data capture)

5. Smart Triggers
• Selective triggers on data changes in a table
• Leverages Enterprise Replication components
• Efficient asynchronous processing
• Uses SQL style WHERE clause and projection list to define and filter what is
triggered
• Setup via the Admin API (JDBC and ODBC have language specific API’s wrapping this)
• Uses the same large object TCP pipe as CDC to send the triggered records
• Record are in JSON format with metadata
• Transaction id, table, user, before/after image for updates
• Useful if you are looking for specific events on a table
• Scales to many watched tables with a few events occurring on each time
• This is not a high throughput system (compared to CDC) or a replication system
(not ensured all rows are processed)

6. Smart Trigger Topology
External Database
MQTT Broker
Java Application
2. Log records processed
matching table triggered
project list filters columns
WHERE clause filters rows
3. Smart Trigger API processes
JSON stream
Processing4. User Application does any filtering/
if needed, then sends data on
1. Insert data into database
ST API

7. Smart Triggers
public class SmartTrigger implements IfmxSmartTriggerCallback {
private final JsonParser p = new JsonParser();
private static final Logger logger = LoggerFactory.getLogger(SmartTrigger.class);
public static void main(String[] args) throws SQLException, InterruptedException {
try(IfxSmartTrigger trigger = new IfxSmartTrigger(
"jdbc:informix-sqli://localhost:9088/bank:user=informix;password=informix");) {
trigger.timeout(5).label("bank_alert"); // optional parameters
trigger.addTrigger("account", "informix", "banktest",
"SELECT * FROM account WHERE balance < 0", new SmartTrigger());
trigger.run(); //run forever
}
}
@Override
public void notify(String jsonString) {
JsonObject json = p.parse(jsonString).getAsJsonObject();
if (json.has("ifx_isTimeout")) {
logger.debug("[SmartTrigger] Server ping: No balance issues");
} else {
json = json.get("rowdata").getAsJsonObject();
logger.debug("{}", json);
logger.warn("[SmartTrigger] ALERT on account #{}. Balance ${}",
json.get("id").getAsInt(),
json.get("balance").getAsInt());
}
}
}

8. Asynchronous Triggers
• Works similar to normal triggers but with a few important differences
• Post commit trigger (data will already be committed before this trigger fires)
• Can target UDRs as well as other tables
• UDR with defined columns
• UDR with a JSON column
• Built on replication technology
• Takes the project and filtering capabilities of replication (same we see with smart triggers)
and processes the filtered rows into a stored procedure
• Work is performed asynchronously (making it different from the usual pre-commit
synchronous triggers of a normal DBMS)
• Scales better, does not disrupt normal database processing
• Different from normal replication or smart triggers, data triggers can route into a UDR.
• Allows on-server processing of data

9. Asynchronous Post Commit Triggers (cont)
• UDRs can be written in 3 languages
• SPL
• Java
• C
• Using a UDR you can process the records anyway you like and then you can store
them back into a table or if you use C or Java UDRs you can ship the data outside the
server
• Example: Using a Java UDR with an MQTT library you can send triggered data to
an MQTT broker

10. Async Post Commit Trigger Topology
UDR
External Database
MQTT Broker
1. Insert data into database
2. Rows filtered,
Sent to UDR
3. UDR processes records,
Can send back to staging tables
Can directly send to external
application

11. V-II Socket Streaming
• A custom C library that takes triggered data from a table and sends it to an MQTT
broker
• https://github.com/IBM-IoT/InformixSparkStreaming
• Uses the Virtual Index Interface (V-II) feature in Informix
• Only sends data to an MQTT broker
• Requires manual compilation on target platforms
• V-II is a synchronous trigger mechanism
• Inserts wait until V-II trigger is completed, which waits on the UDR, which waits
on MQTT to transmit
• Can lead to slow/blocked inserts
• Not Officially supported
• Code is free to use and modify, but not part of the Informix product line

12. Informix Change Data Capture (CDC)
• Processes Informix logical logs (log replay)
• Data processed is sent to the client application over a large object API found in many
drivers
• ESQL/C example in $INFORMIXDIR/demo/cdc
• Java API new in 4.50.xC2
• Supports basic data types as well as LOBs
• Does not support UDT’s
• Supports a large set of record types
• Begin work, commit/rollback, truncate, insert/update/delete records
• Can be used to stream entire database systems for heterogeneous replication
• Fully feature replication can require a lot of work or a 3rd party product

13. CDC Topology
External Database
MQTT Broker
Java Application
1. Insert data into database
2. Log records processed
matching table triggered
project list filters columns
3. CDC API processes byte
stream, converts into usable
object 4. User Application does any filtering/
Processing, then sends data on
if needed.
CDC API

14. • Based on work done with Smart Triggers
• Aims to simplify and reduce the work involved in using CDC
• Working solutions started in under 30 lines of code
• Needs JDBC 4.50.JC2
• Takes in a DataSource object as a constructor
• Can wrap a JDBC URL in an IfxDataSource object
• Must connect to syscdcv1 database
• syscdcv1 database must exist
• Run the $INFORMIXDIR/etc/syscdcv1.sql script
Introducing the new Java API for CDC

15. CDC Java Example Code
public class CDCExample {
public static void main(String [] args) throws Exception {
//You need a datasource for CDC to create connections from
//Simple way is to wrap a URL
IfxDataSource ds = new IfxDataSource("jdbc:informix-sqli://localhost:20290/syscdcv1:user=informix;password=informix");
//Create the builder with the datasource
//Use this to configure what you want to capture
IfxCDCEngine.Builder builder = new IfxCDCEngine.Builder(ds);
//watch as many tables as you need to
//Use format <database>:<owner>.<tablename>
//provide the list of columns afterwards
builder.watchTable("testdb:informix.cdcTable", "a", "b");
builder.timeout(5); //default 5 second timeout
//Build the engine
try(IfxCDCEngine engine = builder.build()) {
//initialize the engine (creates the connections and begins listening for changes)
engine.init();
IfmxStreamRecord record = null;
while((record = engine.getRecord()) != null) {
//Print out the basic record information
System.out.println(record);
//If it is an insert/update/delete, print the column data
if(record instanceof IfxCDCOperationRecord) {
System.out.println(((IfxCDCOperationRecord)record).getData());
}
}
}
}
}

16. Informix Streaming Technology Summary
CDC Smart Trigger Async Trigger V-II Socket Streaming
Row Filtering No Yes Yes No
Projection list Yes Yes Yes No
Target for trigger TCP Pipe/External App TCP Pipe/External App Internal procedure MQTT Broker
Commit type Whole transaction +
rollback
Post commit Post commit Pre commit
Data Format Byte Stream JSON SPL declaration or JSON JSON/MQTT Message
Data Replay ability Yes No On crash/error No
Data Integrity Yes No Yes No
Blob/Clob/Text/Byte
support
Yes No No No
UDT Support No Yes Yes No
Version Introduced 11.50 12.10.xC9 14.10.xC1 11.50 ?
API support ESQL/C
Java (4.50.xC2)
ESQL/C
JDBC
ODBC
SPL
C-UDR
Java-UDR
None

17. • Make smart triggers more resilient to failure
• Better unify the APIs for smart triggers and CDC on the Java side
• Smart triggers a few years old
• Smart triggers could be updated to use the newer style CDC uses
• Looking towards configuration based streaming
• Configuration like JSON rather than use programming
• Combined with REST API to setup triggers pushing to streaming systems
** Subject to change **
Future Direction

18. Questions?