Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

InfiniFlux append protocol

459 views

Published on

http://www.infiniflux.com/download

Append Protocol
(High-Speed Input Protocol)

Published in: Software
  • Be the first to comment

  • Be the first to like this

InfiniFlux append protocol

  1. 1. Append Protocol (High-Speed Input Protocol) www.infiniflux.com
  2. 2. Table of Contents Background CLI API JDBC API Multi Table Insert 1 2 3 4
  3. 3. Background
  4. 4. 4 Background– Limitations of Conventional Protocol Client Server Idle time …… • Limitations of Conventional Synchronous Protocol (Request - Reply): Synchronous protocol is to check the results of executions each time a record was entered. Thus; • Rapidly increase the number of I/O processing in accordance with the number of records • Must check the result of a record each time • Difficult to insert more than 1,000 data per second Client Client Client Server Server Idle time Idle time
  5. 5. 5 Background – Improved Protocol Flush Append Flush Append Append Protocol Diagram • Introduced Optimistic Asynchronous Protocol (Append Protocol) • Pile up input records on the buffer consecutively and flush them when the buffer is full. • Not returning results if append succeeds. • Return error code and message to the client when append was failed. • Able to insert more than millions of records per second • Most of cases are successfully inserted, and return errors rarely. Thus, the improved protocol is the best alternative. Client Client Client No Idle time No Idle time Server Server Server
  6. 6. 6 Background –API Structure Title Contents API for establishing communication channels • Open a channel for appending • Specify a table where data will be inserted • Insert data at a high-speed through this channel API for data insert • Binding column data that conform to the given table schema • Return errors when the length and format are not matched • Guarantee of fast data transfer through buffering API for data flush • Force to send the data through network even when the buffer was not filled. API for setting an error callback • Setting up asynchronous user-callback in order to process error code and message received from the server • Asynchronously operate with data input API for closing communication channel • Close the protocol that is currently appending • Send all the records left on the buffer through network
  7. 7. CLI API
  8. 8. 8 Confirming CLI Installation • Makefile • Header file • Library file Check the files, shown below, in the “install”, “include”, and “lib” directories where the InfiniFlux has been installed. $IFLUX_HOME/install/iflux_env.mk $IFLUX_HOME/include/iflux_sqlcli.h $IFLUX_HOME/lib/libifluxcli.a $IFLUX_HOME/lib/libifluxcli_dll.so
  9. 9. 9 Makefile & Compile • Move to “sample/cli” directory. • Check the contents of Makefile. include $(IFLUX_HOME)/install/iflux_env.mk INCLUDES += $(LIBDIR_OPT)/$(IFLUX_HOME)/include all : sample4_append2 sample4_append2 : sample4_append2.o $(LD_CC) $(LD_FLAGS) $(LD_OUT_OPT)$@ $< $(LIB_OPT)ifluxcli$(LIB_AFT) $(LIBDIR_OPT)$(IFLUX_HOME)/lib $(LD_LIBS) sample4_append2.o : sample4_append2.c $(COMPILE.cc) $(CC_FLAGS) $(INCLUDES) $(CC_OUT_OPT)$@ $< clean : rm -f *.o sample4_append2 • Compile it. Check and compile “$IFLUX_HOME/sample/cli/makefile”. $ cd $IFLUX_HOME/sample/cli $ make
  10. 10. 10 Structure of Append Program SQLAllocEnv SQLAllocConnect SQLDriverConnect SQLAllocStmt SQLAppendOpen SQLAppendDataV2 SQLAppendClose SQLAppendSetErrorCallback SQLFreeStmt SQLDisconnect SQLFreeConnect SQLFreeEnv Extended Functions of InfiniFlux CLI Standard Connect Functions CLI Standard Disconnect Functions Description of “$IFLUX_HOME/sample/cli/sample4_append1.c“
  11. 11. 11 SQLAppendOpen • Open a channel for inserting data into a specified table. • The channel remains open if the channel is not closed. • Able to set up to 1024 statements for each connection. • Use SQLAppendOpen for each statement. SQLRETURN SQLAppendOpen( SQLHSTMT aStatementHandle, SQLCHAR *aTableName, SQLINTEGER aErrorCheckCount ); • aStatementHandle : statement handle to append • aTableName : name of target table to append • aErrorCheckCount : set the number of evaluations to detect server errors based on the number of inputs. If the number is 0, evaluation is not required.
  12. 12. 12 SQLAppendDataV2 • Function to insert data for the channel. • Supported from the InfiniFlux v2.0. • Able to insert data types of both text and binary. SQLRETURN SQLAppendDataV2( SQLHSTMT aStatementHandle, SQL_APPEND_PARAM *aData ); • aStatementHandle: statement handle to append. • aData : a pointer to parameter array, “SQL_APPEND_PARAM” Defined “SQL_APPEND_PARAM” in the “iflux_sqlcli.h”
  13. 13. 13 SQLAppendClose • Close the current channel. • Able to check the number of append success or failure. SQLRETURN SQLAppendClose( SQLHSTMT aStatementHandle, int *aSuccessCount, int *aFailureCount ); • aStatementHandle: statement handle to append • aSuccessCount : the number of successful append • aFailureCount : the number of failed append
  14. 14. 14 SQLAppendSetErrorCallback • Setting up the callback function to be called when an error was occurred while appending. • If this function was not set, the client will ignore even if an error was occurred in the server. SQLRETURN SQLAppendSetErrorCallback( SQLHSTMT aStatementHandle, SQLAppendErrorCallback *aFunc ); • aStatementHandle: statement handle to append • aFunc : a pointer to callback function when an error occurs. typedef void (*SQLAppendErrorCallback)( SQLHSTMT aStatementHandle, SQLINTEGER aErrorCode, SQLPOINTER aErrorMessage, SQLLEN aErrorBufLen, SQLPOINTER aRowBuf, SQLLEN aRowBufLen ); • aStatementHandle: statement handle to append • aErrorCode : 32 bit error code which causes the error • aErrorMessage : the string for the error code • aErrorBufLen : the length of “aErrorMessage” • aRowBuf : the string that has the content of the record that caused the error • aRowBufLen : the length of “aRowBuf”
  15. 15. 15 Other Functions • Additionally, provide useful functions. • Refer to website at http://www.infiniflux.com/document for more information. Name of Functions Function Declaration Description SQLAppendDataByTimeV2 SQLRETURN SQL_API SQLAppendDataByTimeV2( SQLHSTMT aStatmentHandle, SQLBIGINT aTime, SQL_APPEND_PARAM *aData) Like “SQLAppendDataV2()”, it inserts data into a table. However, the value of hidden column, “_arrival_time”, will be used when it was set to a value of a certain time rather than the present time. In other words, send the time of log files as the value of “aTime”. SQLAppendFlush SQLRETURN SQL_API SQLAppendFlush( SQLHSTMT aStatementHandle) Transfer all the data left in the buffer of the current channel to the InfiniFlux server. SQLSetConnectAppendFlush SQLRETURN SQL_API SQLSetConnectAppendFlush( SQLHDBC aHdbc, SQLINTEGER aTimeOption) Set whether to transfer data to InfiniFlux at a specified interval. 0: auto flush off 1: auto flush on SQLSetStmtAppendInterval SQLRETURN SQL_API SQLSetStmtAppendInterval( SQLHSTMT aStatementHandle, SQLINTEGER aValue) When set “auto flush” on by using “SQLSetConnectAppendFlush()”, adjust the interval of flush or set to “auto flush” off. Don’t flush it if “aValue” is 0. The default value is 1000 and unit is “ms”. The value should be multiples of 100.
  16. 16. JDBC API
  17. 17. 17 Confirm the Installation of Library • Library file • Log4j file  Support Java 1.6 version or more • Sample • Makefile Check the file, shown below, in the “lib” directory where the InfiniFlux has been installed. $IFLUX_HOME/lib/iflux.jar $IFLUX_HOME/lib/ifluxLog.jar $IFLUX_HOME/lib/log4j.jar $IFLUX_HOME/sample/jdbc $IFLUX_HOME/sample/jdbc/Makefile
  18. 18. 18 Append Program Structure DriverManager.getConnection Connection.createStatement executeAppendOpen executeAppendData executeAppendClose executeSetAppendErrorCallback ResultSet.close Statement.close Connection.close Extended Functions of InfiniFlux Standard Connect Functions Standard Disconnect Functions Description of “$IFLUX_HOME/sample/jdbc/sample4_append.java”
  19. 19. 19 executeAppendOpen • Open a channel for inserting data into a specified table. • The channel remains open if the channel is not closed. • Able to set up to 1024 statements for each connection. ResultSet IfluxStatement.executeAppendOpen( String aTableName, int aErrorCheckCount ); • aTableName : name of target table to append • aErrorCheckCount: set the number of evaluations to detect server errors based on the number of inputs. If the number is 0, evaluation is not required.
  20. 20. 20 executeAppendData • Function to insert data for the channel. int IfluxStatement.executeAppendData( ResultSetMetaData aRsmd, ArrayList aData ); • aRsmd: information of column metadata about the table • aData: array list of data to append
  21. 21. 21 executeAppendClose • Close the current channel. Run “int IfluxStatement.executeAppendClose();”, and then call “getAppendSuccessCount()” and “getAppendFailureCount()” to check the number of append success or failure.
  22. 22. 22 executeSetAppendErrorCallback • Setting up the callback function to be called when an error was occurred while appending. • If this function was not configured, the client will ignore even if an error was occurred in the server. int IfluxStatement.executeSetAppendErrorCallback( IfluxAppendCallback aCallback); • aCallback : to call “callback” function when an error while appending data IfluxAppendCallback cb = new IfluxAppendCallback() { @Override public void onAppendError(long aErrNo, String aErrMsg, String aRowMsg) { System.out.format("Append Error : [%05d - %s]n%sn", aErrNo, aErrMsg, aRowMsg); } }; stmt.executeSetAppendErrorCallback(cb);
  23. 23. 23 Other Functions • Additionally provide useful functions. • Refer to the website at http://www.infiniflux.com/document for more information. Name of Functions Function Declaration Description executeAppendDataByTime int executeAppendDataByTime(Res ultSetMetaData aRsmd, long aTime, ArrayList aData) Like “executeAppendData()”, it inserts a data into the channel. However, the value of hidden column, “_arrival_time”, will be used when it was set to a value of a certain time rather than the present time. In other words, send the time of log files as the value of “aTime”. executeAppendFlush int executeAppendFlush() Transfer all the data left in the buffer of the current channel to the InfiniFlux server. executeAppendSuccessCount long getAppendSuccessCount() Return the number of records that were appended successfully. executeAppendFailureCount long executeAppendFailureCount() Return the number of records that were failed to append.
  24. 24. Multi Table Insert
  25. 25. 25 Multi Table Insert Structure Description of “$IFLUX_HOME/sample/cli/sample8_multi_session_multi_table.c” • Create thread in the main function, and each thread will process the tasks written below. • Each thread will process Alloc and Free functions respectively against SQLHENV, SQLHDBC, and SQLHSTMT. • Only one statement is allowed to be inserted per table. • The reason for tasks above is that an error occurs while a thread is sharing statement or connection with another thread, it might kill the whole process. If possible, it is safe not to share a resource of thread with others. • When input cycle is irregular and has long interval, conduct “auto timed flush” by using “SQLSetConnectionAppendFlush()” and “SQLSetStmtAppendInterval()”.
  26. 26. The World's Fastest Time Series DBMS for IoT and Big Data www.infiniflux.com info@infiniflux.com InfiniFlux

×