DeepDiveintoAutomatingOracleGoldenGateusingtheNewMicroservices

DeepDiveintoAutomatingOracle
GoldenGateusingtheNew
Microservices
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
VolkerKuhr,SeniorPrincipalProductManager
JingLiu,DirectorofDevelopment
NickWagner,DirectorProductManagement
OracleGoldenGateDevelopment
October2,2017

SafeHarborStatement
Thefollowingisintendedtooutlineourgeneralproductdirection.Itisintendedfor
informationpurposesonly,andmaynotbeincorporatedintoanycontract.Itisnota
commitmenttodeliveranymaterial,code,orfunctionality,andshouldnotbereliedupon
commitmenttodeliveranymaterial,code,orfunctionality,andshouldnotbereliedupon
inmakingpurchasingdecisions.Thedevelopment,release,andtimingofanyfeaturesor
functionalitydescribedforOracle’sproductsremainsatthesolediscretionofOracle.
2

OracleGoldenGate
OracleGoldenGateprovideslow-impactcapture,routing,
transformation,anddeliveryofdatabasetransactions
acrosshomogeneousandheterogeneousenvironmentsin
real-timewithnodistancelimitations.
RealtimePerformance
Extensible&Flexible
Proven&Reliable
Most
Databases
Data
Events
TransactionStreams
Cloud
DBs
Big
Data
SupportsDatabases,BigDataandNoSQL:
*Themostpopularenterpriseintegrationtoolinhistory

Agenda
1
2
NewGoldenGateMicroservicesArchitecture
AutomatingandEmbeddingGoldenGate
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|4

•GoldenGatecomponentsasmicroserviceswithcomprehensiveRESTful
interfaces.
•Enablesremoteandsecureconfiguration,administration,andmonitoring
Simplifieslargescaleandclouddeployments
capabilities.
•EnablesApplicationstoembed,automate,andorchestrateGoldenGate.
5

Administration,Distribution,Receiver,MetricsServiceswithRESTfulServiceInterfaces
NewServicesArchitectureforCloudandLarge-scaleDeployments
ManagerGGSCI
Admin
Server
Source
Database
Target
Database
HTTPSRESTFul
ServiceInterfacesManagerGGSCI
Admin
Server
Metrics
Server
Metrics
Server
SourceTarget
Trail
Files
ExtractTrail
Files
Replicat
Pumps
Distribution
Server
Collectors
Receiver
Server
Service
ManagerService
Manager

NewGoldenGateMicroservices
•AdministrationService
–ReplacesGGSCIandManagerwithasingleadministrationserviceformanagingreplicationprocesses
•Multi-threadedDistributionService
–Replacesmultiplesource-sideExtractPumpswithasingleinstanceservice.
–Lightweightfilteringonly(notransformations)
•Multi-threadedReceiverService
•Multi-threadedReceiverService
–Replacesthemultiplediscretetarget-sideServer/Collectorswithasingleinstanceservice
•PerformanceMetricsService
–Newserviceformonitoringmetricsrelatedtoaparticulardeployment
–Alsoavailableinthetraditionalarchitecture
•ServiceManager
–Newserviceformanagingmultipledeploymentsonalocalhost
7

CommandLine,Browsers,ProgrammaticRESTfulInterfaces
AdministerGoldenGatewithVarietyofClients
Service Interface
Browsers
AdminClient
Trail
Files
ExtractDistribution
Server
HTTPS
Metrics
Server
Service Interface
Anythingvia
ServiceIF
Scripts&
Interpreters
PLSQL
Admin
Server
Trail
Files
ReplicatReceiver
Server
FilesServer
Service
Manager

RESTbasedclientsforRemoteandSecureGoldenGateAdministration
•UsebrowsertoadministerandmonitorGoldenGate
•EachservicehasanembeddedHTML5app
•ThinCommandLineclient-similartoGGSCI
•Connectandadministerlocalandremotedeployments

ExampleRESTfulServiceCalltoCreateExtract
POSThttps://xyz.us.oracle.com:9101/services/v2/processes/extracts/e001
JSONPayload:
SingleCalltocreate,updateparameterfile,register,andstartintegratedExtract
{
"$schema":"ogg:extract",
"credentials":
{
"domain":"OracleGoldenGate",
"alias":"gg1"
},
"status":“running",
"source":
{
"tranlogs":"integrated"
},
"registration":
},
"config":
[
"--Parameterfileforprimaryextract:e001",
"extracte001",
"useridaliasgg1",
"exttrailaa,formatrelease12.3",
"tranlogoptionsexcludetag+",
"eofdelaycsecs10",
"tableu1.*;",
""
],
"registration":
{
"csn":"0.0",
"share":true
},
"targets":
[
{
"name":"aa"
}
]
}

SecurityModel
ClientAuthorization
•IdentityviaSSLusercertificatesor
viausername/password
–IntegratewithSSOconfiguredin
proxy/middleware.
StandardTLS/SSLbasedauthenticationandClientAuthorization
Clients
SSLUser
Certificate
HTTPS
User/
Passwd
proxy/middleware.
•Roles
–SecurityUser
–Administrator
–Operator
–User
11
GoldenGate
Services
SSLServer
Certificate
Authorized
Clients

IndustrystandardHTTP(S)initiatedfull-duplexstreamingprotocol
WebSockets--DefaultDataCommunicationProtocol
Distribution
Service
Receiver
Service
wss://……/
Trail
Files
ExtractTrail
Files
Replicat
•SSLbasedsecurity
•CanseamlesslytraversethroughHTTPforward/reverseproxyservers

Easilybuildself-serviceapplicationswhichautomatesGoldenGate
AllowsApplicationstoEmbedandAutomateGoldenGate
GoldenGate
Services
Deployment
GoldenGate
Services
RESTcallstoconfigure,
administer,andmonitor
GoldenGate
Services
Deployment
GoldenGate
Services
Deployment
GoldenGate
Services
Deployment
CatalogofRESTfulAPIsareavailableforallservices

N-wayactive-activereplicationautomaticallysetup
GoldenGateAutomatedinOracleDatabaseSharding
•Automaticcreationofreplication
processes
•Extracts,Replicats,Distribution
Paths
•AutomaticCDRforresolving
Shard
Catalog/Coordi
nator
AppTier
RoutingTier
Connection
Pool
ShardDirectors
Shardingkey
•AutomaticCDRforresolving
conflicts
•Replicationtopologyisautomatically
reconfigureduponshardingchanges
natorRoutingTier
Data
Tier
…
…

Sharding-AutomaticOracleGoldenGateConfiguration
ShardCatalog1.Definereplicationtopology
GDSCTL>DEPLOY
GDSCTL>createshardcatalog….-replOGG….
GDSCTL>addshardgroup..shgrp1-repfactor3
GDSCTL>addshard-shardgroupshgrp1….
gg_servicehost01:9000/deploy1….
GDSCTL>addshard-shardgroupshgrp1….-
2.Sendcommand
toshards
Shards......
gg_servicehost02:9000/deploy1
gg_servicehost03:9000/deploy1
…..
…….
GDSCTL>DEPLOY

Sharding-AutomaticOracleGoldenGateConfiguration
ShardCatalog1.Definereplicationtopology
5.Updatestatus
OGGshardingorchestration
logicissuesRESTcallstosetup
GoldenGatereplication
GDSCTL>DEPLOY
2.Sendcommand
toshards
Shards
OGG
Services
3.https
Extract
Replicat
Distribution
Path
4.Setupreplication
OGGSharding
Orchestration
Logic
5.Updatestatus
......

ProgramAgenda
AutomatingandEmbeddingGoldenGate
1
2

SetupReplicationusingsimplePL/SQLbuildingsblocks
one_way_replication
create_extract
Database
PL/SQLProcedures
HTTPS
create_extract
create_replicat
create_distpath
….
gg_send_request
Note:WeplantopublishexamplebuildingblocksonOTN
GoldenGate
Services

SEND_GG_REQUEST
UsesUTL_HTTPtomakeRESTcalls
procedure
send_gg_request
(gg_uriINvarchar2
,verbINvarchar2
,payloadINvarchar2
,resp_codeOUTnumber
UTL_HTTP
GoldenGate
Services
HTTPS
,resp_codeOUTnumber
,resp_textOUTNOCOPYvarchar2
)
•URIofGoldenGateServiceEndpoint:https://<gghost>:9001/services/v2/processes/extracts/e001
•HTMLVerb:POST,PATCH,DELETE,GET
•Payload:JSONobjectspecifyingthereplicationprocess
•Response_Code:200(OK),201(CREATED),…
•Response_Text:<detailedResponsefromService>
UTL_HTTP
HTTPS

procedure
create_[extract|replicat]
([ext|rep]_nameINvarchar2
,adminsrv_uriINvarchar2
,db_credentialsINvarchar2
,[ext|rep]_trailINvarchar2
,[ext|rep]_paramsINvarchar2
ProcedurestocreateGoldenGateProcesses
Payload
)
20
procedure
create_distpath
(path_nameINvarchar2
,ds_uriINvarchar2
,src_trailINvarchar2
,rs_uriINvarchar2
,tgt_trailINvarchar2
)
JSON
Payload
send_gg_request

Customizeasneeded
OrchestrationProceduretosetupOne-wayReplication
procedureone_way_replication
(db_srcvarchar2
,gg_srcvarchar2
,gg_trgvarchar2
,db_trgvarchar2
create_extract
create_distpath
create_replicat
…
GenerateERprocessnames
GenerateTrailfilenames
…
,db_trgvarchar2
,tablesvarchar2
,instantiationvarchar2[YES|NO]
,auto_CDRvarchar2[YES|NO]
)
…
DatabaseUtilities
DBMS_DATAPUMP
DBMS_FILE_TRANSFER
DBMS_TTS
...
DBMS_GOLDENGATE_ADM
ADD_AUTO_CDR(…)
RequiresDB12.2+&OGG12.3

SetupOneWayReplication
•OrchestrationPackage
•Databaseexport/import
toinstantiatetarget
•DatabasesandGoldengate
OrchestrationDatabase
BEGIN
add_oneway_replication
(db_src=>′REPDB01′
,gg_src=>′GG_Inst01′
,gg_trg=>′GG_Inst02′
,db_trg=>′REPDB02′
,tables=>′HR.EMP′
,instantiation=>′YES′
•DatabasesandGoldengate
Servicesareup
GoldenGate
Services
GoldenGate
Services
REPDB01REPDB02
,auto_CDR=>’NO’
);
END;

CreateandStartIntegratedExtract
{
"config":[
"ExtractE01AA",
"ExtTrailet",
"UseridAliasgg_src",
"TableHR.EMP;"
],
"source":{
1.CreatingExtract
2.CreatingDistributionPath
3.DatapumpExport
4.DatapumpImport
5.CreatingReplicat
POSTHTTPS://gg_src:8001/....
"source":{
"tranlogs":"integrated"
},
"credentials":{
"alias":"gg_src"
},
"registration":"default",
"begin":"now",
"targets":[
{
"name":"et"
}
]
}
23
GoldenGate
Services
GoldenGate
Services
Extract

CreateandStartDistributionPath
1.CreatingExtract
3.DatapumpExport
4.DatapumpImport
5.CreatingReplicat
{
"$schema":"ogg:distPath",
"name":"AAtoAB",
"description":"distPath1",
"source":{
"uri":"trail://ggsource:8002/
dirdat/et"
},
POSTHTTPS://gg_src:8002/....
GoldenGate
Services
GoldenGate
Services
REPDB01REPDB02Extract
},
"target":{
"uri":"ogg://ggtarget:9003/
dirdat/rt"
},
"begin":{
"sequence":0,
"offset":0
},
"status":"running"
}

DatapumpExport
1.CreatingExtract
3.DatapumpExport
4.DatapumpImport
5.CreatingReplicat
DatapumpExport(EXPDP)
v_dp_handle:=dbms_datapump.open
dbms_datapump.add_file
…
dbms_datapump.set_parameter
dbms_datapump.metadata_filter
dbms_datapump.start_job
GoldenGate
Services
GoldenGate
Services
dbms_datapump.detach
REPDB01REPDB02Extract

DatapumpImport
1.CreatingExtract
3.DatapumpExport
4.DatapumpImport
5.CreatingReplicat
DatapumpImport(IMPDP)
v_dp_handle:=dbms_datapump.open
dbms_datapump.add_file
…
dbms_datapump.set_parameter
dbms_datapump.metadata_filter
GoldenGate
Services
REPDB01REPDB02
dbms_datapump.detach
Extract
GoldenGate
Services

GoldenGateinstantiationSCNfeaturesautomaticallyfilterspre-instantiationchanges
CreateandStartReplicattocompletesetup
1.CreatingExtract
3.DatapumpExport
4.DatapumpImport
5.CreatingReplicat
{
"config":[
"ReplicatR01BA",
"UseridAliasggadmin",
"MapHR.EMP,",
"TargetHR.EMP;"
],
"source":{
POSTHTTPS://gg_trg:9001/....
GoldenGate
Services
GoldenGate
Services
REPDB01REPDB02ExtractReplicat
"source":{
"name":"rt"
},
"credentials":{
"alias":"gg_trg"
},
"checkpoint":{
"table":"ggadmin.ckpt"
}
}

SetupBidirectionalActive-ActiveReplicationfromPL/SQL
Usingthesamehighlevelbuildingblock
OrchestrationDatabaseBEGIN
BEGIN
,instantiation=>′NO′
GoldenGate
Services
GoldenGate
Services
REPDB01REPDB02
,auto_CDR=>’YES’
);
END;
);
END;

Hub&SpokeConfiguration
TwocallsforeveryHub/Spokepair
GG
BEGIN
(db_src=>′HUB′
,gg_src=>′GG_Inst_HUB′
,gg_trg=>′GG_Inst_01′
,db_trg=>′Spoke01′
);
END;
Spoke
GGGG
GG
BEGIN
(db_src=>′Spoke01′
,gg_src=>′GG_Inst_01′
,gg_trg=>′GG_Inst_HUB′
,db_trg=>′HUB′
);
END;
GG
Hub

GlobalReplicationCatalog
•ControlReplicationEnvironmentsinreplicationcatalog
•Create,modifyorremoveReplicationTopologiesfromthiscatalogtable
•CanmanagereplicationwithDMLtotable
–HaveaDMLtriggerexecutethePL/SQLreplicationorchestrationlogic
KeepTrackofReplicationDeployments
DMLTrigger
Orchestration
Logic
GoldenGate
Services
HTTPS
POST
PATCH
DELETE()
ReplicationCatalog
Source
Database
ExtractDist
Path
ReplicatTarget
Database
...
REPDB01E01ABA_to_BR01BAREPDB02
...
REPDB01E02ACA_to_CR01CAREPDB03
...
…..............
...

RetrieveinformationaboutReplicationenvironments
MonitorReplicationfromtheDatabase
ReplicationCatalog
•UseGETRESTfulcalls
•Checkstatus,warnings/errors(ifany),throughput,lag,…
ReplicationCatalog
Source
Database
ExtractDist
Path
ReplicatTarget
Database
......StatusLAG
[s]
Throughput
[Changes/s]
REPDB01E01ABA_to_BR01BAREPDB02
......
OK1.854,673
REPDB01E02ACA_to_CR01CAREPDB03
......
OK1.878,924
…..............
......
.........

ClassicArchitecture
✖✖✖✖Combinationofscripts
–ShellScripts,SQLscripts,…
–ObeyFiles,ParameterFiles,…
GoldenGateMicroservices
✔✔✔✔SimpleandsecureRESTcallsfor
allGoldenGateoperations
EasyOrchestrationwithNewGoldenGateMicroservices
–ObeyFiles,ParameterFiles,…
✖✖✖✖RequiresOSaccesstoDBhosts
–Notsecure
32

•GoldenGatecomponentsasmicroserviceswithcomprehensiveRESTful
interfaces.
Simplifieslargescaleandclouddeployments
capabilities.
•EnablesApplicationstoembed,automate,andorchestrateGoldenGate.
33

AdditionalsessionsandDemos
Sunday,October1
•LiftandShiftWorkloadstoCloudwithOracleDataIntegrationPlatform
Cloud[SUN6653]
•DataMovementbetweenOn-Prem,FusionERPCloud,FusionHCMCloud
andSalesforce[SUN7286]
•AccelerateMigrationtoCloudInfrastructurewithDataIntegrationPlatform
[SUN6896]
Monday,October2
Tuesday,October3
•OracleDataIntegratorProductUpdateandStrategy[CON6654]
•OracleEnterpriseDataQuality:ProductOverviewandRoadmap[CON6656]
•AccelerateCloudOn-BoardingUsingOracleGoldenGateCloudService
[CON6894]
•OracleEnterpriseDataQualityforAllTypesofData[HOL7653]
•OracleDataIntegrationPlatform:aCornerstoneforBigData[CON6655]
•GoldenGate:MAAandBestPracticesforOracle
GoldenGateMicroservices[CON6570]
•OracleGoldenGateProductUpdateandStrategy[CON6897]
Monday,October2
•OracleDataIntegrationPlatformStrategyandRoadmap[CON6646]
•FillingYourDataLakewithPotableData,UsingDataIntegration[CON5465]
•GoldenGate:
DeepDiveintoAutomatingOGGusingthenewMicroservices[CON6569]
•OracleDataIntegrationPlatform:FoundationforCloudIntegration
[CON6650]
•OracleDataIntegrationPlatformEmpowersEnterpriseGradeBigData
Solutions[CON6893]
•OracleDataIntegrationPlatformCloudDeepDive[CON6651]
•OracleGoldenGateCloudService:Real-TimeDataReplicationintheCloud
[HOL7715]
•OracleGoldenGateProductUpdateandStrategy[CON6897]
Wednesday,October4
•APracticalPathtoEnterpriseDataGovernancewithEnterpriseDataQuality
[CON6657]
•OracleDataIntegratorandOracleGoldenGateforBigData[HOL7708]
•IntroductiontoOracleDataIntegrationPlatformCloud[HOL7673]
•AnEnterpriseDatabus:GoldenGateintheCloudWorkingwithKafkaand
Spark(CON6895]
•GoldenGate:BestPractices&DeepDiveonOGG12.3
MicroservicesatCloud[CON6568]
•OracleGoldenGateforBigData[CON6898]
•OracleDataIntegrationPlatformCloudServiceGovernanceEdition
[CON6652]
•OracleSharding:LinearScalability,ExtremeAvailability,andGeo-Distribution
[CON6673]

DeepDiveintoAutomatingOracleGoldenGateusingtheNewMicroservices

DeepDiveintoAutomatingOracleGoldenGateusingtheNewMicroservices

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to DeepDiveintoAutomatingOracleGoldenGateusingtheNewMicroservices

Similar to DeepDiveintoAutomatingOracleGoldenGateusingtheNewMicroservices (20)

Recently uploaded

Recently uploaded (20)

DeepDiveintoAutomatingOracleGoldenGateusingtheNewMicroservices