System design of oracle goldengate 23ai read&learn gg : part2

Introduction

Goldengate software was built in the year 1995 . Since then its architecture has undergone many changes and the latest 23ai introduced a mandatory microservices architecture using rest apis. In this blog I will attempt to create a simple design for goldengate so the readers would get a gist of its architecture.

Problem Statement: We have a database say PRD1 and we want to replicate all transactions of PRD1to REPORT1 . PRD1is a busy OLTP database and REPORT1 is OLAP database , we don’t want to run huge analytical transactions on PRD1 and impact the application but we want to run reports on that data. The solution is to build an app to replicate transactions of PRD1 to REPORT1.

Thought Process: Let us create a clone of PRD1 and name it as REPORT1 and then read all the dml’s running on PRD1 and replay them in REPORT1. To achieve this we will create 2 programs or process , Extract , Replicat and a physical object called Trail Files . To manage extract , replicat and trail files we will need few services like Distribution and Receiver services.

Extract : Extract is a process which we will create on the source database system to capture all the transactions for replay on target database. If source system is oracle it will read from redo logs or archive logs , for postgress it will read from Write-Ahead Logs or WAL , for mongodb it will read from oplog(operations log) . Once we found the real time transactions we will store it somewhere away in a physical file called Trail file.

Trail File: Trail files are physical golden gate objects on disks where transactions read by extract process are written . Data in this trail file is written in a consistent format hence it is used for replication to any supported database like oracle, mysql , postgress , mongodb ,etc. Next we will ship this trail file to a target system with the help of a distribution service.

Distribution Service: We will create it on the source system to manage trail files . With the help of this service we will create a distribution path . Distribution path will let the distribution service know the route to the target system/s where trail files should be sent to be listened by receiver service.

Receiver Service: We will create it on the target system to listen to the incoming trail files from the source system. We will use the web socket protocol , which creates a bridge between source and target system and any newly generated trail file is transferred in real time to the receiver service. Receiver service manages these incoming trail files to be read by the replicat process.

Replicat Process: This process is created on the target system with the sole purpose of reading data from the trail files , constructing sql’s ( dml or ddl ) and applying them on target database and voila !!!

Hope this simple explanation helps you understand the goldengate architecture.

References: https://docs.oracle.com/en/middleware/goldengate/core/23/coredoc