Statistics
The Statistics page offers a centralized and interactive data dashboard with monitoring and reporting features that allow you to track data flow in real-time and get automated error notifications in case something goes wrong. The data dashboard provides an easy view of the most important replication metrics at a minute or even second frequency.
Stats Job
The stats job (hvrstats) is responsible for gathering data that is displayed on the Statistics and pages. The stats job is created in a RUNNING state along with the repository tables during the installation of Fivetran Local Data Processing. If the stats job is not manually suspended, it will always run whenever the hub server is running.
Information for the stats job is gathered from the following files:
Fivetran Local Data Processing log files, in which all data from capture/integrate/refresh jobs is written to. The script called reads information from the log files and pulls them into the statistics system.
Router transaction files. These are the files, to which the capture job transports changes from the source location, while the integrate job reads from the router files and applies changes to the target location. The stats job does not read the transaction files, it only counts them and looks at their names.
*.cap_state file.
The repository table is responsible for maintaining statistical data and consists of a number of columns that store statistical information about data replication. The stats job modifies the table based on the data read from the above files. In particular, the table includes the metric_name and metric_value columns storing data on a variety of metrics captured by Fivetran Local Data Processing, such as capture/integrate latency, captured row counts, integrated change counts.
For a complete list of columns in the repository table and the type of data stored in them, see section .
Stats Aggregation
The stats job performs two types of aggregations when writing to the table:
Scope aggregation
Metrics are taken from HVR log files and are written to table at a certain scope defined by a channel name (column chn_name), location name (column loc_name) and table name (column tbl_name), which can be either named explicitly or regarded as '*' (which means applies to all channels, locations, tables).
For example:
There are 5 'captured inserts' with chn_name='chn1', loc_name='src' and tbl_name='tbl1' and 5 for 'captured inserts' with chn_name='chn1', loc_name='src' and tbl_name='tbl2'. The table will store these values, but it will also store value 10 for tbl_name='*', the sum of both values.
For more information on various scopes that can be defined, see option -s of command showing how the aggregation behaves with different scopes.
Time granularity aggregation
Metrics are gathered/output with a per-minute granularity. That is, the value of captured inserts for one-minute granularity means the number of rows inserted within that minute. These values can be aggregated up to 10 minutes, 1 hour, and 1 day. For more information on the time granularity option, see option -T of command .
There are also different types of aggregation functions. Some metrics are aggregated with the sum function, others - with the min or max functions.
Stats Purging
For the table to stay in a reasonable size, the data needs to be purged from that table. By default, automatic purging of the statistics data from the table is enabled for a hub. This can be configured using the option RETENTION POLICY in or by using the hub property .
To purge the statistics data immediately (as a one-time purge) from the table, use the command .
Viewing Statistics in the User Interface
To open the Statistics page in the user interface, navigate to the left sidebar and click STATISTICS.
The Statistics view contains the following user interface elements.
| Number | UI Element | Description |
|---|---|---|
| 1 | Scope | Allows to inspect historical metrics for a specific channel, location or table. |
| 2 | Graph Range | Allows to select a time period, for which metrics are shown. You can select the time range from the available ones (such as 'Last 10 min' or 'Last 7 days') or you can specify a custom time range by clicking Specify range and selecting the start date/time and end date/time in the Specify Custom Range dialog.
Another way to select a custom time period is to select it on the graph itself by clicking and dragging your mouse from the start time point to the end time point. The graph will then automatically zoom in to the selected time period. The selected time range will be applied to all graphs available in the dashboard. For example, to view statistics from June 25 to June 27, click and drag your mouse on the graph as shown below.  |
| 3 | Metrics | Displays the selected metrics graph in the dashboard or hide the unselected one. For more information on the available metrics, see section . |
| 4 | Hub | Name of the hub for which the statistics is displayed. |
| 5 | Options | The set of options to manage each graph:
|
| 6 | Metrics Legend | Indicates the metrics available on a corresponding graph. Each graph displays different metrics. For more information about the statistics metrics, see .
|
to set this option.Graphs and Metrics
The dashboard contains graphs displaying key performance metrics in your replication environment, such as latency, data volumes, and others. Following are the default graphs displayed on the Statistics page. For a complete list of metrics and their description, click the corresponding link in column List of Metrics.
| Graph Name | Description | List of Metrics |
|---|---|---|
| Latency | Displays the latency information for capturing and integrating changes (). Latency is the time (in seconds) taken for a transaction committed on the source system to be replicated (or committed) on the target system. This graph allows you to analyze the delay in data replication. | |
| Captured Changes Counts | Displays the total number of changes captured. | |
| Integrated Change Counts | Displays the total number of changes integrated. | |
| Integrated Changes (split by Table/Channel) | Displays the total number of changes integrated. This is basically an graph which is split using the metric Integrated Changes and scope Table/Channel. |
The following metrics graphs can be added under the Metrics selector. For a complete list of metrics and their description, click the corresponding link in column List of Metrics.
| Graph Type | Description | List of Metrics |
|---|---|---|
| Router Latency | Displays the latency detected in router files. | |
| Transactions | Displays the total number of transactions. | |
| Durations | Displays the time taken for replication in seconds. | |
| Integrate Burst Duration Average | Displays the average duration of burst integrate cycles. | |
| Speed | Displays the speed of replication. The unit of speed is indicated by the number of captured and integrated rows per a selected unit of time (see option Rate unit above). | |
| Cycles | Displays the number of and cycles. | |
| Byte I/O | Displays the size of captured row data and files in bytes before and after compression. | |
| Compression | Displays the compression ratio for row data. Captured row data is compressed when sent from the capture location to the hub and from the hub to the integrate location. | |
| Replicated Files | Displays the number of replicated files during file replication. | |
| Errors/Warnings | Displays the total number of errors and warnings. | |
| Router Rows | Displays the total number of rows in the transaction files. | |
| Router Bytes | Displays the total size of the transaction files in bytes. | |
| Router Files | Displays the total number of transaction files. |
Comments
0 comments
Please sign in to leave a comment.