Important Notes When Upgrading from HVR 5
This section describes the underlying changes introduced in Fivetran Local Data Processing when compared to the HVR 5.x versions. The fundamental change is that HVRGUI in HVR 5 has been migrated to the web application that comes with a completely new and functionality. Major changes were also introduced to the process architecture, the structure of catalog tables (renamed as ), CLI commands, actions, jobs, import and export system, etc.
- HVR version 6.x is not compatible with the versions 5.x and 4.x. So, ensure that HVR version 6.x is installed on the Hub machine and Agent machine.
- Only installation is supported on the Unix platforms (aix_6.1-powerpc-64bit, solaris_10-sparcv9-64bit).
For the procedure to upgrade from HVR 5 to Fivetran Local Data Processing, see .
Functionality Removed in Fivetran Local Data Processing
- Cloud-licensed agents (cloud-licensed hubs still exist).
- HVR hub on Unix (non-Linux).
- HVR for MacOS platform.
- Documentation set bundled inside our product.
Architecture
| HVR 5 | Local Data Processing |
|---|---|
| HVR hub database contains a set of catalog tables. HVR hub runs on a hub machine and transmits data using HVR's remote protocol. | Local Processing Hub Server has a single repository database that contains repository tables. The repository database can contain multiple hubs (multi-tenancy). Hub is an abstract collection of channels, locations, tables, and actions, etc. Local Processing Hub Server runs on a hub machine and transmits data using the HTTP or HTTPS protocol. Local Processing Hub Server has a set of users that can be authenticated using the following methods:
|
Methods to communicate with HVR 5 (interfaces):
| Methods to communicate with Local Data Processing (interfaces):
|
| HVR GUI runs the Scheduler. | Local Processing Hub Server runs the Scheduler. |
| HVR remote protocol is used to connect between a hub machine and the HVR Agent on a remote source/target machine, and between the HVR GUI and the hub machine. | High-Volume Agent (HVA) protocol is used to connect to an HVA on a remote source/target machine. |
The following image shows the differences (highlighted in blue) between the process architectures in HVR 5 and Local Data Processing.
Licensing
In HVR5, applying an HVR license is a separate step performed after HVR installation. After receiving the license file by email, a user has to save into the HVR_Home/lib/ directory on a hub machine.
In Local Data Processing, applying a license file is part of the Hub System setup workflow ( and ).
Installing Fivetran Local Data Processing
In HVR5, there is only one workflow for installing HVR as a hub or as remote agent.
In Local Data Processing, the redesigned installer for Windows splits the installation process into separate workflows:
Neither: HVR can only be used for command-line control of a remote hub machine.
High-Volume Agent
In Local Data Processing, the is now a part of a unified system setup workflow that allows to easily configure a secure, encrypted and properly authenticated connection from a hub to an agent. This is implemented by introducing agent setup modes and agent authentication methods. For more information about HVA, see section .
Agent Authentication
The following agent are available in Local Data Processing:
Limiting access from hubs with only specific hub certificates, so that the agent only accept connection from a particular hub machine (this allows anonymous connections).
Agent user authentication (using agent user credentials)
Combination of the above two methods
For more information about agent authentication, see section .
Agent Setup Mode
Agent setup mode is used for configuring HVA service. There are two types of agent setup mode:
Time-based setup mode
Token-based setup mode
For more information, see section .
Catalog Tables
The HVR 5 'catalog tables' located inside a hub database are renamed as '' located inside a repository database for Local Data Processing.
- The HVR 5 tables HVR_CONFIG_ACTION, HVR_JOB_RESOURCE*, and HVR_JOB_PARAM were removed in Local Data Processing.
- In Local Data Processing, because a repository database can include multiple hubs (multi-tenancy), a column called hub_name was added as a key column to almost all repository tables for distinguishing between jobs, database objects (channels, locations, tables, etc.) and files relating to a particular hub.
- All changes to the repository tables are tracked in the HVR_COUNTER table.
- New are added, containing information about users (), user properties (), user hub properties (), repository properties (), hub properties (), and location properties ().
- Each (, , , , , , , ) has two time-dimension columns insert_tstamp and delete_tstamp to better manage changes to the repository tables. This time dimensional model allows a user to make interactive changes to a channel definition without immediately introducing them into replication.
HVR 5 allows you to make changes to the definition tables directly using SQL statements, e.g. a user can insert a column into the table. Local Data Processing does not support direct modification of the definition tables using SQL statements. Instead, a user has three options for making such changes: through the User Interface, Command Line Interface (CLI), or REST API.
See section .
The following diagram shows the changes that have been introduced in the repository tables. The tables and columns highlighted in brown are those migrated from HVR 5. The ones highlighted in blue are new tables and columns added in Local Data Processing.
HVR_CONFIG Reorganization
The following image shows the difference between the HVR_CONFIG directory structures in HVR 5 and Local Data Processing. Because of multi-tenancy architecture introduced in Local Data Processing, the HVR_CONFIG directory is now partitioned into hub sub-directories. The benefit of the new HVR_CONFIG structure is that it matches the REST API object naming structure.
HVR_HOME Reorganization
The following image shows the difference between the HVR_HOME directory structures in HVR 5 and Local Data Processing. One significant change is that the agent directory has been renamed to plugin in Local Data Processing. The plugin scripts are stored in the plugin_examples directory, and to activate them, you need to explicitly move them to the HVR_CONFIG/plugin/ directory.
User Interface
The “thick” graphical user interface (HVRGUI) has been replaced with a "thin" web-based user interface. The Local Data Processing is designed to be visually appealing and intuitive providing a better user experience.
One advantage of the new user interface is that it assists both novice and expert users. In the HVR 5 GUI, both the essential features and advanced functionality appeared all at once, whereas in Local Data Processing, they were separated into two layers. A layer of essential functionality is initially visible to a user so that a novice user who has no experience in HVR product could quickly set up a replication channel ready for production. At the same time, for the expert users who have solid knowledge in HVR, the advanced functionality is smartly hidden below the essential features and can be expanded if needed.
Another feature that makes Local Data Processing user interface easy to interact with is that certain input fields that are only required once a user has chosen a specific option or function remain hidden until the user has selected that option, they then expand underneath. This helps users to focus on the things they need to fill over things they might not need.
The Local Data Processing user interface is very informative, it provides short descriptions of basic HVR concepts and options and explanatory workflow that aids a user in setting up a replication channel, creating a location, and understanding the HVR system as a whole.
See section .
Command Line Interface
In HVR 5, the command line requires user credentials to connect to a hub database (e.g. Hvrinit -h sqlserver -u myuser/pwd hubdb mychn). In Local Data Processing, options -h (database class) and -u (database user and password) were removed from all the commands that used to require them.
The Local Data Processing command line interface includes two types of command lines:
- Direct command line - authentication to the HVR system is done using the credentials of an hvr user logged in to a hub machine). In this case, only the hub name is needed to run a command (e.g. hvractivate myhub mychn).
- Remote command line - authentication to the HVR system is done via the HVR's REST interface. In this case, a user needs to use command
-Rto log in, the authentication token is stored for a certain period of time in ~/.config/hvr. Option-Rmust be used to run commands (except ) though the remote command line.
New option --help along with an HVR command displays a short description of all options available for a specific command.
For more information, see section .
Commands
The following commands were either removed or replaced with new commands:
| HVR 5 Commands | Status in Local Data Processing |
|---|---|
| hvrcatalogcreate | Replaced with option -c |
| hvrcatalogdrop | Replaced with option -d |
| hvrcatalogexport | Replaced with |
| hvrcatalogimport | Replaced with |
| hvrcryptdb | Removed |
| hvrfailover | Removed |
| hvrgui | Removed |
| hvrinit | Renamed to |
| hvrlivewallet | Replaced with location property |
| hvrlogrelease | Removed |
| hvrmaint | Split into commands and (internal) hvrmaintjob. See for more details. |
| hvrremotelistener | Renamed to |
| hvrproxy | Currently unavailable, may be implemented in future release. |
| hvrstatistics | Removed |
| hvrswitchtable | Currently unavailable, may be implemented in future release. |
| hvrtestlistener | Renamed to |
| hvrvalidpw | Removed. This authentication plugin for LDAP is not available in Local Data Processing. It was determined that most/all Linux users would integrate their LDAP in their Linux systems for regular login. So in Local Data Processing they could use for LDAP authentication. |
| hvr_boot | Unix reboot steps show script lines instead |
Hvrmaint
The HVR 5 command Hvrmaint has been split into two parts in Local Data Processing:
hvralert
Runs outside HVR Hub Server (by )
Set up inside the or via the command line interface ()
Alert messages have hypertext links
Latency SLAs defined inside a channel
- action with parameter LatencySLA
hvrmaintjob
- Automatically rotates log files
The following are HVR 5 Hvrmaint options that have been converted or removed in Local Data Processing.
| HVR 5 Hvrmaint options | Status in Local Data Processing |
|---|---|
| -email_to=addr1[;addr2] | Moved to alert property |
| -email_from=addr | Moved to alert property |
| -smtp_server=server | Moved to alert property |
| -smtp_port=port | Moved to alert property |
| -smtp_starttls | Moved to alert property |
| -smtp_user=user | Moved to alert property |
| -smtp_pass=pass | Moved to alert property |
| -error_limit=N | Moved to alertproperty |
| -slack_channel=chn | Moved to alert property |
| -slack_webhook_url=url | Moved to alert property |
| -snmp_version=vers | Moved to alert property |
| -snmp_heartbeat | Moved to alert property |
| -snmp_hostname=host | Moved to alert property |
| -snmp_port=port | Moved to alert property |
| -snmp_community=str | Moved to alert property |
| -sns_destination | Moved to alert property |
| -sns_access_key | Moved to alert property |
| -sns_secret_key | Moved to alert property |
| -scan_ignore=patt | Moved to alert property |
| -scan_channel=chn | Moved to alert property |
| -scan_location=loc | Moved to alert property |
| -email_repeat_suppression | Moved to alert property |
| -check_logfile_growth | Removed |
| -disable | Available from the dialog in and command -D |
| -task_group=group | Removed |
| -test_scheduler | Removed. Managed by command . |
| -clear_past_errors | Available from the dialog in and command -C |
| -scan_hvr_out | Removed. Managed by command . |
| -optnm_email_test | Available from the dialog in and command -t |
| -optnm_slack_test | Available from the dialog in and command -t |
| -optnm_sns_test | Available from the dialog in and command -t |
| -optnm_snmp_test | Available from the dialog in and command -t |
| -send_slack_only_when_errors_or_warnings | Moved to alert property =FALSE |
| -email_only_when_errors_or_warnings | Moved to alert property =FALSE |
| -sns_only_when_errors_or_warnings | Moved to alert property =FALSE |
| -send_slack_only_when_errors | Moved to alert property =TRUE |
| -sns_only_when_errors | Moved to alert property =TRUE |
| -email_only_when_errors | Moved to alert property =TRUE |
| -sns_repeat_suppression | Moved to alert property |
| -slack_repeat_suppression | Moved to alert property |
| -email_repeat_suppression | Moved to alert property |
| -latency_limit=dur | Removed |
| -latency_channel=chn | Removed |
| -latency_location=loc | Removed |
| -output=file | Removed |
| -version | Removed |
| -start | Removed |
| -stop | Removed |
| -archive_files=patt | Removed |
| -archive_keep_days=N | Removed |
| -archive_compress | Removed |
| -archive_concat | Removed |
| -archive_dir | Removed |
REST API Interface
HVR REST API is a completely new interface implemented in Local Data Processing providing about 200 Rest API end-points grouped into about 20 'interfaces'. Most REST end-points accept and return JSON requests and responses.
An important purpose of the REST API is that it provides a security layer, as it is responsible for user authorization. Each REST interface is assigned an access level, e.g. you need a ReadExec access level on a hub to post a compare event.
Location Group Membership
In HVR 5, a location can be a member of multiple location groups in a channel.
Local Data Processing imposes the following restrictions on the locations included in a channel:
- a location must be a member of a location group
- a location can only be a member of one location group.
Actions
In Local Data Processing, backslash has been removed before the parameter name of an action. For example,
- HVR 5: in action Capture
- Local Data Processing: in action Capture
The HVR 5 action LocationProperties and its parameters, certain parameters of actions Capture and Integrate, as well as some environmental variables were converted to in Local Data Processing. The complete list of changes are listed in the following sections below.
Action Scope
HVR 5 allows defining actions for a specific scope, e.g. actions can be defined for both a location and a location group, or a location group and a channel. In total HVR 5 allowed 24 combinations of action scopes. In Local Data Processing, this behavior is changed.
In Local Data Processing, the following action scopes are supported:
- Channel scope: all channels (*) or a specific channel
- Location scope: all locations (*) or a location group or a specific location
- Table scope: all tables or a table group or a specific table. Requires a specific channel to be selected in the channel scope.
Actions and defined on a specific location will be effective in a channel only when the location is a member of the channel's location group.
Actions and Parameters
The section lists HVR 5 actions and action parameters that were either converted to the location properties, renamed, removed, or replaced with new parameters in Local Data Processing.
| HVR 5 Actions and Parameters | Status in Local Data Processing | |
|---|---|---|
| Capture | /ArchiveLogFormat | Moved to location property |
| Capture | /ArchiveLogOnly | Moved to location property with value ARCHIVE_ONLY |
| Capture | /ArchiveLogPath | Moved to location property |
| Capture | /CheckpointFrequency | Moved to location property |
| Capture | /CheckpointRetention | Moved to location property |
| Capture | /CheckpointStorage | Moved to location property |
| Capture | /KeyOnlyCaptureTable | Moved to location property |
| Capture | /LogJournal | Split into two separate location properties: |
| Capture | /LogJournalSysSeq | Moved to location property |
| Capture | /LogReadMethod | Moved to location property with values DIRECT, SQL, or LOGMINER |
| Capture | /LogTruncate | Moved to location property |
| Capture | /QuickToggle | Moved to location property |
| Capture | /SupplementalLogging | Moved to location property |
| Capture | /ToggleFrequency | Moved to location property |
| Capture | /TriggerBased | Moved to location property with value DB_TRIGGER |
| Capture | /XLogDirectory | Moved to location property |
| ColumnProperties | /Identity | Removed |
| FileFormat | /AvroCompression | Replaced with new parameter |
| Integrate | /Burst | Moved inside new parameter having two values:
|
| Integrate | /MessageBundling | Moved to location property |
| Integrate | /MessageBundlingThreshold | Moved to location property |
| Integrate | /MessageCompress | Moved to location property |
| LocationProperties | Removed. Action LocationProperties is not available in Local Data Processing, its parameters are either removed or converted to location properties. | |
| LocationProperties | /BulkAPI | Moved to location property |
| LocationProperties | /CaseSensitiveNames | Moved to location property |
| LocationProperties | /CloudLicense | Moved to location property |
| LocationProperties | /IntermediateDirectory | Moved to location property |
| LocationProperties | /Order | Removed |
| LocationProperties | /Proxy | Split into following separate location properties: |
| LocationProperties | /S3Encryption | Moved to location property |
| LocationProperties | /S3Encryption | Moved to location property |
| LocationProperties | /S3Encryption | Moved to location property |
| LocationProperties | /S3Encryption | Moved to location property |
| LocationProperties | /S3Encryption | Moved to location property |
| LocationProperties | /S3Encryption | Moved to location property |
| LocationProperties | /S3Encryption | Moved to location property |
| LocationProperties | /S3Encryption | Moved to location property |
| LocationProperties | /S3Encryption | Moved to location property |
| LocationProperties | /SerialMode | Moved to location property |
| LocationProperties | /SslLocalCertificateKeyPair | Split into two hub properties: |
| LocationProperties | /SslRemoteCertificate | Moved to location property |
| LocationProperties | /StagingDirectoryHvr | Moved to location property |
| LocationProperties | /StagingDirectoryHvr | The scheme moves to location property , and the credentials to appropriate credential properties. |
| LocationProperties | /StagingDirectoryDb | Moved to location property |
| LocationProperties | /StagingDirectoryCredentials | Moved to location property |
| LocationProperties | /StagingDirectoryCredentials | Moved to location property |
| LocationProperties | /StagingDirectoryCredentials | Moved to location property |
| LocationProperties | /StagingDirectoryCredentials | Moved to location property |
| LocationProperties | /StagingDirectoryCredentials | Moved to location property |
| LocationProperties | /StagingDirectoryCredentials | Moved to location property |
| LocationProperties | /StagingDirectoryCredentials | Moved to location property |
| LocationProperties | /StagingDirectoryCredentials | Moved to location property |
| LocationProperties | /StateDirectory | Split into two separate location properties: |
| LocationProperties | /ThrottleKbytes | Removed |
| LocationProperties | /ThrottleMillisecs | Removed |
| Scheduling | /StatsHistory | Moved to hub property |
| TableProperties | /DuplicateRows | Replaced with new parameter |
Jobs
HVR job is a process that performs a certain task, such as activating replication, capturing/integrating changes, refreshing data, comparing data, etc.
| HVR 5 | Local Data Processing |
|---|---|
There are three options for compare and refresh jobs:
| , , and operations are performed by special event-based jobs. The always runs the activate, refresh, and compare jobs. The jobs are controlled by the state of the events in the HVR's event system. This applies to , , and . For example, when you perform the Compare operation, a compare event is created in the HVR's event system, then the mychn-cmp-src-tgt compare job is created in the Scheduler, which runs the compare job. If the mychn-cmp-src-tgt job is restarted from the SUSPEND state, it will resume from where it was interrupted rather than starting from the beginning. When you start the compare operation and there is an existing compare event with the same job name in the PENDING or RUNNING state, then the existing event is canceled (FAILED) by the new compare event. Job Chaining The job chaining option (the Activate Replication dialog in the UI, commands-J and -J) allows a user to manage the sequence of events to be started after replication activation, such as refresh, capture, and integrate. |
Import and Export
Local Data Processing import and export system offers a lot more flexibility and powerful features than that of HVR 5. For example, in Local Data Processing, you can download an action and reload it into a different channel, you can export a file containing properties to connect to a remote agent and share it with your colleagues, you can export changes made to a channel definition into a file and apply those changes to a different hub.
The following table reflects major changes to the HVR export/import system.
| HVR 5 | Local Data Processing |
|---|---|
| Commands Hvrcatalogimport and Hvrcatalogexport | Commands and |
| Export and import to/from XML file | Export and import to/from JSON file |
Export/import of:
| Export/ import of:
|
| Not available | Command loads objects exported from the |
| Not available | Apply Definition Change events:
|
| Not available | Use command to make arbitrary changes to a channel, such as delete or delete a location, rename a channel, add or delete actions, and much more. For example, see section on the page. |
Environment Variables
The following HVR 5 environment variables were converted to the in Local Data Processing.
| HVR 5 Environment Variables | Status in Local Data Processing |
|---|---|
| $HVR_ASM_CONNECT | Split into three location properties: |
| $HVR_ASM_HOME | Moved to location property |
Properties
Properties is a new concept in Local Data Processing that specifies the characteristics/attributes of the HVR components:
The following table shows the place where the properties for each HVR component are stored and the HVR commands for managing these properties.
| Object | Property storage | HVR Commands |
|---|---|---|
| Agent | $HVR_CONFIG/etc/hvragentconfig.conf | |
| Alerts | $HVR_CONFIG/hubs/{hub}/alerts/{alert}.conf | |
| Hub Server | $HVR_CONFIG/etc/hvrhubserver.conf | |
| Hubs | Repository table | |
| License | Repository property .
| |
| Locations | Repository table | |
| Repository | Repository table | |
| Users | Repository tables and |
Snapshots
Local Data Processing allows you to create a and save it to a zip file that stores all the data related to the hub, i.e. the hub snapshot metadata, information about channels, locations, location groups, tables, columns, actions and information from the HVR_CONFIG directory (log files, enroll files, capture state files, catalog cache files, etc.). The snapshot can be used to move the hub to a different repository, for reporting purposes, or for sharing with other users.
Smart Links
dialogs contain links pointing to the relevant documentation pages, as well as links that can navigate you between different user interface pages.
The dialog displays smart links that can navigate you to the relevant UI pages. For example, clicking the name of a compare or refresh event will open the appropriate page.
Comments
0 comments
Please sign in to leave a comment.