Skip to main content

General Documents

Self-Hosted Software Guide

This document outlines the best practices and Cyware's recommendations for managing Cyware software products that are self-hosted by customers. The document includes recommendations and policies related to deployment, backup and restore procedures, monitoring, support, and upgrades.

Web, Application, and Database Servers

The customer is responsible for deploying and maintaining the web, application, and database servers. Before you deploy, review the system requirements and the deployment method prerequisites that are stated in the deployment guide of the respective Cyware products.

Backup and Disaster Recovery

The customer is responsible for all data backup and disaster recovery processes. The processes include:

  • Providing and maintaining the hardware necessary to deploy and operate the software

  • Logical and physical security of all server and network components

  • Designing and maintaining data backup, restoration, and disaster recovery processes

  • Maintaining the database and operating system, including vulnerability and maintenance patching

  • Any fail-over activities and processes to safeguard data in a disaster recovery scenario

General Backup Recommendations

Application and database backups should be scheduled at certain frequencies, based on your specific business requirements. Generally, the higher the transaction volume, the more frequent backups should take place. Taking a backup should be possible online, parallel to normal database update activities to save a transaction-consistent state of the database.

You can take data backups using the following backup concepts:

  • Online Backup: Taking a backup during a normal database update session.

  • Offline Backup: Taking a backup when database updates are disabled (the server is down, in standby mode, or in read-only mode).

  • Full Backup: Taking a backup of the complete database.

  • Partial Backup: Taking a backup of a logical or physical subset of the database.

  • Incremental Backup: Taking a backup of the data that has been changed since the previous backup. A recovery is only possible if a previous full backup is available, as well as all following incremental backups.

Backups can be done monthly, weekly, daily, hourly, by the minute, or more likely a combination of some or all of them. A simple backup plan includes a rotating monthly and weekly backup combined with several hourly backups every day. A more complex scenario includes retaining 12 monthly backups.

Note

Cyware recommends taking a full online backup once a day with a backup retention period of at least 7 days.

To know more about the backup and recovery processes, refer to the respective database documentation.

  • Postgres Backup Guide: https://www.postgresql.org/docs/current/backup.html

  • MySQL  Backup Guide: https://dev.mysql.com/doc/refman/8.0/en/backup-and-recovery.html

General Recovery Recommendations

In addition to taking backups, it is important to restore backups regularly to validate the back up procedures. The best practice is to restore every backup of the production environment to an alternate server and then validate the integrity of the restored databases. This practice provides the following advantages:

  • Validates the recoverability of the backup media.

  • Validates the physical integrity of the databases in the backup, avoiding the problem of false reporting of integrity errors when running integrity checks on volatile databases affected by ongoing updates.

  • Provides a warm copy of the backup, substantially reducing the time required to restore the backup in the event of a disaster.

  • Establishes a last known good backup.

The backup strategies described in this document preserve the physical structure of the database. A clean integrity check of the restored copy ensures that the integrity of the production database was sound at the time of the backup. However, an integrity error detected on the restored copy of a database does not necessarily imply that there are integrity problems on the production database. For example, there could be errors in the backup media. If you discover an integrity error in the restored database, immediately run an integrity check on the production database to verify the integrity of the production system.

Note

Cyware recommends performing a restore validation once per month but not longer than once a quarter.

Infrastructure and Application Monitoring

In a self-hosted installation, the customer should utilize an effective monitoring system to track the performance, health, and availability of the installation. The recommended metrics can be broadly classified into the following two categories:

  • Work metrics

  • Resource Metrics

Work Metrics

Work metrics provide top-level health insights into the infrastructure. These metrics provide an estimate of how much work resources are being produced and whether to raise an alert when specific thresholds are reached or when system failures occur. Some examples of work metrics include request rates and response times.

Resource Metrics

Resource metrics provide deep-level insights into the system’s current state. These metrics are useful for investigative purposes whenever a system fails to operate normally. Some examples of resource metrics include disk space utilization, memory usage, and network availability.

Cyware recommends that self-hosted customers monitor primary metrics that are important for user experience, application, and infrastructure health. Following are some examples to implement the resource metrics monitoring:

  • Request and response duration

  • Number of requests per second

  • Request error rate and HTTP status codes

  • Database connection health

  • Worker/transaction queue length

  • External API request response duration and error rates

  • High memory and CPU utilization

  • Network I/O rates and errors

  • Low disk space and disk failure

Self-Hosted Customer Support Policy

Configuration

To facilitate Cyware’s ability to assist, a self-hosted customer is encouraged to follow the system specification recommendations for the self-hosted installation found in the respective software product deployment guide.

Remote Access

To facilitate Cyware’s ability to assist, a self-hosted customer must provide Cyware with remote access to the software installation, including via a desktop-sharing application.

Scope

Cyware provides general guidance for the customer support activities for the self-hosted software. This does not include performing the following services and are the responsibilities of the customer:

  • Infrastructure planning and deployment

  • Implementation services

  • Configuration services

  • Integration services

  • Customization services and other custom software development

  • Assistance with administrative functions

Self-Hosted Customer Support Responsibilities

To understand the responsibilities of customers and Cyware for various customer support activities, refer to the following table:

Activity

Cyware SaaS

Self-Hosted Software

Planning Phase

Capacity Sizing

Cyware

Customer

Deployment Design

Cyware

Customer

Hardware Acquisition

Cyware

Customer

Production Deployment Phase

Hardware and Operating System Deployment

Cyware

Customer

Virtualization

Not Available

Customer

Load Balancer

Cyware

Customer

Database Setup

Cyware

Customer

Instances Deployment

Cyware

Customer

Integration with SSO, Email, and others.

Cyware

Cyware Assisted

High Availability Build-Out

Cyware

Customer

Disaster Recovery Build-Out

Cyware

Customer

On-Going Production Operations Phase

Instance Cloning

Cyware

Customer

Backup and Restore

Cyware

Customer

Adding Nodes

Cyware

Customer

Application Upgrades

Cyware

Cyware Assisted

Operating System Upgrades

Cyware

Customer

Database Upgrades

Cyware

Customer

Infrastructure Monitoring

Cyware

Customer

Application Monitoring

Cyware

Customer

Application Troubleshooting

Review Server Configuration

Cyware

Customer

Review Application Properties

Cyware

Cyware Assisted

Connection Settings

Cyware

Cyware

Custom Application

Customer

Customer

Custom Integration

Customer

Customer

Infrastructure Troubleshooting

Review Server Resources

Cyware

Customer

Review Network Environment

Cyware

Customer

Review Operating System Configuration

Cyware

Customer

Database Troubleshooting

Review Database Configuration

Cyware

Customer

Review Memory Settings

Cyware

Customer

Performance Review

Cyware

Cyware Assisted

  • Customer designation means the activity is the sole responsibility of the customer. Cyware may provide assistance for the activity pursuant to an agreed upon Statement of Work for Professional Services.

  • Cyware designation means the activity is the sole responsibility of Cyware.

  • Cyware Assisted designation means the activity is primarily the customer’s responsibility with Cyware providing assistance to the customer, as reasonably feasible. Cyware’s assistance is confined to creating an incident, addressing high-level customer questions, providing links to self-help resources, collecting data, and analyzing logs (if feasible).