Self-Hosted Software Guide
This document outlines the best practices and Cyware's recommendations for managing Cyware software products that are self-hosted by customers. The document includes recommendations and policies related to deployment, backup and restore procedures, monitoring, support, and upgrades.
Web, Application, and Database Servers
The customer is responsible for deploying and maintaining the web, application, and database servers. Before you deploy, review the system requirements and the deployment method prerequisites that are stated in the deployment guide of the respective Cyware products.
Backup and Disaster Recovery
The customer is responsible for all data backup and disaster recovery processes. The processes include:
Providing and maintaining the hardware necessary to deploy and operate the software
Logical and physical security of all server and network components
Designing and maintaining data backup, restoration, and disaster recovery processes
Maintaining the database and operating system, including vulnerability and maintenance patching
Any fail-over activities and processes to safeguard data in a disaster recovery scenario
General Backup Recommendations
Application and database backups should be scheduled at certain frequencies, based on your specific business requirements. Generally, the higher the transaction volume, the more frequent backups should take place. Taking a backup should be possible online, parallel to normal database update activities to save a transaction-consistent state of the database.
You can take data backups using the following backup concepts:
Online Backup: Taking a backup during a normal database update session.
Offline Backup: Taking a backup when database updates are disabled (the server is down, in standby mode, or in read-only mode).
Full Backup: Taking a backup of the complete database.
Partial Backup: Taking a backup of a logical or physical subset of the database.
Incremental Backup: Taking a backup of the data that has been changed since the previous backup. A recovery is only possible if a previous full backup is available, as well as all following incremental backups.
Backups can be done monthly, weekly, daily, hourly, by the minute, or more likely a combination of some or all of them. A simple backup plan includes a rotating monthly and weekly backup combined with several hourly backups every day. A more complex scenario includes retaining 12 monthly backups.
Note
Cyware recommends taking a full online backup once a day with a backup retention period of at least 7 days.
To know more about the backup and recovery processes, refer to the respective database documentation.
Postgres Backup Guide: https://www.postgresql.org/docs/current/backup.html
MySQL Backup Guide: https://dev.mysql.com/doc/refman/8.0/en/backup-and-recovery.html
General Recovery Recommendations
In addition to taking backups, it is important to restore backups regularly to validate the back up procedures. The best practice is to restore every backup of the production environment to an alternate server and then validate the integrity of the restored databases. This practice provides the following advantages:
Validates the recoverability of the backup media.
Validates the physical integrity of the databases in the backup, avoiding the problem of false reporting of integrity errors when running integrity checks on volatile databases affected by ongoing updates.
Provides a warm copy of the backup, substantially reducing the time required to restore the backup in the event of a disaster.
Establishes a last known good backup.
The backup strategies described in this document preserve the physical structure of the database. A clean integrity check of the restored copy ensures that the integrity of the production database was sound at the time of the backup. However, an integrity error detected on the restored copy of a database does not necessarily imply that there are integrity problems on the production database. For example, there could be errors in the backup media. If you discover an integrity error in the restored database, immediately run an integrity check on the production database to verify the integrity of the production system.
Note
Cyware recommends performing a restore validation once per month but not longer than once a quarter.
Infrastructure and Application Monitoring
In a self-hosted installation, the customer should utilize an effective monitoring system to track the performance, health, and availability of the installation. The recommended metrics can be broadly classified into the following two categories:
Work metrics
Resource Metrics
Work Metrics
Work metrics provide top-level health insights into the infrastructure. These metrics provide an estimate of how much work resources are being produced and whether to raise an alert when specific thresholds are reached or when system failures occur. Some examples of work metrics include request rates and response times.
Resource Metrics
Resource metrics provide deep-level insights into the system’s current state. These metrics are useful for investigative purposes whenever a system fails to operate normally. Some examples of resource metrics include disk space utilization, memory usage, and network availability.
Cyware recommends that self-hosted customers monitor primary metrics that are important for user experience, application, and infrastructure health. Following are some examples to implement the resource metrics monitoring:
Request and response duration
Number of requests per second
Request error rate and HTTP status codes
Database connection health
Worker/transaction queue length
External API request response duration and error rates
High memory and CPU utilization
Network I/O rates and errors
Low disk space and disk failure
Self-Hosted Customer Support Policy
Configuration
To facilitate Cyware’s ability to assist, a self-hosted customer is encouraged to follow the system specification recommendations for the self-hosted installation found in the respective software product deployment guide.
Remote Access
To facilitate Cyware’s ability to assist, a self-hosted customer must provide Cyware with remote access to the software installation, including via a desktop-sharing application.
Scope
Cyware provides general guidance for the customer support activities for the self-hosted software. This does not include performing the following services and are the responsibilities of the customer:
Infrastructure planning and deployment
Implementation services
Configuration services
Integration services
Customization services and other custom software development
Assistance with administrative functions
Self-Hosted Customer Support Responsibilities
To understand the responsibilities of customers and Cyware for various customer support activities, refer to the following table:
Activity | Cyware SaaS | Self-Hosted Software |
---|---|---|
Planning Phase | ||
Capacity Sizing | Cyware | Customer |
Deployment Design | Cyware | Customer |
Hardware Acquisition | Cyware | Customer |
Production Deployment Phase | ||
Hardware and Operating System Deployment | Cyware | Customer |
Virtualization | Not Available | Customer |
Load Balancer | Cyware | Customer |
Database Setup | Cyware | Customer |
Instances Deployment | Cyware | Customer |
Integration with SSO, Email, and others. | Cyware | Cyware Assisted |
High Availability Build-Out | Cyware | Customer |
Disaster Recovery Build-Out | Cyware | Customer |
On-Going Production Operations Phase | ||
Instance Cloning | Cyware | Customer |
Backup and Restore | Cyware | Customer |
Adding Nodes | Cyware | Customer |
Application Upgrades | Cyware | Cyware Assisted |
Operating System Upgrades | Cyware | Customer |
Database Upgrades | Cyware | Customer |
Infrastructure Monitoring | Cyware | Customer |
Application Monitoring | Cyware | Customer |
Application Troubleshooting | ||
Review Server Configuration | Cyware | Customer |
Review Application Properties | Cyware | Cyware Assisted |
Connection Settings | Cyware | Cyware |
Custom Application | Customer | Customer |
Custom Integration | Customer | Customer |
Infrastructure Troubleshooting | ||
Review Server Resources | Cyware | Customer |
Review Network Environment | Cyware | Customer |
Review Operating System Configuration | Cyware | Customer |
Database Troubleshooting | ||
Review Database Configuration | Cyware | Customer |
Review Memory Settings | Cyware | Customer |
Performance Review | Cyware | Cyware Assisted |
Customer designation means the activity is the sole responsibility of the customer. Cyware may provide assistance for the activity pursuant to an agreed upon Statement of Work for Professional Services.
Cyware designation means the activity is the sole responsibility of Cyware.
Cyware Assisted designation means the activity is primarily the customer’s responsibility with Cyware providing assistance to the customer, as reasonably feasible. Cyware’s assistance is confined to creating an incident, addressing high-level customer questions, providing links to self-help resources, collecting data, and analyzing logs (if feasible).