Skip to main content

Cyware Fusion and Threat Response

Troubleshooting

This topic provides some basic commands that are useful for troubleshooting issues during and after the deployment:

Manage Docker Swarm

Following are some basic Docker Swarm commands that are useful for troubleshooting issues:

  • To check the Docker Swarm Node status, run the following command. This command lists all the nodes of the Docker Swarm Manager.

    docker node ls

    Sample Output

    ID                           HOSTNAME        STATUS  AVAILABILITY  MANAGER STATUS
    41bcef6utixb0l0ca7gxuivsj0    swarm-worker2   Ready   Active
    538ciaotwjuritcdtn9npbnkuz    swarm-worker1   Ready   Active
    6e216jshn25ckzbvmwlnh5jr3g *  swarm-manager1  Ready   Active        Leader

    Note

    The status for each node should be Ready. If the above command times out or throws any error, then you must set up the swarm cluster again.

  • To list all Docker services, run the following command. This command lists services are running in the swarm. The REPLICAS column shows both the actual and desired number of tasks for the service. If the service is in replicated-job or global-job, it will additionally show the completion status of the job as completed tasks over total tasks the job will execute.

    docker service ls

    Sample Output

    ID            NAME      MODE            REPLICAS             IMAGE
    4c8wgl7q4ndfd  frontend  replicated      5/5                  nginx:alpine
    5dmu1ept4cxcf  redis     replicated      3/3                  redis:3.0.6
    6hh08h9uu8uwr  job       replicated-job  1/1 (3/5 completed)  nginx:late
  • To check if containers are up and running, run the following command:

    docker ps
  • If the actual and desired number of replicas for a service is not the same, run the following command to check for the cause:

    docker service ps --no-trunc <service-name>
  • To check the Docker Overlay Network info, run the following command:

    docker network ls
  • To check if the Docker systemd service is running, run the following command:

    systemctl status docker
  • To check the Docker systemd logs, run the following command:

    journalctl -u -f docker.service
  • To view Docker service-wise stats on the current server, run the following command:

    docker stats
  • To restart the Docker service on the server, run the following command. This command restarts all services that run as Docker containers.

    sudo systemctl restart docker

    Note

    You must manually restart the services that do not run as Docker containers.

  • To update a service, run the following command:

    docker service update <service_name> --force
  • To delete a stack, run the following command:

    docker stack rm <stack_name>
  • To go inside the containers, run the following command:

    docker exec -it $(docker ls -q- -f name=<service_name>) bash
  • To go inside a specific container, run the following command:

    docker exec -it <container_id> bash
  • To redeploy a stack, run the following command:

    docker stack deploy -c <compose-file-name> <stack_name>

View Application Logs

Following are some commands to view the application logs of various Respond services:

  • To view the backend application logs, run the following command:

    cd /apps/cyware/logs/cftr_backend/application
    tail -f <logfile.name>
  • To view the Celery beat logs, run the following command:

    cd /apps/cyware/logs/celery-beat
    tail -f <logfile.name>
  • To view the Celery notification logs, run the following command:

    cd /apps/cyware/logs/celery-notification
    tail -f <logfile.name>
  • To view the Celery worker logs, run the following command:

    cd /apps/cyware/logs/celery-worker
    tail -f <logfile.name>
  • To view the frontend logs, run the following command:

    cd /apps/cyware/logs/cftr_frontend
    tail -f <logfile.name>

View Datastore Logs (Docker Version)

Following are some commands to view the logs of the database services that are deployed as Docker containers.

  • To view the Elasticsearch logs, run the following command:

    docker service logs -f db_elasticsearch
  • To view the Redis logs, run the following command:

    docker service logs -f db_redis
  • To view the PostgreSQL logs, run the following command:

    docker service logs -f db_postgres
  • To view the Object Storage logs, run the following command:

    docker service logs -f db_object_storage

View Datastore Logs (Systemd Version)

Following are some commands to view the logs of the database services that are not deployed as Docker containers.

  • To view the PostgreSQL logs, run the following command:

    postgres
  • To view the Redis logs, run the following command:

    tail -f /var/log/redis/*.log
  • To view the Elasticsearch logs, run the following command:

    tail -f /

Check Network Connectivity

Following are some commands to check the network connectivity.

  • To check the connectivity on a TCP port, run the following command:

    telnet <HOSTNAME/IP> <PORT>
  • To check the connectivity on a UDP port, run the following command:

    nc -vz -u <HOSTNAME/IP> <PORT>

Manage Proxy Settings

Following are some commands to manage proxy settings.

  • To view the current proxy settings, run the following command:

    printenv|grep proxy
  • To add proxy settings to the current session, run the following command:

    export http_proxy=<PROXY>
    export https_proxy=<PROXY>
    export no_proxy=<IPs,HOSTNAMES>
  • To add proxy settings for a user, open the ~/.bashrc file and add the following lines:

    export http_proxy=<PROXY>
    export https_proxy=<PROXY>
    export no_proxy=<IPs,HOSTNAMES>

    Note

    You must sign in again to apply the proxy settings.

  • To add global proxy settings, open the /etc/environment file and add the following lines:

    http_proxy=<PROXY>
    https_proxy=<PROXY>
    no_proxy=<IPs,HOSTNAMES>
  • To add proxy settings for the Docker systemd service, open the /usr/lib/systemd/system/docker.service file and add the following lines:

    [Service]
    Environment="HTTP_PROXY=<PROXY>"
    Environment="HTTPS_PROXY=<PROXY>"

    You must restart Docker to apply the proxy settings. To restart the Docker services, run the following command:

    systemctl restart docker

General Troubleshooting Commands

Following are some commands to view the system uptime and system-level logs.

  • To view the uptime of the system, run the following command:

    uptime

    Sample Output

    9:49  up 3 days, 18:56, 2 users, load averages: 2.34 2.97 4.65
  • To view the system-level logs for reboot and more, run the following command:

    tail -f /var/log/messages

View Nginx Logs

If Nginx is installed, run the following command to view the Nginx reverse proxy logs -

tail -f /var/log/nginx/access.log
tail -f /var/log/nginx/error.log