Validator Troubleshooting

Need help solving an issue? Check to see if this has already been answered below. If you can’t find an answer to your issue, reach out to the SKALE Network developer community on Discord.

How do I change my Node’s IP address?

See Changing Node IP in the Node CLI section.

How should validators configure backup and restore?

Validators are responsible for backing up everything on the node. SKALE Network relies on validators to complete the backup procedure for each of the nodes. SKALE Network replicates the data for each SKALE Chain across 16 nodes, adding an additional layer of data availability. If a node goes down for a long time, the other 2/3 of 15 will be responsible for taking the SKALE Chain’s snapshot and shuffling the SKALE Chain in the SKALE Network. If a validator node is down for a long time, it may be reallocated to service new SKALE Chains depending on the length of the downtime.

Please check node backup and sgx_backup to learn more about how to back up the node or SGXWallet, and restore.

Where are the network communications between containers, shared configuration files, and volumes stored?

You can find these details in the skale-node repository and the docker-compose.yml.

How do I access the SKALE Chain’s IP and ports?

You can use the node-cli to view firewall info and SKALE Chain info:

# Show firewall rules for IP and Ports
skale schains show-rules

# Show the SKALE Chain Config
skale schains config [SCHAIN_NAME]

In this file, you can see all the SKALE Chain information:

What are the following Node ports used for?

Port	Description	Notes
3009/TCP	Watchdog service provides sla-agent and SKALE Monitoring/Metrics site with data about all containers’ statuses on all SKALE nodes	Used by nginx as reverse proxy for Flask-based Watchdog
8080/TCP	Used by cAdvisor (Prometheus exporter) - to analyze and expose resource usage and performance data from running docker containers	Running cAdvisor container is optional (MONITORING_CONTAINERS=True)
9100/TCP	Used by Prometheus Node Exporter - to measure host resources such as memory, disk, and CPU utilization, etc.	Running Node exporter container is optional (MONITORING_CONTAINERS=True)
22	Used by the node operator to ssh into the machine
10000–11500/TCP	These ports should be open for communications inside sChains between different nodes.	When there are no sChains, these ports are closed by iptables rules. It’s important not to add another firewall, because sChain configuration is created dynamically as well as iptables rules.

Can I use Kubernetes?

Yes. However, you will need to make sure that your platform is compatible with SKALE. In particular, your platform must allow docker-lvmpy to dynamically partition the external drive for each SKALE chain.

Where can I find the SKALE Node logs?

Validators can run these commands to check their logs in their node:

Here are some logs and commands you can use for troubleshooting or to provide logs to the core team. The most used and first place to look at:

docker logs skale_api

or

docker logs skale_admin

Transaction manager logs:

docker logs skale_transaction_manager

All running containers logs:

skale logs dump [PATH]

SGX certification in validator node:

ls -l ~/.skale/node_data/sgx_certs/

SGX wallet logs:

docker logs runsgx_sgxwallet_1

Node-cli debugging logs:

~/.skale/.skale-cli-log/debug-node-cli.log

Docker-lvmpy logs:

cat /var/log/docker-lvmpy/lvmpy.log

Why is SGX crashing after machine reboot?

SGXWallet may crash after a reboot if you didn’t disable automatic updates. This is because SGXWallet is based on new low-level technology, and kernel updates may break the system. It’s recommended to only update the SGXWallet server if there are critical security fixes.

Node update procedure

Updating system packages is an important part of node maintenance and security. Updates may contain fixes for recent security issues. However, newer updates may be incompatible with SKALE node software. Therefore, updating should be executed carefully.

Caution

Make sure that the infrastructure provider doesn’t update packages after the system is restarted. The following list contains packages that require additional attention. Critical risk (updates should be performed only if there are some messages from the core team):

• kernel updates on SGXWallet server.

Note

Currently, SGX is tested on 4.15-* kernels. It’s best to avoid minor version updates too. High Risk (updates should be avoided in general):

• kernel on SKALE node server
• docker
• docker-compose
• btrfs-progs

Medium Risk (updates should be performed very carefully):

• lvm2
• iptables
• iptables-persistent
• python

General tips

1. Carefully check which packages will be updated when executing apt upgrade. You can use apt list --upgradable

2. Avoid executing apt dist-upgrade

3. Disabling updates for certain packages can be done using apt-mark hold (apt-mark man page). For example, to forbid kernel updates you should run:

   sudo apt-mark hold linux-generic linux-image-generic linux-headers-generic

Another option is to use unattended upgrades (Unattended Upgrades wiki), which allows you to automatically keep the system current with the latest security updates. There is an option to exclude some packages from the update list using the Package-Blacklist section in the configuration file. For example, to disallow Linux kernel updates, you should place the following lines in the /etc/apt/apt.conf.d/50unattended-upgrades file:

Unattended-Upgrade::Package-Blacklist {
        "linux-generic";
        "linux-image-generic";
        "linux-headers-generic";
};

SKALE Network has many resources designed to help you get your questions answered. You can reach out to the SKALE Network developer community on discord, or submit a support request below.

Contact Support