Validator Troubleshooting

Need help solving an issue? Check to see if this has already been answered below. If you can’t find an answer to your issue, reach out to the SKALE Network developer community on Discord.

How do I change my Node’s IP address?

See Changing Node IP

How should validators configure backup and restore?

Validators are responsible for backing up everything on the supernode. SKALE Network relies on validators to complete the backup procedure for each of the supernodes. SKALE Network replicates the data for each SKALE Chain across 16 nodes, adding an additional layer of data availability. If a node goes down for a long time, the other 2/3 of 15 will be responsible for taking the SKALE Chain’s snapshot and shuffling the SKALE Chain in the SKALE Network. If a validator supernode is down for a long time, it may be reallocated to service new SKALE Chains depending on the length of the downtime.Please check supernode backup and sgx_backup to learn more about how to back up the supernode or SGXWallet, and restore.

Where are the network communications between containers, shared configuration files, and volumes stored?

You can find these details in the skale-node repository and the docker-compose.yml.

How do I access the SKALE Chain's IP and ports?

You can use the node-cli to view firewall info and SKALE Chain info:

# Show firewall rules for IP and Ports
skale SKALE Chains show-rules

# Show the SKALE Chain Config
skale SKALE Chains config [SCHAIN_NAME]

In this file, you can see all the SKALE Chain information:

What are the following Supernode ports used for?

Port	Description	Notes
3009/TCP	Watchdog service provides sla-agent and SKALE Monitoring/Metrics site with data about all containers’ statuses on all SKALE supernodes	Used by nginx as reverse proxy for Flask-based Watchdog
8080/TCP	Used by cAdvisor (Prometheus exporter) - to analyze and expose resource usage and performance data from running docker containers	Running cAdvisor container is optional (MONITORING_CONTAINERS=True)
9100/TCP	Used by Prometheus Node Exporter - to measure host resources such as memory, disk, and CPU utilization, etc.	Running Node exporter container is optional (MONITORING_CONTAINERS=True)
22	Used by the supernode operator to ssh into the machine
10000–11500/TCP	These ports should be open for communications inside SKALE Chains between different supernodes.	When there are no SKALE Chains, these ports are closed by iptables rules. It’s important not to add another firewall, because SKALE Chain configuration is created dynamically as well as iptables rules.

Can I use Kubernetes?

Yes. However, you will need to make sure that your platform is compatible with SKALE. In particular, your platform must allow docker-lvmpy to dynamically partition the external drive for each SKALE chain.

Where can I find the SKALE Supernode logs?

Validators can run these commands to check their logs in their supernode:Here are some logs and commands you can use for troubleshooting or to provide logs to the core team. The most used and first place to look at:

docker logs skale_api

docker logs skale_admin

Transaction manager logs:

docker logs skale_transaction_manager

All running containers logs:

skale logs dump [PATH]

SGX certification in validator supernode:

ls -l ~/.skale/node_data/sgx_certs/

SGX wallet logs:

docker logs runsgx_sgxwallet_1

Node-cli debugging logs:

~/.skale/.skale-cli-log/debug-node-cli.log

Docker-lvmpy logs:

cat /var/log/docker-lvmpy/lvmpy.log

Why is SGX crashing after machine reboot?

SGXWallet may crash after a reboot if you didn’t disable automatic updates. This is because SGXWallet is based on new low-level technology, and kernel updates may break the system. It’s recommended to only update the SGXWallet server if there are critical security fixes.

Supernode update procedure

Updating system packages is an important part of supernode maintenance and security. Updates may contain fixes for recent security issues. However, newer updates may be incompatible with SKALE supernode software. Therefore, updating should be executed carefully.

Make sure that the infrastructure provider doesn’t update packages after the system is restarted. The following list contains packages that require additional attention. Critical risk (updates should be performed only if there are some messages from the core team):

kernel updates on SGXWallet server.

Currently, SGX is tested on 4.15-* kernels. It’s best to avoid minor version updates too. High Risk (updates should be avoided in general):

kernel on SKALE supernode server
docker
docker-compose
btrfs-progs

Medium Risk (updates should be performed very carefully):

lvm2
iptables
iptables-persistent
python

General tips

Carefully check which packages will be updated when executing apt upgrade. You can use apt list --upgradable
Avoid executing apt dist-upgrade
Disabling updates for certain packages can be done using apt-mark hold (apt-mark man page). For example, to forbid kernel updates you should run:
sudo apt-mark hold linux-generic linux-image-generic linux-headers-generic

Another option is to use unattended upgrades (Unattended Upgrades wiki), which allows you to automatically keep the system current with the latest security updates. There is an option to exclude some packages from the update list using the Package-Blacklist section in the configuration file. For example, to disallow Linux kernel updates, you should place the following lines in the /etc/apt/apt.conf.d/50unattended-upgrades file:

Unattended-Upgrade::Package-Blacklist {
        "linux-generic";
        "linux-image-generic";
        "linux-headers-generic";
};

SKALE Network has many resources designed to help you get your questions answered. You can reach out to the SKALE Network developer community on Discord, or submit a support request below. Contact Support

Validators

Network Releases

Passive Nodes

Node CLI

Validator CLI

Watchdog