Prestage a Subcloud¶
Before you start an AIO-SX subcloud upgrade or reinstall for the purpose of
restoring the subcloud; the subcloud can be prestaged with OSTree repo
(software updates) and container image archives outside the maintenance window
using the dcmanager CLI. The prestaged data is stored in the subcloud
persistent file system /opt/platform-backup/<sw_version>. This data will be
used when the subcloud is reinstalled next.
Where, the <sw_version> number is the active load of the System Controller.
Note
Only AIO-SX subclouds can be prestaged using the dcmanager CLI.
For information on prestaging a batch of subclouds, see, Prestage Subcloud Orchestration.
About this task
The main steps of this task are:
Ensure prestaging prerequisites are met, see Prestaging Requirements.
Upload the list of container images to prestage. This step is relevant to upgrade and must be performed after the System Controller has been upgraded. See Upload Prestage Image List.
Use dcmanager commands to prestage the subcloud(s).
To increase Subcloud Platform Backup Size using dcmanager CLI, see note.
Prestaging Requirements¶
Prerequisites
Prestaging can be done for a single subcloud or a batch of subclouds via orchestration. See Prestage Subcloud Orchestration.
There are two types of subcloud prestage:
Prestage for upgrade: when the subcloud is running a different (older) load than the System Controller at the time of prestaging.
Prestage for reinstall: When the subcloud is running the same load as the System Controller at the time of prestaging.
Note
Only AIO-SX subclouds can be prestaged using the dcmanager CLI.
Pre-conditions common to both types of prestage:
Subclouds to be prestaged must be AIO-SX, online, managed and free of any management affecting alarms.
Note
You can force prestaging using
--forceoption. However, it is not recommended unless it is certain that the prestaging process will not exacerbate the alarm condition on the subcloud.Subcloud
/opt/platform-backupmust have enough available disk space for prestage data.Subcloud
/var/lib/dockermust have enough space for all prestage image pulls and archive file generation. If the total size of prestage images is N GB, available Docker space should be N*2 GB.
Note
For the prestaged remote subcloud installations, the ISO imported via
load-import --inactive must be at the same patch level as the system
controller. If the system controller is patched after prestaging of
subclouds, you need to repeat the prestaging of each subcloud. This
ensures that the subcloud boot image aligns with the patch level of the
load that needs to be installed on the subcloud.
Warning
If the available docker space is inadequate, some application pods can get evicted due to temporary disk pressure during the prestaging process. The cert-manager application will fail subcloud upgrade if its evicted pods are not cleaned up.
Pre-conditions specific to prestage for upgrade:
The total size of prestage images and custom images restored over upgrade must not exceed docker-distribution capacity.
Prestage images must already exist in the configured source(s) prior to subcloud prestaging. For example, if the subcloud is configured to download images from the central registry; the specified images must already exist in the registry on the System Controller.
Upload Prestage Image List¶
The prestage image list specifies what container images are to be pulled from the configured sources and included in the image archive files during prestaging. This list is only used if the prestage is intended for subcloud upgrade i.e. the System Controller and subclouds are running different loads at the time of prestaging.
The prestage image list must contain:
Images required for subcloud platform upgrade.
Images required for the restore and update or StarlingX applications, currently applied on the subcloud, for example, cert-manager, OIDC, and metrics-server.
If the available docker and docker-distribution storage is ample, prestage image list should also contain:
(Optional) Images required for Kubernetes version upgrades post subcloud upgrade.
(Optional) Images required for the update of end users’ Helm applications post subcloud upgrade.
Note
It is required to determine the total size of all images to be prestaged in advance. Too many images can result in subcloud upgrade failure due to docker-distribution (local registry) out of space error. See the Prerequisites section above for more details.
Procedure
To upload the prestage image list, use the following command after the System Controller has been upgraded.
~(keystone_admin)]$ dcmanager subcloud-deploy upload --prestage-images nn.nn_images.lst +------------------+-----------------+ | Field | Value | +------------------+-----------------+ |deploy_playbook | None | |deploy_overrides | None | |deploy_chart | None | |prestage_images | nn.nn_images.lst| +------------------+-----------------+
Where, the name of the prestage image file can be user defined. However, it is recommended to use the following format <software_version>_images.lst, for example, <21.12_images.lst>.
To confirm that the image list has been uploaded, use the following command.
~(keystone_admin)]$ dcmanager subcloud-deploy show +------------------+-------------------------+ | Field | Value | +------------------+-------------------------+ | deploy_playbook | None | | deploy_overrides | None | | deploy_chart | None | | prestage_images | nn.nn_images.lst | +------------------+-------------------------+
Warning
As prestage images will be pulled from Docker registries currently configured for the subcloud, images in the image list file must not contain custom/private registry prefix.
Single Subcloud Prestage¶
See Prestaging Requirements for preconditions prior to prestaging the subcloud.
~(keystone_admin)]$ dcmanager subcloud prestage subcloud2
Enter the sysadmin password for the subcloud:
Re-enter sysadmin password to confirm:
+-----------------------------+----------------------------+
| Field | Value |
+-----------------------------+----------------------------+
| id | 2 |
| name | subcloud2 |
| description | None |
| location | None |
| software_version | nn.nn |
| management | managed |
| availability | online |
| deploy_status | prestage-prepare |
| management_subnet | 2620:10a:a001:ac01::20/123 |
| management_start_ip | 2620:10a:a001:ac01::22 |
| management_end_ip | 2620:10a:a001:ac01::3e |
| management_gateway_ip | 2620:10a:a001:ac01::21 |
| systemcontroller_gateway_ip | 2620:10a:a001:a113::1 |
| group_id | 3 |
| created_at | 2202-03-18 20:31:16.548903 |
| updated_at | 2202-03-22 18:55:56:251643 |
+-----------------------------+----------------------------+
Rerun Subcloud Prestage¶
A subcloud can be prestaged multiple times. However, only prestaging images will be repeated. Once packages prestaging is successful, this step will be skipped in subsequent prestage reruns for the same software version.
Verify Subcloud Prestage¶
After a subcloud is successfully prestaged, the deploy_status will change to
prestage-complete. Use the dcmanager subcloud show command to
verify the status. The packages directory, repodata directory, and container
image bundles, and md5 file can be found on the subcloud in
/opt/platform-backup/<sw_version>.
Where, the <sw_version> number is the active load of the System Controller.
Troubleshoot Subcloud Prestage¶
If the subcloud prestage fails, check /var/log/dcmanager/dcmanager.log
for the reason of failure. Once the issue has been resolved, prestage can be
retried using dcmanager subcloud prestage command.
Verifying Usage of Prestaged Data¶
To verify that the prestaged data is used over subcloud upgrade, subcloud reinstall, or subcloud remote restore:
Search for the the subcloud name in the log file, for example, subcloud1 from
/www/var/log/lighttpd-access.log. There should not be GET requests to download packages from/iso/<sw_version>/nodes/subcloud1/Packages/.Check subcloud ansible log in
/var/log/dcmanager/ansibledirectory. Images are imported from local archives and no images in the prestage image list need to be downloaded from configured sources.
