Schema Versioning

As RAD contains ASDF compatible schemas the versioning strategy will mostly follow that of other ASDF extensions where:

  • schema modifications trigger creation of a new schema version

  • the old schema is kept unmodified

  • a new tag version is created for the new schema version

  • the new tag version triggers a new manifest version

This allows files created with the old tag to be validated (against the old schema) and opened.

Non-versioned changes

The RAD schemas contain archive database destination and other non-ASDF information including schema keywords:

  • archive_catalog

  • sdf

  • title

  • description

  • propertyOrder

Changes to content stored under these keys will not be trigger a new schema version. Please consult the newest schema version for the most accurate values for these keywords.

Manifest Versioning

New manifests will often be created by:

  • copying the newest manifest version

  • incrementing the manifest version number, id and extension uri version

  • updating the tag definition for that tag/schema change that triggered the manifest version

One exception is if the newest manifest version has not yet been released. In this case the tag definition in the existing (unreleased) manifest can be modified and no manifest version increase is needed.

Creating a New Version

The RAD resources (both schemas and manifests) are located in the src/rad/resources directory for the purposes of the ASDF extension. Meaning that all resources must be located in this directory or have a symlink from this directory to the actual location for the resources. This directory is organized into two subdirectories:

  1. schemas - contains the ASDF schemas or symlinks to the schemas

  2. manifests - contains the ASDF manifests or symlinks to the manifests

While src/rad/resources is where the resources are located for the ASDF extension, not all of the resources are directly located in this directory. Instead, the latest version of any schema is instead located within the latest directory in the top level of the RAD repository, while the latest version of any manifest is located within the latest/manifests directory. These files are linked to from the src/rad/resources directory. Note that the path relative to the latest directory for schemas (latest/manifests for manifests) is the same as the path to the respective symlink relative to the src/rad/resources/schemas for schemas (src/rad/resources/manifests for manifests).

The naming conventions for resources follows two different patterns:

  1. The files within the src/rad/resources directory are named with a version suffix (-a.b.c.yaml) with the latest (highest semantic) version number actually being a symlink targeting a file within latest.

  2. The files within the latest directory end only with the normal .yaml suffix with no version number included. These files are then the target of the version indicated by the symlink file name corresponding to the version number indicated by the URI (id: keyword) within the file itself.

Note

These file naming conventions and the underlying directory structure exist to facilitate a straightforward review process in git for reviewing changes to resources whenever a version number is updated.

This allows for the git diff to directly show the changes to the resource within latest rather than having the new version of the file appear as a brand new file in the git diff. This is facilitated by the fact that the name of the latest file never changes, only its contents, while the files in the src/rad/resources end up with being brand new files or symlink name updates.

Workflow for Updating a Resource Version

Note

We strongly recommend using the RAD Helper Tool’s Bump function to update the version of a given resource. In addition, the tool will also tell you if you need to bump the version of a resource when you attempt to edit it.

The tool will take care of updating all the references to the new version number within the latest.

To update the version of a given resource the following steps should be taken:

  1. Determine the new version number to be used for the resource following the [Semantic Versioning](https://semver.org/) guidelines.

  2. Update the version number in the symlink name in the src/rad/resources directory from the current version number to the new selected (in step 1) version number.

    Note

    The symlink’s target should never be modified as the new target of the symlink will still be the resource’s file with in the latest directory, which will later be modified starting in step 4 to reflect the new version number.

  3. Copy the given file in the latest directory (as is) to the appropriate location in the src/rad/resources modifying the name of the file to include the current version number (e.g. -a.b.c.yaml).

    Note

    This should be done before making any modifications to the contents of any resource file. This is important as the goal is to preserve the exact contents of the resource as it was under the current version number.

  4. In latest update the version number for the URI (id: keyword) to match the new version number selected in step 1 within the file itself.

    Note

    In addition to the id: keyword for manifests (datamodels.yaml), the extension_uri: keyword should also be updated to match the new version number.

  5. Throughout the latest directory update the version number for all related URIs (tag and $ref) to match the version number selected in step 1.

    Note

    This may cause a cascading effect where the version number for other files in the latest directory need to be updated as well. This may necessitate following the version update process for these files as well.

  6. Make the updates to the file in latest with the changes that necessitated the new version.

Note

The RAD unit tests have been designed so that the tests/test_versioning.py tests will fail if a file is modified without updating the version number whenever a version number update would be required.

Therefore, it is suggested that one runs the unit tests after making modifications to a given resource prior to committing those changes. If a test failure occurs then simply stash the changes and then follow the version update process above. The resource changes can then be unstashed and committed as the latest version has now been updated properly.

Note

The unit tests in tests/test_latest.py are designed to ensure that if a version number is updated all the necessitated changes to the resources are made. This includes:

  • Checking that all relevant URIs in latest have been updated to reflect the new version number.

  • Checking the integrity of the symlinks in src/rad/resources to ensure that they are pointing to the correct file in latest.

  • Checking that the old version of the file has not gone missing (in combination with the test_versioning.py tests).

Note

It is important to run the unitests regularly while making changes to the resources to ensure that the versioning process has been followed correctly, as they will guide you through the process of when and how to update resource version numbers.

Old version support

RAD is not yet stable. Efforts will be made to retain support for opening old files. As noted above supporting old versions of files will require keeping several manifest versions and all old schemas. As development continues it may be advantageous to drop support for some old (pre-flight) versions.

Dropping support for pre-flight versions

If it is decided that support for an old (pre-flight) version of a schema will be dropped the following steps will be taken:

  • removal of the unsupported schema versions

  • removal of the unsupported tag versions

  • removal of all manifest versions that contain the dropped schema or tag versions

By following these steps, the unsupported old files can still be opened with asdf.open. When an unsupported file is opened asdf will encounter one or more of the unsupported (and now unknown) tags and issue AsdfConversionWarning describing that the tagged objects are being returned as “raw data structures” (typically a dictionary-like TaggedDict). This will allow users to continue to access the contents of the file and possibly migrate the old file contents to a new supported tag/structure.