dds/docs/guide/catalog.rst

The Package Catalog
###################

``dds`` stores a catalog of available packages, along with their dependency
statements and information about how a source distribution thereof may be
maintained.


Viewing Catalog Contents
************************

The default catalog database is stored in a user-local location, and the
package IDs available can be listed with ``dds catalog list``. This will only
list the IDs of the packages, but none of the additional metadata about them.


.. _catalog.adding:

Adding Packages to the Catalog
******************************

There are two primary ways to add entries to the package catalog.


Adding Individual Packages
==========================

A single package can be added to the catalog with the ``dds catalog add``
command:

.. code-block:: text

    dds catalog add <package-id>
        [--depends <requirement> [--depends <requirement> [...]]]
        [--git-url <url> --git-ref <ref>]
        [--auto-lib <namespace>/<name>]

The ``<package-id>`` positional arguments is the ``name@version`` package ID
that will be added to the catalog. The following options are supported:

``--depends <requirement>``
    This argument, which can be specified multiple times to represent multiple
    dependencies, sets the dependencies of the package within the catalog. If
    the obtained package root contains a ``package.json5``, then the
    dependencies listed here must be identical to those listed in
    ``package.json5``, or dependency resolution may yield unexpected results.

``--git-url <url>``
    Specify a Git URL to clone from to obtain the package. The root of the
    cloned repository must be a package root, but does not necessarily need to
    have the ``package.json5`` and ``library.json5`` files if relying on the
    ``--auto-lib`` parameter.

    ``--git-ref`` **must** be passed with ``--git-url``.

``--git-ref <ref>``
    Specify a Git ref to clone. The remote must support cloning by the ref that
    is specified here. Most usually this should be a Git tag.

    ``dds`` will perform a shallow clone of the package at the specified
    Git reference.

``--auto-lib``
    This option must be provided if the upstream does not already contain the
    ``dds`` files that are necessary to export the library information. This
    can only be specified for packages that contain a single library root at
    the package root.

    The form of the argument is that of ``<namespace>/<name>``, where
    ``namespace`` and ``name`` are the usage requirement keys that should be
    generated for the library.


.. _catalog.adding.json:

Bulk Imports via JSON
=====================

The ``dds catalog import`` supports a ``--json`` flag that specifies a JSON
file from which catalog entries will be generated.

.. note::
    The ``--json`` flag can be passed more than once to import multiple JSON
    files at once.

The JSON file has the following structure:

.. code-block:: javascript

  {
    // Import version spec.
    "version": 1,
    // Packages section
    "packages": {
      // Subkeys are package names
      "acme-gadgets": {
        // Keys within the package names are the versions that are
        // available for each package.
        "0.4.2": {
          // `depends` is an object of dependencies for this
          // particular version of the package. (Optional)
          "depends": {
            // A mapping of package names to version ranges
            "acme-widgets": "^1.4.1"
          },
          // `description` is an attribute to give a string to describe
          // the package. (Optional)
          "description": "A collection of useful gadgets.",
          // Specify the Git remote information
          "git": {
            // `url` and `ref` are required.
            "url": "http://example.com/git/repo/acme-gadgets.git",
            "ref": "v0.4.2-stable",
            // The `auto-lib` is optional, to specify an automatic
            // library name/namespace pair to generate for the
            // root library
            "auto-lib": "Acme/Gadgets",
            // List of filesystem transformations to apply to the repository
            // (optional)
            "transform": [
              // ... (see below) ...
            ]
          }
        }
      }
    }
  }


.. _catalog.fs-transform:

Filesystem Transformations
**************************

A catalog entry can have a set of filesystem transformations attached to its remote information (e.g. the ``git`` property). When ``dds`` is obtaining a copy of the code for the package, it will apply the associated transformations to the filesystem based in the directory of the downloaded/cloned directory. In this was, ``dds`` can effectively "patch" the filesystem structure of a project arbitrarily. This allows many software projects to be imported into ``dds`` without needing to patch/fork the original project to support the required filesystem structure.

.. important::
  While ``dds`` allows you to patch directories downloaded via the catalog, a
  native ``dds`` project must still follow the layout rules.

  The intention of filesystem transformations is to act as a "bridge" that will allow ``dds`` projects to more easily utilize existing libraries.

.. note::
  Filesystem transformations can only be added to catalog entries using the
  :ref:`JSON import method <catalog.adding.json>`. It is not available in the
  command-line import method.


Available Transformations
=========================

At time of writing, there are four main transformations available to catalog entries:

``copy`` and ``move``
  Copies or moves a set of files/directories from one location to another. Allows the following options:

  - ``from`` - The path from which to copy/move. **Required**
  - ``to`` - The destination path for the copy/move. **Required**
  - ``include`` - A list of globbing expressions for files to copy/move. If
    omitted, then all files will be included.
  - ``exclude`` - A list of globbing expressions of files to exclude from the
    copy/move. If omitted, then no files will be excluded. **If both** ``include`` and ``exclude`` are provided, ``include`` will be checked *before* ``exclude``.
  - ``strip-components`` - A positive integer (or zero, the default). When the
    ``from`` path identifies a directory, its contents will be copied/moved
    into the destination and maintain their relative path from the source path as their relative path within the destination. If ``strip-components`` is set to an integer ``N``, then the first ``N`` path components of that relative path will be removed when copying/moving the files in a directory. If a file's relative path has less than ``N`` components, then that file will be excluded from the ``copy/move`` operation.

``remove``
  Delete files and directories from the package source. Has the following options:

  - ``path`` - The path of the file/directory to remove. **Required**
  - ``only-matching`` - A list of globbing expressions for files to remove. If omitted and the path is a directory, then the entire directory will be deleted. If at least one pattern is provided, then directories will be left intact and only non-directory files will be removed. If ``path`` names a non-directory file, then this option has no effect.

``write``
  Write the contents of a string to a file in the package source. Has the following options:

  - ``path`` - The path of the file to write. **Required**
  - ``content`` - A string that will be written to the file. **Required**

  If the file exists and is not a directory, the file will be replaced. If the path names an existing directory, an error will be generated.

Transformations are added as a JSON array to the JSON object that specifies the remote information for the package. Each element of the array is an object, with one or more of the four keys listed above. If an object features more than one of the above keys, they are applied in the same order as they have been listed.


Example: Crypto++
=================

The following catalog entry will build and import `Crypto++`_ for use by a ``dds`` project. This uses the unmodified Crypto++ repository, which ``dds`` doesn't know how to build immediately. With some simple moving of files, we end up with something ``dds`` can build directly:

.. code-block:: javascript

  "cryptopp": {
    "8.2.0": {
      "git": {
        "url": "https://github.com/weidai11/cryptopp.git",
        "ref": "CRYPTOPP_8_2_0",
        "auto-lib": "cryptopp/cryptopp",
        "transform": [
          {
            // Crypto++ has no source directories at all, and everything lives
            // at the top level. No good for dds.
            //
            // Clients are expected to #include files with a `cryptopp/` prefix,
            // so we need to move the files around so that they match the
            // expected layout:
            "move": {
              // Move from the root of the repo:
              "from": ".",
              // Move files *into* `src/cryptopp`
              "to": "src/cryptopp",
              // Only move the C++ sources and headers:
              "include": [
                "*.c",
                "*.cpp",
                "*.h"
              ]
            }
          }
        ]
      }
    }
  }


Example: libsodium
==================

For example, this catalog entry will build and import `libsodium`_ for use in a ``dds`` project. This uses the upstream libsodium repository, which does not meet the layout requirements needed by ``dds``. With a few simple transformations, we can allow ``dds`` to build and consume libsodium successfully:

.. code-block:: javascript

  "libsodium": {
    "1.0.18": {
      "git": {
        "url": "https://github.com/jedisct1/libsodium.git",
        "ref": "1.0.18",
        "auto-lib": "sodium/sodium",
        /// Make libsodium look as dds expects of a project.
        "transform": [
          // libsodium has a `src` directory, but it does not look how dds
          // expects it to. The public `#include` root of libsodium lives in
          // a nested subdirectory of `src/`
          {
            "move": {
              // Move the public header root out from that nested subdirectory
              "from": "src/libsodium/include",
              // Put it at `include/` in the top-level
              "to": "include/"
            }
          },
          // libsodium has some files whose contents are generated by a
          // configure script. For demonstration purposes, we don't need most
          // of them, and we can just swipe an existing pre-configured file
          // that is already in the source repository and put it into the
          // public header root.
          {
            "copy": {
              // Generated version header committed to the repository:
              "from": "builds/msvc/version.h",
              // Put it where the configure script would put it:
              "to": "include/sodium/version.h"
            }
          },
          // The subdirectory `src/libsodium/` is no good. It now acts as an
          // unnecessary layer of indirection. We want `src/` to be the root.
          // We can just "lift" the subdirectory:
          {
            // Up we go:
            "move": {
              "from": "src/libsodium",
              "to": "src/"
            },
            // Delete the now-unused subdirectory:
            "remove": {
              "path": "src/libsodium"
            }
          },
          // Lastly, libsodium's source files expect to resolve their header
          // paths differently than they expect of their clients (Bad!!!).
          // Fortunately, we can do a hack to allow the files in `src/` to
          // resolve its headers. The source files use #include as if the
          // header root was `include/sodium/`, rather than `include/`.
          // To work around this, generate a copy of each header file in the
          // source root, but remove the leading path element.
          // Because we have a separate `include/` and `src/` directory, dds
          // will only expose the `include/` directory to clients, and the
          // header copies in `src/` are not externally visible.
          //
          // For example, the `include/sodium/version.h` file is visible to
          // clients as `sodium/version.h`, but libsodium itself tries to
          // include it as `version.h` within its source files. When we copy
          // from `include/`, we grab the relative path to `sodium/version.h`,
          // strip the leading components to get `version.h`, and then join that
          // path with the `to` path to generate the full destination at
          // `src/version.h`
          {
            "copy": {
              "from": "include/",
              "to": "src/",
              "strip-components": 1
            }
          }
        ]
      }
    }
  }

.. _libsodium: https://doc.libsodium.org/
.. _Crypto++: https://cryptopp.com/