| ``dds`` has been designed from the very beginning as an extremely opinionated | ``dds`` has been designed from the very beginning as an extremely opinionated | ||||
| hybrid *build system* and *package manager*. Unlike most build systems however, | hybrid *build system* and *package manager*. Unlike most build systems however, | ||||
| ``dds`` has a hyper-specific focus on a particular aspect of software | |||||
| development: C and C++ libraries. | |||||
| ``dds`` has a strong focus on a particular aspect of software development: C and | |||||
| C++ libraries. | |||||
| This may sound pointless, right? Libraries are useless unless we can use them | This may sound pointless, right? Libraries are useless unless we can use them | ||||
| to build applications! | to build applications! | ||||
| Indeed, applications *are* essential, but that is "not our job" with ``dds``. | |||||
| Indeed, applications *are* essential, and ``dds`` is able to build those as | |||||
| well. | |||||
| Another design decision is that ``dds`` is built to be driven by automated | Another design decision is that ``dds`` is built to be driven by automated | ||||
| tools as well as humans. ``dds`` is not designed to entirely replace existing | tools as well as humans. ``dds`` is not designed to entirely replace existing | ||||
| Despite the vast amount of work put into build systems and tooling, virtually | Despite the vast amount of work put into build systems and tooling, virtually | ||||
| all developers are using them *incorrectly* and/or *dangerously* without | all developers are using them *incorrectly* and/or *dangerously* without | ||||
| realizing it. Despite this work, we seem to be a great distance from a unified | |||||
| library package distribution and consumption mechanism. | |||||
| realizing it, and we seem to be still a great distance from a unified library | |||||
| package distribution and consumption mechanism. | |||||
| Tabula Rasa | Tabula Rasa | ||||
| *customizability* in favor of *simplicity* and *ease*. | *customizability* in favor of *simplicity* and *ease*. | ||||
| ``dds`` takes a look at what is needed to build and develop *libraries* and | ``dds`` takes a look at what is needed to build and develop *libraries* and | ||||
| hyper-optimizes for that use case. It is also built with a very strong, very | |||||
| optimizes for that use case. It is also built with a very strong, very | |||||
| opinionated idea of *how* libraries should be constructed and used. These | opinionated idea of *how* libraries should be constructed and used. These | ||||
| prescriptions are not at all arbitrary, though. They are built upon the | prescriptions are not at all arbitrary, though. They are built upon the | ||||
| observations of the strengths and weaknesses of build systems in use throughout | observations of the strengths and weaknesses of build systems in use throughout | ||||
| ``dds`` takes a massive divergence at this point. One project using ``dds`` as | ``dds`` takes a massive divergence at this point. One project using ``dds`` as | ||||
| their build system has a nearly identical build process to every other project | their build system has a nearly identical build process to every other project | ||||
| using ``dds``. Simply running :code:`dds build -t <toolchain>` should be enough | |||||
| using ``dds``. Simply running ``dds build`` should be enough | |||||
| to build *any* ``dds`` project. | to build *any* ``dds`` project. | ||||
| In order to reach this uniformity and simplicity, ``dds`` drops almost all | In order to reach this uniformity and simplicity, ``dds`` drops almost all | ||||
| aspects of project-by-project customizability. Instead, ``dds`` affords the | aspects of project-by-project customizability. Instead, ``dds`` affords the | ||||
| developer a contract: | developer a contract: | ||||
| If you play by my rules, you get to play in my space. | |||||
| If you play by the rules, you get to play in this space. | |||||
| .. _design.rules: | .. _design.rules: | ||||
| .. _design.rules.not-apps: | .. _design.rules.not-apps: | ||||
| ``dds`` Is not Made for Complex Applications | ``dds`` Is not Made for Complex Applications | ||||
| =============================================== | |||||
| ============================================ | |||||
| Alright, this one isn't a "rule" as much as a recommendation: If you are | Alright, this one isn't a "rule" as much as a recommendation: If you are | ||||
| building an application that *needs* some build process functionality that | building an application that *needs* some build process functionality that | ||||
| customization features to permit the rules to be bent arbitrarily: Read | customization features to permit the rules to be bent arbitrarily: Read | ||||
| on. | on. | ||||
| ``dds`` contains a minimal amount of functionality for building simple | |||||
| applications, but it is certainly not its primary purpose. | |||||
| ``dds`` *does* contain functionality for building applications, but they must | |||||
| also play by the rules. | |||||
| If you want to build a complex application with ``dds`` that uses lots of | |||||
| platform-specific sources, code generation, and conditional components, a good | |||||
| option is to use an external build script that prepares the project tree before | |||||
| invoking ``dds``. | |||||
| .. _design.rules.change: | .. _design.rules.change: | ||||
| *Your* Code Should Be Changed Before ``dds`` Should Be Changed | |||||
| ================================================================= | |||||
| *Your Code* Should Be Changed Before ``dds`` Should Be Changed | |||||
| ============================================================== | |||||
| The wording of this rule means that the onus is on the library developer to | |||||
| meet the expectations that ``dds`` prescribes in order to make the build | |||||
| work. | |||||
| The wording of this rule means that the onus is on the developer to meet the | |||||
| expectations that ``dds`` prescribes in order to make the build work. | |||||
| If your library meets all the requirements outlined in this document but you | |||||
| still find trouble in making your build work, this is grounds for change in | |||||
| ``dds``, either in clarifying the rules or tweaking ``dds`` functionality. | |||||
| If your project meets all the requirements outlined in this document but you | |||||
| still find trouble in making your build work, or if you *cannot* see *any* | |||||
| possible way for your project to be built by ``dds`` regardless of what changes | |||||
| you make, then it this is grounds for change in ``dds``, either in clarifying | |||||
| the rules or tweaking ``dds`` functionality | |||||
| .. _design.rules.layout: | .. _design.rules.layout: | ||||
| All Code Must Be in Place Before Building | All Code Must Be in Place Before Building | ||||
| ========================================= | ========================================= | ||||
| ``dds`` does not provide code-generation functionality. Instead, any | |||||
| generated code should be generated and committed to the repository to be only | |||||
| ever modified through such generation scripts. | |||||
| ``dds`` does not provide code-generation functionality. Instead, any generated | |||||
| code should be generated by separate build steps before ``dds`` is executed. | |||||
| .. _design.rules.one-binary-per-src: | .. _design.rules.one-binary-per-src: | ||||
| ===================================== | ===================================== | ||||
| Only ``src/`` and ``include/`` will ever be used as the basis for header | Only ``src/`` and ``include/`` will ever be used as the basis for header | ||||
| resolution while building a library, so all ``#include`` directives should be | |||||
| resolution while building a project, so all ``#include`` directives should be | |||||
| relative to those directories. Refer to :ref:`pkg.source-root`. | relative to those directories. Refer to :ref:`pkg.source-root`. | ||||
| All Files Compile with the Same Options | All Files Compile with the Same Options | ||||
| ======================================= | ======================================= | ||||
| When DDS compiles a library, every source file will be compiled with an | |||||
| When DDS compiles a project, every source file will be compiled with an | |||||
| identical set of options. Additionally, when DDS compiles a dependency tree, | identical set of options. Additionally, when DDS compiles a dependency tree, | ||||
| every library in that dependency tree will be compiled with an identical set of | every library in that dependency tree will be compiled with an identical set of | ||||
| options. Refer to the :doc:`guide/toolchains` page for more information. | options. Refer to the :doc:`guide/toolchains` page for more information. |
| unavailable in most setups). Using a Git tag is strongly recommended. | unavailable in most setups). Using a Git tag is strongly recommended. | ||||
| .. seealso:: | .. seealso:: | ||||
| Refer to the documentation on :doc:`/guide/catalog`. | |||||
| Refer to the documentation on :doc:`/guide/remote-pkgs`. |
| Error: Invalid catalog JSON | |||||
| ########################### | |||||
| This error occurs when the JSON data given to import into the package catalog | |||||
| is in some way invalid. Refer to the catalog documentation for a description of | |||||
| the proper JSON format. | |||||
| .. seealso:: | |||||
| :ref:`catalog.adding` |
| Error: A repository filesystem transformation is invalid | |||||
| ######################################################## | |||||
| In ``dds``, a catalog entry can have a list of attached "transforms" that will | |||||
| be applies to the root directory of the package before ``dds`` tries to build | |||||
| and use it. | |||||
| .. seealso:: | |||||
| For information on the shape and purpose of transforms, refer to | |||||
| :ref:`catalog.fs-transform` on the :doc:`/guide/catalog` page. |
| when it is requested. | when it is requested. | ||||
| If such information is not provided, ``dds`` will issue an error. | If such information is not provided, ``dds`` will issue an error. | ||||
| .. seealso:: :ref:`catalog.adding`. |
| It is possible that the intended package *does exist* but that the spelling of | It is possible that the intended package *does exist* but that the spelling of | ||||
| the package name or version number is incorrect. Firstly, check your spelling | the package name or version number is incorrect. Firstly, check your spelling | ||||
| and that the version number you have requested is correct. | and that the version number you have requested is correct. | ||||
| In another case, it is possible that the package *exists somewhere*, but has | |||||
| not been loaded into the local catalog. As of this writing, ``dds`` does not | |||||
| automatically maintain the catalog against a central package repository, so | |||||
| package entries must be loaded and imported manually. If you believe this to be | |||||
| the case, refer to the section on the :doc:`/guide/catalog`, especially the | |||||
| section :ref:`catalog.adding`. |
| whatever exists there before creating the source distribution. | whatever exists there before creating the source distribution. | ||||
| .. warning:: | .. warning:: | ||||
| When using ``dds sdist create`` with the ``--out <path>`` parameter, the | |||||
| When using ``dds pkg create`` with the ``--out <path>`` parameter, the | |||||
| ``<path>`` given **is not the directory in which to place the source | ``<path>`` given **is not the directory in which to place the source | ||||
| distribution, but the filepath to the source distribution itself**! | distribution, but the filepath to the source distribution itself**! | ||||
| distribution in that directory, **the following command is incorrect**:: | distribution in that directory, **the following command is incorrect**:: | ||||
| # Do not do this: | # Do not do this: | ||||
| dds sdist create --out foo/ | |||||
| dds pkg create --out foo/ | |||||
| If you pass ``--replace`` to the above command, ``dds`` will **destroy the | If you pass ``--replace`` to the above command, ``dds`` will **destroy the | ||||
| existing directory** and replace it with the source distribution! | existing directory** and replace it with the source distribution! | ||||
| You **must** provide the full path to the source distribution:: | You **must** provide the full path to the source distribution:: | ||||
| # Do this: | # Do this: | ||||
| dds sdist create --out foo/my-project.dsd | |||||
| dds pkg create --out foo/my-project.tar.gz |
| Building and Using ``dds`` in Another Build System | |||||
| ################################################## | |||||
| One of ``dds``'s primary goals is to inter-operate with other build systems | |||||
| cleanly. One of ``dds``'s primary outputs is *libman* package indices. These | |||||
| package indices can be imported into other build systems that support the | |||||
| `libman`_ format. (At the time of writing there is a CMake module which can do | |||||
| the import, but other build systems are planned.) | |||||
| .. _libman: https://api.csswg.org/bikeshed/?force=1&url=https://raw.githubusercontent.com/vector-of-bool/libman/develop/data/spec.bs | |||||
| .. _PMM: https://github.com/vector-of-bool/PMM | |||||
| .. _CMakeCM: https://github.com/vector-of-bool/CMakeCM | |||||
| .. _lm-cmake: https://raw.githubusercontent.com/vector-of-bool/libman/develop/cmake/libman.cmake | |||||
| .. _build-deps.gen-libman: | |||||
| Generating a libman Index | |||||
| ************************* | |||||
| Importing libman packages into a build system requires that we have a libman | |||||
| index generated on the filesystem. **This index is not generated globally**: It | |||||
| is generated on a per-build basis as part of the build setup. The index will | |||||
| describe in build-system-agnostic terms how to include a set of packages and | |||||
| libraries as part of a build. | |||||
| ``dds`` has first-class support for generating this index. The ``build-deps`` | |||||
| subcommand of ``dds`` will download and build a set of dependencies, and places | |||||
| an ``INDEX.lmi`` file that can be used to import the built results. | |||||
| Declaring Dependencies | |||||
| ====================== | |||||
| ``dds build-deps`` accepts a list of dependency statements as command line | |||||
| arguments, but it may be useful to specify those requirements in a file. | |||||
| ``dds build-deps`` accepts a JSON5 file describing the dependencies of a | |||||
| project as well. This file is similar to a very stripped-down version of a | |||||
| ``dds`` :ref:`package manifest <pkgs.pkgs>`, and only includes the ``depends`` | |||||
| key. (The presence of any other key is an error.) | |||||
| Here is a simple dependencies file that declares a single requirement: | |||||
| .. code-block:: js | |||||
| :caption: ``dependencies.json5`` | |||||
| { | |||||
| depends: [ | |||||
| 'neo-sqlite3^0.2.0', | |||||
| ] | |||||
| } | |||||
| Building Dependencies and the Index | |||||
| =================================== | |||||
| We can invoke ``dds build-deps`` and give it the path to this file: | |||||
| .. code-block:: bash | |||||
| $ dds build-deps --deps-file dependencies.json5 | |||||
| When finished, ``dds`` will write the build results into a subdirectory called | |||||
| ``_deps`` and generate a file named ``INDEX.lmi``. This file is ready to be | |||||
| imported into any build system that can understand libman files. | |||||
| .. note:: | |||||
| The output directory and index filepath can be controlled with the | |||||
| ``--out`` and ``--lmi-path`` flags, respectively. | |||||
| Importing an Index: CMake | |||||
| ************************* | |||||
| .. highlight:: cmake | |||||
| .. note:: | |||||
| This section discusses how to import ``INDEX.lmi`` into CMake, but ``dds`` | |||||
| also has built-in support for generating a CMake targets file. See | |||||
| :doc:`cmake` and :ref:`cmake.pmm` for even simpler integration steps. | |||||
| Supposed that we've generated a libman index and set of packages, and we want to | |||||
| import them into CMake. CMake doesn't know how to do this natively, but there | |||||
| exists a single-file module for CMake that allows CMake to import libraries from | |||||
| libman indices without any additional work. | |||||
| The module is not shipped with CMake, but is available online as a single | |||||
| stand-alone file. The `libman.cmake <lm-cmake_>`_ file can be downloaded and | |||||
| added to a project directly, or it can be obtained automatically through a | |||||
| CMake tool like `PMM`_ (recommended). | |||||
| Getting ``libman.cmake`` via PMM | |||||
| ================================ | |||||
| Refer to the ``README.md`` file in `the PMM repo <PMM_>`_ for information on how | |||||
| to get PMM into your CMake project. In short, download and place the | |||||
| ``pmm.cmake`` file in your repository, and ``include()`` the file near the top | |||||
| of your ``CMakeLists.txt``:: | |||||
| include(pmm.cmake) | |||||
| Once it has been included, you can call the ``pmm()`` function. To obtain | |||||
| *libman*, we need to start by enabling `CMakeCM`_:: | |||||
| pmm(CMakeCM ROLLING) | |||||
| .. warning:: | |||||
| It is not recommended to use the ``ROLLING`` mode, but it is the easiest to | |||||
| use when getting started. For reproducible and reliable builds, you should | |||||
| pin your CMakeCM version using the ``FROM <url>`` argument. | |||||
| Enabling CMakeCM will make available all of the CMake modules available in `the | |||||
| CMakeCM repository <CMakeCM_>`_, which includes `libman.cmake <lm-cmake_>`_. | |||||
| After the call to ``pmm()``, simply ``include()`` the ``libman`` module:: | |||||
| include(libman) | |||||
| That's it! The only function from the module that we will care about for now | |||||
| is the ``import_packages()`` function. | |||||
| Importing Our Dependencies' Packages | |||||
| ==================================== | |||||
| To import a package from a libman tree, we need only know the *name* of the | |||||
| package we wish to import. In our example case above, we depend on | |||||
| ``neo-sqlite3``, so we simply call the libman-CMake function | |||||
| ``import_packages()`` with that package name:: | |||||
| import_packages("neo-sqlite3") | |||||
| You'll note that we don't request any particular version of the package: All | |||||
| versioning resolution is handled by ``dds``. You'll also note that we don't | |||||
| need to specify our transitive dependencies: This is handled by the libman | |||||
| index that was generated by ``dds``: It will automatically ``import_packages()`` | |||||
| any of the transitive dependencies required. | |||||
| More than one package name can be provided to a single call to | |||||
| ``import_packages()``, and ``import_packages()`` may be called multiple times | |||||
| within a CMake project. | |||||
| Using Our Dependencies' Libraries | |||||
| ================================= | |||||
| Like with ``dds``, CMake wants us to explicitly declare how our build targets | |||||
| *use* other libraries. When we import a package from a libman index, the | |||||
| import will generate CMake ``IMPORTED`` targets that can be linked against. | |||||
| In ``dds`` and in libman, a library is identified by a combination of | |||||
| *namespace* and *name*, joined together with a slash ``/`` character. This | |||||
| *qualified name* of a library is decided by the original package author, and | |||||
| should be documented. In the case of ``neo-sqlite3``, the only library is | |||||
| ``neo/sqlite3``. | |||||
| When the libman CMake module imports a library, it creates a qualified name | |||||
| using a double-colon "``::``" instead of a slash. As such, our ``neo/sqlite3`` | |||||
| is imported in CMake as ``neo::sqlite3``. We can link against it as we would | |||||
| with any other target:: | |||||
| add_executable(my-application app.cpp) | |||||
| target_link_libraries(my-application PRIVATE neo::sqlite3) | |||||
| Altogether, here is the final CMake file: | |||||
| .. code-block:: | |||||
| :caption: ``CMakeLists.txt`` | |||||
| :linenos: | |||||
| cmake_minimum_required(VERSION 3.15) | |||||
| project(MyApplication VERSION 1.0.0) | |||||
| include(pmm.cmake) | |||||
| pmm(CMakeCM ROLLING) | |||||
| include(libman) | |||||
| import_packages("neo-sqlite3") | |||||
| add_executable(my-application app.cpp) | |||||
| target_link_libraries(my-application PRIVATE neo::sqlite3) |
| The Package Catalog | |||||
| ################### | |||||
| ``dds`` stores a catalog of available packages, along with their dependency | |||||
| statements and information about how a source distribution thereof may be | |||||
| maintained. | |||||
| Viewing Catalog Contents | |||||
| ************************ | |||||
| The default catalog database is stored in a user-local location, and the | |||||
| package IDs available can be listed with ``dds catalog list``. This will only | |||||
| list the IDs of the packages, but none of the additional metadata about them. | |||||
| .. _catalog.adding: | |||||
| Adding Packages to the Catalog | |||||
| ****************************** | |||||
| The ``dds catalog import`` supports a ``--json`` flag that specifies a JSON5 | |||||
| file from which catalog entries will be generated. | |||||
| .. note:: | |||||
| The ``--json`` flag can be passed more than once to import multiple JSON | |||||
| files at once. | |||||
| The JSON file has the following structure: | |||||
| .. code-block:: javascript | |||||
| { | |||||
| // Import version spec. | |||||
| version: 1, | |||||
| // Packages section | |||||
| packages: { | |||||
| // Subkeys are package names | |||||
| "acme-gadgets": { | |||||
| // Keys within the package names are the versions that are | |||||
| // available for each package. | |||||
| "0.4.2": { | |||||
| // `depends` is an array of dependency statements for this | |||||
| // particular version of the package. (Optional) | |||||
| depends: [ | |||||
| "acme-widgets^1.4.1" | |||||
| ], | |||||
| // `description` is an attribute to give a string to describe | |||||
| // the package. (Optional) | |||||
| description: "A collection of useful gadgets.", | |||||
| // Specify the Git remote information | |||||
| git: { | |||||
| // `url` and `ref` are required. | |||||
| url: "http://example.com/git/repo/acme-gadgets.git", | |||||
| ref: "v0.4.2-stable", | |||||
| // The `auto-lib` is optional, to specify an automatic | |||||
| // library name/namespace pair to generate for the | |||||
| // root library | |||||
| "auto-lib": "Acme/Gadgets", | |||||
| // List of filesystem transformations to apply to the repository | |||||
| // (optional) | |||||
| transform: [ | |||||
| // ... (see below) ... | |||||
| ] | |||||
| } | |||||
| } | |||||
| } | |||||
| } | |||||
| } | |||||
| .. _catalog.fs-transform: | |||||
| Filesystem Transformations | |||||
| ************************** | |||||
| .. note:: | |||||
| Filesystem transformations is a transitional feature that is likely to be | |||||
| removed in a future release, and replaced with a more robust system when | |||||
| ``dds`` has a better way to download packages. Its aim is to allow ``dds`` | |||||
| projects to use existing libraries that might not meet the layout | |||||
| requirements that ``dds`` imposes, but can otherwise be consumed by ``dds`` | |||||
| with a few tweaks. | |||||
| A catalog entry can have a set of filesystem transformations attached to its | |||||
| remote information (e.g. the ``git`` property). When ``dds`` is obtaining a | |||||
| copy of the code for the package, it will apply the associated transformations | |||||
| to the filesystem based in the directory of the downloaded/cloned directory. In | |||||
| this way, ``dds`` can effectively "patch" the filesystem structure of a project | |||||
| arbitrarily. This allows many software projects to be imported into ``dds`` | |||||
| without needing to patch/fork the original project to support the required | |||||
| filesystem structure. | |||||
| .. important:: | |||||
| While ``dds`` allows you to patch directories downloaded via the catalog, a | |||||
| native ``dds`` project must still follow the layout rules. | |||||
| The intention of filesystem transformations is to act as a "bridge" that will allow ``dds`` projects to more easily utilize existing libraries. | |||||
| Available Transformations | |||||
| ========================= | |||||
| At time of writing, there are five transformations available to catalog entries: | |||||
| ``copy`` and ``move`` | |||||
| Copies or moves a set of files/directories from one location to another. Allows the following options: | |||||
| - ``from`` - The path from which to copy/move. **Required** | |||||
| - ``to`` - The destination path for the copy/move. **Required** | |||||
| - ``include`` - A list of globbing expressions for files to copy/move. If | |||||
| omitted, then all files will be included. | |||||
| - ``exclude`` - A list of globbing expressions of files to exclude from the | |||||
| copy/move. If omitted, then no files will be excluded. **If both** ``include`` and ``exclude`` are provided, ``include`` will be checked *before* ``exclude``. | |||||
| - ``strip-components`` - A positive integer (or zero, the default). When the | |||||
| ``from`` path identifies a directory, its contents will be copied/moved | |||||
| into the destination and maintain their relative path from the source path as their relative path within the destination. If ``strip-components`` is set to an integer ``N``, then the first ``N`` path components of that relative path will be removed when copying/moving the files in a directory. If a file's relative path has less than ``N`` components, then that file will be excluded from the ``copy/move`` operation. | |||||
| ``remove`` | |||||
| Delete files and directories from the package source. Has the following options: | |||||
| - ``path`` - The path of the file/directory to remove. **Required** | |||||
| - ``only-matching`` - A list of globbing expressions for files to remove. If omitted and the path is a directory, then the entire directory will be deleted. If at least one pattern is provided, then directories will be left intact and only non-directory files will be removed. If ``path`` names a non-directory file, then this option has no effect. | |||||
| ``write`` | |||||
| Write the contents of a string to a file in the package source. Has the following options: | |||||
| - ``path`` - The path of the file to write. **Required** | |||||
| - ``content`` - A string that will be written to the file. **Required** | |||||
| If the file exists and is not a directory, the file will be replaced. If the | |||||
| path names an existing directory, an error will be generated. | |||||
| ``edit`` | |||||
| Modifies the contents of the files in the package. | |||||
| - ``path`` - Path to the file to edit. **Required** | |||||
| - ``edits`` - An array of edit objects, applied in order, with the following | |||||
| keys: | |||||
| - ``kind`` - One of ``insert`` or ``delete`` to insert/delete lines, | |||||
| respectively. **Required** | |||||
| - ``line`` - The line at which to perform the insert/delete. The first line | |||||
| of the file is line one, *not* line zero. **Required** | |||||
| - ``content`` - For ``insert``, the string content to insert into the file. | |||||
| A newline will be appended after the content has been inserted. | |||||
| Transformations are added as a JSON array to the JSON object that specifies | |||||
| the remote information for the package. Each element of the array is an | |||||
| object, with one or more of the keys listed above. If an object features more | |||||
| than one of the above keys, they are applied in the same order as they have | |||||
| been listed. | |||||
| Example: Crypto++ | |||||
| ================= | |||||
| The following catalog entry will build and import `Crypto++`_ for use by a | |||||
| ``dds`` project. This uses the unmodified Crypto++ repository, which ``dds`` | |||||
| doesn't know how to build immediately. With some simple moving of files, we | |||||
| end up with something ``dds`` can build directly: | |||||
| .. code-block:: javascript | |||||
| "cryptopp": { | |||||
| "8.2.0": { | |||||
| "git": { | |||||
| "url": "https://github.com/weidai11/cryptopp.git", | |||||
| "ref": "CRYPTOPP_8_2_0", | |||||
| "auto-lib": "cryptopp/cryptopp", | |||||
| "transform": [ | |||||
| { | |||||
| // Crypto++ has no source directories at all, and everything lives | |||||
| // at the top level. No good for dds. | |||||
| // | |||||
| // Clients are expected to #include files with a `cryptopp/` prefix, | |||||
| // so we need to move the files around so that they match the | |||||
| // expected layout: | |||||
| "move": { | |||||
| // Move from the root of the repo: | |||||
| "from": ".", | |||||
| // Move files *into* `src/cryptopp` | |||||
| "to": "src/cryptopp", | |||||
| // Only move the C++ sources and headers: | |||||
| "include": [ | |||||
| "*.c", | |||||
| "*.cpp", | |||||
| "*.h" | |||||
| ] | |||||
| } | |||||
| } | |||||
| ] | |||||
| } | |||||
| } | |||||
| } | |||||
| Example: libsodium | |||||
| ================== | |||||
| For example, this catalog entry will build and import `libsodium`_ for use in | |||||
| a ``dds`` project. This uses the upstream libsodium repository, which does not | |||||
| meet the layout requirements needed by ``dds``. With a few simple | |||||
| transformations, we can allow ``dds`` to build and consume libsodium | |||||
| successfully: | |||||
| .. code-block:: javascript | |||||
| "libsodium": { | |||||
| "1.0.18": { | |||||
| "git": { | |||||
| "url": "https://github.com/jedisct1/libsodium.git", | |||||
| "ref": "1.0.18", | |||||
| "auto-lib": "sodium/sodium", | |||||
| /// Make libsodium look as dds expects of a project. | |||||
| "transform": [ | |||||
| // libsodium has a `src` directory, but it does not look how dds | |||||
| // expects it to. The public `#include` root of libsodium lives in | |||||
| // a nested subdirectory of `src/` | |||||
| { | |||||
| "move": { | |||||
| // Move the public header root out from that nested subdirectory | |||||
| "from": "src/libsodium/include", | |||||
| // Put it at `include/` in the top-level | |||||
| "to": "include/" | |||||
| } | |||||
| }, | |||||
| // libsodium has some files whose contents are generated by a | |||||
| // configure script. For demonstration purposes, we don't need most | |||||
| // of them, and we can just swipe an existing pre-configured file | |||||
| // that is already in the source repository and put it into the | |||||
| // public header root. | |||||
| { | |||||
| "copy": { | |||||
| // Generated version header committed to the repository: | |||||
| "from": "builds/msvc/version.h", | |||||
| // Put it where the configure script would put it: | |||||
| "to": "include/sodium/version.h" | |||||
| } | |||||
| }, | |||||
| // The subdirectory `src/libsodium/` is no good. It now acts as an | |||||
| // unnecessary layer of indirection. We want `src/` to be the root. | |||||
| // We can just "lift" the subdirectory: | |||||
| { | |||||
| // Up we go: | |||||
| "move": { | |||||
| "from": "src/libsodium", | |||||
| "to": "src/" | |||||
| }, | |||||
| // Delete the now-unused subdirectory: | |||||
| "remove": { | |||||
| "path": "src/libsodium" | |||||
| } | |||||
| }, | |||||
| // Lastly, libsodium's source files expect to resolve their header | |||||
| // paths differently than they expect of their clients (Bad!!!). | |||||
| // Fortunately, we can do a hack to allow the files in `src/` to | |||||
| // resolve its headers. The source files use #include as if the | |||||
| // header root was `include/sodium/`, rather than `include/`. | |||||
| // To work around this, generate a copy of each header file in the | |||||
| // source root, but remove the leading path element. | |||||
| // Because we have a separate `include/` and `src/` directory, dds | |||||
| // will only expose the `include/` directory to clients, and the | |||||
| // header copies in `src/` are not externally visible. | |||||
| // | |||||
| // For example, the `include/sodium/version.h` file is visible to | |||||
| // clients as `sodium/version.h`, but libsodium itself tries to | |||||
| // include it as `version.h` within its source files. When we copy | |||||
| // from `include/`, we grab the relative path to `sodium/version.h`, | |||||
| // strip the leading components to get `version.h`, and then join that | |||||
| // path with the `to` path to generate the full destination at | |||||
| // `src/version.h` | |||||
| { | |||||
| "copy": { | |||||
| "from": "include/", | |||||
| "to": "src/", | |||||
| "strip-components": 1 | |||||
| } | |||||
| } | |||||
| ] | |||||
| } | |||||
| } | |||||
| } | |||||
| .. _libsodium: https://doc.libsodium.org/ | |||||
| .. _Crypto++: https://cryptopp.com/ |
| .. highlight:: cmake | .. highlight:: cmake | ||||
| Using ``dds`` Packages in a CMake Project | |||||
| ######################################### | |||||
| Easy Mode: Using ``dds`` in a CMake Project | |||||
| ########################################### | |||||
| One of ``dds``'s primary goals is to inter-operate with other build systems | One of ``dds``'s primary goals is to inter-operate with other build systems | ||||
| cleanly. One of ``dds``'s primary outputs is *libman* package indices. These | |||||
| package indices can be imported into other build systems that support the | |||||
| *libman* format. | |||||
| cleanly. Because of CMakes ubiquity, ``dds`` includes built-in support for | |||||
| emitting files that can be imported into CMake. | |||||
| .. note:: | |||||
| ``dds`` doesn't (yet) have a ready-made central repository of packages that | |||||
| can be downloaded. You'll need to populate the local package catalog | |||||
| appropriately. The default catalog file contains a limited set of useful | |||||
| packages, but you may wish to add more for yourself. | |||||
| .. seealso:: Refer to :doc:`catalog` for information about remote packages. | |||||
| .. _PMM: https://github.com/vector-of-bool/PMM | |||||
| .. _CMakeCM: https://github.com/vector-of-bool/CMakeCM | |||||
| .. _lm-cmake: https://raw.githubusercontent.com/vector-of-bool/libman/develop/cmake/libman.cmake | |||||
| Generating a libman Index | |||||
| ************************* | |||||
| Importing libman packages into a build system requires that we have a libman | |||||
| index generated on the filesystem. **This index is not generated globally**: It | |||||
| is generated on a per-build basis as part of the build setup. The index will | |||||
| describe in build-system-agnostic terms how to include a set of packages and | |||||
| libraries as part of a build. | |||||
| ``dds`` has first-class support for generating this index. The ``build-deps`` | |||||
| subcommand of ``dds`` will download and build a set of dependencies, and places | |||||
| an ``INDEX.lmi`` file that can be used to import the built results. | |||||
| Declaring Dependencies | |||||
| ====================== | |||||
| ``dds build-deps`` accepts a list of dependency statements as commnad line | |||||
| arguments, but it may be useful to specify those requirements in a file. | |||||
| ``dds build-deps`` accepts a JSON5 file describing the dependencies of a | |||||
| project as well. This file is similar to a very stripped-down version of a | |||||
| ``dds`` :ref:`package manifest <pkgs.pkgs>`, and only includes the ``depends`` | |||||
| key. (The presence of any other key is an error.) | |||||
| Here is a simple dependencies file that declares a single requirement: | |||||
| .. code-block:: js | |||||
| :caption: ``dependencies.json5`` | |||||
| { | |||||
| depends: [ | |||||
| 'neo-sqlite3^0.2.0', | |||||
| ] | |||||
| } | |||||
| Building Dependencies and the Index | |||||
| =================================== | |||||
| We can invoke ``dds build-deps`` and give it the path to this file: | |||||
| .. code-block:: bash | |||||
| $ dds build-deps --deps dependencies.json5 | |||||
| .. seealso:: | |||||
| When finished, ``dds`` will write the build results into a subdirectory called | |||||
| ``_deps`` and generate a file named ``INDEX.lmi``. This file is ready to be | |||||
| imported into any build system that can understand libman files (in our case, | |||||
| CMake). | |||||
| Before reading this page, be sure to read the :ref:`build-deps.gen-libman` | |||||
| section of the :doc:`build-deps` page, which will discuss how to use the | |||||
| ``dds build-deps`` subcommand. | |||||
| .. note:: | .. note:: | ||||
| The output directory and index filepath can be controlled with the | |||||
| ``--out`` and ``--lmi-path`` flags, respectively. | |||||
| We'll first look as *easy mode*, but there's also an *easiest mode* for a | |||||
| one-line solution: :ref:`see below <cmake.pmm>`. | |||||
| Importing into CMake | |||||
| ******************** | |||||
| We've generated a libman index and set of packages, and we want to import | |||||
| them into CMake. CMake doesn't know how to do this natively, but there exists a | |||||
| single-file module for CMake that allows CMake to import libraries from libman | |||||
| indices without any additional work. | |||||
| The module is not shipped with CMake, but is available online as a single | |||||
| stand-alone file. The `libman.cmake <lm-cmake_>`_ file can be downloaded and | |||||
| added to a project directly, or it can be obtained automatically through a | |||||
| CMake tool like `PMM`_ (recommended). | |||||
| Enabling *libman* Support in CMake via PMM | |||||
| ========================================== | |||||
| Refer to the ``README.md`` file in `the PMM repo <PMM_>`_ for information on how | |||||
| to get PMM into your CMake project. In short, download and place the | |||||
| ``pmm.cmake`` file in your repository, and ``include()`` the file near the top | |||||
| of your ``CMakeLists.txt``:: | |||||
| include(pmm.cmake) | |||||
| Once it has been included, you can call the ``pmm()`` function. To obtain | |||||
| *libman*, we need to start by enabling `CMakeCM`_:: | |||||
| pmm(CMakeCM ROLLING) | |||||
| .. warning:: | |||||
| It is not recommended to use the ``ROLLING`` mode, but it is the easiest to | |||||
| use when getting started. For reproducible and reliable builds, you should | |||||
| pin your CMakeCM version using the ``FROM <url>`` argument. | |||||
| Enabling CMakeCM will make available all of the CMake modules available in `the | |||||
| CMakeCM repository <CMakeCM_>`_, which includes `libman.cmake <lm-cmake_>`_. | |||||
| .. _PMM: https://github.com/vector-of-bool/PMM | |||||
| After the call to ``pmm()``, simply ``include()`` the ``libman`` module:: | |||||
| include(libman) | |||||
| Generating a CMake Import File | |||||
| ****************************** | |||||
| That's it! The only function from the module that we will care about for now | |||||
| is the ``import_packages()`` function. | |||||
| ``build-deps`` accepts an ``--lmi-path`` argument, but also accepts a | |||||
| ``--cmake=<path>`` argument that serves a similar purpose: It will write a CMake | |||||
| file to ``<path>`` that can be ``include()``'d into a CMake project: | |||||
| .. code-block:: bash | |||||
| Importing Our Dependencies' Packages | |||||
| ==================================== | |||||
| $ dds build-deps "neo-sqlite3^0.2.0" --cmake=deps.cmake | |||||
| To import a package from a libman tree, we need only know the *name* of the | |||||
| package we wish to import. In our example case above, we depend on | |||||
| ``neo-sqlite3``, so we simply call the libman-CMake function | |||||
| ``import_packages()`` with that package name:: | |||||
| This will write a file ``./deps.cmake`` that we can ``include()`` from a CMake | |||||
| project, which will then expose the ``neo-sqlite3`` package as a set of imported | |||||
| targets. | |||||
| import_packages("neo-sqlite3") | |||||
| You'll note that we don't request any particular version of the package: All | |||||
| versioning resolution is handled by ``dds``. You'll also note that we don't | |||||
| need to specify our transitive dependencies: This is handled by the libman | |||||
| index that was generated by ``dds``: It will automatically ``import_packages()`` | |||||
| any of the transitive dependencies required. | |||||
| Using the CMake Import File | |||||
| =========================== | |||||
| Once we have generated the CMake import file using ``dds build-deps``, we can | |||||
| simply import it in our ``CMakeLists.txt``:: | |||||
| Using Our Dependencies' Libraries | |||||
| ================================= | |||||
| include(deps.cmake) | |||||
| Like with ``dds``, CMake wants us to explicitly declare how our build targets | Like with ``dds``, CMake wants us to explicitly declare how our build targets | ||||
| *use* other libraries. When we import a package from a libman index, the | |||||
| import will generate CMake ``IMPORTED`` targets that can be linked against. | |||||
| *use* other libraries. When we ``include()`` the generated CMake file, it will | |||||
| generate ``IMPORTED`` targets that can be linked against. | |||||
| In ``dds`` and in libman, a library is identified by a combination of | |||||
| In ``dds`` (and in libman), a library is identified by a combination of | |||||
| *namespace* and *name*, joined together with a slash ``/`` character. This | *namespace* and *name*, joined together with a slash ``/`` character. This | ||||
| *qualified name* of a library is decided by the original package author, and | *qualified name* of a library is decided by the original package author, and | ||||
| should be documented. In the case of ``neo-sqlite3``, the only target is | |||||
| should be documented. In the case of ``neo-sqlite3``, the only library is | |||||
| ``neo/sqlite3``. | ``neo/sqlite3``. | ||||
| When the libman CMake module imports a library, it creates a qualified name | |||||
| When the generated import file imports a library, it creates a qualified name | |||||
| using a double-colon "``::``" instead of a slash. As such, our ``neo/sqlite3`` | using a double-colon "``::``" instead of a slash. As such, our ``neo/sqlite3`` | ||||
| is imported in CMake as ``neo::sqlite3``. We can link against it as we would | is imported in CMake as ``neo::sqlite3``. We can link against it as we would | ||||
| with any other target:: | with any other target:: | ||||
| add_executable(my-application app.cpp) | |||||
| target_link_libraries(my-application PRIVATE neo::sqlite3) | |||||
| Altogether, here is the final CMake file: | |||||
| add_executable(my-application app.cpp) | |||||
| target_link_libraries(my-application PRIVATE neo::sqlite3) | |||||
| .. code-block:: | |||||
| :caption: ``CMakeLists.txt`` | |||||
| :linenos: | |||||
| cmake_minimum_required(VERSION 3.15) | |||||
| project(MyApplication VERSION 1.0.0) | |||||
| .. _cmake.pmm: | |||||
| include(pmm.cmake) | |||||
| pmm(CMakeCM ROLLING) | |||||
| *Easiest* Mode: PMM Support | |||||
| *************************** | |||||
| include(libman) | |||||
| import_packages("neo-sqlite3") | |||||
| add_executable(my-application app.cpp) | |||||
| target_link_libraries(my-application PRIVATE neo::sqlite3) | |||||
| `PMM`_ is the *package package manager*, and can be used to control and access | |||||
| package managers from within CMake scripts. This includes controlling ``dds``. | |||||
| With PMM, we can automate all of the previous steps into a single line. | |||||
| Refer to the ``README.md`` file in `the PMM repo <PMM>`_ for information on how | |||||
| to get PMM into your CMake project. In short, download and place the | |||||
| ``pmm.cmake`` file in your repository, and ``include()`` the file near the top | |||||
| of your ``CMakeLists.txt``:: | |||||
| Additional PMM Support | |||||
| ********************** | |||||
| include(pmm.cmake) | |||||
| The ``pmm()`` function also supports ``dds`` directly, similar to ``CMakeCM`` | |||||
| mode. This will automatically download a prebuilt ``dds`` for the host platform | |||||
| and invoke ``dds build-deps`` in a single pass as part of CMake's configure | |||||
| process. This is especially useful for a CI environment where you want to have | |||||
| a stable ``dds`` version and always have your dependencies obtained | |||||
| just-in-time. | |||||
| The ``pmm()`` function also supports ``dds`` directly, and will automatically | |||||
| download a prebuilt ``dds`` for the host platform and invoke ``dds build-deps`` | |||||
| in a single pass as part of CMake's configure process. This is especially useful | |||||
| for a CI environment where you want to have a stable ``dds`` version and always | |||||
| have your dependencies obtained just-in-time. | |||||
| To start, pass the ``DDS`` argument to ``pmm()`` to use it:: | To start, pass the ``DDS`` argument to ``pmm()`` to use it:: | ||||
| pmm(DDS) | |||||
| pmm(DDS) | |||||
| .. note:: | .. note:: | ||||
| The ``_deps`` directory and ``INDEX.lmi`` file will be placed in the CMake | |||||
| build directory, out of the way of the rest of the project. | |||||
| The ``_deps`` directory and generated CMake imports file will be placed in | |||||
| the CMake build directory, out of the way of the rest of the project. | |||||
| .. note:: | .. note:: | ||||
| The version of ``dds`` that PMM downloads depends on the version of PMM | |||||
| that is in use. | |||||
| The version of ``dds`` that PMM downloads depends on the version of PMM | |||||
| that is in use. | |||||
| This alone won't do anything useful, because you'll need to tell it what | This alone won't do anything useful, because you'll need to tell it what | ||||
| dependencies we want to install:: | dependencies we want to install:: | ||||
| pmm(DDS DEP_FILES dependencies.json5) | |||||
| pmm(DDS DEP_FILES dependencies.json5) | |||||
| You can also list your dependencies as an inline string in your CMakeLists.txt | |||||
| You can also list your dependencies as inline strings in your CMakeLists.txt | |||||
| instead of a separate file:: | instead of a separate file:: | ||||
| pmm(DDS DEPENDS neo-sqlite3^0.2.2) | |||||
| pmm(DDS DEPENDS neo-sqlite3^0.2.2) | |||||
| Since you'll probably want to be using ``libman.cmake`` at the same time, the | |||||
| calls for ``CMakeCM`` and ``DDS`` can simply be combined. This is how our new | |||||
| CMake project might look: | |||||
| This invocation will run ``build-deps`` with the build options, generate a CMake | |||||
| imports file, and immediately ``include()`` it to import the generated CMake | |||||
| targets. ``pmm(DDS)`` will also generate a ``dds`` :doc:`toolchain <toolchains>` | |||||
| based on the current CMake build environment, ensuring that the generated | |||||
| packages have matching build options to the rest of the project. Refer to the | |||||
| PMM README for more details. | |||||
| .. code-block:: | .. code-block:: | ||||
| :caption: ``CMakeLists.txt`` | |||||
| :linenos: | |||||
| cmake_minimum_required(VERSION 3.15) | |||||
| project(MyApplication VERSION 1.0.0) | |||||
| :caption: ``CMakeLists.txt`` | |||||
| :linenos: | |||||
| :emphasize-lines: 4,5 | |||||
| include(pmm.cmake) | |||||
| pmm(CMakeCM ROLLING | |||||
| DDS DEPENDS neo-sqlite3^0.2.2 | |||||
| ) | |||||
| cmake_minimum_required(VERSION 3.15) | |||||
| project(MyApplication VERSION 1.0.0) | |||||
| include(libman) | |||||
| import_packages("neo-sqlite3") | |||||
| include(pmm.cmake) | |||||
| pmm(DDS DEPENDS neo-sqlite3^0.2.2) | |||||
| add_executable(my-application app.cpp) | |||||
| target_link_libraries(my-application PRIVATE neo::sqlite3) | |||||
| add_executable(my-application app.cpp) | |||||
| target_link_libraries(my-application PRIVATE neo::sqlite3) | |||||
| This removes the requirement that we write a separate dependencies file, and we | This removes the requirement that we write a separate dependencies file, and we | ||||
| no longer need to invoke ``dds build-deps`` externally, as it is all handled | no longer need to invoke ``dds build-deps`` externally, as it is all handled | ||||
| by ``pmm``. | |||||
| by ``pmm()``. |
| packages | packages | ||||
| toolchains | toolchains | ||||
| source-dists | source-dists | ||||
| repo | |||||
| catalog | |||||
| pkg-cache | |||||
| remote-pkgs | |||||
| interdeps | interdeps | ||||
| build-deps | |||||
| cmake | cmake |
| Refer to: :ref:`deps.ranges.why-lowest`. | Refer to: :ref:`deps.ranges.why-lowest`. | ||||
| ``dds`` compatible-version ranges are similar to the shorthand range specifiers | ``dds`` compatible-version ranges are similar to the shorthand range specifiers | ||||
| supported by ``npm`` and ``npm``-like tools. There are five (and a half) | |||||
| version range formats available, listed in order of most-to-least restrictive: | |||||
| supported by ``npm`` and ``npm``-like tools. There are four version range kinds | |||||
| available, listed in order of most-to-least restrictive: | |||||
| Exact: ``@1.2.3`` | Exact: ``@1.2.3`` | ||||
| Specifies an *exact* requirement. The dependency must match the named | Specifies an *exact* requirement. The dependency must match the named |
| .. note:: | .. note:: | ||||
| Although headers are not compiled, this does not mean they are ignored. | Although headers are not compiled, this does not mean they are ignored. | ||||
| ``dds`` still understands and respects headers, and they are collected | ``dds`` still understands and respects headers, and they are collected | ||||
| together as part of *source distribution*. | |||||
| together as part of a *source distribution*. | |||||
| .. _pkgs.apps-tests: | .. _pkgs.apps-tests: | ||||
| ********************** | ********************** | ||||
| ``dds`` will recognize certain compilable source files as belonging to | ``dds`` will recognize certain compilable source files as belonging to | ||||
| applications and tests. If a compilable source file stem ends with ``.main`` or | |||||
| ``.test``, that source file is assumed to correspond to an executable to | |||||
| generate. The filename stem before the ``.main`` or ``.test`` will be used as | |||||
| the name of the generated executable. For example: | |||||
| applications and tests, depending on the filenames "stem," which is the part of | |||||
| the filename not including the outer-most file extension. If a compilable source | |||||
| filename stem ends with ``.main`` or ``.test``, that source file is assumed to | |||||
| correspond to an executable to generate. The filename second-inner stem before | |||||
| the ``.main`` or ``.test`` will be used as the name of the generated executable. | |||||
| For example: | |||||
| - ``foo.main.cpp`` will generate an executable named ``foo``. | |||||
| - ``bar.test.cpp`` will generate an executable named ``bar``. | |||||
| - ``cat-meow.main.cpp`` will generate an executable named ``cat-meow``. | |||||
| - ``cats.musical.test.cpp`` will generate an executable named ``cats.musical``. | |||||
| - Given ``foo.main.cpp`` | |||||
| - The stem is ``foo.main``, whose extension is ``.main``, so we will generate | |||||
| an application. | |||||
| - The stem of ``foo.main`` is ``foo``, so the executable will be named | |||||
| ``foo``. | |||||
| - Given ``bar.test.cpp`` | |||||
| - The stem is ``bar.test``, whose extension is ``.test``, so we will generate | |||||
| a test. | |||||
| - The stem of ``bar.test`` is ``bar``, so will generate an executable named | |||||
| ``bar``. | |||||
| - Given ``cat-meow.main.cpp`` | |||||
| - The stem is ``cat-meow.main``, which has extension ``.main``, so it is an | |||||
| application. | |||||
| - The stem of ``cat-meow.main`` is ``cat-meow``, so will generate an | |||||
| executable named ``cat-meow``. | |||||
| - Given ``cats.musical.test.cpp`` | |||||
| - The stem is ``cats.musical.test``, which has extension ``.test``, so this is | |||||
| a text executable. | |||||
| - The stem of ``cats.musical.test`` is ``cats.musical``, so we will generate | |||||
| an executable named ``cats.musical``. | |||||
| - Note that the dot in ``cats.musical`` is not significant, as ``dds`` does | |||||
| strip any further extensions. | |||||
| .. note:: | .. note:: | ||||
| ``dds`` will automatically append the appropriate filename extension to the | ``dds`` will automatically append the appropriate filename extension to the | ||||
| ``src/`` directory must be added to their *include search path*. | ``src/`` directory must be added to their *include search path*. | ||||
| Because the ``#include`` directives are based on the *portable* source root, | Because the ``#include`` directives are based on the *portable* source root, | ||||
| the exactly location of ``src/`` is not important to the content of the | |||||
| the exact location of ``src/`` is not important to the content of the | |||||
| consuming source code, and can thus be relocated and renamed as necessary. | consuming source code, and can thus be relocated and renamed as necessary. | ||||
| Consumers only need to update the path of the *include search path* in a single | Consumers only need to update the path of the *include search path* in a single | ||||
| location rather than modifying their source code. | location rather than modifying their source code. | ||||
| Packages are identified by a name/version pair, joined together by an ``@`` | Packages are identified by a name/version pair, joined together by an ``@`` | ||||
| symbol. The version of a package must be a semantic version string. Together, | symbol. The version of a package must be a semantic version string. Together, | ||||
| the ``name@version`` string forms the *package ID*, and it must be unique | |||||
| within a repository or package catalog. | |||||
| the ``name@version`` string forms the *package ID*, and it must be unique within | |||||
| a repository or local package cache. | |||||
| In order for a package to be exported by ``dds`` it must have a | In order for a package to be exported by ``dds`` it must have a | ||||
| ``package.json5`` file at its package root. Three keys are required to be | ``package.json5`` file at its package root. Three keys are required to be |
| The Local Package Cache | |||||
| ####################### | |||||
| ``dds`` maintains a local cache of packages that it has obtained at the request | |||||
| of a user. The packages themselves are stored as | |||||
| :doc:`source distributions <source-dists>` (``dds`` does not store the binaries | |||||
| that it builds within this package cache). | |||||
| Reading Repository Contents | |||||
| *************************** | |||||
| Most times, ``dds`` will manage the cache content silently, but it may be useful | |||||
| to see what ``dds`` is currently storing away. | |||||
| The content of the cache can be seen with the ``pkg ls`` subcommand:: | |||||
| > dds pkg ls | |||||
| This will print the names of packages that ``dds`` has downloaded, as well as | |||||
| the versions of each. | |||||
| Obtaining Packages | |||||
| ****************** | |||||
| .. seealso:: See also: :doc:`remote-pkgs` | |||||
| When ``dds`` builds a package, it will also build the dependency libraries of | |||||
| that package. In order for the dependency build to succeed, it must have a | |||||
| local copy of the source distribution of that dependency. | |||||
| When ``dds`` performs dependency resolution, it will consider both locally | |||||
| cached packages, as well as packages that are available from any | |||||
| :doc:`remote packages <remote-pkgs>`. If the dependency solution requires any | |||||
| packages that are not in the local cache, it will use the information in its | |||||
| catalog database to obtain a source distribution for each missing package. These | |||||
| source distributions will automatically be added to the local cache, and later | |||||
| dependency resolutions will not need to download that package again. | |||||
| This all happens automatically when a project is built: There is **no** | |||||
| "``dds install``" subcommand. | |||||
| Manually Downloading a Dependency | |||||
| ================================= | |||||
| It may be useful to obtain a copy of the source distribution of a package | |||||
| from a remote. The ``pkg get`` command can be used to do this:: | |||||
| > dds pkg get <name>@<version> | |||||
| This will obtain the source distribution of the package matching the given | |||||
| package ID and place that distribution in current working directory, using the | |||||
| package ID as the name of the source distribution directory:: | |||||
| $ dds pkg get spdlog@1.4.2 | |||||
| [ ... ] | |||||
| $ ls . | |||||
| . | |||||
| .. | |||||
| spdlog@1.4.2 | |||||
| $ ls ./spdlog@1.4.2/ | |||||
| include/ | |||||
| src/ | |||||
| library.json5 | |||||
| package.json5 | |||||
| .. _repo.import-local: | |||||
| Exporting a Project into the Repository | |||||
| *************************************** | |||||
| ``dds`` can only use packages that are available in the local cache. For | |||||
| packages that have a listing in the catalog, this is not a problem. But if one | |||||
| is developing a local package and wants to allow it to be used in another local | |||||
| package, that can be done by importing that project into the package cache as a | |||||
| regular package, as detailed in :ref:`sdist.import`:: | |||||
| > dds pkg import /path/to/project | |||||
| This command will create a source distribution and place it in the local cache. | |||||
| The package is now available to other projects on the local system. | |||||
| .. note:: | |||||
| This doesn't import in "editable" mode: A snapshot of the package root | |||||
| will be taken and imported to the local cache. |
| Remote Packages and Repositories | |||||
| ################################ | |||||
| .. highlight:: bash | |||||
| ``dds`` stores a local database of available packages, along with their | |||||
| dependency statements and information about how a source distribution thereof | |||||
| may be obtained. | |||||
| Inside the database are *package repositories*, which are remote servers that | |||||
| contain their own database of packages, and may also contain the packages | |||||
| themselves. An arbitrary number of package repositories may be added to the | |||||
| local database. When ``dds`` updates its package information, it will download | |||||
| the package database from each registered remote and import the listings into | |||||
| its own local database, making them available for download. | |||||
| Viewing Available Packages | |||||
| ************************** | |||||
| The default catalog database is stored in a user-local location, and the | |||||
| available packages can be listed with ``dds pkg search``:: | |||||
| $ dds pkg search | |||||
| Name: abseil | |||||
| Versions: 2018.6.0, 2019.8.8, 2020.2.25 | |||||
| From: repo-1.dds.pizza | |||||
| Name: asio | |||||
| Versions: 1.12.0, 1.12.1, 1.12.2, 1.13.0, 1.14.0, 1.14.1, 1.16.0, 1.16.1 | |||||
| From: repo-1.dds.pizza | |||||
| Name: boost.leaf | |||||
| Versions: 0.1.0, 0.2.0, 0.2.1, 0.2.2, 0.2.3, 0.2.4, 0.2.5, 0.3.0 | |||||
| From: repo-1.dds.pizza | |||||
| Name: boost.mp11 | |||||
| Versions: 1.70.0, 1.71.0, 1.72.0, 1.73.0 | |||||
| From: repo-1.dds.pizza | |||||
| Optionally, one can search with a glob/fnmatch-style pattern:: | |||||
| $ dds pkg search 'neo-*' | |||||
| Name: neo-buffer | |||||
| Versions: 0.2.1, 0.3.0, 0.4.0, 0.4.1, 0.4.2 | |||||
| From: repo-1.dds.pizza | |||||
| Name: neo-compress | |||||
| Versions: 0.1.0, 0.1.1, 0.2.0 | |||||
| From: repo-1.dds.pizza | |||||
| Name: neo-concepts | |||||
| Versions: 0.2.2, 0.3.0, 0.3.1, 0.3.2, 0.4.0 | |||||
| From: repo-1.dds.pizza | |||||
| Remote Repositories | |||||
| ******************* | |||||
| A remote package repository consists of an HTTP(S) server serving the following: | |||||
| 1. An accessible directory containing source distributions of various packages, | |||||
| and | |||||
| 2. An accessible database file that contains a listing of packages and some | |||||
| repository metadata. | |||||
| The exact details of the directory layout and database are not covered here, and | |||||
| are not necessary to make use of a repository. | |||||
| When ``dds`` uses a repository, it pulls down the database file and imports its | |||||
| contents into its own local database, associating the imported package listings | |||||
| with the remote repository which provides them. Pulling the entire database at | |||||
| once allows ``dds`` to perform much faster dependency resolution and reduces | |||||
| the round-trips associated with using a dynamic package repository. | |||||
| Adding a Repository | |||||
| =================== | |||||
| Adding a remote repository to the local database is a simple single command:: | |||||
| $ dds pkg repo add "https://repo-1.dds.pizza" | |||||
| [info ] Pulling repository contents for repo-1.dds.pizza [https://repo-1.dds.pizza/] | |||||
| This will tell ``dds`` to add ``https://repo-1.dds.pizza`` as a remote | |||||
| repository and immediately pull its package listings for later lookup. This | |||||
| initial update can be suppressed with the ``--no-update`` flag. | |||||
| .. note:: | |||||
| After the initial ``pkg repo add``, the repository is *not* identified by its | |||||
| URL, but by its *name*, which is provided by the repository itself. The name | |||||
| is printed the first time it is added, and can be seen using ``pkg repo ls``. | |||||
| Listing Repositories | |||||
| ==================== | |||||
| A list of package repositories can be seen with the ``pkg repo ls`` subcommand:: | |||||
| $ dds pkg repo ls | |||||
| Removing Repositories | |||||
| ===================== | |||||
| A repository can be removed by the ``pkg repo remove`` subcommand:: | |||||
| $ dds pkg repo remove <name> | |||||
| Where ``<name>`` is given as the *name* (not URL!) of the repository. | |||||
| **Note** that removing a repository will make all of its corresponding remote | |||||
| packages unavailable, while packages that have been pulled into the local cache | |||||
| will remain available even after removing a repository. | |||||
| Updating Repository Data | |||||
| ======================== | |||||
| Repository data and package listings can be updated with the ``pkg repo update`` | |||||
| subcommand:: | |||||
| $ dds pkg repo update | |||||
| This will pull down the databases of all registered remote repositories. If | |||||
| ``dds`` can detect that a repository's database is unchanged since a prior | |||||
| update, that update will be skipped. | |||||
| The Default Repository | |||||
| ********************** | |||||
| When ``dds`` first initializes its local package database, it will add a single | |||||
| remote repository: ``https://repo-1.dds.pizza/``, which has the name | |||||
| ``repo-1.dds.pizza``. At the time of writing, this is the only official ``dds`` | |||||
| repository, and is populated sparsely with hand-curated and prepared packages. | |||||
| In the future, the catalog of packages will grow and be partially automated. | |||||
| There is nothing intrinsically special about this repository other than it being | |||||
| the default when ``dds`` first creates its package database. It can be removed | |||||
| as any other, should one want tighter control over package availability. | |||||
| Managing a Repository | |||||
| ********************* | |||||
| A ``dds`` repository is simply a directory of static files, so any HTTP server | |||||
| that can serve from a filesystem can be used as a repository. ``dds`` also | |||||
| ships with a subcommand, ``repoman``, that can be used to manage a repository | |||||
| directory. | |||||
| Initializing a Repository | |||||
| ========================= | |||||
| Before anything can be done, a directory should be converted to a repository by | |||||
| using ``repoman init``:: | |||||
| $ dds repoman init ./my-repo-dir --name=my-experimental-repo | |||||
| This will add the basic metadata into ``./my-repo-dir`` such that ``dds`` will | |||||
| be able to pull package data from it. | |||||
| The ``--name`` argument should be used to give the repository a unique name. The | |||||
| name should be globally unique to avoid collisions: When ``dds`` pulls a | |||||
| repository that declares a given name, it will *replace* the package listings | |||||
| associated with any repository of that name. As such, generic names like | |||||
| ``main`` or ``packages`` shouldn't be used in production. | |||||
| Listing Contents | |||||
| ================ | |||||
| The packages in a repository can be listed using ``dds repoman ls <repo-dir>``. | |||||
| This will simply print each package identifier that is present in the | |||||
| repository. | |||||
| Importing Source Distributions | |||||
| ============================== | |||||
| If you have a source distribution archive, it can be imported with the | |||||
| appropriately name ``dds repoman import`` command:: | |||||
| $ dds repoman import ./my-repo some-pkg@1.2.3.tar.gz | |||||
| Multiple archive paths may be provided at once to import them all at once. | |||||
| Adding a Package by URL | |||||
| ======================= | |||||
| A repository can also list packages that it does not host itself. Such a package | |||||
| listing can be added "by URL," where the URL tells ``dds`` how to pull the | |||||
| source distribution of the package. Beyond basic HTTP(S) URLs, ``dds`` can also | |||||
| clone packages via ``git``:: | |||||
| $ dds repoman add ./my-repo git+https://github.com/vector-of-bool/neo-fun.git#0.5.2 | |||||
| The above URL tells ``dds`` that it can use ``git clone`` against | |||||
| ``https://github.com/vector-of-bool/neo-fun.git`` and ask for tag ``0.5.2`` to | |||||
| get a source distribution directory that can be imported. Note the fragment on | |||||
| ``git`` URLs! The fragment is required to specify the branch or tag to clone. | |||||
| If the package is available on GitHub, ``dds`` has a shorthand URL for that:: | |||||
| $ dds repoman add ./my-repo github:vector-of-bool/neo-fun/0.6.0 | |||||
| The ``github:`` URL scheme tells ``dds`` to clone from GitHub. A ``github:`` URL | |||||
| must have exactly three path elements to determine *what* to download: | |||||
| ``github:{owner}/{repository}/{branch-or-tag}``. | |||||
| .. note:: | |||||
| The ``github:`` URL lacks an *authority* element, and as such *does not* use | |||||
| the double-slash. | |||||
| .. note:: | |||||
| ``repoman add`` will immediately attempt to pull a source distribution from | |||||
| the given URL so that it may import the package's metadata into its listing | |||||
| database. You cannot add a URL that is not already accessible. | |||||
| Removing Packages | |||||
| ================= | |||||
| A package can be removed from a repository with | |||||
| ``dds repoman remove <repo-dir> <pkg-id>``, where ``<pkg-id>`` is the | |||||
| ``<name>@<version>`` of the package to remove. |
| The Local Package Repository | |||||
| ############################ | |||||
| ``dds`` maintains a local repository of packages that it has obtained at the | |||||
| request of a user. The packages themselves are stored as | |||||
| :doc:`source distributions <source-dists>` (``dds`` does not store the binaries | |||||
| that it builds within the repository). | |||||
| Reading Repository Contents | |||||
| *************************** | |||||
| Most times, ``dds`` will manage the repository content silently, but it may be | |||||
| useful to see what ``dds`` is currently storing away. | |||||
| The content of the repostiory can be seen with the ``repo`` subcommand:: | |||||
| > dds repo ls | |||||
| This will print the names of packages that ``dds`` has downloaded, as well as | |||||
| the versions of each. | |||||
| Obtaining Packages | |||||
| ****************** | |||||
| .. seealso:: See also: :doc:`catalog` | |||||
| When ``dds`` builds a package, it will also build the dependency libraries of | |||||
| that package. In order for the dependency build to succeed, it must have a | |||||
| local copy of the source distribution of that dependency. | |||||
| When ``dds`` performs dependency resolution, it will consider both existing | |||||
| packages in the local repository, as well as packages that are available from | |||||
| the :doc:`package catalog <catalog>`. If the dependency solution requires any | |||||
| packages that are not in the local repository, it will use the information in | |||||
| the catalog to obtain a source distribution for each missing package. These | |||||
| source distributions will automatically be added to the local repository, and | |||||
| later dependency resolutions will not need to download that package again. | |||||
| Manually Downloading a Dependency | |||||
| ================================= | |||||
| It may be useful to obtain a copy of the source distribution of a package | |||||
| contained in the catalog. The ``catalog get`` command can be used to do this:: | |||||
| > dds catalog get <name>@<version> | |||||
| This will obtain the source distribution of the package matching the named | |||||
| identifier and place that distribution in current working directory, using the | |||||
| package ID as the name of the source distribution directory:: | |||||
| $ dds catalog get spdlog@1.4.2 | |||||
| [ ... ] | |||||
| $ ls . | |||||
| . | |||||
| .. | |||||
| spdlog@1.4.2 | |||||
| $ ls ./spdlog@1.4.2/ | |||||
| include/ | |||||
| src/ | |||||
| library.json5 | |||||
| package.json5 | |||||
| .. _repo.export-local: | |||||
| Exporting a Project into the Repository | |||||
| *************************************** | |||||
| ``dds`` can only use packages that are available in the local repository. For | |||||
| packages that have a listing in the catalog, this is not a problem. But if one | |||||
| is developing a local package and wants to allow it to be used in another local | |||||
| package, that can be done by exporting a source distribution from the package | |||||
| root:: | |||||
| > dds sdist export | |||||
| This command will create a source distribution and place it in the local | |||||
| repository. The package is now available to other projects on the local system. | |||||
| .. note:: | |||||
| This doesn't export in "editable" mode: A snapshot of the package root | |||||
| will be taken and exported to the local repository. | |||||
| If one tries to export a package root into a repository that already contains | |||||
| a package with a matching identifier, ``dds`` will issue an error. If the | |||||
| ``--replace`` flag is specified with ``sdist export``, then ``dds`` will | |||||
| forcibly replace the package in the local repository with a new copy. |
| Source Distributions | Source Distributions | ||||
| #################### | #################### | ||||
| A *source distribution* is ``dds``'s primary format for consuming and | |||||
| distributing packages. A source distribution, in essence, is a | |||||
| :ref:`package root <pkgs.pkg-root>` archive that contains only the files | |||||
| A *source distribution* (often abbreviated as "sdist") is ``dds``'s primary | |||||
| format for consuming and distributing packages. A source distribution, in | |||||
| essence, is a :ref:`package root <pkgs.pkg-root>` that contains only the files | |||||
| necessary for ``dds`` to reproduce the full build of all libraries in the | necessary for ``dds`` to reproduce the full build of all libraries in the | ||||
| package. The source distribution retains the directory structure of every | package. The source distribution retains the directory structure of every | ||||
| :ref:`source root <pkgs.source-root>` of the original package, and thus retains | :ref:`source root <pkgs.source-root>` of the original package, and thus retains | ||||
| Generating a source distribution from a project directory is done with the | Generating a source distribution from a project directory is done with the | ||||
| ``sdist`` subcommand:: | ``sdist`` subcommand:: | ||||
| > dds sdist create | |||||
| > dds pkg create | |||||
| The above command can be executed within a package root, and the result will be | The above command can be executed within a package root, and the result will be | ||||
| a gzip'd tar archive that reproduces the package's filesystem structure, but | a gzip'd tar archive that reproduces the package's filesystem structure, but | ||||
| of that package. | of that package. | ||||
| The ``--project=<dir>`` flag can be provided to override the directory that | The ``--project=<dir>`` flag can be provided to override the directory that | ||||
| ``dds`` will use as the package root. The default is the working directory of | |||||
| the project. | |||||
| ``dds`` will use as the package root. The default is the current working | |||||
| directory. | |||||
| The ``--out=<path>`` flag can be provided to override the destination path of | The ``--out=<path>`` flag can be provided to override the destination path of | ||||
| the archive. The path should not name an existing file or directory. By default, | the archive. The path should not name an existing file or directory. By default, | ||||
| directory. | directory. | ||||
| Importing a Source Ditsribution | |||||
| .. _sdist.import: | |||||
| Importing a Source Distribution | |||||
| ******************************* | ******************************* | ||||
| Given a source distribution archive, one can import the package into the local | Given a source distribution archive, one can import the package into the local | ||||
| repository with a single command:: | |||||
| package cache with a single command:: | |||||
| > dds pkg import some-package@1.2.3.tar.gz | |||||
| You can also specify an HTTP or HTTPS URL to download a source distribution | |||||
| archive to import without downloading it separately:: | |||||
| > dds pkg import https://example.org/path/to/sdist.tar.gz | |||||
| Alternatively, if a directory correctly models a source distribution, then | |||||
| that directory can be imported in the same manner:: | |||||
| > dds pkg import /path/to/some/project | |||||
| Importing a package will allow projects on the system to use the imported | |||||
| package as a dependency. | |||||
| .. note:: | |||||
| > dds repo import some-package@1.2.3.tar.gz | |||||
| If one tries to import a package root into the cache that already contains a | |||||
| package with a matching identifier, ``dds`` will issue an error. This | |||||
| behavior can be overridden by providing ``--if-exists=replace`` on the | |||||
| command-line. |
| Specify *additional* link options to use when linking executables. | Specify *additional* link options to use when linking executables. | ||||
| .. note:: | |||||
| ``dds`` does not invoke the linker directly, but instead invokes the | |||||
| compiler with the appropriate flags to perform linking. If you need to pass | |||||
| flags directly to the linker, you will need to use the compiler's options to | |||||
| direct flags through to the linker. On GNU-style, this is | |||||
| ``-Wl,<linker-option>``. With MSVC, a separate flag ``/LINK`` must be | |||||
| specified, and all following options are passed to the underlying | |||||
| ``link.exe``. | |||||
| ``optimize`` | ``optimize`` | ||||
| ------------ | ------------ |
| Now running ``dds build`` will print more output that Catch has generated as | Now running ``dds build`` will print more output that Catch has generated as | ||||
| part of test execution, and we can see the reason for the failing test:: | part of test execution, and we can see the reason for the failing test:: | ||||
| [16:41:45] [error] Test <root>/_build/test/hello/strings failed! Output: | |||||
| [error] Test <root>/_build/test/hello/strings failed! Output: | |||||
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | ||||
| strings is a Catch v2.10.2 host application. | strings is a Catch v2.10.2 host application. |
| .. code-block:: text | .. code-block:: text | ||||
| [12:55:25] [info ] [dds-hello] Link: hello-world | |||||
| [12:55:25] [info ] [dds-hello] Link: hello-world - 57ms | |||||
| [12:55:25] [error] Failed to link executable '<root>/_build/hello-world'. | |||||
| [info ] [dds-hello] Link: hello-world | |||||
| [info ] [dds-hello] Link: hello-world - 57ms | |||||
| [error] Failed to link executable '<root>/_build/hello-world'. | |||||
| ... | ... | ||||
| <additional lines follow> | <additional lines follow> | ||||
| Rebuilding the project will show no difference at the moment. | Rebuilding the project will show no difference at the moment. | ||||
| .. note:: | |||||
| You may also use a ``.jsonc`` or ``.json`` file extension. ``dds`` will | |||||
| search for all of these files, but they will all be parsed as JSON5. | |||||
| .. seealso:: | .. seealso:: | ||||
| Creating a single application executable is fine and all, but what if we | Creating a single application executable is fine and all, but what if we | ||||
| want to create libraries? See the next page: :doc:`hello-lib` | want to create libraries? See the next page: :doc:`hello-lib` |