``dds`` has been designed from the very beginning as an extremely opinionated | ``dds`` has been designed from the very beginning as an extremely opinionated | ||||
hybrid *build system* and *package manager*. Unlike most build systems however, | hybrid *build system* and *package manager*. Unlike most build systems however, | ||||
``dds`` has a hyper-specific focus on a particular aspect of software | |||||
development: C and C++ libraries. | |||||
``dds`` has a strong focus on a particular aspect of software development: C and | |||||
C++ libraries. | |||||
This may sound pointless, right? Libraries are useless unless we can use them | This may sound pointless, right? Libraries are useless unless we can use them | ||||
to build applications! | to build applications! | ||||
Indeed, applications *are* essential, but that is "not our job" with ``dds``. | |||||
Indeed, applications *are* essential, and ``dds`` is able to build those as | |||||
well. | |||||
Another design decision is that ``dds`` is built to be driven by automated | Another design decision is that ``dds`` is built to be driven by automated | ||||
tools as well as humans. ``dds`` is not designed to entirely replace existing | tools as well as humans. ``dds`` is not designed to entirely replace existing | ||||
Despite the vast amount of work put into build systems and tooling, virtually | Despite the vast amount of work put into build systems and tooling, virtually | ||||
all developers are using them *incorrectly* and/or *dangerously* without | all developers are using them *incorrectly* and/or *dangerously* without | ||||
realizing it. Despite this work, we seem to be a great distance from a unified | |||||
library package distribution and consumption mechanism. | |||||
realizing it, and we seem to be still a great distance from a unified library | |||||
package distribution and consumption mechanism. | |||||
Tabula Rasa | Tabula Rasa | ||||
*customizability* in favor of *simplicity* and *ease*. | *customizability* in favor of *simplicity* and *ease*. | ||||
``dds`` takes a look at what is needed to build and develop *libraries* and | ``dds`` takes a look at what is needed to build and develop *libraries* and | ||||
hyper-optimizes for that use case. It is also built with a very strong, very | |||||
optimizes for that use case. It is also built with a very strong, very | |||||
opinionated idea of *how* libraries should be constructed and used. These | opinionated idea of *how* libraries should be constructed and used. These | ||||
prescriptions are not at all arbitrary, though. They are built upon the | prescriptions are not at all arbitrary, though. They are built upon the | ||||
observations of the strengths and weaknesses of build systems in use throughout | observations of the strengths and weaknesses of build systems in use throughout | ||||
``dds`` takes a massive divergence at this point. One project using ``dds`` as | ``dds`` takes a massive divergence at this point. One project using ``dds`` as | ||||
their build system has a nearly identical build process to every other project | their build system has a nearly identical build process to every other project | ||||
using ``dds``. Simply running :code:`dds build -t <toolchain>` should be enough | |||||
using ``dds``. Simply running ``dds build`` should be enough | |||||
to build *any* ``dds`` project. | to build *any* ``dds`` project. | ||||
In order to reach this uniformity and simplicity, ``dds`` drops almost all | In order to reach this uniformity and simplicity, ``dds`` drops almost all | ||||
aspects of project-by-project customizability. Instead, ``dds`` affords the | aspects of project-by-project customizability. Instead, ``dds`` affords the | ||||
developer a contract: | developer a contract: | ||||
If you play by my rules, you get to play in my space. | |||||
If you play by the rules, you get to play in this space. | |||||
.. _design.rules: | .. _design.rules: | ||||
.. _design.rules.not-apps: | .. _design.rules.not-apps: | ||||
``dds`` Is not Made for Complex Applications | ``dds`` Is not Made for Complex Applications | ||||
=============================================== | |||||
============================================ | |||||
Alright, this one isn't a "rule" as much as a recommendation: If you are | Alright, this one isn't a "rule" as much as a recommendation: If you are | ||||
building an application that *needs* some build process functionality that | building an application that *needs* some build process functionality that | ||||
customization features to permit the rules to be bent arbitrarily: Read | customization features to permit the rules to be bent arbitrarily: Read | ||||
on. | on. | ||||
``dds`` contains a minimal amount of functionality for building simple | |||||
applications, but it is certainly not its primary purpose. | |||||
``dds`` *does* contain functionality for building applications, but they must | |||||
also play by the rules. | |||||
If you want to build a complex application with ``dds`` that uses lots of | |||||
platform-specific sources, code generation, and conditional components, a good | |||||
option is to use an external build script that prepares the project tree before | |||||
invoking ``dds``. | |||||
.. _design.rules.change: | .. _design.rules.change: | ||||
*Your* Code Should Be Changed Before ``dds`` Should Be Changed | |||||
================================================================= | |||||
*Your Code* Should Be Changed Before ``dds`` Should Be Changed | |||||
============================================================== | |||||
The wording of this rule means that the onus is on the library developer to | |||||
meet the expectations that ``dds`` prescribes in order to make the build | |||||
work. | |||||
The wording of this rule means that the onus is on the developer to meet the | |||||
expectations that ``dds`` prescribes in order to make the build work. | |||||
If your library meets all the requirements outlined in this document but you | |||||
still find trouble in making your build work, this is grounds for change in | |||||
``dds``, either in clarifying the rules or tweaking ``dds`` functionality. | |||||
If your project meets all the requirements outlined in this document but you | |||||
still find trouble in making your build work, or if you *cannot* see *any* | |||||
possible way for your project to be built by ``dds`` regardless of what changes | |||||
you make, then it this is grounds for change in ``dds``, either in clarifying | |||||
the rules or tweaking ``dds`` functionality | |||||
.. _design.rules.layout: | .. _design.rules.layout: | ||||
All Code Must Be in Place Before Building | All Code Must Be in Place Before Building | ||||
========================================= | ========================================= | ||||
``dds`` does not provide code-generation functionality. Instead, any | |||||
generated code should be generated and committed to the repository to be only | |||||
ever modified through such generation scripts. | |||||
``dds`` does not provide code-generation functionality. Instead, any generated | |||||
code should be generated by separate build steps before ``dds`` is executed. | |||||
.. _design.rules.one-binary-per-src: | .. _design.rules.one-binary-per-src: | ||||
===================================== | ===================================== | ||||
Only ``src/`` and ``include/`` will ever be used as the basis for header | Only ``src/`` and ``include/`` will ever be used as the basis for header | ||||
resolution while building a library, so all ``#include`` directives should be | |||||
resolution while building a project, so all ``#include`` directives should be | |||||
relative to those directories. Refer to :ref:`pkg.source-root`. | relative to those directories. Refer to :ref:`pkg.source-root`. | ||||
All Files Compile with the Same Options | All Files Compile with the Same Options | ||||
======================================= | ======================================= | ||||
When DDS compiles a library, every source file will be compiled with an | |||||
When DDS compiles a project, every source file will be compiled with an | |||||
identical set of options. Additionally, when DDS compiles a dependency tree, | identical set of options. Additionally, when DDS compiles a dependency tree, | ||||
every library in that dependency tree will be compiled with an identical set of | every library in that dependency tree will be compiled with an identical set of | ||||
options. Refer to the :doc:`guide/toolchains` page for more information. | options. Refer to the :doc:`guide/toolchains` page for more information. |
unavailable in most setups). Using a Git tag is strongly recommended. | unavailable in most setups). Using a Git tag is strongly recommended. | ||||
.. seealso:: | .. seealso:: | ||||
Refer to the documentation on :doc:`/guide/catalog`. | |||||
Refer to the documentation on :doc:`/guide/remote-pkgs`. |
Error: Invalid catalog JSON | |||||
########################### | |||||
This error occurs when the JSON data given to import into the package catalog | |||||
is in some way invalid. Refer to the catalog documentation for a description of | |||||
the proper JSON format. | |||||
.. seealso:: | |||||
:ref:`catalog.adding` |
Error: A repository filesystem transformation is invalid | |||||
######################################################## | |||||
In ``dds``, a catalog entry can have a list of attached "transforms" that will | |||||
be applies to the root directory of the package before ``dds`` tries to build | |||||
and use it. | |||||
.. seealso:: | |||||
For information on the shape and purpose of transforms, refer to | |||||
:ref:`catalog.fs-transform` on the :doc:`/guide/catalog` page. |
when it is requested. | when it is requested. | ||||
If such information is not provided, ``dds`` will issue an error. | If such information is not provided, ``dds`` will issue an error. | ||||
.. seealso:: :ref:`catalog.adding`. |
It is possible that the intended package *does exist* but that the spelling of | It is possible that the intended package *does exist* but that the spelling of | ||||
the package name or version number is incorrect. Firstly, check your spelling | the package name or version number is incorrect. Firstly, check your spelling | ||||
and that the version number you have requested is correct. | and that the version number you have requested is correct. | ||||
In another case, it is possible that the package *exists somewhere*, but has | |||||
not been loaded into the local catalog. As of this writing, ``dds`` does not | |||||
automatically maintain the catalog against a central package repository, so | |||||
package entries must be loaded and imported manually. If you believe this to be | |||||
the case, refer to the section on the :doc:`/guide/catalog`, especially the | |||||
section :ref:`catalog.adding`. |
whatever exists there before creating the source distribution. | whatever exists there before creating the source distribution. | ||||
.. warning:: | .. warning:: | ||||
When using ``dds sdist create`` with the ``--out <path>`` parameter, the | |||||
When using ``dds pkg create`` with the ``--out <path>`` parameter, the | |||||
``<path>`` given **is not the directory in which to place the source | ``<path>`` given **is not the directory in which to place the source | ||||
distribution, but the filepath to the source distribution itself**! | distribution, but the filepath to the source distribution itself**! | ||||
distribution in that directory, **the following command is incorrect**:: | distribution in that directory, **the following command is incorrect**:: | ||||
# Do not do this: | # Do not do this: | ||||
dds sdist create --out foo/ | |||||
dds pkg create --out foo/ | |||||
If you pass ``--replace`` to the above command, ``dds`` will **destroy the | If you pass ``--replace`` to the above command, ``dds`` will **destroy the | ||||
existing directory** and replace it with the source distribution! | existing directory** and replace it with the source distribution! | ||||
You **must** provide the full path to the source distribution:: | You **must** provide the full path to the source distribution:: | ||||
# Do this: | # Do this: | ||||
dds sdist create --out foo/my-project.dsd | |||||
dds pkg create --out foo/my-project.tar.gz |
Building and Using ``dds`` in Another Build System | |||||
################################################## | |||||
One of ``dds``'s primary goals is to inter-operate with other build systems | |||||
cleanly. One of ``dds``'s primary outputs is *libman* package indices. These | |||||
package indices can be imported into other build systems that support the | |||||
`libman`_ format. (At the time of writing there is a CMake module which can do | |||||
the import, but other build systems are planned.) | |||||
.. _libman: https://api.csswg.org/bikeshed/?force=1&url=https://raw.githubusercontent.com/vector-of-bool/libman/develop/data/spec.bs | |||||
.. _PMM: https://github.com/vector-of-bool/PMM | |||||
.. _CMakeCM: https://github.com/vector-of-bool/CMakeCM | |||||
.. _lm-cmake: https://raw.githubusercontent.com/vector-of-bool/libman/develop/cmake/libman.cmake | |||||
.. _build-deps.gen-libman: | |||||
Generating a libman Index | |||||
************************* | |||||
Importing libman packages into a build system requires that we have a libman | |||||
index generated on the filesystem. **This index is not generated globally**: It | |||||
is generated on a per-build basis as part of the build setup. The index will | |||||
describe in build-system-agnostic terms how to include a set of packages and | |||||
libraries as part of a build. | |||||
``dds`` has first-class support for generating this index. The ``build-deps`` | |||||
subcommand of ``dds`` will download and build a set of dependencies, and places | |||||
an ``INDEX.lmi`` file that can be used to import the built results. | |||||
Declaring Dependencies | |||||
====================== | |||||
``dds build-deps`` accepts a list of dependency statements as command line | |||||
arguments, but it may be useful to specify those requirements in a file. | |||||
``dds build-deps`` accepts a JSON5 file describing the dependencies of a | |||||
project as well. This file is similar to a very stripped-down version of a | |||||
``dds`` :ref:`package manifest <pkgs.pkgs>`, and only includes the ``depends`` | |||||
key. (The presence of any other key is an error.) | |||||
Here is a simple dependencies file that declares a single requirement: | |||||
.. code-block:: js | |||||
:caption: ``dependencies.json5`` | |||||
{ | |||||
depends: [ | |||||
'neo-sqlite3^0.2.0', | |||||
] | |||||
} | |||||
Building Dependencies and the Index | |||||
=================================== | |||||
We can invoke ``dds build-deps`` and give it the path to this file: | |||||
.. code-block:: bash | |||||
$ dds build-deps --deps-file dependencies.json5 | |||||
When finished, ``dds`` will write the build results into a subdirectory called | |||||
``_deps`` and generate a file named ``INDEX.lmi``. This file is ready to be | |||||
imported into any build system that can understand libman files. | |||||
.. note:: | |||||
The output directory and index filepath can be controlled with the | |||||
``--out`` and ``--lmi-path`` flags, respectively. | |||||
Importing an Index: CMake | |||||
************************* | |||||
.. highlight:: cmake | |||||
.. note:: | |||||
This section discusses how to import ``INDEX.lmi`` into CMake, but ``dds`` | |||||
also has built-in support for generating a CMake targets file. See | |||||
:doc:`cmake` and :ref:`cmake.pmm` for even simpler integration steps. | |||||
Supposed that we've generated a libman index and set of packages, and we want to | |||||
import them into CMake. CMake doesn't know how to do this natively, but there | |||||
exists a single-file module for CMake that allows CMake to import libraries from | |||||
libman indices without any additional work. | |||||
The module is not shipped with CMake, but is available online as a single | |||||
stand-alone file. The `libman.cmake <lm-cmake_>`_ file can be downloaded and | |||||
added to a project directly, or it can be obtained automatically through a | |||||
CMake tool like `PMM`_ (recommended). | |||||
Getting ``libman.cmake`` via PMM | |||||
================================ | |||||
Refer to the ``README.md`` file in `the PMM repo <PMM_>`_ for information on how | |||||
to get PMM into your CMake project. In short, download and place the | |||||
``pmm.cmake`` file in your repository, and ``include()`` the file near the top | |||||
of your ``CMakeLists.txt``:: | |||||
include(pmm.cmake) | |||||
Once it has been included, you can call the ``pmm()`` function. To obtain | |||||
*libman*, we need to start by enabling `CMakeCM`_:: | |||||
pmm(CMakeCM ROLLING) | |||||
.. warning:: | |||||
It is not recommended to use the ``ROLLING`` mode, but it is the easiest to | |||||
use when getting started. For reproducible and reliable builds, you should | |||||
pin your CMakeCM version using the ``FROM <url>`` argument. | |||||
Enabling CMakeCM will make available all of the CMake modules available in `the | |||||
CMakeCM repository <CMakeCM_>`_, which includes `libman.cmake <lm-cmake_>`_. | |||||
After the call to ``pmm()``, simply ``include()`` the ``libman`` module:: | |||||
include(libman) | |||||
That's it! The only function from the module that we will care about for now | |||||
is the ``import_packages()`` function. | |||||
Importing Our Dependencies' Packages | |||||
==================================== | |||||
To import a package from a libman tree, we need only know the *name* of the | |||||
package we wish to import. In our example case above, we depend on | |||||
``neo-sqlite3``, so we simply call the libman-CMake function | |||||
``import_packages()`` with that package name:: | |||||
import_packages("neo-sqlite3") | |||||
You'll note that we don't request any particular version of the package: All | |||||
versioning resolution is handled by ``dds``. You'll also note that we don't | |||||
need to specify our transitive dependencies: This is handled by the libman | |||||
index that was generated by ``dds``: It will automatically ``import_packages()`` | |||||
any of the transitive dependencies required. | |||||
More than one package name can be provided to a single call to | |||||
``import_packages()``, and ``import_packages()`` may be called multiple times | |||||
within a CMake project. | |||||
Using Our Dependencies' Libraries | |||||
================================= | |||||
Like with ``dds``, CMake wants us to explicitly declare how our build targets | |||||
*use* other libraries. When we import a package from a libman index, the | |||||
import will generate CMake ``IMPORTED`` targets that can be linked against. | |||||
In ``dds`` and in libman, a library is identified by a combination of | |||||
*namespace* and *name*, joined together with a slash ``/`` character. This | |||||
*qualified name* of a library is decided by the original package author, and | |||||
should be documented. In the case of ``neo-sqlite3``, the only library is | |||||
``neo/sqlite3``. | |||||
When the libman CMake module imports a library, it creates a qualified name | |||||
using a double-colon "``::``" instead of a slash. As such, our ``neo/sqlite3`` | |||||
is imported in CMake as ``neo::sqlite3``. We can link against it as we would | |||||
with any other target:: | |||||
add_executable(my-application app.cpp) | |||||
target_link_libraries(my-application PRIVATE neo::sqlite3) | |||||
Altogether, here is the final CMake file: | |||||
.. code-block:: | |||||
:caption: ``CMakeLists.txt`` | |||||
:linenos: | |||||
cmake_minimum_required(VERSION 3.15) | |||||
project(MyApplication VERSION 1.0.0) | |||||
include(pmm.cmake) | |||||
pmm(CMakeCM ROLLING) | |||||
include(libman) | |||||
import_packages("neo-sqlite3") | |||||
add_executable(my-application app.cpp) | |||||
target_link_libraries(my-application PRIVATE neo::sqlite3) |
The Package Catalog | |||||
################### | |||||
``dds`` stores a catalog of available packages, along with their dependency | |||||
statements and information about how a source distribution thereof may be | |||||
maintained. | |||||
Viewing Catalog Contents | |||||
************************ | |||||
The default catalog database is stored in a user-local location, and the | |||||
package IDs available can be listed with ``dds catalog list``. This will only | |||||
list the IDs of the packages, but none of the additional metadata about them. | |||||
.. _catalog.adding: | |||||
Adding Packages to the Catalog | |||||
****************************** | |||||
The ``dds catalog import`` supports a ``--json`` flag that specifies a JSON5 | |||||
file from which catalog entries will be generated. | |||||
.. note:: | |||||
The ``--json`` flag can be passed more than once to import multiple JSON | |||||
files at once. | |||||
The JSON file has the following structure: | |||||
.. code-block:: javascript | |||||
{ | |||||
// Import version spec. | |||||
version: 1, | |||||
// Packages section | |||||
packages: { | |||||
// Subkeys are package names | |||||
"acme-gadgets": { | |||||
// Keys within the package names are the versions that are | |||||
// available for each package. | |||||
"0.4.2": { | |||||
// `depends` is an array of dependency statements for this | |||||
// particular version of the package. (Optional) | |||||
depends: [ | |||||
"acme-widgets^1.4.1" | |||||
], | |||||
// `description` is an attribute to give a string to describe | |||||
// the package. (Optional) | |||||
description: "A collection of useful gadgets.", | |||||
// Specify the Git remote information | |||||
git: { | |||||
// `url` and `ref` are required. | |||||
url: "http://example.com/git/repo/acme-gadgets.git", | |||||
ref: "v0.4.2-stable", | |||||
// The `auto-lib` is optional, to specify an automatic | |||||
// library name/namespace pair to generate for the | |||||
// root library | |||||
"auto-lib": "Acme/Gadgets", | |||||
// List of filesystem transformations to apply to the repository | |||||
// (optional) | |||||
transform: [ | |||||
// ... (see below) ... | |||||
] | |||||
} | |||||
} | |||||
} | |||||
} | |||||
} | |||||
.. _catalog.fs-transform: | |||||
Filesystem Transformations | |||||
************************** | |||||
.. note:: | |||||
Filesystem transformations is a transitional feature that is likely to be | |||||
removed in a future release, and replaced with a more robust system when | |||||
``dds`` has a better way to download packages. Its aim is to allow ``dds`` | |||||
projects to use existing libraries that might not meet the layout | |||||
requirements that ``dds`` imposes, but can otherwise be consumed by ``dds`` | |||||
with a few tweaks. | |||||
A catalog entry can have a set of filesystem transformations attached to its | |||||
remote information (e.g. the ``git`` property). When ``dds`` is obtaining a | |||||
copy of the code for the package, it will apply the associated transformations | |||||
to the filesystem based in the directory of the downloaded/cloned directory. In | |||||
this way, ``dds`` can effectively "patch" the filesystem structure of a project | |||||
arbitrarily. This allows many software projects to be imported into ``dds`` | |||||
without needing to patch/fork the original project to support the required | |||||
filesystem structure. | |||||
.. important:: | |||||
While ``dds`` allows you to patch directories downloaded via the catalog, a | |||||
native ``dds`` project must still follow the layout rules. | |||||
The intention of filesystem transformations is to act as a "bridge" that will allow ``dds`` projects to more easily utilize existing libraries. | |||||
Available Transformations | |||||
========================= | |||||
At time of writing, there are five transformations available to catalog entries: | |||||
``copy`` and ``move`` | |||||
Copies or moves a set of files/directories from one location to another. Allows the following options: | |||||
- ``from`` - The path from which to copy/move. **Required** | |||||
- ``to`` - The destination path for the copy/move. **Required** | |||||
- ``include`` - A list of globbing expressions for files to copy/move. If | |||||
omitted, then all files will be included. | |||||
- ``exclude`` - A list of globbing expressions of files to exclude from the | |||||
copy/move. If omitted, then no files will be excluded. **If both** ``include`` and ``exclude`` are provided, ``include`` will be checked *before* ``exclude``. | |||||
- ``strip-components`` - A positive integer (or zero, the default). When the | |||||
``from`` path identifies a directory, its contents will be copied/moved | |||||
into the destination and maintain their relative path from the source path as their relative path within the destination. If ``strip-components`` is set to an integer ``N``, then the first ``N`` path components of that relative path will be removed when copying/moving the files in a directory. If a file's relative path has less than ``N`` components, then that file will be excluded from the ``copy/move`` operation. | |||||
``remove`` | |||||
Delete files and directories from the package source. Has the following options: | |||||
- ``path`` - The path of the file/directory to remove. **Required** | |||||
- ``only-matching`` - A list of globbing expressions for files to remove. If omitted and the path is a directory, then the entire directory will be deleted. If at least one pattern is provided, then directories will be left intact and only non-directory files will be removed. If ``path`` names a non-directory file, then this option has no effect. | |||||
``write`` | |||||
Write the contents of a string to a file in the package source. Has the following options: | |||||
- ``path`` - The path of the file to write. **Required** | |||||
- ``content`` - A string that will be written to the file. **Required** | |||||
If the file exists and is not a directory, the file will be replaced. If the | |||||
path names an existing directory, an error will be generated. | |||||
``edit`` | |||||
Modifies the contents of the files in the package. | |||||
- ``path`` - Path to the file to edit. **Required** | |||||
- ``edits`` - An array of edit objects, applied in order, with the following | |||||
keys: | |||||
- ``kind`` - One of ``insert`` or ``delete`` to insert/delete lines, | |||||
respectively. **Required** | |||||
- ``line`` - The line at which to perform the insert/delete. The first line | |||||
of the file is line one, *not* line zero. **Required** | |||||
- ``content`` - For ``insert``, the string content to insert into the file. | |||||
A newline will be appended after the content has been inserted. | |||||
Transformations are added as a JSON array to the JSON object that specifies | |||||
the remote information for the package. Each element of the array is an | |||||
object, with one or more of the keys listed above. If an object features more | |||||
than one of the above keys, they are applied in the same order as they have | |||||
been listed. | |||||
Example: Crypto++ | |||||
================= | |||||
The following catalog entry will build and import `Crypto++`_ for use by a | |||||
``dds`` project. This uses the unmodified Crypto++ repository, which ``dds`` | |||||
doesn't know how to build immediately. With some simple moving of files, we | |||||
end up with something ``dds`` can build directly: | |||||
.. code-block:: javascript | |||||
"cryptopp": { | |||||
"8.2.0": { | |||||
"git": { | |||||
"url": "https://github.com/weidai11/cryptopp.git", | |||||
"ref": "CRYPTOPP_8_2_0", | |||||
"auto-lib": "cryptopp/cryptopp", | |||||
"transform": [ | |||||
{ | |||||
// Crypto++ has no source directories at all, and everything lives | |||||
// at the top level. No good for dds. | |||||
// | |||||
// Clients are expected to #include files with a `cryptopp/` prefix, | |||||
// so we need to move the files around so that they match the | |||||
// expected layout: | |||||
"move": { | |||||
// Move from the root of the repo: | |||||
"from": ".", | |||||
// Move files *into* `src/cryptopp` | |||||
"to": "src/cryptopp", | |||||
// Only move the C++ sources and headers: | |||||
"include": [ | |||||
"*.c", | |||||
"*.cpp", | |||||
"*.h" | |||||
] | |||||
} | |||||
} | |||||
] | |||||
} | |||||
} | |||||
} | |||||
Example: libsodium | |||||
================== | |||||
For example, this catalog entry will build and import `libsodium`_ for use in | |||||
a ``dds`` project. This uses the upstream libsodium repository, which does not | |||||
meet the layout requirements needed by ``dds``. With a few simple | |||||
transformations, we can allow ``dds`` to build and consume libsodium | |||||
successfully: | |||||
.. code-block:: javascript | |||||
"libsodium": { | |||||
"1.0.18": { | |||||
"git": { | |||||
"url": "https://github.com/jedisct1/libsodium.git", | |||||
"ref": "1.0.18", | |||||
"auto-lib": "sodium/sodium", | |||||
/// Make libsodium look as dds expects of a project. | |||||
"transform": [ | |||||
// libsodium has a `src` directory, but it does not look how dds | |||||
// expects it to. The public `#include` root of libsodium lives in | |||||
// a nested subdirectory of `src/` | |||||
{ | |||||
"move": { | |||||
// Move the public header root out from that nested subdirectory | |||||
"from": "src/libsodium/include", | |||||
// Put it at `include/` in the top-level | |||||
"to": "include/" | |||||
} | |||||
}, | |||||
// libsodium has some files whose contents are generated by a | |||||
// configure script. For demonstration purposes, we don't need most | |||||
// of them, and we can just swipe an existing pre-configured file | |||||
// that is already in the source repository and put it into the | |||||
// public header root. | |||||
{ | |||||
"copy": { | |||||
// Generated version header committed to the repository: | |||||
"from": "builds/msvc/version.h", | |||||
// Put it where the configure script would put it: | |||||
"to": "include/sodium/version.h" | |||||
} | |||||
}, | |||||
// The subdirectory `src/libsodium/` is no good. It now acts as an | |||||
// unnecessary layer of indirection. We want `src/` to be the root. | |||||
// We can just "lift" the subdirectory: | |||||
{ | |||||
// Up we go: | |||||
"move": { | |||||
"from": "src/libsodium", | |||||
"to": "src/" | |||||
}, | |||||
// Delete the now-unused subdirectory: | |||||
"remove": { | |||||
"path": "src/libsodium" | |||||
} | |||||
}, | |||||
// Lastly, libsodium's source files expect to resolve their header | |||||
// paths differently than they expect of their clients (Bad!!!). | |||||
// Fortunately, we can do a hack to allow the files in `src/` to | |||||
// resolve its headers. The source files use #include as if the | |||||
// header root was `include/sodium/`, rather than `include/`. | |||||
// To work around this, generate a copy of each header file in the | |||||
// source root, but remove the leading path element. | |||||
// Because we have a separate `include/` and `src/` directory, dds | |||||
// will only expose the `include/` directory to clients, and the | |||||
// header copies in `src/` are not externally visible. | |||||
// | |||||
// For example, the `include/sodium/version.h` file is visible to | |||||
// clients as `sodium/version.h`, but libsodium itself tries to | |||||
// include it as `version.h` within its source files. When we copy | |||||
// from `include/`, we grab the relative path to `sodium/version.h`, | |||||
// strip the leading components to get `version.h`, and then join that | |||||
// path with the `to` path to generate the full destination at | |||||
// `src/version.h` | |||||
{ | |||||
"copy": { | |||||
"from": "include/", | |||||
"to": "src/", | |||||
"strip-components": 1 | |||||
} | |||||
} | |||||
] | |||||
} | |||||
} | |||||
} | |||||
.. _libsodium: https://doc.libsodium.org/ | |||||
.. _Crypto++: https://cryptopp.com/ |
.. highlight:: cmake | .. highlight:: cmake | ||||
Using ``dds`` Packages in a CMake Project | |||||
######################################### | |||||
Easy Mode: Using ``dds`` in a CMake Project | |||||
########################################### | |||||
One of ``dds``'s primary goals is to inter-operate with other build systems | One of ``dds``'s primary goals is to inter-operate with other build systems | ||||
cleanly. One of ``dds``'s primary outputs is *libman* package indices. These | |||||
package indices can be imported into other build systems that support the | |||||
*libman* format. | |||||
cleanly. Because of CMakes ubiquity, ``dds`` includes built-in support for | |||||
emitting files that can be imported into CMake. | |||||
.. note:: | |||||
``dds`` doesn't (yet) have a ready-made central repository of packages that | |||||
can be downloaded. You'll need to populate the local package catalog | |||||
appropriately. The default catalog file contains a limited set of useful | |||||
packages, but you may wish to add more for yourself. | |||||
.. seealso:: Refer to :doc:`catalog` for information about remote packages. | |||||
.. _PMM: https://github.com/vector-of-bool/PMM | |||||
.. _CMakeCM: https://github.com/vector-of-bool/CMakeCM | |||||
.. _lm-cmake: https://raw.githubusercontent.com/vector-of-bool/libman/develop/cmake/libman.cmake | |||||
Generating a libman Index | |||||
************************* | |||||
Importing libman packages into a build system requires that we have a libman | |||||
index generated on the filesystem. **This index is not generated globally**: It | |||||
is generated on a per-build basis as part of the build setup. The index will | |||||
describe in build-system-agnostic terms how to include a set of packages and | |||||
libraries as part of a build. | |||||
``dds`` has first-class support for generating this index. The ``build-deps`` | |||||
subcommand of ``dds`` will download and build a set of dependencies, and places | |||||
an ``INDEX.lmi`` file that can be used to import the built results. | |||||
Declaring Dependencies | |||||
====================== | |||||
``dds build-deps`` accepts a list of dependency statements as commnad line | |||||
arguments, but it may be useful to specify those requirements in a file. | |||||
``dds build-deps`` accepts a JSON5 file describing the dependencies of a | |||||
project as well. This file is similar to a very stripped-down version of a | |||||
``dds`` :ref:`package manifest <pkgs.pkgs>`, and only includes the ``depends`` | |||||
key. (The presence of any other key is an error.) | |||||
Here is a simple dependencies file that declares a single requirement: | |||||
.. code-block:: js | |||||
:caption: ``dependencies.json5`` | |||||
{ | |||||
depends: [ | |||||
'neo-sqlite3^0.2.0', | |||||
] | |||||
} | |||||
Building Dependencies and the Index | |||||
=================================== | |||||
We can invoke ``dds build-deps`` and give it the path to this file: | |||||
.. code-block:: bash | |||||
$ dds build-deps --deps dependencies.json5 | |||||
.. seealso:: | |||||
When finished, ``dds`` will write the build results into a subdirectory called | |||||
``_deps`` and generate a file named ``INDEX.lmi``. This file is ready to be | |||||
imported into any build system that can understand libman files (in our case, | |||||
CMake). | |||||
Before reading this page, be sure to read the :ref:`build-deps.gen-libman` | |||||
section of the :doc:`build-deps` page, which will discuss how to use the | |||||
``dds build-deps`` subcommand. | |||||
.. note:: | .. note:: | ||||
The output directory and index filepath can be controlled with the | |||||
``--out`` and ``--lmi-path`` flags, respectively. | |||||
We'll first look as *easy mode*, but there's also an *easiest mode* for a | |||||
one-line solution: :ref:`see below <cmake.pmm>`. | |||||
Importing into CMake | |||||
******************** | |||||
We've generated a libman index and set of packages, and we want to import | |||||
them into CMake. CMake doesn't know how to do this natively, but there exists a | |||||
single-file module for CMake that allows CMake to import libraries from libman | |||||
indices without any additional work. | |||||
The module is not shipped with CMake, but is available online as a single | |||||
stand-alone file. The `libman.cmake <lm-cmake_>`_ file can be downloaded and | |||||
added to a project directly, or it can be obtained automatically through a | |||||
CMake tool like `PMM`_ (recommended). | |||||
Enabling *libman* Support in CMake via PMM | |||||
========================================== | |||||
Refer to the ``README.md`` file in `the PMM repo <PMM_>`_ for information on how | |||||
to get PMM into your CMake project. In short, download and place the | |||||
``pmm.cmake`` file in your repository, and ``include()`` the file near the top | |||||
of your ``CMakeLists.txt``:: | |||||
include(pmm.cmake) | |||||
Once it has been included, you can call the ``pmm()`` function. To obtain | |||||
*libman*, we need to start by enabling `CMakeCM`_:: | |||||
pmm(CMakeCM ROLLING) | |||||
.. warning:: | |||||
It is not recommended to use the ``ROLLING`` mode, but it is the easiest to | |||||
use when getting started. For reproducible and reliable builds, you should | |||||
pin your CMakeCM version using the ``FROM <url>`` argument. | |||||
Enabling CMakeCM will make available all of the CMake modules available in `the | |||||
CMakeCM repository <CMakeCM_>`_, which includes `libman.cmake <lm-cmake_>`_. | |||||
.. _PMM: https://github.com/vector-of-bool/PMM | |||||
After the call to ``pmm()``, simply ``include()`` the ``libman`` module:: | |||||
include(libman) | |||||
Generating a CMake Import File | |||||
****************************** | |||||
That's it! The only function from the module that we will care about for now | |||||
is the ``import_packages()`` function. | |||||
``build-deps`` accepts an ``--lmi-path`` argument, but also accepts a | |||||
``--cmake=<path>`` argument that serves a similar purpose: It will write a CMake | |||||
file to ``<path>`` that can be ``include()``'d into a CMake project: | |||||
.. code-block:: bash | |||||
Importing Our Dependencies' Packages | |||||
==================================== | |||||
$ dds build-deps "neo-sqlite3^0.2.0" --cmake=deps.cmake | |||||
To import a package from a libman tree, we need only know the *name* of the | |||||
package we wish to import. In our example case above, we depend on | |||||
``neo-sqlite3``, so we simply call the libman-CMake function | |||||
``import_packages()`` with that package name:: | |||||
This will write a file ``./deps.cmake`` that we can ``include()`` from a CMake | |||||
project, which will then expose the ``neo-sqlite3`` package as a set of imported | |||||
targets. | |||||
import_packages("neo-sqlite3") | |||||
You'll note that we don't request any particular version of the package: All | |||||
versioning resolution is handled by ``dds``. You'll also note that we don't | |||||
need to specify our transitive dependencies: This is handled by the libman | |||||
index that was generated by ``dds``: It will automatically ``import_packages()`` | |||||
any of the transitive dependencies required. | |||||
Using the CMake Import File | |||||
=========================== | |||||
Once we have generated the CMake import file using ``dds build-deps``, we can | |||||
simply import it in our ``CMakeLists.txt``:: | |||||
Using Our Dependencies' Libraries | |||||
================================= | |||||
include(deps.cmake) | |||||
Like with ``dds``, CMake wants us to explicitly declare how our build targets | Like with ``dds``, CMake wants us to explicitly declare how our build targets | ||||
*use* other libraries. When we import a package from a libman index, the | |||||
import will generate CMake ``IMPORTED`` targets that can be linked against. | |||||
*use* other libraries. When we ``include()`` the generated CMake file, it will | |||||
generate ``IMPORTED`` targets that can be linked against. | |||||
In ``dds`` and in libman, a library is identified by a combination of | |||||
In ``dds`` (and in libman), a library is identified by a combination of | |||||
*namespace* and *name*, joined together with a slash ``/`` character. This | *namespace* and *name*, joined together with a slash ``/`` character. This | ||||
*qualified name* of a library is decided by the original package author, and | *qualified name* of a library is decided by the original package author, and | ||||
should be documented. In the case of ``neo-sqlite3``, the only target is | |||||
should be documented. In the case of ``neo-sqlite3``, the only library is | |||||
``neo/sqlite3``. | ``neo/sqlite3``. | ||||
When the libman CMake module imports a library, it creates a qualified name | |||||
When the generated import file imports a library, it creates a qualified name | |||||
using a double-colon "``::``" instead of a slash. As such, our ``neo/sqlite3`` | using a double-colon "``::``" instead of a slash. As such, our ``neo/sqlite3`` | ||||
is imported in CMake as ``neo::sqlite3``. We can link against it as we would | is imported in CMake as ``neo::sqlite3``. We can link against it as we would | ||||
with any other target:: | with any other target:: | ||||
add_executable(my-application app.cpp) | |||||
target_link_libraries(my-application PRIVATE neo::sqlite3) | |||||
Altogether, here is the final CMake file: | |||||
add_executable(my-application app.cpp) | |||||
target_link_libraries(my-application PRIVATE neo::sqlite3) | |||||
.. code-block:: | |||||
:caption: ``CMakeLists.txt`` | |||||
:linenos: | |||||
cmake_minimum_required(VERSION 3.15) | |||||
project(MyApplication VERSION 1.0.0) | |||||
.. _cmake.pmm: | |||||
include(pmm.cmake) | |||||
pmm(CMakeCM ROLLING) | |||||
*Easiest* Mode: PMM Support | |||||
*************************** | |||||
include(libman) | |||||
import_packages("neo-sqlite3") | |||||
add_executable(my-application app.cpp) | |||||
target_link_libraries(my-application PRIVATE neo::sqlite3) | |||||
`PMM`_ is the *package package manager*, and can be used to control and access | |||||
package managers from within CMake scripts. This includes controlling ``dds``. | |||||
With PMM, we can automate all of the previous steps into a single line. | |||||
Refer to the ``README.md`` file in `the PMM repo <PMM>`_ for information on how | |||||
to get PMM into your CMake project. In short, download and place the | |||||
``pmm.cmake`` file in your repository, and ``include()`` the file near the top | |||||
of your ``CMakeLists.txt``:: | |||||
Additional PMM Support | |||||
********************** | |||||
include(pmm.cmake) | |||||
The ``pmm()`` function also supports ``dds`` directly, similar to ``CMakeCM`` | |||||
mode. This will automatically download a prebuilt ``dds`` for the host platform | |||||
and invoke ``dds build-deps`` in a single pass as part of CMake's configure | |||||
process. This is especially useful for a CI environment where you want to have | |||||
a stable ``dds`` version and always have your dependencies obtained | |||||
just-in-time. | |||||
The ``pmm()`` function also supports ``dds`` directly, and will automatically | |||||
download a prebuilt ``dds`` for the host platform and invoke ``dds build-deps`` | |||||
in a single pass as part of CMake's configure process. This is especially useful | |||||
for a CI environment where you want to have a stable ``dds`` version and always | |||||
have your dependencies obtained just-in-time. | |||||
To start, pass the ``DDS`` argument to ``pmm()`` to use it:: | To start, pass the ``DDS`` argument to ``pmm()`` to use it:: | ||||
pmm(DDS) | |||||
pmm(DDS) | |||||
.. note:: | .. note:: | ||||
The ``_deps`` directory and ``INDEX.lmi`` file will be placed in the CMake | |||||
build directory, out of the way of the rest of the project. | |||||
The ``_deps`` directory and generated CMake imports file will be placed in | |||||
the CMake build directory, out of the way of the rest of the project. | |||||
.. note:: | .. note:: | ||||
The version of ``dds`` that PMM downloads depends on the version of PMM | |||||
that is in use. | |||||
The version of ``dds`` that PMM downloads depends on the version of PMM | |||||
that is in use. | |||||
This alone won't do anything useful, because you'll need to tell it what | This alone won't do anything useful, because you'll need to tell it what | ||||
dependencies we want to install:: | dependencies we want to install:: | ||||
pmm(DDS DEP_FILES dependencies.json5) | |||||
pmm(DDS DEP_FILES dependencies.json5) | |||||
You can also list your dependencies as an inline string in your CMakeLists.txt | |||||
You can also list your dependencies as inline strings in your CMakeLists.txt | |||||
instead of a separate file:: | instead of a separate file:: | ||||
pmm(DDS DEPENDS neo-sqlite3^0.2.2) | |||||
pmm(DDS DEPENDS neo-sqlite3^0.2.2) | |||||
Since you'll probably want to be using ``libman.cmake`` at the same time, the | |||||
calls for ``CMakeCM`` and ``DDS`` can simply be combined. This is how our new | |||||
CMake project might look: | |||||
This invocation will run ``build-deps`` with the build options, generate a CMake | |||||
imports file, and immediately ``include()`` it to import the generated CMake | |||||
targets. ``pmm(DDS)`` will also generate a ``dds`` :doc:`toolchain <toolchains>` | |||||
based on the current CMake build environment, ensuring that the generated | |||||
packages have matching build options to the rest of the project. Refer to the | |||||
PMM README for more details. | |||||
.. code-block:: | .. code-block:: | ||||
:caption: ``CMakeLists.txt`` | |||||
:linenos: | |||||
cmake_minimum_required(VERSION 3.15) | |||||
project(MyApplication VERSION 1.0.0) | |||||
:caption: ``CMakeLists.txt`` | |||||
:linenos: | |||||
:emphasize-lines: 4,5 | |||||
include(pmm.cmake) | |||||
pmm(CMakeCM ROLLING | |||||
DDS DEPENDS neo-sqlite3^0.2.2 | |||||
) | |||||
cmake_minimum_required(VERSION 3.15) | |||||
project(MyApplication VERSION 1.0.0) | |||||
include(libman) | |||||
import_packages("neo-sqlite3") | |||||
include(pmm.cmake) | |||||
pmm(DDS DEPENDS neo-sqlite3^0.2.2) | |||||
add_executable(my-application app.cpp) | |||||
target_link_libraries(my-application PRIVATE neo::sqlite3) | |||||
add_executable(my-application app.cpp) | |||||
target_link_libraries(my-application PRIVATE neo::sqlite3) | |||||
This removes the requirement that we write a separate dependencies file, and we | This removes the requirement that we write a separate dependencies file, and we | ||||
no longer need to invoke ``dds build-deps`` externally, as it is all handled | no longer need to invoke ``dds build-deps`` externally, as it is all handled | ||||
by ``pmm``. | |||||
by ``pmm()``. |
packages | packages | ||||
toolchains | toolchains | ||||
source-dists | source-dists | ||||
repo | |||||
catalog | |||||
pkg-cache | |||||
remote-pkgs | |||||
interdeps | interdeps | ||||
build-deps | |||||
cmake | cmake |
Refer to: :ref:`deps.ranges.why-lowest`. | Refer to: :ref:`deps.ranges.why-lowest`. | ||||
``dds`` compatible-version ranges are similar to the shorthand range specifiers | ``dds`` compatible-version ranges are similar to the shorthand range specifiers | ||||
supported by ``npm`` and ``npm``-like tools. There are five (and a half) | |||||
version range formats available, listed in order of most-to-least restrictive: | |||||
supported by ``npm`` and ``npm``-like tools. There are four version range kinds | |||||
available, listed in order of most-to-least restrictive: | |||||
Exact: ``@1.2.3`` | Exact: ``@1.2.3`` | ||||
Specifies an *exact* requirement. The dependency must match the named | Specifies an *exact* requirement. The dependency must match the named |
.. note:: | .. note:: | ||||
Although headers are not compiled, this does not mean they are ignored. | Although headers are not compiled, this does not mean they are ignored. | ||||
``dds`` still understands and respects headers, and they are collected | ``dds`` still understands and respects headers, and they are collected | ||||
together as part of *source distribution*. | |||||
together as part of a *source distribution*. | |||||
.. _pkgs.apps-tests: | .. _pkgs.apps-tests: | ||||
********************** | ********************** | ||||
``dds`` will recognize certain compilable source files as belonging to | ``dds`` will recognize certain compilable source files as belonging to | ||||
applications and tests. If a compilable source file stem ends with ``.main`` or | |||||
``.test``, that source file is assumed to correspond to an executable to | |||||
generate. The filename stem before the ``.main`` or ``.test`` will be used as | |||||
the name of the generated executable. For example: | |||||
applications and tests, depending on the filenames "stem," which is the part of | |||||
the filename not including the outer-most file extension. If a compilable source | |||||
filename stem ends with ``.main`` or ``.test``, that source file is assumed to | |||||
correspond to an executable to generate. The filename second-inner stem before | |||||
the ``.main`` or ``.test`` will be used as the name of the generated executable. | |||||
For example: | |||||
- ``foo.main.cpp`` will generate an executable named ``foo``. | |||||
- ``bar.test.cpp`` will generate an executable named ``bar``. | |||||
- ``cat-meow.main.cpp`` will generate an executable named ``cat-meow``. | |||||
- ``cats.musical.test.cpp`` will generate an executable named ``cats.musical``. | |||||
- Given ``foo.main.cpp`` | |||||
- The stem is ``foo.main``, whose extension is ``.main``, so we will generate | |||||
an application. | |||||
- The stem of ``foo.main`` is ``foo``, so the executable will be named | |||||
``foo``. | |||||
- Given ``bar.test.cpp`` | |||||
- The stem is ``bar.test``, whose extension is ``.test``, so we will generate | |||||
a test. | |||||
- The stem of ``bar.test`` is ``bar``, so will generate an executable named | |||||
``bar``. | |||||
- Given ``cat-meow.main.cpp`` | |||||
- The stem is ``cat-meow.main``, which has extension ``.main``, so it is an | |||||
application. | |||||
- The stem of ``cat-meow.main`` is ``cat-meow``, so will generate an | |||||
executable named ``cat-meow``. | |||||
- Given ``cats.musical.test.cpp`` | |||||
- The stem is ``cats.musical.test``, which has extension ``.test``, so this is | |||||
a text executable. | |||||
- The stem of ``cats.musical.test`` is ``cats.musical``, so we will generate | |||||
an executable named ``cats.musical``. | |||||
- Note that the dot in ``cats.musical`` is not significant, as ``dds`` does | |||||
strip any further extensions. | |||||
.. note:: | .. note:: | ||||
``dds`` will automatically append the appropriate filename extension to the | ``dds`` will automatically append the appropriate filename extension to the | ||||
``src/`` directory must be added to their *include search path*. | ``src/`` directory must be added to their *include search path*. | ||||
Because the ``#include`` directives are based on the *portable* source root, | Because the ``#include`` directives are based on the *portable* source root, | ||||
the exactly location of ``src/`` is not important to the content of the | |||||
the exact location of ``src/`` is not important to the content of the | |||||
consuming source code, and can thus be relocated and renamed as necessary. | consuming source code, and can thus be relocated and renamed as necessary. | ||||
Consumers only need to update the path of the *include search path* in a single | Consumers only need to update the path of the *include search path* in a single | ||||
location rather than modifying their source code. | location rather than modifying their source code. | ||||
Packages are identified by a name/version pair, joined together by an ``@`` | Packages are identified by a name/version pair, joined together by an ``@`` | ||||
symbol. The version of a package must be a semantic version string. Together, | symbol. The version of a package must be a semantic version string. Together, | ||||
the ``name@version`` string forms the *package ID*, and it must be unique | |||||
within a repository or package catalog. | |||||
the ``name@version`` string forms the *package ID*, and it must be unique within | |||||
a repository or local package cache. | |||||
In order for a package to be exported by ``dds`` it must have a | In order for a package to be exported by ``dds`` it must have a | ||||
``package.json5`` file at its package root. Three keys are required to be | ``package.json5`` file at its package root. Three keys are required to be |
The Local Package Cache | |||||
####################### | |||||
``dds`` maintains a local cache of packages that it has obtained at the request | |||||
of a user. The packages themselves are stored as | |||||
:doc:`source distributions <source-dists>` (``dds`` does not store the binaries | |||||
that it builds within this package cache). | |||||
Reading Repository Contents | |||||
*************************** | |||||
Most times, ``dds`` will manage the cache content silently, but it may be useful | |||||
to see what ``dds`` is currently storing away. | |||||
The content of the cache can be seen with the ``pkg ls`` subcommand:: | |||||
> dds pkg ls | |||||
This will print the names of packages that ``dds`` has downloaded, as well as | |||||
the versions of each. | |||||
Obtaining Packages | |||||
****************** | |||||
.. seealso:: See also: :doc:`remote-pkgs` | |||||
When ``dds`` builds a package, it will also build the dependency libraries of | |||||
that package. In order for the dependency build to succeed, it must have a | |||||
local copy of the source distribution of that dependency. | |||||
When ``dds`` performs dependency resolution, it will consider both locally | |||||
cached packages, as well as packages that are available from any | |||||
:doc:`remote packages <remote-pkgs>`. If the dependency solution requires any | |||||
packages that are not in the local cache, it will use the information in its | |||||
catalog database to obtain a source distribution for each missing package. These | |||||
source distributions will automatically be added to the local cache, and later | |||||
dependency resolutions will not need to download that package again. | |||||
This all happens automatically when a project is built: There is **no** | |||||
"``dds install``" subcommand. | |||||
Manually Downloading a Dependency | |||||
================================= | |||||
It may be useful to obtain a copy of the source distribution of a package | |||||
from a remote. The ``pkg get`` command can be used to do this:: | |||||
> dds pkg get <name>@<version> | |||||
This will obtain the source distribution of the package matching the given | |||||
package ID and place that distribution in current working directory, using the | |||||
package ID as the name of the source distribution directory:: | |||||
$ dds pkg get spdlog@1.4.2 | |||||
[ ... ] | |||||
$ ls . | |||||
. | |||||
.. | |||||
spdlog@1.4.2 | |||||
$ ls ./spdlog@1.4.2/ | |||||
include/ | |||||
src/ | |||||
library.json5 | |||||
package.json5 | |||||
.. _repo.import-local: | |||||
Exporting a Project into the Repository | |||||
*************************************** | |||||
``dds`` can only use packages that are available in the local cache. For | |||||
packages that have a listing in the catalog, this is not a problem. But if one | |||||
is developing a local package and wants to allow it to be used in another local | |||||
package, that can be done by importing that project into the package cache as a | |||||
regular package, as detailed in :ref:`sdist.import`:: | |||||
> dds pkg import /path/to/project | |||||
This command will create a source distribution and place it in the local cache. | |||||
The package is now available to other projects on the local system. | |||||
.. note:: | |||||
This doesn't import in "editable" mode: A snapshot of the package root | |||||
will be taken and imported to the local cache. |
Remote Packages and Repositories | |||||
################################ | |||||
.. highlight:: bash | |||||
``dds`` stores a local database of available packages, along with their | |||||
dependency statements and information about how a source distribution thereof | |||||
may be obtained. | |||||
Inside the database are *package repositories*, which are remote servers that | |||||
contain their own database of packages, and may also contain the packages | |||||
themselves. An arbitrary number of package repositories may be added to the | |||||
local database. When ``dds`` updates its package information, it will download | |||||
the package database from each registered remote and import the listings into | |||||
its own local database, making them available for download. | |||||
Viewing Available Packages | |||||
************************** | |||||
The default catalog database is stored in a user-local location, and the | |||||
available packages can be listed with ``dds pkg search``:: | |||||
$ dds pkg search | |||||
Name: abseil | |||||
Versions: 2018.6.0, 2019.8.8, 2020.2.25 | |||||
From: repo-1.dds.pizza | |||||
Name: asio | |||||
Versions: 1.12.0, 1.12.1, 1.12.2, 1.13.0, 1.14.0, 1.14.1, 1.16.0, 1.16.1 | |||||
From: repo-1.dds.pizza | |||||
Name: boost.leaf | |||||
Versions: 0.1.0, 0.2.0, 0.2.1, 0.2.2, 0.2.3, 0.2.4, 0.2.5, 0.3.0 | |||||
From: repo-1.dds.pizza | |||||
Name: boost.mp11 | |||||
Versions: 1.70.0, 1.71.0, 1.72.0, 1.73.0 | |||||
From: repo-1.dds.pizza | |||||
Optionally, one can search with a glob/fnmatch-style pattern:: | |||||
$ dds pkg search 'neo-*' | |||||
Name: neo-buffer | |||||
Versions: 0.2.1, 0.3.0, 0.4.0, 0.4.1, 0.4.2 | |||||
From: repo-1.dds.pizza | |||||
Name: neo-compress | |||||
Versions: 0.1.0, 0.1.1, 0.2.0 | |||||
From: repo-1.dds.pizza | |||||
Name: neo-concepts | |||||
Versions: 0.2.2, 0.3.0, 0.3.1, 0.3.2, 0.4.0 | |||||
From: repo-1.dds.pizza | |||||
Remote Repositories | |||||
******************* | |||||
A remote package repository consists of an HTTP(S) server serving the following: | |||||
1. An accessible directory containing source distributions of various packages, | |||||
and | |||||
2. An accessible database file that contains a listing of packages and some | |||||
repository metadata. | |||||
The exact details of the directory layout and database are not covered here, and | |||||
are not necessary to make use of a repository. | |||||
When ``dds`` uses a repository, it pulls down the database file and imports its | |||||
contents into its own local database, associating the imported package listings | |||||
with the remote repository which provides them. Pulling the entire database at | |||||
once allows ``dds`` to perform much faster dependency resolution and reduces | |||||
the round-trips associated with using a dynamic package repository. | |||||
Adding a Repository | |||||
=================== | |||||
Adding a remote repository to the local database is a simple single command:: | |||||
$ dds pkg repo add "https://repo-1.dds.pizza" | |||||
[info ] Pulling repository contents for repo-1.dds.pizza [https://repo-1.dds.pizza/] | |||||
This will tell ``dds`` to add ``https://repo-1.dds.pizza`` as a remote | |||||
repository and immediately pull its package listings for later lookup. This | |||||
initial update can be suppressed with the ``--no-update`` flag. | |||||
.. note:: | |||||
After the initial ``pkg repo add``, the repository is *not* identified by its | |||||
URL, but by its *name*, which is provided by the repository itself. The name | |||||
is printed the first time it is added, and can be seen using ``pkg repo ls``. | |||||
Listing Repositories | |||||
==================== | |||||
A list of package repositories can be seen with the ``pkg repo ls`` subcommand:: | |||||
$ dds pkg repo ls | |||||
Removing Repositories | |||||
===================== | |||||
A repository can be removed by the ``pkg repo remove`` subcommand:: | |||||
$ dds pkg repo remove <name> | |||||
Where ``<name>`` is given as the *name* (not URL!) of the repository. | |||||
**Note** that removing a repository will make all of its corresponding remote | |||||
packages unavailable, while packages that have been pulled into the local cache | |||||
will remain available even after removing a repository. | |||||
Updating Repository Data | |||||
======================== | |||||
Repository data and package listings can be updated with the ``pkg repo update`` | |||||
subcommand:: | |||||
$ dds pkg repo update | |||||
This will pull down the databases of all registered remote repositories. If | |||||
``dds`` can detect that a repository's database is unchanged since a prior | |||||
update, that update will be skipped. | |||||
The Default Repository | |||||
********************** | |||||
When ``dds`` first initializes its local package database, it will add a single | |||||
remote repository: ``https://repo-1.dds.pizza/``, which has the name | |||||
``repo-1.dds.pizza``. At the time of writing, this is the only official ``dds`` | |||||
repository, and is populated sparsely with hand-curated and prepared packages. | |||||
In the future, the catalog of packages will grow and be partially automated. | |||||
There is nothing intrinsically special about this repository other than it being | |||||
the default when ``dds`` first creates its package database. It can be removed | |||||
as any other, should one want tighter control over package availability. | |||||
Managing a Repository | |||||
********************* | |||||
A ``dds`` repository is simply a directory of static files, so any HTTP server | |||||
that can serve from a filesystem can be used as a repository. ``dds`` also | |||||
ships with a subcommand, ``repoman``, that can be used to manage a repository | |||||
directory. | |||||
Initializing a Repository | |||||
========================= | |||||
Before anything can be done, a directory should be converted to a repository by | |||||
using ``repoman init``:: | |||||
$ dds repoman init ./my-repo-dir --name=my-experimental-repo | |||||
This will add the basic metadata into ``./my-repo-dir`` such that ``dds`` will | |||||
be able to pull package data from it. | |||||
The ``--name`` argument should be used to give the repository a unique name. The | |||||
name should be globally unique to avoid collisions: When ``dds`` pulls a | |||||
repository that declares a given name, it will *replace* the package listings | |||||
associated with any repository of that name. As such, generic names like | |||||
``main`` or ``packages`` shouldn't be used in production. | |||||
Listing Contents | |||||
================ | |||||
The packages in a repository can be listed using ``dds repoman ls <repo-dir>``. | |||||
This will simply print each package identifier that is present in the | |||||
repository. | |||||
Importing Source Distributions | |||||
============================== | |||||
If you have a source distribution archive, it can be imported with the | |||||
appropriately name ``dds repoman import`` command:: | |||||
$ dds repoman import ./my-repo some-pkg@1.2.3.tar.gz | |||||
Multiple archive paths may be provided at once to import them all at once. | |||||
Adding a Package by URL | |||||
======================= | |||||
A repository can also list packages that it does not host itself. Such a package | |||||
listing can be added "by URL," where the URL tells ``dds`` how to pull the | |||||
source distribution of the package. Beyond basic HTTP(S) URLs, ``dds`` can also | |||||
clone packages via ``git``:: | |||||
$ dds repoman add ./my-repo git+https://github.com/vector-of-bool/neo-fun.git#0.5.2 | |||||
The above URL tells ``dds`` that it can use ``git clone`` against | |||||
``https://github.com/vector-of-bool/neo-fun.git`` and ask for tag ``0.5.2`` to | |||||
get a source distribution directory that can be imported. Note the fragment on | |||||
``git`` URLs! The fragment is required to specify the branch or tag to clone. | |||||
If the package is available on GitHub, ``dds`` has a shorthand URL for that:: | |||||
$ dds repoman add ./my-repo github:vector-of-bool/neo-fun/0.6.0 | |||||
The ``github:`` URL scheme tells ``dds`` to clone from GitHub. A ``github:`` URL | |||||
must have exactly three path elements to determine *what* to download: | |||||
``github:{owner}/{repository}/{branch-or-tag}``. | |||||
.. note:: | |||||
The ``github:`` URL lacks an *authority* element, and as such *does not* use | |||||
the double-slash. | |||||
.. note:: | |||||
``repoman add`` will immediately attempt to pull a source distribution from | |||||
the given URL so that it may import the package's metadata into its listing | |||||
database. You cannot add a URL that is not already accessible. | |||||
Removing Packages | |||||
================= | |||||
A package can be removed from a repository with | |||||
``dds repoman remove <repo-dir> <pkg-id>``, where ``<pkg-id>`` is the | |||||
``<name>@<version>`` of the package to remove. |
The Local Package Repository | |||||
############################ | |||||
``dds`` maintains a local repository of packages that it has obtained at the | |||||
request of a user. The packages themselves are stored as | |||||
:doc:`source distributions <source-dists>` (``dds`` does not store the binaries | |||||
that it builds within the repository). | |||||
Reading Repository Contents | |||||
*************************** | |||||
Most times, ``dds`` will manage the repository content silently, but it may be | |||||
useful to see what ``dds`` is currently storing away. | |||||
The content of the repostiory can be seen with the ``repo`` subcommand:: | |||||
> dds repo ls | |||||
This will print the names of packages that ``dds`` has downloaded, as well as | |||||
the versions of each. | |||||
Obtaining Packages | |||||
****************** | |||||
.. seealso:: See also: :doc:`catalog` | |||||
When ``dds`` builds a package, it will also build the dependency libraries of | |||||
that package. In order for the dependency build to succeed, it must have a | |||||
local copy of the source distribution of that dependency. | |||||
When ``dds`` performs dependency resolution, it will consider both existing | |||||
packages in the local repository, as well as packages that are available from | |||||
the :doc:`package catalog <catalog>`. If the dependency solution requires any | |||||
packages that are not in the local repository, it will use the information in | |||||
the catalog to obtain a source distribution for each missing package. These | |||||
source distributions will automatically be added to the local repository, and | |||||
later dependency resolutions will not need to download that package again. | |||||
Manually Downloading a Dependency | |||||
================================= | |||||
It may be useful to obtain a copy of the source distribution of a package | |||||
contained in the catalog. The ``catalog get`` command can be used to do this:: | |||||
> dds catalog get <name>@<version> | |||||
This will obtain the source distribution of the package matching the named | |||||
identifier and place that distribution in current working directory, using the | |||||
package ID as the name of the source distribution directory:: | |||||
$ dds catalog get spdlog@1.4.2 | |||||
[ ... ] | |||||
$ ls . | |||||
. | |||||
.. | |||||
spdlog@1.4.2 | |||||
$ ls ./spdlog@1.4.2/ | |||||
include/ | |||||
src/ | |||||
library.json5 | |||||
package.json5 | |||||
.. _repo.export-local: | |||||
Exporting a Project into the Repository | |||||
*************************************** | |||||
``dds`` can only use packages that are available in the local repository. For | |||||
packages that have a listing in the catalog, this is not a problem. But if one | |||||
is developing a local package and wants to allow it to be used in another local | |||||
package, that can be done by exporting a source distribution from the package | |||||
root:: | |||||
> dds sdist export | |||||
This command will create a source distribution and place it in the local | |||||
repository. The package is now available to other projects on the local system. | |||||
.. note:: | |||||
This doesn't export in "editable" mode: A snapshot of the package root | |||||
will be taken and exported to the local repository. | |||||
If one tries to export a package root into a repository that already contains | |||||
a package with a matching identifier, ``dds`` will issue an error. If the | |||||
``--replace`` flag is specified with ``sdist export``, then ``dds`` will | |||||
forcibly replace the package in the local repository with a new copy. |
Source Distributions | Source Distributions | ||||
#################### | #################### | ||||
A *source distribution* is ``dds``'s primary format for consuming and | |||||
distributing packages. A source distribution, in essence, is a | |||||
:ref:`package root <pkgs.pkg-root>` archive that contains only the files | |||||
A *source distribution* (often abbreviated as "sdist") is ``dds``'s primary | |||||
format for consuming and distributing packages. A source distribution, in | |||||
essence, is a :ref:`package root <pkgs.pkg-root>` that contains only the files | |||||
necessary for ``dds`` to reproduce the full build of all libraries in the | necessary for ``dds`` to reproduce the full build of all libraries in the | ||||
package. The source distribution retains the directory structure of every | package. The source distribution retains the directory structure of every | ||||
:ref:`source root <pkgs.source-root>` of the original package, and thus retains | :ref:`source root <pkgs.source-root>` of the original package, and thus retains | ||||
Generating a source distribution from a project directory is done with the | Generating a source distribution from a project directory is done with the | ||||
``sdist`` subcommand:: | ``sdist`` subcommand:: | ||||
> dds sdist create | |||||
> dds pkg create | |||||
The above command can be executed within a package root, and the result will be | The above command can be executed within a package root, and the result will be | ||||
a gzip'd tar archive that reproduces the package's filesystem structure, but | a gzip'd tar archive that reproduces the package's filesystem structure, but | ||||
of that package. | of that package. | ||||
The ``--project=<dir>`` flag can be provided to override the directory that | The ``--project=<dir>`` flag can be provided to override the directory that | ||||
``dds`` will use as the package root. The default is the working directory of | |||||
the project. | |||||
``dds`` will use as the package root. The default is the current working | |||||
directory. | |||||
The ``--out=<path>`` flag can be provided to override the destination path of | The ``--out=<path>`` flag can be provided to override the destination path of | ||||
the archive. The path should not name an existing file or directory. By default, | the archive. The path should not name an existing file or directory. By default, | ||||
directory. | directory. | ||||
Importing a Source Ditsribution | |||||
.. _sdist.import: | |||||
Importing a Source Distribution | |||||
******************************* | ******************************* | ||||
Given a source distribution archive, one can import the package into the local | Given a source distribution archive, one can import the package into the local | ||||
repository with a single command:: | |||||
package cache with a single command:: | |||||
> dds pkg import some-package@1.2.3.tar.gz | |||||
You can also specify an HTTP or HTTPS URL to download a source distribution | |||||
archive to import without downloading it separately:: | |||||
> dds pkg import https://example.org/path/to/sdist.tar.gz | |||||
Alternatively, if a directory correctly models a source distribution, then | |||||
that directory can be imported in the same manner:: | |||||
> dds pkg import /path/to/some/project | |||||
Importing a package will allow projects on the system to use the imported | |||||
package as a dependency. | |||||
.. note:: | |||||
> dds repo import some-package@1.2.3.tar.gz | |||||
If one tries to import a package root into the cache that already contains a | |||||
package with a matching identifier, ``dds`` will issue an error. This | |||||
behavior can be overridden by providing ``--if-exists=replace`` on the | |||||
command-line. |
Specify *additional* link options to use when linking executables. | Specify *additional* link options to use when linking executables. | ||||
.. note:: | |||||
``dds`` does not invoke the linker directly, but instead invokes the | |||||
compiler with the appropriate flags to perform linking. If you need to pass | |||||
flags directly to the linker, you will need to use the compiler's options to | |||||
direct flags through to the linker. On GNU-style, this is | |||||
``-Wl,<linker-option>``. With MSVC, a separate flag ``/LINK`` must be | |||||
specified, and all following options are passed to the underlying | |||||
``link.exe``. | |||||
``optimize`` | ``optimize`` | ||||
------------ | ------------ |
Now running ``dds build`` will print more output that Catch has generated as | Now running ``dds build`` will print more output that Catch has generated as | ||||
part of test execution, and we can see the reason for the failing test:: | part of test execution, and we can see the reason for the failing test:: | ||||
[16:41:45] [error] Test <root>/_build/test/hello/strings failed! Output: | |||||
[error] Test <root>/_build/test/hello/strings failed! Output: | |||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | ||||
strings is a Catch v2.10.2 host application. | strings is a Catch v2.10.2 host application. |
.. code-block:: text | .. code-block:: text | ||||
[12:55:25] [info ] [dds-hello] Link: hello-world | |||||
[12:55:25] [info ] [dds-hello] Link: hello-world - 57ms | |||||
[12:55:25] [error] Failed to link executable '<root>/_build/hello-world'. | |||||
[info ] [dds-hello] Link: hello-world | |||||
[info ] [dds-hello] Link: hello-world - 57ms | |||||
[error] Failed to link executable '<root>/_build/hello-world'. | |||||
... | ... | ||||
<additional lines follow> | <additional lines follow> | ||||
Rebuilding the project will show no difference at the moment. | Rebuilding the project will show no difference at the moment. | ||||
.. note:: | |||||
You may also use a ``.jsonc`` or ``.json`` file extension. ``dds`` will | |||||
search for all of these files, but they will all be parsed as JSON5. | |||||
.. seealso:: | .. seealso:: | ||||
Creating a single application executable is fine and all, but what if we | Creating a single application executable is fine and all, but what if we | ||||
want to create libraries? See the next page: :doc:`hello-lib` | want to create libraries? See the next page: :doc:`hello-lib` |