Getting Started

TL;DR extraordinaire

At a minimum, you need to:

install the dmd CLI
retrieve some data
- the options available can be found in the Data Collection Patterns cookbook
create the SQLite database for dependency-management-data
import the data

We can do this by running:

go install dmd.tanna.dev/cmd/dmd@latest

# recommended (but optional) - allows renovate-graph to resolve version constraints such as `~> 2.2` to i.e. `2.3.1`. Additionally, populates the `renovate_updates` table, to give you an indication of pending updates across repositories
export RG_INCLUDE_UPDATES='true'
# produce some data that DMD can import, for instance via renovate-graph
# note that the GITHUB_TOKEN must be a fine-grained, or classic, Personal Access Token with either `repo` or `public_repo` scopes
npx @jamietanna/renovate-graph@latest --token $GITHUB_TOKEN your-org/repo another-org/repo
# or for GitLab
env RENOVATE_PLATFORM=gitlab npx @jamietanna/renovate-graph@latest --token $GITLAB_TOKEN your-org/repo another-org/nested/repo

# set up the database
dmd db init --db dmd.db
# import renovate-graph data
dmd import renovate --db dmd.db 'out/*.json'
# then you can start querying it
sqlite3 dmd.db 'select count(*) from renovate'

Installing the `dmd` CLI

Pre-built releases can be downloaded from the GitLab Package Registry.

Alternatively, you can install it from source, provided you have a Go toolchain:

go install dmd.tanna.dev/cmd/dmd@latest

Retrieving the data

As noted above, we need to retrieve data to be imported into DMD.

There are several options available, and to choose the best option for you, check out the Data Collection Patterns cookbook.

In this example, we'll use use renovate-graph, which uses Renovate as the engine for retrieving package data.

We can run the following:

# recommended (but optional) - allows renovate-graph to resolve version constraints such as `~> 2.2` to i.e. `2.3.1`. Additionally, populates the `renovate_updates` table, to give you an indication of pending updates across repositories
export RG_INCLUDE_UPDATES='true'

# produce some data that DMD can import, for instance via renovate-graph
# note that the GITHUB_TOKEN must be a fine-grained, or classic, Personal Access Token with either `repo` or `public_repo` scopes
npx @jamietanna/renovate-graph@latest --token $GITHUB_TOKEN jamietanna/jamietanna deepmap/oapi-codegen
# or for GitLab
env RENOVATE_PLATFORM=gitlab npx @jamietanna/renovate-graph@latest --token $GITLAB_TOKEN tanna.dev/serve jamietanna/tidied

Creating the database and importing the data

Once renovate-graph has executed, you'll see an out directory with one file per repo.

First, we'll create the database:

# or any name, really
dmd db init --db dmd.db

Then, we need to import the data. Notice the quotes around the argument to avoid shell globbing

dmd import renovate --db dmd.db 'out/*.json'

Now our database is ready to go 👏

Generating missing data (optional)

This is an optional step, but for ecosystems like Java, the full dependency tree may not be immediately available.

We can run the following to (try) to fill in the missing dependency tree:

# note that this can take several minutes depending on how many dependencies you have!
dmd db generate missing-data --db dmd.db

Generating advisories (optional)

This is an optional step, but allows us to get some more meaningful information about our dependencies.

We can run the following to set up our advisories:

# optionally fetch community-sourced custom advisories
dmd contrib download

# then generate advisories for all our packages
# note that this can take several minutes depending on how many dependencies you have!
dmd db generate advisories --db dmd.db

Running some queries

Now we've got the data available, we can start to query it.

It's recommended you find your SQLite browser of choice and try the following queries:

-- how many packages have been ingested via renovate-graph
select count(*) from renovate

-- how many pending package updates have been ingested via renovate-graph
select count(*) from renovate_updates

-- how many packages have been ingested via dependabot-graph or through SBOM imports
select count(*) from sboms

-- what are your most popular 10 transitive Go dependencies?
select
  distinct package_name,
  count(*)
from
  renovate,
  json_each(dep_types) as dep_type
where
  package_manager = 'gomod'
  and dep_type.value = 'indirect'
group by
  package_name
order by
  count(*) DESC
limit 10;

And from the dmd CLI, we can also run the following:

# if you've generated the advisories data
dmd report advisories --db dmd.db

dmd report mostPopularDockerImages --db dmd.db
dmd report mostPopularPackageManagers --db dmd.db

Next steps

What's next? You could look at setting up the Git repo to store the data you collect.