Consuming output from Renovate's debug logs

Although the recommended means to retrieve dependency graph information for the Renovate datasource is to use the renovate-graph, it is also possible to consume the debug logs that are available from Renovate.

Note that this may result in suboptimal data collection

When using an existing Renovate setup, this is susceptible to missing data due to the way that Renovate is configured, for instance to require configuration, explicitly opting out a repository or limiting the enabledManagers, all of which will report different data to that of renovate-graph, which ignores any repo-specific configuration.

For instance, let's say that you are running Renovate against your repositories like so:

# newlines added for readability only
env RENOVATE_LOG_FILE=renovate.log \
  npx renovate@latest \
  --token $GITHUB_TOKEN \
  --dry-run=lookup --onboarding false --require-config=ignored \
  jamietanna/jamietanna deepmap/oapi-codegen

This would then produce a log file such as:

{"name":"renovate","hostname":"TheColonel","pid":92906,"level":20,"logContext":"MuGSC6XgB19laFx8mGN5C","config":{},"msg":"File config","time":"2024-02-21T18:37:08.087Z","v":0}
// ...
{"name":"renovate","hostname":"TheColonel","pid":92906,"level":20,"logContext":"MuGSC6XgB19laFx8mGN5C","repository":"jamietanna/jamietanna","baseBranch":"main","config":{"github-actions":[{"deps":[{"depName":"actions/setup-go","commitMessageTopic":"{{{depName}}} action","datasource":"github-tags","versioning":"docker","depType":"action","replaceString":"actions/setup-go@v2","autoReplaceStringTemplate":"{{depName}}@{{#if newDigest}}{{newDigest}}{{#if newValue}} # {{newValue}}{{/if}}{{/if}}{{#unless newDigest}}{{newValue}}{{/unless}}","currentValue":"v2","updates":[{"bucket":"major","newVersion":"v5","newValue":"v5","releaseTimestamp":"2023-12-05T16:50:42.000Z","newMajor":5,"newMinor":null,"updateType":"major","branchName":"renovate/actions-setup-go-5.x"}],"packageName":"actions/setup-go","warnings":[],"sourceUrl":"https://github.com/actions/setup-go","registryUrl":"https://github.com","currentVersion":"v2","isSingleVersion":true,"fixedVersion":"v2"},{"depName":"actions/checkout","commitMessageTopic":"{{{depName}}} action","datasource":"github-tags","versioning":"docker","depType":"action","replaceString":"actions/checkout@v2","autoReplaceStringTemplate":"{{depName}}@{{#if newDigest}}{{newDigest}}{{#if newValue}} # {{newValue}}{{/if}}{{/if}}{{#unless newDigest}}{{newValue}}{{/unless}}","currentValue":"v2","updates":[{"bucket":"major","newVersion":"v4","newValue":"v4","releaseTimestamp":"2023-10-17T15:52:30.000Z","newMajor":4,"newMinor":null,"updateType":"major","branchName":"renovate/actions-checkout-4.x"}],"packageName":"actions/checkout","warnings":[],"sourceUrl":"https://github.com/actions/checkout","registryUrl":"https://github.com","currentVersion":"v2","isSingleVersion":true,"fixedVersion":"v2"},{"depName":"ubuntu","currentValue":"latest","replaceString":"ubuntu-latest","depType":"github-runner","datasource":"github-runners","autoReplaceStringTemplate":"{{depName}}-{{newValue}}","skipReason":"invalid-version","updates":[],"packageName":"ubuntu"}],"packageFile":".github/workflows/rebuild.yml"}]},"msg":"packageFiles with updates","time":"2024-02-21T18:37:12.009Z","v":0}
// ...
{"name":"renovate","hostname":"TheColonel","pid":92906,"level":20,"logContext":"MuGSC6XgB19laFx8mGN5C","repository":"deepmap/oapi-codegen","msg":"32 flattened updates found: actions/checkout, actions/setup-go, actions/checkout, actions/setup-go, actions/checkout, actions/setup-go, actions/checkout, actions/setup-go, github.com/deepmap/oapi-codegen/v2, github.com/getkin/kin-openapi, github.com/go-chi/chi/v5, github.com/gofiber/fiber/v2, github.com/gorilla/mux, github.com/kataras/iris/v12, github.com/labstack/echo/v4, github.com/lestrrat-go/jwx, github.com/oapi-codegen/runtime, github.com/oapi-codegen/testutil, github.com/getkin/kin-openapi, golang.org/x/tools, gopkg.in/yaml.v2, github.com/deepmap/oapi-codegen/v2, github.com/getkin/kin-openapi, github.com/go-chi/chi/v5, github.com/gofiber/fiber/v2, github.com/gorilla/mux, github.com/kataras/iris/v12, github.com/labstack/echo/v4, github.com/oapi-codegen/nullable, github.com/oapi-codegen/runtime, github.com/oapi-codegen/testutil, gopkg.in/yaml.v2","time":"2024-02-21T18:37:16.685Z","v":0}

Notice that this uses the JSON Lines format.

As of DMD v0.89.0, you can now import the file directly using dmd import renovate by providing the path to the resulting log file.

For instance:

dmd import renovate --db dmd.db renovate.log

Importing via log files does not discover the platform

When parsing via Renovate debug log files, it is not possible to discover the Platform which is used across dependency-management-data's database.

This is not of concern if you only have software across a single Git forge, but if you're running across multiple platforms you may need to post-process the data.

As of DMD v0.107.0, you can provide a --platform parameter to override this value, if you know which Platform the debug logs are from.