Add instructions for verifying the attestations using `cosign`

[iainlane]

Thanks for this action and all the work on the whole infrastructure setup. 🙂

I'm just starting to attempt to generate SBOM and provenance attestations (this part using this action), sign my images and push them to the registry. I've been working with a test image of one of our projects: grafana/wait-for-github:iainlane-attestation-test (here is the latest build log).

gh attestation verify works:

laney@melton> GH_DEBUG=1 gh attestation verify -R grafana/wait-for-github oci://grafana/wait-for-github:iainlane-attestation-test
Loaded digest sha256:360587aa9781d76b825f65f56f40738ceaf79b5ef5996b4b121c7544c9f7d662 for oci://grafana/wait-for-github:iainlane-attestation-test
Fetching attestations for artifact digest sha256:360587aa9781d76b825f65f56f40738ceaf79b5ef5996b4b121c7544c9f7d662

* Request at 2024-07-26 08:59:09.609515 +0100 BST m=+0.939990460
* Request to https://api.github.com/repos/grafana/wait-for-github/attestations/sha256:360587aa9781d76b825f65f56f40738ceaf79b5ef5996b4b121c7544c9f7d662?per_page=30
* Request took 254.894208ms
Loaded 1 attestation from GitHub API
Verifying attestation 1/1 against the configured Sigstore trust roots
Attempting verification against issuer "sigstore.dev"
SUCCESS - attestation signature verified with "sigstore.dev"

✓ Verification succeeded!

sha256:360587aa9781d76b825f65f56f40738ceaf79b5ef5996b4b121c7544c9f7d662 was attested by:
REPO                     PREDICATE_TYPE                  WORKFLOW                                       
grafana/wait-for-github  https://slsa.dev/provenance/v1  .github/workflows/build.yml@refs/pull/130/merge

What I'm wondering is how to do the same using cosign. In the build log I can see:

Attestation uploaded to registry
index.docker.io/grafana/wait-for-github@sha256:1fdab504a0700ee12b2537d75d329bee920a88a1f96ee5b20fa9cb749bfc213b

And indeed I can see this reference with e.g. docker buildx imagetools inspect. But when I try to verify or download the attestation I end up with a 404:

laney@melton> cosign --verbose download attestation grafana/wait-for-github:iainlane-attestation-test
# ...
2024/07/26 09:02:35 --> GET https://index.docker.io/v2/grafana/wait-for-github/manifests/sha256-360587aa9781d76b825f65f56f40738ceaf79b5ef5996b4b121c7544c9f7d662.att
2024/07/26 09:02:35 GET /v2/grafana/wait-for-github/manifests/sha256-360587aa9781d76b825f65f56f40738ceaf79b5ef5996b4b121c7544c9f7d662.att HTTP/1.1
Host: index.docker.io
User-Agent: cosign/v2.3.0 (darwin; arm64) go-containerregistry/v0.20.1
Accept: application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.v1+prettyjws,application/vnd.docker.distribution.manifest.v2+json,application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.list.v2+json,application/vnd.oci.image.index.v1+json
Authorization: <redacted>
Accept-Encoding: gzip


2024/07/26 09:02:35 <-- 404 https://index.docker.io/v2/grafana/wait-for-github/manifests/sha256-360587aa9781d76b825f65f56f40738ceaf79b5ef5996b4b121c7544c9f7d662.att (95.813542ms)
2024/07/26 09:02:35 HTTP/1.1 404 Not Found
Content-Length: 169
Content-Type: application/json
Date: Fri, 26 Jul 2024 08:02:35 GMT
Docker-Distribution-Api-Version: registry/2.0
Docker-Ratelimit-Source: redacted
Strict-Transport-Security: max-age=31536000

{"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":"unknown tag=sha256-360587aa9781d76b825f65f56f40738ceaf79b5ef5996b4b121c7544c9f7d662.att"}]}

Error: found no attestations
main.go:74: error during command execution: found no attestations

With my naive understanding, it looks like it's not being found from the tag's manifest. Am I doing something wrong, or is this not expected to work currently?

To make this an issue rather than a question --- if this is possible to do, it'd be a nice example to have in the README 🙂

[bdehamer]

It isn't yet possible to use cosign to verify the attestations generated by this action.

However, this is something that we're actively working on. The incompatibility largely stems from the fact that we're using the Sigstore Bundle format for packaging the attestation, but cosign doesn't yet have support for bundles.

There's a PR open already for adding bundle support to the verify-blob and verify-blob-attestation sub commands. Keep an eye out for additional PRs in the near future.

At soon as this work is complete we will definitely update our docs to show examples of using cosgin to verify our build provenance attestations.

[iainlane]

Cheers for the hint @bdehamer. I saw the linked PR got merged and it prompted me to have a go at this, mainly for curiosity. I didn't actually manage to make it work with cosign, but https://github.com/sigstore/sigstore-go did work in the end 👍

Here's how, for anyone who is interested

I'm sure this isn't the ideal way, even with current tooling, but I was happy to have muddled through. 🙂

$ BUNDLE_MANIFEST_DIGEST=$(oras discover --format json docker.io/grafana/wait-for-github:iainlane-attestation-test | jq --raw-output '.manifests[] | select (.annotations["dev.sigstore.bundle.predicateType"] == "https://slsa.dev/provenance/v1").digest')
$ echo ${BUNDLE_MANIFEST_DIGEST}
sha256:1fdab504a0700ee12b2537d75d329bee920a88a1f96ee5b20fa9cb749bfc213b
$ eval $(jq -r '
  "SIGSTORE_BUNDLE_DIGEST=\(.layers[0].digest | @sh)\n" +
  "IMAGE_INDEX_DIGEST=\(.subject.digest | @sh)"
' <(oras manifest fetch "docker.io/grafana/wait-for-github:iainlane-attestation-test@${BUNDLE_MANIFEST_DIGEST}"))
$ echo ${SIGSTORE_BUNDLE_DIGEST}
sha256:f239c387b258c8bf98c9796af9d5617dba25cf898c42893ef64628c116a84df9
$ echo ${IMAGE_INDEX_DIGEST}
sha256:360587aa9781d76b825f65f56f40738ceaf79b5ef5996b4b121c7544c9f7d662
$ oras blob fetch --output ~/temp/bundle docker.io/grafana/wait-for-github:iainlane-attestation-test@${SIGSTORE_BUNDLE_DIGEST}
✓ Downloaded  application/octet-stream                                                                                                                                                                                                                                                                                                             10/10 kB 100.00%     0s
  └─ sha256:f239c387b258c8bf98c9796af9d5617dba25cf898c42893ef64628c116a84df9
$ go run cmd/sigstore-go/main.go -artifact-digest "${IMAGE_INDEX_DIGEST#sha256:*}" -expectedIssuer https://token.actions.githubusercontent.com -expectedSAN https://github.com/grafana/wait-for-github/.github/workflows/build.yml@refs/pull/130/merge ~/temp/bundle
Verification successful!

{
   "mediaType": "application/vnd.dev.sigstore.verificationresult+json;version=0.1",
   "statement": {
      "_type": "https://in-toto.io/Statement/v1",
      "predicateType": "https://slsa.dev/provenance/v1",
      "subject": [
         {
            "name": "index.docker.io/grafana/wait-for-github",
            "digest": {
               "sha256": "360587aa9781d76b825f65f56f40738ceaf79b5ef5996b4b121c7544c9f7d662"
            }
         }
      ],
      "predicate": {
         "buildDefinition": {
            "buildType": "https://actions.github.io/buildtypes/workflow/v1",
            "externalParameters": {
               "workflow": {
                  "path": ".github/workflows/build.yml",
                  "ref": "refs/pull/130/merge",
                  "repository": "https://github.com/grafana/wait-for-github"
               }
            },
            "internalParameters": {
               "github": {
                  "event_name": "pull_request",
                  "repository_id": "577382444",
                  "repository_owner_id": "7195757",
                  "runner_environment": "github-hosted"
               }
            },
            "resolvedDependencies": [
               {
                  "digest": {
                     "gitCommit": "2cbda36b5cec403a6ae33ee2aec33e52d28a9905"
                  },
                  "uri": "git+https://github.com/grafana/wait-for-github@refs/pull/130/merge"
               }
            ]
         },
         "runDetails": {
            "builder": {
               "id": "https://github.com/grafana/wait-for-github/.github/workflows/build.yml@refs/pull/130/merge"
            },
            "metadata": {
               "invocationId": "https://github.com/grafana/wait-for-github/actions/runs/10106494744/attempts/7"
            }
         }
      }
   },
   "signature": {
      "certificate": {
         "certificateIssuer": "CN=sigstore-intermediate,O=sigstore.dev",
         "subjectAlternativeName": "https://github.com/grafana/wait-for-github/.github/workflows/build.yml@refs/pull/130/merge",
         "issuer": "https://token.actions.githubusercontent.com",
         "githubWorkflowTrigger": "pull_request",
         "githubWorkflowSHA": "2cbda36b5cec403a6ae33ee2aec33e52d28a9905",
         "githubWorkflowName": "Build",
         "githubWorkflowRepository": "grafana/wait-for-github",
         "githubWorkflowRef": "refs/pull/130/merge",
         "buildSignerURI": "https://github.com/grafana/wait-for-github/.github/workflows/build.yml@refs/pull/130/merge",
         "buildSignerDigest": "2cbda36b5cec403a6ae33ee2aec33e52d28a9905",
         "runnerEnvironment": "github-hosted",
         "sourceRepositoryURI": "https://github.com/grafana/wait-for-github",
         "sourceRepositoryDigest": "2cbda36b5cec403a6ae33ee2aec33e52d28a9905",
         "sourceRepositoryRef": "refs/pull/130/merge",
         "sourceRepositoryIdentifier": "577382444",
         "sourceRepositoryOwnerURI": "https://github.com/grafana",
         "sourceRepositoryOwnerIdentifier": "7195757",
         "buildConfigURI": "https://github.com/grafana/wait-for-github/.github/workflows/build.yml@refs/pull/130/merge",
         "buildConfigDigest": "2cbda36b5cec403a6ae33ee2aec33e52d28a9905",
         "buildTrigger": "pull_request",
         "runInvocationURI": "https://github.com/grafana/wait-for-github/actions/runs/10106494744/attempts/7",
         "sourceRepositoryVisibilityAtSigning": "public"
      }
   },
   "verifiedTimestamps": [
      {
         "type": "Tlog",
         "uri": "TODO",
         "timestamp": "2024-07-26T08:24:45+01:00"
      }
   ],
   "verifiedIdentity": {
      "subjectAlternativeName": {
         "subjectAlternativeName": "https://github.com/grafana/wait-for-github/.github/workflows/build.yml@refs/pull/130/merge"
      },
      "issuer": {
         "issuer": "https://token.actions.githubusercontent.com"
      }
   }
}

[tinaheidinger]

A how-to was recently published in the Sigstore blog: https://blog.sigstore.dev/cosign-verify-bundles/

[fmoessbauer]

Based on the howto in the sigstore blog, I created a howto for container images. This is even a bit simpler and also covers the case of bit-by-bit reproducible containers (which are not quite common yet).

Howto verify a container image

For this example, we use the siemens/kas docker container. As this container is 100% bit-by-bit reproducible, you can easily fork the project and let the GitHub Actions CI re-create the exact same containers we ship.

Tools used

• regctl to interact with the raw artifacts of an OCI registry
• cosign >= 2.4.0

Verify the official kas container image

First, download the manifest file a given tag (in this case 4.5) points to:

export TAG=4.5
regctl manifest get --format raw-body ghcr.io/siemens/kas/kas:${TAG} > manifest.json

The container digest is just the sha256 checksum of the index manifest (for multi-arch containers), or the image manifest otherwise. As ghcr.io does not yet support the referrers API, the link between the attestation bundle and the container is done by a tag: The attestation is tagged as sha256-<digest-of-container>. First compute this, then download the attestation with regctl.

DIGEST="sha256-$(sha256sum manifest.json | awk '{ print $1 }')"
regctl artifact get ghcr.io/siemens/kas/kas:${DIGEST} > bundle.json

We have all data needed for the verification.

cosign verify-blob-attestation --bundle bundle.json --new-bundle-format --certificate-oidc-issuer="https://token.actions.githubusercontent.com" --certificate-identity-regexp="^https://github.com/siemens/kas/.github/workflows/release.yml@refs/tags/${TAG}" manifest.json
# Verified OK

Verify reproducible kas container rebuild

Since kas 4.4 both our containers (image manifest) as well as our index manifests are 100% reproducible. This however does not apply to the attestations (for obvious reasons). This can be seen nicely in the following example, were we perform the same steps but against my fork of kas:

regctl manifest get --format raw-body ghcr.io/fmoessbauer/kas/kas:${TAG} > manifest.rebuild.json

Cross-check, that both manifests are identical. The kas container is 100% bit-by-bit reproducible, including the index manifest.

sha256sum manifest.*
# 1bf421652ed769134ec401add491ae6bc6f7b7c6f26c8b1fe12d15e318355563  manifest.json
# 1bf421652ed769134ec401add491ae6bc6f7b7c6f26c8b1fe12d15e318355563  manifest.rebuild.json

But the attestations are not - for obvious reasons

regctl artifact get ghcr.io/fmoessbauer/kas/kas:${DIGEST} > bundle.rebuild.json

now verify with above command

cosign verify-blob-attestation --bundle bundle.rebuild.json --new-bundle-format --certificate-oidc-issuer="https://token.actions.githubusercontent.com" --certificate-identity-regexp="^https://github.com/siemens/kas/.github/workflows/release.yml@refs/tags/${TAG}" manifest.rebuild.json
# Error: failed to verify certificate identity: no matching CertificateIdentity found, last error: expected SAN value to match regex "^https://github.com/siemens/kas/.github/workflows/release.yml@refs/tags/4.5", got "https://github.com/fmoessbauer/kas/.github/workflows/release.yml@refs/tags/4.5"
# main.go:74: error during command execution: failed to verify certificate identity: no matching CertificateIdentity found, last error: expected SAN value to match regex "^https://github.com/siemens/kas/.github/workflows/release.yml@refs/tags/4.5", got "https://github.com/fmoessbauer/kas/.github/workflows/release.yml@refs/tags/4.5"

The subject prefix which contains the workflow is different, as this was build by the GitHub Actions workflow from the fork.

[iainlane]

Looks like the work in sigstore/cosign#3889, when merged, will make this easier with cosign using the --new-bundle-format flag. Online example using that branch:

$ go run github.com/sigstore/cosign/v2/cmd/cosign verify-attestation \
  --certificate-identity https://github.com/grafana/wait-for-github/.github/workflows/build.yml@refs/heads/iainlane/build-multiarch-natively \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  --new-bundle-format \
  --type=slsaprovenance1 \
  grafana/wait-for-github:iainlane-build-multiarch-natively | \
  jq -r '.payload | @base64d | fromjson'

Verification for grafana/wait-for-github:iainlane-build-multiarch-natively --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - Existence of the claims in the transparency log was verified offline
  - The code-signing certificate was verified using trusted certificate authority certificates

{
  "_type": "https://in-toto.io/Statement/v1",
  "subject": [
    {
      "name": "index.docker.io/grafana/wait-for-github",
      "digest": {
        "sha256": "71bf1bc7f123575bb273d263568d6783c3818266ceccb8a0a449e2c34ec454c5"
      }
    }
  ],
  "predicateType": "https://slsa.dev/provenance/v1",
  "predicate": {
    "buildDefinition": {
      "buildType": "https://actions.github.io/buildtypes/workflow/v1",
      "externalParameters": {
        "workflow": {
          "ref": "refs/heads/iainlane/build-multiarch-natively",
          "repository": "https://github.com/grafana/wait-for-github",
          "path": ".github/workflows/build.yml"
          // and so on ...
        }
      }
    }
  }
}

🥳