Create and update PR comments

Stoat introduces the concept of a single, updatable "dashboard" comment in each pull request. This significantly reduces the "tool noise" in a pull request by allowing devs to design a comment template that cleanly lays out all related build information.

Implementing a comment that can be updated by multiple commits and builds without race conditions requires a combination of GitHub actions, a GitHub application, and state that's stored outside of GitHub. Often, devs will use a combination of GitHub actions to attempt to build and updatable comment and end up with a fairly fragile system. Using a lightweight vendor for this system can save a lot of time debugging edge case behavior.

Example Stoat comment aggregating the latest build reports from a Java project (click here to see it live):

Stoat vs. Existing GitHub actions

We commonly get asked "Why use a new action while there are already many other actions that can do similar things?" For example, a user can apply the create-or-update-comment action to maintain a comment on a PR, and post whatever information they would like there. The answer is simplicity. Let's go through a detailed example of the create-or-update-comment action. To guarantee that the information in the comment is always up-to-date, there are three requirements:

• Create a new comment when the PR is opened.
• Update the comment when the PR is updated.
• Update or delete the comment when the comment is no longer needed.

Suppose there is a step, output_information, that generates the information that we want to post to the PR. The complete steps needed to fulfill the above three requirements are:

.github/workflows/post-comment.yaml

# This step generates the information that we want to post to the PR.
# It may also generate nothing, which means the comment is not needed.
- name: Output information
  id: output_information

# If the comment already exists, we want to locate it.
- name: Find the existing comment if it exists
  uses: peter-evans/find-comment@v2
  id: find_comment
  with:
    issue-number: ${{ github.event.pull_request.number }}
    comment-author: "github-actions[bot]"
    # This step uses a unique identifier in the comment body to locate the comment.
    # It is necessary for the remove_deprecated_comment step.
    body-includes: "comment-identifier"

- name: Create or update the comment
  if: steps.output_information.outputs.comment == 'true'
  uses: peter-evans/create-or-update-comment@v2
  with:
    issue-number: ${{ github.event.pull_request.number }}
    comment-id: ${{ steps.find_comment.outputs.comment-id }}
    edit-mode: "replace"
    body: |
      <!--- comment-identifier -->
      The information is: {{ steps.output_information.outputs.information }}

# If the output_information step generates nothing, we still need to
# update the comment to remove the information posted in previous runs.
- name: Remove deprecated comment
  id: remove_deprecated_comment
  if: steps.output_information.outputs.comment != 'true' && steps.find_comment.outputs.comment-id != ''
  uses: peter-evans/create-or-update-comment@v2
  with:
    issue-number: ${{ github.event.pull_request.number }}
    comment-id: ${{ steps.find_comment.outputs.comment-id }}
    edit-mode: "replace"
    # The comment must include a unique identifier so that the
    # comment can be located by the find_comment step.
    # It must match the one in the comment body.
    body: |
      <!--- comment-identifier -->
      The information is no longer relevant.

This example is adapted from a real use case at Airbyte. You can see that this is complicated and only works for one GitHub job. Aggregating information across multiple jobs would run into race conditions and likely require an external database of some kind.

In contrast, here is the equivalent Stoat config, Stoat template, and example GitHub workflow:

.stoat/config.yaml

---
version: 1
enabled: true
comment_template_file: ".stoat/template.hbs"
plugins:
  json:
    post-comment:
      # The post-comment outputs a JSON file with the information that we want to post to the PR.
      path: output.json

.github/workflows/post-comment.yaml

# appended at the end of the GitHub job
- name: Run Stoat Action
  uses: stoat-dev/stoat-action@v0
  if: always()

.stoat/template.hbs

The information is: {{ plugins.json.post-comment.value.information }}

Four GitHub action steps are replaced with the one Stoat action and a small amount of config. Also, any number of actions can push data to Stoat and the comment can aggregate the results without difficulty, so almost all the complexity here can be reused.