What an AI code review checks: bugs, security and your conventions
A Solemly review checks three things on every pull request: bugs and logic errors, security vulnerabilities, and problems against your team's own conventions. With Jira connected it also checks whether the change does what its ticket asks. Each finding is double-checked before it's posted and carries a severity from CRITICAL to LOW.
Checked against the product on 2026-11-03.
The three areas
| Area | What it looks at |
|---|---|
| Code review | Bugs, logic errors, and problems against your team's conventions. |
| Security | Vulnerabilities and unsafe patterns, reported on the Solemly / Security check. |
| Acceptance | Only when Jira is connected: whether the change does what its linked ticket asks. |
What the reviewer can see
- The related files and the project structure around the change, not just the diff.
- Your instruction files:
CLAUDE.md,AGENTS.md,REVIEW.md, Cursor rules and similar, read at the pull request's commit. - The repository's code index of functions, classes and imports (Python, JavaScript, TypeScript, Go and Java), so a change is checked against the callers and definitions around it.
- With cross-repo context on, the callers of a changed function in your organization's other repositories.
- With agentic review on, an opt-in mode, the reviewer searches and reads your codebase before concluding.
Severities
| Severity | Meaning |
|---|---|
CRITICAL | Must be fixed: a serious bug, data loss or an exploitable security issue. |
HIGH | Should be fixed before merging. |
MEDIUM | Worth fixing. |
LOW | Minor. |
NIT | Style nitpicks. Dropped by default, so they don't reach your pull request. |
CRITICAL and HIGH findings can block a merge when the repository blocks on them.
What keeps a review quiet
- Findings are checked again before they're posted, and ones Solemly can't back up are dropped.
- Findings rated
NITare left out. - A large change gets its most important issues, not a wall of comments.
- Your instruction files can say what not to comment on, such as formatting your formatter already enforces.
- A
.solemly.yamlor the repository's Review rules can raise the minimum severity, capLOWfindings and leave paths out. - Reviews of older commits are dropped when you push again, so you don't get stale comments.
Files that are never reviewed
Lockfiles, third-party and build directories such as vendor/ and node_modules/, minified code and source maps, test snapshots, binary files and files marked as generated are left out of every review. SVG files are reviewed, since they can contain script.
What you get back
On GitHub, one native pull request review: a summary, inline comments on the changed lines with one-click suggested fixes where they fit, and check runs. On GitLab and Bitbucket, one note or comment per review listing every finding with its file, line and severity, and commit or build statuses.
| Check | Reports |
|---|---|
Solemly / Review | The review: complete, or blocking when a high-severity finding is open and the repository blocks on them. |
Solemly / Security | The security review, on its own check. |
Solemly / Summary | Whether a summary of the change was written into the pull request description. |
Solemly / Acceptance | With Jira connected, whether the change does what its linked ticket asks. |
Questions
Does it comment on formatting?
Not if you tell it not to. Instruction files can say what to ignore, such as formatting your formatter already enforces, and NIT findings are dropped by default.
How many findings does a review post?
Only the ones it can back up. Findings are double-checked before they're posted, nitpicks are left out, and a large change gets its most important issues rather than a wall of comments. A .solemly.yaml can raise the minimum severity or cap LOW findings.
Does it review generated files and lockfiles?
No. Lockfiles, third-party and build directories, minified code, test snapshots, binary files and files marked as generated are left out of every review. SVG files are reviewed, since they can contain script.
Can it see code outside the diff?
Yes. It reads related files and the project structure, uses the repository's code index of functions, classes and imports, and with cross-repo context on, sees callers in your organization's other repositories.
In the docs
Related
Try it on your next pull request
Free for 14 days with every feature, no credit card. Install it on a repository and the next pull request gets reviewed.