To alleviate this burden and make the code review process more efficient, we propose AI Code Reviewer.
The Reality of Code Reviews
Code Reviews serve multiple purposes. In the industry, we like to say that it increases the security of the code merged, but there is no real measure to prove that. However, code reviews are full of suggestions regarding code style and structure. Things like “extract this to a variable/method”, “change this name to be more descriptive”, or “log a message here”. Let’s say, that 80% of the comments are about things that are common, easy to spot, and regarding general knowledge, and 20% are real to-the-point comments, questioning logic and possible runtime issues. Then, if we can tackle the first group, we can substantially cut the needed time to review a given piece of code.
Enter AI Code Reviewer.
How the AI Code Reviewer Works
Since Large Language Models were trained on swaths of code and internet pages, we can leverage them to apply the general “how to code” knowledge during Code Review. Then, we allow ourselves to focus on the logic and runtime considerations, knowing AI had already dealt with “the usual”.
The idea is straightforward. Take the file changes from the Merge Request, send them to OpenAI API, with proper prompts, and receive the review comments. Since, at Astrafy, we use Gitlab, we wrapped the script in Gitlab CI and it is triggered during Merge Request events, such as opening one, or updating one with additional commits.
We can use different roles when dealing with OpenAI API. We use system prompt to explain how we want ChatGPT to behave. There we say all about its role as the code reviewer, in which format we will send the diff, and what are we expecting as its output. Then as the user prompt, we send the actual diffs marked with file paths. We also decided to use GPT 3.5 in order not to drain GPT 4 messages too fast. You can have a look at the code via the following link:
We found that satisfying AI Code Reviewer is impossible. It will generate remarks as long as there are any changes. Learn to sift through its remarks to find the valuable ones, instead of blindly following them.
It makes it easier to separate the code review into two parts. In the first part, you can use the AI Code Reviewer in order to find out where you can improve on the syntax and structure. After that, in the second part, ask a human to review the code, disregarding the comments from AI. This way you’re not caught in back and forth with AI and colleagues to add all the changes.
Remember that using this tool equals sending your code diffs to OpenAI, which is not everywhere encouraged if not outright forbidden. Make sure you can use it, before employing it full scale in your organization.
Large diffs are problematic. ChatGPT has a token limit, which makes it impossible to send arbitrarily large code review changes. You can modify the code to split it into multiple requests, however, when done so, you will receive multiple separate reviews, instead of one. In large code reviews, context is everything. The ability to follow the changes from one place to another is crucial. If you automatically split the diffs based on arbitrary numbers, you run into the risk of having incoherent reviews.
Super small reviews are also problematic. ChatGPT will attempt to generate, for example, 8 distinct comments based on your 3 lines changed. You might want to introduce the lower limit of lines changed that qualifies for AI Review.
AI Review Code
The code is simple and extensible. It boils down to three steps:
Prepare a message with lines changed, that ChatGPT should review
Send the message as well as instructions what to do with it as prompts
Add the response as the review comment.
For full example see a public repository with this hooked up on every Merge Request here. Below is the link to script with majority of the logic.
AI Reviews will slowly become a thing in the industry. Anything that improves efficiency and cut costs will. It’s worth noting that it will not replace human code reviewers but instead supercharge code reviews with AI guidance. While code reviews are sometimes bypassed or inefficiently reviewed, AI code reviewer will solve that by guiding, preparing and pointing out potential wrong practices in the code. In the end, it just adds one fully automated step in a merge request: