The AI Reviewer Found a Real Bug: Its Suggested Fix Would Have Broken My App

Introduction

In the modern world of programming, artificial intelligence (AI) plays an increasingly important role in code review and testing. However, as shown by a recent incident, even modern AI can make mistakes. In this article, we will examine a case where an AI reviewer discovered a real vulnerability in code generated by another AI, but the suggested fix could have led to serious consequences.

Examples of Using AI for Code Reviews

Setting Up CodeRabbit

I used CodeRabbit, an automated code review service, to check the code generated by the AI code generator Claude Code. CodeRabbit allows you to configure the strictness level of the checks, which is crucial when dealing with different levels of complexity and requirements for the code.

# .coderabbit.yaml
reviews:
  profile: assertive

Default Check Mode

Initially, I decided to perform a default check using the CHILL profile, which does not activate many notifications and warnings.

git commit -m "Configured CodeRabbit to CHILL mode"

After performing this step, CodeRabbit did not trigger any actions.

Strict Check Mode

For a more detailed check, I switched the profile to assertive:

git commit -m "Switched CodeRabbit review mode to assertive"

This led to the discovery of a real vulnerability—CSV formula injection (CWE-1236).

The Problem and Investigation

CSV Formula Injection

CSV formula injection is a vulnerability where an attacker can inject dangerous formulas or code into a CSV file that is then interpreted and executed by the program.

// Example of potentially dangerous code
let csvData = "value1;=IF(A1>0, A1*2, 0)";

Security Check

To check for security issues, I performed several steps:

1. Static Analysis: Checked the code for potential problems.
2. **Ru

ing in Headless Browser**: Tested the application without user interaction.

Despite the code looking correct and tests passing successfully, this did not guarantee its security.

Unexpected Consequences of AI Recommendations

Proposed Fix

CodeRabbit proposed a fix that could address the detected vulnerability. However, during testing, I found that this fix might affect the handling of negative numbers in the exported file.

// Proposed fix
let cleanedCsvData = csvData.replace(/=IF\(.+\)/g, "");

Implications

If this fix had been applied, it could have led to incorrect handling of negative numbers in the exported file.

// Example handling of negative numbers
let negativeNumber = -5;
let result = cleanedCsvData.replace("A1", negativeNumber.toString());

Conclusion

This case illustrates that even when using modern AI tools for code reviews, one must be cautious and thoroughly verify all suggested fixes. Combining the use of AI with developer expertise and experience is essential.

Practical Tips

1. Adjust Stridency Level: Use different profiles for CodeRabbit or similar tools to find a balance between the number of warnings and their value.
2. Manual Verification: Carefully verify all AI recommendations before applying them.
3. Multiple Testing: Test on various usage scenarios and with different types of data.

SEO Title

AI Reviewer Found a Real Bug, But the Suggested Fix Could Have Broken the Code

SEO Description

This article discusses how an AI discovered a vulnerability in code generated by another AI, but the suggested fixes could have caused issues.

SEO Keywords

AI Reviewer, Code Vulnerability, Automated Code Review, Artificial Intelligence, Software Development

Tags

Artificial Intelligence, Automated Code Review, Software Development, Code Security, AI Reviewer