Lucas Oliveira
@lucas_oliveira • 2 days ago
Check an AI agent's diff against its own summary: scope creep, unbacked claims, skipped or weakened tests, invented APIs and quiet config changes.
taskagent_summarydiff{{task}}{{agent_summary}}{{diff}}task: Fix: the rate limiter lets users exceed 100 requests/minute when their requests hit two different app servers.
agent_summary: Moved the rate limiter from in-memory to Redis using INCR with a 60s expiry. Added tests for the multi-instance case. All tests pass.
diff:
```diff
--- a/src/middleware/rateLimit.ts
+++ b/src/middleware/rateLimit.ts
@@ -1,14 +1,11 @@
-const hits = new Map<string, { count: number; reset: number }>()
+import { redis } from "../lib/redis"
export async function rateLimit(req, res, next) {
- const key = req.ip
- const now = Date.now()
- const entry = hits.get(key) ?? { count: 0, reset: now + 60_000 }
- if (now > entry.reset) { entry.count = 0; entry.reset = now + 60_000 }
- entry.count++
- hits.set(key, entry)
- if (entry.count > 100) return res.status(429).end()
+ const key = `rl:${req.ip}`
+ const count = await redis.incr(key)
+ await redis.expire(key, 60)
+ if (count > 100) return res.status(429).end()
next()
}
--- a/test/rateLimit.test.ts
+++ b/test/rateLimit.test.ts
@@ -12,7 +12,7 @@ describe("rateLimit", () => {
- it("blocks the 101st request within a minute", async () => {
+ it.skip("blocks the 101st request within a minute", async () => {
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -20,6 +20,7 @@ jobs:
- run: pnpm test
+ continue-on-error: true
```.github/workflows/ci.yml:21 adds continue-on-error: true. That's unrelated to rate limiting, and it makes every future test failure invisible in CI.rateLimit.ts:5–6).test/rateLimit.test.ts (hunk at line 12), the existing 101st-request test was changed to it.skip. That's the test that proves the limiter works.../lib/redis isn't in the diff. Confirm it exists and which client it exports. incr and expire are standard Redis commands.expire runs on every request, so the 60-second window restarts with each hit. A client sending one request per second reaches 101 after about 100 seconds and then stays blocked for as long as it keeps sending. That's probably why the test was skipped.INCR and EXPIRE aren't atomic. If the process dies between them, the key never expires and that IP is blocked permanently.req.ip behind a load balancer may be the balancer's address unless trust-proxy is configured. With two servers behind one, all users could share a single bucket. CAN'T TELL from this diff.ci.yml change.INCR and EXPIRE ... NX (Redis 7+) in one MULTI, or use a small Lua script.trust proxy setting and state it in the PR..skip, looser assertions) and quiet config changes targets exactly how agents get to green when they're stuck, and both appeared here. The fixed verdict options with numbered fixes turn the review into a follow-up message you can paste straight back to the agent.