Online community — the model build-off
Three models, one byte-identical planner package — same source hash, same generator — so every difference below is the model, not the brief. A community looks like the easiest product in the catalogue: spaces, posts, a member list, some events. What decides it is who may read what about whom. The members are strangers to each other, moderators judge reports about people they know, and the member directory is a list of real people's contact details. So the question is not whether the app has a directory. It is whether one member can read another's email address, whether the export of that list is guarded as tightly as the list itself, and whether a reported member can find out who reported them. One of the three guarded the export and left the list wide open. The other two closed that gap by construction — and neither did it because a check in its own test suite asked.
How the builds ranked.
Hearthside
Best OverallThe one to copy. It got the directory right by design, not by accident.
- Best at
- Treating the member directory as the confidential record it is. The directory query never selects an email address at all, and the same visibility rule is applied at four separate surfaces — the list, a profile opened by its id, a member's connections and search — so there is no second door into what the first one withholds. A member who hides from the directory returns not-found rather than forbidden, so nobody can confirm they exist. The members table and its export are refused to ordinary members and hosts alike, and the moderation queue and its export agree the same way. We created a second community to test separation between them and found none leaking. Its demo sign-in is double-locked: in production it stays off even if the setting that enables it is left on. And its own suite of 115 checks re-ran cleanly against the database it shipped.
- Watch for
- No login throttling. Ten wrong passwords went through in under three seconds with no lockout, no backoff and no refusal. Two things genuinely soften it: every attempt costs a quarter of a second of password hashing, and an unknown address fails in the same time with the same message as a real one, so it cannot be used to discover accounts. But nothing stops a patient attacker, and nothing tells the operator an account is under attack.
Gatherwell
Best HardeningThe most defended build in the programme. It arrived with two operational faults.
- Best at
- Defences the others did not build. It is the only one of the three with working login throttling — twelve attempts, then a refusal. It also enforces cross-site request protection: every change must come from its own origin or it is refused. Retried requests are recognised and replayed rather than applied twice, concurrent edits are caught instead of silently overwriting each other, and every audit entry is written in the same transaction as the change it records. The directory problem is closed structurally: the member list and its export are literally the same request, so they cannot drift apart. A demo visitor gets not-found for the real community, not forbidden, so they cannot even learn it exists. Its reply-level reporting is defended inside the database itself, which refuses to store a report whose recorded source disagrees with the reply it names.
- Watch for
- Two operational faults. Deleting its database to start fresh left the application unable to start at all, because a leftover file told it the tables already existed. And its own verification could not be reproduced against the build it delivered — 64 of 76 checks passed in our hands, the rest depending on data from its own build session. Smaller: it answered only on the machine it ran on, so it could not be put on a hosted address without changing code; its login limit counted per email address only; and a member who hid from the directory vanished even for the moderators responsible for them.Fixed since audit — all of it was repaired against our findings and re-verified, and it was rescored 92 → 97. Every database starting point we tried now boots — deleted, empty, populated, and imported. Its full suite passes 76 of 76 twice in a row with no preparation, and when we deliberately broke a permission check in our copy, the suite caught it. Hosting is now configurable while staying local-only by default, a second login limit per source was added, and staff can now see hidden members, clearly labelled as hidden.
CrowdCraft
Leanest StackModeration done right, and the member list left wide open beside it.
- Best at
- Getting the vertical's usual failure right. The moderation queue is closed to ordinary members and to hosts, while filing a report stays open to every member, which is exactly the right split, and the export of the member list is locked down even more tightly than the queue. Every privileged surface we tried refuses an anonymous visitor, sign-out revokes the session on the server, and passwords are properly hashed. It runs on four production dependencies with a clean audit. Its own 44 checks re-ran cleanly in our hands.
- Watch for
- The member list hands every signed-in member all sixty member records with email addresses included — the community manager's and the moderator's among them — while its own export of the same list refuses even a moderator. One path treats the directory as confidential, the other does not. Beside it, the demo sign-in that grants any role to an anonymous visitor cannot be switched off, because the only setting the application reads is its port; there is no login throttling; and the session cookie is not marked secure.
No public demo for this build.
Models tested: Claude Fable 5.1, GPT-6 Astra and Gemini 3.8 Flash — all three from the same planner package, so all three numbers are directly comparable. Scores are as found at audit, after each build had run its own acceptance pass. Test data that a build's own QA runs left behind in its delivered database is reported back to the build but not scored, and all three builds were treated the same way on that. GPT-6 Astra was later repaired against our findings and re-verified (92 → 97); that is described under the build, but the score stays the as-found one, because that is what the model produced unaided. Two of the three are published as live demos. The Gemini 3.8 Flash build is not: its folder was changed after the audit, so what exists now is no longer the build that was scored. These grade correctness and security, not visual design or workflow usability.