What breaks in AI-written software
AI is good at writing code and poor at deciding what the code should be. The failures we are called in to fix are consistent: a data model shaped by screens rather than by the business, login without proper record-level permissions, queries that work on fifty rows and collapse on fifty thousand, and none of the things the law requires because none of them appear on screen. The code usually works. It is the shape underneath that does not.
Start with what it genuinely does well
Any account of this that begins with the failures is selling something. AI tools write good code, and by 2026 that is not controversial. Give one a well-described job with a clear boundary and it will produce something a competent developer would recognise and largely approve of.
The jobs it does properly are the ones where the answer is knowable from the question. A single function with defined inputs and outputs. The scaffolding around a form. Tests for code that already exists. A one-off script to reshape a spreadsheet. Explaining an unfamiliar codebase to the person who inherited it. A working prototype to show a room full of people what you mean, which is genuinely valuable and used to cost thousands.
What it cannot do is know things it was not told. Everything below follows from that one limitation, and none of it is a flaw in the model.
The failure is almost never the code
Software built a screen at a time acquires a data model a screen at a time. You ask for a booking page and get a bookings table. You ask for a customer list and get a customers table. You ask for invoices next week and get an invoices table that stores the customer's name and address again, because at that moment nothing connected them.
Six weeks later the same person exists three times with three spellings, and there is no answer to the question of which one is right. Nothing has crashed. Every screen works. The system simply cannot tell you what it earned from a customer last year, and it never will without being taken apart.
A developer designs the data model first because it is the expensive thing to change. Screens are cheap and get replaced constantly. The structure underneath them is set in the first fortnight and lived with for years. AI builds outward from what you asked to see, which is exactly the wrong direction, and it will not warn you because you never asked it to design a business.
This is the failure that costs the most to fix, and it is the one nobody notices while it is happening.
Login is not permission
Ask for a login and you will get one, and it will generally be sound: hashed passwords, sessions, a reset flow. That is authentication, and it answers the question of who you are.
Authorisation answers a different question. Can this particular user see this particular record? That depends entirely on rules the model was never given, so it does what looks reasonable and checks that somebody is logged in.
The result is the most common serious flaw we find. The page loads a record from the address bar and shows it to anyone with an account. Change the number at the end of the URL and you are reading another customer's file, another parent's child, another company's invoices. The application behaves perfectly. It was never told that those records belong to somebody.
This class of flaw has sat at the top of the industry's API security lists for years precisely because it cannot be caught by testing whether the software works. It works. You catch it by asking, of every screen, which user is allowed to see this, and then checking that the code asks the same question.
What only appears under real data
A system demonstrated with twenty records tells you almost nothing about a system holding twenty thousand. These are the four that arrive in order:
- Queries that repeat. A list page fetches the list, then fetches related information once per row. Fifty rows is fifty extra queries and nobody notices. Five thousand rows is a page that times out. It is the single most common performance fault in generated code.
- No indexes. Searching works instantly on a small table because everything is small. The cost appears months later and looks like the server being slow rather than like a missing line written a year earlier.
- Two people at once. The last place in a class, two staff editing one record, a payment retried when the page hangs. Handling this requires deliberately thinking about simultaneity, and it is never visible in a demonstration because demonstrations have one user.
- Money and time. Currency held in ordinary decimal numbers drifts by pennies that will not reconcile. Dates stored without proper time zone handling break twice a year when the clocks change, and the bug lands in a register at 8am on a Sunday in March.
The parts the screen never shows
A generated system does the visible job. The obligations attached to that job are invisible, so they are absent.
If you hold personal data you need a lawful basis for holding it, a retention period after which it goes, a way to find everything about one person when they ask, and a way to delete it. None of that is a screen, so none of it gets built. A system with no delete path is not a small gap. It is an obligation you cannot meet.
If you take payments, card details must never touch your system at all. The correct pattern hands the customer to the payment provider and keeps you out of scope entirely. Code that accepts a card number into your own database is a serious problem, and it is the kind of thing a generated system will do willingly if you describe the screen rather than the constraint.
Then there is the ordinary operational layer. Who changed this record and when. Where the backups are and whether anyone has restored one. Where the data physically sits. What happens when a member of staff leaves. All of it invisible, all of it assumed by your insurer and your clients.
A safe way to use it
The answer is not to avoid these tools. We use them daily and they have made parts of this work faster. The answer is to keep a person accountable for the decisions that outlive the code.
- Design the data model before the screens, on paper, with someone who has done it before. It is an afternoon and it is the afternoon that matters most.
- Write down the permission rules as sentences: a parent sees their own children and nothing else. Then have someone check the code against the list, screen by screen.
- Test with realistic volume before launch, not after. Load the number of records you expect in year two.
- Get an independent review before real money or real personal data goes near it. A day of someone experienced reading the code is cheap against the alternative.
- Ask how you would change it in a year, and if nobody can answer, that is the answer.
When to keep what you have built
Plenty of it is fine and should be left alone. If a tool is used only by your own staff, holds no personal data beyond names, has a handful of users and would take a morning to recreate, it does not need us and it does not need rewriting. Internal calculators, reporting scripts, things that reshape a spreadsheet: keep them, and enjoy the fact that they cost you a weekend.
The threshold is not sophistication. It is exposure. The moment a system holds other people's personal data, takes money, or becomes something the business cannot trade without, it has stopped being a prototype and the questions above all apply at once.
If you have built something that has crossed that line and you are not sure what is underneath it, a review is a short piece of work and usually a reassuring one. Email hello@flitzen.co.uk and say what you built and what it holds.
Common questions
The code is usually correct. The risks sit around it: a data model shaped by screens rather than by the business, missing record-level permissions, no handling for real data volumes, and none of the retention, deletion or audit obligations that never appear on a screen. Used for a first draft with a person accountable for those decisions, it is fine. Used to produce a whole system unsupervised, it reliably produces all four.
Missing authorisation. The system checks that you are logged in but not that the record you are asking for belongs to you, so changing an identifier in the address bar shows another customer's data. It is invisible in testing because the software works exactly as intended.
Usually repeated queries and missing database indexes. A list page that fetches related data once per row is imperceptible at fifty records and fatal at five thousand, and an unindexed search is instant on a small table. Both are written once and only surface as the data grows.
You can, but the legal obligations are not visible in the output and will not be built unless someone asks for them: a lawful basis for holding the data, a retention period, a way to answer a subject access request, and a way to delete. Have those specified and checked by a person before the system holds anyone real.
Not automatically. Internal tools with few users and no personal data are worth keeping. The line is exposure rather than sophistication: once a system holds other people's data, takes payments, or the business cannot trade without it, it needs the data model, permissions and obligations checked properly.
Where to go next
Want a straight answer about your own setup?
Tell us what you are running now and what is going wrong with it. We will say what we would do, including when the answer is that you do not need us.