Building Business Systems with Generative AI: Where to Draw the Line Between Prototype and Production

How Far Can You Move Quickly with AI?

For a company without developers on staff, even a small internal tool used to start with finding a vendor and asking for a quote.

Today, for narrow, well-bounded jobs — a request form, a summary report, drafting, a first response to internal questions — it is increasingly possible to build a working prototype yourself and try it out in a short time.

The value of that change is not that a finished product appears at once. It is that you can try something before deciding.

What got shorter, though, is mainly the distance to a prototype. The responsibility of handling other people’s real data, and of keeping a system running as part of someone’s daily work, did not get lighter in the same way.

The important thing is not to hold prototypes and production to the same standard.

Narrowing that territory is not the goal here. If anything, being clear about where the review starts is what lets you move through everything else without hesitating.

Implementation steps and tool selection are out of scope for this article.

Working Is Not Proof of Safety

Ask generative AI to “add a login feature” and you may well get a login screen, and be able to sign in with a correct ID and password, within a short time.

On screen, the feature looks finished.

What you have confirmed, however, is only the happy path: that it behaves as intended when the correct information is entered.

Is access refused when another user’s ID is specified? Are credentials stored in a safe way? What happens when unexpected input arrives? Logging in the normal way tells you none of this.

Whether someone can log in is a question of authentication. What data and functions that person may access is a question of authorization. Confirming that authentication succeeded is not the same as confirming that other users’ data stays out of view.

Among the implementations generative AI proposes, some are safe, and some work correctly without being safe. The awkward part is that the look of the screen and a happy-path check make the two hard to tell apart. Both succeed on screen, and both come with a tidy explanation.

A defect that breaks and stops is easy to notice. An implementation that runs while being unsafe looks like a success.

This division of responsibility is stated in the vendor’s own documentation.

In its security documentation for Claude Code, Anthropic frames permissions as something the user grants, and states that the user is responsible for reviewing the safety of proposed code and commands before approving them (checked August 2026).

This does not mean the tool offers no protection.

The same document describes starting from read-only permissions in manual mode, asking for confirmation before operations that make changes, and mechanisms for running commands inside filesystem and network boundaries. Which mode you start in depends on your plan and settings.

Even with those protections, the role of reviewing content before approval does not leave the user. Protection on the tool’s side and review by the user are not substitutes for each other.

That review requires more than care.

What counts as a warning sign can depend heavily on having worked in the area before. Where someone who has implemented authentication many times would stop, a person without that experience is more likely to move on the moment the screen works.

Proposing and Executing Are Different

Having AI produce a suggestion and letting it operate your environment are two different things. What makes the difference is less the kind of product than how it is used and what permissions it holds.

With a suggestion, a person still decides whether to adopt it. Grant permission to operate the environment without approval, however, and there is no second choice after execution.

So checking the quality of suggestions is not enough. The scope of what the AI can execute must be limited in advance.

In April 2026, at the SaaS startup PocketOS, an agent in the AI coding tool Cursor (using Anthropic’s Claude Opus 4.6) deleted the production database.

According to the founder’s account, the deletion took about nine seconds.

Bookings and new sign-ups from the previous three months became temporarily unavailable, and the response — including the effect on customers’ own operations — ran to more than 30 hours. The founder also published the agent’s after-the-fact output, in which it listed the rules it had broken.

By the founder’s account, the agent found a token sitting in an unrelated file and used it. According to Railway’s explanation, that token was long-lived, carried account-wide permissions, and could operate every workspace the account belonged to. As a result, a single API call removed the production data.

There are two lessons here.

In other words, the normal path to recovery became unavailable at the same moment as the incident, and recovery depended on the provider’s separate arrangements.

Following the incident, Railway introduced grace periods for deletions via the API and for the deletion of backups. It also indicated plans to revise its screens so that the scope of a token’s permissions is easier to grasp at creation time (April 29, 2026).

These measures address permissions and reversibility, not how instructions are worded.

Nor can this incident simply be read as “external services are safer.” The incident happened on an external service, and how far permissions can be narrowed, and how quickly things can be restored, depend on the provider’s design as well.

Even so, the responsibility to verify the provider’s mechanisms and to add any protections that are missing remains with you.

An instruction that says “please do not delete this” is not a restriction on permissions. To make it a real boundary, the AI must not have access to it in the first place.

The founder likewise says that irreversible operations should have a confirmation step, and has continued using AI since the incident.

What should be stopped is not the use of AI, but execution without review.

The Happy Path Leaves Things Unchecked

The PocketOS incident was about execution permissions granted to an AI. Around the same period, incidents were also disclosed that show how information can become visible through other routes.

Being able to log in and seeing only what you are allowed to see are different things.

In April 2026, Lovable, a platform for building AI applications, disclosed that chat history and source code in public projects had been viewable by other users who knew the link.

The cause was a regression introduced by a change in February 2026, which unintentionally restored access that had previously been restricted. Private projects and the company’s cloud product were reported as unaffected.

According to reporting, the range researchers examined included cases where database credentials had been written directly into source code.

The failure was not limited to the defect itself: the process for escalating a report to the right people also failed.

By the company’s account, a valid report reached its external bug bounty program on February 22. Because internal documentation was out of date, however, it was closed without being escalated to the security team. When the researcher went public on April 20, the company fixed it within two hours.

Lovable has acknowledged the defect, the failure to handle the report, and that its initial public statement treated the problem too lightly.

A report that arrives but never reaches the right person does not lead to a response.

Visibility settings, too, can be left in place unnoticed.

In May 2026, WIRED and Axios reported the results of a large-scale study by the security firm RedAccess into assets built with AI development tools and left publicly accessible.

The study identified thousands of assets that were viewable with effectively no authentication, some of which appeared to expose medical records, financial information, internal documents, and logs of conversations with customers. The researchers also cited public-by-default settings in some tools as a contributing factor.

The companies, for their part, disputed both the methodology and the characterization of what “public” means.

Replit said it had been given less than 24 hours before publication, and explained that whether an app is public or private is a setting the user chooses.

Wix, the parent company of Base44, said the URLs needed to verify the claims were not provided, and argued that being in a public state does not in itself indicate a platform defect.

Lovable told Axios that the URLs and technical details needed to investigate were missing, and told WIRED that how an app is configured is ultimately the creator’s responsibility. Netlify did not respond to requests for comment.

The publications independently confirmed that some of the examples shown were in fact public. They were not able to confirm that everything visible was genuine sensitive information.

The counts and the nature of individual records call for caution. But the study and the reporting agree on one point: there were cases where content could be seen simply by visiting a URL, with no sophisticated attack involved.

Development activity itself can create another path for data exposure.

Testing against real customer data. Pasting actual records into a prompt. Handing credentials to an AI tool.

Depending on the contract, the product, and the settings, records of those exchanges may remain in conversation histories, logs, or on local devices. Information can leak from there without an attacker breaking into the production system.

The causes above are all different: permissions granted to an AI, a defect on the platform side, visibility settings, and the use of real data during development.

What they share is this: confirming that the happy path works is not the same as confirming the scope of permissions, authorization, visibility settings, or whether the system and its data can be restored.

A standard other than “does it work” is needed.

These three cases alone say nothing about whether incidents are becoming more or less common. Permissions, authorization, visibility, and recovery have been on the checklist since long before AI.

What changed is not the questions, but how quickly you arrive at them.

PocketOS and Lovable are both software companies. The subjects of the RedAccess study, by contrast, included organizations across a range of industries and experience levels.

This is not a problem that can be explained by any one company’s or engineer’s lack of skill.

Expertise reduces oversights. What separates outcomes, though, is whether you have decided what to permit, understand the blast radius, and have a route by which someone notices when something is wrong.

In Production, “Finished” Means Something Else

In a prototype, you first confirm that the user-facing features behave as intended.

In production, that is not enough. Preparations on the technical and operational side are also needed — for example:

• Who can access which data
• If something is deleted by mistake, to what point and by what means it can be restored
• Whether it is possible to determine afterward who did what
• Whether someone can keep maintaining it after the person who built it moves on
• If an incident is suspected, who investigates and who decides whether external notification is required

These are the parts users do not see.

Even with every on-screen feature in place, if these preparations are missing, the system is not ready for production.

And most of them cannot simply be added later.

Who can access which data depends on how the data is held and how permissions are designed. How far you can roll back is largely settled once the backup method and retention period are chosen.

Addressing these issues later can mean significant rework, depending on the architecture.

The Secure Software Development Framework from the National Institute of Standards and Technology (NIST) likewise treats security as something built into the normal development process rather than added after development.

“Finished” changes meaning in production not only because there are more features, but because the design needed to protect the system, restore it after failure, and keep it operating becomes part of the job.

Mapping the Questions Before You Go Live

The following organizes the points above into a form you can work through for your own situation.

Recording your own answer, the person responsible, and the date checked against each item makes it easier to revisit when the architecture or operations change later.

Data and Environments

What to check	Main areas involved (examples)
Is there data you could avoid collecting or storing in the first place?	Business process / data design
Have you confirmed the applicable laws, contracts and internal standards, the purpose of use, subcontractors, and retention periods?	Legal and contracts / data management
Are the database, runtime environment and credentials separated between prototype and production?	Cloud / DB / secret management
Can development use fictitious or properly generated synthetic data? If data derived from real data is required, have you confirmed the re-identification risk and its legal treatment?	Test data management
Is the practice to keep production credentials and real data out of AI development tools? Where they are required, have you confirmed the contractual terms, the scope of retention, and who can access it?	AI tool contracts and settings / internal rules
Are credentials absent from source code, conversations with AI, and plaintext files on devices? Are there procedures for safe storage, rotation and revocation?	Secret management / permissions / devices

Permissions and Execution

What to check	Main areas involved (examples)
Is the design such that an AI agent performing operations cannot reach the production database or deployment path directly? Where it must, have you limited the target, permissions and duration, and provided approval, logging and revocation?	Permissions / deployment path
Is each user’s scope of access enforced on the server side, not only in what the screen displays?	API / DB / access control
Have you tested that attempts to access another user’s data are rejected?	Testing / record the result
Do payments, data transmissions, and deletions have limits on amount, volume, and scope, along with safeguards against duplicate execution, approval requirements, and a means of reversal or recovery?	Business rules / application code / external services

Detection, Logging, and Recovery

What to check	Main areas involved (examples)
Can abnormal or unauthorized operations be detected, stopped, and reported to the people who need to know?	Monitoring and alerting / incident response
Does production have deletion protection or delayed deletion, with backups separated so that they are not lost in the same incident?	Cloud / DB / backup design
Have you defined the acceptable data loss and recovery time, and actually rehearsed a recovery?	Backup / recovery procedures / business continuity
Have you decided which operations are logged, how long logs are kept, and who can read them?	Log design / retention and access management
Is there a route by which vulnerability reports and anomaly reports from outside reliably reach the right person?	Reporting channel / internal procedures

Ongoing Operation and Accountability

What to check	Main areas involved (examples)
Is there someone other than the creator who can maintain the system and stop it when needed?	Operating structure / maintenance
Is there a named first responder for a suspected incident, and a defined path for decisions?	Incident response / internal structure
Before going live, has someone with the expertise the data and features require reviewed it?	Internal review / external review

”Could We Avoid Holding It at All?” Comes First

There is a reason the choice not to collect or store data sits at the top of the first table.

Japan’s My Number can be changed where it is recognized that there is a risk of the number having been leaked and used improperly, either at the person’s request or by the authority of the head of the municipality.

The U.S. Social Security Administration may also assign a new number in limited circumstances, such as when a person continues to suffer harm from using the original number despite attempts to resolve the problem.

Both, however, require a formal procedure. And changing the number does not undo the disclosure of information that has already leaked.

So before asking how to encrypt something, ask whether it needs to be collected and stored at all.

For My Number in particular, the permitted purposes and the creation of files containing it are restricted more tightly than for ordinary personal data.

Cardholder data follows the same logic.

PCI DSS sets technical and operational requirements for environments that store, process or transmit cardholder data. If your system can be arranged so that it does none of the three, the scope that applies can be reduced.

Outsourcing payment processing does not remove the need to manage the provider, or the compliance obligations that remain on your side.

Data you never collect cannot leak from your own systems or processing paths. The less data you store, process, and transmit, the fewer exposure paths you have to manage.

Keeping the Review from Becoming a Formality

Filling in the tables is not the point.

Authorization and recovery contain questions that are hard to spot without practical experience in the area. The review also needs real authority: if a problem is found, there must be a clear process and someone empowered to pause the launch.

If a problem can be found but not acted on, the review tends to become a formality.

Define the criteria for production readiness before the review begins. If they are set afterward, they tend to bend toward what has already been built.

Whether the creator and the reviewer need to be different people depends on the likely blast radius and on the independence that laws and contracts require. Where the knowledge does not exist in-house, consider a review by an outside specialist.

An external review does not in itself guarantee safety, but it is a way to supply knowledge the company does not have.

What varies with company size is mainly who performs the review and how extensive it needs to be. The questions themselves — access control, recovery, logging, accountability — do not disappear because a company is small.

Nor is this something you do once.

Each time the data you handle, the permissions you grant, payment processing, integrations with external services, or the architecture changes, revisit the questions that apply. Adjust the depth of the review to the change and its likely impact.

The Boundary Exists So You Can Move Fast

The point of varying the depth of review is not to narrow what you can build.

While it stays vague how much needs checking and where, everything starts to look vaguely dangerous. Small prototypes that touch neither real data nor production permissions end up caught in the same debate, and stall.

The boundary is not there to declare everything outside it unconditionally safe. It is there so that work with a small blast radius can move quickly and without hesitation.

When an incident occurs in production, the consequences extend beyond slowing or halting AI adoption.

A production incident can disrupt operations, create recovery costs, require explanations to customers and partners, trigger contractual or legal obligations, and damage trust. The business itself can be affected.

And if the lasting internal lesson becomes simply that “the problem happened because we used AI,” even uses that could have proceeded safely tend to stop.

Safety-focused design, then, is not there to hold back the use of AI. It is there to make incidents less likely, to limit the damage when one occurs, and to keep the technology usable as part of the business.

Move fast where AI makes things fast.

Where users or the business may be affected, build the ability to protect data, restore systems, and explain what happened.

These two are not in conflict. Drawing the boundary is what makes both possible.

About the Information in This Article

This article provides a general overview of the questions involved in moving an internal tool built with generative AI from prototype to production, based on information reviewed as of August 2026.

Regulations, guidelines, standards and product specifications may change. The laws that apply, the contractual obligations, and the measures required also vary with the industry, the data handled, the regions involved, and the architecture.

The considerations and checkpoints in this article do not establish or guarantee the safety, legality, or contractual compliance of any particular system.

For actual decisions, consult the latest primary sources and assess your situation with input from the relevant professional perspectives, including legal, security, infrastructure, and operations.

The main sources are as follows.

Regulations and standards: Personal Information Protection Commission, Japan, “Guidelines for the Proper Handling of Specific Personal Information (Business Operators Edition)” (partially revised June 2025); Digital Agency of Japan, “Frequently Asked Questions about the My Number System”; U.S. Social Security Administration, “Can I change my Social Security number?”; National Institute of Standards and Technology (NIST), “Secure Software Development Framework (SSDF) Version 1.1 / SP 800-218” (February 2022); PCI Security Standards Council, “PCI DSS” and the council’s FAQ on applicability to merchants that fully outsource payment processing.

Vendor documentation: Anthropic, “Claude Code — Security” documentation.

Incidents: For PocketOS, the account published by founder Jer Crane, the official post-incident explanation from the platform provider Railway (April 29, 2026), and reporting by Euronews, ABC News and Fast Company (April 2026). For Lovable, the company’s official post-incident statement “Our response to the April 2026 incident” (April 22, 2026) and reporting by The Next Web. For the RedAccess study of publicly accessible assets, the research itself and reporting by WIRED and Axios (May 7, 2026).