Penetration Test or Production Readiness Review: Which Do You Need?

Why do people confuse the two?

Both end with a report of security findings, and both are things a buyer may ask about before signing. The questions they answer are different, though. A penetration test asks whether an attacker can get in today. A production readiness review asks whether the software was built to a recognised standard and what would need to change before real users rely on it.

When a customer asks for security evidence, a penetration test can look like the obvious first step. As the first check a new application gets, though, it is an expensive way to find a hard-coded API key or a missing permission check, and it leaves most of the code unexamined. This guide explains what each one covers, what each one misses, and the order that gets the most from both.

What is a penetration test?

The National Cyber Security Centre (NCSC) defines penetration testing as ‘a method for gaining assurance in the security of an IT system by attempting to breach some or all of that system’s security, using the same tools and techniques as an adversary might.’

In practice, a tester works against a running environment within an agreed scope and time. For an application, the scope usually covers:

• The web or mobile application itself
• The application programming interfaces (APIs) behind it
• Sign-in, session handling and the boundaries between user roles
• The hosting and network exposure, if that is in scope

The output is a report with each finding rated by risk, how it was found and how to fix it. Buyers usually ask for a summary letter rather than the full report, confirming the scope, the date, who carried out the test, and whether the findings have been fixed.

What a penetration test is good at

A tester sees the system the way an attacker does. They can find problems that only exist once everything is deployed together, such as a misconfigured storage account, a debug endpoint left switched on, or two low-risk issues that combine into a serious one. That view of the running system is what a code review cannot give you.

What a penetration test cannot tell you

The NCSC is clear about the limits. It says a penetration test ‘can only validate that your organisation’s IT systems are not vulnerable to known issues on the day of the test.’ It also says penetration testing should be viewed as a way of gaining assurance in your vulnerability management processes, ‘not as a primary method for identifying vulnerabilities.’

A test is time-boxed, so the tester follows the most promising paths rather than reading every line of code. Parts of the application they never reach are not tested, and anything you change after the test is not covered by it.

What is a production readiness review?

A production readiness review checks the application’s code and delivery practices against recognised standards, before or soon after launch. It answers a launch question: is this safe to put in front of users, and what evidence can we show?

The main reference for the code is the OWASP Application Security Verification Standard (ASVS), which the Open Worldwide Application Security Project (OWASP) describes as a set of security requirements for designing, developing and testing web applications. ASVS has three levels. Level 2 suits most business-critical applications, Level 1 is a reasonable floor for an early prototype, and Level 3 is for the highest-risk systems.

A review typically covers:

• Static code analysis: automated scanning of the source code for known weakness patterns
• Secure coding validation: an engineer checking authentication, authorisation, input handling, secrets and data protection against ASVS requirements
• Dependencies: whether third-party packages are maintained and free of known critical vulnerabilities
• Accessibility: an automated scan of key journeys against the Web Content Accessibility Guidelines (WCAG) 2.2 AA
• Tests and delivery: whether automated tests check the business logic that matters, and whether changes reach production through a pipeline

Our production readiness checklist for AI-built apps sets out the individual items, each mapped to the ASVS chapter or other reference it comes from.

What a review is good at

A reviewer can see everything that is in the repository, including parts of the application a tester never reaches. That makes a review good at finding whole classes of problem. Examples include an authorisation check missing from every endpoint written in the same style, secrets committed to the code history, and a dependency with a known vulnerability that has not yet been exposed.

It also covers areas a penetration test does not set out to examine, such as accessibility, test quality and whether the code can be maintained. In our experience, AI coding tools leave predictable gaps in the code, such as secrets, missing authorisation checks and tests that assert very little, because the tools optimise for a working demo.

What a review cannot tell you

A review works from the code, so it cannot see how the deployed system behaves. Full verification at any ASVS level also includes configuration and architecture requirements that a code review cannot reach on its own. A credible review report lists which requirements were checked and which were out of scope, rather than claiming compliance with the whole level.

A review is also not independent testing in the sense a buyer usually means when they ask for a penetration test. If the request is for a penetration test, a code review does not satisfy it.

How do they compare?

Which should come first?

Run the review first, fix what it finds, and then commission the penetration test. There are three reasons for this order.

First, it follows the NCSC’s own framing. A penetration test is meant to confirm that your own process for finding and fixing vulnerabilities works, not to be that process. The NCSC also recommends sharing a current vulnerability assessment with the testers at scoping, and a review gives you one.

Second, it gets more from the tester’s days. If the test finds secrets in the code or a missing permission check, you have paid a specialist to find something a review would have caught. The time they spent on it was not spent looking for harder problems.

Third, it gives you a cleaner summary letter. Our guide to the security evidence enterprise buyers ask for makes the same point. Close the obvious issues before the test, so the letter a buyer reads shows remediated findings rather than a long list of basic ones.

When do you need a penetration test?

You need one when someone you answer to asks for it, and when the stakes justify an independent attacker’s view. The common triggers are these:

• An enterprise or public sector buyer asks for one. Security questionnaires often ask for the date and scope of your last penetration test. The Cloud Security Alliance’s Consensus Assessments Initiative Questionnaire (CAIQ) asks whether you have a process for periodic, independent, third-party testing. An annual external test is the usual expectation once you sell to enterprise.
• A standard you are subject to requires it. For example, PCI DSS (the card payment industry’s data security standard) requires penetration testing at least every 12 months. It also requires a test after significant changes to systems in its scope.
• You work for government or critical national infrastructure. The NCSC recommends that government organisations use testers and companies in CHECK, its scheme for assured penetration testing companies. CHECK was developed for central government, the wider public sector and critical national infrastructure.
• The system has changed significantly since the last test. A new sign-in flow, a new public API or a move to new hosting changes what an attacker can reach.

Where does Cyber Essentials Plus fit?

Cyber Essentials Plus is neither a penetration test of your application nor a code review. It independently tests five technical controls across your organisation’s IT infrastructure and cloud services. The NCSC’s Requirements for IT Infrastructure v3.3 state that ‘bespoke and custom components of web applications are out of scope.’

It is worth having, and many buyers ask for it first. It does not answer the question a penetration test or a production readiness review answers about your own application. Our production readiness checklist explains what Cyber Essentials Plus covers in more detail.

When is a review enough on its own?

A review on its own is a reasonable choice when nobody you answer to has asked for a penetration test and the application is at an early stage. Examples include a prototype in front of a small group of pilot users, or an internal tool that holds no sensitive data and is not reachable from the internet.

It is not enough once a buyer, insurer, regulator or standard asks for penetration test evidence. Nor is it enough once the application holds personal data at scale, takes payments, or would cause serious harm if it were breached. At that point you need both, and the review makes the test more useful.

How do you choose a penetration tester?

Look for a firm whose accreditation your buyers will recognise, and agree the scope before you agree the price. These checks help:

• Accreditation. CREST accredits companies that provide penetration testing. Its accreditation requires evidence covering company processes, service methodologies and data security practices. For government work, check whether the buyer expects a CHECK company.
• Scope. Make sure the test covers the application, its APIs and its sign-in flows, not only your marketing website. Agree whether the tester gets test accounts for each user role.
• Access. Decide whether the tester works without inside knowledge, with partial knowledge such as API documentation, or with the source code. More access usually means more coverage in the same number of days.
• Retest. Ask whether the price includes retesting the findings you fix, so the summary letter can say they are closed.
• Reporting. Ask for an example report, and check that it gives clear steps to reproduce each finding and a summary letter you can share with buyers.

What about AI-driven penetration testing?

Some providers now offer penetration testing carried out partly or largely by AI agents, and some market it as fast enough to run several times a year. More frequent testing can be useful between full tests. Before you rely on it to satisfy a buyer, ask the buyer whether they will accept it. Ask the provider what accreditation it holds and whether a person verifies each finding. If the buyer’s questionnaire asks for a test by an accredited firm, check that the provider holds that accreditation before you buy.

How do we approach this?

We carry out production readiness reviews, and the same team can fix what they find. We do not carry out penetration tests. When a customer needs one, we recommend commissioning it from a CREST-accredited tester, and we fix its findings alongside our own.

Our Production Readiness Review is a fixed-fee, two-week review for AI-built and AI-generated applications, and we review code we did not build as well as our own. Senior engineers validate the code against OWASP ASVS, Level 2 by default for business-critical systems. They use our own suite of checks, built from the findings of penetration tests on previous projects, so issues that real tests have found are checked for on every review. Static code analysis runs with GitHub Advanced Security and our in-house rules.

You receive an evidence pack listing what was checked, what was out of scope and every finding by severity, with a prioritised plan. Fixes run as tickets under managed support for AI-built software or as a separately quoted piece of work. Once the application is live, our guide to keeping supported software secure explains how the checks continue under managed support.