A couple of weeks back while traveling I had an opportunity to spend some time with an old colleague from my penetration testing days. Once upon a time we used to spend much of our time on the road, living out of suitcases, probably giving the TSA fits and generally living la vida Sneakers. I'm out of that particular game these days because it's just not my bag anymore. The colleague in question is more or less on the management side of things at that particular company. Contrary to what one might reasonably assume, however, we didn't spend a whole lot of time reminiscing about the good old days, nor did we complain about all those kids on our respective lawns. What we did do was have a conversation that I've been ruminating on since I got home.
A lot of business entities ask and pay for penetration tests - a team of relatively tame hackers goes to town on their infrastructure with little to no insider knowledge to see what they can get into (within certain limits, usually) and the client uses the results as their roadmap to figure out what they need to fix. To a certain extent, this makes sense - sometimes the stuff that's broken doesn't make its presence known until somebody stumbles across it and gives it the business. But... the way these things usually go is, the client fixes everything the red team tore through like a thermite lance through a baby's crib and that's about it. They usually don't touch anything else, even to see how it stood up to second- and third- order effects. And this is a pretty serious problem, as evidenced by the overall state of information security in the last quarter century.
To a lesser extent, the purpose of a pentest is to discover stuff that the client either didn't know or forgot they had because the very act of going in blind means you have to probe everywhere to get your bearings. Sometimes on an engagement you'll run into legacy equipment that can't be patched or decomissioned, and if they try to reconfigure it the service will go casters up on you. A lot of really strange, highly specialized equipment falls into this category; GPS signal simulators come immediately to mind. Often, more extreme measures, such as airgapping will need to be taken to protect them. You might discover some long-gone intern's skunkworks project still chugging away, forgotten by everyone. Occasionally you'll stumble across one or two machines that everybody knows about because they're in the rack and everyone sees them, but they're not in any formal inventory or documentation so they don't get regular maintenance. You know... the sort of thing a hacker is likely to stumble across when they go poking around inside the network, shank it, and use it as a base of operations to infiltrate the rest of the network.
Just don't get me started on anesthesia machines.
Security assessments, on the other hand, are kind of the red-headed stepchild of the security industry. They're not sexy. They're not exciting. You don't get to break stuff (and if you do there's usually hell to pay). A lot of the assessment process involves reading documentation (when there is any), asking lots of seemingly dumb questions to folks in IT (and occasionally developers), and once in a while running a scan or two with the client hovering nearby. The overall goal is very different from that of a penetration test: Assessments try to gauge the overall state of things when compared to some set of standards, like the US Government Configuration Baseline (anonymized link), NIST 800-53 (anonymized link), or PCI DSS (anonymized link). Those baselines are not complete, and there's no way they ever can be complete because every environment's a little different. Sometimes the baselines intersect in no visible way, shape, or form with reality as we know it because they're usually not written by the technically minded.
While assessments are limited in that they usually don't do a whole lot of active probing (compared to the amount of paperwork you usually have to sift through), what some folks don't much like to admit is that pen testing isn't "complete" either in some of the same ways that security assessments aren't. Security assessments try to look at the environment at a high level and figure out what the average is. One of the questions the process asks is, "If you went diving around in that network, what could you reasonably expect to find?" Another question assessments try to answer is "How well do they know their own environment?" You probably won't find some of the more exotic or forgotten stuff because it might not be in any documents the client provides, so it might go unnoticed by all concerned. If you don't know what you have, you can't protect it or even know what shape it's in. You wouldn't necessarily know that there are some things you should do to improve vast swaths of the environment in one go (such as configuring your switches to put all of your VoIP phones on their own VLAN). You definitely wouldn't know if somebody had put a rogue wireless access point on your network unless you stumbled across it (or found out the hard way that the one you were using wasn't yours).
Pen tests, on the other hand don't normally hit everything in the environment but they are more likely to find the feral devices within the operation's sphere of reference. They're usually fairly tightly scoped ("You're going after OLYMPIA ORACLE only, leave everything else alone. If you so much as sneeze near the S/390 I'll kick you ass so hard, your descendents for three generations will write in their journals that, from time to time they experienced incredible pain in their butt cheeks for no discernable reason."), if only so that the run can be finished in a reasonable amount of time. An actual attacker can afford to spend weeks or months carefully exploring a network of hundreds or even thousands of systems in a network. Pen testers are usually (but not always) contractors, and so don't have the luxury of spending that kind of time because they're on the hook for a certain number of hours (including writing up their findings). They are there to get a job done, not spend a couple of days noodling around with a weird Xenix box they smoked out of some random workstation's hosts file. Sometimes a physical test is included to test the personnel and the layers of defenses between an attacker and the target (after all, physical access trumps everything). This usually means breaking out the lockpicks, scoping out the lobby for a day or two to determine how employees dress on site, and actually going in. If you happened to catch Tiger Team when it was on about a decade ago it's actually pretty accurate in most respects when it comes to this aspect of pen testing. (s01e01) (s02e02) (fun fact!)
I think you could make the case that, at some point you'd need both. A good assessment can take about a month, and is going to involve comparing everything to some set of standards. It's for improving your baseline - get an inventory of everything you have and what's running, figure out what standard operating procedures you have and what processes you need, figure out what you can automate (because craft error is a thing and it's easy to make mistakes when you're editing configs on the tenth server in a day). Figure out how to monitor what you have, if only to keep tabs on when a hard drive blows. Then bring in the big guns with the get-out-of-jail-free cards to see what they can make happen. Sweep the floor before you bring in the mop, if you want to look at it another way. But that's easy for me to say - I've not been on the management side, just the hired gun side. I didn't have to worry about budgeting or getting buy-in from the folks who use those systems or doing paperwork, I just had to not meddle with stuff I wasn't supposed to and try not to break things permanently.