How can spies from democracies compete with spies from autocracies?

Economist international editor Edward Lucas devotes 4,000+ words in the new issue of Foreign Policy to the changing landscape of state espionage in the 21st century; it's not particularly well-organized (if there's a reason for the order in which his thoughts are laid out, I couldn't find it), but despite that, it's well worth a read, even if there's lots I don't agree with here.

Lucas's main question is whether autocracies are going to win the surveillance race, especially in the face of increased civil society pressure for limits on mass surveillance in democracies. He's obviously conflicted on the issue -- he says "Western democracies need the intelligence services to defend open societies against Putinismbut not at the price of self-Putinization" -- but he's also clearly convinced that spies for democratic states are fighting with one hand tied behind their back relative to their autocratic counterparts.

That said, he's also critical of spy agencies' unwillingness to use careful forensic work on public sources in order to understand the world, basically accusing them of wanting to take shortcuts through wiretapping and dragnet surveillance because studying public sources is hard. But as he recounts, when one of his Economist colleagues was sued for libel by a Russian oligarch that he'd accused of attaining his position and wealth because of his relationship to Putin, the Economist was "able to spend hundreds of thousands of dollars on a detailed, forensic investigation of a segment of the energy market that we believed our target was manipulating" -- but that while "a spy chief from another Western country told me that finding a few hundred thousand dollars in cash to bribe a North Korean would be no problem" there was no hope of getting the same sum to spend on "statisticians and lawyers."

In the meantime, Lucas points at the worrying trend of spies leaving government service to work for commercial military/surveillance contractors who are used to circumvent democratic limits on surveillance, while simultaneously becoming very rich and politically powerful, and thus able to lobby for the expansion of these kinds of programs. Read the rest