22. Venona

(This entry is a summary of a talk given on July 9, 2010by Nigel West on board Queen Mary 2, headed from New York to Southhampton, UK. Disclaimer: Any errors or omissions in the following are totally my own and should not be attributed to Nigel West.)

This was an Anglo-American cryptographic project that succeeded in decoding more than 2,000 Soviet messages exchanged between Moscow and various diplomatic posts overseas between 1940 and 1949. It had several codenames but is better known as VENONA. [See West, Nigel (2002). Historical Dictionary of International Intelligence. Lantham, MD: The Scarecrow Press, p. 264].

The word VENNONA has no meaning, being a codeword only. The original codeword was BRIDE. Its disclosure brought a chill to many people because it was so closely held (only declassified in 1995. See http://www.nsa.gov/public_info/declass/venona/dated.shtml ] This project was responsible for the identification of scores of Soviet agents across the world and was the key to many famous espionage cases in the United States.

VENONA began in 1943, without mandate, at Arlington Hall Codebreakers had solved the problem of acquiring Japanese traffic and breaking the code. Time-on-hand gave the opportunity to look at other things. The codebreakers turn attention to the “Russian problem,” breaking Soviet codes.

Soviets used commercial cable companies to communicate with Moscow. Messages were encrypted and sent by wire. This was a huge advantage, since a clean (original) copy was retained at the sending office. Acquiring traffic required neither interception nor relying on copies of originals (which could be very difficult to read.)The large amount of traffic, and the clarity of the text, made cryptology much easier.

Soviets used one-time pads (OTP) for encryption. They believed the OTP was unbreakable. The first step was converting letters (Cyrillic) to numbers. Next a page was selected from the OTP. One by one, the numbers of the clear message were modularly added to successive numbers from the OTP. Decryption in Moscow would reverse the process, using modular subtraction to get the clear text.

As discussed in a previous blog, producing OTPs requires generation of random numbers. However machine production of random numbers requires some algorithm (at least in the past. In a previous blog entry a method for producing a truly random distribution was presented where no algorithm is needed.) When there is an underlying algorithm it is conceivable to recover the algorithm and deduce the random sequence.

The Soviets must have had some kind of problem generating random numbers. Added to this, more than one page of OTPs were produced. When the Nazi’s attacked the USSR, production and distribution of pads was difficult. Second, the bills of lading had to be widely used to alert what was on the ships. Thus, having observed multiple uses of OTP pages, and the huge traffic in bills of lading, deciphering the encryptions was given a significant boost.

How or why duplicate pages existed is not known. One may surmise that only a few people had responsibility for producing the pads, since they had to be closely held. Under pressure to produce more and more pads, and the difficulty of producing random numbers, it is likely that a preparer simply used carbon paper to produce multiple pages, then inserted pages in new pads in random order. We have no evidence that this is true but it is a most likely scenario. This was catastrophic for the NKVD, GRU, Naval GRU, since each used OTPs. Trying to match up clear text of manifest with encrypted traffic was a sure way into the messages. A very important clue and entryway were the dates of information on manifests. Gradually, over two years, the encrypted version matched up with clear text. That allowed for teasing out the OTP edition, and results began to appear.

The first cracked NKVD message contained a list of 24 names, people working at Los Alamos. This was extremely important since even the existence of the Manhattan Project was closely held. Soviets discussing the Manhattan project meant there was a Russian spy in Los Alamos.

Acquiring such valuable information showed the value of the VENONA project. Over 2000 messages were broken. From 1941-1948 the BRIDE project continue since so much information was in the traffic. A huge effort was set up to find out who was the spy and identify him. (Actually there were at least two spies, Ted Hall and Klaus Fuchs. They may have been more, as many continue to suspect.) The Soviet codeword for the Manhattan Project was found to be ENORMOUS.

In cryptography, a message is seldom entirely broken by itself. Messages are not only encrypted but they are encoded, where codewords are used to further cover something closely held. As more messages are decrypted some of these codewords are also broken, leading to identification of, say specific names. As a codeword was broken all messages containing it were updated. Sadly for historical purposes, we now have only the best versions of the traffic. We cannot see how the information evolved.

We learned to read quite a bit of Soviet traffic from New York, San Francisco, Washington, DC, and a few more sites. We were able to look at other REZIDENTURA. We shared information with the UK and were able to get much further help from the Brits and from Swedish crypts. The Swedes kept all the traffic the Soviets sent even though they could not read it.

Traffic from Moscow to Canberra was hugely important. [See West, Nigel (2002). Historical Dictionary of International Intelligence. Lantham, MD: The Scarecrow Press, p. 18].

The Australians (with the US) operated a secret installation at Pine Gap, outside Alice Springs. (See http://www.bibliotecapleyades.net/sociopolitica/sociopol_pinegap08.htm) The Canberra traffic was contemporaneous and allowed identification of real people. From that, contacts could be identified. Together, efforts led to developing a string or cluster of people involved in the act. The Australians at that time actually had no counterespionage units, until MI5 urged on. As the counterespionage effort there got underway it was identified as “The Case” to conceal the actual source (VENONA).

There was also considerable consternation that scientists passing secrets might be continuing in espionage or that they recruited others to replace them. An unidentified spy BARON (see http://www.freerepublic.com/focus/f-news/1503011/posts) was at Bletchley Park. It never was discovered who it was passing secrets to the USSR. GSHQ (Government Communications Headquarters, the NSA of Great Britain; see Nigel West (2002), p. 117)

VENONA also revealed changes in codenames. It took 4.5 years to find REST from Los Alamos. Counterintelligence narrowed it to one person who fit all the data. It was Klaus Fuchs. By the time he was ferreted out (1949) he had returned to the UK to work at Harwell. How could he be dealt with? It was out of the question to use VENONA in court or its value would be lost. MI5 did get Fuchs to self-present. He acknowledged he had passed information. He was arre3asted, pled guilty, believing he was \doing the right thing. He knew that he was already known to MI5 from VENONA.
Donald Maclean was also identified from information from VENONA and known by the codenames GOMER, HOMER, G. He vanished before he could be arrested. Since Maclean was known to be close to Philby, suspicion immediately fell on Kim Philby.

Harry gold (GOOSE) was identified. That led to David Greenglass and his wife (BUMBLEBEE and WASP.) Their interrogations revealed Ethel Rosenberg, married to Julius (ANTENNA and then LIBERAL.) In retrospect the codenames are often semi-transparent to help the handlers remember who the sources are.

Alger Hiss and his wife were also identified. Hiss had attended the Yalta conference and advised President Franklin Roosevelt on spheres of influence. As it happened, and was later determined, there were about 300 spies in the US, many of them around FDR. It was also true in Britain. Two were known as NOBILITY and MONTAGUE. INTELLIGENCE may have been Holbein (also RESERVIST, and MINISTER, never specifically identified.) Many of these are discussed at the NSA website (http://ww.nsa.gov).

The recent identification of 10 Russian sleeper spies in the US shows it never ends. These have been taken out of the picture (as discussed early in this blog.) They may have disclosed some information. Forensics tracing may reveal more. Locking up a spy has no operational value. However, exchanging them gave the US a great advantage, since the Russians sent to us, for example Alexander Zaporozhsky, Gennady Vasilenko, Sergei Skripal, and Igor Sutagin. Some may cooperate to a degree, others have no interest in handing over anything. Zaporshsky helped identify the traitor, Hansen. Vasilenko (a name included by Ames) was totally innocent. He was charged in Moscow with terrorism since he had hunting rifles in his garage. Very unusual for the Russians to sent people as these out of the country. Sutagin is a nuclear scientist, and will be interesting to talk with.