Title: TURMOIL/APEX/APEX High Level Description Document

Release Date: 2014-12-28

Description: This undated page from NSA’s internal WikiInfo describes some of the agency’s techniques for capturing IPSec Virtual Private Network (VPN) traffic: see the Der Spiegel story Prying Eyes: Inside the NSA’s War on Internet Security, 28 December 2014.

Document: TURMOIL/APEX/APEX High Level Description Document - Wikiinfo

The accredited security level of this system is: TOP SECRET//SI-GAMMA/TALENT


Level Description Document

From Wikiinfo



• 1 (S//SI//REL) APEX: Active-Passive Integration with TURMOIL

o 1.1 (U) Motivation
o 1.2 fU) Background

■ 1,2.1 (Ul IPSEC VPN/IKE/ESP Protocols


■ 1.2.3 (U) FASHIONCLEFT Protocol/CDR

■ 1.2.4 (S//SI//REL1 TURMOIL Existing Capability - VPN

■ 1,2.5 fS//SI//REL) TURMOIL Existing Capability - VoIP

■ 1.2.6 (U) TURBINE


o 1.3 (U) APEX

■ 1,3.1 (TS//SI//REL) HAMMERMILL exfil operation in APEX

■ 1.3.2 (S//SI//REL1 Follow-on processing paths within TURMOIL

■ 1.3.3 (S//SI//REL) APEX mission applications
o 1.4 fUl APEX Command/ Control Development

o 1.5 (S//SI//REL) APEX Application development

■ 1.5.1 (TS//SI//REL) VPN phases

■ 1.5.2 (TS//SI//REL) VoIP phases

■ 1.5.3 (TS//SI//REL) Dataflow phases

• 2 Goals bv Spin

o 2.1 fU) Spin 15 goals
o 2.2 (U) Spin 16 goals
o 2.3 fUl Spin 17 goals
o 2.4 (U) Spin 18 goals
o 2.5 fUl Spin 19 goals

[edit] (S//SI//REL) APEX: Active-Passive

1 of 12

TURMOIL/APEX/APEX High Level Description Document - Wikiinfo

lllLUyiUtlUU TV1L11 1 Ulli'lWlJj

DC Radar Task 4, Modernizing VoIP and VPN

[ed|t] (U) Motivation

(TS//SI//REL) APEX describes the cross-organizational effort to achieve a
capability for shaping of TAO active collection from HAMMERMILL routers to
TURMOIL, a midpoint passive collector. The collection missions for this effort
are targeted VoIP collection (HAMMERCHANTI and collection of IPSec VPN
traffic (HAMMERSTEIN1. The desire to have the active-passive integration
capability is driven by three key motivations: scaling the exploitation of
streaming content, enhancing the likelihood of success for the VPN capability,
and eventually creating new capabilities that can't be achieved separately. In

• Routers exfiltrate content in real-time as a packet stream rather than as a
file. Current TAO backend collectors for HAMMERMILL are expected to
be strained with a large number of streaming packet exfil sessions. The
current backend capacity to receive HAMMERMILL streams is expected
to limit the number of simultaneous VoIP or VPN streams that can be

• The capability to decrypt selected IPSEC/ESP streams is built into
TURMOIL, as is the capability to route back to CES the IPSEC key
exchanges necessary to recover selected keys. Directing IPSEC VPN
traffic to TURMOIL puts it where it can be most efficiently used.

• Integration with TURMOIL and TURBINE will enable HAMMERMILL
missions to leverage TURMOIL selection and targeting rather than
operating stand-alone. This would make implants lighter, and less
complex, at the expense of possibly more exfilled traffic.

(TS//SI//REL) The end-state goal of the APEX development is to

• Achieve the real-time exfil of HAMMERMILL active collection and direct
it to a TURMOIL passive collector that can recognize the traffic, unwrap
the packets from the TAO protocol, and restore the packets to their
original state.

• Perform appropriate processing/forwarding of the unwrapped content to
corporate repositories, and optionally perform further target
identification and traffic selection in TURMOIL

• Engage TURBULENCE storage and analytic processes for delivery of
content to analysts

• Enable TURBINE dynamic control of both HAMMERMILL and TURMOIL,
allowing near-real-time implant tasking based on feedback from

[edit] (U) Background

2 of 12

TURMOIL/APEX/APEX High Level Description Document - Wikiinfo

[edit] (U) IPSEC VPN/IKE/ESP Protocols

(TS//SI//REL) IPSec describes a suite of protocols for creation of VPN tunnels
between devices. IKE is the protocol used to exchange cryptographic
parameters and establish a secure tunnel. ESP is the protocol that performs
the packet-by-packet encryption. A wide variety of algorithms of varying
strengths may be used within IPSEC. CES generally requires the packets from
both sides of an IKE exchange and knowledge of the associated pre-shared key
(PSK) in order to have a chance of recovering a key for the corresponding
cipher (ESP). A major goal of APEX is to access two sides of key exchanges for
traffic of interest.


(TS//SI//REL) HAMMERMILL is the base capability for implants on a family of
routers. Built on HAMMERMILL are a suite of mission applications for
collection of various types of traffic. HAMMERMILL uses FASHIONCLEFT to
exfil a copy of targeted data as a packet stream.

(TS//SI//REL) HAMMERMILL 2.0 is deployed and is commanded by a custom
command interface. Targeting information must be delivered via these
manually initiated commands to the HAMMERMILL application.
HAMMERMILL 2.5 is designed to accept command and control via
CHIMNEYPOOL messages and is awaiting testing.

(TS//SI//REL) HAMMERSTEIN is a HAMMERMILL application module that is
tasked to collect all packets that match a 5-tuple filter of source and
destination IP address, source and destination port, and protocol.
HAMMERSTEIN can collect IKE and ESP based on a 5-tuple filter.

HAMMERCHANT is an application module that identifies VoIP (H.323/SIP)
signaling passing through the router, extracts the user identifiers, and collects
the call if one of the users corresponds to an entry on its target list.

[edit] (U) FASHIONCLEFT Protocol/CDR

(TS//SI//REL) FASHIONCLEFT is the protocol used by HAMMERMILL and
other TAO implants to deliver collected data back to the TAO Common Data
Receptor (CDR). FASHIONCLEFT precedes an exfil session with a strongly
encrypted Session Announcement that describes the parameters of the exfil

[edit] (S//SI//REL) TURMOIL Existing Capability - VPN

(TS//SI//REL) TURMOIL is a passive filtering and collection device targeting
high-speed networks. Within TURMOIL are existing (passive) IPSEC
processing capabilities, processing IKE and ESP packets identified by the
TURMOIL front-end filtering.

3 of 12

TURMOIL/APEX/APEX High Level Description Document - Wikiinfo

(TS//SI//REL) The passive VPN metadata generation capability receives IKE
packets from TURMOIL'S front-end packet filtering and creates a metadata
record for each key exchange seen, destined eventually for the TOYGRIPPE
database. This process can be deployed anywhere a TURMOIL is deployed.

(TS//SI//REL) The "PIQ blade" processes targeted IKE and ESP. This capability
cannot be deployed to all TURMOIL sites, only to those that meet CES security

(TS//SI//REL) The PIQ blade performs two main tasks. It checks the IKE source
and destination addresses against a target list stored in KEYCARD. If one of
these is targeted, it sends the key exchange packets back in near-real-time to
CES databases via a connection directly to the CES Attack Orchestrator (AO).

(TS//SI//REL) The PIQ blade also receives ESP traffic. When a new targeted
tunnel is observed, the PIQ blade asks the AO if a key is available and buffers
the ESP traffic as long as it can while waiting for a key. If a key is returned,
the ESP is decrypted and the underlying IP packets are made available for
other application processing in TURMOIL. Because ESP encrypts packet-
by-packet, the PIQ blade can begin decryption of a session when a key arrives,
even if the earliest packets have fallen out of the buffer.

[edit] (S//SI//REL) TURMOIL Existing Capability - VoIP

(S//SI//REL) TURMOIL also contains existing capability for processing VoIP
traffic, including SIP and H.323. Capabilities for other VoIP protocols are in
development. The passive capability relies on seeing the signaling setting up a
call, extracting identifiers related to the calling and called parties, looking up
these identities for targeting information (via a KEYCARD lookup), and
collecting targeted traffic. TURMOIL also has the capability to generate
metadata records for all calls, not just targeted ones, for the FASCIA database.

[edit! (U) TURBINE

(TS//SI//REL) TURBINE is focused on command and control of covert implants.
It can execute automated workflows and is designed to communicate with
TURMOIL as well as other devices, enabling integration of active and passive
information in near-real-time. TURBINE communicates with covert implants
via CHIMNEYPOOL messages over a covert infrastructure, and with TURMOIL
and other devices via ISLANDTRANSPORT.


(TS//SI//REL) The TURMOIL component that creates metadata records from
all IKE key exchanges forwards files of these metadata records to the
PRESSURE WAVE database. METROTUBE is a framework for hosting analytic
processes that operate on items in PRESSURE WAVE. The VPN Analytic is a
METROTUBE-hosted process that is triggered when a new file of VPN

4 of 12

TURMOIL/APEX/APEX High Level Description Document - Wikiinfo

metadata arrives in PRES SURE WAVE. This process extracts the file, converts
the records into a format that TOYGRIPPE ingests, and forwards the result to
TOYGRIPPE. To support APEX, all three of the components manipulating the
metadata - the TURMOIL metadata extractor, the VPN analytic and the
TOYGRIPPE database - require modification to support the additional
information about the data source that APEX provides (see Analytic
Challenges from APEX L ).

[edit] (U) APEX

(TS//SI//REL) Currently HAMMERMILL exfils its packets by sending both the
FASHIONCLEFT session announcement and the exfilled packet stream to one
of the Common Data Receptor addresses. APEX changes the operation of

[edit] (TS//SI//REL) HAMMERMILL exfil operation in APEX

• The destination address for HAMMERMILL exfil is set to direct the traffic
to a link that a TURMOIL can see.

• The TURMOIL APEX components are configured with the HAMMERMILL
destination address and any ports needed to identify potential
HAMMERMILL traffic with a 5-tuple filter. The APEX components are also
provided with parameters needed to decrypt the FASHIONCLEFT
protocol and reconstruct the exfilled packets to their original form.

• The APEX code unwraps the FASHIONCLEFT protocol, reconstructs the
packets and directs them to an application-specific process (currently
VoIP or VPN). This process is responsible for properly handling the
application metadata delivered in the Session Announcement (e.g.
to/from phone numbers for VoIP), the exfilled packets, and the
configuration information in the TURMOIL configuration file to mark and
direct datato the appropriate path for further processing.

[edit] (S//SI//REL) Follow-on processing paths within TURMOIL

(TS//SI//REL) After APEX unwraps FASHIONCLEFT, the reconstructed packets
can be directed to one of two follow-on processing paths within TURMOIL,
depending on the nature of the application: recirculation or forwarding.

(TS//SI//REL) The recirculation path presents the APEX-unwrapped packets
and metadata to the TURMOIL filtering components as if they arrived on
TURMOIL'S passive input. The recirculated packets can then engage
TURMOIL'S passive processing and selection.

(TS//SI//REL) The forwarding path bundles the reconstructed packets together
with appropriate metadata and sends the bundle home through the normal
TURMOIL forwarding path to PRESSUREWAVE, where further analytic
processes may be run to prepare the data for corporate consumption.

5 of 12

TURMOIL/APEX/APEX High Level Description Document - Wikiinfo

[edit] (S//SI//REL) APEX mission applications

(TS//SI//REL) Initial APEX missions are collection via TURMOIL of
HAMMERSTEIN exfil of VPN traffic, and HAMMERCHANT exfil of targeted

(TS//SI//REL) HAMMERSTEIN IPSEC exfil is an example of traffic for which
there is an existing TURMOIL passive application. By recirculating the
packets, the passive VPN applications will be engaged to perform their normal
functions, though the packet stream will carry additional metadata related to
the APEX processing.

(TS//SI//REL) HAMMERCHANT exfil is an example of traffic that is
pre-selected and needs no further selection processing. Furthermore, the VoIP
signaling is not exfilled along with the collection; information such as calling
party and called party are carried as metadata in the FASHIONCLEFT Session
Announcement. Recirculation cannot engage the passive VoIP components to
perform formatting for the backend databases because these passive
processes rely on seeing the SIP or H.323 signaling. The APEX process will
take the HAMMERCHANT sessions, extract needed metadata from the Session
Announcements to attach to the session, and forward directly back to the
PRESSURE WAVE repository for analytic processing.

(TS//SI//REL) The initial APEX proof-of-concept missions - HAMMERSTEIN for
VPN and HAMMERCHANT - will develop and demonstrate both of these paths

[edit] (U) APEX Command/ Control Development

(TS//SI//REL) To enable APEX, both TURMOIL and TURBINE must be supplied
with configuration information and tasking. TURMOIL must have at minimum
an implant ID, the address that this particular implant will put in the
destination address of all its packets (and possibly ports), a key to unwrap the
FASHIONCLEFT protocol, a TAO case notation to assign to the exfil, and
appropriate TAO classification markings for the exfil. Other configuration
metadata for the implant (PDDG, SIGAD, zip codes, etc.) may also be required.

(TS//SI//REL) In its end-state, command and control of both the
HAMMERMILL implant and the TURMOIL APEX components will be
performed through TURBINE. TURBINE will he responsible for configuring
the implant with an address to send its traffic to and configuring a receiving
TURMOIL with the parameters it needs to identify and process that mission's
traffic. TURBINE will be able to receive information from TURMOIL
processing of exfilled traffic, evaluate using a workflow, and deliver updated
tasking to HAMMERMILL. Development to this end-state will occur in three

• C&C phase 1: manual configuration: HAMMERMILL is configured via its
existing command interface. Simultaneously TURMOIL APEX is provided
a configuration file for this HAMMERMILL mission. A human is

6 of 12

TURMOIL/APEX/APEX High Level Description Document - Wikiinfo

responsible for keeping the two in sync.

• C&C phase 2: semi-automatic configuration for an already established
presence: TURBINE receives the parameters for a mission and in an
automated fashion configures both the HAMMERMILL implant and the
corresponding TURMOIL APEX components. The TURBINE-
HAMMERMILL interface will use CHIMNEYPOOL RPC commands, and
HAMMERMILL 2.5 is the HAMMERMILL version that can receive
CHIMNEYPOOL commands and is required for phase 2. HAMMERMILL
2.5 will be available only for low-end MIPS platforms.

• C&C phase 3: dynamic targeting for an already established presence:
TURBINE sets initial configuration parameters as in Phase 2. Exfilled
traffic is evaluated by TURMOIL components for potential additional
targeting information. TURMOIL messages to TURBINE to dynamically
target a particular flow through the router. An example is receiving an
IKE key exchange (and possibly a few initial packets), evaluating the IP
addresses to decide if the VPN being set up corresponds to a target, and
messaging back to HAMMERMILL to capture and exfil the corresponding

(TS//SI//REL) Phase 3 of control must be managed so that router exfil does not
exceed the tolerable bandwidth limits set by OPSEC and operational concerns.
TURBINE may need to implement additional workflows in support of Phase 3
to monitor and control exfil volume. These workflows will have to be designed
to fit the metrics and monitoring information that HAMMERMILL and APEX
are capable of providing.

[edit] (S//SI//REL) APEX Application development

[edit] (TS//SI//REL) VPN phases

(TS//SI//REL) The development of the VPN process via APEX will proceed in a
phased fashion.

• VPN phase 1: IKE metadata only: IKE packets are exhled to TURMOIL
and unwrapped by APEX, recirculated and presented to the VPN
processing components. Metadata is extracted from each key exchange
for the CES TOYGRIPPE metadata database. This database is used by
SIGDEV analysts to identify potential targets for further exploitation.

• VPN phase 2: Targeted IKE forwarding: In addition to the metadata
generation, the IKE addresses are looked up in KEYCARD. If either IP
address is targeted, the key exchange packets are forwarded to the CES
Attack Orchestrator.

• VPN phase 3: Static (manual) tasking of ESP: HAMMERSTEIN may be
statically (manually) tasked to forward targeted ESP The exfiltrated ESP

7 of 12

TURMOIL/APEX/APEX High Level Description Document - Wikiinfo

packets are unwrapped by APEX, recirculated and presented to the VPN
processing components for potential decryption.

• VPN phase 4: ESP dynamic targeting: Based on the value returned by
KEYCARD, the ESP for a particular VPN may be targeted as well.
TURMOIL will send HAMMERSTEIN (via TURBINE) the parameters for
capturing the ESP for the targeted VPN.

[edit] (TS//SI//REL) VoIP phases

(TS//SI//REL) HAMMERCHANT currently maintains its own list of targeted
VoIP entities. It extracts identities from SIP or H.323 signaling, checks against
its target list, and exfils only the voice content for targeted calls. This model
could be expanded in the future using HAMMERSTEIN.

• VoIP phase 1: HAMMERCHANT Collect: HAMMERCHANT targeted VoIP
RTP exfil is captured by TURMOIL. An APEX forwarding component
bundles the voice packets into a file, attaches appropriate metadata, and
delivers to PRESSUREWAVE. A variant of the passive VoIP analytic will
be triggered to prepare the exfil for corporate delivery.

• VoIP phase 2: HAMMERCHANT Survey: HAMMERCHANT monitors all
VoIP SIP and H.323 signaling and exfiltrates all call signaling metadata to
TURMOIL. An APEX component puts the call signaling metadata into an
ASDF record and publishes it to the TURMOIL AsdfReporter component.

• VoIP phase 3: HAMMERSTEIN: VoIP signaling is captured by
HAMMERSTEIN using port information. The signaling is exfiled to
TURMOIL and unwrapped. This signaling is then presented to the normal
TURMOIL VoIP processes. The metadata process will create metadata
records for FASCIA. The collection process will extract identifiers from
the signaling and check against KEYCARD to decide if the identifiers are
targeted. If either calling party identifier or called party identifier is
targeted for active exfil, then an extended TURMOIL VoIP component will
extract the ports for the voice conversations and will send this
information as a 5-tuple filter to HAMMERSTEIN, via TURBINE.

(TS//SI//REL) Implementation of the VoIP Phase 2 and 3 processes will be
driven by mission need. The VoIP Phase 3 process should be advantageous in
allowing HAMMERMILL to potentially expand beyond SIP and H.323 VoIP
collection without additional code development in the implant, leveraging
passive processing code.

[edit] (TS//SI//REL) Dataflow phases

• Dataflow Phase 1: Dataflow tested: Collected IKE is sent to CES
database. Collected targeted data is deposited into PRESSUREWAVE and
analysts can guery and view the data.

8 of 12

TURMOIL/APEX/APEX High Level Description Document - Wikiinfo

• Dataflow Phase 2: Migrate additional HAMMERSTEIN voice flows to
PRESSURE WAVE. Exfil all signaling via port selection and let the normal
TURMOIL selection process select it.

• Dataflow Phase 3: Institutionalize APEX capability and continue to
migrate other voice flows to PRES SURE WAVE.

[edit] Goals by Spin

[edit] (U) Spin 15 goals

(TS//SI//REL) The minimal goal for Spin 15 is to achieve VPN phase 1 - IKE
metadata extraction using C&C phase 1 - manual configuration. The APEX
VPN goal is complete when a CES / TAO VPN analyst validates the SRI and the
IKE metadata content in TOYGRIPPE. See TURMOIL Spin 15
Storv/Requirements r.

(TS//SI//REL) A highly desirable goal for Spin 15 is to achieve VoIP phase 1 -
HAMMERCHANT capture - with C&C phase 1 - manual configuration. The
APEX VoIP goal is complete when a VoIP analyst validates that RTP bundled by
TURMOIL is correctly processed by the VoIP analytic and stored in

[edit] (U) Spin 16 goals

(TS//SI//REL) The minimal goals for Spin 16 are

• to achieve VPN phases 2 and 3 - targeted IKE forwarding and static
tasking of ESP using C&C phase 2 - semi-automatic configuration of

• to achieve VoIP phase 1 - HAMMERCHANT targeted VoIP RTP exfil is
captured by TURMOIL.

• to achieve Dataflow phase 1 - Collected IKE is sent to CES database.
Collected targeted data is deposited into PRESSURE WAVE and analysts
can query and view the data.

[edit] (U) Spin 17 goals

(TS//SI//REL) The minimal goals for Spin 17 are

• to achieve C&C phase 3 and VPN phase 4 - ESP dynamic targeting
using TURBINE.

• to achieve VoIP phase 2 - HAMMERCHANT Survey.

• to achieve Dataflow Phase 2 - Migrate additional HAMMERSTEIN voice

[edit] (U) Spin 18 goals

9 of 12

TURMOIL/APEX/APEX High Level Description Document - Wikiinfo

(TS//SI//REL) The minimal goal for Spin 18 is to achieve VoIP phase 3 -
HAMMERSTIEN exfil'd VoIP signaling is processed by an extended TURMOIL
VoIP component which will extract the ports for the voice conversations and
will send this information as a 5-tuple filter to HAMMERSTEIN, via TURBINE.

[edit] (U) Spin 19 goals

(TS//SI//REL) The goal for Spin 19 is to institutionalize the APEX capability
and continue to migrate other voice flows to PRESSUREWAVE.

/APEX/APEX Hiah Level Description Document"

Category: APEX


Derived From: SI Classification Guide, 02-01, Dated: 20060711
and NSA/CSSM 1-52, Dated: 20070108
Declassify On: 20320108

10 of 12

TURMOIL/APEX/APEX High Level Description Document - Wikiinfo

Go Search


• Main Page

• Community portal

• Recent changes

• Random page

• Help


• What links here

• Related changes

• Upload file

• Special pages

• Printable version

• Permanent link

social software tools

• TournalNSA

• Tapioca

• Connexions


• SpySpace

• Round Table

• Wikilnfo-NF

partner wikis

• Intellipedia

• CSEC wiki

• GCHQ wiki

• DSD wiki

• GCSB wiki

11 of 12

TURMOIL/APEX/APEX High Level Description Document - Wikiinfo

Derived From: SI Classification Guide, 02-01, Dated: 20060711

and NSA/CSSM 1-52, Dated: 20070108

Declassify On: 20320108


12 of 12


Click to send permalink to address bar, or right-click to copy permalink.

Un-highlight all Un-highlight selectionu Highlight selectionh