Title: XKeyScoreTabs XKS Development
Release Date: 2014-06-18
Description: This page from GCHQ’s internal GCWiki provides a guide to the NSA’s XKeyScore tool: see the Der Spiegel article The NSA in Germany: Snowden’s Documents Available for Download, 18 June 2014.
Document: XKeyscoreTabs XKS Development
Jump to: navigation. search
Getting __ .
an ---- „
* 1 XKS Ungrades
* 2 Guidance un micropluuins
* 3 Types of XKEYSCQRE
o 3.1 Traditional
o 3.2 Staua 2
o 3.3 Deep Dive
* 4 Skinny XKS
 XKS Upgrades
XKS is upgraded fortnightly un Thursday
mumings between OfJOO-1100. If you can't Lug
un or use the tool during this period, its
because of this.
 Guidance on
As you know, you can create microplugins to do
different things: some perform advanced detection
techniques to find types of traffic which can't he
detected by keywords or regular expressions alone.
Others identify and extract data fields into XKS's
• XKEYSCQRE Main Page
* XKS (ffl scale on SSE
* Getting Strong-Selected Content Into XKS
• Getting an XKS Account
• Using XKEYSCQRE
■ XKEYSCQRE Training
* XKEYSCDRE Development
. XKEYSCQRE Contacts
■ XKS News Archive
• XKS Requirements
• XKS Searches user guide
■ XKS Results user guide
* XKS Annrovnl nrocess
. NFV in XKS
■ Fmmndnn from XKS
* Auto made Promotion from XKS
* XKS for CINE
■ NSAXKovscoro Using XKS tor CNE
• XKS Tech Dicdonarics
• Mastering The Internet
• T ra nsform ino An a Iv si s
• SD Heme
v ■ d ■ ç
In the latter case, the extracted content fragments are stored in the metadata table for 30 days.
It wiJJ depend on the precision and nature of the search criteria you have used as to how strongly
- or weakly - selected that content will be.
If you are going to use search criteria that will extract data about people and store that in the
metadata table, please consult QPPLEG before doing so. They will wish to understand the nature
and scope of any data being stored in case it includes at least the names of individuals and the
majority of the data is not believed to relate to probable intelligence targets. This would make
this data particularly sensitive.
In addition, a quarterly check is now being made on all new microplugins which add data to the
1 of 4
metadata table to ensure they meet UK legal and policy requirements.
Please also he aware that usually micro plugins are automatically shared with at least NSA and
may also get deployed to other 2P XKS. By mid-2OIL a new version of XKS should have heen
deployed where individual microplugins will still he deployed to every XKS, hut they can he
tagged not to run on certain XKS. The only exception is where you deploy a micro plug in only to
GTE's XKS fleet: these will not he visible to 2P partners.
 Types of XKEYSCORE
There are currently three different types of XKS:
* Stage 2
* Deep Dive
They differ principally on where in the processing chain they sit, whether the data they receive
has already heen sessionised or not and whether they ingest all of the data they receive or
whether they apply rules to only ingest some data.
When XKS was first developed it was used to receive data from low data rate signals being
processed through WEALTHYCLUSTER (WC). WC session ised all the data on the Jink and
presented it all to XKS. All data was ingested into XKS.
GCHQ has traditional XKS aL many of our sites, including all of our Comsat, Terrestrial and SMO
sites. The EREPO XKS is also a traditional XKS, though in that case data has heen softly selected
at the implant and sessionisation takes place in TERRAIN, rather than WC.
fedill Singe 2
For higher data rates, a "Stage 2“ XKS was developed to ingest data from TURMOIL. TURMOIL
passes 5% of the packets to XKS which XKS then sessionizes. TURMOIL decides which 5% of
packets to pass based on the following criteria:
* strong selection
* subnet promotion
* technology promotion
* e-mail domains
* persona session promotion (where if a strong selector is seen, 10 minutes'or 10 MB of data
* persona session collection (where the data is collected and forwarded to NSA's PIN WALE
but is also passed to the XKS)
This data is then sent to the Stage 2 XKS. All other data is lost.
Only JPC (MUSCULAR) at GCHQ uses a Stage 2 XKS.
fedill Deep Dive
Deep Dive XKS was developed to prove that sessionisation at 10G data rates was possible. First it
sessionises all data on a link. Then it promotes data using the GENESIS selection language to
identify data types where we assess there is potential intelligence value and ingests those. The
promotion process can make one of three decisions:
* Block data that is legally not allowed to he in the system - ie UK-UK traffic
■ Allow data that is known to he wanted through use of promotion rules
* And then to drop any data that doesn't meet either of these
One of the experiments in TINT is seeking to identify where the best balance lies between what
is kept and what is not. A factor in deciding how much data to keep is the scale of storage
capacity that can he provided.
GCHQ already operates a number of Deep Dive XKS:
2 of 4
3 of 4