Title: XKEYSCORE Search Forms

Release Date: 2015-07-01

Document Date: 2009-03-01

Description: This 49-page March 2009 NSA presentation explains how to conduct searches within XKeyScore: see the Intercept article XKEYSCORE: NSA’s Google for the World’s Private Communications, 1 July 2015.

Document: xks-search-forms-p1-normal.gif:
XKEYSCORE
Search Forms

March 2009

El



• *1



... ■





OK

l 1001

SECRET//COMINT//REL TO USA, AUS, CAN, GBR, MZL

1-52

DATED: 20070108
DECLASSIFY ON: 20320108xks-search-forms-p2-normal.gif:
SECRET//COMINTORELTO USA, AUS. CAN, GBR. NZL

Standard search fields

Wildcards

■ * - multiple characters anywhere in word

■ _ - single character anywhere in word

■ Some fields are auto-wildcarded - the field name will have
a * before and/or after it

Operators

■ Boolean AND, OR - do not use in the same field

■ ! - NOT (e.g. !joe)

■ - comparison (e.g. >00080)

■ regex: - regular expression (e.g. regex:[0-9]*)

■ Enter I to require a field to be non-empty

SECRET//COMINT//RELTO USA, AUS, CAN, GBR, NZLxks-search-forms-p3-normal.gif:
SECRET//COMINT//RELTO USA. AUS. CAN. GBR. NZL

Special (full-text) search fields

Google-like syntax - just list your terms and the
query will return sessions that match any of them
Wildcards only allowed at the end of a word
Search terms must be at least 4 characters

Use + or - to require that a word must or must not
be present

Use to find an exact phrase
Use () for grouping

You can still use “classic” syntax - we convert it
for you

SECRET//COMINT//RELTO USA, AUS, CAN, GBR, NZLxks-search-forms-p4-normal.gif:
SECRET//COMINT//RELTO USA. AUS. CAN, GBR. NZL

Special (full-text) search fields

Vs

f

■ Examples:

Search terms Returned results...
apple banana contain ‘apple or ‘banana’ or both
+apple +juice contain both apple and ‘juice
+apple -macintosh contain apple’ but not macintosh’
+apple +(turnover strudel) contain apple AND either ’turnover’ or ‘strudel’
apple* contain words like ‘apple’ or ‘apples’ or applesauce' or applet’
"apple juice” contain the exact phrase ‘apple juice'

SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZLxks-search-forms-p5-normal.gif:
SECRET//COMINT//RELTO USA, AUS. CAN, GBR. NZL

This plug-in has no data!

Under development

— Menu items and search forms may show up
before a plug-in goes “live” in the field

Limited deployment

■ Some sites run different sets of plug-ins

Populated by front end

■ Some plug-ins simply database metadata
provided by the system that feeds XKS, and not
all sites are set up the same way

SECRET/ZCOMINT/tRELTO USA, AUS, CAN, GBR, NZLxks-search-forms-p6-normal.gif:


SECRETi/COMINT/mELTO USA. AUS. CAN. GBR. NZL

Simple Loggingxks-search-forms-p7-normal.gif:
SECRET//COMINT//RELTO USA. AUS. CAN. GBR. NZL

Full Log DNI

One record for every session processed

Collection fields

- SIGAD

■ Casenotation

■ Session ID (UUID)

Protocol fields

■ MAC addresses

■ IP addresses

■ Port numbers

SECRET//COMINT//RELTO USA, AUS, CAN, GBR, NZLxks-search-forms-p8-normal.gif:
SECRET//COMINT//RELTO USA, AUS, CAN. GBR. N2L

Full Log DNI

Application ID fields

□ Application Name - full application ID

■ Application Type - top level of application ID

□ Application Info - extra info

n Appid+fingerprints - full application ID plus any
matching fingerprints

Example:

□ Application name: mail/webmail/yahoo

■ Application type: mail

■ Application info: viewFolder_webmail

SECRET//COMINT//REL TO USA, AUS, SECRET//COMINT//RELTO USA, AUS, CAN, GBR. NZL

Full Log DNI

Fields populated by other plug-ins

□ Username (from User Activity)

■ Category hits (from Category DNI)

■ Client IP/X-Forwarded-For (from Web Proxy)

Most Full Log search fields are available on
every other search form

SECRET//COMINT//RELTO USA, AUS, CAN, GBR, NZLxks-search-forms-p10-normal.gif:
SECRET//COMINT//RELTO USA, AUS. CAN, GBR. NZL

Email Addresses

Anything that looks like an email address
Searchable fields

■ Email username - the part before the @ only!

■ Domain - the part after the @

■ Subject - email subject, if present

Example:

Sender: serl@yahoo.com

MIME-Version: 1.0

Subject: check this out

Date: Tue, 02 Jan 2007 13:27:31 -0000

Message-ID:

From: "User One"

To: "User Two”

SECRETI/COMINTOREL TO USA, AUS, CAN, GBR, NZLxks-search-forms-p11-normal.gif:
SECRET//COMINT//RELTO USA, AUS. CAN, GBR. NZL

Logins & Passwords

Anything that looks like a login or password
Searchable fields
- Username
Password

Examples:




USER badguy
PASS asdf123

SECRETI/COMINTOREL TO USA, AUS, CAN, GBR, NZLxks-search-forms-p12-normal.gif:
SECRET//COMINT//RELTO USA, AUS. CAN, GBR. NZL

Phone Numbers in DNI

Anything that looks like a phone number
Searchable fields
■ Phone number

Number type (fax, telephone, mobile, etc.)

Example:

John Smith
Executive Assistant
Phone: 555-1234
Fax: 555-2345

SECRETI/COMINTOREL TO USA, AUS, CAN, GBR, NZLxks-search-forms-p13-normal.gif:
Dictionary

Scanning

xks-search-forms-p14-normal.gif:
SECRET//COMINT//RELTO USA. AUS. CAN. GBR. NZL

Alert

Log of sessions tipped to TRAFFICTHIEF
Searchable fields

□ Target (strong selector)

■ Weight (confirmed/unconfirmed)

Other fields

Permutation that triggered the tip
(DECODEORDAIN)

□ Copy of XML document sent to TRAFFICTHIEF

SECRET//COMINT//RELTO USA, AUS, CAN. GBR, NZLxks-search-forms-p15-normal.gif:
SEC RET//COMINT//RE L

Category - DNI

V

Category hits from CADENCE and other
dictionaries

Searchable fields

■ Dictionary

■ Category

■ Keywords

Target (TRAFFICTHIEF)

SECRET//COMINT//RELTO USA, AUS, CAN, GBR, NZLxks-search-forms-p16-normal.gif:
SECRET//COMINT//RELTO USA. AUS. CAN. GBR. NZL



User Activity

SECREr//COMINT//REL TO USA, AUS, CAN, GBR, NZL

J

xks-search-forms-p17-normal.gif:
SECRET//COMINT7/RELTO USA, AUS, CAN, GBR. NZL

User Activity

Metadata from applications with a strong
selector-webmail, chat, webcam
Searchable fields

■ Active username (“search value”)

■ Activity - what the active user was doing

■ Attribute type - type of metadata

■ Attribute value - metadata value

□ Source - which plug-in provided the data

SECRET//COMINT//RELTO USA, AUS, CAN, GBR, NZLxks-search-forms-p18-normal.gif:
SECRET//COMINTORELTO USA. AUS. CAN, GBR. NZL

User Activity

Example:

Usemame Activity Source Attribute type Attribute value
badguy@yahoo viewFolder_webmail appproc app_provider Yahoo
badguy@yahoo viewFolder webmail appproc appjype webmail
badguy@yahco viewFolder_webmail appproc direction cl ent
badguy@yahoo viewFolder webmail appproc previous user user@yahoo
badguy@yahoo viewFolder_webmail appproc user_realn yahoo
badguy@yahoo viewFolder webmail appproc via squid/2.b
badguy@yahoo viewFolder_webmail appproc x-forwardedjp 10.0.123.45
badguy@yahoo viewFolder webmail appproc yahooBcoobadguy@yahoo viewholder_webmail appproc yahooGSS asdf1234asdf
asdf1234asdf viewFolder_webmail appproc user_realn yahooGSS
asdf1234asdf viewFolder webmail appproc yahoo badguy@yahoo

SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZLxks-search-forms-p19-normal.gif:
SECRET.7C0MINT//RELT0 USA. AUS. CAN. GBR. NZL

I

Document

Processingxks-search-forms-p20-normal.gif:
Extracted Files

Log of files transmitted as email
attachments, web uploads, etc.
Searchable fields
Filename
File extension
File type/MIME type

SECRET//COMINT//RELTO USA, AUS, CAN, GBR, NZLxks-search-forms-p21-normal.gif:
SECRET//COMINT//RELTO USA. AUS.

Document Tagging

¥

Document bodies and email bodies are labeled
with hits from a custom second-level dictionary
Idea: “embassy” by itself is not so interesting, but
inside a Word document, maybe it is
Searchable fields

■ Filename

■ Tech name (tag/category) - government, monetary,
proliferation, satellite, wireless, etc.

Tech value - word or phrase that hit
Note: also called Tech Strings search

SECRET//COMINT//RELTO USA, AUS, CAN, GBR, NZLxks-search-forms-p22-normal.gif:
SECRET//COMINT//RELTO USA, AUS. CAN, GBR. NZL

Document Metadata

Metadata from Office docs, videos, etc.
Searchable fields

- Filename and extension, document type

■ Author, organization

■ Language
Unique ID

■ Creation/modification timestamps

: Flash of the entire document and any embedded
images

SECRET//COMINT//RELTO USA, AUS, CAN, GBR, NZLxks-search-forms-p23-normal.gif:
PDF Metadata

Metadata from PDF documents
Searchable fields
■ Unique ID
Filename
Title

■ Author, creator, producer

■ Version

■ Language

Also available in Document Metadata
search

SECRET//COMINT//RELTO USA, AUS, CAN. GBR, NZLxks-search-forms-p24-normal.gif:
SECRET.7C0MINT//RELT0 USA. AUS. CAN. GBR. NZL

Protocol

Processingxks-search-forms-p25-normal.gif:
SECRET//COMINT7/RELTO USA. AUS. CAN, GBR. NZL

Blackberry

Id numbers and payload info from
Blackberry devices
Searchable fields

■ Source and destination PIN and BES

■ Direction

Payload type and encoding

SECRET//COMINT//RELTO USA, AUS, CAN, GBR, NZLxks-search-forms-p26-normal.gif:
SECRET//COMINT7/RELTO USA. AUS. CAN. GBR. NZL

Cellular DNI

Metadata from DNI over cellular modems

Searchable fields

■ IMSI. TMSI, IMEI, MCC, RAC, TLLI, etc.

■ Cell ID, Tunnel ID, Access point

■ Latitude, longitude

■ Spotbeam, direction

Limited deplyment - populated in SOTF by
front end

SECRET//COMINT//RELTO USA, AUS, CAN, GBR, NZLxks-search-forms-p27-normal.gif:
SECRET//COMINT//RELTO USA, AUS. CAN, GBR. NZL

Cisco Passwords

Logs Cisco router passwords
Searchable fields

Password

Decoded password (simple obfuscation with
known key)

SECRET//COMINT//RELTO USA, AUS, CAN, GBR, NZLxks-search-forms-p28-normal.gif:
HTTP Activity

Metadata from HTTP (WWW) traffic
Searchable fields

■ Host, URL file path, URL query string

■ Search terms - parsed from URLs for common search
providers (Google, Yahoo)

■ Language, character encoding

■ Referrer

■ User-Agent

■ Cookies

■ Server type (Apache, etc.)

■ Via - proxy info

■ Geolocation Info - e.g. city names from weather reports

SECRET//COMINT//RELTO USA, AUS, CAN, GBR, NZLxks-search-forms-p29-normal.gif:
Metadata from IKE (Internet Key
Exchange) sessions

Searchable fields

■ Version

■ Vendor ID

Encryption parameters - key length, field size,
group curve, etc.

■ Cookies
= Nonce

SECRET//COMINT//RELTO USA, AUS, CAN, GBR, NZLxks-search-forms-p30-normal.gif:
SECRET//COMIWT7/RELTO USA, AUS. CAN, GBR.

IRC Café Geolocation

QUIT messages from IRC - Internet cafés often
configure their IRC clients to advertise the café’s
street address

Searchable fields:

■ Username

■ Nick name

■ Café address (the QUIT message)

- Café IP

Example:

:nickname!~username@10.0.27.134 QUIT :Quit: MainStreet
Internet Cafe, 350 Main Street, P4 , 512M, Webcam, MP3,
128Kbps

SECRET//COMINT//RELTO USA, AUS, CAN, GBR, NZLxks-search-forms-p31-normal.gif:
SECRET//COMINT//RELTO USA, AUS. CAN, GBR. NZL

Passport detection

Y

Detect images of passports (code from R6)
OCR machine-readable information

Searchable fields

□ Original filename
Passport detection score

□ Info from machine readable area - name,
passport number, issuing state, DOB, expiration,
etc.

Under development

SECRET//COMINT//RELTO USA, AUS, CAN, GBR, NZLxks-search-forms-p32-normal.gif:
SECRET//COMINT//RELTO USA, AUS, CAN, GBR. NZL

Radius Logs

¥

Metadata from RADIUS sessions for dial-
up authentication and IP assignment
Searchable fields
Username
Phone number
■ IP address
- Account information

SECRET//COMINT//RELTO USA, AUS, CAN, GBR, NZLxks-search-forms-p33-normal.gif:
RBGAN

SECRET//COMINT7/RELTO USA, AUS, CAN, GBR. N2L

Metadata from RBGAN satellite internet
terminal collection
Searchable fields
Username
IMEI

□ Latitude and longitude
■ Spotbeam and direction

Limited deployment - populated by front
end

SECRET//COMINT//RELTO USA, AUS, CAN. GBR, NZLxks-search-forms-p34-normal.gif:
Metadata from RTP audio and video
sessions

Searchable fields
Payload type
. SSRC

Number of bytes and packets
Timestamps and sequence numbers
The RTP formatter in the session viewer
can decode certain payload types into
playable audio or video

SECRET//COMINT//RELTO USA, AUS, CAN, GBR, NZLxks-search-forms-p35-normal.gif:
Metadata from SIP (Session Initiation
Protocol) used for VoIP setup, etc.

Stored as multiple type-value pairs per
session

Searchable fields

3 Message type

□ Attribute type (call-id, content-type, from, to,
user-agent, via, etc.)

■ Attribute value

□ Subsession ID

SECRET//COMINT//RELTO USA, AUS, CAN. GBR, NZLxks-search-forms-p36-normal.gif:
SSL

SECRET//COMINT//RELTO USA, AUS, CAN. GBR. N2L

TTr

Metadata from SSL sessions
Searchable fields

■ Version

Encryption parameters - key length, modulus,
exponent, etc.

□ Signature info

■ Certificate info

SECRET//COMINT//RELTO USA, AUS, CAN. GBR, NZLxks-search-forms-p37-normal.gif:
TOR Log

r

Logs any identified TOR routers used for
anonymizing Internet traffic
Searchable fields

■ TOR from server

■ TOR to server

■ Router nickname

SECRET//COMINT//RELTO USA, AUS, CAN, GBR, NZLxks-search-forms-p38-normal.gif:
SECRET//COMINT//RELTO USA, A

Web File Transfer

Log of uploads and downloads from public file-
sharing sites (rapidshare, depositfiles, etc.)
Searchable fields

■ Filename
File size

■ Number of downloads

■ Uploader

- Username and password

Under development (GCHQ/MHS)

SECRET//COMINT//RELTO USA, AUS, CAN, GBR, NZLxks-search-forms-p39-normal.gif:
Log of X-Forwarded-For IP addresses and other

leaked public/private IP

Currently contains XFF plus leaked info from

STUN and Google Earth

Searchable fields

■ Internal from IP

■ Internal to IP

■ External from IP

■ External to IP

■ Source - plug-in that provided the info

■ Network path - chain of XFF addresses

SECRET//COMINT//RELTO USA, AUS, CAN, GBR, NZLxks-search-forms-p40-normal.gif:
//COMINT//RELTOI

Wireshark

Metadata from various protocols processed by the

wireshark library

Protocols

■ Routing - BGP, OSPF

■ VoIP - H225, Skinny, Clarent, Megaco, SCTP

■ Net management - SMB, SNMP

■ Tunneling - GTP

Searchable fields

■ Protocol

■ Field name

■ Field value

SECRET//COMINT//RELTO USA, AUS, CAN, GBR, NZLxks-search-forms-p41-normal.gif:
Metadata from WLAN collection
Searchable fields

■ Channel

■ SSID

■ BSSID

■ MAC addresses

■ Username

■ Private IP

■ Limited deplyment - populated in SOTF by front
end

SECRET//COMINT//RELTO USA, AUS, CAN, GBR, NZLxks-search-forms-p42-normal.gif:


SECRET//COMINT//RELTO USA. AUS. CAN. GBR. NZL

Miscellaneousxks-search-forms-p43-normal.gif:
Call Logs

DNR metadata from JUGGERNAUT,
CERF, FASCIA, DURT, etc.

Searchable fields
Phone numbers
■ Signaling type
OPC, DPC, CIC, IMSI

Limited deployment

SECRET//COMINT//RELTO USA, AUS, CAN, GBR, NZLxks-search-forms-p44-normal.gif:
SECRET//COMINTWRELTO USA, AUS. CAN, GBR. NZL

Network Logs

Network metadata from MOONSHINE logs
Searchable fields

- Net type
ESSID, BSSID
■ Channel

■ Carrier

■ Latitude and longitude

Limited deployment

SECRET//COMINT//RELTO USA, AUS, CAN, GBR, NZLxks-search-forms-p45-normal.gif:
CNE

SECRET//COMINT//RELTO USA. AUS. CAN. GBR. NZL

/.WlliPl r

CNE data from TAO
Searchable fields
Project name
■ Collection technique
Filename and extension

Limited deployment (xks-cne)

SECRET//COMINT//RELTO USA, AUS, CAN, GBR, NZLxks-search-forms-p46-normal.gif:
SECRET//COMINT//RELTO USA, AUS, CAN. GBR. N2L

Registry

Windows registry data from TAO (CNE)
Searchable fields
■ Collection technique
Hive

Key, subkey, value

Limited deployment (xks-cne)

SECRET//COMINT//RELTO USA, AUS, CAN, GBR, NZLxks-search-forms-p47-normal.gif:
Custom Search
Formsxks-search-forms-p48-normal.gif:
SECRET//COMINT7/RELTO USA, AUS, CAN, GBR. NZL

Simple Search

Simple way to search for usernames, IP
addresses, and machine ID cookies

Just enter your search term and select
what type of thing it is, and the form sends
it to User Activity or HTTP Activity as
appropriate

SECRET//COMINT//RELTO USA, AUS, CAN, GBR, NZLxks-search-forms-p49-normal.gif:
Problem: XKS may have info about
“badguy@yahoo” in Email Addresses, User
Activity, Logins & Passwords, etc.

Solution: submit multiple searches from a single
form

Enter the username and select which databases
to search, and the form translates that into the
proper queries

Similar MultiSearches for IP addresses and MAC
addresses

Optional: merge results into one table

SECRET//COMINT//RELTO USA, AUS, CAN, GBR, NZL


Download Document

XKEYSCORE Search Forms (xks-search-forms-p1-normal.gif)

Download Document

XKEYSCORE Search Forms (xks-search-forms-p2-normal.gif)

Download Document

XKEYSCORE Search Forms (xks-search-forms-p3-normal.gif)

Download Document

XKEYSCORE Search Forms (xks-search-forms-p4-normal.gif)

Download Document

XKEYSCORE Search Forms (xks-search-forms-p5-normal.gif)

Download Document

XKEYSCORE Search Forms (xks-search-forms-p6-normal.gif)

Download Document

XKEYSCORE Search Forms (xks-search-forms-p7-normal.gif)

Download Document

XKEYSCORE Search Forms (xks-search-forms-p8-normal.gif)

Download Document

XKEYSCORE Search Forms (xks-search-forms-p9-normal.gif)

Download Document

XKEYSCORE Search Forms (xks-search-forms-p10-normal.gif)

Download Document

XKEYSCORE Search Forms (xks-search-forms-p11-normal.gif)

Download Document

XKEYSCORE Search Forms (xks-search-forms-p12-normal.gif)

Download Document

XKEYSCORE Search Forms (xks-search-forms-p13-normal.gif)

Download Document

XKEYSCORE Search Forms (xks-search-forms-p14-normal.gif)

Download Document

XKEYSCORE Search Forms (xks-search-forms-p15-normal.gif)

Download Document

XKEYSCORE Search Forms (xks-search-forms-p16-normal.gif)

Download Document

XKEYSCORE Search Forms (xks-search-forms-p17-normal.gif)

Download Document

XKEYSCORE Search Forms (xks-search-forms-p18-normal.gif)

Download Document

XKEYSCORE Search Forms (xks-search-forms-p19-normal.gif)

Download Document

XKEYSCORE Search Forms (xks-search-forms-p20-normal.gif)

Download Document

XKEYSCORE Search Forms (xks-search-forms-p21-normal.gif)

Download Document

XKEYSCORE Search Forms (xks-search-forms-p22-normal.gif)

Download Document

XKEYSCORE Search Forms (xks-search-forms-p23-normal.gif)

Download Document

XKEYSCORE Search Forms (xks-search-forms-p24-normal.gif)

Download Document

XKEYSCORE Search Forms (xks-search-forms-p25-normal.gif)

Download Document

XKEYSCORE Search Forms (xks-search-forms-p26-normal.gif)

Download Document

XKEYSCORE Search Forms (xks-search-forms-p27-normal.gif)

Download Document

XKEYSCORE Search Forms (xks-search-forms-p28-normal.gif)

Download Document

XKEYSCORE Search Forms (xks-search-forms-p29-normal.gif)

Download Document

XKEYSCORE Search Forms (xks-search-forms-p30-normal.gif)

Download Document

XKEYSCORE Search Forms (xks-search-forms-p31-normal.gif)

Download Document

XKEYSCORE Search Forms (xks-search-forms-p32-normal.gif)

Download Document

XKEYSCORE Search Forms (xks-search-forms-p33-normal.gif)

Download Document

XKEYSCORE Search Forms (xks-search-forms-p34-normal.gif)

Download Document

XKEYSCORE Search Forms (xks-search-forms-p35-normal.gif)

Download Document

XKEYSCORE Search Forms (xks-search-forms-p36-normal.gif)

Download Document

XKEYSCORE Search Forms (xks-search-forms-p37-normal.gif)

Download Document

XKEYSCORE Search Forms (xks-search-forms-p38-normal.gif)

Download Document

XKEYSCORE Search Forms (xks-search-forms-p39-normal.gif)

Download Document

XKEYSCORE Search Forms (xks-search-forms-p40-normal.gif)

Download Document

XKEYSCORE Search Forms (xks-search-forms-p41-normal.gif)

Download Document

XKEYSCORE Search Forms (xks-search-forms-p42-normal.gif)

Download Document

XKEYSCORE Search Forms (xks-search-forms-p43-normal.gif)

Download Document

XKEYSCORE Search Forms (xks-search-forms-p44-normal.gif)

Download Document

XKEYSCORE Search Forms (xks-search-forms-p45-normal.gif)

Download Document

XKEYSCORE Search Forms (xks-search-forms-p46-normal.gif)

Download Document

XKEYSCORE Search Forms (xks-search-forms-p47-normal.gif)

Download Document

XKEYSCORE Search Forms (xks-search-forms-p48-normal.gif)

Download Document

XKEYSCORE Search Forms (xks-search-forms-p49-normal.gif)

e-Highlighter

Click to send permalink to address bar, or right-click to copy permalink.

Un-highlight all Un-highlight selectionu Highlight selectionh