Title: XKEYSCORE Application IDs

Release Date: 2015-07-01

Description: This undated NSA presentation gives a technical introduction to the treatment of AppIDs within XKeyScore: see the Intercept article XKEYSCORE: NSA’s Google for the World’s Private Communications, 1 July 2015.

Document: xks-application-ids-brief-p1-normal.gif:
Classification: TOP SECRET//COMIIMT//ORCON//REL TO USA, FVEY//20291123

Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291123

M - 1 * * ' ~ . 1a— Ætl MK

xks-application-ids-brief-p2-normal.gif:
Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291

Basic Syntax

Syntax:

• Very C-like.

function( 'name', level, ) = 'search terms and pattern';

Two valid search functions appid and fingerprint:

• appid('chat/icq', 8.5, wireshark='icq', chatproc=‘ICQ’) =

/[*o]icq/c and $icq;

• fingerprint('fingerphnt/phone/nokia/generic', 7.0) =
'user-agent: nokia' or
'profile: ';

Classification: TOP SECKET//COMINT//ORCON//REL TO USA, FVEY//20291123xks-application-ids-brief-p3-normal.gif:
Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//2

Naming Conventions



XKS Appid's are named using a
pseudo directory convention.

/applica tion_ type/sub_ type/name

Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291123xks-application-ids-brief-p4-normal.gif:
Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291

Levels

Levels are 1.0 - 9.9 with lower numbers having a higher priority. This
allows multiple signatures to match a piece of traffic and only the
most specific appid will be applied. An example might be:

9.9 Yahoo

9.8 Yahoo/chat

9.7 Yahoo/chat/incoming

Since the Yahoo/chat/incoming has the lowest level, the traffic will be
labeled as yahoo/chat/incoming

Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291123xks-application-ids-brief-p5-normal.gif:
Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//202911:

Basic Search Patterns

XKEYSCORE supports Boolean operations and regular
expressions

Raw text must be encapsulated between single quotes

• 'search term'

Terms can be combined with Boolean logic

• 'search term' and 'another term'

• 'search term' or 'another term'

Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291123xks-application-ids-brief-p6-normal.gif:
Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291123

Binary and Reaex Patterns

Binary patterns can be represented by putting a \x in front of each binary
value

• '\xff\xff\x00\x02'

Note: Unlike C, no double back slashing required
/regex/



Classification: TOP SECRET//COMINT//ORCON//REL TO USA. FVEY//20291123xks-application-ids-brief-p7-normal.gif:
Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291

CHAINWORDs

- ... ................................—

You can assign a pattern to a variable (CHAINWORD) and reuse
the variable in many patterns.

• $sip = 'via: sip' and 'cseq:' and 'SIP/2'c;

Now we can use this variable in future definitions:

• appid('voip/sip‘, 7.2 ) = $sip;

• appid('voip/sip/invite\ 6.9) = $sip and 'INVITE';

Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291L23xks-application-ids-brief-p8-normal.gif:
Classification: TOP 5ECRET//COMINT//ORCON//REL TO USA, FVEY//2029112

Built in functions

ip( expr ) Matches against an IP Address looks in to address and from address in the session headere • ip( '10.10.10.1');
toport( expr ) Matches against the Destination/To port. Note this must be a numeric representation of a port. • tOport( 1920 );
fromport( expr ) Matches against the Source/From port. Note this must be a numeric representation of a port. • fromport( 80 );
port( expr ) Matches against the either port. Note this must be a numeric representation of a port. •port( 6667 );
next_protocol( expr) Matches against the integer version of the next protocol. • next_protocol( 250 );
protocol ('text') Will only work for IP next protocol names as defined in the IANA next protocol numbers document • protocol (’tep');

Classification: TOP SECRET//COMINT//ORCON//REL TO USA. FVEY//20291123xks-application-ids-brief-p9-normal.gif:
Classification: TOP 5ECRET//COMINT//ORCON//REL TO USA, FVEY//202911

Built in functions

email address(sel) permutes just like strong selector (just like

DECODEORDAIN

mac_address(addr) Tasks a mac address

smac(addr)

dmac(addr)

ip(addr) tasks this IP address (either to or from)

fromjp(addr) tasks this IP address only when it is the originator

tojp(addr) tasks this IP address only when it is the destination

Classification: TOP SECRET//COM INT//ORCON//REL TO USA, FVEY//20291123xks-application-ids-brief-p10-normal.gif:
Classification: TOP 5ECRET//COMINT//ORCON//REL TO USA, FVEY//2029112:



More built in functions

first(expr) Matches against a pattern at the beginning of the session
Ipos(expr) Matches against a pattern at the beginning of each line (\n)
pos( expr ) expression occurs at offset X in the session • pos('Heilo') == 5, • pos(/Good.*Grief/) <= 10
between( expr ) • betwccnCHclIo', 'World', 10, 100} Separation between 'Hello' and 'World' is greater than or equal to 10 bytes and less than or equal to 100 bytes This is the same as using the following regular expression: • /Heiio.'term'c Docs a case sensitive match of the term
'term'u Treats the term as UTF-16

Classification: TOP SECRET//COMINT//ORCON//REL TO USA. FVEY//20291123xks-application-ids-brief-p11-normal.gif:
Classification: TOP 5ECRET//COMINT//ORCON//REL TO USA, FVEY//20291



Predefined Chainwords

There are a number of chainwords predefined for convenience:

$tcp

$udp

$icmp

$sctp

$rpc

$arp

$ssl

$http_cmd

$http

$http_get

$http_put

$http_post

• shttp delete

• shttp trace

• $http_head

• Shttp_options

• $http_partial

• Svbulletin

• Smime_type

• $user_agent

Classification: TOP SECRET//COM INT//ORCON//REL TO USA. FVEY//20291123xks-application-ids-brief-p12-normal.gif:
Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291

Example________________________________________

appid('voip/sip/IMS', 6.0, wireshark='sip') =

(’via: sip' or 'v: sip') and ’cseq:’ and (
'p-access-network-info:' or
'p-called-party-id:' or
’p-charging-vector:’ or
’p-charging-vector-addresses:’ or
’p-media-authorization:’ or
’security-verify:’ or
’security-server:’ or
’security-client:' or
'service-route:' or
'record-route:' and 'pcscf or
'record-route:' and 'scscf' or
’contact:' and 'pcscf or
'contact:' and 'scscf or
'proxy-authorization:' and 'pcscf or
’proxy-authorization:’ and 'scscf' or
'path:' and 'pcscf or
’path:' and 'scscf
);

Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291123xks-application-ids-brief-p13-normal.gif:
Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291:

Example________________________________________

rr

appid('voip/skinny/keep-alive', 3.0, wireshark='skinny') =
toport(2000) and

first('\x04\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00')

appid('voip/skinny/keep-alive-ack', 3.0, wireshark=’skinny') =
fromport(2000) and

first('\x04\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00')

appid('voip/skinny(port2000)', 9.9, wireshark='skinny') =
port(2000);

Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291123xks-application-ids-brief-p14-normal.gif:
Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291:

Example

appid('chat/yahoo', 6.0, chatproc='Yahoo') =

(('YCHT'c and Syahoo chat) or first('YCHT'c)) and not
port(5050);

appid('chat/icq', 8.5, wireshark=’icq', chatproc='ICQ') =
/[Ao]icq/c and Sicq;

appid('chat/icq', 9.0, wireshark^'icq', chatproc='ICQ') =
first('icq') and not port(25);

fingerprint(’encryption/moujahedeen', 7.0) =

'begin+gimf+asrar+el+moujahedeen' or
'begin gimf asrarel moujahedeen’;

Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291 123xks-application-ids-brief-p15-normal.gif:
Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291

Example

appid(,mail/smtp/to_server'/ 8.5, direction = $from_server,
wireshark^smtp’) =
toport(25) and
( first(’helo') or
first('ehlo’) or
first('data') or

(lpos('To: 'c) and lpos('From:'c)) or
IposCQUIT'c) or
lpos('mail from:’) or
lpos('rcpt to:') );

Classification: TOP SECRET//COM INT//ORCON//REL TO USA, FVEY//20291123xks-application-ids-brief-p16-normal.gif:
Classification: TOP 5ECRET//COMINT//ORCON//REL TO USA, FVEY//20291123

Example

$gmail = 'D=(top.js&&top.)S.imt)?function(d){top.js.P(window'c or <br />first(’POST /gmail'c) or <br />first('GET /gmail'c) or <br />'GMAIL_AT=’c or <br /> <br />/SID = [A-Za-zO-9\-\_]{87> = ;Domain=\.google\.com/c or <br /> <br />'GMAIL_STAT='c or <br /> <br />'[[V'ctV'c or <br /> <br />'S^gmail^c or <br /> <br />'ain=\'mail.google.com'c or <br /> <br />'<title>Gmairc or <br /> <br />'GMAIL_RTT='c or <br /> <br />'GMAIL_LOGIN='c or <br /> <br />'\nServer: GFE/'c; <br /> <br />appid(’mail/webmail/gmair, 8.0, webproc='Gmair) = <br /> <br />$gmail; <br /> <br />Classification: TOP SECRET//COM INT//ORCON//REL TO USA. FVEY//20291123xks-application-ids-brief-p17-normal.gif: <br />Classification: TOP 5ECRET//COMINT//ORCON//REL TO USA, FVEY//20291 <br /> <br />Append Option <br /> <br /># append the mime_type and HTML title to any of these applds.. <br />PARAMS = append=$mime type, append2=$http info, <br />append3=$doc_title; <br /> <br />$web = "web"; <br /> <br />appid('http/proxy_to_server', 9.1, $web, direction=$proxy_to_server) <br />$webproxy to server; <br /> <br />appid('http/proxy to client', 9.1, $web, direction=$proxy to client) = <br />$webproxy_to_client; <br /> <br />Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291123xks-application-ids-brief-p18-normal.gif: <br />Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291: <br /> <br />Tvoe Option <br /> <br />Third parameter is the type; if missing, it takes up to the first slash as <br />the type <br /> <br />appid('http/response’, 9.2, $web) = <br /> <br />$http and <br /> <br />not ('x-cache' or 'x-forward' or 'get/' or <br />'post /' or 'get http' or 'post http'); <br /> <br />appid('http/response/partial', 9.1, $web) = <br /> <br />$http and $http_partial; <br /> <br />Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291L23xks-application-ids-brief-p19-normal.gif: <br />Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291 <br /> <br />Appid utility___________________________________________ <br /> <br />r appid options: --help --list-all this help message list all the application/fingerprint names and levels <br /> <br />—list-appids list all the application names (no fingerprints) <br /> <br />-list-fingerprints list all the application names (no appids) <br />-list-types list all the application types <br /> <br />—list-levels -unit-test list all the application levels perform unit tests with data in the heirachy 'datadir', with files matching 'filespec' <br />—quiet don't print any load messages <br /> <br />—appid_fname arg location of appid.cfg <br /> <br />-input-file arg input file to test <br /> <br />—datadir arg The test data directory. Defaults to <br /> <br />$(XSCORE_TEST_DATA DIR)/appids <br />—filespec arg ( = .*\.ul24) A regular expression to match against files to <br />check <br /> <br />—noexit arg (=0) do not stop on the first error <br /> <br />Classification: TOP SECRET//COMINT//ORCON//REL TO USA. FVEY//20291123xks-application-ids-brief-p20-normal.gif: <br />HOOt 101 <br />1001 I OOI <br /> <br />I 1001 tool I <br /> <br />Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291123 <br /> <br />id Validation <br /> <br />appic sample.ul24 <br />Loading appds <br /> <br />->Loading : /home/oper/xkeyscore/config/dictionarie$/appid/appid_definition$.cfg <br />->Loading : /home/oper/xkeyscore/config/dictionaries/appid/anorymizer.appid <br />->Loading : /home/oper/xkeyscore/config/dictionaries/appid/bulletin_board.appid <br />->Loading : /homc/oper/xkcyscorc/conflg/dlctfonarles/appid/tao vpn.appld <br />->Loading : /home/oper/xkeyscore/config/dictionaries/appid/tcmolp.appid <br />->Loeding : /home/cper/xkeyscore/config/dictionaries/appid/terminal.appid <br />->Loading : /home/cper/xkeyscore/config/dictionanes/appid/voip.appid <br />->Loading : /home/oper/xkeyscore/config/dictionaries/appid/appid_definitions.cfg <br />Finished loading appids <br />Filename: sample.ul24 <br /> <br />Appid: encryption/https <br /> <br />Total Size: 19.36Kbits <br />Total Time: O.Olsecs <br />Rate: 1.936Mbits/s <br />Overall performance: <br /> <br />Total Time: O.Olsecs <br />Total Bits: 0.01936Mbits <br />Overall Rate: 1.936Mbits/s <br /> <br />Classification: TOP SECRET//COM INT//ORCON//REL TO USA. FVEY//20291123xks-application-ids-brief-p21-normal.gif: <br />Classification: TOP 5ECRET//C0MINT//0RC0N.//REL TO USA, FVEY//20291 j 23 <br /> <br />- -. R <br /> <br /><entry> <br /> <br /><Lype> bcolear: </Lype> <br /> <br /><name> cbar./l rc </name> <br /> <br /><oar.«gory> applicat.ion_i <br /><infcrciiation> <br />application type> chat <br /><level> $.1 </lev®l> <br /> <br /><action> tipplicc.tion_id <br /></iriiortnaLlon> <br /> <br /><select> <br /> <br />KW-' <POS=C>nick' +• <br /> <br />KW= * < POS=C >isen1 -t- <br />KW =' <E’OS=C>wbois ’ + <br /> <br />KW-'<P05-C>isirc’+ <br /> <br />KW-'CPOS-Oiro' l <br />kw- ' <POS-C>join ’ +■ <br /> <br />KW-' <E'OS-C>aul.h ' + <br /> <br />KW-'<POS~C>crypt’+ <br /> <br />KW=' <.Pi)3=C>pir.g' i <br />KW=' <fc>OS=C>poi:g • * <br /> <br />KW-'<POS-C>privmsg1+ <br />KW='<POS=C>notice awth' + <br /> <br />KW=' ire * &KK= * privmsg ’ &KVi=' net ice1 + <br />KW=*error :your <br /></select> <br /> <br /><ceseiect> KW-’ap COl’-O'SKW-'user' <br /></encry> <br /> <br />appidi’cha^/lrc', 8.5, wireshark-'ire*, ch&Lproc-1IRC') - <br />‘orivmgg *; <br /> <br />appid(’chat/irc*, 8.L, wircsliark-'ire', eh£tproc='IRC'3 = <br />r.ot {port (110} ar.d 'user'] and <br />( f' rsr. (’rri c< ') c r first ('i son ') cr <br />first('whois ') cr first(’isirc ') or <br />l!rs_(’irc ') cc first(’join ') or <br />first(’auth ’) cr first ('crypt ') or <br />first (’ping ') cr first Cpor.g 1) or <br />first('privmsg '5 or <br />firs-('no-ice aalh’) or <br />l1 i rc1 a~id 1 privmgg ’ and ’notice 1 ) <br />or ’error :yoi:' <br /> <br />) ; <br /> <br />:/deselect-- <br /> <br />Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291123


Download Document

XKEYSCORE Application IDs (xks-application-ids-brief-p1-normal.gif)

Download Document

XKEYSCORE Application IDs (xks-application-ids-brief-p2-normal.gif)

Download Document

XKEYSCORE Application IDs (xks-application-ids-brief-p3-normal.gif)

Download Document

XKEYSCORE Application IDs (xks-application-ids-brief-p4-normal.gif)

Download Document

XKEYSCORE Application IDs (xks-application-ids-brief-p5-normal.gif)

Download Document

XKEYSCORE Application IDs (xks-application-ids-brief-p6-normal.gif)

Download Document

XKEYSCORE Application IDs (xks-application-ids-brief-p7-normal.gif)

Download Document

XKEYSCORE Application IDs (xks-application-ids-brief-p8-normal.gif)

Download Document

XKEYSCORE Application IDs (xks-application-ids-brief-p9-normal.gif)

Download Document

XKEYSCORE Application IDs (xks-application-ids-brief-p10-normal.gif)

Download Document

XKEYSCORE Application IDs (xks-application-ids-brief-p11-normal.gif)

Download Document

XKEYSCORE Application IDs (xks-application-ids-brief-p12-normal.gif)

Download Document

XKEYSCORE Application IDs (xks-application-ids-brief-p13-normal.gif)

Download Document

XKEYSCORE Application IDs (xks-application-ids-brief-p14-normal.gif)

Download Document

XKEYSCORE Application IDs (xks-application-ids-brief-p15-normal.gif)

Download Document

XKEYSCORE Application IDs (xks-application-ids-brief-p16-normal.gif)

Download Document

XKEYSCORE Application IDs (xks-application-ids-brief-p17-normal.gif)

Download Document

XKEYSCORE Application IDs (xks-application-ids-brief-p18-normal.gif)

Download Document

XKEYSCORE Application IDs (xks-application-ids-brief-p19-normal.gif)

Download Document

XKEYSCORE Application IDs (xks-application-ids-brief-p20-normal.gif)

Download Document

XKEYSCORE Application IDs (xks-application-ids-brief-p21-normal.gif)

e-Highlighter

Click to send permalink to address bar, or right-click to copy permalink.

Un-highlight all Un-highlight selectionu Highlight selectionh