Title: XKEYSCORE System Administration

Release Date: 2015-07-01

Document Date: 2012-12-01

Description: This 143-page NSA presentation from December 2012 provides a general introduction to the architecture, operation and administration of XKeyScore: see the Intercept article XKEYSCORE: NSA’s Google for the World’s Private Communications, 1 July 2015.

Document: xks-system-administration-p1-normal.gif:
TOP SECRET II SI II REL TO USA. AUS. CAN. GBR. NZL

December 2012

t-—v.—r----------------

TOP SECRET // Si II REL TO USA, AUS, CAN, GBR, NZLxks-system-administration-p2-normal.gif:
TOP SECRET II Si II REL TO USA, AUS, CAN. GBR, NZL

2xks-system-administration-p3-normal.gif:
^Introduction to XKEYSCORE
VPurpose and Capabilities
VData Flow

VWhat is a Cluster?
VXKEYSCORE Databases

TOP SECRET II $1 II REL TO USA, AUS, CAN. GBR, NZLxks-system-administration-p4-normal.gif:
TOP SECRET // SI II REL TO USA. AUS. CAN. GBR. NZL

■ XKEYSCORE performs filtering and selection
to enable analysts to quickly find information
they need based on what they already know.

■ XKEYSCORE also performs SIGDEV
functions such as target development to allow
analysts to discover new sources of
information.xks-system-administration-p5-normal.gif:
■ XKEYSCORE processes data at field sites,
where it is collected, and allows analysts from
all over the world to query it.

■ At field sites, the XKEYSCORE software can
run in clusters of few or many servers, giving it
the ability to scale in both processing power
and storage.

■ All processing is plugin or fingerprint based,
which allows new capabilities to be quickly
deployed to support operational needs.

TOP SECRET IIJ

I REL TO USA, AUS; CAN. GBR, NZLxks-system-administration-p6-normal.gif:
■ XKEYSCORE is a Computer to Computer
(C2C) exploitation system.

■ It is a fully distributed processing and query
system.

■ XKEYSCORE can run on multiple servers.

■ Plugin and fingerprint architecture allows new
capabilities to be quickly deployed.xks-system-administration-p7-normal.gif:
■ XKEYSCORE is typically Installed with Red Hat
AS5u8 operating system. The suggested disk set up
is:

• Set up separate partitions for / (root), /var,

/tmp, and lexport/data

m XKEYSCORE clusters can be composed of three
different functionalities, which are:

• One host acts as the web server/user interface, etc...

• Another host normally runs as the real-time processing unit

• Other host acts as the search or query system.

■ Hybrid system can perform multiple roles on one
server, which enables efficient registration.

• process_data_parent

• 1 query_proc

TOP SECRET II $1II REL TO USA, AUS CAN. GBR, Ixks-system-administration-p8-normal.gif:
TOP SECRET II SI II REL TO USA. AUS. CAN. GBR. NZL

Data Flow (High-level)

ZXr

The backend is where the raw data for
XKEYSCORE is processed; that is, we
receive information from our sources (e.g.
WEALTHYCLUSTER2), process it, and store
it into a database.

TOP SECRET II SI II REL TO USA, AUS, CAN. GBR, NZLxks-system-administration-p9-normal.gif:
■ A cluster is comprised of one master server
and one or more slaves.

■ All slaves in a cluster have their own copy of
configurations (/opt/xkeyscore/config) files via
the xks rsync push_config cronjob.xks-system-administration-p10-normal.gif:
Data Flow - Databases

There are two types of databases on an
XKEYSCORE system: insert (¡0) and query
(qO)

i pra«ts„tart

// \




Î ■ >! iO ' * qO
^ ^

NOTE: sotfjnput proc is now called, sotf dlst

processdataN’s are now called, process data parent

TOP SECRET II Si II REL TO USA, AUS, CAN. GBR, NZL

10xks-system-administration-p11-normal.gif:


■ file_input_proc and sotf_dist take in sessions
from the front-end and load balances them
across multiple process_data_parent’s.

■ process_data_parent is responsible for
processing sessions and extracting metadata

■ xksmetaingester takes the metadata from
the process_data_pa rent’s and writes it to the
insert database, ¡0

■ register_metadata_tables takes completed
insert tables, indexes them, and moves them
to the query database, qO

TOPSECRET/

I REL TO USA, AUS, <

11xks-system-administration-p12-normal.gif:
TOP SECRET II Si II REL TO USA, AUS CAN. GBR, NZL

12xks-system-administration-p13-normal.gif:
Lesson Objectives

V Operating System Services

-«/HTTPD

VMYSQL
v* NFS
VAUTOFS

V Mount Points

V/xks_data

V Directory Structure

TOP SECRET II $1II REL TO USA, AU$, CAN. GBR, NZLxks-system-administration-p14-normal.gif:
IIIREL TO USA. AUS. CAN. GBR. NZL

Operating System Services

■ XKEYSCORE is typically installed on servers
running Red Hat 5u8 operating system.

■ This section discusses common operating
system services used during XKEYSCORE
operation.

14xks-system-administration-p15-normal.gif:
■ The http daemon is needed for the web-based
GUI, viewing content, and is required on all
servers.

■ The master server is the web server and the
slaves retrieve content through HTTPS.

TOP SECRET Hi

15xks-system-administration-p16-normal.gif:
■ The mysql daemon is a SQL-based database
server for processing, querying, and is needed

for the XKEYSCORE GUI.

■ It is required on all servers for administration,
processing, and querying metadata in
databases.

TOP SECRET // $1 II REL TO USA, AU$, CAN. GBR, NZL

16xks-system-administration-p17-normal.gif:
■ Mounting a directory uses the NFS service.

■ NFS allows file systems that physically reside
on one computer to be shared by other
computers on the network.

■ The machine with the hardware containing the
directory must allow the hardware to be made
available to other machines.

■ Required on all computers for clustering.

TOP SECRET It $1II REL TO USA, AU$, TOP SECRET // SI II REL TO USA. AUS. CAN. GBR. NZL

■ /etc/exports

• /export/data/xkeyscore master(rw) slave(rw)

• /opt/xkeyscore/config/loadserver * *(rw)xks-system-administration-p19-normal.gif:
■ Computers requiring shared access to the
/export/data/xkeyscore directory must be told
where to find the directory.

• This is accomplished via automounting.

■ The autofs daemon listens for computers
trying to connect to the directories, or mounts,
that it is responsible for.

■ The mounts are dropped after a time out, but
autofs remounts the drive when drives need
to be accessed.

TOP SECRET II $1II REL TO USA, AU$, CAN. GBR, NZL

19xks-system-administration-p20-normal.gif:
■ For a clustered XKEYSCORE, automounts
must be set up on all of the computers in the
cluster.

■ auto.master and auto.data files in the /etc
directory must be edited or created.

■ When finished, the mounted directories on the
remote machines can be accessed.

■ The oper account should have full read/write
permissions on all shared drives.

TOP SECRET//J

.TO USA, AUS,<

20xks-system-administration-p21-normal.gif:
OP SECRET //SI IIREL TO USA. AUS. CAN. GBR. NZL

■ auto.master - designates mount points on the local
computer and the directory to mount on the remote
server.

• Example:

■ auto.data - enables all servers to see the
/export/data/xkeyscore directory on other machines
and locate databases, archived, data, and
MAILORDER directory.

• Example:

► xks1 -rw,soft,intr,tcp xks1 :/export/data/xkeyscore

► xks2-rw,soft,intr,tcp xks2:/export/data/xkeyscore

► /xks_data

/etc/auto.data ~timeout=60

TOP SECRET II $1II REL TO USA, AU$, CAN. GBR, NZL

21xks-system-administration-p22-normal.gif:
Directory Structure

/opt/xkeyscore/ - contains all of the
XKEYSCORE software. Software includes
the GUI, processing, scripts, and
configurations.

• bashrc - XKEYSCORE environment variables
file.

• beacon/ - contains the beacon perl script
(shm_beacon.pl) and a link to the beacon
configuration file (shm_beacon.config).

• bin.shells/ and bin.shells/sysadmin - contains
miscellaneous bash, python, and C shell scripts.

• build/ - contains libraries and plug-ins.

• install/ - contains installation scripts.

TOP SECRET II $1II REL TO USA, AU$, CAN. GBR, NZL 22xks-system-administration-p23-normal.gif:
Directory Structure

/opt/xkeyscore/config/ - consists of sub-directories
and each contain configuration files for building and
running XKEYSCORE.

• crontab/ - contains the master and slave crontab file.

• dictionaries/ - contains the dictionary files for the filtering,
selection, TRAFFICTHIEF, CADENCE, fist tables, and any
other local dictionaries.

• misc/ - contains miscellaneous per-plug-in configuration
files, (i.e. sotf_input_proc.xml).

• plugins/ - contains event handler configuration files for
each of the plugins (default.xml).

• www/- contains web configuration files and xscore.cfg.

• SERVICE/ - contains the config files for all the services
needed by XKEYSCORE (httpd, php, mysqld, etc.)

TOP SECRET II Si II REL TO USA, AUS, CAN. GBR, NZL

23xks-system-administration-p24-normal.gif:
Directory Structure

ir

■ /opt/xkeyscore/www/ - contains the contents
of the web front end.

• docs/ - contains documents viewable through the
XKSGUI.

• html/ - contains web pages and scripts that are
not on the secure server.

• secured/ - contains web pages and scripts that
are on the secure server including:

► crons/ - location of cron job scripts

• src/ - contains source code for the XKS GUI.xks-system-administration-p25-normal.gif:
TOP SECRET II SI IIREL TO USA. AUS. CAN. GBR. NZL

¥

• /export/data/xkeyscore/ - is used for both internal
databases and metadata archive databases, input,
output, and archiving of data.

• archives/ - (optional) destination for processed content

• inputs/ - (optional) used for file based input

• mysql/ - location of the MySQL database consisting of
admin, insert, and query databases.

• outputs/ - (optional) contain the following sub-directories:

► mailorder/ - pickup point

► mailorder_working/ - file creation point before being moved to
mailorder/

TOP SECRET II Si II REL TO USA, AUS, CAN, GBR, I

25xks-system-administration-p26-normal.gif:
)P SECRET//SI//REL

. TO USA. AUS. CAN, GBR. NZL

7

■ /xks_data/ - logical mount point for all other
XKEYSCORE (including itself)
/export/data/xkeyscore.

• / - mount point for the hostname’s
local directory /export/data/xkeyscore (referenced
by host name).

► All servers must export their /export/data/xkeyscore
directory and mount this on the / directory
for each hostname of each machine, including itself.

TOP SECRET // $1II REL TO USA, <

, AUS, CAN. GBR, I

26xks-system-administration-p27-normal.gif:
TOP SECRET II Si II REL TO USA, AUS, CAN. GBR, NZL

27xks-system-administration-p28-normal.gif:
TOP SECRET II SI IIREL TO USA. AUS. CAN. GBR. NZL

■»/ Accessing the GUI

VExiting a Session
-/ Main Menu Bar

V MyXKS

V Admin

-/•Computer Resources Option
VStart and Stop Processing
-»/Run a Process Manually

V Users
-/ Search

•«/ Workflow Central
-/ Results
-/ Fingerprints

TOPSECRET/

I $1II REL TO USA, AUS, CAN. GBR, NZL

28xks-system-administration-p29-normal.gif:
DP SECRET II SI IIREL TO USA. AUS. CAN. GBR. NZL

fit*iMKIM:

ZXr

7

■ In the address field of a web browser, type
https://.
m PKI’s or a UserlD and password are required.
After successfully launching a new session,
the XKEYSCORE WELCOME window
appears.

• Note: Compatible web browsers for XKEYSCORE
version 1.5 are:

► Internet Explorer is not supported

► Firefox/3.0.* and above

TOP SECRET/

I $1II REL TO USA, AUS, CAN. GBR, t

29xks-system-administration-p30-normal.gif:
>10 01

TOP SECRET II SI II REL TO USA. AUS. CAN. GBR. NZL

Accessing the GUI



TOP SECRET II Si II REL TO USA, AUS CAN. GBR, NZL

30xks-system-administration-p31-normal.gif:
)P SECRET II SI IIREL TO USA, AUS. CAN. GBR. NZL



■ The main menu bar across the top of the
window has menus that, when selected, each
has additional options available in a drop
down menu form.

XHySS Xftta ¿to *\Swch fttukHwCdlril .touts $Firgeipnnts ;Elaggng ^SMistts iEastnj ©Hip

TOPSECRET/

I $1II REL TO USA, AUS, CAN. GBR, f

31xks-system-administration-p32-normal.gif:
TOP SECRET II SI II REL TO USA. AUS. CAN. GBR. NZL

I OPTION DESCRIPTION
Home Returns to the main page
MyXKS Can edit user settings, disable/enable access to databases, edit a search form search setting, and restore default settings.
Admin Computer resources. Input Directories, Category Throttle, Search DBs. and DB Registration settings.
Users Contains User Accounts, Clearances, Privileges, Send Email, Users Online, MyAuditees, My Audit Logs, and All Audit Logs.
Search Provides different search query forms, such as email addresses, category, full log, and user activity.
Workflow Central Request, modify, and view standing queries that will execute at a specified time or interval
Results Can search personal searches by date time, query type, query name, output table, and user.
Fingerprints Fingerprint builder and viewer.
Map Brings up Google Earth
Help Help Documentation, XK Forum. Account Maintenance, and About XKEYSCORE

TOP SECRET II Si II REL TO USA, AUS, CAN. GBR, NZL

32xks-system-administration-p33-normal.gif:


TOP SECRET II SI II REL TO USA. AUS. CAN, GBR. NZL

Admin Menu

SUB-MENU OPTION DESCRIPTION
Computer Resources Allows for process configuration and management.
Input Directories Contains the configuration for file-based Input directories.
Category Throttle Edit CADENCE quota limits by category and/or fist table.
Search DBs Configuration for query databases which are queried when a search is submitted.
DB Registration Contains the mapping from insert databases to query database.
News Add, modify, delete mandatory and home page News.

TOP SECRET II Si II REL TO USA, AUS, CAN, GBR, NZL

33xks-system-administration-p34-normal.gif:
TOP SECRET II SI II REL TO USA, AUS. CAN, GBR. NZL

Computer Resources



The Processing->Computer Resources

option from the ADMIN menu allows control of
the entire daemon-styled, or continuously
running, processes for XKEYSCORE.

■ Processes appears in a table following the
convention:


xkeyOl process_data_parent

TOP SECRET II $1II REL TO USA, AUS, <

34xks-system-administration-p35-normal.gif:
TOP SECRET II SI II REL TO USA. AUS. CAN GBR. NZL

Computer Resources

Computer Resource Window Process Table

¿kttr* XfV/»S A**™ X UK" Q, IT WaWtoffCtntiil , Fr4*»ir#t* □ Stafates « rtet»

« H.%. *

fil.T^n Ffcor

XI0

aL3Aoc*Mr*

30 !*€*£ Ottoarto
S Ttrotlk

Ssmt^oo*

2ce«wtf*ü(r

*£!«***

S CtMVMUden OK*«

2R
f* 2p y Ute
** C wtoW
H? 9+t*>
t^Quö', frc/te-

31
APP LAUWCHER STOPPS) APP

Computer Rttcwtci

«►i« Md Ae*Kr»5* AfpUiKhw h FmrUr»o

ACWN P*oc Hart Crop ar> Mifi Prop ot A/«M«ni PVofraniO CCM*ttrtd*tf Clift-« Mm C*tf«4r« Slwta4 ;i*e• ttdSttW • QXryjfXK ms HJM RlN 2013-11-2717:3397 0 2012-11-2717 3247 0
• tfaftroOl ch+zt jMiORter xf prp 31019 HJM 201311-2717:3337 0 2012-11-3717 32470
• tfatoiOl ^sjWa^tdtr 31041 MJH RlN 2013-11.2717:3357 0 2012-11.3717 33470
• cft****tZ4WK«jn 31053 VUN IW 2012-11-2717 3537 0 2012-11-27 17 52470
• IMomOI »jeryjfap«** 1311 HJM RlN 3012-13-0321.00090 2012-12-032103070
• *».rfvij*cc 31134 HJM RlN 201311 2717:22*7 0 201 211 2717 32470
• UdavOi a».ft*tofur«*ar 133» HJM RlN 2012-11-2721:0113 0 2012-11-2721 01130
• ttdroOl U* fl0Cl24t«rv«r 311» HJM rin 201211-2717 33970 2d Ml-2717 3247 0
• tMfluOl 31124 HJM RUN 2012-11-2717:3357 0 201M1-2717.32470
• IbfcroOi c»J«r• «4S'tOi rts_«r#«f_rtrt* 311» HJM RUN 2012H-27173357 0 2012-11-271732470
• IfefcroOl lU*«"Oi roA^ccrjarvc •««>**» . ^ A— A a ^ ^ ^ ^ - A - CCvi^JT /C;|UiAJi4rorwvu 31140 31i«3 HM HJM RlN RlN 3013-11-2717 3357 0 2012-11-2717 33470 3012-11-2717.3397 0 201*41471733500





TOP SECRET II Si II REL TO USA, AUS, CAN, GBR, NZL

35xks-system-administration-p36-normal.gif:
TOP SECRET II SI IIREL TO USA.

5. CAN. GBR. NZL

V

¿Wife

TJSr

■ The xks_app_launcher process runs on all
servers from the inittab.

■ It tells the computer which program to run by
looking at its tasking host.

■ /opt/xkeyscore/config/www/xscore.cfg

• The config file specifying the location of the
tasking database.

■ Processes can be stopped, started, edited, or
deleted from the Computer Resources
window.

TOP SECRET Hi

I REL TO USA, AUS, <

. GBR, t

36xks-system-administration-p37-normal.gif:
5. CAN. GBR. NZL



■ Add a new process - click Add

■ Edit a process - click Stop in the ACTION
column, then click Edit.

■ Delete process - click Stop in the ACTION
column, then click Delete.

■ Stop the App Launcher - disables the
xksappjauncher on every host.

TOP SECRET/

.TO USA, AUS,<

37xks-system-administration-p38-normal.gif:
TXT

Visual cues in the form of colors are used to
help identify activities performed by
XKEYSCORE and serve as status indicators
for monitoring purposes.

• Red - indicates processes have been stopped

• Green - indicates processes are running

• Yellow - indicates processes are starting

• Orange - indicates processes are being stopped

• White - indicates processes won’t start
Visual cues are also available in the
COMMANDED STATUS and STATUS
columns of the table.

TOPSECRET/

I REL TO USA, AUS, <

. GBR, txks-system-administration-p39-normal.gif:
Resources-Start/Stop Processi

i

r

■ It may be necessary to stop or start processes
for troubleshooting or for a graceful server
restart.

■ Individual processes and programs - Click
Stop in the ACTION column. To start it, click
Run.

■ To stop all individual programs, select
ACTIONS->Start/Stop Resources. Enter the
program name in PROGRAMS field, then click
OK.

■ Can use ‘xks proc’ actions and commands to
do the same functionxks-system-administration-p40-normal.gif:
Resources-Start/Stop Processi

All Processing - select START/STOP
Resources from the ACTIONS drop-down
menu, leave the PROGRAMS and ON
HOSTS fields to their defaults, click OK.

Specifying programs or hosts - select
STOP or START, enter a wildcard expression
such as * or! in the PROGRAMS or HOSTS
field, and click OK.

• Example: process*

Alternatively, in a terminal window can run:
• xks proc stop process*

TOP SECRET III

.TO USA, AU$,<

40xks-system-administration-p41-normal.gif:
Run a Process Manually

It may be necessary to run a process manually for
troubleshooting purposes. To run a process
manually:

1. Launch the GUI and log on as oper or admin.

2. Click ADMIN > Processing > Computer Resources

3. Click Stop in the ACTION column for the process.

4. Open a terminal window and ssh to the host running the
process, as the user 'oper'.

5. Type ps -ef | grep to verify

that the process is stopped.

6. Type

Example:

query proc

--loglevel debug

TOP SECRET II Si II REL TO USA, AUS, CAN, GBR, NZL

41xks-system-administration-p42-normal.gif:
TO USA. ■

& Home XMyXKS X Admin £

| Navigation Flter 2 ¿0

=3 Uwr Accounts
Clea-ances
SPrivlegej
=3SerdEmal
3 Use* Onto»

■ This menu is only accessible to users with
system administration privileges.

■ An SA can add/modify user accounts, add
groups, clearance levels, privileges, and
email users from this menu.

TOPSECRETj

. TO USA,,

i, GBR, I

42xks-system-administration-p43-normal.gif:
TOP SECRET // SI II REL TO USA. AUS. CAN. GBR. NZL

■ From the main menu bar, click MyXKS to view
your profile, accesses, privileges, auditors,
settings, fingerprints, workflows, and recent
results.

■ Right click on any search form name to add a
shortcut for that search form.

FulLogDNE HTTP Activity My Rngerpiints Vtf Workflows My Recent

Resets

n r



th

Ftofie

3 My Recent F.escfcs
SJPioftexks-system-administration-p44-normal.gif:
TOP SECRET II SI IIREL TO USA. AUS. CAN GBR. NZL

Search Menu

■ From the main menu bar, click SEARCH.
Menu options display in the vertical pane on
the left.

TOP SECRET II Si II REL TO USA, AUS, CAN, GBR, NZL

44xks-system-administration-p45-normal.gif:
FOP SECRET // SI II REL TO USA. AUS. CAN, GBR. NZL

Search Menu

When choosing a plugin type from the menu
options, the only data searched is the data that
was identified as a hit when the plugin was

processed
SUB-MENU OPTION DESCRIPTION
Category DNI Searches dictionary category hits.
Full Log DNI Searches all sessions received by XKEYSCORE.
User Activity Enables a user to search by a user’s activity. Example: a user can find a hotmail user’s msnMallToken

TOP SECRET//Si/.'REL TO USA, AUS CAN GBR, NZL 45xks-system-administration-p46-normal.gif:
/.Wife

rtr

All searches are conducted on database
tables where the results of the XKEYSCORE
engine are stored.

Each row of a database table contains values
from an individual session that was identified
as a hit by XKEYSCORE when that plugin or
microplugin processed the session.

Each search type query is related to a plugin
or microplugin, which performs the metadata

extraction.

46

TOPSECRET/

I REL TO USA, AUS, <

. GBR, txks-system-administration-p47-normal.gif:
/.Wm

rr

v

■ Search details can be accessed from the
Search status window by clicking Details.

. CURRENT SEARCH DETAILS window
displays and allows the user to watch a query
run through the appropriate databases.

■ RESULTS link in the main menu bar can be
used to display a list of all previous search
results.

■ Queries operate in parallel on each host.

TOP SECRET Hi

.TO USA, AU$,<

47xks-system-administration-p48-normal.gif:
• ouno ■ tnoo i y aoo-r i of*T_

Show Ou»rv

oí .-■ « • w

P«ur* MkJ» rinlrhwd
I Slwluv

=_1 Al aWavi'OI yA-»«w

ryi /li«K«vr'OS ^r»/n*w

I --I V». vwrO? .r y ^ iw»i«/

I ~ I »»iwi/ov'H •. -4■ «-4» in->í-.v53 AUKCVrOJI Cr _^| AI« Kwwr na i.fO/w« .1
I— ~1 /llxKovru*»: doummory/now
íAixKcvrOl: cio/n*w

i m= I Al » I« a wiTin- r
I — I /llxIovrUü: ciU/hcw

* —* AlAKwy»OÓ.MW«.IMHM»M y /• IW vv

i 4l*KAv«n/ r^nA^^w

I■■ Ï AlxkovrU/íciírummorv/ocnt
r=| Aiy.kvvfOö. «.iOX iww

t Ji—■ ÄUI*A«*«nfl !r« vv

l»n AI->«vrOO «OAi»w

I -1 Alyk. vrOCi. ll¥UfiK»lv*r y A • w -»«y

I at^l AIXWAVri II «-4I IA->^w
I SJ AI-I¡ Al Vkvvr I I
.5 I Alxuovn i f^auiTMTiQ* y/r»ow

C^Zl AI-KCV»' i ? f4n/n*w

Al . k «

í= I AixHovr'i ui: c4U/ncw
. = 1 AlKKAvn j: qcuminArY/aon*
i «■'I
í — 1 AlxKovrl 4:qoumiT>ory/novv

I --‘I AImK% wr 1 Ib. b|OA «www

lnr=l 4lwkAvr1 ^ «mm^ryAlr»l«l-

Alxkcvr l C-: ciO/n*w

I -i Al.*.k « vi « 0.h|kuiiMHMi y A.IVJI (W

í m 1 Al *k AVI 1 / r 4I i/n^w

Al xkrvr I 7: cu?umm*M-v/hwvv

• —i aiwwavi 1 n rtn I 0 373 1

I — ‘I /»l-kwr I ^ci*»MM»**«»rv«i»yv
r=I Al w 1 *=■. *-lO A tww

l»-^l /TIaWävi -I M- rj AI jimriAi y/nrin«

igl aiI Al vkvvr 20. (.iW-inxitv^yA iwvv

i — I /iixhov».íi í^SZI AUKC.vr'j i f|«.i*.n»r.A»v/nAW

C4ia*»vmc4

r»rl



TOP SECRET /í Si II REL TO USA, AUS, CAN. GBR, NZL

48xks-system-administration-p49-normal.gif:
■ From the main menu bar, click RESULTS to
retrieve the results of previous queries.

■ By changing the start and stop dates, queries
performed between those dates can be
viewed.

■ If the query name is known, it can be entered
in the QUERY_NAME field.

■ If the USERID is known, it can be entered.

■ When complete, a window displays with the
matching queries.

TOP SECRET Hi

.TO USA, AU$,<

49xks-system-administration-p50-normal.gif:
>10 01 101 W' 'ODI I«jl

1001 I 001 tool
1001 loot 1001 11®'®'»’
oiooi icolWA Wbi
O 10 01 ,0* 1001 1 001

>101 010001 wo* ,(>10I* 01 101910
) 101.

HI

li’4

TOP SECRET// SI REL TO USA. AUS CAN GBR. NZL

TOP SECRET U Si II REL TO USA, AUS, CAN, GBR, NZL

50xks-system-administration-p51-normal.gif:
V XKEYSCORE Process Data Flow

V Processing Programs

V Query Processes

V Other Processes

V Cronjobs
Vcrontabxks-system-administration-p52-normal.gif:
TOP SECRET II SI II REL TO USA. AUS. CAN GBR. NZL

-End Process Data Flow

TOP SECRET II Si II REL TO USA, AUS, CAN, GBR, NZL

52xks-system-administration-p53-normal.gif:
Processing Programs

Processing programs are the main processes that extract metadata from
the traffic and then database the information in insert databases.

PROGRAM DESCRIPTION
file_input_proc Scans for new input files, (before processing moves the file to the .tmp directory of the input directory specified)
sotf_dist Listens for incoming SOTF sessions
process_data_parent Processes all new files discovered by file_input_proc or sotf dist; optionally archives content and databases metadata. Parent process loads all dictionaries and starts up, then forks child processes which do the actual processing.

TOP SECRET U Si II REI_ TO USA, AUS, CAN. GBR, NZL

53xks-system-administration-p54-normal.gif:
■op SECRET II SI IIREL TO USA. AUS. CAN. GBR. NZL

Y/^

• This process replaces process_dataO through process_dataX

■ The “parent" process starts up and loads all the dictionaries, and then “forks"
child processes which actually do the processing

■ Parent acts similar to the xks_app_launcher, managing restarts for the
children when they die

■ When dictionaries are modified, parent reloads them and restarts the children

■ “xks proc” will show an “X/Y" number next to process_data_parent

• This is the number of children currently running, over the number that
should be running (based on the xks.config num_data_processors setting)

• pdp will show up yellow anytime X != Y and green when everything is
running normally

• This means when you first (re)start pdp, it will show yellow while it is
loading the dictionaries, because none of the actual child process_data's
are running yet

■ “xks proc" will report extra or missing process_dataX with a PID of 0

• Can't tell what PID missing process_data is suppose to have, because its
managed by the parent now

TOP SECRET II Si II REL TO USA, AUS, CAN, GBR, NZL

54xks-system-administration-p55-normal.gif:
TOP SECRET//SI//REL TO USA. AUS. CAN. GBR. NZL

Query Processes

■ Query processes are processes that search
and submit all necessary tables for the
analysts queries.

PROGRAM DESCRIPTION
querydispatch Submits search jobs to search databases and propagates the status of the search and results back to the web server
query_proc Searches through all the necessary tables for the analysts queries.

TOP SECRET II Si II REL TO USA, AUS, CAN. GBR, NZL

55xks-system-administration-p56-normal.gif:
■ Other process which is run from the
Application Launcher.

■ mailorder_proc - polls the

/export/data/xkeyscore/outputs/mailorder_working
directory by default. Then renames and moves
mailorder files to

/export/data/xkeyscore/outputs/mailorder.

56xks-system-administration-p57-normal.gif:
■ xks_meta_ingester - streams metadata over
socket. This process improves database
performance. Instead of each xscore_proc
writing to the database independently, they
stream their metadata over socket to the
metajngester, which combines it by plugin
and writes to the database.

• Reduces the number of connections to MySQL and
gives better control over table size.

TOP SECRET II $1 II REL TO USA, AU$, CAN. GBR, NZL

57xks-system-administration-p58-normal.gif:
register_metadata_tables - moves tables
from processing database of XKEYSCORE
system to query database.

• Works against the uberjndex table

• uber_index->table_name, base_table_name, join_table

• Base table - contains common information
amongt tables (full log xxx xxxxx table)

• Extension table - extends the base table

• Registration process takes place in two phases:

► Register all base tables

► Register all extension tables that have had its base table
registered

TOPSECRETi

. TO USA, i

J, GBR, I

58xks-system-administration-p59-normal.gif:
TOP SECRET II SI IIREL TO USA. AUS. CAN. GBR. NZL

■ signal acquisition loopback - process that
feeds modified packets back into the system.

• Front-end for packet recursion or any other
process that feeds modified packets back into the
system

► Reinjects back to front-end - xfip

► Process is completel independent

TOP SECRET II Si ll\

TO USA, AUS,

CAN. GBR, I

59xks-system-administration-p60-normal.gif:
■ mpmr_server - this is the map-reduce server
for microplugins, which runs the “Reducer”
portion of GENESIS v5 microplugins.

■ Runs outside the normal processing flow, and
will not affect the rest of the system.

■ It has a telnet port (5850) just like an
xscore_proc.xks-system-administration-p61-normal.gif:
TOP SECRET // SI IIREL TO USA. >

. AUS. CAN. GBR. NZL

¿Wife

■ correlation_server_0 - in-memory
map-reduce server for correlation engine.

■ Each machine has one correlationserver, and
every process_data_parent connects to every
correlationserver

• xscore_proc - 8GB by default

• uses port 4321

TOPSECRET«I

I $1II REL TO USA, AUS, CAN. GBR, t

61xks-system-administration-p62-normal.gif:
TOP SECRET II SI II REL TO USA. AUS. CAN. GBR. NZL

T

m xks_comms_server - a more efficient way to

communicate with hosts within and outside an

XKS cluster (not currently implement)

• Automatically handles configuration for talking
between slaves, master and overlord at site

• Configuration is needed to connect to the “peer”
on the path towards, other sites

• Comms configuration lives in
$XSCORE_DIR/config/comms/comms.config

• Supports a “quality of service” which “fairly”
distributes available bandwidth to the services that
are using comms

TOP SECRET II $1 II REL TO USA, AUS,

CAN, GBR, I

62xks-system-administration-p63-normal.gif:
OP SECRET If SI IIREL TO USA. AUS. CAN. GBR. NZL

r

■ xkscommsserver

• Allow and Peer rules have a “network” parameter
which the comms systems uses to determine an
“inside” and an “outside” in proxies.

• Comms system will only accept connections from
address ranges it has been specifically configured
to allow.

• Every between 2 comms servers connection
should have:

► “bandwidth_rule” on each side, name doesn’t matter but
both rules should usually have same bandwidth cap

► “allow” rule on one side with a reciprocal “peer” rules on
the other side

TOP SECRET II $1II REL TO USA, AUS, CAN. GBR, I

63xks-system-administration-p64-normal.gif:
TOP SECRET II SI IIREL TO USA. AUS. CAN, GBR. NZL

Other Processes

■ xks_comms_server

• Example: If we have a site named “US-123”
connecting to xks-central over a 1 Mbps link,
US-123’s config would be:

bandwidth[world] = 1Mbps

peer[00] = address=xks-central.corp.nsa.ic.gov, port=2412,
bandwidth=world, network=external

And xks-central would have:
bandwidth[us123] = 1Mbps

allow[00] = address=xkey-master.us123, bandwidth=us123,
network=internal

TOP SECRET II Si II REL TO USA, AUS, CAN, GBR, NZL

64xks-system-administration-p65-normal.gif:
■ Other process which is run from the
Application Launcher.

■ GUId - rescans content against fingerprints
when a user clicks to view the content of a
session.

■ tomcat.sh - web server used to host XKS
GUI

■ sotftod124server - downloads sessions

• Gets called from the GUId process

• Works with any downloaded traffic that is SOTF

TOP SECRET // $1II REL TO USA, AU$, CAN. GBR, NZL

65xks-system-administration-p66-normal.gif:
TOP SECRET //SI IIREL TO USA. AUS. CAN, GBR. NZL

Statistic Processes

■ Other process which is run from the
Application Launcher.

■ xks_server_stats - sends to
xks_system_monitor on Master and generates
stats about the server itself.

• CPU usage, memory usage, disk space, disk I/O,
network traffic, etc.

• Stats are fed to xks_system_monitor and the
system monitor does magic with them.

66xks-system-administration-p67-normal.gif:
5. CAN. GBR. NZL

rr

7

m xks_system_monitor - collects stats
messages from all over the system (front-end
and back-end and the server Itself) and
summarizes them for forwarding. Optionally it
can database stats locally.

TOPSECRET/

I REL TO USA, AUS, <

. GBR, t

67xks-system-administration-p68-normal.gif:
■ XKEYSCORE uses a number of cron jobs to
perform tasks.

CRONJOB DESCRIPTION
age_off_new.php Ages off metadata and content when the disk is near capacity, or when thresholds have been met.
xks update_dictionaries Pulls updates from various sources.
xks rsync pushconfig Copies the /opt/xkeyscore/config directory to the slaves.
rwc_post_to_pu b. py Once an hour kicks off an update request

TOP SECRET II Si II REL TO USA, AUS, CAN. GBR, NZL

68xks-system-administration-p69-normal.gif:
■ Crontab is the program used to install,
uninstall or list the tables used to drive the
cron daemon.

■ The crontab consists of

• age_off_new.php

• xks update_dictionaries

• xks rsync push_config

• rwc_post_to_pub.py

TOP SECRET II $1II REL TO USA, AU$, CAN. GBR, NZL

69xks-system-administration-p70-normal.gif:
TOP SECRET II SI IIREL TO USA. AUS. CAN. GBR. NZL

■ age_off_new.php

• Options:

► -debug : extra debug statements in the output

► -info : extra info statements in the output

► -task db : explicitly state that the machine is a task host

► -web_db : explicitly state that the machine is a web host

► -nosleep : use if you want to run now

• This process ages off tables and archived data
based on the settings in the xks.config file and
the percentage of disk space used.

TOP SECRET II $1II REL TO USA, AUS, CAN. GBR, NZL

70xks-system-administration-p71-normal.gif:
TOP SECRET II SI IIREL TO USA. AUS. CAN. GBR. NZL

V

m xks update_dictionaries

• This process pulls the necessarily files from
various sources to update the dictionary.

• Configure /opt/xkeyscore/config/xks.config

► ^dictionaries]
dictionary]!}] = type=royale, \
src=sftp://tssi_1vey:tssi_fvey@
xks-control/home/tssi_fvey/xks_dict_update.tar.gz, \
dest=update/xks_dict_update.tar.gz, \
action[0]=“cd

$XSCORE_DIR/config/dictionaries/update;$XSCORE_DIR/config/
dictionaries/update/dup_install.pl >/dev/null 2>&1“
dictionary]"!] = type=cadence

TOP SECRET II SI II REL TO USA, AUS, CAN. GBR, NZL

71xks-system-administration-p72-normal.gif:
■ xks rsync push_config

• Transfers Master configurations to its slaves.

• Excludes dot files, “httpd/logs”,
loadserver/packages”, “httpd/log”

• force: option to xks to force push_config when not
on the master



72xks-system-administration-p73-normal.gif:
TOP SECRET II SI II REL TO USA. AUS. CAN, GBR. NZL

Zr

■ rwc_post_to_pub.py

• The automatic starProc process is as follows:

► Hour 1: master asks whoever (say xks-control) for an
update, gets the rpm, installs It, there Is much rejoicing.
The slaves asks the master for the rpm at the same time
the master asks xks-control, but obviously the master
doesn’t have it, so nothing happens.

► Hour 2: everyone asks for an update again, this time the
master has the rpm, the slaves download it and install
and there is much rejoicing.

► The rpm is installed and process_data_parent’s are
restarted as soon as the rpm is downloaded on a given
machine.

TOP SECRET II Si II REL TO USA, AUS, CAN, GBR, NZL

73xks-system-administration-p74-normal.gif:
TOP SECRET II Si II REL TO USA, AUS, CAN. GBR, NZL

74xks-system-administration-p75-normal.gif:
V What is a DeepDive?

V Why DeepDive?

V What does a DeepDive look like?

V Front-End Processes

V xFIP

V Promoter

75xks-system-administration-p76-normal.gif:
fOP SECRET // SI IIREL TO USA. AUS. CAN, GBR. NZL

DeepDive XKEYSCORE

XKEYSCORE packet processing solution

• XKEYSCORE’s software handles all packet processing

• No upfront filtering prior to XKEYSCORE

• XKEYSCORE “promoter” tries to promote richest/most
interesting traffic

► All Strong Selectors

► Full take ASDF (User Activity metadata)

► Subset of GENESIS signatures

■ List managed by XKEYSCORE team in concert with
collection managers and site engineers

• 20% - 30% of site traffic is fully processed and can be
found via XKEYSCORE search

► Typically does not include unknown or uninteresting
protocolsxks-system-administration-p77-normal.gif:
Why DeepDive?

Access to most relevant DNI data supporting SlgDev and
collection missions. Enables new mission capabilities (e.g.,
Correlation)

Session promotion can be synchronized and managed based on
Genesis signatures, traditional tasking selectors and available
resources

► Provides better scaling

► Drop unwanted data. Keep the rest and make decisions
later and more accurately

Better control of the processing space

► Instantiate new mission capabilities and dataflows quickly

► Troubleshooting and monitoring made easier

Need access to “raw" packets to support new mission (e.g.,
Cyber, Bulk Crypt)

• Sessions can be displayed as Packet Bundles like Wireshark

TOP SECRET II Si II REL TO USA, AU$, CAN, GBR, NZL

77xks-system-administration-p78-normal.gif:
DeepDive

TOP SECRET II SI IIREL TO USA. AUS. CAN, GBR. NZL

What does a DEEPDIVE look like?

• XKEYSCORE full-take session processor (Back End)

• High speed packet ingest: an end-to-end solution

• Intelligent filtering to vary the proportion of traffic retained

TOP SECRET II Si II REL TO USA, AUS, CAN, GBR, NZL

78xks-system-administration-p79-normal.gif:
TOP SECRET // SI II REL TO USA. AUS CAN GBR. NZL

XKEYSCORE Front-End Processes

What It's called What It does What it means
Packet Splatter Ingests packets (from files, from the network, from a capture card) in a variety of formats. If it's a packet stream, ft can probably be fed into a DEEPDIVE.
xFip Fast reassembly of TCP/IPv^, UDP/IPv4 streams*, and TCP/lPv6 and UDP/IPv6 streams*. DEEPDIVC sessionizes everything
METTLESOME Reassembly of streams from less common protocol stacks. before making a keep/drop decision.
Promoter Rule-based filtering of reassembled sessions, based on keyword, country code or appid/flngerpmt. DEEPDIVE intelligently chooses the most useful traffic for retention.
Defrag Fully rebuilds sessions** Enough content available to do full decoding/document descent at the Back End

TOP SECRET II Si II REL TO USA, AUS, CAN, GBR, NZL

79xks-system-administration-p80-normal.gif:
■ Packet bundles

• Preserves original packets and packet order

• Preserves information that is lost during sessionization

• Original pcap available in the XKS Viewer

■ Packet API

• Microplugins can iterate over raw packets

• Microplugins can use information that is lost during
sessionization

► E.g. timestamps, flags, checksums

■ Packet fingerprints

• Fired based on observations xFip has made

► E.g. large sequence gaps, TTL variation

TOP SECRET U Si II REL TO USA, AUS, CAN, GBR, NZL

80xks-system-administration-p81-normal.gif:
TOP SECRET II SI II REL TO USA. AUS. CAN. GBR. NZL

Filters sessions prior to back end processing

• keywords, regex, country code, appids*

• SIGDEV: promotion rather than strong selection

Set the focus of the back end

• traffic types of interest

• regions of interest

• legal/policy constraints

Set the width of the access aperture

• promote 20% of 20 signals?

• promote 100% of 4 signals?

Set the length of data retention

• promote 20% and keep for 3 days?

• promote 30% and keep for 2 days?

allow appid chat.*

allow country_cod« PK

block countrycode US-US

TOP SECRET II Si II REL TO USA, AUS, CAN, GBR, NZL

81xks-system-administration-p82-normal.gif:
TOP SECRET II SI II REL TO USA. AUS CAN GBR. NZL

TOP SECRET II si II REL TO USA, AUS, CAN. GBR, NZLxks-system-administration-p83-normal.gif:
TOP SECRET // SI II REL TO USA. AUS. CAN. GBR. NZL A

V Usage
^options

V General Commands

V Services

V Actions

V Options

83xks-system-administration-p84-normal.gif:
xks Options

TOP SECRET //SI II REL TO USA. AUS. CAN GBR. NZL

■ Usage: xks [options]

• Try 'xks help ' to get help on a specific
service or action

• General commands:

► services - list available services

► actions - list available actions

► dependencies [invert] - show service dependencies

► help [items] - print help on services or actions



Services (specify one or more service names or
‘all’):

► start

► stop

► restart

► status

► setup

- start the specified services

- stop the specified services

- restart the specified services

- print the status of the specified services

- setup/configure/fix the current xks install

TOP SECRET U Si II REL TO USA, AUS, CAN, GBR, NZL

84xks-system-administration-p85-normal.gif:
xks Actions

TOP SECRET //SI II REL TO USA. AUS. CAN. GBR. NZL

Actions:

► accounts_report - sends an email containing accounts usage to the
specified users
► add_admin - sets up a local Linux user to administer XKS
► change_db_password - changes the XKS database user’s password and
updates all references to it
► cluster - cluster actions
► compile_genesis - compiles GENESIS signatures
► disk_check - get raid and disk status
► ext4_format - format $XSCORE_DATA_DIR partition and convert
to ext4 filesystem
► ext4_upgrade - convert to ext4 filesystem while preserving
contents of SXSCORE DATA DIR (no formatting)
► fetch - fetch a remote file
► force_register - force metadata table registration
► info - show cluster information
► install_slave - install a slave machine in this cluster
► local_tagging - checks and/or loads tagging file

TOP SECRET II Si II REL TO USA, AUS, CAN, GBR, NZL

85xks-system-administration-p86-normal.gif:
TOP SECRET II SI II REL TO USA. AUS. CAN, GBR. NZL

xks Actions

Actions:

► monitor - view XKS monitoring messages via activemq
► mpmr_register - force mpmr table registration
► mysqls - run a mysql script
► onail - run a command on all machines in this cluster
► powertower - configure or run a powertower command
► proc - control XKS processes on this cluster
► query - display query status or submit a query
► query_dispatch - command line interface to the XKS
query_dispatcher(s).
► rac - access remote admin ports
► reload dictionaries - force running processes to reload dictionaries
► rsync - push configs or files to slaves
► search_fields - populates user settings with search fields
► $how_config - show values from xksconfig for specified keys
► switch - query or rebalance data switch
► sync_accounts - synchronize user accounts (except for
classifications)
► tail - view realtime xks logs
► tasking dump - print out the contents of the xkTasking and
xksTasking_voip databases

TOP SECRET II Si II REL TO USA, AUS, CAN, GBR, NZL

86xks-system-administration-p87-normal.gif:
TOP SECRET II SI II REL TO USA. AUS. CAN, GBR. NZL

xks Actions

Actions:

► top - display system performance
► update_dictionaries - update all XKS dictionaries
► update_gui_help - update the ’help' pull downs in GUI
► users - display the users currently logged into the GUI
► version - show XKS version information
► watchdog - check and (re) start essential XKS processes.
► workflow - manually submit a workflow

TOP SECRET II Si II REL TO USA, AUS, CAN, GBR, NZL

87xks-system-administration-p88-normal.gif:
xks Options

TOP SECRET II SI II REL TO USA. AUS. CAN, GBR. IMZL

Options:

► -verbose : print extra information to the screen

► -debug : used for debugging script problems

TOP SECRET II Si II REL TO USA, AUS, CAN, GBR, NZL

88xks-system-administration-p89-normal.gif:
■OP SECRET //SI II REL TO USA. AUS. CAN. GBR. NZL

xks - General Commands

• Type: xks help services

► This will list all available services:

■ first - initialization service that runs before all others

■ virus_scanner- sets up virus scanner, assuming tarballs are
present.

■ ftpd - enables ftp on the master if mailorder is enabled

■ distcc - sets up distributed compiler service

■ slash_proc - setup optimal /proc parameters

■ myricom - handles installation and configuration 10GigE
network cards

■ home - sets up the home directory for the xks user account

■ gcc - check there is a working compiler on the system

■ upgrade - updates configuration files when upgrading to a
new version of xks

■ bashrc - sets up bash environment variables

■ beacon - sets up xks monitoring beacon based on xks.config

■ tt- checks connectivity to TRAFFICTHIEF server

TOP SECRET II Si II REL TO USA, AUS, CAN, GBR, NZL

89xks-system-administration-p90-normal.gif:
rOP SECRET //SI IIREL TO USA. AUS. CAN. GBR. NZL

Sr

• Type: xks help services

► This will list all available services:

■ sendmail - configures sendmail for use with xks

■ role_files - this service installs role-specific files

■ issue - sets up the DoD mandatory login warnings

■ royale_with_cheese - setups automatic updates

■ ntpd - configure ntp based on xks.config

■ linksummary - sets up xks link summary GUI

■ nfsd - sets up xks-specific nfs mounts

■ server_certs - sets up server certificates for SSL applications

■ openoffice - installs and configures OpenOfflce for use in the
xks GUI

■ init_d - sets up the xks init.d services

■ resolver - sets up resolver config

■ php - sets up PHP related stuff. Except php.ini

■ httpd - sets up xks-specific httpd configuration

TOP SECRET II Si II REL TO USA, AUS, CAN, GBR, NZL

90xks-system-administration-p91-normal.gif:
•OP SECRET ft SI U REL TO USA. AUS. CAN, GBR. NZL

xks - General Commands

Type: xks help services

► This will list all available services:

■ www- sets up GUI configuration files

■ voip - sets up voip processing

■ crond - ensures xks can use cron and sets up xks cron jobs

■ sshd - configures the secure shell service for use with xks

■ license - checks for a valid license file and if one isn't found
prints a message

■ syslog - configures the syslog service for use with xks

• all xks processes log to /var/log/xks.log

■ dictionaries - checks status of any configured dictionaries

■ cluster_check - checks network connectivity across the
cluster

■ autofs - start, stop, restart automounts

■ loadserver- start, stop, and setup loadserver

■ directories - sets up directories used for xks

■ auditd - no help available

TOP SECRET II Si II REL TO USA, AUS, CAN, GBR, NZLxks-system-administration-p92-normal.gif:
TOP SECRET II SI IIREL TO USA. AUS. CAN, GBR. NZL

xks - General Commands

• Type: xks help services

► This will list all available services:

■ Idap - no help available

■ mysqld - sets up the mysql server for use with xks

■ disks - checks status of disk partition used by xks

■ databases - maintains database scheme consistency

■ local tasking - reapplies local tasking if necessary

■ workflows - sets up xks default workflows

■ category_throttle - overrides default category throttle settings based
on overrides specified in xks.config

■ enrichment_tomcat - sets up enrichment tomcat java application
server

■ plugin_setup - populate plugin database tables from xml files, appy
default plugin config specified in xks.config, apply overrides from
xks.config, regenerate plugin config files from database

■ crdb - no help available

■ tomcat - sets up tomcat java application server

■ clickstream - sets up clickstream service

TOP SECRET II Si II REL TO USA, AUS, CAN, GBR, NZL

92xks-system-administration-p93-normal.gif:
TOP SECRET II SI IIREL TO USA. AUS. CAN GBR. NZL

xks - General Commands

Type: xks help services

► This will list all available services:

■ filejnput - sets up directories and database entries needed for file-
based input

■ age_off_db - synchronizes the database (xs_task_db.age_off) with
xks.config's settings for content and metadata. The values in the
database will be unconditionally overwritten with those found in
xks.config

■ db_connectivity - verifies connectivity to critical databases

■ pdf-sets up xpdf language packs

■ ul_age_off- sets the maximum data retention time to a little over an
hour in UL mode.

■ mDNSResponder - sets up mDNSResponder for use with SOTF
input

■ appjauncher - controls the xks app launcher, which is responsible
for monitoring xks processes and starting/stopping them as
commanded from the GUI

■ processes_setup - configures xks processes based on specifications
in xks.config

■ comms - sets up the XKS communications system configuration

93xks-system-administration-p94-normal.gif:
TOP SECRET // SI II REL TO USA. AUS, CAN. GBR. NZL

V

• Type: xks help services

► This will list all available services:

■ endace - handles all the Installation and configuration for
Endace Dag packet capture cards

■ last - cleanup service that runs after all others

TOP SECRET II Si II REL TO USA, AUS, CAN, GBR, NZL

94xks-system-administration-p95-normal.gif:
• start

► xks start mysql

• stop

► xks stop httpd

• restart

► xks restart nfs

• status

► xks status autofs

• setup

► xks setup plugins

TOP SECRET II Si II REL TO USA, AUS, CAN, GBR, NZL

95xks-system-administration-p96-normal.gif:
TOP SECRET II SI II REL TO USA. AUS. CAN. GBR. NZL

xks - Actions

• xks onall ‘ps -ef | grep xscore | grep -v grep’

• xks force registerla.uj

• xks rsync pushconfig -force

► Usage: xks rsync
[push_config|push_compiled|push_slaves|push]


• xks update_dictionaries

► Usage: xks update dictionaries
[test|check|print|force|help] |

• xks version •

• xks info

TOP SECRET II $1II REL TO USA, AUS, CAN. GBR, I

96xks-system-administration-p97-normal.gif:
no oi ,0‘ w
1001 I ooi *«

TOP SECRET //SI II REL TO USA. AUS. CAN GBR. NZL

xks - Actions

• xks query servers

operfltlxksvr 0 1 run] $ xks qua ry servers
tlxksvr02 : tjO 3« 148 9 8n S 4 w 2012-12-OS 16:07: 22
tlHksvr(J3 : qü 23ûj Os On Ow 2Û12-12-OS 17 : SS: 02
Clxks v r O4 : ClxksvrUb : tlxksvrO? : cjO 230a 0= On O w 2012-12-OS 17 : SS: 02
tlxkavr08: tlxk=vrÛ9: cjO Oa Os 173n 2w 2012-12-OS 17 : SS: 02
tlxkxvrlO : cjO 230-e» 0» On 0 w 2012-12-05 17:55: 02
tlxksvrll:qO 230â 03 On Ow 2012-12-OS 17 :SS: 02
tlxksvr 12 : cjO 225a Os 4n Ow 2012-12-05 17:55: 02
a=awaiting dispatch, s=sent, n=ne w, w=vjorking tiw*8tA«p show« Msrli-at submitted but unfinished query leurrent, time : 2012—12—05 18: 02: 09

TOP SECRET II Si II REL TO USA, AUS, CAN, GBR, NZL

97xks-system-administration-p99-normal.gif:
TOP SECRET II SI II REL TO USA. AUS. CAN GBR. NZL

xks proc full

I [oper@tlxkavrQl run] S xka proc full
1 app launcher status: RUN (pid 30705:
id hostname program arguments commanded actual pid
1-5 tlxksvrOl cadcncc_tosking_proc —myfdi XYD —pddg IE --dig... RUN RUM 31136
4 tlxksvrOl check_mailorder_site. php RUN RUN 31019
723 tlxksvrOl clickstreanservice.sh RUM RUN 31053
654 tlxksvrOl cnrichment-tomcat.sh RUN RUN 31200
9 tlxkavrOl file_input_proc RUM RUN 31104
1 tlxksvrOl GUId RUM RUN 9335
b4B tlxksvrOl nail* rde r_proc —eepydir /expert/data/xkey... RUM RUN 31140
8 tlxksvrOl quory_dispatch RUM RUN 17549
3 tlxksvrOl query_proc RUM RUN 3096S
193 tlxksvrOl regist e r_met adat atables —loglevel error RUM RUN 31143
70? tlxksvrOl signal^cquisitionbase -t generic_paeket_to_bundle... RUM RUN: 4/4 31236
12 tlxksvrOl aetftedl24sorv«r RUM RUN 31108
4 Cl tlxksvrOl st rong_selector_targeting RUM RUN 31145
13 tlxksvrOl tomcat.ah RUM RUN 31124
653 tlxksvrOl xX2_cowi3_3orv«r RUM RUN 31148
5 tlxksvrOl jfV.3_neta_in«jc5ter RUM RUN 31051
4 62 tlxksvrOl xks_serv*r_stat8 RUM RUN 31138
11 tlxksvrOl xfcs_system_uonitor RUM RUN 13986
"24 tlxksvc02 co c relat ion_se rvc c_0 --loglevel debug RUM RUN 13946
31 tlxksvr02 i*pmr_Rorvor RUM RUN 7150
”30 tlxksvr02 p roce 3s_dat a_pa re nt —inax-nem 20 RUM RUN: 4/4 22661
30 tlxksvr02 query_proc RUM RUN 14754
187 tlxk*vr02 register_metadat*_t«bleji --loglevel error RUM RUN 14994
710 tlxkavrQ2 aign*l_*cquisition_b«se -f gen«ric_pack«t_to_bundlo. .. P.UH RUN: 4/4 14996
41 tlxksvr02 signal_acquisition_loopback -f packet_aux.config -i loo... RUM RUN: 6/6 14990
2 59 tlxksvr02 sotf_dist RUM RUN 1484S
67? tlxksvr02 xkc_comms_se rve r RUM RUN 14992
38 tlxksvr02 xks_neta_ingester RUM RUN 14970
473 tlxksvr02 xks_s*rver_stats RUM RUN 14988

TOP SECRET II Si II REL TO USA, AUS, CAN, GBR, NZL

99xks-system-administration-p100-normal.gif:
no oi

1001 i ooi too

TOP SECRET // SI IIREL TO USA. AUS. CAN GBR. NZL

xks Example

• xks query

[operStlHksvrOl run]$ xks query
id user type search start search stop duration status
66250201 http_parser 00:00 12/2/12 00:00 12/6/12 00:00:10 ongoing
€6250183 full_log 00:00 12/3/12 00:00 12/6/12 00:00:17 ongoing
66250155 full_log 00:00 12/4/12 00:00 12/6/12 00:00:40 ongoing
€6250127 geo_info 00:00 11/30/12 00:00 12/6/12 00:01:16 ongoing
€6250352 enailaddresses 00:00 11/21/12 00:00 12/6/12 00:03:36 ongoing
66249873 full_log 22:00 12/3/12 21:59 12/4/12 00:11:31 ongoing
66249660 full_log 00:00 12/2/12 00:00 12/6/12 00:18:57 ongoing
€6244233 category 00:00 11/5/12 00:00 12/6/12 00:42:17 ongoing
66244135 full_log 00:00 11/28/12 00:00 12/6/12 00:44:30 ongoing
€6244009 http_parser 00:00 11/5/12 00:00 12/6/12 00:48:49 ongoing
€6243967 http_parser 00:00 11/5/12 00:00 12/6/12 00:49:34 ongoing
66243855 documentjnetadata 00:00 11/21/12 00:00 12/6/12 00:50:48 ongoing
66243785 correlation 00:00 11/5/12 00:00 12/6/12 00:52:46 ongoing
€6243463 correlation 00:00 11/21/12 00:00 12/6/12 00:56:13 ongoing
66243071 enail_addresses 00:00 11/1/12 00:00 12/6/12 01:08:24 ongoing
€6242973 user_activity_exif 00:00 11/21/12 00:00 12/6/12 01:12:03 ongoing
66242413 http_parser 00:00 11/28/12 00:00 12/6/12 01:26:32 ongoing
66242315 full_log 00:00 12/4/12 00:00 12/6/12 01:30:52 ongoing
There are 18 queries in progress

TOP SECRET II Si II REL TO USA, AUS, CAN. GBR, NZL

100xks-system-administration-p101-normal.gif:
>10 01
1001 I 001

TOP SECRET II SI II REL TO USA. AUS, CAN GBR. NZL

xks Example

xks query detail

[opetStlxfcevrOl run]i xks query detail idefidiSlCfS
juecy Sirae ry Oeand:
Type: eaiil addresses Searc.oin: iron CO:CO 11/3C/12 to 00:00 12/6/12
Duration: 00:01:29 Priority: 5 Csacel: N(o) Ksx Results: lOCOO Ksx Tine: 6030 Duery SCI Ken.: *a*tev3 *
Classification: S,IS/SI,M«A XCrosn, IS,HCS, S/SMIU3C0LAK, 3EL USA.NSA riDrOP.N,SI,C,R
Rtri: »KERB ¿«Latin* >* '2012-11-30 00:00:00' AMI dm.tin. 'iu.ry Status
ho it database status
tlxksvrOl qO finished
tlxksvt02 qO ongoing
tlxksvt03 qO finished
tlxksvrOt qO finished
tlxksvrW qO finished
tlxksvr06 qO finished
tlxksvrOT qO finished
tlxksvr08 qO finished
tlxksvrO? qO finished
tlxkxvril qO finished
tlxksvclZ qO finished
tlxkxvr!3 qO finished
tlxksrrli qO finished
tlxksvrlO * 2 finished

TOP SECRET II Si II REL TO USA, AUS, CAN. GBR, NZL

101xks-system-administration-p102-normal.gif:
TOP SECRET II Si II REL TO USA, AUS, CAN, GBR, NZL

102xks-system-administration-p103-normal.gif:
TOP SECRET II SI IIREL TO USA. AUS. CAN. GBR. NZL

V Executables
Vonall
-/xks onall
Vxks monitor
Vsotf_stat
Vxks top

V Web Status

V Additional Monitoring





TOP SECRET // $1 // REL TO USA, AUS, CAN. GBR, NZL

103xks-system-administration-p104-normal.gif:
■ System monitoring can be performed from the
command line using the following executable

commands:

• mysqls

• onall

• xks onall

• sotf_stat

• xks top

TOP SECRET // $1II REL TO USA, AU$, CAN. GBR, NZL

104xks-system-administration-p105-normal.gif:
The mysqls bash shell script can be used to
execute MySQL statements from in the
/opt/xkeyscore/bin.shells/sysadmin/mysqls
directory. The most commonly used options in
mysqls are:

• status - displays file-based input statistics.

• speed - displays the total file based input
processing rate (Mbps)

• speedl - displays file-based input processing rate
(Mbps) per input source.

• speed2 - displays file-based input processing rate
(Mbps) per xkeyscore processing server.

• count - displays the count of input files in the
new, working, error, and done states.

105xks-system-administration-p106-normal.gif:
TOP SECRET II SI II REL TO USA. AUS. CAN. GBR. NZL

■ mysqls status

[oper^tlxksvrOl run]5 mysqls status

status count!*) sum(filesize) priority

bit rate_Mbps

NULL

TOP SECRET U Si II REL TO USA, AUS, CAN, GBR, NZL

106xks-system-administration-p107-normal.gif:
// SI II REL TO USA. AUS CAN GBR. NZL

■ xks onall ‘xks mysql status’

[ope r 0c 1 xks vr 0 1 run) $ xks onall ’xks myscfl status’

Do you want to exeoute "xks ny»ql status" onall'? [y | n] y

11 xks vrOl

status mysqld

myscjld is running

tlxksvr02

status raysqld

mysqld is cunning;

tlxkavr03 -*-*■-*

status raysqld

mysqld is cunning

— — — ~ 11xksvrO4 — — — ^

status raysqld

mysqld is «running

tlxksvrOS

status mysqld

mysqld is «running

TOP SECRET II Si II REL TO USA, AUS, CAN. GBR, NZL

107xks-system-administration-p108-normal.gif:
TOP SECRET II SI II REL TO USA. AUS CAN GBR. NZL

xks monitor

■ This script will monitor your front-end
processes.

■ Type: xks montor or xks monitor h to

receive the help menu

Vl'CVCJr*AD C
i nouu Command Name Description
c config Configure this utility
d dataflow_al1 FrontEnd Dataflow Menu
b dataflow_bo BackEnd Dataflow Honu
in, h menu View this menu
a packet_splatter Packet Acquisition [Front End)
P process_data Process Data [Back End]
q quit Quit/Exit
s servers Server Stats (CPU, 10, etc. )
t sotf_input SOTF Input [Back End] ’xks top’ replacement
f xf ip Sessionization [Front End]

TOP SECRET II Si II REL TO USA, AUS CAN, GBR, NZL

108xks-system-administration-p109-normal.gif:
TOP SECRET // SI II REL TO USA. AUS. CAN GBR. NZL

xks monitor

■ Type: xks montor f to receive xfip stats

KKBYB03RK BvaaionizAtion (o«rv«r Vi*w '4')

Cji>»ix>r «ion KAt* < M t*|>* ) 1.0*41 r,<4 K ICKNAllty» (k^cr » count(vkt> t»vmt 1 r r«4H4nt«%
7UMH410 i dooooo 0.00 A. 00 0. A6 A. DA A o. aa 0.00
/LHU'^UtlUtUUOO 0.00 O. OO a. ao o. uu o u. uu a. uu

m:*1: D. OO (KCV: itWl,

TOP SECRET II Si II REL TO USA, AUS, CAN. GBR, NZL

109xks-system-administration-p110-normal.gif:
■ The sotf_stat command is used to display the
SOTF (streaming object transfer format) input
statistics for an entire cluster.

■ The statistics include total number of
process_data’s running on the cluster,
session input rate (sessions/sec), total bytes
input (Mbps), and total bytes output to
process_data(s) (Mbps).

TOP SECRET // $1II REL TO USA, AU$, CAN. GBR, NZL

110xks-system-administration-p111-normal.gif:
■ To execute the sotf_stat script:

• Log on to the server and open a terminal window.

• Type sotf_stat because the command is in the
path

• Type s to toggle the summary statistics view from
total statistics to individual host statistics.

• Type q to quit the program

TOP SECRET II $1II REL TO USA, AU$, CAN. GBR, I

111xks-system-administration-p112-normal.gif:
■ The sotf_stat script lists the hostname,
number of process_data’s currently running,
Mbps, number of sessions, and number of

bytes.

r—XKEYSCORE SOTF Statistics

Hostname #In #OA/#OC Mbps In Sess In Bytes In Sess Q MaxBlk
mhxkssvr02 7 4/4 18.66 410920147 4818112974976 0 0
mhxkssvr03 3 4/4 16.15 410121822 4783549865004 0 0
mhxkssvr04 3 4/4 16.65 410444622 4781320276992 0 0
mhxkssvr05 3 4/4 15.79 409831857 4759939303920 0 1

1—PRC: 15/ 15 Rate: 64.52 Mbps Sessions: 1641358767 Bytes: 19143289212772-

TOP SECRET II Si II REL TO USA, AUS, CAN, GBR, NZL

112xks-system-administration-p113-normal.gif:
TOP SECRET // SI IIREL TO USA. AUS. CAN, GBR. NZL

■ The xks top script lists the hostname, Mbps
sotf rate, number of process_data’s running,
the % of CPU, and % of 10 wait.

hostnane sotf iprocs cpu* iowaiti
mhxIcssvcOl -0.00 0 0.53 0.02
phxkssvc02 21.08 4 12.88 7.94
pxkssvi:03 13.55 4 13.35 7.28
mhxkssvi:04 14.97 4 14.50 8.63
rahxkssvc05 14.13 4 17.14 8.01
TOTAL - 63.74 16 11.68 6.38

TOP SECRET II Si II REL TO USA, AUS, CAN. GBR, NZL

113xks-system-administration-p114-normal.gif:
TOP SECRET // SI IIREL TO USA. AUS. CAN GBR. NZL

IOOI I 001-001 |01 WWl>| - -

Additional Monitoring

■ xks tail

I (oper@mhxkssvr02 •*)$ xks tail

pec 5 18:27:29 mhxkssvr02 register_metadata_tables[13877] : cregister
st (automatic?) repair failed

Dec 5 18:27:29 mhxkssvr02 register_metadata_tables(13877) : cregister
st (automatic?) repair failed

Dec 5 18:27:29 mhxkssvr02 register_metadata_tables(13877) : cregister
st (automatic?) repair failed

Dec 5 18:27:59 mhxkssvr02 register_metadata_tables(13877) : cregister
st (automatic?) repair failed

Dec 5 18:27:59 mhxkssvr02 register_metadata_tables(13877) : cregister
st (automatic?) repair failed

Dec 5 18:27:59 mhxkssvr02 register_metadata_tables(13877) : cregister
st (automatic?) repair failed

Dec 5 18:28:08 mhxkssvr02 sotf_dist(13986) : csotf_dist_t> NOTICE: cu
Dec 5 18:28:29 mhxkssvr02 register_metadata_tables(13877) : cregister
st (automatic?) repair failed

Dec 5 18:28:29 mhxkssvr02 register_metadata_tables(13877) : cregister
st (automatic?) repair failed

Dec 5 18:28:29 mhxkssvr02 register_metadata_tables(13877) : cregister
[st (automatic?) repair failed__________________________________________

TOP SECRET II Sl II REL TO USA, AUS, CAN. GBR, NZL

114xks-system-administration-p115-normal.gif:
TOP SECRET II Si II REL TO USA, AUS, CAN. GBR, NZL

115xks-system-administration-p116-normal.gif:
«/ Common Troubleshooting techniques
VFull Disk
«/Sotf Problems
«/MySQL

«/Processing Problems
«/Outputs
«/Query Problems
«/Directory Permissions

116xks-system-administration-p117-normal.gif:
■ /var/log/xks.log (xks tail) - Relevant error
messages can be viewed in this file. This
directory may fill the disk, some known
reasons are:

• process_data has lost its connection with the
sotf_dist and is continuously trying to reconnect to
sotf_dist.

• nfs error may have occurred and a detailed
message can be found in the file
/var/log/messages.

• Corrupt tables in the insert database.

• Check to make sure the age_off_new.php cronjob
aged off old metadata and content.

TOP SECRET II $1II REL TO USA, AU$, CAN. GBR, NZL 117xks-system-administration-p118-normal.gif:
■ /export/data/xkeyscore/inputs

• If there are too many files in the directory:

► fileinputproc may be running improperly or not at all.
Verify that file input proc is running from the command
line type:

■ ps -ef | grep file_ | grep -v grep

■ xks proc

► The file_input_proc may need to be restarted.

■ No new files in the directory:

• The directory may not be cross-mounted properly,
if automounting is used.

.TO USA, AU$,<

118xks-system-administration-p119-normal.gif:
TOP SECRET // SI IIREL TO USA. AUS. CAN. GBR. NZL

■ /export/data/xkeyscore/mysql/iO or i1

• If /export/data/xkeyscore/mysql/iO or ¡1 are filling and
qO and/or q1 maintains its size,
register_metadata_tables may not be working
properly.

► Restart process and watch the databases to see if it is
transferring files or run the process by hand to
troubleshoot further.

• If /export/data/xkeyscore/mysql/qO or q1 is filling, the
age_off_new.php script may be running improperly or
not at all.

► First run the command: ps -ef grep age_

■ If script isn't running, try running it by hand.

■ If script is running, then stop script and try running it by hand to
see if there are any errors.

TOP SECRET II $1II REL TO USA, AUS, CAN. GBR, NZL

119xks-system-administration-p120-normal.gif:
rOP SECRET //SI I! REL TO USA. AUS. CAN, GBR. NZL

SOTF Problems

Can an sotf_input_proc run with a file_based
file_input_proc?

• Yes. Both input types can run on XKEYSCORE given that
each are independently configured correctly.

■ Can file-based input be disabled so that only
sotfjnput is processed?

• If moving from file-based input to sotfjnput, and no
additional file-based input is expected, the plug-in for file-
based input, db_input_file_handler, should be disabled.

• From the TERMINAL WINDOW:

► Stop all the processes : xks stop all

► Change /opt/xkeyscore/config/xks.config to set fllejnput to ‘no’.

► Setup the config : xks setup plugins, xks setup processes

► Rsync change to slaves : xks rsync push_config

► Restart process_data’s : xks proc restart pdp

TOP SECRET II Si II REL TO USA, AUS, CAN, GBR, NZL

120xks-system-administration-p121-normal.gif:
)P SECRET II SI IIREL TO USA. AUS. CAN. GBR. NZL

■ Is XKEYSCORE receiving input?

• To verify whether XKEYSCORE is receiving input,
run the sotf_stat command to get the current
input statistics.

• If no connection is visible, from the command line:

1. Type telnet localhost 5042

2. Output statistics for the specified sotf_dist

3. If running, type ps -ef | grep sotf_dist

4. Determine if sotf_dist’s are listening on the
specified port:

Type telnet localhost 5040

If command is refused, the sotf_dist is not listening on the port.
Continue with step 5.

5. Type netstat -a | grep 5040

If a connection is established for this port then most
likely the sotf_dist is listening on this port.

TOP SECRET II $1II REL TO USA, AUS, CAN. GBR, NZL

121xks-system-administration-p122-normal.gif:
SOTF Problems continued

netstat will tell if...

• sotf dist is listening for connections

• If connections have been made to the sotf_dist

• If we are “backing up”- i.e., if sotf_dist is running
but has no process_data’s connected to it, it won’t
be able to send data anywhere, so eventually its
network receive queue will get large.

► Ideally, the receive queue should always be 0.xks-system-administration-p123-normal.gif:
SOTF Problems continued

Is the process_data_parent running?

• At least one process_data must be running and
synchronized with the sotf_dist for it to receive
input.

► If problems continue, run the sotf_dist in a terminal to
further troubleshoot and identify error messages.

TOP SECRET//:

. TO USA, i

J, GBR, I

123xks-system-administration-p124-normal.gif:
MySQL - D124 file troubleshooting

Symptom: A lot of errors or too many
errors display when performing the
command

‘mysqls status’:

1. First try, mysqls cleanup, in a terminal
window.

2. Type mysqls status

3. Type mysqi xs task_db; to log into MySQL
database and use the xs_task_db database.

4. Execute the following command: delete
from tar files where starus="error";

5. Exit out of the MySQL database

6. Type mysqls status

There sh©uk*TnG*totta@rTJfir

r

■ The heart of the XKEYSCORE processing
engine is the xscore_proc with related plugins.

■ Input to the xscoreproc is either file-based
and from an file_input_proc, or streaming from
an sotf_input_proc.

■ After processing, the written metadata to the
insert databases can be sent to a follow on
system for additional processing.

TOPSECRET/

.TO USA, AU$,<

. GBR, t

125xks-system-administration-p126-normal.gif:
Processing Problems continued..

m How many process_data’s should be
running on a host?

• From the XKEYSCORE GUI:

► Click ADMIN > Processing > Computer Resources

► Determine how many process_data’s are configured to
be running on the specified host.

TOP SECRET II $1II REL TO USA, AUS, CAN. GBR, NZL

126xks-system-administration-p127-normal.gif:
Processing Problems continued

How many xscore_proc’s are actually
running on a host?

• Log onto the XKEYSCORE server and open a
terminal window.

• Type ps -ef | grep xscore | grep -v managed

IfcT-lfoifö '1$ [< -it 1 ref xsrn | re? *t io;?: i.f.i ¡XL USE IS 3k3! (7:38:21 sere v.x -mi --te* act HD -«Serai nfii -4 iO -toe 1ml • -okM 4 to as: --ki-k 3) -to
: -±ilq±l m --relfcil I'M -tlfcc 3 ill--iilto S710 Cft ¿54*2 tiSH ti 1$£ ’ (0:05:1: Esrejc« Aili-l --pet 581 -4 iO -iKiicElml - -and 4 to_ttU -bmc 3) —loglB li e:nt
■fit $9$ 23615 IMS! (0:05:9 bsscjhx iiitf-2 -pet 582 ~t> iO -nriicelml • --entnd 4 tor«: -b:-k 3) -I05]« i. m
Cfst 2*710 22561 X 1$.*:5(0:05:9 nzsj(fir m m ¿1 m l rn± nmjmt JttIM --pet KM “«tenBljffiidjaplfi -4? iO -toelml - --cnal 4_to_r«:. -kkc 31 -logic* Tiernt

TOP SECRET II Si II REL TO USA, AUS, CAN. GBR, NZL

127xks-system-administration-p128-normal.gif:
Processing Problems continued..

• xks_app_launcher is running, but not
starting processes specified in the
Computer Resources window?

• This may indicate that the xks_app_launcher is
defunct. Use the kill command to kill the
app launcher and its related sub-processes:

► Typepkill -f app_

■ If a PID is not being specified, use the pkill command. The -f
option kills all of the sub-processes.

► Type ps to look for the new xks app launcher process.

TOPSECRETi

TO USA,,

J, GBR, I

128xks-system-administration-p129-normal.gif:
rOP SECRET U SI IIREL TO USA. AUS. CAN. GBR. NZL

Processing Problems continued..

r

• If, after performing the procedures, the
xks app launcher is still not starting applications:
► In a terminal window, manually run the problem process
to see if there are any error messages.

► The xks_app_launcher on any host is dependent on the
access of the xs_task_db.proc_resources database
table on the master. Verify that the specified host can
access the master’s database and /opt directory.

► On the slave system type mysqi xs_task_db
-h

■ performs a remote MySQL server login

TOP SECRET II $1II REL TO USA, AUS, CAN. GBR, I

129xks-system-administration-p130-normal.gif:
Processing Problems continued.

r

• To test the xscore proc, type:

telnet

Optional commands to assist trouble shooting are:

• sbr - prints the processing rate for the single

xscore-proc.

• sh - displays dictionary hit statistics.

• ss - displays statistics on the internal plug-in

processing rates.

• help - there are many commands and can be
described in the help menu.

TOPSECRETj

TO USA,,

J, GBR, I

130xks-system-administration-p131-normal.gif:
TOP SECRET II SI II REL TO USA.

. CAN. GBR. NZL



■ If the process_data_parent continues to deny
access through the command port, and input
still has not started processing, check the
input source.

■ Run the process in a terminal window with the
argument -loglevel debug, to view debug
messages.

■ The command port also provides processing
rates and statistics for troubleshooting
performance issues, outages, and general
administration issues.

TOP SECRET/

.TO USA, AUS,<

131xks-system-administration-p132-normal.gif:
5. CAN. GBR. NZL

rr

7

m /export/data/xkeyscore/outputs/mailorder

■ If there are no new files in the MAILORDER
directory, MAILORDER may not be working
properly. Possible causes are that:

• Files are being written to the wrong directory or it
is not configured properly

• Permissions on the MAILORDER directory will not
allow MAILORDER to move files

TOP SECRET//J

.TO USA, AUS,<

, GBR, t

132xks-system-administration-p133-normal.gif:
■ Query dispatch is the process that submits
search jobs to search databases and
propagates the status of the search and the
results of the search back to the web server.

■ After submitting a new query, Search Status
window displays a summary listing query
name, date and time submitted, number of
databases complete, and number of results.

TOP SECRET II $1II REL TO USA, AU$, CAN. GBR, NZL

133xks-system-administration-p134-normal.gif:
TOP SECRET II SI IIREL TO USA. AUS. CAN. GBR. NZL

V

■ The query never moves to the finished state.

• If a database outage or a comms outage occurs,
results will not be reported from the single system.
However, results from all other databases will
return properly with the query results, but they will
not appear in this state.

TOP SECRET II $1II REL TO USA, AUS, CAN. GBR, I

134xks-system-administration-p135-normal.gif:
TOP SECRET // SI II REL TO USA. AUS. CAN. GBR. NZL

V

■ Query job status is stuck in
awaiting_disbatch.

• If a status appears stuck in this state, the
query_dispatch may not be running on the web
server. To determine whether it is running:

► Type ps -ef | grep query_

• If the process is not running, restart it from the
XKEYSCORE GUI or troubleshoot the
xks_app_launcher.

TOP SECRET II $1II REL TO USA, AUS,

CAN. GBR, I

135xks-system-administration-p136-normal.gif:
OP SECRET II SI IIREL TO USA, AUS. CAN. GBR. NZL

¥

• Another cause of this scenario is that a query
database may have hung up the query dispatch
process. Check the progress of queries on the
query database hosts by viewing the table
sdb_queryJobs in the query database, which
tracks the status of queries:

► Typenysql qO

► Type select status,count(*) from sdb_query_jobs
where group by status;

• The select statement displays the current state of
the queries on the query host. If many more
queries appear in the new state when compared
to other query databases, begin troubleshooting
the problem query_proc on the specified query
database.

TOP SECRET II $1II REL TO USA, AUS, CAN. GBR, I

136xks-system-administration-p137-normal.gif:
TOP SECRET II SI IIREL TO USA. AUS. CAN. GBR. NZL

¥

■ The query is in the sent state, but never
appears in new.

• After the query_dispatch process disbatches the
query, the status is moved to sent. A query moves
to the new state when the query has been placed
in the query processing queue on the query_host.

• If a query does not move to the new state in a
reasonable amount of time, the connectivity of the
database should be tested.

TOP SECRET II SI II REL TO USA, AUS, CAN. GBR, I

137xks-system-administration-p138-normal.gif:
OP SECRET II SI IIREL TO USA. AUS. CAN. GBR. NZL

¥

• To check the progress of queries on the query
database hosts, view the table sdb_queryJobs in
the query database, which tracks the status of
queries:

► Typ© nysql qO

► Type select status, count (*) from sdb_query_jobs

• The select statement displays the current state of
the queries on the query host. If many more
queries appear in the new state when compared
to other query databases, begin troubleshooting
the problem query_proc on the specified query
database.

where group by status

TOP SECRET II $1II REL TO USA, AUS, CAN. GBR, NZL

138xks-system-administration-p139-normal.gif:
TOP SECRET // SI II REL TO USA. AUS. CAN. GBR. NZL

¥

• The query appears in the new state, but
never finishes.

• query is in the new state, has been received by
the query host and placed in a queue waiting to be
processed.

• Queries can become backlogged with a large
number of queries waiting in the new state,
though the query_proc is processing the queries
properly. It is hard to predict the time to work off a
query backlog, but using the following select
statement the status of queries for the current day
can be checked for processing trends.

top secret //si m

TO USA,,

, AUS, CAN. GBR, I

139xks-system-administration-p140-normal.gif:
TOP SECRET II SI IIREL TO USA. AUS. CAN, GBR. NZL

Query Processing

■ To display the number queries in each state for
the current day:

• Type

select status,

count(*),datetime_submitted,(UNIX_TIMESTAMP(now()) -
UNIX TIMESTAMP(datetime_submitted))/3600 from
sdb query jobs where (datetime submitted>(now() - INTERVAL
'1' DAY)) group by status;

■ To display the number of queries processed per
hour for the current day:

• Type

select status,count(*)/24 AS queries_per_hour from
sdb_query_jobs where cancel!="C" and

(datetime_submitted>(now() - INTERVAL ’1' DAY)) AS Backlog
group by cancel;

■ If processing properly, queries can take hours, if not
days, to complete based on the backlog and the
processing trends.

TOP SECRET II Si II REL TO USA, AUS, CAN, GBR, NZL

140xks-system-administration-p141-normal.gif:
Retrieving Metadata and Content

■ Queries complete but there are no results.

• If queries complete, but no results are visible,
verify that the date range of the query coincides
with the collection date of the data. If using test
data, test the query system by putting the start
date range at a year or two older to assure it is not
old test data.

• Verify that query metadata is in the query
database by checking the contents of the
/export/data/xkeyscore/mysq!/{query_db}/
directory.

TOP SECRET//:

. TO USA, i

J, GBR, I

141xks-system-administration-p142-normal.gif:
Retrieving Metadata and Content

■ Queries complete and metadata returns,
but there is no content.

• The metadata in the XKEYSCORE viewer
displays the host and directory path of the content
file. Verify the content file exists using the Is -I
command. Trace a dataflow issue if the file does
not exists. If the content file exists, confirm the
httpd daemon is started on all slave systems. To
confirm the httpd daemon:

• 1. Type su - oper

• 2. Type xks status httpd

• 3. If the daemon is not on, type xks start
httpd

TOPSECRETj

. TO USA, i

J, GBR, I

142xks-system-administration-p143-normal.gif:
■ To troubleshoot problems with metadata or
content from a query, it will be necessary to
retrieve the actual content, since recreating
the problem is very difficult. This can be
accomplished from the XKEYSCORE GUI.
Click RESULTS and begin a search of the
questionable queries.

TOP SECRET // $1II REL TO USA, AU$, CAN. GBR, NZL

143xks-system-administration-p144-normal.gif:
TOP SECRET //SI II REL TO USA. AUS, CAN. GBR. NZL

TOP SECRET II si II REL TO USA, AUS, CAN, GBR, NZL

144


Download Document

XKEYSCORE System Administration (xks-system-administration-p1-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p2-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p3-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p4-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p5-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p6-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p7-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p8-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p9-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p10-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p11-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p12-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p13-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p14-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p15-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p16-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p17-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p18-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p19-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p20-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p21-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p22-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p23-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p24-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p25-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p26-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p27-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p28-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p29-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p30-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p31-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p32-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p33-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p34-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p35-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p36-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p37-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p38-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p39-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p40-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p41-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p42-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p43-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p44-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p45-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p46-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p47-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p48-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p49-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p50-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p51-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p52-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p53-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p54-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p55-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p56-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p57-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p58-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p59-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p60-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p61-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p62-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p63-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p64-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p65-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p66-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p67-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p68-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p69-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p70-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p71-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p72-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p73-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p74-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p75-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p76-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p77-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p78-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p79-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p80-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p81-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p82-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p83-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p84-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p85-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p86-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p87-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p88-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p89-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p90-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p91-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p92-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p93-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p94-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p95-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p96-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p97-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p99-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p100-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p101-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p102-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p103-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p104-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p105-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p106-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p107-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p108-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p109-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p110-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p111-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p112-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p113-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p114-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p115-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p116-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p117-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p118-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p119-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p120-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p121-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p122-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p123-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p124-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p125-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p126-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p127-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p128-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p129-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p130-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p131-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p132-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p133-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p134-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p135-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p136-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p137-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p138-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p139-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p140-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p141-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p142-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p143-normal.gif)

Download Document

XKEYSCORE System Administration (xks-system-administration-p144-normal.gif)

e-Highlighter

Click to send permalink to address bar, or right-click to copy permalink.

Un-highlight all Un-highlight selectionu Highlight selectionh