Title: Making Things Measurable: Technology Trending Challenges and Approaches

Release Date: 2014-12-28

Document Date: 2012-06-06

Description: This 40-page NSA presentation for the June 2012 SIGDEV conference includes a ranking of cryptographic protocols in order of ‘risk’ the they pose to the agency’s operations: see the Der Spiegel story Prying Eyes: Inside the NSA’s War on Internet Security, 28 December 2014.

Document: TOP SECRET//COMINT//REL FVEY//20340601

SIGDEV Conference 2012

(U) Making Things Measureable:

Technology Trending Challenges and

Approaches

June 2012

TOP SECRET//COMINT//REL FVEY//20340601

Derived from NSA/CSSM 1
Dated 20070108
Declassify On: 2037050

TOP SECRET//COMINT//REL FVEY//20340601

Overview (U)

• (U) Setting the Stage

- Strategic Surprise, Priority Needs, Definitions

• (U) Making Things Measurable

- Emerging Technology Discovery

- Technology Use Discovery

• (U) Challenges

- Complexity

- Getting data is only step 1

- Visualization

- Building outreach and engagement

TOP SECRET//COMINT//REL FVEY//20340601

TOP SECRET//COMINT//REL FVEY//20340601

CT Trends Focus Questions (U)

(U) Does NSA CT know what technologies,
communications products and applications, and
modus operandi are being used by terrorists,
terrorist groups, or in locations of interest?

(U) Does NSA CT know what emerging
technologies, communications products and
applications, and modus operandi are likely to be
used by terrorists, terrorist groups, or in locations

of interect?

Prevent Strategic
Surprise

TOP SECRET//COMINT//REL FVEY//20340601

TOP SECRET//COMINT//REL FVEY//20340601

CT Trends Focus Questions (U)

(U) Does NSA CT know what technologies,
communications products and applications, and
modus operandi are being used by terrorists,
terrorist groups, or in locations of interest?

(U) Does NSA CT know what emerging
technologies, communications products and
applications, and modus operandi are likely to be
used by terrorists, terrorist groups, or in locations

(U/RbL) What°&P/eret?y asking is:

Can we tell which ones are likely to become

---------------a priority.need?

TOP SECRET//COMINT//REL FVEY//20340601

TOP SECRET//COMINT//REL FVEY//20340601

Risk Management for SIGINT

Threats (U)

• (S//REL) Threat to SIGINT Capability

- A behavior or technology that has the potential to have a negative
impact on NSA's capability to provide SlGINT on a Terrorism Target

• (U) Use Risk

The possibility that a particular threat will be adopted by Terrorist targets

• (S//REL) Indications and Warning

- Early warning of high impact threats to prevent surprise to key
stakeholders and reduce risk from Terrorist adoption of technology that
would adversely affect SIGINT production

r (5//REL) NSA's ability TO manage risk '
is directly proportional to our ability

'---------to detect threats----------'

TOP SECRET//COMINT//REL FVEY//20340601

TOP SECRET//COMINT//REL FVEY//20340601

The data-driven approach

"Count what is countable, measure what is measurable, and make measurable that

which cannot be measured"

_ Galileo (17th century astronomer)

"When you can measure what you are speaking about, and express it in numbers,
you know something about it; but when you cannot measure it, when you cannot
express it in numbers, your knowledge is of a meagre and unsatisfactory kind"

_ Lord Kelvin (discovered absolute zero)

"Not everything that counts can be counted, and not everything that can be counted

"You cannot manage what you cannot measure"
Bill Hewlett (co-founder of Hewlett-Packard)

counts"

- Albert Einstein

TOP SECRET//COMINT//REL FVEY//20340601

So... what is a (CT) trend?

A trend is a measurement ofoccurrepce

(S//REL) Comparing the behavior of a single target...

- Pattern-of-life

- Modus Operandi

- Technology Usage

.to the behaviors seen within the target space

Multiple targets, within and across the entire CT enterprise

- Over a period of time

TOP SECRET//COMINT//REL FVEY//20340601

Prediction and Identification of
Priority Needs Prevents Strategic

Surprise (U)

Known

Identify issues that are
emerging into and
rising within the target
space

Rising

Emerging

TOP SECRET//COMINT//REL FVEY//20340601

TOP SECRET//COMINT//REL FVEY//20340601

Making Things Measurable

irn

im§Fain|

Tsghn§r§p§ Th§y|ht t§a§l§F§

T§ghnlgal Tgghnglefie

in u§e

TOP SECRET//COMINT//REL FVEY//20340601

TOP SECRET//COMINT//REL FVEY//20340601

Innovation Phases (U)

Adoption

Experimentation

Interest

TOP SECRET//COMINT//REL FVEY//20340601

TOP SECRET//COMINT//REL FVEY//20340601

Technology Adoption Factors

JU)

TOP SECRET//COMINT//REL FVEY//20340601

Optics (U)

• (S//REL) Optic #1: Emerging Technology Discovery

Focused primarily on interest and experimentation phases of innovation

- Watching the Watchers

- Weaker indicators

- New technologies

• (S//REL) Optic #2: Technology Use Discovery

Focused primarily on adoption phase of innovation

- Owning the Known

- Stronger indicators

- New targets

12

TOP SECRET//COMINT//REL FVEY//20340601

TOP SECRET//COMINT//REL FVEY//20340601

Analytics and Processes (U)

Producti

Technical Thought

Producti

Technical Thought

Production Element

FORTREND - Extremist

Production Element

FORTREND - Extremist

* Producti
_____on _ _ _ _

Technical Thought

Production Element
Scannina Note Proioct
FORTREND - Extremist

Administrative Response

Administrative Response

Administrative Response

Seized Media * Seized Media * Seized Media

TOP SECRET//COMINT//REL FVEY//20340601

Optic #1: Emerging Technology

Discovery (U)

• (S//SI//REL) Emerging Technology & Behavior Discovery

Detection of interest, experimentation, knowledge transfer or direction using
content, metrics approaches

Currently using deskside & virtual engagement to leverage TOPI analyst
initiative to discover, prioritize, and work against "strongest" indicators

- Leverages inherent TOPI expertise and functions of traffic processing/translation/tasking
etc..

- Embedded analysts, virtual relationships: production "customers"

- Currently identifying, tracking 'technical' thought leaders

- Technical sub-forums, scanning notes measurements

- Administrative emails (No-Reply etc..)

- Forum links, uploaded/downloaded files

Goal: Generate Prioritized Input (techs/behaviors) for Research

TOP SECRET//COMINT//REL FVEY//20340601

Optic #2: Technology Use

Discovery (U)

. (S//SI//REL) Technology Use and Behavior Discovery

_ "Stratactical" data sets

_ Includes target-specific data point for each item (e.g. selector)

_ Discovery of target behavior by identifying technology use patterns, trends, and/or
anomalies in:

_ User-agents (browsers, OS, devices)

_ Tasking (new tasking, total tasking)

_ Network, Protocol usage (Active User metrics)

_ Visited URLs, web searches

_ Process lists, pre-fetch logs, registry entries, software logs
_ Hardware usage (smartphones, tablets, SD cards)

_ Currently using various tools (XKEYSCORE, SEEKER, BIONICTURTLE, JEMA, JOLLYROGER,
MARINA, TUNINGFORK, QFDs, etc...) and approaches with multiple cloud analytics in
varying stages of development and/or planning

Goal: Generate Prioritized Input (techs/behaviors) for Research

15

TOP SECRET//COMINT//REL FVEY//20340601

TOP SECRET//COMINT//REL FVEY//20340601

Measurement Drives Research (U)

(S//REL) Triage begins with target indicators of a new technology

Derived from either optic: Emerging or Use Discovery

Interest, Experimentation, Use, Knowledge Transfer, Metric, etc...

Target a Technology a Do other targets use this technology?

This is the central defining question for Trends Analysis:

Do other CT targets use this technology?

TOP SECRET//COMINT//REL FVEY//20340601

TOP SECRET//COMINT//REL FVEY//20340601

Weak vs. Strong Indicators: Brutal

Triage (U)

Exper • Exper • Exper

Previous/Low • Previous/Low • Previous/Low

Log files, traffic • Log files, traffic

TOP SECRET//COMINT//REL FVEY//20340601

Log files, traffic

TOP SECRET//COMINT//REL FVEY//20340601

The Wicked Problem Aspect (U)

(S//REL) Defining the problem is the first (wicked)

problem

- Triage Stage 1

- Initial priority: (single) target + initial understanding of technology

- Implications Research

- What does the product/service do?

- Current NSA capabilities to detect, collect, exploit, analyze?

- Do any other CT targets use it?

- Triage Stage 2

- Updated priority: target(s) + updated understanding of tech/USSS

- Validated Next Steps

- As needed: capabilities/access development requirements

- Reporting: internal, CIR, e-gram; Gaps report; prioritization w/in tech category

18

TOP SECRET//COMINT//REL FVEY//20340601

TOP SECRET//COMINT//REL FVEY//20340601

Goal: Periodic Reporting

Vehicle (U)

. (U//FOUO) Move beyond ad hoc task responses to routine deliverables

. (U//FOUO) Overcoming volume challenge

- Huge variety of inputs, massive numbers in each
_ Prioritization

- Visualization

. (S//REL) Moving threats to a simple Risk Assessment model

- Borrows methodology from models used for executive purposes elsewhere in agency

- (FAMT, Geopolitical Technology Trends Matrix, TAO...)

- Opportunities, threats handled separately

TOP SECRET//COMINT//REL FVEY//20340601

TOP SECRET//COMINT//REL FVEY//20340601

Capabilities Development Risk

M^Eriv /I h

TRIVIAL MINOR MODERATE MAIOR CATASTROPHIC
Impact Loss/lack of Loss/lack of Loss/lack of Loss/lack of Near-total
insight to small insight to insight to large insight to loss/lack of
> aspect of target significant component of majority of insight to target
to production communications aspect of target target communications
, presence target communications, communications , presence
Mse communication presence , presence
s, presence
Risk
v
Current Highest Priority Target Use Document tracking Fivewes, Facebook chat presentation Mail.ru, TeamViewer, Join.me OTR, Tor, Smartphones, Zoho.com webmail, TrueCrypt Tor+ Trilight Zone + Cspace + ZRTP VoIP client on Linux

Current
Operational
Target Use

Current Low
Priority/Previou
s Higher
Priority Target
Use

■ Technical

Thought Leader
Recommendati
ons,

Pvnerimentatio

TOP SECRET//COMINT//REL FVEY//20340601

TOP SECRET//COMINT//REL FVEY//20340601

Capabilities Development

Impact

>

to production

Use

Risk

v

Curre
Highest P
Target

Cui rent
Operational
Target Use

Current Low
Priority/Previou
s Higher
Priority Target
Use

Technical
Thought Leader
Recommendati
ons,

Experimentatio

TRIVIAL MINOR MODERATE MAIOR CATASTROPHIC
Loss/lack of insight to small aspect of target communications , presence Loss/lack of insight to significant aspect of target communication s, presence Loss/lack of insight to large component of target communications, presence Loss/lack of insight to majority of target communications , presence Near-total loss/lack of insight to target communications , presence
document tracking Fivewes, Facebook chat Mail.ru, TeamViewer, Join.me OTR, Tor, Smartphones, Fastmail, Tor+ Trilight Zone + Cspace + ZRTP VoIP
m i p.e ent~tion rypt client on Linux

(TS//SI//REL) With
rare exceptions,
application-specific
solutions are only
built based on these
two criteria

TOP SECRET//COMINT//REL FVEY//20340601

TOP SECRET//COMINT//REL FVEY//20340601

Capability Development
Challenges (U)

(TS//SI//REL) With rare exceptions,
application-specific solutions are only
built based on these two criteria????

. In resource-restrained environment, development of capabilities against likely-
to-increase in priority applications is trumped by standing requirements
driven by known priority applications

. Capabilities development response to current/priority technology threats occurs
normally w/in existing resources - but response does not scale, either to the
industry or to multiple crises

TOP SECRET//COMINT//REL FVEY//20340601

Simplifying the Risk Matrix (U)

Impact

>

to production

Use

Risk

TRIVIAL MINOR MODERATE MAIOR CATASTROPHIC
Loss/lack of insight to small aspect of target communications , presence Loss/lack of insight to significant aspect of target communication s, presence Loss/lack of insight to large component of target communications, presence Loss/lack of insight to majority of target communications , presence Near-total loss/lack of insight to target communications , presence

Current

Highest Priority
Target Use

Current Low
Priority/Previou
s Higher
Priority Target
Use

Technical
Thought Leader
Recommendati
ons,

Experimentatio

n

2nd Quadrant

4th Quadrant

TOP SECRET//COMINT//REL FVEY//20340601

TOP SECRET//COMINT//REL FVEY//20340601

Adding in the Solution

Impact

>

to stakeholders

Use

Risk

v

Current Low
Priority/Previou
s Higher
Priority Target
Use

Technical
Thought Leader
Recommendati
ons,

Experimentatio

4th Quadrant

TOP SECRET//COMINT//REL FVEY//20340601

t

nal

se

Current

Highest Priority
Target Use

Automated
solutions not
required;
manual
workflows
sufficient:
XKS fingerprint

Minimum
automated
solutions
required:
realm creation,
simple
extraction,

presents. ■

Significant,
routine capdev
required:
STARPROC
capability, CES
detectors,
endpoint

■ctoaftepiaiiPB',

Requires
focused inter-
office capdev:
Lead/Program
manager
needed, SPF
required, inter-
. .ofiisetfpfip. i

2nd Quadrant

TRIVIAL

MINOR

MODERATE

MAIOR

CATASTROPHIC

Requires SID-
level attention:
FAMT, LAE;
WPMO, AMOD
portfolio
integration

TOP SECRET//COMINT//REL FVEY//20340601

Examples: Jan-February 2012

/TQ//QI//REM

Impact

>

to production

Use

Risk

Current

Highest Priority
Target Use

Current Low
Priority/Previou
s Higher
Priority Target
Use

Technical
Thought Leader
Recommendati
ons,

Experimentatio

n

TRIVIAL

Loss/lack of
insight to small
aspect of target
communications
, presence

MINOR

Loss/lack of
insight to
significant
aspect of
target

communication
s, presence

MODERATE

Loss/lack of
insight to large
component of
target

communications,

presence

MAIOR CATASTROPHIC
Loss/lack of Near-total
insight to loss/lack of
majority of insight to target
target communications
communications , presence , presence



TeamViewer

Join.Me
LaplinkGold

Hi

I a

Purematrlmony
______com

Zemana Antl-
Keylogger

..........I.........

Web.de
Cspace
Redphone

TOP SECRET//COMINT//REL FVEY//20340601

se

TOP SECRET//COMINT//REL FVEY//20340601

Goal: EmergingTechnology

Snapshot(U)

• (U) Executive version - snapshot of top items only

• (S//REL) Overcoming the challenges of prioritization and volume
is still only 50% of the problem

• (S//REL) Stated Preference:

Breakdowns by target/target set

- Preserve opportunity vs. threat

- Identify HUMINT sources for collaboration

TOP SECRET//COMINT//REL FVEY//20340601

TOP SECRET//COMINT//ORCON//REL FVEY//20340601

Emerging Technology
Snapshot(U)

Target/Org Tech Quadrant
AQSL courier TAILS
GIMF TAILS 1
AQ media TrueCrypt
S2I42 Join.Me
LT, S2I42 TeamViewer 2
LT Laplink
TTL Extremist version of Tor Opportu nity
AQ media Encrypted Webmail Source

(TS//SI//REL) Full details available as needed

TOP SECRET//COMINT//REL FVEY//20340601

TOP SECRET//COMINT//REL FVEY//20340601

Emerging Technology
Snapshot(U)

• (S//REL) Monthly Emerging Technology Snapshot

- 1-3 page Snapshot (6 page max if previous month data
included) to CT leadership

- Snapshot + supporting full data to MICROEXPANSE

- Underlying processes in alpha stage

- Stopgap until maturation of multiple efforts

- Data Explorer, ECHOBASE

- Inclusion of FAA/PRISM in GM-Halo

TOP SECRET//COMINT//REL FVEY//20340601

TOP SECRET//COMINT//REL FVEY//20340601

End Results - Tactical &

Strategic (U)

. (S//REL) Tactical Outcomes

- Lead Generation

- Target Development

- Target Discovery

- Behavior Detection

- Access Prioritization

. (S//REL) Strategic Outcomes

- Prioritization for Capabilities Development

- Driven by target priority: single target + volume of targets

- Prioritized within tech category, target (set) category

- Overall CT product line prioritization

TOP SECRET//COMINT//REL FVEY//20340601

TOP SECRET//COMINT//REL FVEY//20340601

Challenges (U)

. (C//REL) Complexity

- Understand target, technology, & SIGINT system

. (S//SI//REL) Getting data is only step 1

- Getting a data set is like to getting a new bearer to analyze

. (U) Visualization

- Excel tops out at a million rows...

. (TS//SI//REL) Clean data

- Targets vs. Selectors

TOP SECRET//COMINT//REL FVEY//20340601

TOP SECRET//COMINT//REL FVEY//20340601

Overcoming Complexity (U)

"siGiNT^

Fingerspitzengefühl

• Literally "fingertip feeling"

• Empathy, sensitivity, tact

• Ability of military commanders
to react rapidly

CT Trends TeainChnolog^chn°|ogy

SIGDEV analysts
Partner/Enablers

Must understand tech threat
implications, provenance and
structure of data to manipulate,
interpret it

TOP SECRET//COMINT//REL FVEY//20340601

TOP SECRET//COMINT//REL FVEY//20340601

Getting Data is Step 1 (U)

TOP SECRET//COMINT//REL FVEY//20340601

TOP SECRET//COMINT//REL FVEY//20340601

Getting Data is Step 1 (U)

TOP SECRET//COMINT//REL FVEY//20340601

TOP SECRET//COMINT//REL FVEY//20340601

Getting Data is Step 1 (U)

TOP SECRET//COMINT//REL FVEY//20340601

TOP SECRET//COMINT//REL FVEY//20340601

Visualization (U)

(TS//SI//REL) Excel tops out at a million rows...

- 19 branches, 30+ target sets, ~200 realms, -800 domains, -45000
selectors = 1 million rows/~2.5 weeks for summarized active user events
from E012333 alone

- Spreadsheets are good, but not everyone knows how to use a
pivot table

- Each dataset can easily provide 4-5 or more pivoted looks for
each branch/target set = minimum 100-150 slides

((S//REL) Intent is to routinely produce\
multiple large datasets on a monthly
basis for collection management,
research purposes

TOP SECRET//COMINT//REL FVEY//20340601

TOP SECRET//COMINT//REL FVEY//20340601

Visualization (U)

(S//REL) Analysts work at the selector level
• Leadership wants data presented at the target level

(S//REL) Automated population of technology, behavior
information in analyst workflow tools, databases

(S//REL) Each separate visualization task takes
manpower, time away from operational analysis

TOP SECRET//COMINT//REL FVEY//20340601

TOP SECRET//COMINT//REL FVEY//20340601

Clean Data (U)

(S//SI//REL) Metrics will only provide a near-accurate picture: ground truth will always be the domain of
the TOPI and based on content

(S//SI//REL) Some selectors (accurately) map to multiple targets, multiple teams, multiple organizations

(S//SI//REL) Some selectors simply don't have a known target, only a target set

(S//REL) Need to correlate across widely different datasets requires creation of normalized bridge
datasets (e.g. comparing executables to domains)

(S//SI//REL) TKB/UTT are victims of years of "fill in the blank" freeform data entry; very slowly being
addressed (~2015?)

TOP SECRET//COMINT//REL FVEY//20340601

TOP SECRET//COMINT//REL FVEY//20340601

Rising Strategic Issues (U)

• (TS//SI//REL) Encrypted Webmail Services

Atabmail, Zoho, Safe-mail, Fastmail, HMA Mail

• (TS//SI//REL) Remote Desktop Viewers/Remote Access Tools

- TeamViewer, Join.me, Cybergate

• (TS//SI//REL) Aggregators/Over-the-Top Messaging Services

- WhatsApp, Nimbuzz, eBuddy

TOP SECRET//COMINT//REL FVEY//20340601

TOP SECRET//COMINT//REL FVEY//20340601

What Next? (U)

. (S//SI//REL) Continue to build, strengthen, expand:

- internal workflows, research and discovery capabilities

- collaboration with production elements

- Operational support via embedded analysts at NSAW

- Tradecraft, technical support virtually with extended enterprise

- partnerships with FVEY SIGDEV community

- Establish and expand dialogue opportunities

- "Failure Sharing" - tradecraft sharing and operational deconfliction

. (S//REL) Technology Trends MyNoc

TOP SECRET//COMINT//REL FVEY//20340601

TOP SECRET//COMINT//REL FVEY//20340601

Question

s?

TOP SECRET//COMINT//REL FVEY//20340601

Comment

s?

e-Highlighter

Click to send permalink to address bar, or right-click to copy permalink.

Un-highlight all Un-highlight selectionu Highlight selectionh