Title: Making Things Measurable: Technology Trending Challenges and Approaches
Release Date: 2014-12-28
Document Date: 2012-06-06
Description: This 40-page NSA presentation for the June 2012 SIGDEV conference includes a ranking of cryptographic protocols in order of ‘risk’ the they pose to the agency’s operations: see the Der Spiegel story Prying Eyes: Inside the NSA’s War on Internet Security, 28 December 2014.
Document: TOP SECRET//COMINT//REL FVEY//20340601
SIGDEV Conference 2012
(U) Making Things Measureable:
Technology Trending Challenges and
Approaches
June 2012
TOP SECRET//COMINT//REL FVEY//20340601
Derived from NSA/CSSM 1
Dated 20070108
Declassify On: 2037050
TOP SECRET//COMINT//REL FVEY//20340601
Overview (U)
• (U) Setting the Stage
- Strategic Surprise, Priority Needs, Definitions
• (U) Making Things Measurable
- Emerging Technology Discovery
- Technology Use Discovery
• (U) Challenges
- Complexity
- Getting data is only step 1
- Visualization
- Building outreach and engagement
TOP SECRET//COMINT//REL FVEY//20340601
TOP SECRET//COMINT//REL FVEY//20340601
CT Trends Focus Questions (U)
(U) Does NSA CT know what technologies,
communications products and applications, and
modus operandi are being used by terrorists,
terrorist groups, or in locations of interest?
(U) Does NSA CT know what emerging
technologies, communications products and
applications, and modus operandi are likely to be
used by terrorists, terrorist groups, or in locations
of interect?
Prevent Strategic
Surprise
TOP SECRET//COMINT//REL FVEY//20340601
TOP SECRET//COMINT//REL FVEY//20340601
CT Trends Focus Questions (U)
(U) Does NSA CT know what technologies,
communications products and applications, and
modus operandi are being used by terrorists,
terrorist groups, or in locations of interest?
(U) Does NSA CT know what emerging
technologies, communications products and
applications, and modus operandi are likely to be
used by terrorists, terrorist groups, or in locations
(U/RbL) What°&P/eret?y asking is:
Can we tell which ones are likely to become
---------------a priority.need?
TOP SECRET//COMINT//REL FVEY//20340601
TOP SECRET//COMINT//REL FVEY//20340601
Risk Management for SIGINT
Threats (U)
• (S//REL) Threat to SIGINT Capability
- A behavior or technology that has the potential to have a negative
impact on NSA's capability to provide SlGINT on a Terrorism Target
• (U) Use Risk
The possibility that a particular threat will be adopted by Terrorist targets
• (S//REL) Indications and Warning
- Early warning of high impact threats to prevent surprise to key
stakeholders and reduce risk from Terrorist adoption of technology that
would adversely affect SIGINT production
r (5//REL) NSA's ability TO manage risk '
is directly proportional to our ability
'---------to detect threats----------'
TOP SECRET//COMINT//REL FVEY//20340601
TOP SECRET//COMINT//REL FVEY//20340601
The data-driven approach
"Count what is countable, measure what is measurable, and make measurable that
which cannot be measured"
_ Galileo (17th century astronomer)
"When you can measure what you are speaking about, and express it in numbers,
you know something about it; but when you cannot measure it, when you cannot
express it in numbers, your knowledge is of a meagre and unsatisfactory kind"
_ Lord Kelvin (discovered absolute zero)
"Not everything that counts can be counted, and not everything that can be counted
"You cannot manage what you cannot measure"
Bill Hewlett (co-founder of Hewlett-Packard)
counts"
- Albert Einstein
TOP SECRET//COMINT//REL FVEY//20340601
So... what is a (CT) trend?
A trend is a measurement ofoccurrepce
(S//REL) Comparing the behavior of a single target...
- Pattern-of-life
- Modus Operandi
- Technology Usage
.to the behaviors seen within the target space
Multiple targets, within and across the entire CT enterprise
- Over a period of time
TOP SECRET//COMINT//REL FVEY//20340601
Prediction and Identification of
Priority Needs Prevents Strategic
Surprise (U)
Known
Identify issues that are
emerging into and
rising within the target
space
Rising
Emerging
TOP SECRET//COMINT//REL FVEY//20340601
TOP SECRET//COMINT//REL FVEY//20340601
Making Things Measurable
irn
im§Fain|
Tsghn§r§p§ Th§y|ht t§a§l§F§
T§ghnlgal Tgghnglefie
in u§e
TOP SECRET//COMINT//REL FVEY//20340601
TOP SECRET//COMINT//REL FVEY//20340601
Innovation Phases (U)
Adoption
Experimentation
Interest
TOP SECRET//COMINT//REL FVEY//20340601
TOP SECRET//COMINT//REL FVEY//20340601
Technology Adoption Factors
JU)
TOP SECRET//COMINT//REL FVEY//20340601
Optics (U)
• (S//REL) Optic #1: Emerging Technology Discovery
Focused primarily on interest and experimentation phases of innovation
- Watching the Watchers
- Weaker indicators
- New technologies
• (S//REL) Optic #2: Technology Use Discovery
Focused primarily on adoption phase of innovation
- Owning the Known
- Stronger indicators
- New targets
12
TOP SECRET//COMINT//REL FVEY//20340601
TOP SECRET//COMINT//REL FVEY//20340601
Analytics and Processes (U)
Producti
Technical Thought
Producti
Technical Thought
Production Element
FORTREND - Extremist
Production Element
FORTREND - Extremist
* Producti
_____on _ _ _ _
Technical Thought
Production Element
Scannina Note Proioct
FORTREND - Extremist
Administrative Response
Administrative Response
Administrative Response
Seized Media * Seized Media * Seized Media
TOP SECRET//COMINT//REL FVEY//20340601
Optic #1: Emerging Technology
Discovery (U)
• (S//SI//REL) Emerging Technology & Behavior Discovery
Detection of interest, experimentation, knowledge transfer or direction using
content, metrics approaches
Currently using deskside & virtual engagement to leverage TOPI analyst
initiative to discover, prioritize, and work against "strongest" indicators
- Leverages inherent TOPI expertise and functions of traffic processing/translation/tasking
etc..
- Embedded analysts, virtual relationships: production "customers"
- Currently identifying, tracking 'technical' thought leaders
- Technical sub-forums, scanning notes measurements
- Administrative emails (No-Reply etc..)
- Forum links, uploaded/downloaded files
Goal: Generate Prioritized Input (techs/behaviors) for Research
TOP SECRET//COMINT//REL FVEY//20340601
Optic #2: Technology Use
Discovery (U)
. (S//SI//REL) Technology Use and Behavior Discovery
_ "Stratactical" data sets
_ Includes target-specific data point for each item (e.g. selector)
_ Discovery of target behavior by identifying technology use patterns, trends, and/or
anomalies in:
_ User-agents (browsers, OS, devices)
_ Tasking (new tasking, total tasking)
_ Network, Protocol usage (Active User metrics)
_ Visited URLs, web searches
_ Process lists, pre-fetch logs, registry entries, software logs
_ Hardware usage (smartphones, tablets, SD cards)
_ Currently using various tools (XKEYSCORE, SEEKER, BIONICTURTLE, JEMA, JOLLYROGER,
MARINA, TUNINGFORK, QFDs, etc...) and approaches with multiple cloud analytics in
varying stages of development and/or planning
Goal: Generate Prioritized Input (techs/behaviors) for Research
15
TOP SECRET//COMINT//REL FVEY//20340601
TOP SECRET//COMINT//REL FVEY//20340601
Measurement Drives Research (U)
(S//REL) Triage begins with target indicators of a new technology
Derived from either optic: Emerging or Use Discovery
Interest, Experimentation, Use, Knowledge Transfer, Metric, etc...
Target a Technology a Do other targets use this technology?
This is the central defining question for Trends Analysis:
Do other CT targets use this technology?
TOP SECRET//COMINT//REL FVEY//20340601
TOP SECRET//COMINT//REL FVEY//20340601
Weak vs. Strong Indicators: Brutal
Triage (U)
Exper • Exper • Exper
Previous/Low • Previous/Low • Previous/Low
Log files, traffic • Log files, traffic
TOP SECRET//COMINT//REL FVEY//20340601
Log files, traffic
TOP SECRET//COMINT//REL FVEY//20340601
The Wicked Problem Aspect (U)
(S//REL) Defining the problem is the first (wicked)
problem
- Triage Stage 1
- Initial priority: (single) target + initial understanding of technology
- Implications Research
- What does the product/service do?
- Current NSA capabilities to detect, collect, exploit, analyze?
- Do any other CT targets use it?
- Triage Stage 2
- Updated priority: target(s) + updated understanding of tech/USSS
- Validated Next Steps
- As needed: capabilities/access development requirements
- Reporting: internal, CIR, e-gram; Gaps report; prioritization w/in tech category
18
TOP SECRET//COMINT//REL FVEY//20340601
TOP SECRET//COMINT//REL FVEY//20340601
Goal: Periodic Reporting
Vehicle (U)
. (U//FOUO) Move beyond ad hoc task responses to routine deliverables
. (U//FOUO) Overcoming volume challenge
- Huge variety of inputs, massive numbers in each
_ Prioritization
- Visualization
. (S//REL) Moving threats to a simple Risk Assessment model
- Borrows methodology from models used for executive purposes elsewhere in agency
- (FAMT, Geopolitical Technology Trends Matrix, TAO...)
- Opportunities, threats handled separately
TOP SECRET//COMINT//REL FVEY//20340601
TOP SECRET//COMINT//REL FVEY//20340601
Capabilities Development Risk
M^Eriv /I h
TRIVIAL MINOR MODERATE MAIOR CATASTROPHIC
Impact Loss/lack of Loss/lack of Loss/lack of Loss/lack of Near-total
insight to small insight to insight to large insight to loss/lack of
> aspect of target significant component of majority of insight to target
to production communications aspect of target target communications
, presence target communications, communications , presence
Mse communication presence , presence
s, presence
Risk
v
Current Highest Priority Target Use Document tracking Fivewes, Facebook chat presentation Mail.ru, TeamViewer, Join.me OTR, Tor, Smartphones, Zoho.com webmail, TrueCrypt Tor+ Trilight Zone + Cspace + ZRTP VoIP client on Linux
Current
Operational
Target Use
Current Low
Priority/Previou
s Higher
Priority Target
Use
■ Technical
Thought Leader
Recommendati
ons,
Pvnerimentatio
TOP SECRET//COMINT//REL FVEY//20340601
TOP SECRET//COMINT//REL FVEY//20340601
Capabilities Development
Impact
>
to production
Use
Risk
v
Curre
Highest P
Target
Cui rent
Operational
Target Use
Current Low
Priority/Previou
s Higher
Priority Target
Use
Technical
Thought Leader
Recommendati
ons,
Experimentatio
TRIVIAL MINOR MODERATE MAIOR CATASTROPHIC
Loss/lack of insight to small aspect of target communications , presence Loss/lack of insight to significant aspect of target communication s, presence Loss/lack of insight to large component of target communications, presence Loss/lack of insight to majority of target communications , presence Near-total loss/lack of insight to target communications , presence
document tracking Fivewes, Facebook chat Mail.ru, TeamViewer, Join.me OTR, Tor, Smartphones, Fastmail, Tor+ Trilight Zone + Cspace + ZRTP VoIP
m i p.e ent~tion rypt client on Linux
(TS//SI//REL) With
rare exceptions,
application-specific
solutions are only
built based on these
two criteria
TOP SECRET//COMINT//REL FVEY//20340601
TOP SECRET//COMINT//REL FVEY//20340601
Capability Development
Challenges (U)
(TS//SI//REL) With rare exceptions,
application-specific solutions are only
built based on these two criteria????
. In resource-restrained environment, development of capabilities against likely-
to-increase in priority applications is trumped by standing requirements
driven by known priority applications
. Capabilities development response to current/priority technology threats occurs
normally w/in existing resources - but response does not scale, either to the
industry or to multiple crises
TOP SECRET//COMINT//REL FVEY//20340601
Simplifying the Risk Matrix (U)
Impact
>
to production
Use
Risk
TRIVIAL MINOR MODERATE MAIOR CATASTROPHIC
Loss/lack of insight to small aspect of target communications , presence Loss/lack of insight to significant aspect of target communication s, presence Loss/lack of insight to large component of target communications, presence Loss/lack of insight to majority of target communications , presence Near-total loss/lack of insight to target communications , presence
Current
Highest Priority
Target Use
Current Low
Priority/Previou
s Higher
Priority Target
Use
Technical
Thought Leader
Recommendati
ons,
Experimentatio
n
2nd Quadrant
4th Quadrant
TOP SECRET//COMINT//REL FVEY//20340601
TOP SECRET//COMINT//REL FVEY//20340601
Adding in the Solution
Impact
>
to stakeholders
Use
Risk
v
Current Low
Priority/Previou
s Higher
Priority Target
Use
Technical
Thought Leader
Recommendati
ons,
Experimentatio
4th Quadrant
TOP SECRET//COMINT//REL FVEY//20340601
t
nal
se
Current
Highest Priority
Target Use
Automated
solutions not
required;
manual
workflows
sufficient:
XKS fingerprint
Minimum
automated
solutions
required:
realm creation,
simple
extraction,
presents. ■
Significant,
routine capdev
required:
STARPROC
capability, CES
detectors,
endpoint
■ctoaftepiaiiPB',
Requires
focused inter-
office capdev:
Lead/Program
manager
needed, SPF
required, inter-
. .ofiisetfpfip. i
2nd Quadrant
TRIVIAL
MINOR
MODERATE
MAIOR
CATASTROPHIC
Requires SID-
level attention:
FAMT, LAE;
WPMO, AMOD
portfolio
integration
TOP SECRET//COMINT//REL FVEY//20340601
Examples: Jan-February 2012
/TQ//QI//REM
Impact
>
to production
Use
Risk
Current
Highest Priority
Target Use
Current Low
Priority/Previou
s Higher
Priority Target
Use
Technical
Thought Leader
Recommendati
ons,
Experimentatio
n
TRIVIAL
Loss/lack of
insight to small
aspect of target
communications
, presence
MINOR
Loss/lack of
insight to
significant
aspect of
target
communication
s, presence
MODERATE
Loss/lack of
insight to large
component of
target
communications,
presence
MAIOR CATASTROPHIC
Loss/lack of Near-total
insight to loss/lack of
majority of insight to target
target communications
communications , presence , presence
TeamViewer
Join.Me
LaplinkGold
Hi
I a
Purematrlmony
______com
Zemana Antl-
Keylogger
..........I.........
Web.de
Cspace
Redphone
TOP SECRET//COMINT//REL FVEY//20340601
se
TOP SECRET//COMINT//REL FVEY//20340601
Goal: EmergingTechnology
Snapshot(U)
• (U) Executive version - snapshot of top items only
• (S//REL) Overcoming the challenges of prioritization and volume
is still only 50% of the problem
• (S//REL) Stated Preference:
Breakdowns by target/target set
- Preserve opportunity vs. threat
- Identify HUMINT sources for collaboration
TOP SECRET//COMINT//REL FVEY//20340601
TOP SECRET//COMINT//ORCON//REL FVEY//20340601
Emerging Technology
Snapshot(U)
Target/Org Tech Quadrant
AQSL courier TAILS
GIMF TAILS 1
AQ media TrueCrypt
S2I42 Join.Me
LT, S2I42 TeamViewer 2
LT Laplink
TTL Extremist version of Tor Opportu nity
AQ media Encrypted Webmail Source
(TS//SI//REL) Full details available as needed
TOP SECRET//COMINT//REL FVEY//20340601
TOP SECRET//COMINT//REL FVEY//20340601
Emerging Technology
Snapshot(U)
• (S//REL) Monthly Emerging Technology Snapshot
- 1-3 page Snapshot (6 page max if previous month data
included) to CT leadership
- Snapshot + supporting full data to MICROEXPANSE
- Underlying processes in alpha stage
- Stopgap until maturation of multiple efforts
- Data Explorer, ECHOBASE
- Inclusion of FAA/PRISM in GM-Halo
TOP SECRET//COMINT//REL FVEY//20340601
TOP SECRET//COMINT//REL FVEY//20340601
End Results - Tactical &
Strategic (U)
. (S//REL) Tactical Outcomes
- Lead Generation
- Target Development
- Target Discovery
- Behavior Detection
- Access Prioritization
. (S//REL) Strategic Outcomes
- Prioritization for Capabilities Development
- Driven by target priority: single target + volume of targets
- Prioritized within tech category, target (set) category
- Overall CT product line prioritization
TOP SECRET//COMINT//REL FVEY//20340601
TOP SECRET//COMINT//REL FVEY//20340601
Challenges (U)
. (C//REL) Complexity
- Understand target, technology, & SIGINT system
. (S//SI//REL) Getting data is only step 1
- Getting a data set is like to getting a new bearer to analyze
. (U) Visualization
- Excel tops out at a million rows...
. (TS//SI//REL) Clean data
- Targets vs. Selectors
TOP SECRET//COMINT//REL FVEY//20340601
TOP SECRET//COMINT//REL FVEY//20340601
Overcoming Complexity (U)
"siGiNT^
Fingerspitzengefühl
• Literally "fingertip feeling"
• Empathy, sensitivity, tact
• Ability of military commanders
to react rapidly
CT Trends TeainChnolog^chn°|ogy
SIGDEV analysts
Partner/Enablers
Must understand tech threat
implications, provenance and
structure of data to manipulate,
interpret it
TOP SECRET//COMINT//REL FVEY//20340601
TOP SECRET//COMINT//REL FVEY//20340601
Getting Data is Step 1 (U)
TOP SECRET//COMINT//REL FVEY//20340601
TOP SECRET//COMINT//REL FVEY//20340601
Getting Data is Step 1 (U)
TOP SECRET//COMINT//REL FVEY//20340601
TOP SECRET//COMINT//REL FVEY//20340601
Getting Data is Step 1 (U)
TOP SECRET//COMINT//REL FVEY//20340601
TOP SECRET//COMINT//REL FVEY//20340601
Visualization (U)
(TS//SI//REL) Excel tops out at a million rows...
- 19 branches, 30+ target sets, ~200 realms, -800 domains, -45000
selectors = 1 million rows/~2.5 weeks for summarized active user events
from E012333 alone
- Spreadsheets are good, but not everyone knows how to use a
pivot table
- Each dataset can easily provide 4-5 or more pivoted looks for
each branch/target set = minimum 100-150 slides
((S//REL) Intent is to routinely produce\
multiple large datasets on a monthly
basis for collection management,
research purposes
TOP SECRET//COMINT//REL FVEY//20340601
TOP SECRET//COMINT//REL FVEY//20340601
Visualization (U)
(S//REL) Analysts work at the selector level
• Leadership wants data presented at the target level
(S//REL) Automated population of technology, behavior
information in analyst workflow tools, databases
(S//REL) Each separate visualization task takes
manpower, time away from operational analysis
TOP SECRET//COMINT//REL FVEY//20340601
TOP SECRET//COMINT//REL FVEY//20340601
Clean Data (U)
(S//SI//REL) Metrics will only provide a near-accurate picture: ground truth will always be the domain of
the TOPI and based on content
(S//SI//REL) Some selectors (accurately) map to multiple targets, multiple teams, multiple organizations
(S//SI//REL) Some selectors simply don't have a known target, only a target set
(S//REL) Need to correlate across widely different datasets requires creation of normalized bridge
datasets (e.g. comparing executables to domains)
(S//SI//REL) TKB/UTT are victims of years of "fill in the blank" freeform data entry; very slowly being
addressed (~2015?)
TOP SECRET//COMINT//REL FVEY//20340601
TOP SECRET//COMINT//REL FVEY//20340601
Rising Strategic Issues (U)
• (TS//SI//REL) Encrypted Webmail Services
Atabmail, Zoho, Safe-mail, Fastmail, HMA Mail
• (TS//SI//REL) Remote Desktop Viewers/Remote Access Tools
- TeamViewer, Join.me, Cybergate
• (TS//SI//REL) Aggregators/Over-the-Top Messaging Services
- WhatsApp, Nimbuzz, eBuddy
TOP SECRET//COMINT//REL FVEY//20340601
TOP SECRET//COMINT//REL FVEY//20340601
What Next? (U)
. (S//SI//REL) Continue to build, strengthen, expand:
- internal workflows, research and discovery capabilities
- collaboration with production elements
- Operational support via embedded analysts at NSAW
- Tradecraft, technical support virtually with extended enterprise
- partnerships with FVEY SIGDEV community
- Establish and expand dialogue opportunities
- "Failure Sharing" - tradecraft sharing and operational deconfliction
. (S//REL) Technology Trends MyNoc
TOP SECRET//COMINT//REL FVEY//20340601
TOP SECRET//COMINT//REL FVEY//20340601
Question
s?
TOP SECRET//COMINT//REL FVEY//20340601
Comment
s?