Title: Black Hole Analytics
Release Date: 2015-09-25
Document Date: 2009-09-01
Description: This GCHQ briefing from September 2009 describes the tools then available to analyse metadata collected on a massive scale: see the Intercept article Profiled: From Radio to Porn, British Spies Track Web Users’ Online Identities, 25 September 2015.
Document: add-sd-blackhole-p1-normal.gif:
BLACK HOLE ANALYTICS
ADD/SD briefing, September 2009
TOP SECRET STRAP 1add-sd-blackhole-p2-normal.gif:
TOP SECRET STRAP1
Contents
■ Describe the new breed of C2C tools
developed in Applied Research (TR)
■ Scaling them up through the Next
Generation Events (NGE) projectadd-sd-blackhole-p3-normal.gif:
TOP SECRET STRAP1
Next Generation Events Roadmap
GTAC f
Convergence 1
Cloud upscale
I ntcmcl Profiling 2
I ntcmcl
Profiling 1
-*0 * lOg
-2G0 experts
Capability development
(including cA)
Large-scale contact
chaining
Data mining and
profilingadd-sd-blackhole-p4-normal.gif:
TOP SECRET STRAP1
NGE Analytical Capabilitiesadd-sd-blackhole-p5-normal.gif:
TOP SECRET STRAP1
**0
Some useful definitions
• Presence Event
• Describes that an identifier was online, on an IP address, at a point in
time.
• Simple, atomic element.
• Can be very easily developed so are very diverse
• Very useful!
• TDI = Target Description Identifier
• Has a type, eg Yahoo-Y-Cookie
• And a value, eg tom123@yahoo
• GTDI = Generic Target Detection Identifier
• Expands the TDI concept to telephony e.g. A GSM location update
messageadd-sd-blackhole-p6-normal.gif:
TOP SECRET STRAP1
Some more definitions
• BLACK HOLE: The large flat file storage where all the data sits
• After initial processing, and before being manipulated and correlated
and loaded into the QFD database tables
• QFD: Question Focused Dataset (Database)
- One for each tool: MUTANT BROTH, AUTO ASSOC etcadd-sd-blackhole-p7-normal.gif:
TOP SECRET STRAP1
GOICjadd-sd-blackhole-p8-normal.gif:
TOP SECRET STRAP1
^0
QFD desktop...so far...
Enables analysts to:
Create a profile of a target's online activities (Mutant Broth)
Find other identifiers for a target (Auto Assoc)
Create a social network (Social Animal)
Investigate websites or web forums of interest (Karma Police,
Infinite Monkeys, HRMAP)
Find out who has been searching the web and for what
(Memory Hole)
Find out who has been looking at what on Google Earth
(Marbled Gecko)add-sd-blackhole-p9-normal.gif:
Converged QFD desktop...by Dec...
■ Enables analysts to:
■ Create a profile of a target's online activities alongside
telephony ( Evolved Mutant Broth)
■ Find alternative identifiers across telephony and the internet
(Hard Assoc)
■ Create a social network including telephony (Evolved Social
Animal)
■ Find out what has been happening in real time (Samuel Pepys)
All available on the desktop via Looking Glass plug-ins and Web
GUIsadd-sd-blackhole-p10-normal.gif:
TOP SECRET STRAP1
«f
Success story -TSS2 Nocon
■ Slides removed - contact for full
versionadd-sd-blackhole-p11-normal.gif:
TOP SECRET STRAP1add-sd-blackhole-p12-normal.gif:
TOP SECRET STRAP1
Capability Dev Workspace
Oh. this looks interesting -
what does an analyst think?
Easy access to data,
however it is collected
Use live data to
w-ork on real problems
Put the latest results
Processing power
ready to use
improvement
Hmm. new data... I wonder what
happens if I mine it like this...
I ve just developed collection of
Will this NSA algorithm help?
Is that better?
I'll put that
processing on a
few more
bearers...
Collection
engineer/
research
team
Analytics'^
developer
Or this?
OK. but could you just...
Good, but how about...
Could you tweak it to...
Analyst with an
operational problem
Oh heck. I've caught
Bin Laden
This data might be best viewed like this: take a look
Is that better? L_
m just add in some of those results...
User-interface
specialistadd-sd-blackhole-p13-normal.gif:
TOP SECRET STRAP1
Data Mining at scale
Experiments with Cloud ongoing to enable
analysts to request and run MOAGs (large TNN
graphs) from the desktop
Distillery will enable analysts to spot real time
changes to the data at scale (e.g. detection of
impossible travel)
Joint Collaboration Environment (lnnov8) -
trialling running of large scale analytics using
both GCHQ and NSA data
add-sd-blackhole-p14-normal.gif:
TOP SECRET STRAP1
Future Implications
■ We shall be able to:
• easily monitor changes in our targets' profiles and networks
■ develop and trial new capabilities using real life analytical
experiments
■ respond quickly in a crisisadd-sd-blackhole-p15-normal.gif:
Questions?
TOP SECRET STRAP1
|/BLAZING_SADDLES
/NGE BLACK HOLE

Download Document
Black Hole Analytics (add-sd-blackhole-p1-normal.gif)
Download Document
Black Hole Analytics (add-sd-blackhole-p2-normal.gif)
Download Document
Black Hole Analytics (add-sd-blackhole-p3-normal.gif)
Download Document
Black Hole Analytics (add-sd-blackhole-p4-normal.gif)
Download Document
Black Hole Analytics (add-sd-blackhole-p5-normal.gif)
Download Document
Black Hole Analytics (add-sd-blackhole-p6-normal.gif)
Download Document
Black Hole Analytics (add-sd-blackhole-p7-normal.gif)
Download Document
Black Hole Analytics (add-sd-blackhole-p8-normal.gif)
Download Document
Black Hole Analytics (add-sd-blackhole-p9-normal.gif)
Download Document
Black Hole Analytics (add-sd-blackhole-p10-normal.gif)
Download Document
Black Hole Analytics (add-sd-blackhole-p11-normal.gif)
Download Document
Black Hole Analytics (add-sd-blackhole-p12-normal.gif)
Download Document
Black Hole Analytics (add-sd-blackhole-p13-normal.gif)
Download Document
Black Hole Analytics (add-sd-blackhole-p14-normal.gif)