Title: Black Hole Analytics

Release Date: 2015-09-25

Document Date: 2009-09-01

Description: This GCHQ briefing from September 2009 describes the tools then available to analyse metadata collected on a massive scale: see the Intercept article Profiled: From Radio to Porn, British Spies Track Web Users’ Online Identities, 25 September 2015.

Document: add-sd-blackhole-p1-normal.gif:
BLACK HOLE ANALYTICS

ADD/SD briefing, September 2009

TOP SECRET STRAP 1add-sd-blackhole-p2-normal.gif:
TOP SECRET STRAP1

Contents

■ Describe the new breed of C2C tools
developed in Applied Research (TR)

■ Scaling them up through the Next
Generation Events (NGE) projectadd-sd-blackhole-p3-normal.gif:
TOP SECRET STRAP1

Next Generation Events Roadmap

GTAC f

Convergence 1

Cloud upscale

I ntcmcl Profiling 2

I ntcmcl
Profiling 1

-*0 * lOg

-2G0 experts

Capability development
(including cA)

Large-scale contact
chaining

Data mining and
profilingadd-sd-blackhole-p4-normal.gif:
TOP SECRET STRAP1

NGE Analytical Capabilitiesadd-sd-blackhole-p5-normal.gif:
TOP SECRET STRAP1

**0

Some useful definitions

• Presence Event

• Describes that an identifier was online, on an IP address, at a point in
time.

• Simple, atomic element.

• Can be very easily developed so are very diverse

• Very useful!

• TDI = Target Description Identifier

• Has a type, eg Yahoo-Y-Cookie

• And a value, eg tom123@yahoo

• GTDI = Generic Target Detection Identifier

• Expands the TDI concept to telephony e.g. A GSM location update
messageadd-sd-blackhole-p6-normal.gif:
TOP SECRET STRAP1

Some more definitions

• BLACK HOLE: The large flat file storage where all the data sits

• After initial processing, and before being manipulated and correlated
and loaded into the QFD database tables

• QFD: Question Focused Dataset (Database)

- One for each tool: MUTANT BROTH, AUTO ASSOC etcadd-sd-blackhole-p7-normal.gif:
TOP SECRET STRAP1

GOICjadd-sd-blackhole-p8-normal.gif:
TOP SECRET STRAP1

^0

QFD desktop...so far...

Enables analysts to:

Create a profile of a target's online activities (Mutant Broth)
Find other identifiers for a target (Auto Assoc)

Create a social network (Social Animal)

Investigate websites or web forums of interest (Karma Police,
Infinite Monkeys, HRMAP)

Find out who has been searching the web and for what
(Memory Hole)

Find out who has been looking at what on Google Earth
(Marbled Gecko)add-sd-blackhole-p9-normal.gif:
Converged QFD desktop...by Dec...

■ Enables analysts to:

■ Create a profile of a target's online activities alongside
telephony ( Evolved Mutant Broth)

■ Find alternative identifiers across telephony and the internet
(Hard Assoc)

■ Create a social network including telephony (Evolved Social
Animal)

■ Find out what has been happening in real time (Samuel Pepys)

All available on the desktop via Looking Glass plug-ins and Web
GUIsadd-sd-blackhole-p10-normal.gif:
TOP SECRET STRAP1

«f

Success story -TSS2 Nocon

■ Slides removed - contact for full

versionadd-sd-blackhole-p11-normal.gif:
TOP SECRET STRAP1add-sd-blackhole-p12-normal.gif:
TOP SECRET STRAP1

Capability Dev Workspace

Oh. this looks interesting -
what does an analyst think?

Easy access to data,
however it is collected

Use live data to
w-ork on real problems

Put the latest results

Processing power
ready to use

improvement

Hmm. new data... I wonder what
happens if I mine it like this...

I ve just developed collection of

Will this NSA algorithm help?

Is that better?

I'll put that
processing on a
few more
bearers...

Collection

engineer/

research

team

Analytics'^

developer

Or this?

OK. but could you just...

Good, but how about...

Could you tweak it to...

Analyst with an
operational problem

Oh heck. I've caught
Bin Laden

This data might be best viewed like this: take a look
Is that better? L_
m just add in some of those results...

User-interface

specialistadd-sd-blackhole-p13-normal.gif:
TOP SECRET STRAP1

Data Mining at scale

Experiments with Cloud ongoing to enable
analysts to request and run MOAGs (large TNN
graphs) from the desktop

Distillery will enable analysts to spot real time
changes to the data at scale (e.g. detection of
impossible travel)

Joint Collaboration Environment (lnnov8) -
trialling running of large scale analytics using
both GCHQ and NSA data

add-sd-blackhole-p14-normal.gif:
TOP SECRET STRAP1

Future Implications

■ We shall be able to:

• easily monitor changes in our targets' profiles and networks

■ develop and trial new capabilities using real life analytical
experiments

■ respond quickly in a crisisadd-sd-blackhole-p15-normal.gif:
Questions?

TOP SECRET STRAP1

|/BLAZING_SADDLES
/NGE BLACK HOLE