Title: QFDs and BLACKHOLE Technology behind GCHQ/INOC

Release Date: 2015-09-25

Document Date: 2009-03-01

Description: This GCHQ presentation from March 2009 describes the BLACKHOLE database used to store raw metadata records from TEMPORA, with about 10 billion records being added each day: see the Intercept article Profiled: From Radio to Porn, British Spies Track Web Users’ Online Identities, 25 September 2015.

Document: qfd-blackhole-technology-behind-inoc-p1-normal.gif:
QFDsand BLACHOLE
Technology behind GCHQ/INOCqfd-blackhole-technology-behind-inoc-p2-normal.gif:
TRAP2 COMINT

Probes Flat Store Analytic engines

Question specific
DBs (INOC today)

Data-mart extraction

MESME/Netezza etc,

Clouds

<10G

© Crown Copyright. All rights reserved. This ¡nformaion is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under
other UK nformaton legislation. Refer disclosure requests to GCHQ on

Contains Intellectual Property owned andfor managed by GCHQ. The material may be disseminated throughout toe recipient organisation, tut GCHQ permission must

fce obtained for dissemination outside the organisation.

Slide2 UK SECRET STRAP2 COMINT ORCONqfd-blackhole-technology-behind-inoc-p3-normal.gif:
ECRET STRAP2 COMINT ORCON

Flat data store - not a database

~ 1,100,000,000,000 events since 6th August 2007

- 53.3TB compressed

- 217TB uncompressed

- 47% of data is from last 3 months

19%

~10 Billion events per day

Cheap system:

- £1 k per TB of storage

- £20k per 10Gbps probe

- ~£400k for 10x10Gbps prototype

■ HTTP

Websearch
SMTP
Webmail
CNE Server Log
Social networking
Instant Messenger
Routing Protocols

bject to exemption uniter

© Crown Copyright. All rights reserved. This infcrmaion * exempt from disclosure under the Freedom of hfl
other UK nformaton legislation. Refer disclosure requests to GCHQ on

Contains Intellectual Property owned andtor managed by GCHQ. The material may be disseminated throughout the recipient organisation, tut GCHQ penrisson must

be obtained for dissemination outside the organisation.

Slide 3 UK SECRET STRAP2 COMINT ORCONqfd-blackhole-technology-behind-inoc-p4-normal.gif:
__________________________________________

Questions - INOC Today

Who, Where, When, What? - Online presence (TDI)
MUTANT BROTH
Who, which website(s)? - Online browsing activity
| KARMA POLICE

Who, which fora?- Bulletin board usage
INFINITE MONKEYS
Where? - Google maps/earth usage

© Crown Copyrigt] [cserved.Thrs inrcrmSTcnes exen?nw\ osaosure under the Freedom o< Information Act 2000 and may be subject to exemption under

formation legislation. Refer disclosure requests to GCHQ on
Contains Intellectual Property owned andfor managed by GCHQ. The material may be disseminated throughout the recipient ergarisation, tut GCHQ penrissen must
te obtained for dissemination outside the organisation.

Sl.de 4 UK SECRET STRAP2 COMINT ORCONqfd-blackhole-technology-behind-inoc-p5-normal.gif:
ECRET STRAP2 COMINT ORCON

Questions - INOC Today

Who, Where, When, What? - Online presence (TDI)

MUTANT BROTH

Who, which website(s)? - Online browsing activity
| KARMA POLICE

Who, which fora?- Bulletin board usage

INFINITE MONKEYS

Where? - Google maps/earth usage

... And many more

O Crewo Ccpyrigtl Jcservcd. his rWSTcn s cxcirpTtrcm oisciusurc under the Freedom cit Information Act 2COO and may be subject to exemption under

Stjpji u .reformation legislation. Refer disclosure requests to GCHQ on

Contains Intellectual Property owned andtor managed by GCHQ. The material may be disseminated throughout the recipient organisation, tut GCHQ penrisstn must mbb
be obtained for dissemination outside the organisation.

deS UK SECRET STRAP2 COMINT ORCONqfd-blackhole-technology-behind-inoc-p6-normal.gif:
__________________________________________

Questions - INOC Today

Too many specific questions... Too much for analysts to learn.
Need analyst systems to be simple and intuitive.

Goc

Let analysts find & simply understand their targets easily...
drill in to systems only when they need to

© Crown Copyright. All rights reserved. This informaien is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under
orher UK nformaton legislation. Refer disclosure requests to GCHQ on

Contains Intellectual Property owned andtor managed by GCHQ. The material may be disseminated throughout the recipient organisation, tut GCHQ penrisson must

te obtained for dissemnation outside the organisation.

Slide6 UK SECRET STRAP2 COMINT ORCONqfd-blackhole-technology-behind-inoc-p7-normal.gif:
End Product Report» contributed by:

□ Grade C
■ Grade B

□ Grade A

TRAP2 COMINT OR

AR Prototypes -EPR Stats - Feb 2008 - Jan 2009

>3 Crown Ccpjtrpl. All qftiis reserved. This infbrmaSon s exempt from disclosure under the Freedom of Information Act 2000 and may bo subject to exemption under

orher UK ntorroalon Relation. Refer disclosure requests to GCHQ on

Contains Intellectual Property owned and'or managed by GCHQ. The material may be disseminated throughout the recipient organisation, tut GCHQ permisstxi must

te obtained for dissemination outside the organisation.

■ UK SECRET STRAP2 COMINT ORCON

Slide 8

qfd-blackhole-technology-behind-inoc-p8-normal.gif:
Nuntxr Ol.»'»

ET STRAP2 COMINT ORCO

Nuntai ol uMrtohotuoAd > 10 quarts*, try«««

«her UK ntermaton legislation. Refer disclosure requests to GCHQ
Contains Intellectual Prcpetty wned andOr managed by GCHQ. The material may fcc dissemiryred throoghoct tie recipient

to obtained for dissemination outside tte organisation.

Slide9 UK SECRET STRAP2 COMINT ORCON

'Q pemssen must









e-Highlighter

Click to send permalink to address bar, or right-click to copy permalink.

Un-highlight all Un-highlight selectionu Highlight selectionh