Title: Data Stored in BLACK HOLE

Release Date: 2015-09-25

Description: This undated page from GCHQ’s internal GCWiki provides technical information about the agency’s BLACK HOLE repository: see the Intercept article Profiled: From Radio to Porn, British Spies Track Web Users’ Online Identities, 25 September 2015.

Document: data-stored-in-black-hole-p1-normal.gif:

The maximum classification allowed on GCWiki is TOP SECRET STRAP1 COMINT. Click to report
inappropriate content.

For GCWiki help contact: webteam

Support page

Data stored in BLACK HOLE

From GCWiki

Jump to: navigation, search

This page describes the data which is stored in
BLACK HOLE, where it comes from and how-
much of it there is. Some of the information is
too highly classified to be contained here, or
falls under the Applied Research intellectual
property restrictions. Nonetheless, as much as
possible is published here.


• 1 What data BLACK HOLE contains

o 1.1 Types of data
o 1.2 Classification of the data

• 2 Where the data in BLACK HOLE
comes from

• 3 How much data there is

o 3.1 Volumes in BLACK HOLE
o 3.2 Volumes in the datamarts

What data BLACK
HOLE contains

feditl Types of data

The events created cover wcbmail. email
transfers, ftp, chat, internet browsing, website
logins, vbulletin web fora, web cams, gaming,
social netw'orking -- and the list is growing.

The data is primarily generated by B13B high-rate bearer processing modules, and so the exact types
depend on which modules arc running at the time.

[edit] Classification of the data


BLACK HOLE: The Knowledge

• WhaLitJs

• Where it came from

• What is in it

o What data it contains
o How much data there is

• How to use it

o How to connect to it
o How to find the file you want

• Who uses it. and whv

o Applied Reseach uses
o SIGINT Development uses
o IPT uses

• How it is built

o Logical connectivity
o Physical system
o Security issues

• What_could_happenjo_it

o Applied Research enhancements
o SIGINT Development intentions
o Engineering interest

• What we like about it

o Good things
o Things that could be better
o Exploding myths
o Wish list

v- d- £

The classification of the data in BLACK HOLE ranges from SECRET STRAP 1 to TOP SECRET
STRAP2 UK EYES ONLY. Most data should be covered by TOP SECRET STRAP2 and there may.data-stored-in-black-hole-p2-normal.gif:
inadvertently, be ECI data (but this is not expected, as we are not actively doing things covered by ECI).
The data includes fields which have been produced by enrichment (such as, for example, target details)
and this can increase the classification.

feditl Where the data in BLACK HOLE comes from

Take a look under the covers of BLACK HOLE to find out where the data comes from.

feditl How much data there is

feditl Volumes in BLACK HOLE

The total amount of (compressed) data entering BLACK HOLE is approximately 200GB/day.

feditl Volumes in the datamarts

Prototype Current retention time / months Current database size / TB Estimated size for six months retention / TB Current database size / billion rows Estimated database size for 6 months retention / billion rows
MUTANT BROTH 4 7.7 11.55 21.7 32.55
KARMA POLICE 3 6.8 13.6 17.8 35.6
MEMORY HOLE 0.5 0.6 7.2 2.2 26.4
AUTOASSOC 6 0.1 0.1 11 11
SOCIAL ANIMAL 1 1.6 9.6 1 6
HRMAP 3 7 14 16 32
INFINITE MONKEYS 6 0.02 0.02 0.2 0.2
Total 23.82 56.07 69.9 143.75

Please note - all of these values are approximate.

Retrieved from |



)ata stored in BLACK HOLE”

I ( nuil rhonchooh Find)


Click to send permalink to address bar, or right-click to copy permalink.

Un-highlight all Un-highlight selectionu Highlight selectionh