Title: GCHQ Analytic Cloud Challenges
Release Date: 2015-09-25
Document Date: 2012-05-14
Description: This GCHQ presentation dated 14 May 2012 describes the agency receiving “more than 50 billion events [metadata records] per day” and some of the tools available for analysing that mass of data: see the Intercept article Profiled: From Radio to Porn, British Spies Track Web Users’ Online Identities, 25 September 2015.
Document: gchq-analytic-cloud-challenges-p1-normal.gif:
GCHQ Analytic Cloud Challenges
Innovation Lead for Data, Analytics &
Visualisation Engineering
This information is exempt from Disclosure under the Freedom of Information Act 2000 and may be subject
to exemption under other UK information legislation. Refer disclosure requests to GCHQ on I
14/5/2012
UK TOP SECRET STRAP 1gchq-analytic-cloud-challenges-p2-normal.gif:
Approximately 50 Billion Events Per
Day from 250+ xlOG bearers
• Scale Target for May 2012 is
approximately 320 x 10G with
a capacity for 100 BEPD
Volume by site
Typical retention is six
months, sometimes reduced
to cope with volume increase
26/3/2012 Ref: 18171507
UK TOP SECRET STRAP 1
2gchq-analytic-cloud-challenges-p3-normal.gif:
U« KOHI Itiw-
TDB Events PC OV-1 • Events High Level Concept Graphic
Purpose
A high level view on
GCHQ event (unselected
meta-data) capabilities
and uses
OV-1« Hgt» L«««l Oj-»*•«»:««•! CofK«|rt ******** VJ 14*1 o«m«f tewagai
IM*C**» »IHM« gchq-analytic-cloud-challenges-p4-normal.gif:
Query Focused Dataset
What are QFDs
• A database designed for answering a
single (or few) analytic question
• Initially adopted from
research, engineering tasked to
mature and scale ASAP
• The engineering has been
incremental based on priority
• Library of components and design
patterns developed
• Scale using appropriately sized
database instances, federation
using service tiers
• Deployment of new instances and
updates streamlined
• Corporate support
• New QFDs still being produced by
engineering and others using the
standard components
• Some now provided interactive
access to datasets generated by
HADOOPS batch analytics
QFD Facts and Figures
• Approximately 100 QFD instances
deployed for 16 different QFDs ♦
more for Ul etc
• Each with events (or results)
appropriate for the questions they
answer
• Hardware shared depending on
QFD, all storage is shared
• Driven by the need for a flexible
platform at large scale with low
overall cost
• Redhat Enterprise Linux, Oracle DB
(DB needs are simple)
• Latest generation HP BL456c 67
blades, 24 core, 64GB RAM, EVA
storage "200TB usable
• Most QFD instances have 70TB
usable capacity
• Total storage 15PB raw, 11PB usable
26/3/2012 Ref: 18171507
UK TOP SECRET STRAP 1
4gchq-analytic-cloud-challenges-p5-normal.gif:
A selection of the major QFDs
Name Description Questions Answered Physical &ude*Cheh
AUTOASSOC BuU unsdccted TDI-TDI correlations with confidence scores. What other TC*s belong to your target ? What technologies your target is using ? 2»1 instances, each SO70TB storage
Evolved Mutant Broth Identify »hen certain TOIs appear in traffic which indicate target usage and their location. Telephony and C2C data provide a converged view. Where has my target been? What kind of communications devices has my target been using? 1D»S instances, each 70TB storage
Hard Assoc Provide strongly correlated selectors for both C2C and Telephony traffic taken from TDIs appearing n the same packet Are there any alternative C2C or Telephony selectors for my target? 3»2 instances, each 70TB storage
HRMap Host-rcferrer relationships • information about how people get to websites, including Inks folowed and direct accesses. How do people get to my website of interest and where do they go to next ? What wcbsrtcs have been visaed from a gwen IP? S*3 instances, each 70TB storage
KARMA POUCE Whtch TDIs ha»« been seen at approximately the same time, and from the same computer, as voits to websites. Which websites your target visits, and »hen/where those visits occurred. Who visas suspicious websites, and when/wticre those visits occurred. Which other websites are vented by people who visit a suspicious website. Which IP address and web browser were being used by your target when they vented a website. 11»7 instances, each 70TB storage, 3»1 correlator instances
SOCIAL ANTHROPOID Converged comms events allowing you to see who your targets have communicated with via phone, over the nternet, or usng converged channels (e.g. sending emails from a phone or makng voice cals over the internet). What commiaiications your target is engaged in. Who has your target been communicating with. What commimications have occurred using a particular locator (IP address, cell tower, etc). &»3 instances, each 70TB storage
26/3/2012 Ref: 18171507
UK TOP SECRET STRAP 1
5gchq-analytic-cloud-challenges-p6-normal.gif:
HADOOP Clusters
Facts & Figures
• Three batch analytic clusters
(HADOOP clouds)
• Each contains unique, unselected
events, reference and results data
• Redhat Enterprise Linux, Apache
HADOOP (Map Reduce)
• Each with ^900 data nodes, plus
Ul/ingest/egress servers
• Each node 8 core, 64GB
RAM, 6x1TB
• Each 6PB raw storage, 1.5PB for
events (3x replication and results)
• The three clouds could, in
theory, store 24 Trillion events @140
BEPD and 6 months retention period
Usage
• 200 capability developer accounts
• Many more users able to access
applications
• Guiding light > 150 users
• First Contact, Cloudy Cobra > 50
• Jobs during working hours
• Operationally ad-hoc
(experimental, search, one time
use)
• Development of new capability
• Schedules Jobs run overnight
• Sustained, operational applications
• Increasing expectation for fully
supported, automated jobs such as
Rumour Mill
26/3/2012 Ref: 18171S07
UK TOP SECRET STRAP 1
6gchq-analytic-cloud-challenges-p7-normal.gif:
Physical Overview
Private Cloud
• Data Nodes
• HADOOP Master Nodes (Job Track & Name Node)
Edge Nodes connect to Computer Hall Network
• Ingest Nodes
• Application Nodes (provide user interface - web)
• Innovation Nodes (cap dev login to servers)
26/3/2012 Ref: 18171S07
UK TOP SECRET STRAP 1
7
ID^o j>4>gchq-analytic-cloud-challenges-p8-normal.gif:
Data Ingest
Ingest Design Goals
• Store all data received, i.e.
don't risk losing important
fields by normalising to single
file format.
• Partition by directory structure
• Partition by data type to
simplify common queries.
• Partition by ingest date to
allow incremental analytics.
• Partition by security
compartment.
• Horizontal scaling by adding
new hardware.
Data Formats
• TLV- Tag Length Value
• Comma Delimited
• Fix Position
• Single Line & Multi-Line
• Multiple Variations
• Actor Action
• Many sources and many types
26/3/2012 Ref: 18171507
UK TOP SECRET STRAP 1gchq-analytic-cloud-challenges-p9-normal.gif:
Analytic use of data
Security
• Data access controlled across 2
dimensions:
• Usability: Operational (standard
events) or Controlled (special
purpose e.g. test)
• Classification: Multiple buckets -
Open (TSS2), a few compartments,
general CIOs Sensitive
• Capability developer access
restricted by file/directory
permissions
• User facing applications apply fine
grained security on per record basis
Silver Library
• Abstraction layer to separate analytic
development from storage
dependencies
• Pluggable parsers for new formats
• Record parsers for event data held in
HDFS
• Applications still need to know field
labels (function calls)
• Applications often add additional
abstractions
• Input and Output handlers to read
and write data based on type
• Utility functions to aid development
• Simple search and filter routines
• Events Product Centre maintained
26/3/2012 Ref: 18171507
UK TOP SECRET STRAP 1
9gchq-analytic-cloud-challenges-p10-normal.gif:
Example Analytics
•Glorified grep driven by GUI - find eventi that contain
uler search term
Guiding Light
• Ml information types/vdumesof traffic on bearer
Golden Axe
•Generated list of suspected clone mobSe phones (IMEI
grey list)
Tribal Carnem
•Uses Radius logs to identify & collect activity for IP session
Public Anemone
•Geolocation based on web-based map searches
Epic Fail
•Identifies careless use of TOR networks
Sterling Moth
• IP summarisation tool using c2C presence events
Foghorn
• find non-targets using targets machines
And many more....
26/3/2012 Ref: 18171S07
UK TOP SECRET STRAP 1gchq-analytic-cloud-challenges-p11-normal.gif:
Rich Visualisation
Desktop applications for rich
visualisation and analysis - Eclipse RCP
LOOKING GLASS client platform, FIRE
ENGINE question-based federated
access to events and reference data
sources
Pilots
- NGCC Contact chaining with
C2C/converged event data
- QFD Federator
- NG Geo: Converged geo events
analysis
Next stage to learn from pilots and
build the tool for everyone - ALPHA
CENTAURI
26/3/2012 Ref: 18171507
UK TOP SECRET STRAP 1
11gchq-analytic-cloud-challenges-p12-normal.gif:
Rumour Mill - Push Analytics
Rumour Mill is a dashboard that
will:
- Enable analysts to prioritise new
work as it arrives from customers by
easily finding out "what does GCHQ
already know"
- Enable analysts to monitor existing
work to spot when something
happens that would change their
priorities
First level results, list of questions
against identifiers. Simple yes/no
answers
Click on a yes to the second level -
drills into the detail
Many questions are derived from
cloud based analytics run each day
against the current identifier list
26/3/2012 Ref: 18171507
UK TOP SECRET STRAP 1
12gchq-analytic-cloud-challenges-p13-normal.gif:
Sharing & Collaboration
Other SIA and foreign partners
Data (bulk & query) and technology exchange
Two major components:
• Web user interfaces (VAIL) on GCHQ servers but
accessible from the partner site. Interactive query of
QFDs. Allow exposure of GCHQtradecraft.
• Brokering services. Sustained access for interactive query
of GCHQ data integrated into partner tools.
26/3/2012 Ref: 18171507
UK TOP SECRET STRAP 1
13gchq-analytic-cloud-challenges-p14-normal.gif:
Future Options for Event
Processing
26/3/2012 Ref: 18156927
UK TOP SECRET STRAP 1
14gchq-analytic-cloud-challenges-p15-normal.gif:
Key Challenges
Affordable, continuing
scale of our capabilities
• All dimensions, cost, power, cooling, space, storage, bandwidth, processing etc
• As we share with more partners, the access demands increase
• Enable more delivery by collaborating with others
Enabling our analysts to
cope with complexity
and volume of data
• Need to streamline understood workflows to save analysts time
• Need to push data to analysts, sometimes with low latency
• Enable next level analysis, behavioural, pattern of life, noticing
change, predictions
Complexity and pace of
analytic development
• Suitably skilled people are hard to find m-house and externally
• Must be able to cope with mixed maturity capabilities
• Workload separation, processing contention, cross site analytics
Agility, resilience and
maturity of platforms
• Keep pace with development, respond to community demands
• Understand and match the evolving business expectations of maturity
• Support appropriately, agreed strategy for resilience
26/3/2012 Ref: 181S6927
UK TOP SECRET STRAP 1
ISgchq-analytic-cloud-challenges-p16-normal.gif:
Approach to challenges
Deploy more of the
same while maturing
Explore new
technology
Know the value of our
data
Understand usage of
capabilities
Stay flexible and
increase collaboration
• Suitably refreshed, will answer Immediate scaling challenge
• Reduce support burden and streamline new capability development
• Implement Improvements to existing architecture
• Non-relational, distributed databases; QFD consolidation & convergence
• Research initiatives (A)SEM/HAKIM/STREAMING, and commercial offerings
• Collaboration opportunities CLOUDBASE/ACCUMULO
• Don't generate/ingest data we don't use, filter or de-dupe upstream
• Be smart about retention, be smart about need for Interactive availability
• Distil the raw data to generate rich information sets for analysts
• Automate the simple workflows with summaries and push analytics
• Optimise capabilities for their use and access load
• Mature, provide resilience and support as appropriate
• Enable collaboration and lower the entry bar for capability developers
• Stay vendor neutral, continue to use open standards, open software
• Enhance the benefit for both research & engineering of working together
26/3/2012 Ref; 181S6927
UK TOP SECRET STRAP 1
16gchq-analytic-cloud-challenges-p17-normal.gif:
Understand usage of capabilities
Use Case Class Mapping
Large proportion of queries for
most of the analysts
Target, Unknown
Query
Target Based Discovery
TugattaMd
Development
Target Tracking
Small proportion of queries for
some of the analysts
Target, Unknown
Query
Anomaly Detection
Behaviour-based
Discovery
Target. Known
Query
• Developed jointly between GCHQ
and NSA to understand
- the benefits of our current
capabilities
- where respective strengths and
weaknesses exist
• Provides a clear set of drivers for
architectural evolution
- Missing capabilities
- Suboptimal use of capabilities
- Opportunities for collaboration and
reuse
26/3/2012 Ref: 18156927
UK TOP SECRET STRAP 1
17gchq-analytic-cloud-challenges-p18-normal.gif:
v—
Existing coverage of use cases
Known Target, Known
Query
Streaming. Rumour Mill and QFD capabilities (possibly assisted by cloud analytic)
Could select data subset based on target, could pre-calculate results
Large usage and automation suggests need for optimised capability
Known Target, Unknown
Query
Not possible with existing QFD capabilities
Needs new analytic or more Indexes on target selected data
Large usage and needs suggest a new capability
Unknown Target, Known
Query
Core QFD capability, possibly populated by cloud analytic
1 Full unselected data set required but what level of Interactive query?
1 Is It acceptable that older data be made available non-lnteractively?
Unknown
Target, Unknown Query
1 Use batch analytic platform for low level search or new analytic
Could a heavily indexed store provide a responsive capability?
• Possibly only over a subset of data - recent data only?
26/3/2012 Ref: 18156927
UK TOP SECRET STRAP 1
18gchq-analytic-cloud-challenges-p19-normal.gif:
Value assessment
What measures of value?
• Duplicated over time or across bearers
• Value decreases with age
• Value decreases after processing (SPAM, summaries)
• Use by analysts and analytics
What actions to take?
• Do filtering upstream or don't generated
• Investigate use of streaming capabilities for filtering or sampling
• Discard immediately after first stage processing
• Age off and discard more selectively
• Periodically evaluate data types to understand usage
Need to agree possibilities with operations and experiment
26/3/2012 Ref: 181S6927
UK TOP SECRET STRAP 1
19gchq-analytic-cloud-challenges-p20-normal.gif:
Technology candidates
QFD consolidation and HAKIM
Existing QFD drawbacks
• Some duplication between QFDs (~10%)
• Need new QFDs to answer new questions
Need a consolidated database with multiple indexes and flexible additions
• HAKIM is a research prototype to do just that
• Unification of data, associated data kept together
• Quick and flexible addition of new data types and indexes
• Scalable and cost effective
Other candidates exist and some HAKIM components could be replaced
• Oracle DB or distributed, non-relational database?
• Convergence with HADOOP stack HBASE/ACCUMULO
Engineering is working with research to develop to the next stage
26/3/2012 Ref: 18156927
UK TOP SECRET STRAP 1
20gchq-analytic-cloud-challenges-p21-normal.gif:
Technology candidates
The "skinny" cloud (Laurel)
What is it?
• A batch analytic HADOOP cluster
• Contains all the data but for a short retention period
What would it be used for?
• An ideal place to do incremental analytics
• Answers the cross-site analytic problem
• Could be dedicated to sustained usage
• Provides some resilience by duplicating recent data
What are the challenges?
• Can we transfer and ingest all the data
Engineering could potential build this using on-order kit
26/3/2012 Ref: 181S6927
UK TOP SECRET STRAP 1
21gchq-analytic-cloud-challenges-p22-normal.gif:
Technology candidates
The GCHQ "core" (Hardy)
What is it?
• A HADOOP cluster with Map/Reduce and interactive query/analytics capabilities
• Probably using CLOUDBASE/ACCUMULO and reusing NSA knowledge
What would it be used for?
• A place for data and summaries promoted from the bulk stores
• Known Target, Know Query and some Known Target, Unknown Query
• Optimised for major use case and suitable for data sharing
• Provides some resilience by duplicating important data
What are the challenges?
• GCHQ has limited expertise with CLOUDBASE/ACCUMULO this technology
• The promotion analytics and criteria are not developed
Engineering could potential build this using on-order kit
26/3/2012 Ref: 18156927
UK TOP SECRET STRAP 1
22gchq-analytic-cloud-challenges-p23-normal.gif:
SM TO* NCXr
TDB Events PC SV-1 • Skinny cloud, GCHQ Core and QFD consolidation
Clc-KJ
I AH tt* data at
ttW* MWJld
Q -rv »UfOM ,
Bu»W« 1"
IftHAButW. w c*«a H **£
E»tntt
AJ E*««9 k*
Scociv E«#N T,i#a* C«nUd«MM Qlt>*
Cp«r«to'A Trial XARIMi
On* #o«*n
Ot«Y
HiAZU
Palpóse
Sbcwa possitk (KW a'ch tortee V* GCHQ Everra
He«* conpororrs
Sknrry douJ afi ir* dsra *cm Cor üw bu*, beW rv ***
Un* A |im:m tú do dttly. «»uwfmuUl arutplc» en bufe
e*«ni5 Penen so*/. on>y &*>wanpacr/ckefe a u**6t* COpy c/ Un dilA fur a Wo»! 3tym pacied
QMng SOTO remenee
GCHQ Co«* - VJuatfe «ven» ar* seccied |NSA A
GCHQ MMk) tf/ ¿a* )laa ¿nd ocekd «rito ir* co* Tr*
con» e fKt irtnwvy a p9U? *H toveb anar* 1c*. r*rt>ec an
hUracf «* du«> focotfrxy Ir» AAJtkn to ir* f*w w\íftt
wrmíTfi W b?4d and sbp«*d » USA lor1mygtwn
hio tbíar icott Al tbo inpcrjfi: ova«* día rad Vw tr*
nxms rvtjnxr pened pPOttfng ro« lenco.
Censo Malee Qf D • TN> non stojo Qf Os. ow
moro teaiei« aro rrctowtu heneen
8V1 «no*w *r*e A
M(>llWd U4I7*1} 14 4* t*
0*mi
