Title: CSEC ITS/2NE Cyber Threat Discovery

Release Date: 2015-02-25

Document Date: 2010-01-01

Description: This 2010 CSEC document complete with speakers’ notes shows that visits to Canadian government websites were monitored, and hundreds of thousands of emails to government email addresses retained every day, for cybersecurity purposes: see the CBC News article CSE monitors millions of Canadian emails to government, 25 February 2015.

Document: TOP SECRET

1^1 Communications Security Centre de la sécurité

Establishment Canada des télécommunications Canada

CSEC ITS/N2E
Cyber Threat Discovery

I



DISCOCON 2010



m/4

A CND Perspective

# *1
I



Safeguarding Canada’s security through information superiority

Préserver la sécurité du Canada par la supériorité de l’information

Canada

TOP SECRET

■ * ■ Communications Security Centre de la sécurité

Establishment Canada des télécommunications Canada

N2E Cyber Threat Discovery

• CSEC/ITS Discovery: Context & Sitrep

• Current Capabilities

• CND Metadata Analysis at Scale

Safeguarding Canada’s security through information superiority

Préserver la sécurité du Canada par la supériorité de l’information

Canada

TOP SECRET

1^1 Communications Security Centre de la sécurité

Establishment Canada des télécommunications Canada

N2E Context

• Within CSEC/ITS, CND operations concentrated in N2

- Core: incident response team (alerts -> analysis -> mitigation advice)

- Malware/RE & VR teams

• Discovery has always been a required activity within N2

- IR often takes precedence

• N2E established to focus effort on discovery

- Hunting vs. firefighting: new threats, techniques, tradecraft

- Iterations of hypothesis based research/analysis

• What evidence is there in the data of compromised systems?

• What new threats or techniques can we find operating against us?

- Development of effective anomaly detection/heuristic techniques

• How can we better detect these new threats in the future?

• Shape capability development priorities

Safeguarding Canada’s security through information superiority
Préserver la sécurité du Canada par la supériorité de l’information

Canada

TOP SECRET

1^1 Communications Security Centre de la sécurité

I . I Establishment Canada des télécommunications Canada

N2E Sitrep

• Team formally established earlier this year

- Growing to 7; hiring underway, strong candidates in pipeline

• Excellent access to full take data

• Analytical environment improving

• Progress on policy support

- Use of intercepted private communications

- Sharing mechanisms

Safeguarding Canada’s security through information superiority
Préserver la sécurité du Canada par la supériorité de l’information

Canada





TOP SECRET

l+l

Communications Security Centre de la sécurité

Establishment Canada des télécommunications Canada



Our Haystack



Three individual clients

- 10’s - low 1000’s of signature based alerts each day
(majority false positive)

Soon: “official" Internet gc.ca aggregation point

Full pcaps (retention: days to months)

- l’s - 10’s TB of passively tapped network traffic each day

Metadata (retention: months to years)

- 10’s - 100’s GB of non-indexed, textual, network metadata / day (descrip

Safeguarding Canada’s security through information superiority
Préserver la sécurité du Canada par la supériorité de l’information

Canada

TOP SECRET

1^1 Communications Security Centre de la sécurité

Establishment Canada des télécommunications Canada

Toolset

• Homemade wrappers for heuristic detection tools

• Popquiz/Slipstream for metadata

• And...

$ find | xargs | grep | sed |
cut | awk | sort | uniq -c |
perl | python

Safeguarding Canada’s security through information superiority
Préserver la sécurité du Canada par la supériorité de l’information

Canada

TOP SECRET

■ ^ ■ Communications Security Centre de la sécurité

Establishment Canada des télécommunications Canada


^ I

Homemade Wrappers: PonyExpress

SMTP processing

8Ball wrapper to process all email attachments

• Cuts and re-rebuilds SMTP session from full packet captures

• Extracts and logs metadata from SMTP and RFC822

• Extracts attachments and sends it to a cluster of 8Ball services

- Wrapper able to manage other deep scan tools (e.g. AV)

- Aggregates results from scans asynchronously

• Generates alerts for analysts to look at with full SMTP pcap

• Catches new implants or first stage delivered using attachments

Safeguarding Canada’s security through information superiority
Préserver la sécurité du Canada par la supériorité de l’information

Canada

TOP SECRET

1^1 Communications Security Centre de la sécurité

Establishment Canada des télécommunications Canada

_____

Homemade Wrappers: Stripsearch

Automated binary analysis

• Recursively runs binary through the following tools:

• Correlation with file repository and reports

• Future: greater degree of automation

Safeguarding Canada’s security through information superiority
Préserver la sécurité du Canada par la supériorité de l’information

Canada

TOP SECRET

■ ^ ■ Communications Security Centre de la sécurité

Establishment Canada des télécommunications Canada

Metadata

• Standard Flows (with protocol guessing)

• DNS

- Queries / Answers and beaconing

• SMTP

- RFC 788 (SMTP), RFC 822 (2822, 5322) Internet Message
Format

• HTTP

- All server-side headers

- User-Agent summary

- List of POST without a GET, URLs and PE downloads

Safeguarding Canada’s security through information superiority

Préserver la sécurité du Canada par la supériorité de l’information

Canada

TOP SECRET

1^1 Communications Security Centre de la sécurité

Establishment Canada des télécommunications Canada

Metadata: ¡findmask

Safeguarding Canada’s security through information superiority

Préserver la sécurité du Canada par la supériorité de l’information

Canada

TOP SECRET

l+l

Communications Security Centre de la sécurité

Establishment Canada des télécommunications Canada

Noteworthy Catches

WMI-based implant

• Trend Micro has a good report this type of implant

http://us.trendmicro.com/imperial/md/content/us/trendwatch/researchanda

• Uses WMI for persistence instead of Registry or Boot sector

- Reside in ActiveScriptEventConsumer

- First seen in September 2009

See RFI_20091023_280_1 and IXR_20100909_732_1

- Caught with Pony Express

- Stripsearch fuzzy correlation linked it with previous reports

Safeguarding Canada’s security through information superiority
Préserver la sécurité du Canada par la supériorité de l’information

Canada



il



TOP SECRET

■ ^ H Communications Security Centre de la sécurité

Establishment Canada des télécommunications Canada

■i

Noteworthy Catches

Google IP addresses used as a sleep command

• From DNS metadata

- When call back domain was queried, a Google IP was used as a sleep
command. Searching for Google IPs in DNS metadata revealed new
TROPPUSNU domains and infected workstations.

Suspicious RDP and TOR sessions

• From Flow metadata we identified:

RDP

- Incoming RDP sessions from various locations to the RDP service (uses
the same certificate)

- Still under investigation
TOR

- Was picked-up as a suspicious burst of outgoing SSL connections going to
several locations from a single source

Safeguarding Canada’s security through information superiority
Préserver la sécurité du Canada par la supériorité de l’information

Canada







12

TOP SECRET

1^1 Communications Security Centre de ia sécurité

Establishment Canada des télécommunications Canada

Noteworthy Catches: TOR...

• Most likely a breach of Internet usage policy

• All SSL CERTS were:

- Self-signed

- Certificate has a validity window of just 2 hours (e.g. 101017152421Z
to 101017172421Z)

- Issuer name appears to be a randomly generated fully qualified
domain name (FQDN), unique for each destination IP (e.g.
www. b2wwzd u vdc5jyty3s7 .net)

- Common Name (CN) field appears to be a randomly generated FQDN
unique for each session (e.g. www.xehzhl3cip.net)

Safeguarding Canada’s security through information superiority

Préserver la sécurité du Canada par la supériorité de l’information

Canada

TOP SECRET

1^1 Communications Security Centre de la sécurité

Establishment Canada des télécommunications Canada

Noteworthy Catches: TOR





All sessions for October 17 extracted from pcap using the following
ngrep HEX string:

$ ngrep -I pcap -tq -X/^<5A170D313031303i:

T 2010/10/17 15:49:00.699087
(...

vdc5jyty3s7. net0(...x...)

.X.X.43:61469 [AP]

..10101715242ÜZ..10101^172421Z0%1#0!..U....WWW.b2wwzdu

- The string is the end of validity period (Date only)

Snort signature could be generated for looking at CERTS with
validity period starting/ending same day

Safeguarding Canada’s security through information superiority
Préserver la sécurité du Canada par la supériorité de l’information

Canada

TOP SECRET

■ ^ ■ Communications Security Centre de la sécurité

Establishment Canada des télécommunications Canada

^^—'11II' ' ^—EZZ - -.

CND Metadata Analysis at Scale

Selected SAWUNEH Results

www jolyon.co.uk



Safeguarding Canada’s security through information superiority
Préserver la sécurité du Canada par la supériorité de l’information

Canada

15

TOP SECRET

1^1 Communications Security Centre de la sécurité

Establishment Canada des télécommunications Canada

General Problem: Information Overload

• What is ‘at scale’ for CSEC CND?

• Problem: too much data

- Acquired, retained, summarized, analyzed, presented to analysts

• Opportunities at all tiers of system. Selected SAWUNEH topics
help in one or more of these areas

Safeguarding Canada’s security through information superiority

Préserver la sécurité du Canada par la supériorité de l’information

Canada

TOP SECRET

1^1 Communications Security Centre de la sécurité

Establishment Canada des télécommunications Canada

What is SAWUNEH?

• Literally: ‘Summer Analytic Workshop Up North EH’

- Annual data analysis workshop hosted at CSEC by crypt-math shop

- 2010 thrust was CND

- Reps from each of the 5 eyes + cleared researchers

• Helps fill the (sparsely inhabited) plane that often exists between
OpsDev and R&D

• Special Assets: Netezza, Cray XMT

Safeguarding Canada’s security through information superiority

Préserver la sécurité du Canada par la supériorité de l’information

Canada

TOP SECRET

1^1 Communications Security Centre de la sécurité

Establishment Canada des télécommunications Canada

Topic 1: Correlating ‘E-Mail, Web, Flow’ Metadata

• E-mail based content delivery attacks against GC *very* common

• CND capability rapidly developed & deployed; heavily relies on attachment
scanning

• The inevitable evasion

- Attacks now commonly use email as inducement to URL visit, instead of direct
content delivery.

• Too many benign hyperlinks delivered over email

- Need to reduce metadata presented to front line analysts

Safeguarding Canada’s security through information superiority
Préserver la sécurité du Canada par la supériorité de l’information

Canada

TOP SECRET

l+l

Communications Security Centre de la sécurité

Establishment Canada des télécommunications Canada

Topic 1: Correlating ‘E-Mail, Web, Flow’ Metadata

Reducing Email URL metadata volume delivered to front line analysts

• Select highly suspicious subset of Email URL metadata based on
correlation

• Only present ‘Email URL’ metadata if:

- Email was inbound

- Contained one or more hyperlinks

- Hyperlink nominated as suspicious

- Hyperlink was actually visited by a recipient.

- Provide flow and http metadata for resulting HTTP session

Safeguarding Canada’s security through information superiority

Préserver la sécurité du Canada par la supériorité de l’information

Canada

TOP SECRET

■ ^ ■ Communications Security Centre de la sécurité

Establishment Canada des télécommunications Canada

Topic 1: Correlating ‘E-Mail, Web, Flow’ Metadata

SAWUNEH approach

• Import EMAIL, URL, FLOW metadata into separate tables on the
NETEZZA appliance (distributed db)

Safeguarding Canada’s security through information superiority
Préserver la sécurité du Canada par la supériorité de l’information

Canada

TOP SECRET

1^1 Communications Security Centre de la sécurité

Establishment Canada des télécommunications Canada

‘Email + Web’ attack detection
SAWUNEH Style



Findings:

• 50% of the unique results were malicious

• Significantly lower false positive rates compared to URL inspection only

• Result provides anomaly tip but also context at fingertips for analyst

Next Steps:

• Better definition of ‘suspicious’ URL

• Automate extraction of web content from session for analysts

• Automate feeding of web payloads into content scanning system = partial mitigation
of the original evasion. Higher analyst productivity

Safeguarding Canada’s security through information superiority

Préserver la sécurité du Canada par la supériorité de l’information

Canada

TOP SECRET

l+l

Communications Security Centre de la sécurité

Establishment Canada des télécommunications Canada



Topic 2: Malicious Attachment Prediction

Problem:

• Our current email scanning system fully processes every email it sees in a modular
fashion

• Works very well today, but we can’t afford to scale this approach ‘as-is’ to meet
future requirements.

• Large proportion of time spent in *late* stages of processing., especially deep scan

Possible Approach:

• Can we predict which emails will score in the deep scan based only on metadata
extracted during earlier processing phases?

• If so, can we drop those that have low probability of deep scan success?

Safeguarding Canada’s security through information superiority

Préserver la sécurité du Canada par la supériorité de l’information

Canada

TOP SECRET

l+l

Communications Security Centre de la sécurité

Establishment Canada des télécommunications Canada

Malicious Attachment Prediction

• Given metadata from each processing phase, build a predictive model to selectively
promote emails with high probably of non-zero scoring in 8-ball based on features
extracted early in processing

• Choice of Predictive Model

- Random Forests: decision tree based classifier (Brieman 2001)

• Take a feature set (Xr... Xp) and a corpus of training samples with *known*
classification as input. Generate Random Forest of decision trees to be used to
prediction classification

• New samples pass through previously trained forest which yields probability that
deepscan will be nonzero

Safeguarding Canada’s security through information superiority

Préserver la sécurité du Canada par la supériorité de l’information

Canada

TOP SECRET

1^1 Communications Security Centre de la sécurité

Establishment Canada des télécommunications Canada

Malicious Attachment Prediction
Data Reduction vs. ‘Interesting’ Data Loss

Safeguarding Canada’s security through information superiority
Préserver la sécurité du Canada par la supériorité de l’information

Canada

TOP SECRET

l+l

Communications Security Centre de la sécurité

Establishment Canada des télécommunications Canada

Malicious Attachment Prediction

Result:

• 85% data reduction with 1-3% loss of ‘interesting’ emails

- Can discard majority of emails with minimal loss in positive hits

• Prediction with model is relatively cheap (10,000s features per second)

Next steps:

• Extract those features which contribute most to prediction and incorporate
appropriate filters in each stage of email processing system with aim to discard
early and often

• There are likely a few simple features contributing disproportionally to predictive
power: flow size, attachment size, attachment type etc

Safeguarding Canada’s security through information superiority

Préserver la sécurité du Canada par la supériorité de l’information

Canada

TOP SECRET

■ ^ ■ Communications Security Centre de la sécurité

Establishment Canada des télécommunications Canada

(Mini) Topic 3: PE Masquerade Detection

• Multiple threat actors observed masquerading Windows PE downloads using
‘benign’ filename extensions (jpg, gif, htm etc)

• We log metadata on HTTP sessions containing a PE header in content of first
packet (modules available in both SLIPSTREAM and POPQUIZ)

- Exclude entries with suffixes common to PE downloads (exe, bin, php, asp)

- Provide analysts contextual metadata & 1 click access to payloads

• Easy to see entries that jumped out: file extensions such as .jpg, .gif highly
suspicious, often malicious

• Also filter by ‘uniqueness’ over time to eliminate AV / OS updates

• Next Step: automate extraction of pe masquerades and push through content
scanning system

Safeguarding Canada’s security through information superiority
Préserver la sécurité du Canada par la supériorité de l’information

Canada

TOP SECRET

■ ^ ■ Communications Security Centre de la sécurité

Establishment Canada des télécommunications Canada

Safeguarding Canada’s security through information superiority

Préserver la sécurité du Canada par la supériorité de l’information

Canada

TOP SECRET

■ ^ ■ Communications Security Centre de la sécurité

Establishment Canada des télécommunications Canada





Safeguarding Canada’s security through information superiority
Préserver la sécurité du Canada par la supériorité de l’information

Canada

TOP SECRET

1^1 Communications Security Centre de la sécurité

Establishment Canada des télécommunications Canada

Final Notes

SAWUNEH outcomes made a few things clearer to us

• Our days of plain old grep as primary search tool are nearing an end

• Relational DB’s provide us with a few key benefits:

- Simple indexing of existing metadata (sub second searches)
ability to correlate easily across ‘primary’ metadata outputs

- significantly faster hypothesis testing

• Depending on scale, DB over head may be too costly

- In such cases, still very useful tool for discovery work on finite snapshots

- Case be used to test correlation hypothesis on finite data sets before spending significantly more cycles
implementing a streaming / on-line version.

• Simple correlation techniques can provide high yield metadata to analysts

Islands of primary metadata are becoming unwieldy

- Post processing of primary metadata (correlation, newness, uniqueness, reduction techniques etc)
becoming a requirement to mitigate information overload.

Safeguarding Canada’s security through information superiority
Préserver la sécurité du Canada par la supériorité de l’information

Canada

TOP SECRET

■ ^ ■ Communications Security Centre de la sécurité

Establishment Canada des télécommunications Canada

Thanks

cse-cst.gc.ca

|@cse-cst.gc.ca

cse-cst.gc.ca

Safeguarding Canada’s security through information superiority

Préserver la sécurité du Canada par la supériorité de l’information

Canada

TOP SECRET

1^1 Communications Security
■ T ■ Establishment Canada

Centre de la sécurité

des télécommunications Canada

CSEC ITS/N2E
Cyber Threat Discovery

s' DISCOCON 2010
A CND Perspective
M&S/V - M ii
E' *
W s •
\ 1
Safeguarding Canada's security through information superiority n J1*1 Préserver /a sécurité du Canada par la supériorité de ¡'information V-^di Idvld

i

TOP SECRET

■ ^ ■ Communications Secunty Centre de la sécurité
|t| Establishment Canada des télécommunications Canada

N2E Cyber Threat Discovery

• CSEC/ITS Discovery: Context & Sitrep

• Current Capabilities

• CND Metadata Analysis at Scale

Safeguarding Canada's security through information superiority
Préserver la sécurité du Canada par la supériorité de l'information

Canada

2

TOP SECRET

■ ^ ■ Communications Security Centre de la sécurité

■ ” ■ Establishment Canada des télécommunications Canada

N2E Context

• Within CSEC/ITS, CND operations concentrated in N2

- Core: incident response team (alerts -> analysis -> mitigation advice)

- Malware/RE & VR teams

• Discovery has always been a required activity within N2

- IR often takes precedence

• N2E established to focus effort on discovery

- Hunting vs. firefighting: new threats, techniques, tradecraft

- Iterations of hypothesis based research/analysis

• What evidence is there in the data of compromised systems?

• What new threats or techniques can we find operating against us?

- Development of effective anomaly detection/heuristic techniques

• How can we better detect these new threats in the future?

• Shape capability development priorities

Safeguarding Canada's security through information superiority
Préserver la sécurité du Canada par la supériorité de l’information

Canada

3

TOP SECRET

1^1 Communications Security Centre de la sécurité

■ t ■ Establishment Canada des télécommunications Canada

~ ......

N2E Sitrep

• Team formally established earlier this year

- Growing to 7; hiring underway, strong candidates in pipeline

• Excellent access to full take data

• Analytical environment improving

• Progress on policy support

- Use of intercepted private communications

- Sharing mechanisms

Safeguarding Canada's security through information superiority
Préserver la sécurité du Canada par la supériorité de l'information

Canada

TOP SECRET

hi

Our Haystack

Three individual clients

- 10’s - low 1000’s of signature based alerts each day
(majority false positive)

Soon: “official" Internet gc.ca aggregation point

Full pcaps (retention: days to months)

- l’s - 10’s TB of passively tapped network traffic each day
Metadata (retention: months to years)

- 10's - 100’s GB of non-indexed, textual, network metadata / day (descrip

Safeguarding Canada’s security through information superiority
Préserver la sécurité du Canada par la supériorité de l'information

Canada



Notes on gc.ca:

•System capacity at 400TB per month

•Expected to start at 150TB per month (68 departments) but expected to
grow to capacity the next 2 years (>100 departments)

5

TOP SECRET

Communications Security Centre de la sécurité

Establishment Canada des télécommunications Canada



• Homemade wrappers for heuristic detection tools

• Popquiz/Slipstream for metadata

• And...

Toolset

$ find | xargs | grep | sed
cut | awk | sort | uniq -c |
perl | python

We have a bunch of heuristic tools which are good at detecting unknown
malware. They are mainly wrappers around tools like 8Ball and metadata
produced out of raw network traffic.

Detection from wrappers handled through alert management system but
metadata analyzed manually.

Using good old unix text filters!!!

Safeguarding Canada's security through information superiority
Préserver la sécurité du Canada par la supériorité de l’information

Canada

6

TOP SECRET

■ ^ ■ Communications Security Centre de la sécurité

■ t ■ Establishment Canada des télécommunications Canada

Homemade Wrappers: PonyExpress
SMTP processing

8Ball wrapper to process all email attachments

• Cuts and re-rebuilds SMTP session from full packet captures

• Extracts and logs metadata from SMTP and RFC822

• Extracts attachments and sends it to a cluster of 8Ball services

- Wrapper able to manage other deep scan tools (e.g. AV)

- Aggregates results from scans asynchronously

• Generates alerts for analysts to look at with full SMTP pcap

• Catches new implants or first stage delivered using attachments

Safeguarding Canada’s security through information superiority
Preserver la sécurité du Canada par la supériorité de l’information

Canada

•Currently processes 400,000 emails per day

•These are all emails that were previously filtered by Anti-SPAM, Anti-
Virus softwares

•Out of these, 400 gets promoted to the alert system

•1% of these alerts are worthy of reporting to client

•With gc.ca coming

•PonyExpress will be in front of Anti-Virus, Anti-SPAM due to
location of collection points.

•Looking at deploying appliances in front of PonyExpress

TOP SECRET

1^1 Communications Security Centre de la sécurité

■ t ■ Establishment Canada des télécommunications Canada

•J-.,.;:::-':-;;" 'i'tfwi'ihmmmm——— ...........____________________;___shhmh

Homemade Wrappers: Stripsearch

Automated binary analysis

• Recursively runs binary through the following tools:

• Correlation with file repository and reports

• Future: greater degree of automation

Safeguarding Canada's security through information superiority
Préseiver la sécurité du Canada par la supériorité de l'information

Canada

•Modules are run in parallel and some sequentially: Filters are run first to
exclude known good

•Each module can be selectively disabled or enabled

•Recursive means when 8Ball finds an embedded file, it extracts it and runs it
through stripsearch independently

•Currently processes our malware repository (~1500 samples) in lh30 minutes.
•Capable of processing folders and subfolders

TOP SECRET

■ ^ ■ Communications Secunty Centre de la sécurité

I t| Establishment Canada des télécommunications Canada

Metadata

• Standard Flows (with protocol guessing)

• DNS

- Queries / Answers and beaconing

• SMTP

- RFC 788 (SMTP), RFC 822 (2822, 5322) Internet Message
Format

• HTTP

- All server-side headers

- User-Agent summary

- List of POST without a GET, URLs and PE downloads

Safeguarding Canada’s security through information superiority
Préserver la sécurité du Canada par la supériorité de l'information

■! WÊÊÊÊiÊÊiÊÊÊmmÊm■

SMTP comes from PonyExpress
All others from popquiz

Canada

HTTP server headers and User-Agent summary are based on the first payload
packet so when headers are using more than one packet data gets truncated.
Sorting yelds strange side effects like showing nice staircase-like output

User-Agent: b
User-Agent: bla
User-Agent: blab
User-Agent: blabl
User-Agent: blabla

Found some SQL injection attempts in Server String: ‘DROP TABLE
servertypes; —

9

TOP SECRET

1^1 Communications Security Centre de la sécurité

|t| Establishment Canada des télécommunications Canada

Metadata: :findmask

ËSSBNÜ

Safeguarding Canada's security through information superiority
Préseiver la sécurité du Canada par la supériorité de l'information

Canada

When we have more than 7-10 unique known strings in the same flow its
usually worth investigating. Anything less is most likely a false positive.

Looks for XORed strings such as:
•Windows API calls
•Part of Registry keys

10

TOP SECRET

■ ^ ■ Communications Security Centre de la sécurité ,

I" | Establishment Canada des télécommunications Canada

'' ........

Noteworthy Catches

WMI-based implant

• Trend Micro has a good report this type of implant

http://us.trendmicro.com/imperial/md/content/us/trendwatch/researchanda

• Uses WMI for persistence instead of Registry or Boot sector

- Reside in ActiveScriptEventConsumer

- First seen in September 2009

See RFI_20091023_280_1 and IXR_20100909_732_1

- Caught with Pony Express

- Stripsearch fuzzy correlation linked it with previous reports

Safeguarding Canada's security through information superiority
Préseiver la sécurité du Canada parla supériorité de l'information

mm

Canada

Since first version, we saw increase in sophistication

At first WMI event created through a script run by
CSCRIPT.EXE

Now, is able to create the WMI event from the binary

TOP SECRET

■ ^ ■ Communications Security Centre de la sécurité

■ t ■ Establishment Canada des télécommunications Canada

Noteworthy Catches

Google IP addresses used as a sleep command

• From DNS metadata

- When call back domain was queried, a Google IP was used as a sleep
command. Searching for Google IPs in DNS metadata revealed new
TROPPUSNU domains and infected workstations.

Suspicious RDP and TOR sessions

• From Flow metadata we identified:

RDP

- Incoming RDP sessions from various locations to the RDP service (uses
the same certificate)

- Still under investigation
TOR

- Was picked-up as a suspicious burst of outgoing SSL connections going to
several locations from a single source

Safeguarding Canada's security through information superiority
Preserver la sécurité du Canada par la supériorité de l'information

Canada

12

TOP SECRET

■ ^ ■ Communications Security Centre de la sécurité

|t| Establishment Canada des télécommunications Canada

Noteworthy Catches: TOR...

• Most likely a breach of Internet usage policy

• All SSL CERTS were:

- Self-signed

- Certificate has a validity window of just 2 hours (e.g. 101017152421Z
to 101017172421Z)

- Issuer name appears to be a randomly generated fully qualified
domain name (FQDN), unique for each destination IP (e.g.
www.b2wwzduvdc5jyty3s7.net)

- Common Name (CN) field appears to be a randomly generated FQDN
unique for each session (e.g. www.xehzhl3cip.net)

Safeguarding Canada’s security through information superiority
Préseiver la sécurité du Canada par la supériorité de l'information

Canada

13

TOP SECRET

■ ^ ■ Communications Secunty

■ T I Establishment Canada

Centre de la sécurité

des télécommunications Canada

Noteworthy Catches: TOR...

• All sessions for October 17 extracted from pcap using the following
ngrep HEX string:

vdc5jyty3S7. net0(...x„.)

- The string is the end of validity period (Date only)

• Snort signature could be generated for looking at CERTS with
validity period starting/ending same day

The easiest signature would be to have snort look at starting end ending in the
same month. However, the signature will have to be changed every month.

A search for October only looking at CERTs ending in October (1010) resulted
in a lot of false positives but easily managed by greping start date of October.
Only one false positive left.

Safeguarding Canada's security through information superiority
Préseiver la sécurité du Canada par la supériorité de l'information

Canada

14

14

TOP SECRET

■ ^ ■ Communications Security Centre de la sécurité

I t ■ Establishment Canada des télécommunications Canada

i ; 'i-'. ~T~~ Ï

CND Metadata Analysis at Scale

Selected SAWUNEH Results

Safeguarding Canada's security through information superiority
Préseiver la sécurité du Canada par la supériorité de l’information

Canada

15

15

TOP SECRET

■ ^ ■ Communications Security Centre de la sécurité

■ T I Establishment Canada des télécommunications Canada

General Problem: Information Overload

• What is ‘at scale’ for CSEC CND?

• Problem: too much data

- Acquired, retained, summarized, analyzed, presented to analysts

• Opportunities at all tiers of system. Selected SAWUNEH topics
help in one or more of these areas

Safeguarding Canada's security through information superiority
Préserver la sécurité du Canada par la supériorité de l'information

Canada

Today:

1-10’s TB pcap retained each day

10’s-100’s GB of metadata retained each day

Future:

metadata continues to increase linearly with new access points
attempt to hold pcap steady at current rates through smarter retention policies

Tiers:

16

TOP SECRET

1^1 Communications Secunty
I t ■ Establishment Canada

Centre de la sécurité

des télécommunications Canada

What is SAWUNEH?

• Literally: ‘Summer Analytic Workshop Up North EH’

- Annual data analysis workshop hosted at CSEC by crypt-math shop

- 2010 thrust was CND

- Reps from each of the 5 eyes + cleared researchers

• Helps fill the (sparsely inhabited) plane that often exists between
OpsDev and R&D

• Special Assets: Netezza, Cray XMT

Many researchers but focus is ‘applied’

Netezza: distributed data appliance made available to facilitate workshop
OpsDev vs R&D rather than Ops vs R&D

Safeguarding Canada's security through information superiority
Préseiver la sécurité du Canada par la supériorité de l'information

Canada

■■■■■■■■■■■

17

17

TOP SECRET

■ ^ ■ Communications Security

■ t ■ Establishment Canada

Centre de la sécurité

des télécommunications Canada

Topic 1: Correlating ‘E-Mail, Web, Flow’ Metadata

• E-mail based content delivery attacks against GC *very* common

• CND capability rapidly developed & deployed; heavily relies on attachment
scanning

• The inevitable evasion

- Attacks now commonly use email as Inducement to URL visit, instead of direct
content delivery.

• Too many benign hyperlinks delivered over email

- Need to reduce metadata presented to front line analysts

Safeguarding Canada’s security through information superiority
Préserver la sécurité du Canada par la supériorité de l'information

Canada

PonyExpress:

Yields many detections with low false positive rates
Commercial sector has responded in similar fashion

Email as inducement

Delivery of HTTP URL over E-mail, exploit is delivered via
HTTP

18

TOP SECRET

1^1 Communications Security Centre de la sécurité

■ t ■ Establishment Canada des télécommunications Canada

Topic 1: Correlating ‘E-Mail, Web, Flow’ Metadata

Reducing Email URL metadata volume delivered to front line analysts

• Select highly suspicious subset of Email URL metadata based on
correlation

• Only present ‘Email URL’ metadata if:

- Email was inbound

- Contained one or more hyperlinks

- Hyperlink nominated as suspicious

- Hyperlink was actually visited by a recipient.

- Provide flow and http metadata for resulting HTTP session

Safeguarding Canada's security through information superiority
Préserver la sécurité du Canada par la supériorité de l'information

Canada

Initial definition of ‘suspicious’ was naive: suffix dictionary
Have actual visit helps by:
more likely to be successful attack

means analyst likely has access to related web traffic to assist with triage

19

TOP SECRET

■ ^ ■ Communications Security Centre de la sécurité

■ T ■ Establishment Canada des télécommunications Canada

.............IP......... I ........ 'I'Pi'n'i

Topic 1: Correlating ‘E-Mail, Web, Flow’ Metadata
SAWUNEH approach

• Import EMAIL, URL, FLOW metadata Into separate tables on the
NETEZZA appliance (distributed db)

• Single SQL Query to fill requirement:

Safeguarding Canada's security through information superiority
Préserver la sécurité du Canada par la supériorité de l’information

Canada

I

20

TOP SECRET

■ ^ ■ Communications Security

■ t ■ Establishment Canada

Centre de la sécurité

des télécommunications Canada

Email + Web’ attack detection
SAWUNEH Style

Findings:

• 50% of the unique results were malicious

• Significantly lower false positive rates compared to URL inspection only

• Result provides anomaly tip but also context at fingertips for analyst

Next Steps:

• Better definition of 'suspicious' URL

• Automate extraction of web content from session for analysts

• Automate feeding of web payloads into content scanning system = partial mitigation
of the original evasion. Higher analyst productivity

Possible ‘suspicious’ definitions: URLs tagged with ‘ObfJavaScript, FlashContent, LedToDownload’ etc

Correlated on URL ‘trees’ not isolated URLS (otherwise redirects, remote hrefs etc defeat scanning)..

But somewhat expensive (not sure if this is solved in feasible way ?)

primary metadata sources too large for discovery... need value-add reductions
reduced ‘working sets’ of metadata = higher yield for analyst time.

They evade by moving payload delivery to different session. We follow them and (attempt) to close evasion.

Safeguarding Canada's security through information superiority
Preserver la sécurité du Canada par la supériorité de l'information

Canada

21

21

TOP SECRET

■ ^ ■ Communications Security Centre de la sécurité

It| Establishment Canada des télécommunications Canada

Topic 2: Malicious Attachment Prediction

Problem:

• Our current email scanning system fully processes every email it sees in a modular
fashion

• Works very well today, but we can’t afford to scale this approach ‘as-is’ to meet
future requirements.

• Large proportion of time spent in *late* stages of processing., especially deep scan

Possible Approach:

• Can we predict which emails will score in the deep scan based only on metadata
extracted during earlier processing phases?

• If so, can we drop those that have low probability of deep scan success?

Safeguarding Canada's security through information superiority
Préseiver la sécurité du Canada par la supériorité de l'information

ö

Canada

Fully process = sessionize, smtp, rfc822, mime, deep
scan(8Ball) of all attach...etc

Optimization axiom: find your hotspot

22

TOP SECRET

1^1 Communications Security Centre de la sécurité

■ T ■ Establishment Canada des télécommunications Canada

Malicious Attachment Prediction

• Given metadata from each processing phase, build a predictive model to selectively
promote emails with high probably of non-zero scoring in 8-ball based on features
extracted early in processing

• Choice of Predictive Model

- Random Forests: decision tree based classifier (Brieman 2001)

• Take a feature set (X,.... Xp) and a corpus of training samples with *known*
classification as input. Generate Random Forest of decision trees to be used to
prediction classification

• New samples pass through previously trained forest which yields probability that
deepscan will be nonzero

Safeguarding Canada's security through information superiority
Préseiver la sécurité du Canada par la supériorité de l'information

Canada

Phases: (ip, tcp, smtp, rfc822, mime etc)

Classifier:

Based on probability of scoring non-zero in deep scan

In our case: email metadata types + > 10000
previously scanned samples with full metadata
including overall scan score used to build the Forest

Note: Only build/train forest once. Subsequently you
are just passing metadata through and getting an
P(email scores non-zero)

23

TOP SECRET

■ ^ ■ Communications Security Centre de la sécurité

■ ▼ ■ Establishment Canada des télécommunications Canada

Malicious Attachment Prediction
Data Reduction vs. ‘Interesting’ Data Loss

Safeguarding Canada's security through information superiority
Préserver la sécurité du Canada par la supériorité de l'information

Canada

24

TOP SECRET

l+l

Communications Security Centre de la sécurité

Establishment Canada des télécommunications Canada

m



Malicious Attachment Prediction

Result:

• 85% data reduction with 1-3% loss of ‘interesting’ emails

- Can discard majority of emails with minimal loss in positive hits

• Prediction with model is relatively cheap (10,000s features per second)

Next steps:

• Extract those features which contribute most to prediction and incorporate
appropriate filters in each stage of email processing system with aim to discard
early and often

• There are likely a few simple features contributing disproportionally to predictive
power: flow size, attachment size, attachment type etc

Safeguarding Canada's security through information superiority
Préseiver la sécurité du Canada par la supériorité de l'information

Canada

Could be used either permanently, or simply as a learning tool to find which
features you might want to filter on.

Emphasis on feature predictors available early in processing (flow, smtp, 822
header rather than MIME content etc)

25

TOP SECRET

l+l

Communications Security
Establishment Canada

Centre de la sécurité

des télécommunications Canada

WÊBi

■HSSe-“”

(Mini) Topic 3: PE Masquerade Detection

• Multiple threat actors observed masquerading Windows PE downloads using
‘benign’ filename extensions (jpg, gif, htm etc)

• We log metadata on HTTP sessions containing a PE header In content of first
packet (modules available In both SLIPSTREAM and POPQUIZ)

- Exclude entries with suffixes common to PE downloads (exe, bin, php, asp)

- Provide analysts contextual metadata & 1 click access to payloads •

• Easy to see entries that jumped out: file extensions such as .jpg, .gif highly
suspicious, often malicious

• Also filter by ‘uniqueness’ over time to eliminate AV / OS updates

• Next Step: automate extraction of pe masquerades and push through content
scanning system

Safeguarding Canada's security through information superiority
Préseiver la sécurité du Canada par la supériorité de l'information

Canada

26

TOP SECRET

1^1 Communications Security Centre de la sécurité

■ t ■ Establishment Canada des télécommunications Canada

Safeguarding Canada's security through information superiority

Préseiver la sécurité du Canada par la supériorité de l’information

Canada





TOP SECRET

1^1 Communications Security Centre de la sécurité

It | Establishment Canada des télécommunications Canada



Safeguarding Canada's security through information superiority

Préserver la sécurité du Canada par la supériorité de l'information

Canada

28

TOP SECRET

1^1 Communications Security Centre de la sécurité

■ t I Establishment Canada des télécommunications Canada

Final Notes

SAWUNEH outcomes made a few things clearer to us

• Our days of plain old grep as primary search tool are nearing an end

• Relational DB’s provide us with a few key benefits:

- Simple indexing of existing metadata (sub second searches)

- ability to correlate easily across ‘primary’ metadata outputs

- significantly faster hypothesis testing

• Depending on scale, DB over head may be too costly

- In such cases, still very useful tool for discovery work on finite snapshots

- Case be used to test correlation hypothesis on finite data sets before spending significantly more cycles
implementing a streaming / on-line version.

• Simple correlation techniques can provide high yield metadata to analysts

- Islands of primary metadata are becoming unwieldy

- Post processing of primary metadata (correlation, newness, uniqueness, reduction techniques etc)
becoming a requirement to mitigate information overload.

Safeguarding Canada's security through information superiority
Préserver la sécurité du Canada par la supériorité de l'information

Canada

TOP SECRET

■ ^ ■ Communications Security Centre de la sécurité

It | Establishment Canada des télécommunications Canada

Thanks

cse-cst.gc.ca

^m@cse-cst.gc.ca

Safeguarding Canada’s security through information superiority
Préserver la sécurité du Canada par la supériorité de l'information

Canada

30

e-Highlighter

Click to send permalink to address bar, or right-click to copy permalink.

Un-highlight all Un-highlight selectionu Highlight selectionh