What We Learned from Event RPE-003: The Wolf in Sheep's Clothing

What We Learned

RPE-003: The Wolf in Sheep's Clothing


Description

The Fables of Aesop by Joseph Jacobs published in 1894 includes the tale of the wolf in sheep's clothing. Just as in this tale, DreamPort is playing the Sheperd trying to spot the wolf in the flock. DreamPort plans to execute an RPE competition where participants design, implement or enhance user activity monitoring (UAM) solutions for detecting live and recent insider threat attacks or unauthorized activities. DreamPort is interested in identifying UAM solutions that employ advanced real-time analysis of multiple data sources for detecting unauthorized activities. DreamPort will again use HQ (from our first RPE) but this time setup a we will use live portions of the HQ environment consisting of Windows and Linux machines (endpoints and servers) and use human actors to interact with this environment in real-time. During the RPE, one or more of these actors will be 'bad' and will execute unauthorized actions at any time. Participants must install their solutions within HQ and attempt to detect these bad actors as quickly as possible. The solution(s) which detect all of the unauthorized actions will be deemed a winner'.

DreamPort acknowledges multiple solutions are available today for UAM, but we are explicitly interested in solutions which offer predictive monitoring (configuration-less) features and not just policy-based (ALLOW/DENY) monitoring features.

Participant solutions should offer alerts for overtly prohibited activities and activities which fall outside normal operating behavior.

Overview

  • User Activity Monitoring (UAM)
    • Events were either incidents and attacks
  • Five Teams:
    • Booz Allen/NVIDIA/CrowdStrike
    • IBM
    • Jazz Networks
    • LogicHub
    • Splunk/SRC Technologies
  • 25 Events Spread Across 20 Hours of Competition (2 Rounds)

High-Level Findings

  • Companies with agents performed better.
  • Majority of incidents covered five categories/challenges:
    • identify obvious plain text threats
    • identify malicious/suspicious USBs added to network.
    • identify foreign computer(s) on network or added to network.
    • identify external connections
    • identify incidents after loss of log data
  • Teams, in aggregate, detected 80% of incidents/attacks.
  • At least two teams detected ~50% of incident/attacks.
  • Majority of teams detected 24% of incidents/attacks.
  • Majority of reports were false positives.

General System Deficiencies

  • Lacked consistent ability to identify foreign computer on network or added to the network.
  • Lacked consistent ability to identify malicious/suspicious USBs added to network.
  • Lacked consistent ability to identify obvious plain text threats and incidents (e.g., file names, email contents, computer name).
  • Lacked consistent ability to identify external connections (e.g., website, IP).
  • Lacked consistent ability to identify incidents after loss of log data.

How many teams detected each incident/attack?

Scores by Incident
Detections vs. Reports

Results

  • Best Performance with Agent: Jazz Networks
  • Best Performance without Agent: IBM
  • Highest Number of Detections: Jazz Networks
  • Lowest False Positive Rate: Booz Allen
  • Best Performance by Category:
    • Obvious plain text threats: Jazz Networks
    • Malicious/suspicious USBs added to network: Jazz Networks
    • Foreign computer(s) on network or added to network: Jazz Networks
    • External connections: LogicHub
    • Incidents after loss of log data: Jazz Networks
  • Best Overall Performance: Jazz Networks

Congratulations to all of the participants!

Special Thanks To

Federal Business Council, Inc. (FBC) specializes in producing conferences and trade show events at Federal Government locations throughout the United States. Each month thousands of federal employees attend FBC events to evaluate the latest advances in technology, military hardware, training, and other product areas, as well as update their sources for future requirements.

Founded in 1976, FBC has conducted more than 4,000 on-site expositions and conferences for the Department of Defense, Intelligence Community, and civilian agencies. Over the last 40 years, FBC has become a comprehensive resource for marketing to the Federal Government. To learn more, visit https://www.fbcinc.com/.

Questions?

For any RPE-001 questions or concerns, please contact us.

Participants

Booz Allen Hamilton
ESI
Fuse Solutions
IBM
Jazz Networks
LogicHub
PCI
Visionist Inc