RPE-014: Rossum's Universal Robots

Events

RPE-014: Rossum's Universal Robots

---

And the winner is…

Congratulations to Technica Corp for winning this RPE!

Description

This event is open to all colleges, universities, individuals and non-traditional participants who have experience in machine learning applications to network packet capture (PCAP) for automated anomalous behavior identification. We have $10,000 (USD) in prizes to give away to the best performers as determined by us!

Wikipedia and Matt Simon from Wired write that in 1920, Czech writer Karel Capek wrote the play Rossum's Universal Robots or Rossumovi Univerzální Roboti. This play is believed to be the introduction of the word robot to the English language. In this play, roboti (robot) are produced in a factory as artificial beings that resemble androids we may envision from Hollywood instead of the traditional bots of metal, plastic and servos. These roboti start rebelling and end up killing nearly all humans on the planet. In this play, it's easy to spot the roboti, but here at DreamPort, spotting the cyber attacker even after the fact let alone while attacks are in progress, can be much harder.

Overview

Machine Learning influences almost everything MISI does at DreamPort, and we want to bring new people into the fold. In honor of DreamValley's second anniversary, we created this challenge which instead of 'rapid', we prefer to think of as:

• Robot prototype event
• Researching prototype event

For those of you who have not visited DreamValley, in 2019 we began construction of a scale-model city operated by programmable logic controllers (PLC) which we have named DreamValley. Check out a photo below of DreamValley 'at night' (with the lights out). There are now eight (8) separate real-world systems inside of this scale-model city that are controlled by different PLC devices. We recently completed a structured event in which attackers were invited to attack city infrastructure from known locations while a defensive team was alerted to the incoming attacks. Background traffic, our surprises, and simulated users all make the defenders job hard but the most important point for you is that all of this traffic has been captured and is begging to be analyzed. There are more than seventy-five physical hosts and depending on our scenario hundreds of virtual hosts in this network. We have PLCs from:

• UniPi
• Mitsubishi
• Siemens
• AutomationDirect
• Schneider
• AllenBradley
• PhoenixContact

We are challenging interested participants to:

1. Analyze gigabytes of known good and known bad (organized distinctly) traffic from the DreamValley experiment network.
2. Receive and label (with MISI help) gigabytes of known bad traffic from DreamValley.
3. Build a classifier of anomalous network events using the algorithms of your choice.
4. Build a single script/executable to ingest new PCAP (on your test days) using your ML 'model to determine if and where attacks inside of DreamValley are being launched.

Remote Participation

This event is well-suited for remote participation although you must ensure you can download large amounts of PCAP files. For live-fire exercises we may ask you to connect to our virtual private network.

How Will It Work

Thanks to KeySight, VMWare and other partners, we are capturing all traffic on the DreamValley network. To participate in this RPE you have to download the initial 'known good' and 'known bad' PCAP. We are currently determining the most effective mechanism to distribute this PCAP but for now we will be using Amazon S3. To be added as an eligible host you must communicate your public facing IP address you will use to download the PCAP from DreamPort. You should assume that this traffic comes from hosts where network time synchronization is used (for those hosts who support it).

You should begin analysis of this PCAP immediately. Can you identify all transmitting hosts? Can you determine protocols and ports? Next, what are the most effective features for your models? Which algorithms are the most effective?

At this point, we will meet virtually to discuss labeling the PCAP data. We know all our hosts, assets and thanks to our recent events, we know who the attackers were. We can provide detail on each host manufacturer, purpose and how these hosts connect with each other. We can provide details on where attackers were located and what they did.

Now, it's on you. You need to work. Select your features, build a representative model score to see if you can spot the past attacks. Wash, rinse, repeat. If you can make this work. Get ready for the evaluation. We discuss this later.

Requirements

Here are the requirements you must meet if you wish to be considered a valid contender for the prizes:

1. You must produce a single machine learning model object. If you are using separate algorithms (instead of a fused model) with separate inputs, you still must architect your code to decompress, unpack or the like from a single input file. There is no upper limit on size of this file.
2. You must produce a single script or executable capable of training new PCAP and analyzing suspect PCAP. Everything should be done from one (1) script. There is no strict limit on which programming language you choose.
3. The 'analyze' output function of your script/program must supply the IP address or MAC address of the suspected attackers. You are free to supply a collection of hosts but if you do, you must use a numerical value to indicate confidence or likelihood of which host(s) are most believed to be attackers.
4. You must turn over copies of the source code and binary model(s) used for analysis to be eligible for prize money. You will retain intellectual property rights, but you are granting MISI and United States Cyber Command unrestricted rights to use and modify your creation.

Schedule

This event is going to take place over time. Here is the proposed schedule which is subject to change:

• September 17: Registration Closes.
• September 17: MISI releases "known-good" & "known-bad" PCAP to participants.
• September 21: Q&A & Data Labeling Session.
• October 19-20: MISI holds virtual call to evaluate progress and provide additional Q&A, as necessary.
• November 2-5: MISI holds individual virtual meetings with registrants for evaluation of progress.

After the final step, participants will be evaluated on their performance which we discuss next.

Evaluation

Because we have a prize pot, we must ensure we evaluate each participant fairly. We have baselined DreamValley and will conduct a set of attacks on the hosts. We will not be telling you which attack and won't be telling you where we launch from. We will launch a minimum of five (5) attacks to include two (2) that are not located in the PCAP you will get for training. Each attack will be delivered to you as PCAP. You must run your analysis code against this PCAP using the model you create and return the output. We will be asking you to run this analysis live in a virtual meeting screenshare so we can observe the output.

The principal evaluation criteria is two-fold:

• Did you correctly identify the attacker IPs?
• Did you correctly identify the target IPs?

We identify the following additional formal criteria for ranking performance. Notice these are not identified as requirements, but we use these to a means to identify stellar performers. Do not work on these features instead of the actual hard requirements as we discuss in the previous section.

• Are you able to determine and report the intermediate hosts in an attack?
• Are you able to classify and report the protocols (not ports) used in the attack?

The prize breakdown at this time is as follows (this is subject to change):

1. $5,000.00 USD for first place
2. $3,000.00 USD for second place
3. $2,000.00 USD for third place

Tie Breaker

In the event that multiple parties are evaluated as first place we will use the following concept to break the tie. No machine learning model should be considered concrete and static. You may have better data tomorrow then you have today. Does your model contain a retraining feature? If we offer additional detail on new PCAP can you incorporate this new data to produce a new model? In the event of a tie, the winner will be the party whose model can be retrained on the fly.

If no parties choose to work on re-training, the winner will be determined by MISI evaluation.