Event RPE-002: The Turing Test
RPE-002: The Turing Test
Date: December, 2018 | Location: DreamPort Facility in Columbia MD
In honor of Alan Turing, DreamPort plans to put on their own 'Turing Test'. In 1950, Alan Turing conceived of the idea to identify in a conversation between two entities, which entity was a robot, and which was human. Our DreamPort Turing Test will be an RPE competition where participants implement an automated process to interact with a Microsoft Windows machine just as a human user may do with the goal being to fool a human judge who is monitoring target computers via Remote Desktop Protocol (RDP) or Virtual Network Computing (VNC) into thinking a normal user is interacting with that machine and not an automated program or process. This RPE will use the DreamPort HQ network which will include both virtual and physical endpoints, servers and common services including connectivity to the Internet.
Participants should plan on using a fully patched Microsoft Windows 7 Ultimate as the target environment and will have substantial time to interact with the target for setup or instrumentation. There are no restrictions on the type of implementation (e.g. script, program, etc.) that a participant may use, and they can assume their program, script or service will have system level access to the target machine. The judges will not be allowed to browse the hard drive(s) of the target machine(s) or view the System Manager (process listing) of the target(s). They may only watch the desktop, but this means they can view the system tray if they choose.
Participants must try to implement as many user behaviors as possible such as:
- Opening/Closing programs
- Browsing the Internet
- Creating content (e.g. Microsoft Word)
- Communicating via Email
- Printing documents
It is important to point out that we are striving for an evolution in approaches typically used in automated testing such as simple automation of a web browser or desktop program. When a participant solution browses the Internet, it should try to mimic user behavior as much as possible. A Selenium script may start a web browser and click on a few links, but it will likely click on the same links every time and will run much faster than a human. We are searching for a solution that will start a web browser and make decisions about websites and links that are potentially different each time. Does the solution search on a term and then click on results? Does the solution view a link aggregation site like a news and click on a random series of links each time?
Participants should strive to implement advanced features such as creating social media content or chatting in real time. The target systems will have the following applications installed:
- Microsoft Office 2016
- Google Chrome
- Mozilla Firefox
- Tor Browser Bundle
- Adobe Acrobat Reader
In this RPE, a small number of judges will be shown a series of displays (3-5) and must decide which computer is an automated process and which are normal users. DreamPort will provide a small number of participants in a separate room or area and inform them when the judging process begins, they should begin to interact with their computer(s). Judges will have a small window of time (e.g. 5 minutes) of viewing to determine which computer is being controlled via automated means and which are not. The judging process will occur in a series of rounds where any system identified as being controlled automatically is ejected from the competition. Judges will be cautioned to watch the performance of the target machines. Any machine that slows down in a noticeable manner may raise the judge's suspicion. For this reason, participants should exercise caution to not implement a resource intensive component or program.
- Use a network service for issuing updates, libraries, scripts or other binary files
Participants may not:
- Infect or install target computer with remote control software or malware and use as means of seemingly automated control
- Play movie of pre-recorded automated interaction
Participants should strive to produce an executable and associated or required binary files. This solution should be capable of manipulating the host system in a manner similar to how a human may use the same system including moving the mouse around in a non-random fashion. There are no limitations on the architecture of the participant solution (x86 versus amd64) and no limitations on the compiler or programming language. We suggest the footprint of the installed file(s) be kept low but there is no restriction on static versus compiled code and participants should feel free to use Microsoft .Net (e.g. C#) if they wish as well. Participants will be able to install their executable in such a way that it will start automatically when the machine boots. The target machines will be configured to login without any password required using an administrator account. To put this another way, judges will not see any target system during the boot process, only after the machine(s) boot and login to the desktop.
This RPE requires participants have the at least intermediate the following skills:
- Windows API (e.g. User Interface Automation)
- C/C++, C#
- .NET Framework
- Visual Studio
The following skills are suggested:
- Machine Learning
- AutoIt Scripting
Hints and Suggestions
Learning from our first RPE, we will offer the following suggestions:
- Machine Learning has a direct application!
- Train your programs?