Event RPE: The Turing Test
RPE: The Turing Test
Date: TBD. | Location: DreamPort Facility in Columbia MD
In honor of Alan Turing, DreamPort plans to put on their own 'Turing Test'. In 1950, Alan Turing conceived of the idea to identify in a conversation between two entities, which entity was a robot, and which was human. Our DreamPort Turing Test will be an RPE competition where participants implement an automated process to interact with a Microsoft Windows machine just as a human user may do with the goal being to fool a human judge who is monitoring target computers via Remote Desktop Protocol (RDP) or Virtual Network Computing (VNC) into thinking a normal user is interacting with that machine and not an automated program or process. This RPE will use the DreamPort HQ network which will include both virtual and physical endpoints, servers and common services including connectivity to the Internet.
Participants should plan on using a fully patched Microsoft Windows 7 Ultimate as the target environment and will have substantial time to interact with the target for setup or instrumentation. There are no restrictions on the type of implementation (e.g. script, program, etc.) that a participant may use, and they can assume their program, script or service will have system level access to the target machine. The judges will not be allowed to browse the hard drive(s) of the target machine(s) or view the System Manager (process listing) of the target(s). They may only watch the desktop, but this means they can view the system tray if they choose.
Participants must try to implement as many user behaviors as possible such as:
- Opening/Closing programs
- Browsing the Internet
- Creating content (e.g. Microsoft Word)
- Communicating via Email
- Printing documents
It is important to point out that we are striving for an evolution in approaches typically used in automated testing such as simple automation of a web browser or desktop program. When a participant solution browses the Internet, it should try to mimic user behavior as much as possible. A Selenium script may start a web browser and click on a few links, but it will likely click on the same links every time and will run much faster than a human. We are searching for a solution that will start a web browser and make decisions about websites and links that are potentially different each time. Does the solution search on a term and then click on results? Does the solution view a link aggregation site like a news and click on a random series of links each time?
Participants should strive to implement advanced features such as creating social media content or chatting in real time. The target systems will have the following applications installed:
- Microsoft Office 2016
- Google Chrome
- Mozilla Firefox
- Tor Browser Bundle
- Adobe Acrobat Reader
Participants are encouraged to bring functional prototypes to the start of this RPE.
Participants will be evaluated by a team of 3-5 personnel who monitor all multiple screens visually for a set period not to exceed 10 minutes. Not all of these screens will be RPE participants, some will be driven by actual humans. Judging personnel will be looking for behavior across all of the screens that does not appear to be human and will call out a screen as such if they suspect they are watching an automated bot rather than a human operator. Judges will not be able to see the users 'over the shoulder' but rather they will be watching a remote desktop display of the target. Judges will also not be able to view the process list, system tray or any other details about each running station that could give away if a system is being automatically controlled instead of manual interaction from a human.
Since there will be more than one human actor driving a station on this network, the judges must determine during the time window which screens are not being driven by a human user and instead by an automated solution.
Evaluation will not begin until the end of day 2 of the event. After this, evaluation will take place every 80 minutes for 10 minutes or until all participating teams have been eliminated. Participants will get a 10-minute warning prior to the start of the next evaluation window where they must ensure their solution is running as expected.
Because of this, participants are encouraged to modify or enhance their approaches to escape the next round of evaluation. The remaining solution will be declared the winner.
The expected solution from this RPE is a configurable, functional prototype which can interact with a Microsoft Windows installation in a realistic way. No solution exists that can automate and imitate every aspect of a human interacting with a computer, but we want to advance the state of the art in that direction.
Solutions can be scripted or compiled but they can only rely on the built-in functions of Microsoft Windows or resources installed along with the solution (e.g. configuration files, executable code, etc.). Solutions can require special items or hardware be attached to the test system using commonly available interfaces only (e.g. USB 2 or 3).
This RPE requires participants have the at least intermediate the following skills:
- Windows API (e.g. User Interface Automation)
- C/C++, C#
- .NET Framework
- Visual Studio
The following skills are suggested:
- Machine Learning
- AutoIt Scripting