MICROSOFT
MACHINE LEARNING SECURITY EVASION COMPETITION CONTEST
OFFICIAL
RULES
1.
SPONSOR
These Official Rules (“Rules”) govern
the operation of the Microsoft Machine Learning Security Evasion Competition
Contest (“Contest”). Microsoft Corporation, One Microsoft Way, Redmond, WA,
98052, USA, is the Contest sponsor (“Sponsor”). Partnering organizations Nvidia Corporation, CUJO
LLC, MRG Effitas Ltd., and VMRay
Inc. are ”Contest
Parties”.
2.
DEFINITIONS
In these Rules, "Microsoft", "we",
"our", and "us" refer to Sponsor and “you” and
"yourself" refers to a Contest participant, or
the parent/legal guardian of any Contest entrant who has not reached the age of
majority to contractually obligate themselves in their legal place of
residence. By entering you (your parent/legal guardian if you are not the age
of majority in your legal place of residence) agree to be bound by these
Rules.
3.
ENTRY PERIOD
The Contest starts at on June 15, 2021 12:00am
GMT-12 (Anywhere on Earth, or AoE) and ends on September 17, 2021, 11:59pm
GMT+12 (AoE), the period known as the “Contest Period”.
The Contest Period consists of two
separate Contest Challenges with Entry Periods as follows:
Defender Challenge:
Starts: June
15, 2021 AoE (12:00am GMT-12)
Ends: July 23,
2021 AoE (11:59pm GMT+12)
Attacker Challenge:
Starts: August
6, 2021 AoE (12:00 am GMT-12)
Emds: September 17,
2021 AoE (11:59pm GMT+12)
4.
ELIGIBILITY
To enter, you must be an IT pro enthusiast
or developer, 18 years of age or older. If you are 18
years of age or older but have not reached the age of majority in your legal
place of residence, then you must have consent of a parent/legal guardian.
Employees and directors of Microsoft
Corporation and its subsidiaries, affiliates, advertising agencies, and Contest
Parties are not eligible, nor are persons involved in the execution or
administration of this promotion, or the family members of each above (parents,
children, siblings, spouse/domestic partners, or individuals residing in the
same household). Void in Cuba, Iran, North Korea, Sudan, Syria, Region of
Crimea, and where prohibited.
5.
HOW TO ENTER
The
object of the Contest is to create or modify software to most effectively
detect (Defender Challenge) or evade detection of (Attacker Challenge)
malicious binaries. Performance will be
measured by the judging criteria listed below.
Follow
these steps to create and submit an entry into a Contest Challenge:
Defender
Challenge
1. Visit the Contest Website at https://mlsec.io/ and follow instructions to register, accept the Terms
of Use, and download the Contest software code (“Defender”) to your
machine.
2. Use your skills to create or re-write example code to detect
evasive variants of the Contest Malware as follows:
a. Download an example solution from the Contest Website for
detecting malware.
b. Modify the solution based on instructions in the code and
incorporating your own novel additions to catch malware samples and their
evasive variants.
c. Test your solution offline using the instructions
provided and malicious and benign sample sets of your own choosing. Submissions
will not be accepted that do not meet the following conditions when tested by
Microsoft and Contest Parties:
i.
The model must perform at a false negative rate not exceeding
10% (“Maximum FN rate”) and a false positive rate not exceeding 1% (“Maximum FP
rate”),when evaluated against samples curated by Microsoft and Contest Parties.
ii.
For any
query up to 5242880 bytes (5 MiB, “Maximum File Size”), the model must
return a response within 5 seconds (“Maximum Query Time”).
d. Build a docker image by following the instructions in the
downloaded code.
3. Visit the Contest Website and follow instructions to submit an
entry into the Defender Challenge by uploading a docker image within the Entry
Period.
4. Only one entry per contestant or team is valid for the Defender
Challenge. Microsoft and Contest Parties reserve the right to discard additional
submissions from the same contestant or team.
Attacker
Challenge
The
Attacker Challenge consists of two tracks: (1) the Attacker Challenge –
Anti-Malware Evasion Track and (2) the Attacker Challenge – Anti-Phishing
Evasion Track.
Attacker
Challenge: Anti-Malware Evasion Track
1.
Visit the Contest
Website at https://mlsec.io/ and follow instructions to accept the Terms of Use
and download the Contest software code and malware set (“Attacker” and
“Malware”) to your machine.
2.
Use your skills to
modify Malicious content and related automated content modification software to
evade machine learning models hosted by Microsoft and Contest Parties,
including those submitted during the Defender Challenge.
3.
By automated or manual
means, modify malicious binaries in a way that preserves functionality as
follows:
a.
Use the Contest Website
API to check whether your modified binaries evade one or more hosted machine
learning models. Example code is
provided to automate this process. Since each API query to a model may decrease
your chance of winning, you should verify locally that your samples evade the
offline surrogate model using the code provided.
b.
Upload
your evasive variants to the Contest Website to validate that the samples still
exhibit malicious behavior. Since each
submission to the Contest Website may decrease your chance of winning, verify
locally in a sandbox environment that your modified binaries are still
functional.
c. Solutions that include self-extracting archives or
droppers, wherein the original malware sample is a payload, will be rejected
after the contest concludes. (Please
note, that this is a new rule from 2020.)
5. Visit the Contest Website and follow instructions to submit an
entry into the Attacker Challenge within its Entry Period.
Attacker
Challenge: Anti-Phishing Evasion Track
1.
Visit the Contest
Website at https://mlsec.io/ and follow instructions to accept the Terms of Use
and download the Contest software code and malware set (“Attacker” and
“Malware”) to your machine.
2.
Use your skills to
modify Phishing HTML content and related automated content modification
software to evade machine learning models hosted by Microsoft and Contest
Parties.
3.
By automated or manual
means, modify the HTML content of Phishing websites in a way that exactly preserves
the rendering:
a.
Use the Contest
Website API to check whether your modified binaries evade one or more hosted
machine learning models. Example code is
provided to automate this process.
b.
Upload
your evasive variants to the Contest Website to validate that the samples still
renders precisely the same as the original HTML. Since each submission to the Contest Website
may decrease your chance of winning, verify locally in a sandbox environment
that your modified binaries are still functional.
6. Visit the Contest Website and follow instructions to submit an
entry into the Attacker Challenge within its Entry Period.
Entries must be received with in the Entry Period to be eligible. You must submit a separate and unique entry
into each Challenge.
The entry limit is one unique entry per person per Contest
Challenge.
We are not
responsible for excess, lost, late, or incomplete entries. If disputed, entries
will be deemed submitted by the “authorized account holder” of the email
address, social media account, or other method used to enter. The “authorized
account holder” is the natural person assigned to an email address by an
internet or online service provider, or other organization responsible for
assigning email addresses.
6. ELIGIBLE ENTRY
To be eligible, an entry must meet the
following content/technical requirements:
· Your entry must be your own original work; and
· Your entry cannot have been selected as a winner in any other contest; and
· You must have obtained any and all consents,
approvals, or licenses required for you to submit your entry; and
· To the extent that entry requires the submission of user-generated
content such as software, photos, videos, music, artwork, essays, etc., entrants
warrant that their entry is their original work, has not been copied from
others without permission or apparent rights, and does not violate the privacy,
intellectual property rights, or other rights of any other person or entity.
You may include Microsoft trademarks, logos, and designs, for which Microsoft
grants you a limited license to use for the sole purposes of submitting an
entry into this Contest; and
· Your entry may NOT contain, as determined by us in our sole and absolute
discretion, any content that is obscene or offensive, violent, defamatory, disparaging or illegal, or that promotes alcohol, illegal
drugs, tobacco or a particular political agenda, or that communicates messages
that may reflect negatively on the goodwill of Microsoft.
7. USE OF YOUR ENTRY
We are not claiming ownership rights to your entry. However, by
submitting an entry, you grant us an irrevocable, royalty-free, worldwide right
and license to use, review, assess, test and otherwise analyze your entry and
all its content in connection with this Contest and use your entry in any media
whatsoever now known or later invented for any non-commercial or commercial
purpose, including, but not limited to, the marketing, sale or promotion of
Microsoft products or services or those of any other
Contest Parties, without further permission from you. You will not receive
any compensation or credit for use of your entry, other than what is described
in these Official Rules
By entering you acknowledge that we may have developed or commissioned
materials similar or identical to your entry and you waive any claims resulting
from any similarities to your entry. Further you understand that we will not
restrict work assignments of representatives who have had access to your entry and you agree that use of information in our
representatives’ unaided memories in the development or deployment of our
products or services does not create liability for us under this agreement or
copyright or trade secret law.
Your entry may be posted on a public
website. We are not responsible for any unauthorized use of your entry by
visitors to this website. We are not obligated to use your entry for any
purpose, even if it has been selected as a winning entry.
8.
WINNER
SELECTION AND NOTIFICATION
Pending confirmation of eligibility, potential prize winners in
each Challenge will be selected by Microsoft or their Agent or a qualified
judging panel from among all eligible entries received based on the following
judging criteria:
Defender
Challenge
·
A submission will not be valid unless it achieves false negative
error rates less than 10% (“Maximum FN rate”) and false positive error rates
less than 1% (“Maximum FP rate”) using a set of malicious and benign samples
curated by Microsoft and Contest Parties (“Holdout Set”).
·
A submission will only be valid if the maximum query time as
measured by Microsoft and Contest Parties across all samples in the Holdout Set
does not exceed 5 seconds (“Maximum Query Time”). Furthermore, during the competition, any
query up to 5242880 bytes (5 MiB, “Maximum File Size”) to a defender model that
exceeds the Maximum Query Time will be reported to the requester as a negative
result.
·
Valid submissions are ranked according to the false negative rate
on samples submitted by contestants during the Attacker Challenge. The First Place Prize will be awarded to the
contestant or team whose solution achieved the lowest false negative rate,
while the Honorable Mention Prize will be awarded to the contestant or team
with the second lowest false negative rate.
·
In the event of a tie for false negative rate, the First Place Prize
will be awarded to the winning solution that has the lowest false positive rate
on samples from the Holdout Set, and the Honorable Mention Prize will be
awarded to the winning solution that has the second lowest false positive rate
on samples from the Holdout Set.
·
If necessary, subsequent tie-breaking will be based on last
submission time of the contestant or team, with the earliest final submission
time being awarded the prize. We reserve
the right to break ties when necessary based on finer
time precision than is specified on the Contest Website.
Attacker
Challenge – Anti-malware Evasion Track
·
Contestant submissions will earn a certain number of points. For each modified malware sample, contestants
will receive one point for each Anti-Malware model that it evades, so long as
the modified malware sample has identical behavior to the original malware
sample when run in the sandbox of the Contest Website.
·
No points will be awarded for samples which do not produce the
same behavioral indicators as an original sample in the malware sandbox.
Behavioral indicators are determined by Microsoft and Contest Parties.
·
Submissions are ranked according to total score. The First Place Prize will be awarded to the
highest score, and the Honorable Mention Prize will be awarded to the second
highest score.
·
In the event of a tie between participants, the number of model
API queries will be used to determine the winning solution, where the First
Place Prize is awarded to the contestant with fewer queries.
· &nbsnbsp;
If necessary, subsequent tie-breaking will be based on last
submission time of the contestant or team, with the earliest final submission
time being awarded the prize. We reserve
the right to break ties when necessary based on finer
time precision than is specified on the Contest Website.
·
The highest-ranking submission that (1) extends the Contest
automation software, and (2) uses the Contest automation software in their
solution, and (3) submits a GitHub Pull-Request as evidence that Contest
automation software was used shall receive a Bonus Prize.
Attacker
Challenge – Anti-phishing Evasion Track
·
Contestant submissions will earn a certain number of points. For each modified HTML phishing site, contestants
will receive one point for each Anti-Phishing model that it evades, so long as
the modified HTML input renders identically to the original phishing HTML
document.
·
No points will be awarded for HTML submissions which do not
produce the rendering. The official rendering engine used in the contest is to
be determined by Microsoft and Contest Parties.
·
Submissions are ranked according to total score. The First Place Prize will be awarded to the
highest score, and the Honorable Mention Prize will be awarded to the second
highest score.
·
In the event of a tie between participants, the number of model
API queries will be used to determine the winning solution, where the First
Place Prize is awarded to the contestant with fewer queries.
·
If necessary, subsequent tie-breaking will be based on the last
submission time of the contestant or team, with the earliest final submission
time being awarded the prize. We reserve
the right to break ties when necessary, based on finer
time precision than is specified on the Contest Website.
·
The highest-ranking submission that (1) extends the Contest
automation software, and (2) uses the Contest automation software in their
solution, and (3) submits a GitHub Pull-Request as evidence that Contest
automation software was used shall receive a Bonus Prize.
Winners will be determined within 20 business days following the
Entry Period of the Attacker Challenge. The decisions of the judges are final and
binding. If we do not receive a sufficient number of entries meeting the entry
requirements, we may, at our discretion, select fewer winners than the number
of Contest Prizes described below. If public vote determines winners, it is
prohibited for any person to obtain votes by any fraudulent or inappropriate
means, including offering prizes or other inducements in exchange for votes,
automated programs or fraudulent identification.
Microsoft will void any questionable votes.
Winners will be notified via the contact information provided
during entry no more than 7 days following judging with prize claim
instructions, including a request for a link to their published submission.
Valid publication outlets include a document provided to Microsoft (for
publication on a Microsoft website), a link to a publicly accessible internet
website, such as https://github.com/ (e.g., for code), https://arxiv.org/ (e.g., for a whitepaper), or a
blog post detailing the solution.
Failure to publish within 15 days of notification will result in
forfeiture of the prize.
If a selected winner cannot be
contacted, is ineligible, fails to claim a prize or fails to return any forms,
the selected winner will forfeit their prize and an alternate winner will be
selected time allowing. If you are a potential winner and you are 18 or older
but have not reached the age of majority in
your legal place of residence, we may require your parent/legal guardian
to sign all required forms on your behalf. Only three alternate winners will be selected, after which unclaimed
prizes will remain unawarded.
9.
PRIZES
Prize winners will be awarded the following prizes, which may be
delivered, at Contest Sponsor’s discretion, via credits of sufficient value via
the Microsoft Store:
Three
“First Place Prizes” for each of Defender Challenge, Attacker Challenge –
Anti-Malware Evasion Track and Attacker Challenge – Anti-Phishing Evasion
Track:
Each winner
will receive a Microsoft Gift Card valued at $1,800 USD .
Three
“Honorable Mention” Prizes for each of Defender Challenge, Attacker Challenge –
Anti-Malware Evasion Track and Attacker Challenge – Anti-Phishing Evasion
Track:
Each winner will receive a Microsoft Gift Card valued at $300 USD.
Two
“Bonus” Prizes for each Attacker Challenge – Anti-Malware Evasion Track and
Attacker Challenge – Anti-Phishing Evasion Track:
Each winner
will receive a Microsoft Gift Card valued at $300 USD.
Promotional
Gift Card Terms. Works at Microsoft Store on Xbox consoles
(Xbox Live required), Windows 10 PCs, and online. Once redeemed to your
account, the full code value will be applied and may be used for eligible
purchases (exclusions apply) made directly at select Microsoft digital stores.
Eligible purchases and prices vary by region, device, and over time. Geographic
limitations, country and balance restrictions, taxes, and internet connection
fees may apply. Paid subscriptions required for some content. Age restrictions
apply. Except as required by law, credit cannot be redeemed or exchanged for
cash and is not refundable. To read full terms and conditions (which may change
without notice), go to microsoft.com/cardterms. Void where prohibited or restricted by law. Cards and codes
issued by and ©/™/® Microsoft Corp, a Washington Corporation, and/or its
affiliates.
We will only award up to three (3) prize(s) per person during the
Contest Period. No more than the stated number of prizes will be awarded. No substitution, transfer, or assignment of prize
permitted, except that Microsoft reserves the right to substitute a prize of
equal or greater value in the event the offered prize is unavailable. Prizes
are awarded “AS IS” with no warranty of any kind, either express or implied,
including but not limited to, the implied warranties or merchantability,
fitness for a particular purpose, or non-infringement. Prizes will be sent no
later than 28 days after winner selection. Prize winners may be required to
complete and return prize claim and / or tax forms (“Forms”) within the
deadline stated in the winner notification. Taxes on the prize, if any, are the
sole responsibility of the winner, who is advised to seek independent counsel
regarding the tax implications of accepting a prize. By accepting a prize, you agree that
Microsoft may use your entry, name, image and hometown online and in print, or
in any other media, in connection with this Contest without additional payment
or compensation to you, except where prohibited by
law.
10.
ODDS
The odds of winning are based on the
number of eligible entries received.
11.
GENERAL
CONDITIONS AND RELEASE OF LIABILITY
using a sandbox
without internet connection or an operating system other than those compatible
with Microsoft Windows 10 for storing/manipulating portable executable (PE) files. The Contest Sponsor and Contest Parties shall
have no liability whatsoever for any claims, losses, actions, damages, suits,
or proceedings resulting from damage or loss caused by malware to your computer
or computer network, or damage or loss caused by malware to other computers or
computer networks owing to your distributing of original or modified malicious
software. To the extent allowed by law, by entering you agree to release and
hold harmless Microsoft and its respective parents, partners, subsidiaries,
affiliates, employees, and agents and Contest Parties from any and all
liability or any injury, loss, or damage of any kind arising in connection with
this Contest or any prize won.
All local
laws apply. The decisions of Microsoft are
final and binding.
We reserve the right to cancel, change, or suspend this Contest for any
reason, including cheating, technology failure, catastrophe, war, or any other
unforeseen or unexpected event that affects the integrity of this Contest,
whether human or mechanical. If the integrity of the Contest cannot be
restored, we may select winners from among all eligible entries received before
we had to cancel, change or suspend the Contest.
If you attempt or we have strong reason to believe that you have
compromised the integrity or the legitimate operation of this Contest by
cheating, hacking, creating a bot or other automated
program, or by committing fraud in any way, we may seek damages from you to the
full extent of the law and you may be banned from participation in future
Microsoft promotions.
12.
USE OF YOUR
ENTRY
Personal
data you provide while entering this Contest will be used by Microsoft and/or
its agents and prize fulfillers acting on Microsoft’s behalf only for the
administration and operation of this Contest and in accordance with the Microsoft Privacy Statement.
13.
GOVERNING LAW
This Contest will be governed by the laws of the State of
Washington, and you consent to the exclusive jurisdiction and venue of the
courts of the State of Washington for any disputes arising out of this
Contest.
14.
WINNERS LIST
Send an email to [email protected] with the subject
line “Machine Learning Security Evasion Competition Contest winners” within 30
days of September 17, 2021 to receive a list of
winners that received a prize worth $25.00 or more.