Important Notice: Public User Email Address Disclosure
Please be aware of phishing attempts. Emails from BitMEX are sent from “[email protected]” and “[email protected]”. BitMEX will never ask you to transfer any funds, nor ask for your password.
Hello,
On Friday, November 1 at 06:00 UTC, many of our users received an email which contained the email addresses of other users in the To: field. This was a general email update to our users about
upcoming changes to the weighting of our indices.
As a result, many BitMEX user email addresses, including a large number of inactive addresses, were disclosed to other users in small batches. No other information was disclosed.
We apologise for the concern this caused. This email will provide you with information about what happened and how we can assist you.
What happened?
BitMEX is a global business that sends emails to many different email providers. Email deliverability itself is a multi-layered problem, involving decades of work in building sender reputation systems and automatic spam filters. Unfortunately, this makes the job of large services such as BitMEX difficult at times: we only send mass emails to all users at rare events. We intend to keep a high signal-to-noise ratio, and only send email when absolutely necessary.
The
index change we published on 1 Nov was of sufficient importance - it will impact pricing of all of our products - that we felt it necessary to inform all BitMEX users about it. However, bulk mail sends such as this are difficult to orchestrate correctly on a global scale, to all recipients. Some mail servers, especially the global arms of large brands like Yahoo and 163, have very tight controls that are often triggered when we send large amounts of mail. For system notifications such as withdrawals, password resets, and liquidations, it is imperative that the customer receives mail dependably.
To remedy this, we built an in-house system to handle the necessary rendering, translation, staging, and piecemeal (as not to trigger rate limits) sending of important email. BitMEX has not sent an email to every customer at once
since 2017, and much has changed since then. When we initiated the send, it became clear that it would take upwards of 10 hours to complete, and there was a desire on the team to ensure users received the same material information on a more reasonable timescale.
To handle this, the tool was quickly rewritten to send single SendGrid API calls in batches of 1,000 addresses. Unfortunately, due to the time constraints, this was not put through our normal QA process. It was not immediately understood that the API call would create a literal concatenated To: field, leaking customer email addresses. As soon as we became aware, we immediately prevented further emails from being sent and have addressed the root cause.
BitMEX is a company that takes engineering seriously, and we are disappointed that this lapse in care has resulted in unwanted disclosure for our customers. We believe that processes, not engineers, are to blame for these failures. Our processes failed here, and we are working around-the-clock to revamp them and to ensure that even the simplest-looking code changes are put under strict review.
Additionally, and unrelated to this action, the BitMEX Twitter account was accessed by an external individual. The account was back under BitMEX control within 6 minutes and resecured, and the event is under security review.
Beyond email addresses, no personal or account information has been disclosed. At no point were any of our core systems at risk.
Who was affected?
Most BitMEX users were affected by this action. You can self-diagnose your exposure with the following steps:
- If you received an email about the index change, and your email was the only one listed in the To: field, you were not affected.
- If you received the index change email, and you saw multiple addresses in the To: field, you were affected.
- If you did not receive an index change email, but you received this email, it is best to assume that you were affected. While the system was cut-off before it completed entirely, many recipients began marking BitMEX emails as spam, understandably out of hope that it would stop further emails. This caused deliverability issues at some hosts, causing mail not to be delivered. Unfortunately, someone else in your batch may have received the email, exposing your email address.
- The deliverability issues caused by the spam reporting caused some follow-up password resets to be delayed for several hours. Our operation teams remedied this by 06:00 UTC on Nov 2.