Brief incident in November 25th
Incident Report for Boost AI Search & Discovery
Postmortem

What happened and what we have done to solve it

On November 25th, at around 4 AM EST, while monitoring the system, we detected some unknown suspicious stress tests, which showed many requests coming from one IP. Therefore, we decided to react immediately by implementing a Rate Limit by IP to prevent this kind of situation. The process caused a short downtime.

Our engineers figured them out immediately and fixed them. The system was back to normal within some minutes.

Lesson learned

We do understand the importance of this period to your business and we are trying our best to maintain the best performance of the system. Please rest assured that we keep monitoring the system 24/7 to make sure you will have a successful sales season.

Besides, we have applied checklists for firewall and rate limit deployment to avoid downtime during deployment.

Posted Nov 30, 2022 - 09:19 UTC

Resolved
Brief downtime during deployment of Rate Limit by IP feature to prevent unknown suspicious stress tests.
Posted Nov 25, 2022 - 04:00 UTC