Admin Permissions Lost
Incident Report for Neon Fundraise
Postmortem

We would like to share more details with our customers about the events that occurred with the release of New Admin v1 last week. There were two issues introduced with the upgrade:

  • Multiple admin notifications were sent out for the same activity;
  • Some tabs in the admin panel were hidden.
It is important to note that neither of these issues affected donors, fundraisers, or visitors to your sites. We apologize for any confusion this update caused.

What happened
After the release, we noticed some administrators were receiving multiple notifications for the same activity. Upon further investigation, we discovered that every time a notification was sent, the recipient’s email was re-added to the cached queue for that same notification. This resulted in the system compounding the number of emails that certain administrators received — especially for campaigns with high activity in the last week. We didn’t catch this issue during our QA testing due to the fact that our QA cache is constantly being refreshed.

Additionally, some customers reported that certain tabs in their admin panel were missing. Others reported that they were not seeing all of their campaigns in the New Admin interface. Both of these issues were related to a change in the way admin tabs and roles are managed in the New Admin.

How we resolved it
We released minor fixes on Wednesday, March 15, and again on Thursday, March 16. However, because the more serious issues listed above required a server release, we released those fixes on Saturday, March 18.

After the notification queue was fixed, emails were no longer compounded in the system. While this was a straightforward fix, it took some time to catch the bug in the production environment. At this time, this issue is resolved, and no administrators should be experiencing this issue.

The admin panel tabs issue was fixed by making changes to the new administrator roles. Now that this fix has been released, our revamped Admin Roles structure is working correctly in New Admin v1, which adds more flexibility and options for organizations when managing administrators.

Summary
We realize that the admin interface is an important part in managing your daily fundraising. We would like to apologize again for the impact these issues had to your fundraising operations. New Admin v1 is a major update and we thank our customers for the tremendous support we have received throughout, and after, the upgrade. We have been receiving a great deal of positive feedback on the updated design and functionality. We will continue to strive to serve your needs and improve our products.

Posted Mar 20, 2017 - 15:10 PDT

Resolved
This incident has been resolved.
Posted Mar 16, 2017 - 10:37 PDT
Monitoring
Admin Users who were missing some tabs will now see them. If you are still having trouble please be sure to let us know at http://support.rallybound.com
Thanks again for your patience.
Posted Mar 16, 2017 - 10:14 PDT
Identified
We've identified one glitch and we're issuing a patch. Admin Users who were missing some tabs will see them after this emergency release.
Thanks for your patience.
Posted Mar 16, 2017 - 10:04 PDT
Investigating
We've received some reports related to our admin update that some admin permissions have been lost. We are looking into this and will update here as soon as we find out more.
If you do submit a ticket to support (http://support.rallybound.com), please do include a screenshot.
Thank you.
Posted Mar 16, 2017 - 07:37 PDT