Inability to display some RollWorks Reporting data (RollWorks Only)
Incident Report for NextRoll
Resolved
We’re happy to report that Journey Stages has been taken out of maintenance.

Many thanks to the RollWorks Engineers who worked tirelessly to get this product back up and working for everybody!

This will be the last communication about this escalation.

Thank you.
Posted Aug 17, 2021 - 17:08 PDT
Update
As a follow-up to Friday’s email, we wanted to share an update on the Journey Stages outage, which is related to some underlying ClickHouse issues.

We have found and fixed the root cause for one issue which was due to a bug in ClickHouse itself. That was patched over the weekend, and investigation is continuing on the root cause of one other issue. We have made some infrastructure changes that we believe will better position us against future issues, and we are testing these changes in production now.

As a reminder, no data has been lost. Customer access in the production UI has been limited out of an abundance of caution while engineering finishes testing the back-end changes we have made.
Posted Aug 16, 2021 - 12:13 PDT
Update
Engineering is continuing to work towards a resolution on Journey Stages. While the issue continues to be ongoing the page will remain in maintenance mode.

The expected resolution for this issue is likely to be on or around this Wednesday, August 18, 2021. We will continue to update you as new information becomes available
Posted Aug 13, 2021 - 15:11 PDT
Update
The issues we’ve been experiencing with Journey Stages data specifically have unfortunately returned.

As a result the Journey Stages page has been put into maintenance mode so any user navigating to that page will see a "We are under maintenance" screen.

We will be following-up later today with more info regarding this maintenance.
Posted Aug 13, 2021 - 12:08 PDT
Update
This is just a quick update that June & May Journey Stages data is now available in the dashboard.

April's Journey Stages data backfill is in progress and we will provide an update once this has been complete.
Posted Aug 13, 2021 - 10:50 PDT
Update
In efforts to immediately allow users to load Journey Stages in the most recent days, data is being backfilled starting from the most recent months.

With that note, data for August & July is now available for Journey Stages. June will be shortly available, followed by May and so forth. For any date that is unavailable you will simply see a "?" and a warning "unable to get accounts".
Posted Aug 12, 2021 - 10:32 PDT
Identified
RollWorks reporting pages related to Playbooks, New Influenced Opportunities and Account Card features will now fully load.

The Journey Stages page is still affected and we are currently working on its resolution.
Posted Aug 11, 2021 - 06:26 PDT
Investigating
RollWorks reporting related to Playbooks, Journey Stages, New Influenced Opportunities and Account Card features are all affected. Pages that display this data will now only partially load (omitting the related reporting data).

We're working on a fix.
Posted Aug 10, 2021 - 19:04 PDT
This incident affected: RollWorks (RollWorks Impact Overview Dashboard, RollWorks Playbooks).