Formset & Systemset being impacted by Zendesk Field Display Bug

Incident Report for Cloudset

Postmortem

Zendesk confirmed the incident - Service Incident - October 5th, 2022

The identified cause of Zendesk Marketplace Support Apps to render incorrect ticket field values.

While Zendesk breakages can occur from time-to-time, Cloudset will continue to dialog with Zendesk about how to avoid such occurrences, and in particular, the process and timelines to alarm and fix such issues.

Posted Oct 06, 2022 - 07:54 UTC

Resolved

Zendesk have successfully rolled back the ZAF deploy. Formset and System set are now running again successfully.

Posted Oct 05, 2022 - 15:13 UTC

Update

Zendesk Engineering Support are currently scoping a roll-back and escalating. We'll update you again as this firms up.

Apologies for the inconvenience, we know our US customer base is starting to get hit by this.

Posted Oct 05, 2022 - 14:20 UTC

Identified

This issue has been identified by Cloudset and acknowledged by Zendesk to be an issue with the ZAF Hide Option Values.

There has also been wider community based reports of this issue - https://support.zendesk.com/hc/en-us/community/posts/5005346071066-Zendesk-App-get-ticketFields-options-and-optionValues-incorrect

A request has been made to roll-back any recent deployment to restore ZAF Hide Option Values, and in turn restore full operation of Formset and Systemset.

Posted Oct 05, 2022 - 13:35 UTC

Update

We are starting to narrow down the issue with the ZAF to be the inability to hide field option values using the ZAF API.

Posted Oct 05, 2022 - 09:08 UTC

Investigating

We have received numerous reports of partial failure of Formset and Systemset that can prevent field values, managed by Cloudset apps, being selected. In other cases, due to apps use of save hooks, ticket updates can't be performed.

We advise all Formset and Systemset customers who are experience issue to confirm by appending ?noapps to their ticket URLs. If this confirms the issue, we further recommend DISABLING Formset/Systemset to allow tickets to be processed.

We recognise that running without Cloudset apps impacts operational performance and quality of data.

We working with Zendesk team to correlate and determine a resolution.

Further updates will be posted on progress to resolve.

Posted Oct 05, 2022 - 08:43 UTC

This incident affected: Cloudset Apps Services (Cloudset Apps Web Services).