Building a Better Satisfaction Survey: Part 2 – Reporting and Measures

Welcome to the second part of my Satisfaction Survey rebuild. In the first section, we got under the hood and made some fundamental changes to the way the satisfaction survey handles the data it receives. We're now going to take a look at the kind of reporting that our previous modification makes possible. The goals for this article are to:

1. Examine the existing reporting capabilities
2. Introduce additional reporting made possible by modifying the way data is handled in Part 1
3. Introduce a methodology for controlling the survey sample size

The Out-of-the Box Report

While the information provided in the standard satisfaction report is certainly valuable, it is by no means a complete picture of the questions asked (and hopefully answered) in the survey. While providing a 50,000-foot view, the data presented isn't actionable at an individual level, and likely is not even usable as a Key Performance Indicator in most environments. The report is hampered by the limited amount of information available - a problem that no longer exists after the modifications made in the first part of this guide.

Figure 1: The out-of-the-box satisfaction report

Re-thinking the Satisfaction Survey Report(s)

In order for us to properly report on the results of our satisfaction surveys, we need to understand what they contain. The answers provided in these surveys speak to much more than simply an overall rating, they speak to individual components of an end-user's experience dealing with support.

Figure 2: Why can't they all look like this?

If all of our surveys came back looking like the one above, an overall rating would be all we needed to gain a clear insight into the experience we're providing our customers. More likely, however, there will be survey results that look more like the one below:

Figure 3: Why couldn't they all look like the other one?

Using More of the Data

For example, understanding how your users view the responsiveness of the support group can be very enlightening when compared to the response SLA or SLO scores. Did we consistently meet or exceed a response SLA, but the users report themselves as mostly unsatisfied? Did we consistently breach the SLA, but the users were mostly satisfied? The answers to questions like these can be taken into account when writing the appropriate SLA for next year.

As another example, if satisfaction scores are part of an individual employee's key performance measures, looking beyond simply the overall score can provide information that will help the development of said worker.

Consider the following scenario: During their employee evaluations, Nick Burns and Susie Queue both received an overall rating of 4 for last quarter's surveys. This is valuable information, but knowing that Nick had a 3 for "professionalism and courtesy" (move!) and Susie a 3 for "information provided" would allow for more targeted recommendations from their reviewer.

Figure 4: An example report leveraging the new tables (one of the questions has been changed to gather feedback against the resolution SLO)

Report Parameters

Below are the local parameters used for the example report above:

Figure 5: The %worker% local parameter

Figure 6: The %date1% local parameter

Figure 7: The %date2% local parameter

Report Query

Below is the query used for the example report above:

select COUNT(distinct v.[workitem_number]) AS 'Sample Size',
		avg(v1.[workitem_satisfaction_overall]) AS 'Overall Score',
		avg(v2.[workitem_satisfaction_responsiveness])AS 'Responsiveness',
		avg(v3.[workitem_satisfaction_expertise]) AS Expertise',
		avg(v4.[workitem_satisfaction_information]) AS 'Information',
		avg(v5.[workitem_satisfaction_professionalism]) AS 'Professionalism',
                         5 as 'Scale'
from (select [workitem_number] 
		from hd_workitem_current_view
		where workitem_auxdata like '%<name>custom_survey_complete</name>%<value>true</value>%'
			and [assigned_to_worker_full_name] LIKE %%worker%%
			and [workitem_created_on] >= %%date1%%
			and [workitem_created_on] < DATEADD(dd,1,%%date2%%)
) v,
	(select [workitem_satisfaction_overall] 
		from hd_workitem_current_view 
		where [workitem_satisfaction_overall] <> 0
			and [assigned_to_worker_full_name] LIKE %%worker%%
			and [workitem_created_on] >= %%date1%%
			and [workitem_created_on] < DATEADD(dd,1,%%date2%%)
) v1,
(select [workitem_satisfaction_responsiveness] 
		from hd_workitem_current_view 
		where [workitem_satisfaction_responsiveness] <> 0
			and [assigned_to_worker_full_name] LIKE %%worker%%
			and [workitem_created_on] >= %%date1%%
			and [workitem_created_on] < DATEADD(dd,1,%%date2%%)
) v2,
(select [workitem_satisfaction_expertise]
		from hd_workitem_current_view 
		where [workitem_satisfaction_expertise] <> 0
			and [assigned_to_worker_full_name] LIKE %%worker%%
			and [workitem_created_on] >= %%date1%%
			and [workitem_created_on] < DATEADD(dd,1,%%date2%%)
) v3,
(select [workitem_satisfaction_information] 
		from hd_workitem_current_view 
		where [workitem_satisfaction_information] <> 0
			and [assigned_to_worker_full_name] LIKE %%worker%%
			and [workitem_created_on] >= %%date1%%
			and [workitem_created_on] < DATEADD(dd,1,%%date2%%)
) v4,
(select [workitem_satisfaction_professionalism]
		from hd_workitem_current_view 
		where [workitem_satisfaction_professionalism] <> 0
			and [assigned_to_worker_full_name] LIKE %%worker%%
			and [workitem_created_on] >= %%date1%%
			and [workitem_created_on] < DATEADD(dd,1,%%date2%%)
) v5
HAVING avg(v1.[workitem_satisfaction_overall]) IS NOT NULL
		and avg(v2.[workitem_satisfaction_responsiveness])IS NOT NULL
		and avg(v3.[workitem_satisfaction_expertise]) IS NOT NULL
		and avg(v4.[workitem_satisfaction_information]) IS NOT NULL
		and avg(v5.[workitem_satisfaction_professionalism]) IS NOT NULL

Determining the Percentage of Surveys Returned

You may have noticed in part 1 when we were creating new fields in the workitem table, one of them was called "satisfaction_trigger." This table can be used to set and track which incidents send a survey. The built-in "Send Satisfaction Survey Randomly" notify rule performs a random number generation, then sends the survey if that number was below a certain threshold. Unfortunately, there is no good way to tell which incidents actually received a survey due to the nature of how the triggered rules are stored in the database.

As an alternative, consider leveraging this new "satisfaction_trigger" field and a series of incident rules instead. Upon ticket creation, the same randomization routine can be performed, with the result of setting "satisfaction_trigger" to '1.' During the course of the incident, other factors may come about requiring that number be set back to '0' so the survey is not sent. A notify rule can be configured to send the satisfaction e-mail anytime an incident closes having a "satisfaction_trigger" value not equal to '0.'

Figure 8: Setting the satisfaction trigger on 25% of incidents as they are opened

Figure 9: Leveraging the satisfaction trigger in a notify rule

Once this has been implemented, determining the percentage of surveys returned is as simple as dividing the number of surveys received by the number of incidents with a "satisfaction_trigger" value of '1' for a given time interval and multiplying by 100.

Understanding Your Sample Size

Obviously, we know how many surveys were completed for any given reporting period, but how many were sent? Not surprisingly, this information can matter on multiple levels. It gives us insight into how much we are "bugging" our users for answers, how often our users are participating in the survey, and the percentage differential. Knowing the percentage of surveys that are returned versus the total tickets allows us to calculate our confidence level and interval - the projected validity of the survey results plus or minus a percentage margin for error (+/- n%). For example, given 1000 incidents closed in a given time period we received 200 completed surveys. While there are varying calculations and I've rounded the number slightly, this should give us a 90% (+/- 5%) confidence level that the sample results were representative of the whole.

By controlling which incidents receive a satisfaction survey (we may not want to include administrative, time tracking, or various other internal incident types) and tracking how many reports are sent versus returned, the number of surveys sent can be adjusted as necessary to meet the target confidence interval. I am by no means a statistician, but more information on sample size and confidence intervals (including tools for calculating these things) can be readily found online.

Conclusion

Thanks for reading! This concludes my articles on modifying the satisfaction survey. I hope that it has provided some insight into the amount and quality of data that can be obtained with just a few small modifications to the satisfaction survey.