MSAT/OSAT Product & User Satisfaction
Principal Researcher I Insurance, Banking & Finance I North America
PROJECT OVERVIEW
A members-only insurance and banking institution relied heavily on Overall and Member Satisfaction Surveys (OSAT/MSAT) to track internal performance, benchmark, and set new goals focused on improving the member experience.
​
Despite the organization ranked within the top percentile comparative to peer groups for member satisfaction, a new executive team issued a comprehensive audit and review of all OSAT/MSAT surveys used across five key business units: Auto, Home, Bank, Property & Casualty, and Card. The goals were to a) identify areas for improvement, removal, or additions to the surveys, and b) ensure surveys were effectively aligned with each business unit’s objectives (e.g., "fit for purpose"), along with the organization overall.
​
Through collaboration with the company's internal audit teams, I streamlined existing surveys, reduced the total number, and refined language, structure, and question type to boost user feedback, increase engagement, and improve response rates.
1 study
4 months
30+ participants
Topic Areas
Bank
Card
Life
Auto
Home
Business Objectives
Key goals of the organization
With a new CEO and executive leadership team, the organization wanted to set new internal benchmarks for measuring organizational and member satisfaction across all business units, service areas, and product lines.
New leadership wanted the organization to land within the top 1% of market segment customer and organizational satisfaction. This meant not only devising surveys to measure user satisfaction (which already existed), but reviewing those methods (e.g., actual surveys) to assess fit-for-purpose, structure, and overall effectiveness.
An additional request of executive leadership was to assess existing surveys currently used by competitor peer sets and other market/industry leaders - examining similarities, gaps, and variabilities between MSAT and OSATs used in house and those used across markets.
TEAMS & COLLABORATION
Audit Director
Principal UX Researcher
Audit Associate I
Audit Associate II
Business Unit Leads
Tools & Software
Tools used on this project
Research Approach
With MSAT and OSAT scores already clocking in the 90th percentile, executive leadership wanted to "add some distance" between themselves and the nearest competitor(s) by further refining, expanding, and improving their existing customer satisfaction surveys - making sure they stay ahead. My research approach was a combination of examining existing survey efficacy and appropriateness, analyzing competitor and peer set groups, and collaborating with audit and business unit leads to refine, improve, and distribute a more effective user measurement tool.
All business.
KICK OFF
My first priority was to partner with the audit director to inform each business unit lead of the intended project goal; assessing current surveys used within each BU; their structure, intent, purpose, and if they currently align with organizational goals and objectives set by the CEO.
SURVEY INVENTORY & AUDIT
Next, was a deep dive into current MSAT and OSAT surveys currently used across each BU.
MARKET RESEARCH
Following inventory and audit, a deep dive into market and industry competitors helped identify similarities and differences between in-house MSAT/OSAT surveys and competitor surveys. Similarities and differences were earmarked and passed to audit team leads.
AUDIT TEAM ALIGNMENT
Finally, survey analysis and results were to be shared with audit team leads. Audit director and staff would collaborate with business unit leads to review data, highlight areas of misalignment, make adjustments, and distribute surveys during quarterly distribution schedules.
Each survey was analyzed on four key metrics:
Survey structure (length, question type, etc.)
​
Survey intent and goal
​
Alignment with current business goals and objectives
​
Audience appropriateness
Weeks 1 - 2
Months 1 - 2
Month 4
Months 2 - 3
Methodologies
The research techniques used over the course of the project
Internal Survey Research
20+ internal MSAT/OSAT surveys currently deployed across all business units across the organization were analyzed for alignment to current goals and objectives, efficacy, structure, and fit-for-purpose.
Market & Industry Research
Market research and industry analysis was performed through competitor and peer set reports supplied through Boston Consulting Group. This list served as a springboard for analyses.
Data & Findings
Data captured, insights produced, and what was delivered
Becoming the 1%; A matter of finding the right balance.
Data gathered from market research was segmented based on business unit and survey type: auto, home, bank, life, card.
Markey Surveys
Across each business unit comparative analyses was performed highlighting similarities and differences between the organization's surveys and those of competitor peer groups.
In-House Surveys
Overlapping similarities and major differences in survey intent, question type, structure, length, and format were "bucketed" across each BU - kept in tallied lists for all surveys.
This list was used to aid internal audit teams when assessing survey fit-for-purpose and whether surveys were presently meeting organizational goals and objectives.
Key Insights & Takeaways
FIT FOR PURPOSE
Some surveys were identified as being "recycled" over multiple years despite organizational goals shifting. Surveys were strongly recommended to be re-audited, reviewed, and modified on a yearly to semi-yearly cadence to avoid misalignment between survey intent and company goals.
OPTIONS MATTER
Question type and answer options matter. Additional forced choice response options were suggested to be able to better quantify responses. Whereas open ended/text box options were viewed as "discretionary" (e.g., I can skip it, so I will) by customers when quantitative data needed to be supplemented.
LENGTH
Survey length was significantly reduced across multiple product lines, with and average reduction of 21%. Survey length reduction was recommended to avoid survey fatigue and increase the likelihood of members actually completing all survey questions.
FREQUENCY
Some surveys (particularly auto and home), were sent multiple times throughout the year to the same member. Research uncovered members sent multiple surveys throughout the year generally do not complete them (e.g., response rates decease as surveys sent increase).
Deliverables
Executive leadership and business unit leads received:
Analysis of 60 competitor surveys
​​​
20+ individual question recommendations
​​​
1 market analysis & peer competitor report​
​
1 executive summary with additional areas of exploration for future phases