Synthesized UX Insights Improved AI-Generated Designs
In two tests, AI-generated designs based on UX insights and recommendations met more predefined criteria than designs based on raw comments. The average gap was 8.4 percentage points.
On November 20, 2026, Nielsen Norman Group published an article comparing interfaces generated by AI using two types of research context. In two tasks, designs based on insights and recommendations met more predefined criteria than designs based on raw user comments. The average gap was 8.4 percentage points.
How Comments and Recommendations Were Compared
The model was given two tasks: create a single-page form for rebooking a canceled flight and a multi-step checkout for an online store. The task brief and design-system information stayed the same within each task. What changed was the research context the model received.
In one condition, it received about 100 short user comments for each task. In the other, it received insights and recommendations drawn from those same comments. The researcher used an AI tool to help analyze the comments but decided which insights and recommendations to include. In the second condition, a step of human interpretation stood between the feedback and the generated interface.
Eight design variations were generated for each task and each type of context. A person and a language model evaluated them blind against a predefined list of criteria. The final score represented the share of criteria met; the published results use the human ratings. After calibration, the reviewers agreed on roughly 96% of decisions across the two tasks.
What Changed in the Designs
Insights and recommendations performed better in both tasks, and the difference was statistically significant: p < 0.001. The gap was 11.4 percentage points for the rebooking form and 5.5 for checkout. These figures measure performance against a specific list of criteria, not overall interface quality.
In the canceled-flight comments, several users mentioned loyalty-program miles. None of the designs generated directly from the comments explained whether the cancellation would affect a passenger’s miles. Every design generated from the insights and recommendations included that information.
For checkout, users complained that items disappeared from their carts. Fewer than half of the designs based on raw comments kept the item list and order total visible at every step. Almost all the designs based on insights and recommendations did. The gap was smaller here, possibly because the brief already identified some shopper needs: applying a discount code and receiving a gift within two days.
The comparison has limits. The datasets were small—about 100 short comments per task—and likely cleaner than a typical stream of feedback. Both the form and checkout are also common interface patterns. The authors suggest that preliminary analysis might be more useful for less familiar tasks with larger, more fragmented datasets, but that remains to be tested. It is also not yet clear how much of the improvement came from the insights and how much from the recommendations.
What This Means for Website Projects
If a team uses AI to produce designs, handing it a collection of feedback does not replace deciding which problems matter and how to address them. A comment may describe a user’s difficulty precisely without pointing to the right interface change. Different comments may contradict one another or have no bearing on the scenario at hand.
A practical step is to separate observations from insights, then turn those insights into recommendations for generating and evaluating a design. That means considering not only what users say but also product goals, business needs, and technical constraints. In this comparison, the researcher made the final selection of insights and recommendations, even though they used an AI tool to analyze the comments. The model therefore received guidance it lacked when working from individual remarks.
This is not a reason to discard the source data or skip reviewing the finished design. The finding is narrower: in the two tasks studied, insights and recommendations were more useful for generation than comments alone. The authors also plan to investigate whether the effect holds for interviews, usability-testing notes, and surveys, as well as for other models and ways of providing context. The need to review AI output in UX work also came up in an article about a survey of UX professionals.
My View: Insights Need to Make It Into the Specification
I wouldn’t give a model feedback without first deciding on priorities. It’s easy to overlook a detail such as showing the cart contents at every step of a design; that ambiguity becomes even more apparent when the design is translated into components. To me, this comparison makes the case for a specification that states what problem each element solves and which states it should appear in. Generation does not relieve the team of responsibility for those decisions.
Sources
Where the news comes from. The text is a retelling in the author’s own words; the facts come from the source, the opinion is the author’s.
- Can AI Generate Good Designs from Raw User Data? www.nngroup.com
Product designer
I work with digital products and design systems in Lisbon. I keep track of how design decisions hold up when they become real components.
All posts by the authorRelated articles
UX Survey: AI Speeds Up Work, but Its Outputs Still Need Checking
A Nielsen Norman Group survey finds that AI helps UX professionals complete many tasks faster. But its outputs need checking, and automation alone cannot speed up work with users.
AI Respondents Matched a UX Survey’s Main Finding but Gave Fewer Distinct Answers
AI respondents and people both named UX as the most common term for this kind of work. But people offered more than 50 alternatives, while synthetic participants gave about a dozen.
OpenAI’s API and Google Found Only Some of the Pages ChatGPT Cited
Google and the OpenAI Responses API found only about a quarter of the pages ChatGPT cited. Measuring visibility in AI answers means distinguishing between matching websites and matching URLs.
Discussion 5
Orkhan Latifov
For a small site project, the practical risk is that recommendations can sound settled while still hiding trade-offs. I’d want each one in the brief to include the evidence behind it, the affected page or flow, and how we’ll know it was addressed; otherwise the team may spend time implementing a polished interpretation that does not match what parents actually need.
Fedor Ustenko
@orxan.latif, I’d add a check against the original comments before treating a recommendation as settled. In the study, the insights and recommendations were distilled from those comments, so the improved designs show that synthesis can help the model meet predefined criteria—not that the synthesis captured every important need or trade-off. For the flight rebooking and checkout flows, I’d keep the source evidence traceable and test the resulting design with users, especially where a recommendation rules out an alternative.
Łukasz Górski
@fedir_ust, do you mean checking the recommendations against the comments before implementation, or validating them with users too? The study’s two tasks only measured predefined criteria, so those checks seem to answer different questions.
Inês Rocha Post author
Both: check the synthesis against the comments before implementation to catch omissions, then validate the design with users; the study’s predefined criteria measure neither whether the synthesis captured every need nor whether the solution works for users.
Beatrice Norkutė
I’m not convinced the 8.4-point gap shows that synthesized insights are inherently better input. The model may simply have been given a more directive brief: recommendations already narrow the space of plausible designs, while raw comments leave more interpretation to the model. That’s a useful finding about prompt structure, but it doesn’t tell us whether the resulting interfaces better serve users in practice. Were the criteria independently defined and were the designs tested with users, or was success limited to matching the predefined checklist?