Serendipity Systems: Architecting Personalization Systems at Scale

Think of the last innovative enterprise you interacted with. Can you think of one where some form of recommendation or personalization was not part of the experience? Recommendations are no longer product features, service attributes, gimmicks, or nice-to-haves. They are the central organizing design principle of modern experiences. Personalization wasn’t an add-on for Amazon, Uber, Netflix, or Airbnb. It was core to their experiences because personalization was core to their business model. We’re moving to a world where not doing personalization is a recipe for guaranteeing underperformance and obsolescence. So, how do we do this well?

Serendipitous influence

Architecting modern personalization systems isn’t just about data and algorithms. Recommendation architectures are those that persuade and influence our choices. The choices they present and how they are presented change not only how we discover products and experiences but how they promote internal self-reflection. Done well, recommendation and personalization are systems of “serendipitous influence.” The best recommendations are those that inspire you in a way you didn’t expect. They make you wish you’d gotten that recommendation earlier. Fostering serendipitous delight consistently becomes the hallmark for enterprises that want to provide delightful experiences. In our experiences in building such systems, we find there are four key dimensions that help firms elevate from personalization to serendipity:

Data

1. A consolidated view of “customers” and the enterprise is critical for consistent experiences. Data silos are likely to result in incongruent experiences as customers traverse channels and inconsistent recommendations between content, products, and services.
2. The proliferation of data across applications, devices, and brands makes it critical that data is connected, otherwise the customer is more likely to ask “Why don’t you know who I am?”
3. Resolution of unknown to known users is also critical in maximizing the total market of customers that you can personalize for. Amazon’s internal data lake (Project Andes) was the start of creating a producer consumer “data mesh” to democratize its insights on customers. This enables Amazon teams to use data across the entire ecosystem to recognize customers across channels and devices (e.g., from the web to mobile, to Prime Video, and to Alexa devices).

Intelligence

1. The best personalization systems are built from strong data-powered cultures. Literacy in data science, AI, and ML is necessary, but not sufficient. Teams need to be literate in utilizing insights to drive action.
2. Linking analytics to outcomes and a holistic set of KPIs is needed to monitor the health of recommendation systems and understand their impact (intentional or otherwise). YouTube found optimizing on clicks created click-bait recommendations that resulted in poor experiences and less user engagement versus optimizing for time spent watching videos.
3. Rules-based strategies help to “fake it till you make it” but they struggle at scale and often underperform due to biases in rules and an inability to iterate quickly. Personalization systems should allow employees to build campaigns to optimize based on business goals as opposed to rules (e.g., maximize revenue, minimize my carbon footprint through fewer shipments). This allows employees to be more strategic and creative while permitting underlying analytical models to make decisions on data volumes and patterns that would otherwise go unnoticed.

Amazon.com is a different store for every customer. Its Personalization Platform (P13N) allows business teams to set strategies and filters to leverage personalization algorithms optimized against business outcomes (e.g., improve conversion, drive engagement). The system also understands when signals like customer intent are changing. If I searched for shoes in the morning but now, I’m searching for toothpaste, my recommendations should understand my intent and needs have shifted, and consumer goods and staples are more relevant than athleisure products. These signals are optimized against business or experience KPIs which allows for dynamic recommendations to improve business outcomes while boosting the customer experience.

Design

1. Experimentation leads to better personalization. Design is key to better experimentation. The best recommendation cultures view experimentation of systems as an end-to-end exercise from data and analytics to UI/UX development to creative development and qualitative research.
2. Recommendation Experience Design (RXD) as a competency is integral to designing systems that create the proper nudges and in understanding the intended and unintended consequences of personalization. Done well, these teams are a blend of technologists, behavioral scientists, creatives, and experience designers.
3. Volume of experimentation can be a valuable KPI unto itself.
   “If you double the number of experiments you do per year, you’re going to double your inventiveness.” – Jeff Bezos.
4. Without trust in the system, it’s hard to get adoption. Without adoption the system will not survive. There’s a fine line between helping to make someone aware of a system and navigating their choices and manipulating them. Systems that manipulate are unlikely to survive.

Through user experience research, Stitch Fix found that people couldn’t judge what they would like from clothing images alone. Customers might say from an image they don’t like something, but when they interacted with it or put it on, they would find they loved it. This gave encouragement to be more aggressive in sending products that algorithms suggested a customer would like even if the customer stated they weren’t interested in those types of products.

Orchestration

1. Orchestrating intelligence and actions for interventions across the customer journey is essential to consistent experiences. Enterprises need a view of the critical journeys (if not all journeys), and this requires organizations to easily orchestrate data and analytics internally and externally to empower employees to improve customer experiences.
2. Employee experiences can be as critical as customer experiences. If employee experience is an afterthought (and poorly done) then adoption of the system will be low and the corresponding pace of experimentation will suffer (if it occurs at all).

Stitch Fix’s orchestration of customer and merchandising data enabled the development of Hybrid Designs, its internal AI-driven design group. The apparel designs are a true collaboration built on the orchestration of human intelligence and artificial intelligence that helped Stitch Fix generate 2021 revenue of $2.1 billion, with over four million active users

Recommendation systems that are thoughtfully architected across these four dimensions drive differentiated and innovative experiences through more experimentation and greater degrees of adoption within the enterprise and by customers. This results in systems that evolve from a goal that’s transactional – “Will I buy this?” – to those that make us wonder “How did I live without this?” What does the future hold? The expectation from consumers is some form of personalization. Moving forward, discovery should be like talking with a friend who knows you so well that they can anticipate your needs. This is a world where we’ve democratized access to serendipity, and firms should provide such experiences or be left behind. It is either serendipity at scale or obsolescence with haste.

INNOVATION TAKEAWAYS

Personalization is key
Personalization is no longer a nice to have, it’s critical for maintaining a competitive advantage.

Serendipity tops it
The best recommendations are those that inspire and delight you in a way you didn’t expect.

Four dimensions
Data, intelligence, design, and orchestration are the key dimensions for architecting innovative, serendipitous personalization systems.

Cover it all
End-to-end experimentation is critical in architecting and designing personalization systems that actually make an impact on consumer and employee experiences.

Interesting read?

Capgemini’s Innovation publication, Data-powered Innovation Review | Wave 4 features 18 such articles crafted by leading Capgemini and partner experts sharing inspiring examples of it – ranging from digital twins in the industrial metaverse, “humble” AI, serendipity in user experiences, all the way up to permacomputing and the battle against data waste.. In addition, several articles are in collaboration with key technology partners such as  Alation, Cognite, Toucan Toco, DataRobot, and The Open Group to reimagine what’s possible.