01

Designing a near-future physical and digital product

An exploration of multimodal communication through an edible chocolate-based language. This concept was developed independently for my IxD master’s program thesis.

Research, UX, UI & Service Design

The overview

The brief

The research

The project development

The conclusions

“émoi” is a service and "data gustation" interface that allows users to communicate their feelings through an edible chocolate-based language using qualities of taste, texture, shape and colour.

The concept uses a variety of digital systems (ie. e-commerce, sentiment analysis algorithms) to reinvent the long-standing tradition of sharing and gifting chocolates as an expression of interpersonal feelings and exchanges.

This edible representation of the user’s emotions is created using a communication language based on research into multimodal systems and the human gustatory and oral somatosensory experience.

Role throughout project

This project was carried out independently.

• Design research - prototyping - UX/UI/interaction design - service design

• Content design - content production - content writing

• Communication - storytelling - copywriting

• Media production - motion design - visual design - audio production

The tools used were

• Primary, secondary and tertiary sourced research

• Literature review

• Competitor / state-of-the-art analysis

• User research

• Desk research

• Interviews

• Scenarios

• Personas

• Prototyping

• User flow

• System design

• Service blueprinting

• Storytelling

Awards

Core77 Design Awards 2016 "Student Notable" in Speculative Concept

Shortlisted for IxDAwards 2016

Events

Lift India 2016

Lift Conference 16

News

Times of India

Bangalore Mirror

20 minuti, Svizzera

The brief

The project was produced as part of my master’s thesis. The assigned brief was to define and answer a research question and hypothesis.

The general direction I chose to work on was to imagine a near future world / product / service.

The project’s point of departure was to explore senses other than the visual modality as a way of communicating and representing data. I relied on a breadth of existing research into multi-sensory communication to investigate the possibilities alternate mediums had to offer.

My findings lead me to tackling one of the lesser used mediums as an exploration in speculative design: taste. Taste is versatile and multimodal in itself too, combining elements of smell and haptics through its texture or “mouthfeel”.

Through researching the possibilities and applications for this concept, I eventually arrived at the final form of the project - ‘visualising’ human sentiment through chocolate as an edible medium.

Research question

Signals present in the natural world convey messages, changes in states, as well as illustrate alarm mechanisms. Can such features be used to implement multi-sensory modes of communication that go beyond the screen and standard notification systems?

Hypothesis

It is highly probable that data representation can be effectively communicated through “natural” signals since our bodies have evolved to interpret and understand these signals over millennia.

The first step in answering my research question and defining my project direction involved diving into the existing literature on the topic of multi-modal forms of communication.

Our modern world tends to default on communicating to us visually. But research shows that multi-modal methods of “display” are not only more effective, but that we humans are in fact naturally better equipped at interacting with our world multi-modally.

The studies referenced also revealed that there truly is a capacity to create languages through diverse multisensory systems. And so a future where we can receive and understand digital data languages using a diverse set of sensory receptors may indeed be a conceivable near future.

GUIs fall short of embracing the richness of human senses and skills people have developed through a lifetime of interaction with the physical world.

- Hiroshi Ishii, MIT professor and Tangible UI pioneer

“Only humans invent and design their own communication environment [and] it is only recently that we have been creating wholly invented contexts for our communication.”
-Judith Donath

In her 2007 publication in the MIT Press entitled “Signals, cues and meaning,” Judith Donath uses signalling theory to discuss how new technologies shape communication and the social world. This theory of evolutionary biology states that, at its very core, our living planet and its inhabitants rely on an eclectic range of explicit and implicit signals that inform and shape our world and its continuous evolution. This intrinsic communication takes place in the form of expressions between persons, mating calls of animals in their habitat, or even trees shedding their auburn leaves to bloom with flowers again the following season.

With the most recent technological explosion, we’ve created a multitude of new digital environments, from mobile computing, to online video chat, to video games and multiplayer role-playing games, each with their own particular toolbox of signals and conventions.

Our digital products are still in the infancy of their potential - unable to take into account considerations like context or medium to make communicating with us and our other devices as natural as nature itself.

As early as the 90s, barely at the cusp of the internet age, the team at Xerox PARC were discussing the limitations and possibilities a networked world would be bringing us. Even back then, there was already talk about how our display systems are too divorced from one another and from the environment and spaces in which they existed.

Mark Weiser, one key figure in that Xerox PARC team and a man who is considered to be the father of ubiquitous computing, described this in his seminal publication “The Computer of the 21st Century.” He states that “the most profound technologies are those that disappear. They weave themselves into the fabric of everyday life until they are indistinguishable from it.” Just as with driving, the more we get used to understanding the processes and signals involved, the more the tasks become second nature and disappear into the periphery. Weiser considered this concept to be the key to realizing the seamless integration of “ubiquitous computing”.

He likened the state of computing at the time of publication to how writing was once a “new technology” available only to a select number of scribes. Whereas “today this technology is ubiquitous in industrialized countries. Not only do books, magazines and newspapers convey written information, but so do street signs, billboards, shop signs and even graffiti. […] The constant background presence of these products of "literacy technology" does not require active attention, but the information to be conveyed is ready for use at a glance.”

Though we have come a long way from the time of his paper’s publication, the merging of our devices with our environments has not yet reached that state of indistinguishable seamlessness he describes. Yes, the world is more connected than ever, but there is still a distinct lack of “calm,” unobtrusive integration between our devices and spaces that allows things to “disappear” into the background. Yes, the world is more digital than ever, but it seems each one of our devices is still awkwardly screaming for our attention like a child that hasn’t yet learned how to speak.

Today’s GUI systems do not take advantage the diversity of human sensory capabilities, a set of skills that has evolved through eons of interaction to best engage and interact with the natural world that surrounds us. This is the crux of what MIT’s Hiroshi Ishii and many of today’s leading thinkers attest to.

Mark Weiser also noted that “[…] although ubiquitous computers may employ sound and video in addition to text and graphics, that does not make them "multimedia computers." Today's multimedia machine makes the computer screen into a demanding focus of attention rather than allowing it to fade into the background”. “Even the most powerful notebook computer, with access to a worldwide information network, still focuses attention on a single box,” he said.

Indeed, the screen-only interface system is a transitional one at best. As interface designer and future of technology researcher Bret Victor states in his “Brief Rant on the Future of Interaction Design”, suggesting that GUIs on screens are the technology of the future “is like claiming that black-and-white is the future of photography. It's obviously a transitional technology. And the sooner we transition, the better”.

This is why I chose multisensory display systems as the topic for the future digital communication method I would explore - a place where colours, textures, tastes and sensations are coded with information to form a language. Just as today’s billboards and signs now carry “codes” that we collectively understand.

Tangible interfaces can be intuitively coded to create a multisensory language. For example, a researcher exploring haptic visualization explores using nominal (qualitative) and ordinal (quantitative) values to communicate and access data. She discusses how shapes, patterns and orientations can be effectively used to define ‘tangible IDs’ to assign names to groups or types of data, and how all other non-shape related variables can be used to express ordinality (order). Examples of this include the use of resistance on a door to portray the “accessibility” level a person is given when entering a space, or levels of vibration that convey seismic activity levels of an earthquake.

The human haptic system can process a complex variety of tactile variables that can be cutaneous (touch) or kinaesthetic (body movement). Cutaneous senses include mechanoreceptors that can interpret changes in patterns and pressure as well as differences in vibration types like flutter, buzzing, or thermoreceptors that can recognize variations in temperature. These cutaneous elements can also be identified visually using variables like size, shape, texture/grain, orientation, and elevation. While kinaesthetic variables can produce sensations that include resistance, friction, and kinaesthetic location.

The future of our experiences may utilize our body’s “input sources” and our brains powerful processing capability to decipher new communication messages from our surroundings. This scenario’s plausibility is supported by recent breakthroughs in research areas such as “sensory substitution” where participants with auditory and vision impairments can regain perception by learning sensory languages communicated through haptic, auditory and impulse devices transmitted through unconventional parts of their body.

There are, for example, already deaf individuals who can decipher “sound messages” using a vibration language through a custom-made vest, blind persons who can “see“ with their tongue and a monochromatic man who can’t see any colour at all but hear them instead.

To explore how multisensory communication can be applied to a cross-modal language I referred to work of researcher Charles Spence of Oxford University into something he called the “Bouba/Kiki Effect”. The “Bouba/Kiki Effect” is a phenomenon where people were shown to consistently associate specific letter pronunciation sounds with specific shapes. The basic idea behind the “Bouba/Kiki Effect” is that people generally associate vowels like those present in the word “Bouba”, which create a round-shaped mouth and widen the vocal tract in pronunciation (a, o, u), with round amoeba-like shapes. While vowels like those in the word “Kiki” that stretch our lips out horizontally and narrow our vocal tract (i, e) are associated with sharp, polygon-like shapes.

Most importantly, the“Bouba/Kiki Effect” has been shown to take place in the taste modality too. For example, “Bouba” has been commonly associated with sweet flavours, while “Kiki” was more linked to bitter tastes. Apparently, people had also consistently identified mint chocolate as being more "kiki" than regular chocolate”.

This particular piece of research was applied in “Kitchen Theory,” a multimodal experiment in cuisine, where Spence, along with Chef Jozef Youssef, created presentations that applied “bouba/kiki” to food. They experimented with flavour, colour, shape, texture as well as sound. The chef’s creation is based on a study of participants from China, India, Malaysia, and the USA that found that participants consistently associated colour and flavour values. Black was associated with bitter, green with sour, pink with sweet, and white with salty.

Back to section top

The next step in defining my project direction involved selecting an appropriate medium based on the previous research investigation findings.

I chose to focus on the gustatory sense in particular because it is uniquely multisensory. While in the mouth, it engages taste as well as more “haptic” sensations - also called our somatosensory system. This includes variables like texture and temperature. At the same time food lies in the tangible world so the classic haptic variables like moisture, texture, and temperature can be engaged using the hands too. And of course, food also inherently carries olfactory attributes - so you can perceive it through smell. All these elements could work together to create a complex multidimensional experience produced through a gustatory display.

Chocolate takes full advantage of the gustatory experience’s multimodal capacity. Unlike other foods with either subtle or very specific flavours, chocolate comes in many flavours, texture, and colours that can all further lend themselves to a multimodal visualization of the data. 

Chocolate is also easily accessible and loved by people in general. People are familiar with what it should taste like and it is eaten by the majority of the population. And though it is used to celebrate and commemorate occasions, its context transcends any one particular situation; it can be served with a meal, with a coffee or just as a snack, at home, in the office or at an event. It also does not impose a certain time for its serving and can be eaten multiple times a day. 

As a medium, chocolate is also inherently easy to manipulate compared to other foods which may require complicated and extensive cooking processes. Chocolate can also be easily mixed with a variety of ingredients. It is also relatively easy to transport and to store.

Finally, I thought it would be very interesting to address the long-standing, long known relationship between food and mood. Our emotions have a unique connection with our food experience. Whether it be how customs and traditions seem to always be centred around food, or our eating habits relating to psychology (think self soothing and comfort food). And chocolate has historically been a key player in this relationship.

Back to section top

The following step in researching my project direction was the ideation and iteration phase. Using a survey and a series of user interviews I was able to pinpoint which areas of my initial ideas worked and which didn’t work. This was key to developing and improving upon the concept. And through more user and industry expert interviews I was able to further refine the final project concept.

Concept: Team cognition

The initial concept’s main idea was to use multimodal visualization through the chocolate to tackle “team cognition” and promote positive team dynamics in the workplace. The chocolates would represent:

1. The internal emotion: the collective sentiment of the members of a team

2. Team performance: the effectiveness of the team’s performance

3. The external emotion: the collective sentiment of audience, customers or followers

The system would eventually aggregate enough data to be able to have some predictive capacities and a general ability at measuring the “health” of the team and their output capacity.

Syntax: The Dimensional Approach

I researched an emotion representation language to serve as the basis of the communication syntax I could use for the concept. In this iteration, the system would use an emotion to flavour language based on Wilhelm Wundt’s (1905) “dimensional approach” which divides the emotional spectrum into three basic dimensions of valence, arousal and tension to cover the diverse range of experienced emotions. This would be used in conjunction with the aforementioned “bouba/kiki effect”.

Concept: Edible news reactions

The second concept’s aim was to create a simple interface that could teach users about the multisensory visualization based the gustatory sense. The user would input a news topic by scanning it from their phone or a newspaper and an algorithm would mine online comments related to the topic to calculate the audience’s dominant sentiment towards the topic to produce an appropriate chocolate combination.

Syntax: The Geneva Emotion Wheel

For this concept I explored a different emotion representation language that seemed more appropriate for the context. Here I referenced an emotion to flavour language based on Klaus R. Scherer’s “Geneva Emotion Wheel” which gets more specific than the previously mentioned dimensional approach. Again, this would be used in conjunction with the “bouba/kiki effect” concepts.

• Interviewees found the “Team Cognition” concept difficult to understand.

• They found that the “Edible News Reactions” concept didn’t fit the medium - what does news and the general public’s “emotion” have to do with chocolate?

• They found that chocolate is more likely associated with personal, one-to-one emotional experiences that occur in more direct and intentional interactions between people.

• Interviewees and prospective users described how they found the idea of communicating one’s emotion to others through chocolate intriguing.

“It would be so intriguing for people to discover emotions of bosses or peers through the shape and taste of a chocolate”

- Interview participant

Industry expert interviews

I interviewed 4 chocolate artisans and 1 hotelier consultant in the local industry to gain valuable insights into their behaviours and needs and those of their customers.

• Chocolate has an ability to invoke experiences of deep emotions in people.

• Artisanal chocolates in particular are given as presents in special occasions, holidays and celebrations, to express affection, greetings, or condolences.

• Chocolate artisans separate themselves from mainstream products through ingredient quality, creation creativity and uniqueness, and an ability to customize and personalize chocolate design and flavours.

Tristan has been running his artisanal chocolate shop for 17 years, and he is passionate about his art. He says that the art of chocolate is an emotional experience for both him and his clients. He particularly loves experimenting with the diverse possibilities available in creating chocolate.

His creations of include dark chocolate, milk chocolate, and white chocolate. But he also explores intriguing and exciting combinations created after multiple experiments and trials and errors with a diverse variety of ingredient.

He’s created chocolates with sea salt, peppers, chillis, and sesame, ginger, green tea, mint, and many more - which he keeps stocked among many other ingredients in the back of his shop for an endless choice of variety.

Concept: Émoi, a gifting service that uses flavour and mouthfeel to communicate feelings and messages

I developed this concept iteration to take into account my research and interviews findings. Since artisanal chocolate’s natural context and function is in the form of a gift offered personally in special occasions, It would make most sense for a visualization of emotions through chocolate to be part of this experience too. By displaying the emotional state of its ‘user’ in the form of chocolate, an emotionally expressive, personalised and unique experience is created for both the sender and the recipient.