Where is UI design headed?

I had my first encounter with generative AI in 2016, with the Deep Dream Generator created by Alexander Mordvintsev. It’s an image generation tool that is built using Convolutional Neural Network (CNNs) that are useful for image analysis and pattern recognition. I fed some images to the generator, and it was supposed to create something entirely new from the visual data. It processed the forms and figures in the images I had uploaded and created a collection of blurry ambiguous compositions. There was a button to create further variations of the compositions until you found what you liked. I could barely understand what those forms meant and could only faintly discern what it appeared to be. It looked like the machine was grappling with the task of understanding all the input it received, as if it was struggling to make sense of it all. Generative AI was still very primitive back then. It worked well with images of nature, though it struggled with human figures. I found it fascinating nonetheless.

Here are some of the images I created.

I was curious to know more. For my next task, I decided to take a deep dive into understanding how neural networks worked. During my research on image processing algorithms, I came across technologies like Computer Vision and subsequently, Extended Reality (XR). I discovered how both these technologies presented a vast field for exploration and contained multitudes of possibilities to build innovative experiences. My imagination ran wild and I found it strenuous to gather my thoughts, I couldn’t fathom how our lives might change if these technologies became powerful and accessible to the general public.

Where does User Interface fit into this story?

Hold on, we’re getting there.

Fast forward to today, we sit in front of our electronic devices, struggling to catch our breath as we scroll through each new development in AI and how it is facilitating efficient workflows for various other technologies, including XR. With each passing day, humanity’s vision of the future, good or bad, is approaching faster than most people anticipated.

You’ve all probably seen plenty of these but just to put things into perspective I recreated some of the images I had generated in 2016. I used Deep Dream Generator once again after 7 years, and this was the output.

With all of the mind-blowing things happening in the world of AI, I as a designer, have two big questions. First, a question that concerns many designers — Will my job profile become obsolete? Second, How rapidly will User Interface (UI) and User Experience (UX) transform in the coming years?

The first question is rather pointless because there’s not much we can do but speculate. Whether AI will adjust into or completely replace existing systems is something that is difficult to predict. And so, I chose to pursue the second question.

I wanted to approach this topic with a comprehensive understanding of how emerging technologies are shaping UX, the currents trends, and how the users are adapting to these changes. I put these topics in conjunction with the developments in AI and XR as I envision great experiences being built if we combine these technologies and harness their offerings effectively. And while some of the ideas may have been discussed previously, I present them with a more grounded approach by bringing in the perspectives of our existing experience with technology.

All right then, let’s begin!

As humans have become increasingly familiar with electronic devices over the last decade, many companies’ priorities have now shifted towards providing a great UX with their products and services. With the integration of technologies like AI, they are constantly experimenting and building solutions on a pursuit to provide the best experience. Let’s look at some examples.

When we think about advanced Human Computer interaction (HCI) powered by Artificial Intelligence, the facial recognition system in our devices can be a great example. Facial recognition completely eliminated the password-protected phone unlocking method, thereby reducing the user’s effort to a mere glance. Similarly, features like Voice Assist enabled users to perform several tasks without even touching their device. Another example is gesture recognition. Although relatively new and not used as widely, gesture recognition has significantly reduced users’ efforts with its fast and efficient interactions.

With each example we see a trend towards making technology less like a tool that is a separate entity and more like an extension of the users. It might be clear by now where I’m going with this. A favourable UX has minimum to no interface interaction. This makes technology more accessible and overall enjoyable to use. However, the examples mentioned above are making the interface completely “invisible”.

So, how do designers go about achieving this kind of UX? One of the most important guiding principles of experiences like these is the concept of Natural User Interfaces (NUI). NUI’s objective is simple — designing intuitive interactions in order to get as close as possible to how humans engage with the physical world. Some other technologies that are truly living up to the definition of NUI are XR experiences which include Augmented Reality (AR), Virtual Reality (VR) and Mixed Reality (MR). The 3 dimensional and environmental context provided by these technologies makes interacting with digital information more intuitive and immersive than any other method. The spatial awareness and contextual interface puts the experience at the forefront, and the traditional interface becomes secondary.

These two parallel developments in the field of alternate interfaces have very similar objectives. By leveraging the principles of NUI, their ultimate goal is to make digital interactions feel as natural as possible.

But how are users reacting to this?

Technological developments may not need to wait for users, but their true value is only realised when users find it beneficial enough to make it a part of their lives. In order to understand how users might perceive innovation in upcoming technologies, we should first understand how they have perceived innovations in the past.

With every new technology we have always witness interesting gradients of curiosity, excitement and skepticism amongst the public. But, in the end, what has mattered most is how it has improved and simplified people’s lives. When touchscreen interfaces were first introduced, many people questioned its longevity. Ergonomically, it was inferior to its predecessor, the button keypad. Button keypads provided tactile feedback and were more precise whereas touchscreens increased the chances of accidental touches and smudges. Despite these drawbacks touchscreens were eventually widely accepted because of their configurability, flexibility and adaptability, both in terms of development and usage.

This is a great example to understand how often tradeoffs are made in technologies for the ‘greater good’. When users are shown the ‘greater good’ they will make adjustments to their habits and expectations as they integrate a new technology into their lives. The remaining drawbacks in these technologies are mitigated with incremental improvements over the years through user research.

Adoption of XR

XR has a long standing history that predates mobile phones and even the internet. However, It’s potential was only realised when the supporting infrastructure improved with robust computing power, reduced cost and size of hardware, and commercial viability. Despite this, XR is still in its early stages of adoption. Let’s understand why that is the case.

I worked at a company that had various XR offerings. After working on several projects as a UI/UX designer I realised one thing, there will always be a significant learning curve for users no matter how strong their digital literacy is. And this will continue to happen until we see products powered by XR being fully integrated into our lives. To have strong conviction in a design, simply talking to users and taking their feedback in an incubated environment isn’t enough. It’s a painstaking process of longitudinal research which is only possible when these technologies are ubiquitous. While the awareness and interest in XR have grown overtime and we have witnessed great potential in sectors like retail, entertainment, education and beyond, there are several challenges and barriers in its practical application that exist even today.

XR is still expensive, both to build and to experience. Most people still need assistance while navigating through these experiences. There is also a very limited amount of quality content that is available and it severely lacks diversity. This is because the XR content development pipeline is very complex. From content generation, optimisation and animation to content publishing, consumption and management, all stages require large amounts of resources, bulky softwares and devices. Even if we look at its intended usage, we would notice limitations. There’s either a barrier between the user and the virtual experience or a barrier between the user and their surroundings. For example, holding up a smartphone or tablet to be able to view AR experiences, or, wearing VR headsets, where a user’s spatial awareness of their actual surroundings is completely eliminated. MR compensates for these drawbacks to an extent by merging the real and virtual surroundings but its current application is extremely limited. These barriers hinder the seamless integration of XR into regular use.

We often view new achievements in these technologies as the next frontier of HCI that has the potential to transform the way we engage with digital content. But we only see it’s “potential” through archetypes. Only when we look at the ground reality do we understand that there’s still a long way to go. This is where AI has played a transformative role, it has changed the paradigm by making advanced technology accessible, affordable and highly efficient.

Adoption of AI

Let’s revisit the examples I had discussed in the previous section. Technologies like voice assist, facial recognition and gesture control have simplified user interactions significantly. They have done this to such an extent that older people and people with disabilities are able to perform tasks independently on their devices. Overall, these technologies have been received positively and are used widely. In 2017, Pew Research Center study found that 46% of US adults use virtual assistants on a daily basis. By 2020, up to 51% of American citizens used AI-powered voice assistants as their main source of information, and the numbers continue to grow.

Artificial Intelligence has also led to the development of Wearable AI which is incorporated in Smart Watches and Glasses. They collect data about the user’s environment and their activities and use that data to provide users with information, insights, recommendations and much more. Smart Glasses have not been able to capture user’s interest yet due to a of lack of compelling practical examples, high cost and technical limitations. But, Smart Watches have become increasingly popular since their introduction. The convenience of using a hands-free device along with AI powered features like voice control, health tracking, smart notifications and recommendations make Wearable AI very appealing.

We should also be mindful of the fact that most AI powered technologies work behind the scenes to enhance products and services that users might already be familiar with. AI has not introduced radically different formats of interactions which has also contributed to its high adoption rates.

Form all the topics discussed above, we understand how adoption of new technologies is not just dependent on how original or disruptive they are. It is also dependant on a combination of factors like convenience, cost, and perceived benefits. Technologies can only be accepted once users are able to see how they are beneficial in their immediate lives. AI has largely been successful in doing so, but XR has faced many barriers. Continuous developments in hardware are attempting to address these barriers. But, if we put that aside, there are still many scattered pieces that need to be put together in terms of creating solid use cases and quality content. This is where the recent developments in AI can change the course for XR in the coming years. In fact, discussions on XR are already gaining momentum as people are seeing these two technologies slowly converge.

A lot has happened in the past year in AI. This section briefly covers some selected developments that are hinting towards the interface of the future.

An interesting development in the category of Wearable AI was announced by Humane while I was still writing this piece. They’re developing a screen-less device which is built entirely from the ground up for AI. It uses a tiny projector to display information on the user’s hand and majority of the tasks are performed using only voice commands. It allows the users to truly stay present in their surroundings — no screens, no buttons, no distractions. With it’s AI powered and completely customised insights, the device enhances everyday interactions for the users as they explore the world.

Another notable development is taking place in the world of body implants. The Brain Computer Interface (BCI) is a device that enables users to interact with external devices using brain signals. It has been under development for decades with successful trials being run on monkeys and pigs. AI and Machine Learning have been crucial tools for the developments in BCI. Read more about this here. Recently, The FDA approved the human trials involving a BCI chip developed by Elon Musk’s Neuralink, a venture with the mission to develop a BCI that improves human communication in light of AI. If the trials are successful, people would be able to perform various tasks on a given device by just thinking about them.

In both these examples, the interfaces function as natural extensions of a person’s cognitive and physical abilities, making them great examples of NUI. But what about the content within these interfaces? What would they look/feel like? XR is brining some interesting ideas to the table.

As we saw earlier, XR has been able to demonstrate how immersive and contextual interfaces are highly intuitive. We have also seen how the success of XR technologies would rely on a robust ecosystem of applications, games, and experiences that cater to different interests and needs. This is where AI is stepping in and we are witnessing unprecedented progress in these technologies.

For example, Moth+Flame’s new generative AI platform enables users to create VR content on the fly. It was recently used by Investco, a global investment firm where they were able to build a custom VR training experience that took only 72 hours to develop. Several new AI plugins for different content generation softwares are also accelerating the processes. AI plugins for Blender (a 3D content creation software) are enabling users to create textures, animations and 3D objects using only text prompts. Apart from this, many independent creators have already explored new workflows by combining different tools and softwares to create XR content within hours. For instance, using BariumAI and Midjourney to create 3D assets and textures and combining them with platforms like 8th Wall (a WebAR content development platform).

The hardware is shrinking into hands-free devices and it’s becoming easier to create interactive XR content. When the barriers of content creation and hardware are completely out of the way, our primary goal would be to present new and innovative usage scenarios. In other words, showing the users the ‘greater good’ so that they’re willing to make the transition from their old devices. And just like smartphones, the new devices would become omnipresent rapidly.

All the interfaces we’ve discussed till now are still stand-alone experiences, and they can possibly head in their own unique directions in the future. The various versions of the ‘interface of the future’ are still arbitrary at this stage. And though all these technologies are improving the user’s experience in their own ways, there is an overall lack of cohesiveness if we look at the human species and how they interact with the world around them.

None of the interfaces mentioned above fit into our evolutionary context on their own. Hence, to be able to envision the future, we need to be mindful of the aspects that hold significance for humans in comprehending and engaging with the world. I have defined this by addressing 3 primary aspects — Contextual Information, Control and Choice, and Easy Management. By addressing these three very important points, we will move closer to the future interface that is about to unravel in front of us.

Contextual Information- Spatial Interface

Visual information is the primary source that shapes our perception, builds trust and credibility, holds our attention and evokes emotional responses. Information that is presented to us in the right context is even more effective in doing so, and XR has promised compelling experiences with its immersive visuals.

Now place the new developments in XR alongside something like Google’s Geospatial Creator, a tool that will allow users to place spatially anchored AR content anywhere on the planet — and we can slowly start to build a picture. With XR we can expect to see information displayed spatially as we navigate the world. The interface would be largely divided into two parts. The first interface would be anchored to the users, a personalised dashboard that follows the user everywhere. The second interface would be anchored to various objects and locations in the world, providing users with all the relevant information they need to know as they interact with their surroundings.

Additionally, with the help of AI, the information would be tailored to how a person has previously interacted with the world, creating a unique experience for every individual. The interface will continuously improve itself as the users interact with it overtime.

Control and Choice- Flexible Modality

We have seen how technologies like voice assist and facial recognition are now used on a daily basis by many, but there is always an option to not use them. Users can still type in their passwords or perform tasks manually. Some user prefer more control and visibility over the interface while others don’t.

Technology cannot be enforced on users, hence the interface of the future will give the users several options to choose from. Users would be able to switch modalities based on their requirements. They would have options to choose between a voice activated or a visual touch-based interface or, interact with an AI powered virtual assistant. Users would be able to use a traditional 2D interface, an interactive gesture based XR interface or even transport the themselves into a virtual environment, all using the same device.

The interface will be multi-sensory with visuals, sounds and haptics, and it will give the users the option to fully configure a personalised UX for themselves. This will also be applicable for the hardware a user would pick depending on their needs. Speaking of which, lets see what the hardware might look like.

Easy to Manage- Wearable Ecosystem

The concept of wearable devices has become extremely popular as people are able to perform common tasks using a hands-free device. However the new devices will be even more inconspicuous. The device of the future will not make itself the centre of attention but will silently assists the users from the background as they interact with the world.

In the future, we would purchase a collection of wearable devices or small implants attached to the skin that would be almost or completely invisible. From this collection of devices, we would choose the what we need depending on the nature of the content we want to consume, and remove what we don’t need. Each device would serve its own purpose. For example, wearing contact lenses for the visual feed, placing hearing devices inside our ears for sound or small finger bands for haptic feedback and gesture interactions.

With AI, the devices will learn and mould themselves according to the user’s needs. This will ensure that the device truly becomes an extension of the user, seamlessly integrating into their daily lives. The future of wearable devices will be unobtrusive, and would empower users to effortlessly interact with technology while maintaining a natural and immersive connection with the world around them.

As we venture into the future of UI design, one thing is evident — technology is moving towards more seamless and invisible interactions and NUIs are paving the way for this transformation. The convergence of AI and XR holds a great promise for the future. It’s an exciting time for technology as it is being carefully crafted to coexist harmoniously with human beings. But, there is also a cloud of fear that looms over us. At times, technology has been detrimental to humanity causing destruction and harm. And the more powerful the technology, the more profound its effects.

Hence, as technology gets more advanced, it will also become important for us to periodically step back and contemplate the future we want to build for ourselves.

In understanding the intricacies of humanity’s pursuits, it becomes clear that technology is a small fragment in the grand scheme of human aspiration. Our primary need lies in the quest for a life of true quality which goes beyond the instant gratification and transient desires that are brought by technology. Quality life encompasses many other intangible qualities like the fulfilment of our emotional, intellectual and spiritual aspirations and coexistence with fellow human beings. Technology should be built to aid us in this pursuit, not steer us away from it.

We should cherish and value these intangible qualities that make us human. Only when we do this can we create a future where technology serves our fundamental goal: a meaningful life of purpose and fulfilment.

1. Emerging ExG-based NUI Inputs in Extended Realities: A Bottom-up Survey (2021); Kirill A. Shatilov, Dimitris Chatzopoulos, Lik-Hang Lee, Pan Hui
   https://dl.acm.org/doi/abs/10.1145/3457950?sid=SCITRUS
2. AI-assisted content pipelines for XR (2023); Kuldeep Singh, Raju Kandaswamy
   https://www.thoughtworks.com/en-in/insights/blog/machine-learning-and-ai/ai-content-xr
3. A Short Report on Multi-Touch User Interfaces (2010); Patrick Lindemann
   https://www.medien.ifi.lmu.de/lehre/ws1011/mmi2/mmi2_uebungsblatt1_loesung_lindemann.pdf
4. VR Firm Moth+Flame Opens AI Generative Content Beta (2023); Demond Cureton
   https://www.xrtoday.com/virtual-reality/vr-firm-mothflame-opens-ai-generative-content-beta/
5. Nearly half of Americans use digital voice assistants, mostly on their smartphones (2017), Kenneth Olmstead
   https://www.pewresearch.org/short-reads/2017/12/12/nearly-half-of-americans-use-digital-voice-assistants-mostly-on-their-smartphones/