May 7th 2020 marked the first Inside Xbox event to show off gameplay running on the Xbox One X Series, Microsoft’s latest gaming console. There was much hype surrounding this event, unfortunately, many felt disappointed by what was shown. Aaron Greenberg the General Manager of Games Marketing had to say about this:
Had we not said anything & just shown May Inside Xbox show like we did last month, I suspect reactions might have been different. Clearly we set some wrong expectations & that’s on us. We appreciate all the feedback & can assure you we will take it all in & learn as a team. 🙏🏻💚
— Aaron Greenberg 🙅🏼♂️❎ (@aarongreenberg) May 8, 2020
Is failure bad?
We live in a world where success is celebrated and failure is seen by some as a disaster. Not scoring that point, not passing that exam, or getting that deal can hurt but it can also stop someone from trying again.
“I have not failed. I’ve just found 1,000 ways that won’t work.” Thomas Edison.
Thomas Edison is regarded as America’s most recognised and prolific inventor. In his pursuit in developing an efficient incandescent lamp, he tried over 1,000 different ways with each one failing, only to finally succeed. Sometimes in life, the greatest opportunity to learn comes from failing, getting up and trying again.
In Agile, failure is just a result of a test, there is no social stigma or negativity around it. By failing and you have learnt something you may not have learnt otherwise.
“What good is an idea if it remains an idea? Try. Experiment. Iterate. Try again. Change the world.” Simon Sinek Together is Better
Failure in an Agile Environment
The purpose of this post is to show how we can leverage user research coupled with a testing mindset to help to exceed user expectation and create a buzz around the Xbox One Series X and its games.
Agile can be defined in two different ways:
- The ability to react to an ever-changing environment. When technology introduces new opportunities, markets collapse, or new markets open up, having the ability to react quickly amd take advantage of the opportunity is being agile.
- To take what you don’t know (the known unknown) and use a systematic approach to test different hypothesis to make it become a known known.
In this post we will look at the second approach, that is to develop a set of hypotheses, build a minimum viable product (MVP) to test and use the results for either iterative or deliver to the end users.
The importance of user research
It is sometimes very easy to forget the maxim,
“you are not your customer”
To help eliminate such biases, user research helps to understand user behaviour, their needs, their motivations through testing, observation and interviewing.
What is user research within Agile?
If Agile is the process by which we can develop and test new ideas, user research is the way we can help generate ideas to test from the users.
What are we looking to test?
Jerret West, CVP of Gaming Marketing at Xbox said, “starting with the May 7th episode of Inside Xbox, we will be showcasing what happens next in the world of Xbox, every month, which is why we’re calling it ‘Xbox 20/20.’”
By having an event every month until release provides the perfect opportunity to test and iterate.
To illustrate the idea of using user feedback and iterating we will look at two different examples:
- Ray Tracing – a hardware feature for the Xbox Series X.
- Halo Infinite – the launch of a new game.
Hardware – Ray Tracing on Xbox Series X
The Xbox Series X is the first Microsoft console to feature Ray Tracing which has traditionally been reserved for high-end PCs.
What questions do users have and where does the information exist?
Gathering information online
In this example, we are looking for users who are discussing ray tracing. This may include content on Quora.com, Twitter, Microsoft Answers and the comment section in YouTube videos. What is important is to identify what questions are people asking? What makes people excited about Ray Tracing? And how are people justifying the purchase of a new graphic card or a next-generation console to experience Ray Tracing?
Asking questions – slicing and dicing data
We have all seen Twitter polls, voting in Instagram Stories, emails with questionnaire etc. there are many ways to get answers to questions. We may want to ask questions to specific audiences which can be done by using different social accounts, time of day of posting or paid postings where we have even more control on who participates.
What if we took this a stage forward? Microsoft has significant amount of user data on which games are played, which apps are used and which consoles have been purchased by user. We could create different audience segments:
- Users who have purchased Xbox S and Xbox X.
- Users who buy multiple games per month.
- Users who play less than 4 hours of gamer per month.
- Users by gamerscore.
It can then be possible to target user in console (having surveys on the dashboard) or over email. Questions can be asked can include if they know what Ray Tracing is? How important are graphics relative to other feathers a new console has to offer? Or if they have any specific questions on Ray Tracing?
In terms of the audience we can take two approaches:
- Select the audience which is most likely to purchase the Xbox Series X and target those.
- Look for the audience that share similar properties which is largest in in size relative to other audiences. By optimising a campaign for this audience will also have the greatest impact on the whole audience.
Developing a hypothesis and MVP
Now that you have data, it is about developing hypothesis from the data and finding, ‘quick and dirty’ ways to test. What is important to remember:
- A hypothesis is just an idea, not every idea will work. Don’t become attached to an idea until you have data to prove its validity.
- Your hypothesis may not be the best.
- It is all about getting feedback and iterating.
- The solution may not fit all (e.g. ways people learn or consume media.)
- The endpoint is a decision based upon on data you currently have on how to present Ray Tracing for the Series X launch period. Things change over time.
Let us for a moment walk through an example to see how this could work. We may for example have a segment of the audience who does not know what Ray Tracing is but regard improved graphics as the most important factor when buying a new generation console.
We could test showing videos of Ray Tracing on and off versus screenshots of games with and without Ray Tracing, to see which one better communicates how Ray Tracing can make graphics look better. Or is there a place to show both video and images? Taking that a stage further by iterating, what if we had an engineer talk how Ray Tracing works before or after showing the video/images? Rather then using an engineer what if we used a studio head talked about the power of Ray Tracing in building games? The objective here is to test multiple different scenarios with users to find the one that is most optimal. This format can then be used at the reveal event.
Launching a New Game – Halo Infinite
Halo is a system seller and the second biggest IP after Minecraft. We know that Halo Infinite will be shown at the Xbox 20/20 event in July and the game is due to be released this year.
Let’s set ourselves a different question, post the reveal event how quickly can we adapt to user feedback and produce marketing material? Usually, in any marketing campaign, we define all the marketing materials required in advance and have them ready for the event. What if we took a different stance, and decided we needed to be flexible and respond to user feedback during the actual reveal?
The following is a an example how we can build a team that has the capability to respond rapidly to user feedback.
Predict
We are going look at two sources to try to predict how fans will react to the reveal. Internal users and historical reactions to new video game reveals.
Internal users
Bring together employees from Microsoft/Xbox who are experienced in storytelling but have not worked on the Halo reveal in a ‘room’ together to show the Halo reveal to. The objective is to highlight potential problems with the reveal.
I would recommend using the format called ‘Braintrust’ pioneered by Pixar, where everyone’s’ opinion is equal, no one is to be blamed for failure and people are encouraged to use Radical Candor to identify the root causes of problems. To learn more about Braintrust I would suggest reading this article on fastcompany.com or reading the book Culture Code by Daniel Coyle
Having this critical feedback can be used to identify which problems need to be resolved before the reveal and which are a potential problem that we need to help mitigate during the reveal event.
Historical reactions
There have been many First Person Shooters in the past, there include previous releases of Halo including the more recent release of Halo Master Chief Collection on the PC. There have had reveals, trailers, gameplay videos with lots of user feedback. Take all that user feedback and identify common questions, common disappointments, recommendation I.e. ‘I wish we could see more of….’
Building a multi-disciplined team
The idea is to put together a team with different skills so they can act independently from all the other teams. The team should consist of people who have the necessary skill and experience to produce any marketing material they require. The team may include graphic designers, video designers, content writers, engineers, video editors etc. The team should be independent and able to self organise in a way that works for them. In software development we would call this team a Scrum team. In our example we can say we have an Agile team.
Their first objective is to understand what the Halo Infinite reveal contains, take the feedback from the internal users and the data from the historical reactions and understand possible pitfalls.
Building a content library
Once you have all these possible scenarios, the scenarios need to be ranked according to the chance of it happening on the day of the reveal and the effort required to produce something. Let’s look at a possible example of this playing out.
Let’s say the July event is 90 minutes long. Halo Infinite is shown in the first 5 minutes and there is space reserved for the last 5 minutes for a follow up to the Halo Infinite reveal. Within minutes of the reveal the feedback online is negative indicating an area of disappointment, for example, the lack of multiplayer gameplay. The team has 80 minutes event to rectify it and present something at the end of the show.
It may seem scary at first producing a something in a very limited time-space but with all things, it is all about practice and having retrospectives to see how you can improve. Questions that could come up in the retrospective could include:
- What additional content needs to be added to the content library, that we can repurpose?
- Which Halo staff need to available to interview on the day if needed?
- What skills does the team need to learn?
- How does the team perform under pressure?
- What can we do to ease the pressue?
- What are the impediments that need to be removed?
- What backs up can we put in place?
Each time you run a scenario the team become quicker because they know what works. Through the retrospective they have highlighted the bottlenecks and subsquent dealt with them. And on the day of the reveal and they become aware of what is needed, they function like a well oiled machine, relaying on each other’s capability to deal with what ever hit’s them.
Conclusion
The post looked at two different ways we can apply a framework of taking user feedback, testing and optimising as we learn. What is key to note, there is no absolute right solution, it taking the data you have, making educated guesses and testing with real users. When you build a business that Agile in both senses it will learn from it mistakes and takes it successes and look to amplify them.
Image Credit
Thanks to Zohre Nemati for sharing their work on Unsplash.
Photo by David Travis on Unsplash
