Designing for the Untestable
Originally written for #AltDevBlog
Sometimes you’re asked to design for the untestable scenario. For instance, design a system for 10,000 players to asynchrously interact in a persistent competitive world with progression mechanics that plays out over 3 months.
Disclaimer: The entire time you are reading this remember one basic truth or else everything else contained herein is useless.
Focus on second-to-second play first. Nail it. Move on to minute-to-minute, then session-to-session, then day-to-day, then month-to-month (and so on). If your second-to-second play doesn’t work, nothing else matters. Along these lines, if your day-to-day fails, no one will care about month-to-month, either.
- Brenda Brathwaite
WebPad: Bring your Own Controller
Going to take a break from my Useless Designer series to talk about a personal project which has me very excited. While judging Dare I came across a good idea which could be brilliant: using a mobile device as the controller in a shared space. The idea was shown by Lucky Ghost using the iPhone and a native iOS app. However, Apple would never let it through the AppStore and not everyone likes Apple. The idea was good but limited, but then a solution hit me there and then and I excitedly shared it with them.
The game hosts a web-service providing HTML5 WebApp with WebSockets transforming any modern mobile device into the controller.
Stop being the Useless Designer: Programming
A bit to the left, No a bit to the right. Mmmm maybe if we make it blue?
Let’s face it, there is nothing more annoying than being bossed about by someone who is useless.?
So here are three simple rules.
- Work with them in the trenches.
- Everyone in the trenches has to be useful.
- Supplement don’t Replace?
So acquire some “Hard Skills” fast and be useful. This is a multi-part post for some places to start developing those “Hard Skills”.?
Stop being the Useless Designer
Part 1: Excel & Formulas
A bit to the left, No a bit to the right. Mmmm maybe if we make it blue?
Let’s face it, there is nothing more annoying than being bossed about by someone who is useless.?
So here are three simple rules.
- Work with them in the trenches.
- Everyone in the trenches has to be useful.
- Supplement don’t Replace?
So acquire some “Hard Skills” fast and be useful. This is a multi-part post for some places to start developing those “Hard Skills”.?
Though I encourage you to jump into your own tunnels of exploration. I hope this is the first of a multi-part post focusing on various tools or hard skills for designers. Introducing a tool or skill, then getting you interested.
THE BEST PLACE TO START IS YOUR IN-HOUSE TOOLS!!!
Feedback in Production
Originally written for #AltDevBlogADay
As game developers we know the importance of Feedback in our games for players, but so often we forget the importance of feedback for ourselves. Implementing good feedback loops in your production cycle can be just as important.
Two stories highlight this for me. The first is personal, a team for a game I was working on was having the critical bug count communicated to them daily in scrums, and key issues pinned to the wall. The engagement with the lists was low, until one programmer asked for a graph. Since the coloured stack graph went up enjoyment is much higher and conversations happen around the graph (and supporting lists).
Edit: Found Source for this article"You Wouldn't Like Me When I'm Angry" by Nick Waanders
In an article I read a programmer spoke about a game they were working on and the artists not paying attention to the frame rate. The number just didn’t matter to them. So he replaced the fps counter with a series of photos of his face: Happy, content, angry. The improvement was immediate, because the information was visual and the granularity reduced for clearer communication.
General Rule: For every role in the production cycle question what information is critical or a good performance indicator and try to expose feedback in the most appropriate format and make the feedback loop as short as possible.
Claire 
Below are just some quick ideas and areas to try hit
Programmers
- Compile or Pack times: The length of this feedback loop directly impacts the productivity of your team.
- Code Reviews: Not only for quality of code but is the individual receiving adequate feedback on their work. A mentor system or pair programming system can help here, don’t try dump it all on your leads, they won’t have time. Resulting in a breakdown on the feedback loop.
- Feedback on Failure: Are your error messages consistent throughout the business? Can you implement a dump system, or display a call-stack. In the case of failure are you providing the coder with enough quality information to solve the issue? In larger companies, does the error communicate which area of responsibility the error is occurring in?
- Live Analytics: In the case of your programmers working on live systems do they have access to realtime easy to process visual data on player patterns, which functions are performance heavy or what areas of suffering above average load?
Artists
- Immediate Preview: As an artist can I, without support from others, quickly view my assets in-game.
- Assets Streaming: Can an artist update an asset without rebooting the game, removing the need to navigate menus and get to the point of the game where they can see the asset.
- Performance Data: Do I get understandable visual feedback on texel to pixel ratio, overdraw density or other technical data in a fashion that is accessible to me?
Designers
- Real-time Variables: Can a designer alter values in game on the fly to see how they feel?
- Stats Data: Can I receive feedback from any play-through of the game done by QA, Focus Testers, or coders in the form of visual reports or recorded numbers?
- Live Analytics: Do they have access to real-time easy to process visual data on player patterns, purchases of items, drop of rates and general user flow?
- UI Heatmap: Can the game generate a heat-map of cursor flow?
QA
- Build Dates: Can I quickly visual identify the build number and date?
- Bug Feedback: When a bug is returned to QA am I provided with enough information to reassess the bug and is the communication clear?
Production Staff
- Work Done: Does a report quickly identify work done and work remaining at present?
- Visible Build: Can I access a recent working version of the game?
- Time Spent: Can I identify areas of the work which have a higher than expected cost, or large investment in them.
- Graphs: It sounds cliché, but a picture is worth a 1000 words. Live graphs prefably on a web system internally will make you life so much easier.
- Team Time: Is there a frequent structured process for me to receive feedback from the team and provide them with feedback?
Board, CEO, VP, and other Stake Holders
- Visible Progress: Am I able to see visible progress in an easy visual format?
- Team Issues: Are HR items (such as sickness), morale, rework to content, areas of uncertainty communicated with progress reports?
- Target visibility: Am I aware of the big goals for the next development target(s) and able to assign a confidence level to delivery?
Battle Plan: Shipping Someone’s Problem Child
Getting it out There
In my relatively short time in the industry I seem to have attracted a few fixer-uppers, reboots, ports and all round oh do you mind just finishing it off jobs. Some of the biggest of these I have been Lead Programmer or Lead Designer on. Through mistakes, and successes I now have a rough battle plan for getting these jobs done that I thought I would share as my first post.
To clarify we are not referring to projects with a stable team and direction that you take over, or are promoted in charge of. We are talking about those deals where a wet deposit of mud is dropped on your desk, and you are told to find the diamond inside.
1. Run Away if you can
First off, run away! No seriously escape is your best option. These jobs are scary, messy and with ten hidden problems for every problem you can see. Unless there is a really really good reason to do this then just leave it, its not worth your time.
Okay you think it may be worth your time? First question has someone signed some paper-work and committed you before a full in-depth review, if so slap or knock out the foolish suit with no brains. Try avoid commitments, and deadlines at first, you need some time to inspect this project with your full team.
2. Investigate & Negotiate
Find the stake holders, and if possible the developers originally involved and follow the path that lead to the current state. Treat all previous work on the project as if it was done by a genius savant who forgets his trousers. The people before you are likely to be intelligent and experienced but assume they make silly mistakes.
Find out the stake-holders goals, and what they want from the project. Whittle it down as much as possible and kill any sacred cows you can. The more you have the option to change and throw out the more options you have.
Ask to get back to them, keep the lines of communication open. The most common cause of a project failure of this kind is a lack of communication or vision. Understand what they want, keep them in the loop because I guarantee you that you only understand half of what they want, and they only understand a tenth.
3. Rip & Roar
Is there a playable build? Get some pizza, drinks and your team together. Play it play it and play it some more. Understand the product, read the documentation, pour through the assests and then rip into them.
Give the team a chance to say what is terrible about the game, what should be burned with fire, and stuff that just doesn’t make sense. Be sure to keep them on track with the stakeholder vision, but don’t stop them from tomatoes at the goals, let them question it.
Write all this down in a public space, such as a whiteboard. It keeps everyone aware of the challenges that need addressing.
4. Analyse
Pick through the design, art and code. Often the design of a derailed project will have conflicts, and missing pieces of data. Get your design team to dig deep into the numbers, and systems.
The code review strategy depends on the size of the team. I suggest strike teams of 3-5 programmers with specialist areas, or general on a small team, to attack all the core systems. Compare it to the documentation, if there is any, but more importantly map it out yourselves. Don’t let them do any commits yet, just map out the maze.
I cannot speak for the art process but generally its been a case of compiling a style bible, and clarifying the style. Hopefully someone else can give more advice here ;)
Highlight areas of risk, and I can guarantee for every one you find you missed three. You should now have replaced your rip apart boards, with risk boards.
5. Rebuild
Talk with the team about time-scales and build a plan to rebuild. Leave yourself time to iterate, and for the un-missed risks. This is pretty much like a normal game process just with a very unstable base.
This is normal project management, and I don’t want to tell everyone how to suck eggs. The only real challenge, which is hopefully address by the previous two steps, that the team buy in is much harder to get on a rework than a fresh concept.
6. Set Expectations
Now is the time to go back to the stakeholders and set expectations. The worst mistake you can make, and one I’ve made, is to try gloss over some risks.
Everyone should know the risks, the costs and what they are likely to get at the end of the project. I guarantee the costs are higher, and the risks are higher. The only reason to do this from a business perspective is to recover investment. Don't be afraid to say it's not worth it and kill it now.
Sometimes though the reason is because someone believes there is a diamond in that piece of mud they dropped on your desk. It’s a great feeling to find that diamond, and worth the work.
Also now would be the correct time to sign the piece of paper. 
Claire
Postscript:
Here are some general tips for you all, hope they help.
- Lines of Communication are so important in problem projects, be sure of them.
- Don’t pollute the project with blame, hence the rip it sessions. It gets all that negative air out there then rub it out and replace it with plans.
- It’s tempting to throw out massive blocks of code, but a lot of time can be wasted. Instead I encourage decoupling efforts first so you can isolate problem areas.
- Some core design may need to be reworked, but encourage your designers to isolate risk areas first.
- Don’t be afraid to prototype! Yes I know the project has already started but prototypes give a stable clear vision of a problem and a possible solution.