Large Scale Project Management

This is an excerpt from the Article

Large Scale Project Management
by Michael Saladino






Projects across the industry are becoming more complex and in few genres is this more evident than in massively multiplayer games. Publishers are pushing for a shorter turn-around on their investments, hardware complexity is increasing and end users are demanding larger and more interactive content. All of this results in ever-growing development teams. At major studios like Electronic Arts, Sony, and Microsoft, teams over a hundred people are becoming the status quo. This gem will look at what the new project management looks like in terms of team hierarchy, rigorous scheduling, strong process, and resource handling. I will propose a collection of guidelines that I have used myself in directing large scale projects at Microsoft and now at Electronic Arts.

Building Your Hierarchy

For decades, computer games were developed by small teams with a few dozen people at the most. This helped promote the idea that a couple of managers could deliver a game in its entirety. One producer, one lead engineer, one art director and two dozen artists and programmers building the game was how most products were being constructed. But with larger, modern teams comes the necessity for hierarchy to maintain direction and order. Deciding on your hierarchy or organization chart is the first step in building a stable and productive development team. But what does hierarchy really mean for a large scale project?

First and foremost, hierarchy is a way of reducing the number of direct reports to a single manager. This is normally done by selecting sub-leads below the manager. If you have one lead engineer and twenty programmers all reporting to him, you need to select sub-leads in order reduce the communication load on the manager. Pick four or five engineers along major functionality lines such as multiplayer, AI, game mechanics, front-end, graphics, and systems. The goal should be for each sub-lead to have two to four reports and those four or five sub-leads all report to the lead engineer. This is the most fundamental purpose of hierarchy: reducing communication requirements by empowering sub-leads to take ownership.

By reducing the number of direct reports one manager has to deal with, you are clearly reducing his daily and hourly communication needs. Studies in the military, arguably one of the greatest achievements in management structure, find that an individual manager functions best when he’s only handling three or four people directly. From my experience in game development, I find that someone can effectively produce assets for a game whether art or code and maintain three or four direct reports. Once you hit six to eight, the manager’s ability to produce drops extremely low as they spend their days managing almost exclusively. And if you cross over into double digit reports, an individual’s ability to even manage effectively is affected. Therefore, keep direct reports in the single digits. If the manager is also expected to produce assets for the game, keep them below five; otherwise, they are a pure manager.

While the direct result of hierarchy is improved communication handling, there are other indirect and subtle improvements to take note. Workers in small groups tend to feel more individual empowerment. When a person is in a cube farm and is treated just like the other thirty engineers surrounding him, the person can begin to lose identity. A feeling like an assembly line worker can take hold which is usually detrimental to the person’s overall production. Creating sub-leads in control of small groups can keep a person feeling like a major contributor even if it’s only in the small group. Most engineers would rather identify as one of three engineers building AI rather than one of thirty engineers build a game. It’s simply more personal.

Of course, this benefit also has a draw back which is people becoming isolated from the game as a whole. If an artist does nothing but paint household object textures for months on end, it can be easy to lose the forest for the trees. The detail that they’re focused on grows in size until it becomes the whole world which can be good in the short term but is often bad for the long. Remember to always keep people attached to the overall product. Have weekly team meetings where you show off the game as an end-user would experience it so every person can look and say “I did that part”. If you think about this issue in a psychological way, people in the game industry were most likely all smart children in their school days. Therefore, take them back to the joy of seeing their work posted on a board with a gold star for everyone to see. Even if the grown-up cynic in them openly scoffs, a part of their childhood will appreciate the attention.

Delivery vs. Functional

Once your team grows to the point where more hierarchy is required, the question becomes how do I divide the team? If you imagine a small start-up with six engineers and twenty artists most people would split the team up into functional groups. Five of the engineers would report to one lead. The art team would be cut up into modelers, texture painters, musicians, and animators each with a corresponding lead. And this layout would work well as I’ve experienced it many times in my past. But when you’re game requires ten times this many people, does a functionally based organization still make sense?

Do all the administrative assistants at IBM report to a single lead administrative assistant? Do all the engineers at Microsoft report to a single lead engineer? Can you imagine what would happen if they did even with a deep sub-lead hierarchy like I described above? It would be abject chaos because a functional hierarchy is useful at handling functionally dependant communication but little else. This causes a hierarchy based on functional lines to not scale well. Our true tasks in constructing a game are not functionally pure but instead require constant cross-functional communication. So let’s try something different: a hierarchy based on delivery instead of department.

A delivery group for a game would be the front-end team. This group would definitely include engineers and Photoshop artists. It might also include usability experts, 3d modelers, and Flash scripters. We now have many different disciplines all reporting to a single front-end manager with possible functional sub-leads beneath like a lead front-end engineer. The team is given a single mandate from the team leader: deliver the front-end needed to ship the game. The front-end manager would drive all external information gathering such as the front-end design documents and technical certification requirements from 3rd party console. But the majority of information needed to build this unit has been isolated within the walls of the front-end team and only team wide critical information ever needs to be floated out of the group.

Other delivery groups could include game mechanics, systems, multiplayer, or a stand-alone server. These are areas that can be sliced from the whole and driven with minimal external communication because a delivery group is cross-functional by nature. Included in the multiplayer group are the levels teams that will construct the multiplayer levels. There are modelers, designers, and texture painters. We include the engineers that will build the client/server architecture and extend the single-player experience to include multiplayer. By making these divisions through the overall team, you can reduce the problem from 160 people making one product to four teams of 40 making four products. And if you draw the lines cleanly and have a strong management team at the top of those four products, you’ll be able to keep a rolling integration of these components during development.

Pure Art Directors

But delivery groups aren’t the best solution for all problems. While they keep a smaller team focused on a subset problem, you do wind up with an unfortunate long-term issue with people having managers that aren’t of their discipline. For instance, imagine an animator being assigned to a group that’s going to deliver the melee system for a role-playing game. His manager is handling different disciplines and may not really understand an animation skill set or career. How will this relate to the animator’s ability to get promotions? Perhaps the animator is significantly better than how the melee system turns out due to engineering issues. How will the animator’s pure talent be seen unless a more senior animator is somehow in the equation to rate his long-term value to the company?

This is why functional leadership is needed. Notice I did not say management because these are two very different beasts. To manage as I’ve said before is to handle information flow. Leadership is to inspire your team and motivate them to become better than they currently are. An art director or even more specifically an animation director for my above example needs to be in place to help drive quality and identify long-term career growth for the functional team. No one reports to the art director, technical director, or game design director because they are the holders of the pure art. They push the quality limits without having their hands in the schedule or management flow. So by including these functional leaders in the team you allow managers to drive delivery teams while allowing pure art owners to help inspire individuals with a functional language that they share.

Producer vs. Development Manager

The next line we need to identify comes in the form of the producer. Many small companies that I’ve worked with over the years have a producer come up with ideas, schedule them, and then push the team to maintain vision and deadlines. What a schizophrenic position. To be expected to both develop brilliant new ideas and have the resolve to cut them when needed. No wonder so many titles scream madly past their delivery dates. It’s like allowing an author to edit their own book or giving a director complete control over the final cut. Maybe there are a few true geniuses where this is a good idea but for the rest of the human race, it’s a recipe for disaster. The idea man will always want the final result to be grander in its ultimate pursuit for perfection and you want him thinking that way. You need another person to be the scheduler, manager, and all-around “get it done” man. Enter the development manager.

A development manager is a producer’s alter ego. When the producer is blue skying, the development manager is there to attach the anchor of reality. The development manager’s job is to maintain the schedule and manage the time of his reports. A successful development manager seldom gives the producer exactly what he wants but instead masters the art of delivering 90% of what the producer wants but at half the cost. Each delivery group should have a development manager at the helm making sure that milestones are being met at the same time as pushing to deliver the best approximation of the producer’s vision. It’s a checks and balance system in its truest form. Why expect these two halves to exist in a single person when you can instead hire two people each of who are uniquely suited for the positions. (Plus the arguments between the two heads are now externalized and are fun to watch.)

Quality Assurance

The last detail in initial team layout I’m going to discuss is quality assurance (QA) also known simply as test. Test is regularly the most overlooked aspect of game development. But as with everything I’ve mentioned so far as projects increase in size this is another detail that can no longer be ignored. With larger games come larger test requirements. The sheer number of options in online and especially massively multiplayer games creates test matrices that can barely be fathomed. This is why attaching testers to a team earlier is better and also why our industry as a whole must start treating QA as a viable job and long term career.

The first major step with integrating test is at a company level. Most companies sadly still view testers as 19 year olds that can barely spell. And unfortunately, if that’s who the company expects in the positions then that’s what the company will pay for and that’s what the company will hire. However, major software development companies like IBM and Microsoft have known for decades that software test is a career for mature QA experts. It’s a real job with a real skill set often staffed with people from engineering. A strong test group needs strong organizational management, a solid engineering department, and plenty of headcount. Normally, only the last item is supported by companies and only in the form of cheap contract labor. This reality needs to change across our entire industry.

If you can convince your company that you need to integrate test early, hire a few engineer testers. They are engineers that know how to build test cases in order to stress test bare code systems. They can architect automated testing suites to limit the human equation when dealing with massive test matrices. Sit them with a deliverable team and get them involved in the day to day business of the product. A test engineer working with his team should be involved in the architecting process with an eye towards automation and how certain decisions might increase or decrease required hands-on test time. A correct decision early in the engineering process can save hundreds of hours for testers at the end of the project.

If you found this post interesting you should checkout the full article in the book Massively Multi-Player Game Development 2.