Crunch: does it help and how do you know?
by Eric Byron on 07/29/16 10:44:00 am
In the great debate amongst the games industry regarding crunch the one fundamental questions that doesn’t seem to be answered is: does crunch actually help a game development project be more successful?
In our original survey and articles, the Game Outcomes Project measured game project success using an aggregate score of 4 factors:
- Return on investment
- Critical acclaim
- Internal satisfaction
- Project delays
From our 2014 survey results, our fourth article concluded that teams that reported having crunched during the project were generally less successful than teams that reported little or no crunch. So why do so many still swear by crunch as a necessary part of game development?
Did we ask the wrong questions or make false assumptions about what success is? There are many in the games industry that believe working long hours is an inevitable part of the creative process and those who are passionate cannot and should not be constrained to working 40 hours a week. On the other hand, there is a lot of scientific evidence regarding the ill effects on productivity when working long hours. More importantly, there seems to be complete void of any real data to validate that those working long hours are actually as productive and creative as they would be working less hours.
Alex St. John, in response to an interview with Kate Edwards, recently described those wanting to avoid crunch as having a “wage slave” attitude and basically said we should feel privileged to work 80 hours a week. You can read his blog article here:
Alex St. John 在最近回應 Kate Edwards 的訪問中認定那些不想加班的人是"薪水奴隸"，而且我們應該感覺一周工作八十小時是一種恩賜。請可以透過部落格的文章閱讀他的看法。
We need less opinions and more data!
We’ve been talking with leaders in the games industry trying to gather more evidence to support or dispel the theory that crunch is a necessary and beneficial practice when making games or other creative endeavors. One problem we have run into right away is a problem with defining crunch itself. There are those who believe crunch should be defined as extended periods of mandatory overtime: 3 or more weeks of 60+ hours each week. Evan Robinson, author of the article “Why Crunch Mode Doesn’t Work: 6 Lessons”, is of the opinion that anything beyond 40 hours per week is crunching. Dave Swanson, a director of engineering for Electronic Arts, commented that what people call crunch now is nothing compared to the hours he worked when he started in game development in the early days of Madden Football at the Tiburon Studio in Orlando, Florida.
我們與遊戲產業內的領導者深談，想找到更多證據來支持或質疑這種"創意開發或遊戲開發時，加班有益"的理論。首要的問題就是我們如何定義甚麼叫作加班（超時工作）。有人認為加班應該定義為被命令長時間處於超時工作的狀態（而不是偶發的超時）：也就是超過三周都在六十小時以上工作。Evan Robinson－<六堂課告訴你為何加班不可行>的作者則認為只要每周超過四十個小時就叫加班。Dave Swanson－美商藝電的工程監督則認為現在大夥所謂的加班與他當時在奧蘭多的 Tiburon Studio 開始做 Madden Football 時的工時簡直就是一碟小菜。
We also want to acknowledge before going further with this discussion that there is a distinction between voluntary and mandatory overtime. Within our survey there was a statistical difference, in terms of project outcomes, when respondents indicated that overtime was voluntary. Voluntary overtime is not surprisingly less harmful but there are still studies that suggest, any time we work long hours, fatigue effects the quality of our work. This seems to be especially true for “knowledge workers” who have to make decisions or solve complex problems. In game development this may translate to more bugs or poor technical decisions that result in performance issues or generally poor quality or stability of the game. We’ve included some links at the bottom of the article where you can go review some of the research that’s been done. Don’t just take our word for it.
Let’s talk a little bit about why game teams crunch. Seppo Hellava, Co-Founder and Creative Director at Wonderspark, made a direct correlation to ship dates. He was proud to tell us that his teams don’t crunch because they do not set hard deadlines. They do this by working in short iterations with frequent small releases. This minimizes the risk of introducing a major problem with any release, allows them to quickly change course if they learn something new; either from development or from those playing the game. Because there are no hard deadlines, they can manage the workload and focus on just the next iteration, not a long feature list with pressure to get some major release out for a marketing window.
接著我們來談談為何遊戲團隊會加班。Seppo Hellava－Wonderspark 的創意監督及共同創辦人，訂定了發售日的計算方式。他自豪地說他的團隊不加班，因為他們的發售日期是彈性的。他們透過快節奏的短期循環來開發及釋出產品。這樣的作法降低了大問題發生的風險，讓他們能夠快速學習新知－不論是從開發中或從試玩中，同時立即改變方向。為他們沒有明確的時程底線，他們可以調整工作時程，只專注在下一循環的工作上，而非讓長期的工作清單及因為營運目標而設定的發售時程造成團隊的壓力。
Of course, this doesn’t work for many game studios with pressure from publishers or marketing to ship a game for Christmas or release within a very short window to hit some specific target market or hit some revenue goal in a particular fiscal period. It can be extremely hard to avoid some degree of crunch when faced with a hard deadline and too much work left to do with limited time to do it. Joe Hoff, Director of Software Engineering at InContact and a former Development Director at EA Sports, put it this way:
“Crunch is caused by designers trying to shove 10 pounds of shit in a 5-pound bag. They will always insist on more features than can possibly be done within the available capacity.”
當然，對於很多接受發行商或營運單位壓力－要在聖誕節發售或是在下一個會計結算前就要達到特定的營收－的遊戲開發工作室並不能這樣運作。當面對固定發售日的時程而還有很多工作必須完成的情形下極難避免一定程度的加班。Joe Hoff 前美商藝電運動開發監督及 InContact 的軟體工程部門監督說道："加班是肇因於設計師總是試著把十公升的大便裝到五公升垃圾袋。設計師總是堅持要有更多的特色，比目前能做到的都來的多。"
Yes, there are those that believe crunch is the result of poor management and a willingness to over commit, promising features that will force the team to work overtime to deliver them. Counter point: Warren Spector is famously quoted as saying:
“Crunch will always exist in studios that strive for quality. Look, I’m sure there have been games made without crunch. I’ve never worked on one or led one, but I’m sure examples exist.”
沒錯，加班是由差勁管理及過度承諾造成，這樣的情形逼迫團隊超時工作交付成果。但也有可能是如 Warren Spector 所說："加班會永遠存在於追求品質的工作室。聽著，我確信一定存在某些遊戲團隊不需要加班，但我還沒有碰過或帶過這樣的專案。"
We talked to Kate Edwards, the Executive Director of the International Game Developers Association (IGDA), about crunch in the games industry. Kate is passionate on this subject and the IGDA recently announced an initiative called the Crunch Comp Initiative. The IGDA has been studying quality of life in the games industry for years, running an annual Developer Satisfaction Survey (DSS). In 2015 the DSS respondents reported the following:
“62% indicated that their job involved crunch time; 58% said they were in crunch more than twice in the last two years; and 61% said that crunch time is expected at their workplace.”
我們與 IGDA 的執行監督 Kate Edwards 談過遊戲產業的加班問題。Kate 對這個議題充滿熱情，而且 IGDA 最近才公布"Crunch Comp Initiative"這個業界自清行動。IGDA 多年來已經研究了遊戲產業內的生活品質問題，每年都針對開發者做問卷統計。在 2015 年的問卷結果表示："百分之六十二的回答者有加班（crunch）狀況；其中百分之五十八在最近兩年這樣的情形超過兩次；百分之六十一的人說在他們的工作環境加班是被期待的。"
In our interview, Kate also stated, “crunch is almost entirely avoidable, even in creative works, because it goes back to that creative vision and that creative vision includes a certain agreed upon feature set. Someone has to draw a very hard and difficult line and say, ‘that is a great idea but we are not going to do it in this version’.”
She commented that in the DSS, respondents often blame crunch on inexperienced management. When she lectures she has a slide that says: “Crunch time: In every other industry it’s called it ‘poor project management’”.
We talked a little bit about the conflict between creative leadership and project management, who are trying to hold that line and say “no more scope” but the creative leadership is fighting to add more features (recall Joe Hoff’s quote above). Who wins the struggle between creative leadership and the passion to make the game the best it can be and project management, who know that scope creep can cause the team to crunch?
Kate commented that we have to make sure that when considering new scope, that could cause a team to have to crunch, that leadership has to really recognize the impact on people’s lives. She also reminded us that many of these companies don’t compensate folks for working later so it should not just be assumed that because the new feature will be awesome that people should be willing to work extra hours to make it happen. She also suggested that the decisions are often made by managers who aren’t going to work long hours with the staff they are expecting to crunch.
The 2015 DSS found that as many as 37% of developers are not compensated for extra hours they work.
And the debate goes on… Many teams crunch and many leaders make a conscious decision regarding scope versus capacity, believing that crunching will allow them to publish a better, more successful game. So how do we prove one way or another if crunch is beneficial or harmful to most game development projects? We need data. We have a proposal for a way that you can do your own data gathering so you know what the real affect is of working long hours on your project.
We believe that productivity can be measured as combination of three measurable factors:
Here’s how this works, and it isn’t rocket science.
Volume simply measures the amount of work being done during some period of time. You can use whatever unit of measure makes sense for you. It might be story points, work items, bugs fixed, man days of estimated work… You almost surely are already collecting this data point.
Efficiency is simply the amount of effort required to deliver the volume of work you are measuring. You may already collect this data too. For example, if your team delivers 20 story points with 5 guys working 40 hours in a one-week sprint your efficiency rating looks like this.
5 * 40 = 200 man days of work time divided by 20 units = 10 hours per unit. You have an efficiency rating of 10.
This assumes that all 20 story points were delivered with no defects or work remaining. This is not very realistic which is why we also need to measure quality to really understand the true productivity.
The easiest way to add this component to the calculation is to measure the amount of time that members of the team spend on bug fixing or rework. You don’t have to be precise in this measurement but if you ask folks to honestly ball park the amount of time they spend each day on bug fixing you can get a good indication of how much effort goes into resolving issues.
Using the same example from above with 5 guys working 40 hours in a one-week sprint, let’s say we identify that they each spent about 4 hours fixing bugs. That is 20 hours total or about 10% of the total time spent on bug fixing.
We subtract the bug fixing percentage and calculate the “productivity” rating. (10 – 10%) = 9.
Now we have a bench mark. In a good one-week sprint, with 5 guys we should hit a productivity rating of 9. If you measure this over time you will expect to see some fluctuation, hopefully with productivity increasing as the team gets more experienced and learns to work together more effectively.
The trick is to see what happens when people start working longer hours. We expect to see an initial increase in the volume of work completed with no significant change in the overall productivity rating. The question you need to answer is: what happens after 2 or 3 weeks in a row of working longer hours? Research in other industries has suggested that by the third week you will see all three metrics (volume, efficiency and quality) decline. Meaning, as people get tired they get less efficient and they will generate poorer quality work, be less innovative and make poorer decisions.
If you believe that crunch is okay and a natural way of working in creative endeavors than we implore you to measure productivity in this way and see for yourself what is really happening. Research suggests that after as little as 3 weeks of working 60 hours per week individuals may actually produce less than if they had worked 40 hours each week during that same period. Don’t you want to know if crunch really works? Measure it and find out! Even better; measure it and then share your aggregate data with the Game Outcomes Project and the IGDA so we can share what you’ve learned with the broader industry.
假如你相信加班在創意工作時很正常，那麼我們希望你能測量這個生產力，自己看看會發生甚麼事。研究指出在每周六十個小時工作下，僅需要三周，每個人的產量就會比每周工作四十個小時還要低。你難道不想要證明加班有用嗎？自己測量看看吧！更建議你測量之後，與我們 “遊戲專案為何成功團隊" 及 IGDA 分享，那麼我們也會把你的數據列在我們的研究之中分享給業界。
We believe, from both the Game Outcomes Project survey and from the IGDA DSS, that most people working in the games industry have some period of crunch each year. We concluded from the Game Outcomes Project survey that for those that participated in the survey; teams that crunch have less successful projects than those that don’t.
我們相信，從 “遊戲專案為何成功" 的問卷到 IGDA 的報告，大多數在遊戲產業工作的人都每年都曾經加班工作。從問卷中得到家班的團隊卻獲得較少的成功成果。
From the personal experiences of members of the Game Outcomes Project team, and industry leaders we have spoken with, there appears to be a correlation between fixed ship dates and issues with managing scope, that contribute strongly to the need to crunch. Teams that can avoid hard deadlines or can manage scope well are less likely to rely on crunch to get their game released.
For those teams that expect to crunch, and intentionally commit to scope that they know will require crunch probably have no real data to know if crunch actually gives them the boost in productivity they want. It is also possible that crunch causes harm to employees that results in physical and mental health issues, but that’s a topic for another article.
It is possible to measure productivity and, if your teams crunch, you should want to know if the practice is effective or not. Maybe, “you can’t handle the truth!” Stop just believing that we should all expect, and be happy to work 80 hours a week and find out if that is really the best way to be successful.
Question we still need to address:
How long does it take to recover from crunching? If you work 80 hours for 1 or 2 weeks (or months) and then reduce your hours back to something closer to 40 hours, how long before you have recovered?
What if we never have a regular schedule and simply work 60 hours per week all the time? If we don’t have a baseline metric of productivity at 40 hours per week, how do we know if 60 is more or less productive?
How important is diversity of activities to creativity? There are some that would argue that people who work too many hours lose their ability to innovate and need outside activities to stimulate their creativity. How do we know when that is a factor for our staff?
We are here to help and to learn with you. We want to improve the industry that we work in and are all passionate about.
The Game Outcomes Project is an independent industry/academic partnership. It is driven entirely by intellectual curiosity. Our mission is to survey a large number of game developers and statistically analyze the connections between game development team culture and project outcomes.
The Game Outcomes Project team currently includes Paul Tozour, Lucien Parsons, Zhenghua “Z” Yang, NDark Teng, Eric Byron, Ben Weber, Karen Buro, Joe Frontiera, and Carlos Puertas.
“遊戲專案為何成功"團隊成員有 Paul Tozour，Lucien Parsons，Zhenghua “Z” Yang，NDark Teng，Eric Byron，Ben Weber，Karen Buro，Joe Frontiera，以及 Carlos Puertas。
Leaders who contributors to this article included: Seppo Helava, Joe Hoff, Clinton Keith, Keith Fuller, Dave Swanson, Andre Thomas, and Kate Edwards.
此篇文章訪談的領導者：Seppo Helava，Joe Hoff，Clinton Keith，Keith Fuller，Dave Swanson，Andre Thomas，及 Kate Edwards。
IGDA resources on quality of life
IGDA Developer Satisfaction Survey
Interview with Kate Edwards
Kate Edwards 的訪問
Clinton Keith’s blog – “Death March Crunches: 10 Causes and Solutions”
Clinton Keith 的部落格文章：邁向死亡的加班衝刺：十個原因及解法
Danial Cook article on productivity
Danial Cook 生產力的相關文章
Sleep deprivation study from the US Army
“Proof that Positive Work Cultures Are More Productive” from the Harvard Business Review
“Psychosocial working conditions and the utilization of health care services”
Evan Robinson included links to several studies within his crunch article, which is linked within the article above.
前述 Evan Robinson 的文章連結中包含了數個研究。