Two guys made an MMO: The Growtopia Postmortem
by Seth Robinson on 06/19/13 10:26:00 pm Featured Blogs
（2013 年年中 Seth Robinson 撰寫他們 2013 年初上市作品 Growtopia）
So, being the master of game marketing that I am, I’ve carefully been totally silent here on my blog about my latest game, despite being released five months ago.
The game now regularly hits 3,000 players online at once and has been financially a huge success. (At least by my humble standards!) It’s been a top 1000 grossing US App Store game for the past four months and doing similarly well on Android – all without being featured and zero paid advertising.
11-9-2014 Update: We’ve grown! We now regularly hit 40,000+ players online, have 5+ million users and around 300K daily active users. 80+ million worlds have been created by users.
Today I’m finally sitting down to document how this project took shape as well as the trials and tribulations.
September 8th, 2012: Six pictures
After my highly competitive multiplayer online game Tanked was more or less finished, I began thinking aboutduct a new game. Something that would capitalize on the networking client/server experience I’d gained but apply it to a fresh idea with simpler controls that would be more accessible.
在我上一個多人競爭的線上遊戲 Tanked 之後，我開始思考一個新遊戲，這從我撰寫連線機制的經驗而生，想把它擴展到一個更簡單操控的點子上．
Despite Tanked being 3D, I decided to do the game in 2D – faster development and we could target even very crappy mobile devices.
儘管前一個作品 Tanked 是三維，這次我決定做二維遊戲．這樣開發速度更快，而且可以讓更多普通手機遊玩．
With Tanked, I felt my fatigue in “doing it all, alone" was the cause of my non-existent marketing efforts (I’ve STILL never done a youtube video for Tanked? REALLY?!?!), as well as reluctance to add the final layers of polish. I needed a partner, someone to provide a mutual motivation as well as handle the artistic needs of the project.
在做 Tanked 的時候我發覺自己做又沒有市場營銷幫忙令人疲倦（記得 Tanked 似乎沒有一個官方釋出的影片）．尤其是在開發過程中的最後打磨階段．我需要一個能夠互相砥礪又能提供美術素材的夥伴．
So with that in mind, I presented six mock-up screenshots to long-time IRC pal and fellow indie, Mike Hommel, aka Hamumu to entice him to come aboard.
因此我把這六張概念圖發給了我的朋友 Mike Hommel 也就是 Hamumu．並邀請他加入．
Here are the original shots (created using stolen sprite rips mostly) for your viewing displeasure (I’m no artist!):
[圖右下：玩家間互動甚至 PVP 的方式]
We changed and improved the ideas represented in these mockups over time, but stayed focused on the general goals.
He agreed to give it a shot. Like so many indie collaborations, I suspect he was worried we’d never reach beta. Hell, me too!
I think I predicted 4-6 months to finish it. This is usually where I would chuckle at my foolish naivety and admit it took years longer than expected but… not this time?!
The reason things went so smoothly? Well, I guess probably because we had very few “unknowns".
In the past, we’d both already made 2d tile based platformers, collision code, networked games, mysql based projects and websites, and done cross platform development. Using my Proton SDK insured we could run on eight platforms (nearly) out of the box.
在這之前，我們都已經做過二維平台遊戲，也有了碰撞，網路，資料庫，專案，網站，以及跨平台的開發經驗．使用 Proton 的函式庫，我們能夠釋出幾乎八個平台．
We figured the only real unknown was what our max player limit would be and how things would scale up. I had guessed conservatively, 600 players online would be our max. Luckily, I was wrong, the original server actually was able to handle much more.
The above shot shows four instances running around in the same world. All graphics are just DrawRects for now. The collision is incredibly simple and as a result gives us very few problems later as we tweak it for special tiles and items later. The game is written in C++ with GL/GLES.
上圖是針對相同地圖產生四個客戶端的繪圖結果，透過方格計算，碰撞其實無比簡單，因此省下了之後的很多麻煩．遊戲是用 C++ 及 GL/GLES 寫的．
I think one reason this project went so fast is I was able to fight the urge to go overboard on everything like I usually do. There are no fancy 2d physics. I mean hey, we’re not making Limbo here. It would be overkill. I think it helps that I had already done the box2d physics platformer thing in a previous project so I didn’t really feel like I had something to prove, if that makes sense.
我認為能夠這麼快速釋出的原因在於我急切地在任何方面拋棄掉負擔．沒有複雜的二維物理．因為我們不是在做 Limbo 這種產品．那種碰撞對我們就是過度設計．我在之前專案就已經把平台遊戲方塊碰撞的物理程式寫好了．我不需要在這個專案證明甚麼．
I’ve never actually met Mike in person, he’s in California and I’m in Japan. But if anything, that probably helped – we work on the code base (he programs too!) in (roughly) day/night shifts which help avoid svn code conflicts.
我從未面對面見過 Mike 本人，他人在加州，我在日本．但如果要找出一件加速效率的事情的話，我猜是時差，因為兩個人不會同時改程式，這樣可以避免程式碼版本控制的衝突．
We use IRC and Google Docs to communicate – spreadsheets for tracking hours and financial stuff, and the drawing board to share screenshots and ideas. The best part is we have a full history of everything we’ve done, that’s where I’m getting most of the screenshots for this article!
我們使用即時通訊軟體跟 Google 文件溝通．用試算表紀錄財務，用繪圖板來分享截圖跟點子．最棒的是我們紀錄了整個開發歷史，所以我才能現在給你們看這些截圖．
介面開始於 Hamumu 的概念圖
In roughly five weeks, the game looks pretty similar to its current form.
November 30th, 2012: The game is stealth-released on Android as a free beta after three months of development
在 2012 年十一月底：這個遊戲經過開發數個月祕密地在安卓上外測．
We were unable to use the original name we wanted (Buildo) as someone else was sort of already using it. After a VERY painful few days we finally agree on Growtopia. I really couldn’t get anything done during that period, strange that stress of finding a game name could grind everything to a halt like that. I was never totally satisfied with the name, but it’s very searchable so that’s a perk. The domain name was taken so we had to live with growtopiagame.com.
我們本打算用 Buildo 這個名稱，但是這名稱已經被用了．在一段非常痛苦的幾天後我們同意釋出的名稱就叫 Growtopia．那段時間我幾乎無法做任何事，選個好名字讓我們暫停了所有開發．我並不完全滿意於這個名字，但是在搜尋上有優勢．所以網站的名稱就決定了．
IAP is disabled initially as we expected to reset the universe a few times. (We ended up never having to do that)
There were no “World Locks" yet, so people tended to clump together in little towns, locking their areas with the smaller lock types.
Despite the total stealth release (I didn’t even mention the beta on my website) the player base quickly grows. It’s the perfect design for PR slackers like us, very viral in nature due to the social aspects.
January 9th, 2013: The game is released on iOS, and the word beta is removed from the Google Play description
The release! We get a boost from Touch Arcade, Pocket Gamer and IndieGames who all post our teaser video. (we sent it to them and a few other sites, that was pretty much it for our marketing campaign)
正式釋出，Touch Arcade，Pocket Gamer，以及 IndieGames 都貼出我們寄出的的試玩影片，正如我們的行銷計畫．
We stress out as we watch the users online grow. 200 users, 300 users.. 600 users online at once! Will the server die? We upgrade our server three times in two weeks to handle the increase, wondering if after the initial surge we’ll need to go back to the cheaper one. Servint, our data center, did a great job of quickly migrating us from server to server as needed.
Life after release, where we are now
Like a newborn, we found the game screaming for our non-stop attention. We find ourselves constantly putting out fires and dealing with issues. 600 hundred forum posts a day to read, hundreds of daily support emails.
There is no finish line, there is no “done" – we’ve basically worked full time on the game since its release.
Our game has an extremely sensitive virtual economy that could be decimated in only hours by a rampant bug.
A single server crash can cause an hour of lost work per player online. A single day roll-back on a busy day now would mean instantly vaporizing SEVEN HUMAN YEARS worth of effort.
These things weighed very heavily on me in the beginning. I had trouble sleeping and would check the server throughout the night to make sure it was still running correctly. After a while though, you sort of reach a certain level of numbness/comfort with it all.
I guess I’ll forgo the usual “what went right/wrong" and just illuminate the most important/damaging events and how we dealt with them.
Our first penis
Well, we knew it had to happen. I appeared in their world and gave them five minutes to remove it, but secretly I was sort of proud, it’s sort of a developer achievement to have someone miss-use your creative tools like this.
As our player count grew, we found our player discipline tools inadequate. Over time we beefed them up.
The problem with tape
Instead of simply muting people, we thought it would be much cooler to visually duct-tape their mouth shut. When they attempt to speak, only muffled noises come out.
The end result wasn’t what we expected. Being duct-taped quickly became the goal. When players saw us, they’d mob us yelling obscenities, hoping to be taped. We had to make a new rule: “If you try to get taped on purpose, we’ll just ban you" to keep it under control.
The item duping
Due to a bug a patch introduced, it became possible to log on with two instances of the same player for a short while if you did it at EXACTLY the perfect time. Within hours a group of Asian players had figured it out and were duping like mad. I remember using google translate on Growtopia related Korean message boards trying to figure out the method used.
Knowing it was happening, but not understanding the modus operandi was VERY stressful, we had to spy and watch them doing it. We figured it out and patched the server. We hand deleted the ill-gotten gains as best we could (we didn’t want to roll back!), but it quickly becomes impossible to track as items are traded. For a while, we had the server report anyone with > 100 world locks and we’d just assume they were holding duped items and punish the account. Nowadays that wouldn’t work because many legitimate players have earned that much wealth through normal gameplay.
The bedrock bug
One day fairly early in Growtopia’s lifetime (I think it was still in the wide beta on android) we made a horrible mistake with user security levels which let anybody destroy bedrock and white doors. (Normally a player can’t destroy these tiles, as means player can fall out of the world at the bottom, and can’t enter worlds the normal way)
Twenty minutes had passed before I’d noticed the frenzied broadcasts that excitedly shared the bug – not only were people destroying normally impenetrable blocks, those blocks were giving seeds, which would grow into trees that would eventually yield more of the forbidden blocks, letting the user place them!
This bug really didn’t cause too much damage, but we were running around fixing holes in levels and replacing missing entrance doors for weeks. We also added a filter that would remove those items from peoples inventories.. but I wouldn’t be surprised if there is still a bedrock tree growing somewhere out there and you do still see the occasional bedrock piece missing from older worlds.
The IAP hackers – “Someone is giving away world locks"
Like most developers, I’ve grown used to the idea knowing that people are pirating my games. I don’t stress too much, it doesn’t ruin the experience of others for the most part.
But if thousands of dollars worth of gems are stolen via fraudulent means, it has a real impact on us as it is pumped into the game economy.
We’d watch would-be thieves dump their contraband in remote locations. We’d wait (invisibly) and see who came to pick it up, track down the persons main hideout and ban them all. Eventually we upgraded our server to do full IAP receipt checking, but life had a certain cops ‘n robbers taste to it for a while.
We still have issues with stolen credit cards and chargebacks, but it’s now at manageable levels for the most part.
The Tapjoy hack
Tapjoy is a great way to let players watch a 20 second commercial, to buy gems using their time, and only if they choose to. (They get 90 gems, we get 5 cents, or whatever) However, due to me using it wrong (similar to IAP, I was using my code from previous single player games I worked on rather than custom stuff with full server verification) it was very vulnerable to client hacks, as illustrated in the image.
I wasted a lot of time writing code to detect hacks and such, when I should have just re-implemented Tapjoy the correct way, which I eventually did anyway. It now runs flawlessly.
我花了很多時間來偵測這些駭客行為，我應該直接把 Tapjoy 的部分重寫才對，因為最後還是真的重寫了．現在終於運行正常．
The virtual economy made item trading and bartering very important. The original release had no secure way to trade, only “drop trading". Naturally scamming was prevalent. We told people “Never drop trade with people you don’t know!"
We introduced a robust trading system as quick as we could.
The blueberry hack
Due to a server vulnerability, it was possible to hack a client to give you unlimited blueberries. (Basically, when using it to eat, a special case happened where it didn’t check if you actually had any left)
We were perplexed why a certain group of players had so many blueberries, it didn’t make sense… someone eventually emailed us and explained the trick and we quickly patched it. Thanks, sir.
The wall hack
It turns out the first thing that people do in a game like this is use a memory scanning utility (a “trainer") to locate important memory addresses (speed, X/Y position, etc) and modify them. We realized wall hacking would be possible, but didn’t think it was a big deal due to our lock/ownership systems. (If someone gets into your locked area, it doesn’t really matter, the server wouldn’t let them take anything)
However, players liked to drop items around their worlds, which presented juicy targets for wall hackers. We quickly combated this with clever server + client hacks which worked pretty well, but in the end we removed all doubt by adding full A* path finding on every single player movement. (this wasn’t possible for speed reasons until the V2 server upgrade.. more on that later)
The black day – the rollback
But it wasn’t the hackers or credit card fraud that did the worst damage. It was us!
On February 23th our worst nightmare became a reality. We’d inadvertently made some changes that caused certain very cheap items to give out a high number of gems when “recycled", this created an infinite money loop. As soon as it was noticed, someone “broadcasted" it which meant every single person online knew how to earn unlimited gems. The entire economy was trashed in only minutes.
We had to roll back almost 24 hours worth. (We do automated backups daily.. and sometimes I do extra backups before a serious patch, but today I didn’t for some reason)
That weekend was dubbed “Apology Weekend" and we worked quickly to add perks and gems bonuses to try to make it up to the players. If you ever see a “Rollback Plaque", it was earned by someone on that weekend. We wrote a program to restore all gem purchases that had been wiped, with double the gems.
This was a wake-up call to more seriously test things before patching. It’s tough trying to surprise your audience with something new (we’re very secretive) and yet wanting to properly test it. We now do have a fully functional beta test server that we can route normal players into.
That day was easily the absolute worst for me personally. I’m happy to say we haven’t had another roll-back yet.
Too many users
In mid April we hit 2,000 concurrent users on weekends and our server began to buckle. Round trip to punch something could take a full second and people were constantly being disconnected. We added live profiling support and narrowed down the slow down to world creation and enet packet sorting.
We decided I would write a “V2 server" upgrade in a separate svn branch that would take advantage of multiple cores; it made no sense that our hardware had 16 cores and 32 threads but our entire Growtopia server process was run in a single thread.
Meanwhile, Mike would handle running active game and adding items and preparing to move them over to the new server.
同時 Mike 會負責處理線上的問題，準備搬移伺服器．
We had a lot riding on the V2 server, would it really solve our problems or fail miserably? It’s very difficult to fake 2,000 users, so we released a client upgrade that could smoothly switch between the old server and the new beta one.
When things looked good, we just pushed a button and switched the real database over to the new server, if it died, we could just switch them back. Anyway, it worked beautifully and we haven’t outgrown it yet.
Here’s how I explained it to users on our Facebook page: (it was actually only a software change, we didn’t change hardware. But this was the result as far as latency and gameplay)
The freemium dilemma
After railing against slimy abuses of the freemium model myself, I vowed to do it ‘right’. Here is what we did:
在對自己的免費模型檢討（連結： https://www.codedojo.com/?p=1251 ）之後，我傾向做正確的事，以下就是我們的作法：
Only one soft currency, that can be earned in the game.
No paywalls, you really can get every single thing in the game without buying stuff. But more than that, the game is fun and works fine without paying – no slogging along at a snail’s pace and grinding required to do stuff.
$10 is the biggest IAP.
$30 limit per day limit to help situations like “my kid bought $500 of stuff accidentally", as far as I know, no major mobile freemium companies do this, but really, they all should! Actually, Apple should probably send a “ARE YOU SURE? Click OK to allow more IAP in this game" type email after anybody spends too much in a game to verify it wasn’t a mistake or a kid.
No annoying push messages, spamming, email sharing, etc
No popups asking you to buy stuff. No pressure selling. Don’t buy stuff, you really don’t have to.
The expensive status/vanity items don’t have any special powers as compared to the cheap items, they just look different, which in a situation with hundreds of thousands of players to compare items with, is a real value, of sorts. The rarity creates the value.
We don’t want whales, we want schools of tiny fish who will pop in a few bucks now and then. If a customer ends up spending 5 cents an hour that would be great, and I think a decent value.
We want to make enough to continue working on the game and adding content/events instead of the normal “release and forget" model.
A few times we saw someone making daily purchases that seemed strangely high and emailed their Google purchasing account email directly to verify that everything was ok. (We thought it might be a kid using a parents’ device) I don’t think we ever got a reply. I think Apple and Google should be doing the emailing really, they have more and better information than we do. We will disable IAP on any account by request. (the option won’t even show up in the store in that case)
有幾次我看到玩家有異常購買的情況我都會寄信問 Google 的購物帳戶確認是否有問題．（我們以為是未成年人誤刷）而從未獲得回應．我真的認為 Apple 與 Google 應該這樣做，他們有比我們更多的資訊．如果有這樣的情況我們可以關閉遊戲內的內購．（這個選項在後台根本不存在）
I think overall we “did it fairly right", but I still have moral reservations about this model and know that we’re exploiting certain kinds of vulnerable individuals due to the nature of open-ended purchases, even if you don’t need them. In a single player game I don’t think it would ever be justified to extract $100 in IAP, but in an MMO where we have players with 1,000+ hours logged and who purchase presents for others, I think it isn’t totally unreasonable for an informed adult to spend that much.
But what is the line?
This is game where a kid can do a /broadcast and instantly be visited by fifty or more REAL people hanging on his every word because a prize is going to be given away. He or she can run any kind of game within Growtopia and have complete control over these guests, ordering them around and banning at will. I’m proud we created mechanics that allow something this amazing but also humbled by the damage this power could do to someone vulnerable. My own son (he’s nine) has spent $40 of his own money in Growtopia over the 200+ hours he’s played. He’s given most of it away to his “friends".
這個遊戲是讓小孩子能夠廣播，拜訪其他世界及真實朋友的世界來獲取獎勵．在 Growtopia 中有各種可以遊玩的遊戲．可以掌控對自己世界的所有權．我很自豪於創造了這樣的機制，讓人可以在不傷害其他人的自由下遊玩．我自己的小孩也花了他自己的零用錢四十美金以及兩百小時在這個遊戲上．大部分都是跟他的朋友互動．
If he’s learned a real friend doesn’t require a World Lock, maybe it was a worthy investment.
Giving away too much
A problem with our single currency is you can make around $3 of IAP per hour farming in the game. Our richest player has never spent a dime, and has $1000 usd worth of gems. I’m sort of proud of that on one hand, it illustrates our game is REALLY free and not a trick. But on the other hand, I don’t know what he should spend his 1,000,000+ gems on. If we add a 100,000 gem “Vanity item" in the store, eight year olds are also going to be pressured to use IAP to buy that essentially worthless item to keep up with the Jones’. (Worthless as far as functionality, but .. well, the value as trading power due to the rarity and cost is very real in a multiplayer society like this one)
It will be a challenge to figure all this out.
Email support system woes
Originally we handled support emails by replying and bcc’ing eachother. This .. doesn’t scale up very well. Eventually we setup with desk.com and now we can give much better service. When a customer forgets to tell us his GrowID, we can just look through his history and grab it.
一開始我們用郵件來做客戶服務．在遊戲變大的時候表現的不是很好．最後我們架設了 desk.com 來提供更佳的服務．客戶回信忘記寫帳戶名稱的時候我們還是能夠從歷史紀錄中取得．
It’s not yet written, John!
All I know for sure is I chose the perfect person to collaborate with and our combined efforts have definitely resulted in superpowers. Llike when spandex wearing heroes stand close together they can shoot larger balls of lightning. It’s like that but totally different.
In the five months since Growtopia’s release we’ve had 3.5 million+ worlds created by nearly 400,000 users. Last Saturday we set the record with 65,000+ hours logged in a single day.
在這五個月隨著 Growtopia 的釋出，我們有了超過三百五十萬個世界，四十萬個玩家帳戶．上個週六登入的玩家有六萬五千人．
I don’t see any shortage of ways to improve the heart of Growtopia, which is about giving useful tools to the player to let him or her be creative with both strangers and friends; but each addition and tweak is now taking increasingly long to test and safely deploy due to the size and scope of the existing user base. We can’t make any mistakes now, especially with huge additions going in this weekend. No rollbacks.
Growtopia was created by Seth A. Robinson and Mike Hommel.