104276-why-do-we-have-unintended-new-bugs-and-issues-after-every-patch

Page 1, Page 2

Content
{| style="width: 100%;"

How many source code branches does your team work in (not meant as a snarky tone, just genuinely curious)? At any given time our code environments look like the following: Dev (Drop 7), Dev (Drop 6), Dev (Drop 5) | Staging (Drop 4), PTR (Drop 3), Live (Drop 2) Our Dev environments run CI (continuous integration) with hundreds of our devs checking in work on a daily basis. For shipping environments, we have some proprietary tech we use for integrations and code promotions (our Build team rocks and they work really hard). Branches iterate on a monthly basis (imagine a conveyor belt moving from left to right.) We also have a fairly aggressive monthly Live release cycle which can be seen as follows:

1 Content Drop We iterate our branch numbers when this occurs and this represents major changes in our database structures, system overhauls, new features coming online, etc. Typically, large scoped features and high risk changes are deployed with Drops (Sabotage, Blighthaven, The Defile, etc.) Content drops contain literally thousands of changes made by hundreds of developers while the branch resides on our Dev environment

2 - 4 Weekly Patches We iterate build numbers and not branch numbers with these Smaller and manageable changes are deployed (sometimes we get whacky and throw something big in) These only include promoted code (we also have another system we use for fixing things directly on shipping which I can't delve into details on)

~# Hotfixes (unknown number of hotfixes depending on severity of issues) We do not iterate build numbers externally when hotfixes occur Small and targeted fixes for exploits/critical issues occur during hotfixes These only include promoted code (we also have another system we use for fixing things directly on shipping which I can't delve into details on)

Here's just a few examples of how something may break:

We don't catch something in QA (hey, we're not perfect and we're the first to admit that, but damn do we try hard and are we pretty diligent in what we get.  FYI our QA team is really on the ball with a lot of things and they deserve a badge and a hat.) During the build and/or code promotion process to Live a code dependency or integration issue occurs and we break an unforseen system (typically Build/QA catches this) System X comes online and System Y depends on it in such an obscure way that until the code is live in a production environment we don't see the issue (today's crashes are a good example of this, we needed a LOT of players online doing a LOT of different things for the system to stress to the point that causes the break)

It's a lot to juggle and hopefully that helps shed some light on things. Cougar also has a good post on patches here: https://forums.wildstar-online.com/forums/index.php?/topic/16069-carbine-processes-patch-types/ Cheers

Edited August 1, 2014 by Virtue


 * }
 * }

{| style="width: 100%;"

I don't think they're rushing to keep up with non-ideal deadlines, they're trying to keep up with player expectations. Which is admirable, but foolhardy; they'll never fix bugs and improve situations fast enough for us. The problem with being an MMORPG game developer at launch is that, if you're listening to the forums, the game was expected to be perfect when it launched, so all the player base expects the fixes will at least be done by the end of the week and the fixes will all be perfect coming out the other end. I think, unfortunately, instead of being intentional hyperbole on most people's part, some of them are actually believing that rhetoric themselves. I'm grateful Carbine's taken player concerns in such considerably large swallows and they're testing as much as they can on their release schedule, but they have to come to Earth and realize that they will never, ever be fast enough to be above scorn and ridicule. Then there are things that happen because the game engine is new and a lot of the tools they are using are new. Take for example the bug that reset everyone's world boss progress in the attunement quests to zero. To know that would happen, the team would either have to intuitively know that it would happen, or they'd have to see it while testing that, if someone was killing world bosses in in-house testing or on the PTR, the fix wiped out the quest. The game's a bit too new to know it intuitively yet (they said the system they tripped is a developer tool that automatically adjusts if they have to change a quest, but it shouldn't have worked there), and nobody on the PTR just happens to be testing that phase of attunement to know that. And finally, of course, bugs in your bug fixes, like bugs everywhere, are just going to happen. That's why WoW routinely gets hotfixes and patches right after content drops, and they have staff on orders of magnitude larger than Carbine, a much more familiar system for them, and a lot more leeway from the fanbase to test. They're some of the best developers on the face of the planet at Blizzard, and it still happens that the touch-ups you apply, no matter how much they're tested, will inevitably run afoul of something once player pressure has been applied.


 * }
 * }

{| style="width: 100%;"

QFT. If Dev's were perfect, we tech support people would have a lot less work to do. Call it "job security."


 * }
 * }

{| style="width: 100%;"

lol, so you program for nuclear power plants? seriously. if you dont, can you with no doubt say that they dont have bugs? that they dont find mistakes after a software update? they cant afford mistakes, but unless you code for them can you honestly say... they...dont...happen?


 * }
 * }

{| style="width: 100%;"

I've read some hilarious things on these forums but this, this almost made me wee a little bit I laughed that hard.


 * }
 * }

{| style="width: 100%;"

This is all people need to know.


 * }
 * }

{| style="width: 100%;"

Sophmoric rhetoric at it's best. News flash; the software run in nuclear powerplants is basic as it gets; and it is still rather buggy. There are actual metrics for how bugged software is (usually in some iteration of the form X bugged lines of code per Y units of code). There is NO robust commercial grade software in which there are no bugs, period. This goes for MMO's, operating systems, systems control software for nuclear, energy grids, financial transactions, etc etc etc. MMO's are massive dynamic programs with a scale and complexity rivalled by only a few other classes of programs when raw scale, interactivity, and update requirements are considered.


 * }
 * }

{| style="width: 100%;"

ROFL!!!! God, I'm sorry, I quoted the wrong post. I was trying to respond to the quote above yours. This one.

I need to go to sleep. I was seriously looking around like, "I... I don't know what that has to do with that." I apologize.


 * }
 * }

{| style="width: 100%;"

You guys are the ones saying "how hard can it be" and a Carbine has clearly stated why their can be issues, yet you still *cupcake*ing say "omg that's bullshit it doesn't/shouldn't work that way cause I know a guy who works on these shitty iphone apps that *cupcake*ing suck ass and he says it doesn't work that way." So your defense about my response is the biggest load of hypocrite crap you could ever spout... Left hand needs to meet right hand yo cause clearly they have no idea what they are each *cupcake*ing doing at the moment.


 * }
 * }

{| style="width: 100%;"

Absolute statements, all the kids are making them these days.


 * }
 * }

{| style="width: 100%;"

personal responsibility for a game not working as advertised? do you even understand the things you say?


 * }
 * }

{| style="width: 100%;"

Software *design* is about designing software where coupling(dependencies between components that should be independent) is kept to a minimum. In a well designed system changing something generally only affects things directly related to it, or if they affect farther things, it's via a chain effect. That's not to say that it's easy, it isn't, but when things are breaking unrelated things that's a sign of a poor design. Another thing that's common in others but less so in mmo's is unit testing. Testing that all systems work as you expect them to. This is a lot harder for some parts, but it should be absolutely possible to automatically test many of the systems. At least it is if coupling is minimized and systems are compartmentalized. Game programming is kind of the wild west of programming: it's a more creative enterprise so it's quite common for people to play fast and loose with these rules.Tbh, these rules are boring as hell, but they're what keeps the madness at bay when doing complex integrations. And regardless of the above one thing is sure: for a monthly patch at the very least, there should be a bunch of people giving a once over of all the primary systems: check each ability is working as planned. The issues mentioned in the original post should have been caught. The servers dying due to circumstances that couldn't be tested? That's not so easy and can be forgiven, but stuff like the stealth and wild barrage are pretty obvious and should've been easily  caught. This wasn't a hotfix, it was a monthly fix which should have had a decent amount of testing


 * }
 * }

{| style="width: 100%;"

Can you provide screenshots and detailed notes of all the testing you did yourself on the PTR, I mean most of these patches were on the PTR before release presumably with the same bugs. Just send me all the notes you took and copies of all the bug reports you submitted while you were on the PTR testing. You were the last line of defense for the bugs, I blame you for patches being released with bugs, I mean since we're playing the blame game and all.


 * }
 * }

{| style="width: 100%;"

I enjoyed reading this post as I pretty much share the same opinion. Maybe now our little Chua will be able to understand what you meant before. You really have to take it slow with some people.


 * }
 * }

{| style="width: 100%;"

General Motors wants word with Human about recalled cars Toyota wants word about speeding Prius


 * }
 * }

{| style="width: 100%;"

or the loot bug that went on and on


 * }
 * }

{| style="width: 100%;"

apparently at carbine every branch releases something and the other branch is not aware of what just happened. You and me have a different understanding of mainline and how products are being released or shared by departments/branches. It seems to me that at the moment one department blames the other and devs running all over the forum or livestream trying to fix what the other person just typed. It makes no sense lol.


 * }
 * }