It’s no secret that I’m a Star Citizen backer. As I’ve written about many times, it’s what brought me back to PC gaming. I jumped on the bandwagon back in late 2012 when the kickstarter started and if nothing else, it’s opened up a world I’d forgotten about for years and helped me discover a multitude of other games as I immersed myself in the gaming community. I’ve made new friends and the experience overall has been thoroughly enjoyable.
Since that fateful day in November 2012, I’ve watched over the years as the game has endured both its triumphs and its controversies. From those first excited days of seeing the hangar module get produced and realising I didn’t have a DX11 graphics card to be able to try it, to grumbles over delays (dogfighting module, social module, persistent universe, Squadron 42 vertical slice anyone?) and of course the various controversies including Derek Smart declarations of financial collapse, Crytek lawsuits, mega expensive game packages costing tens of thousands of dollars and more. Star Citizen has had it all. I’ve watched a lot of content, spent a lot of time in game, moaned at delays along with the rest of the community, been to CIG events and largely followed the game where I can (time permitting).
Over the years I’ve gotten to meet several employees of Cloud Imperium Games and have managed to get some online or physical face time with the people making the game. Several years ago I wrote a series of articles explaining at a high level how the software development process works (I’m a Product/Project Manager by trade, although I work in electronic trading in finance) and speculatively relating it to the development of Star Citizen. Well thankfully, I was recently given the opportunity to get some face time with Eric Kieron Davis, the LA Studio Director and Production Lead for CIG to talk about the development process and how things work there.
If you’re unfamiliar with the software development process, feel free to have a look at the series I wrote previously (part 1, part 2, part 3, part 4 and part 5) if you want to understand more. Without further ado, The Wccftech Exclusive Interview with Eric Kieron Davis!
The Star Citizen Interview
Wccftech: Thanks for taking the time to speak to me Eric! I wrote a series of articles about the software development process several years ago and speculatively associated with some of what I assumed to be the development process at CIG on Star Citizen so it’s great to get the opportunity to speak with you and verify some of this. So let’s start out with a simple one, are you guys using some form of Agile or Scrum?
Eric Kieron Davis: What I assume Adrian is that we’re going to talk mostly about the development practices, not the support teams and other areas. I joined in 2015 and the project has naturally evolved. What we determined early on is that we wanted to allow for the teams in each location to work closely together but in a system that works well globally. When we first started, you remember we had each studio was like a pod and operating in a silo. We’ve made hires over the years so those key words that you just used like Agile and Scrum are deeply embedded now and I’m having daily conversations with product owners and scrum masters. We have story points and as you know with the new quarterly release plan, every single release we have more data points which guide the timelines, so we’re taking that data and using it to project and say “here’s where we see 3.3 and 3.4”.
Over time as we have people that are with the company for longer, you get to know people and what they can and can’t deliver. It helps that as we get to know people, I know if I give something to “Jill”, she’s going to slam it out in a day, but if I give it to “Chuck” maybe he’s not so strong in that area and it’ll take him a few weeks.
So learning the team, the management structure, the code, these are all factors to developing the product so as we go through these cycles and these teams form, it naturally falls into place and you don’t have to think so hard anymore and you know that generally you’re going to have certain tasks going to one person or team naturally vs others.
The quarterly release cycle we have now, a little restructuring is a positive thing and it is helping us get better at delivering.
W: So talking about versions and releases, what’s your branching strategy? Presumably you have a working main that everyone contributes to, at some point in time you take a cut and say this is feature complete then you’ll stabilise it, merge and reconcile for the next one? How does it work with Star Citizen?
EKD: Adrian, you just literally said what I was gonna say. That’s exactly what we do, there’s no magic in that. When I got here, we used to call our main repository “main” in perforce funnily enough. We’ve renamed it so it’s called “gamedev” now which is our branch which holds everything. When we get ready to start releasing into a branch, we’ll call it SC3.2 branch and that’s our quarterly release. We have an inclusion/exclusion process to make sure certain assets go in or stay out if they aren’t ready and so it starts going from this big jumbly mess to this nice refined package that we’ll ultimately release to PTU.
For instance, we did a bunch of fixes to the gravlev system, we didn’t just want that to be on the 3.1 branch and then Squadron doesn’t have it so we’re naturally putting it back in to gamedev. Generally pretty much everything is going back unless we do what’s called a “no merge” if for example it’s something we put into the 3.2 release but we want to do a bit of a better implementation in the future so we don’t want to put it back to gamedev because it may break other systems we’re working on.
W: Yep I know that! My teams do a lot of hacks as well!
EKD: Hah! We have to be careful, it’s a term that a lot of people don’t like unless you know how to do it and how it works. Doing things in this way can meet objectives to give the community what they want but then we engineer it properly after.
W: So I’m curious, you guys have been doing the quarterly release thing for a while for Star Citizen and it feels like you’re getting into a regular rhythm now, how long are your merges back to main taking?
EKD: It’s a pretty quick process now, we had this of course back in the version 2 days and so a process for it has been around for a while. In general it’s pretty quick but it depends on the release and what we’re trying to achieve with that release. If it was a really big one and we kind of bit off more than we could chew but got everything in there in the end, that can take more time to put back into gamedev. We’ll have the merge happening throughout the PTU and live phase, but with 3.2 we were talking days. It has to be because we know we have to move quickly right for the next branch.
We’ve got Sean Tracy, another name you know, guys like him are super focussed on the technical health of our builds and maintaining our gamedev repository in a good state.
W: So you’ve reminded me of another topic I wanted to discuss when you talk about technical health. One of the things I always hate when I change jobs, you’re digging and digging and trying to see how deep the rabbit hole goes. Invariably, if you’ve got a project with a bunch of legacy code, you’re going to find there’s a bunch of technical debt in there which needs to be paid. As someone who joined in 2015, what was your impression of the level of technical debt in the Star Citizen code base and what’s your honest assessment of the level that’s in the code base today?
EKD: With a project like this, and you know Chris (Roberts) himself is a coder and he’s up every night poring through the code, the man is a machine. Back before my time, there was an element of “we’ve gotta get the hangar module out” or “we’ve gotta get the arena commander module out”. When you want to release that, there are some items that you put in place and they work and then someone says “we’d better write that one down for later!”
You used the negative term “tech debt”, honestly when I first joined, there was what I’d consider a normal amount. It wasn’t like I was “whoah! This is all tech debt!” and the team did a good job of keeping track of stuff in the backlog that we needed to address. What’s been really cool is that as we’ve scaled the team and continued development, we’ve brought a lot of that stuff into our 2 week periods (sprints) and the rate of tech debt has gone down significantly over the last 2 years.
We always have a long list of things we want to do right and we know we’ve just got to keep paying off that tech debt. Item 2.0 was a big one for us, that wasn’t the perfect version we wanted and every time we do a release we’re doing an upgrade to it. It’s become a very healthy attitude of getting ahead of our tech debt and we’re continuously improving there.
W: One of the things we always hear is about the shared code base between Squadron 42 and Star Citizen and it has surprised the community at times that the single player stuff hasn’t come sooner. How much of a dependency is there from the single player game on stuff like Item 2.0 and network bind culling and things like that considering it’s obviously single player and not online?
EKD: It’s a shared code base but obviously there are some things that will be more critical than others and you’re right in that because it’s a single player game, the online components aren’t so important. That much said, there is a lot in Star Citizen that is going to be utilised in Squadron 42 so it’s an interesting question, a lot of stuff I can’t talk about but ultimately we want to make it the best game we can. At a high level, people look at Star Citizen and see “$200 million, 500 people” and ask why it isn’t out yet.
W: Yep, the old “9 women can’t have a baby in a month”.
EKD: Yeah, that’s my favourite example of productivity I use all the time.
W: I don’t want to discuss specific people as there has been some controversy over it in the past, but staff churn is a fact of life for any company. How are you guys finding your baseline velocity affected with staff churn and people leaving/new people joining and needing to be onboarded etc?
EKD: In the old days it was pretty dependant on who it was that was leaving. If it was a concept artist who was got a job offer to work on the next Iron Man suit, maybe we had another guy who could pick things up pretty quickly but if it’s an engineer who is deeply embedded in one specific code area, it was harder and we’d take a velocity hit. I think it was more of a problem when we were smaller and everyone was hyper-focussed on one area but as we’ve scaled and grown, we have a much greater ability now to shift work around. As we’ve grown the company and are all making careers here and want this to be the place we work for the rest of our lives, we’re really sharing knowledge and the follow the sun development model also helps because we’re all working on multiple components.
There are places you can always get better and improve upon but we’re always working towards that.
W: One of the challenges is you always want a degree of cross-pollination in development teams so that you don’t have “that one guy” who knows everything and maybe gets run over by a bus, but cross pollination also takes time so it’s always a bit of a balancing act. Are you guys doing sprint planning as global teams?
EKD: It’s global and it has to be. We found years ago that if we weren’t doing this stuff globally, we’re just going to hit blockers and we’ll need to wait for someone to wake up in another office to help remove a problem. Some stuff of course happens locally because we gain efficiency like that but we have tools to help us and we are using Confluence and Jira, we have shared boards so you won’t have somebody sitting in another office going “I didn’t know that was coming!” so it has to happen globally at some level. We’re obviously not getting all of our developers in one room.
We may realise “oh we have to do this piece in 3.3 because it’s super crucial to 3.4” everyone should be aware of that and that’s the way we’ve designed the company to make sure that knowledge spreads.
W: Slightly tricky one you probably won’t want to answer! I’ve worked in several places where you’ve got some guy, he’s really senior in the company and he’s a coder and he was one of the founders and does what he wants to do and goes in and says “this shouldn’t work like that!” and he does whatever he wants in the code base. In many ways, everyone in the company reports to Chris, but at the same time, if Chris is cutting code for Star Citizen, he needs to report to someone. So what happens in that? Is Chris coming into sprint planning and being allocated tasks and his tickets are on a Jira board where everyone can see what he’s doing? Is someone peer reviewing his code? Or is he like “nah, f**k all that, I’m doing what I want and you guys deal with the fallout”?
EKD: That’s a tricky one! What’s really important to us at all levels, especially at the leadership side is to keep as closely in touch with the product as we can and it takes a lot of effort to manage that. But on the coding side, absolutely, everyone has a peer review, Paul Reindell is our Star Citizen Engineering Director and we have other insanely talented engineers and Chris works very closely with them. There is definitely a level of maintained respect at all levels. Sean Tracy knows everything about Perforce, everything in our engine and he’s a super smart guy. He said he was going to try to do a check in request that would purge the entire database just to see what would happen and thankfully it’s blocked! We have checks and balances everywhere. We do a lot of testing and there’s a complete sense of shared responsibility across everything that we’re doing.
Back in the day maybe things were different when there are 10 people, but as we’ve grown, we need to ensure that the creative process still allows for our independent natures but still works in such a way that we can get things out with better timelines and higher quality and everyone is on board with that.
W: So if I was an employee at CIG working on Star Citizen, I could login to your Jira and see tickets that Chris Roberts is working on? He’s adhering to the process? He doesn’t strike me as a process guy…
EKD: Maybe… I’m kidding! Of course. You have to understand, this grew from nothing. If you go work at Activision, they have everything in place to design and build games. For us, in the early days it was just hire people and get stuff done. So we’re building not just a game, not just a company, but also all the processes to make this stuff work. Would I take a junior engineer and give them some really complicated R&D or architectural work or would I give it to my CEO or Tony Zurovec? The answer is obvious. When you’re producing any product, there’s the R&D phase which is going to be a bit looser and you prototype so these are things I’m going to put different resources on to those kinds of tasks.
There’s an art to producing and working to get the best out of people that’s not necessarily written in a book, particularly when you’re working on creative stuff.
W: Earlier on in the project, you guys did a reasonable amount of outsourcing for Star Citizen. I’ve had some experience in this area and it’s an extremely useful tool, but at the same time, the degree of precision required to get it producing appropriately is extremely high. Apart from Turbulent who are more focussed on the website, are there any core game components still being outsourced and how do you control them if so?
EKD: As you know, there are strengths and weaknesses to both outsourcing and going in house. When we started, we did some outsourcing to try to ramp things quicker and that also allowed us to learn what our strengths were internally. As we ramped studios, now it’s like “I have this insanely talented team in Germany and they know this stuff inside out and I talk to them every day and they’re in my Jira and in my processes” so we use that. We obviously still use some outsourcing now but it’s less core game dev and much tighter controlled. We’ve seen stuff that we outsource now we’re able to get the best usage out of it and turn things around really quickly.
W: Talking about Germany, the guys there are mostly ex-Crytek so it feels like there are what I’d call “centres of excellence” around the company. As much as there is global operation and teamwork with handoffs, are there things which are always going to go to Germany or stay in LA?
EKD: Absolutely, but we don’t want too much of that as we don’t want to make too many dependencies on one person or group.
W: Many thanks for your time Eric! This has been a great introduction to the process of Star Citizen and CIG, perhaps see you at an event sometime!
EKD: Absolutely, and see you in the ‘verse!
