Technical AMA!
marktillery
Member, Administrator
Hey everyone,
I'm marktillery, one of the engineers behind our little game. I get a lot of technical questions in the chat, so I thought I'd do a tech AMA. Post your questions here and I and the other engineers will try to answer as many as we can!
Thanks for playing!
Comments
(Assuming this is technical enough)
How do towers decide on what to target?
I get that they target mercs over squadunits, but is there some other criteria? This is partially brought to mind based on a targeting bug I ran into where it couldn't decide and never shot.
Does a game have a server or is it all client synchronous? Are you happy with the current architecture, or might that evolve as well?
How far would you like to take modding? Just assets, or UI and gameplay?
On the modding subject, do you have plans to expose hooks for writing custom AIs? I seem to recall that a fan mod (Sorian AI) for Supreme Commander greatly improved the single player skirmish mode for that game.
Not a big tech guy, but you seemed lonely over here, so here is a 'technical' question. You guys said that it was codded with C++ and coffeescript. Is coffeescript and Javascript different? Or is it just something the cool kids call Javascript? Thanks for being awesome and for the game we played :D
Not on the team, but I can answer this one (though it has been a while since I've looked at Coffeescript). Coffeescript is its own language, but it compiles to Javascript. It's got different syntax, and some people like it better. There shouldn't be any technical benefits: Coffeescript generates Javascript so you could have just written Javascript in the first place. In other words, if you write the following in Coffeescript:
It's the same as the following Javascript:
First off, thanks for doing this AMA. I love the technical side of things, it's my favorite, as an IT guy :D
Each test you are adding more users, so you run the risk of reaching server limitations. What steps have you done in preparation to solidify the server and make sure you have no major stress related issues? Have you experienced any stress related issues so far? (TW3 seemed to have none, from a user perspective!)
What was the source of the initial PvP TW3 bugsplats? I believe the logs said something about a missing file?
Also...what are you guys using for CI and CD?
Dang, where's the edit button. I'm actually interested in which tools you're using for the whole ALM stack (Requirements, Planning, Test Case definitions and execution plans, results, defects, CI, CD, etc.). I'm an architect/tech lead in product lifecycle IT for a large company (we give people the option of multiple on premise tools for each layer and then integrate them all together), so I'm always interested in what smaller companies do.
@ShrikeGames Targeting is a pretty complicated topic - there are some basic target priority behaviors that you've mentioned, but these behaviors don't always play out in a perfectly clean way. Firstly, it would be pretty hard to write down a set of targeting rules that don't contain any contradictions, that everybody agreed on, and that worked in every situation. Secondly, even if you managed to do that, there's severe performance limitations to how you can do targeting, so you might not be able to express those rules in code that would run fast enough in practice. So the short answer is that we try to make targeting pretty good, and pretty fast. If you've got targeting bugs though, please post replays so we can see them!
@wondible The networking is lockstep, aka synchronous. There is a server but all it does is pass messages back and forth. Essentially its like old-school peer-to-peer lockstep, except without a lot of the limitations of that method, most notably that we don't have to stall the whole game if one player is late delivering an input message. Lockstep comes with some tradeoffs: Its easy to implement and it gives you replays and observer mode almost for free. (Live observing will be returning to Atlas soon). On the other hand, it makes supporting reconnects harder, and it puts latency in between player clicks and unit movement. (See http://gafferongames.com/networking-for-game-programmers/what-every-programmer-needs-to-know-about-game-networking/ for more info).
@wondible @AceAl As for modding, we haven't talked about that a whole lot, since we're just really focused on making the game fun. But we'll definitely be releasing as many of our internal tools as we can (certainly the map editor!), and we'd love to see a lot of community created content make it into the game eventually.
@Pandapownium @Kwaai Yeah, coffeescript is just syntactic sugar on top of javascript. We have a couple of blog posts about our experiences with it: http://blog.artillery.com/2014/06/working-with-coffeescript-class-names.html and http://blog.artillery.com/2014/06/working-with-coffeescript-common-gotchas.html. Overall it's been a pretty good experience. My two big gripes are that it interferes with debugging and stack traces, and that the compile times eventually become a nuisance when your project reaches a certain size.
@Abyss Because we're making a lockstep game, we don't have a whole lot of super difficult server problems. Most of the complexity in our servers is all the stuff that happens before the game starts - the game client systems (friends, partying, queuing, match-making, etc). The gameserver itself does little more than send packets back and forth between all the players in a game. Ultimately, this means we can handle growth by horizontal scaling, which is a fancy way of saying "turning on more servers".
@wolverinero79 It was a weird edge case where we used a script to modify the playtest manifest (which is the thing that tells your computer what files to download in order to run the game) from our master branch instead of the playtest branch, which caused a compatibility problem due to a new feature that had been added in master, but not merged into the playtest branch.
Regarding your project planning/management questions, we have a very lightweight process. A lot of the processes that you'd use for something with a specific business purpose (e.g. "write test cases to make sure this tax software computes AMT correctly" or "Build a simulator that shows that this software can maintain the temperature of this reactor vessel within the specified range") don't really help a lot in gaming.
Our single product requirement is that the game should be fun. This is a vague, nebulous goal, so our approach is to do things that make it easy to do lots of gameplay experiments very quickly. Processes that are designed to prevent mistakes often slow things down, so they aren't right for us. We're not building a nuclear missile control system, so we can afford to make mistakes, as long as we fix them quickly.
Of course, we do have a lot of problems with very specific and well-defined technical requirements, e.g. "the gameserver should not crash" or "the map editor needs to be faster". However, most such problems that we encounter can be solved by one or two engineers in less than a month, so they still don't need a lot of process.
I'm curious where the graphics layer lives for Atlas, and how much it utilizes hardware. Is WebGL still a thing? do you have access to that or use other similar toolsets in the browser-descended client?
@CohLysion WebGL is basically a javascript API wrapping OpenGLES 2.0, which is basically just a subset of modern OpenGL. Additionally, emscripten (https://github.com/kripken/emscripten) has a mostly-complete OpenGLES 2.0 implementation built on top of WebGL. So, even before we switched to the native client, we had already ported our renderer to C++ and OpenGLES 2.0. When we switched to native, we only needed minor changes in order to run on desktop OpenGL 3.3 for OS X, and for Windows, we just use ANGLE (https://github.com/google/angle) to translate OpenGLES to Direct 3D, which is what Chrome and Firefox use underneath their WebGL implementations anyway.
This approach has worked out really well. We get to use the most preferred graphics API on each OS (using OpenGL on windows is a bit dicey), and we don't have to maintain multiple renderers or multiple sets of shaders. The downsides are that we don't get direct access to all the crazy features available in recent desktop OpenGL or D3D11. However, many of the features we might want (like multiple render targets or instanced drawing) are available via GLES 2.0 extensions anyway, so its not a big problem for us.
TL;DR: We didn't have to change very much in terms of rendering for the native client.
Q: So what was the impetus to go native?
A (since I looked it up and you've answered this before): http://blog.artillery.com/2015/09/artillery-native-game-client.html
@AceAl yeah, that about sums it up :D
I remember Abdi talking about GUI woes, are you guys running into poor reflow and/or selector performance with react? What's the main bottleneck at the moment?
@Burdock I'm not an expert on the guts of the UI, but basically anything we do that causes a layout can be pretty bad. However, the bigger problem right now is that we can't de-couple our UI frame rate from the game frame rate. We have a feature request with Coherent that will make it possible to do that, and will vastly improve the situation, as UI frame skips won't cause frame skips in the game.
I have a whole bunch, hope you don't mind!
What does your standard engineer's toolchain look like in terms of OS, VCS, IDE?
What kind of automated integration testing are you doing on a per-build basis, if any?
Was there any internal conflict about the decision to create a standalone client, rather than being purely web based, or was the transition obvious?
What kind of rough priority is there for an API to dig into matches and/or replays?
Have there been any sort of performance measurements between giving a unit order and when the lockstepped response occurs, minus latency? Is this a concern among the development team?
Are you doing any sort of filtering of information at the server level in order to prevent maphacks, a la HoN and Dota2? If not, is this something that could feasibly be added in later without requiring extensive reworks?
What pushed various members of the team to leave Google and move to a startup environment?
Regarding pathing, did you guys build it from the ground up or base it off A*?
In addition to Cycles question, are you using standard nav meshes, Recast, or some other tech?
@Decency @Cycle @Burdock hey guys! Really sorry for not noticing your questions here. Its been a busy few weeks. I will follow up tomorrow.
@Decency
@Cycle Both - we use a heavily modified version of A*, which we built from scratch. It operates on a navmesh that we build automatically from the map by using a constrained delaunay triangulation. This approach is pretty similar to what starcraft does, as explained in this GDC talk: http://www.gdcvault.com/play/1014514/AI-Navigation-It-s-Not. (May require GDC vault account to view). We also have a lot of mechanisms for inter-unit coordination to make pathing in crowds work.
@Burdock See above ^^.
@marktillery absolutely fascinating, thanks for the response!
Favorite term from following some of the above references: "Simplicial Complex"
Remembered my followup question, regarding (in part) #6 above and hacking and or breaking the game: Do you foresee at any point putting together an outside testing group with the mandate to break or hack the game/servers/outgame systems, etc.?
@CohLysion Honestly we just haven't thought about it too much. It seems like a potentially great idea, but we have such a huge amount of work to do right now that it just isn't on our radar screens right now.
Are the current system requirements about what you expect the launch version to have, or are you planning on doing more optimization? Also, are they more intensive then what you originally expected?
I don't remember the sign up survey very clearly now, but I believe it had me list my system specs (2011 Mac Air, 1.7 GHz Intel i5, 4 GB RAM, Intel HD 3000 Graphics), which are not even close to being able to run the game as is (1 FPS on replays and games).
@John Hey there - sorry you're not able to run the game. A 1.7Ghz i5 is on the low side, but it should get around 30fps. We test the game on a 1.8Ghz i5 with HD4000 (although that's a PC) and get 30-50fps. Can you make sure you're on the lowest possible settings? (Use the "Integrated" preset).
Hi, I'm having issues downloading the AtlasSetup.exe, getting the "Failed - Network error" error message. Is this a server side issue? I can download things from other websites just fine, but this atlas client just doesn't want to download.
Perhaps this wasn't the best place to ask this question, I'm going to shoot an email to the support crew if I'm still having issues in a few more tries. Thanks for doing this ama!
@HotBassFishin yeah emailing support@artillery.com is the way to go with these things
I just switched to integrated as well as lowest settings and I'm still getting 1 FPS. I'll have access to a desktop later today that I will try it on.