We have normality. I repeat, we have normality. Anything you still can't cope with is therefore your own problem.- Douglas Adams (The Hitchhiker's Guide to the Galaxy)
The steps are mostly documented in a previous bulletin Service Outages . Long story short, Google (whose maps-API we're obviously using for showing the moving map) decided to disable support for Fusion Tables -layer, which the map was using to render airport markers based on users' search criteria. Third party developers were given ample time (over a year in fact) to move into some alternate solution - and we had our home-baked server-side marker rendering code ready to kick in as soon as the support for Fusion Tables was actually dropped ... we didn't want to do it in advance, because we knew (or rather - were seriously expecting) that switching from Google's enormous render farms to using our own server to do all the heavy lifting would not be as performant as people were grown to expect.
Then December 3rd arrived, Fusion Tables went the way of the Dinosaurs right on schedule, we rolled out our own renderer and ... pretty much immediately ran into all sorts of trouble. Even though the home-baked renderer had been thoroughly tested by the whole team (consisting of me, myself and I) ... it wasn't really scaling up the way it was intended. In fact, just a couple of simultaneous users on the map brought the server on its knees so - something needed to be done fast. The "throw more hardware on the problem" (i.e. getting a new server) was out of the question, so the aforementioned three membered team of infinite monkeys decided to begin optimising our rendering code. This along with fixing a couple of bugs that had gone unnoticed, took about a day (since this is done in addition to day-job, so no-one really is in charge of the site 24/7) and ... then we started collecting statistics. Finding out where the various bottlenecks of the system were was necessary so that those could get rid of first.
On the following weeks, we identified most of the more serious bottlenecks, optimised the database queries, got rid of some ancient third-party dependencies the site had (like jQuery and loDash which are pretty much obsolete these days). And ... since the whole rendering process on smallest zoom levels (i.e. largest map views with most markers on screen at the same time) was still the most significant bottleneck of all, it was necessary to introduce tile cacheing. This of course means exchanging compute time to storage space. If you have a 100% similar query conditions that has already been rendered, you will receive the cached tile. This should help initial map views to load a lot faster - but might introduce some cache invalidation issues on tile borders (which is where we still have problems with marker Z-index trashing ... working on that:) or when the data gets updated on gateway.
Against our better judgement we even implemented a new "required feature"-filter, "airport needs to have Ground Traffic" which had been visible on the UI for a long time - just disabled. And ... of course, we changed the map style to a slightly more "X-Plane -like map". For what it is worth, people seem to like the new gray-/blue-scale terrain map over the more "realistic" roadmap style we had before.
Well, the intention is to keep collecting performance data at least over the holidays (because - well - funnily, a lot of gateway artists choose to do new stuff over any holiday period:) ... if not too many sore thumbs are found, the map might get a more automated update schedule (in the past with Fusion Tables, updating the map has been a semi-manual process; all the boring details - airport rendering, data update precalculation, etc. - have been automated, but since Fusion Tables API was a bit problematic, actually updating the map markers data on Fusion Tables was done manually about once a week ... now that all of the data is on our own server, this could be further automated ... but we really want to follow what impact this will have on performance - after all, if the performance suffers from too frequent data updates - e.g. because of problems with cache invalidation - I think it isn't worth it).
So ... Happy Holidays to all Gateway Artists! Keep doing what you're doing ... You're the best! ... Peace & Love!