Memory Leaking Camps, Encounters, and Skill Items

Hey Folks! I continued my memory leak investigation today, and I think I found some culprits. It appears campsites, encounter items, and skill selection are behaving badly.

The skill selection is an odd one, as I use a temporary skill item to check if the player has completely filled their skill or trait boxes. And for whatever reason, those test items remain in memory even after quitting. This doesn't have a huge impact, but adds-up if you play multiple new games in a session. This wasn't too hard to fix.

More worrying was the campsite leak. Each hex explored by players or NPCs loads the one or more camps for that hex. And those camps never get removed from memory. Fair enough during a single game, as they all must exist along with their hexes. However, when quitting, they remained in memory. And for longer games, this can mean a lot of unnecessary camps in memory. I think I've got a fix ready for this, too.

Even worse, every encounter item on every encounter page is getting stuck in memory. When you think about how many encounter screens you see in a game, and how many choices come up, this is a lot of items we're talking about. And that's not counting combat. If this includes combat, we're talking dozens of items per-turn! That's fire-hose leaking right there.

And unfortunately, this is harder to fix. Those same encounter items include a mix of temporary items specific to each encounter and more general items from the player's inventory. (Or more accurately, clones of them for encounter use only.) Simply destroying all of them can accidentally destroy important player items (like skills, tools, etc.).

What's worse, the way that encounter screens are resolved, I can't just destroy them each turn without getting null pointers. (Usually from trying to discharge any used tools during the turn advancement.) So I have to be really careful about when I destroy them, not just which ones. Sheesh!

But if this is as bad as it seems, this could mean plugging a major hole in memory management. So there's at least that silver lining. Here's hoping we can find a fix for it!

iPad Testing, Memory Leaks

Hey Folks! Took some time this morning to try out the latest build on iPad, and it's getting pretty close. Almost no new bugs logged, and the worst new one I found was pretty minor. (More annoyance than game-breaking.) For reference, this was a 1.5 hour session, complete with septicaemia. No crashes, no glitches, no problems other than some frustrating item fiddling. And even then, this was during my attempts to use tiny pill bottles without using the zoom feature.

Not bad!

However, one major problem did show up, even if it didn't break the game: memory. By the time I quit, the game was using over 300MB, and that's a bit much for a tablet. (It'd probably crash some older ones.) What's more, returning to the main menu only dropped that to 237MB, which is a far cry from the 71MB it uses on first load.

So the memory problem is pretty real.

Not as bad as before, though. And frankly, 1.5 hours is a pretty long session for a single game on mobile. Coupled with autosave, I probably wouldn't even mind a crash after that long, provided I just reload and continue. But we're going to see what we can do to minimize that chance anyway. Tiago says Apple can be particularly strict about leaks, so I'll take his word for it.

In that vein, I've decided to see what I can do to help. My initial approach was to use hxScout to watch memory and see if I could notice patterns. Unfortunately, I think running hxScout causes some sawtooth memory usage that makes it hard to see in realtime. And my cursory digging-down through the per-frame allocations didn't tell me much.

Trying a different tack, I decided to test spawning and despawning creatures to see if they leave residual memory allocations. And after some trial and error, I was able to spawn and despawn creatures over and over with barely any net memory footprint change.

Still, memory increases over time, whether starting, playing, and quitting over and over, or even during a single game.

I decided next to see if it might be items. So far, I'm tracking the number of items created, initialized, and destroyed. And some interesting data is coming to light. No items are being destroyed in-game, which is one thing. Maybe normal (short games, no lost items), but also maybe items that should be cleaned-up.

More immediately interesting is that 80-120 items are not being cleaned-up after quitting the game. Compared to the hundreds created in a single game, this isn't much. But it might be a place to start. So tomorrow, I'll see if I can get more info on which items these are, and find out why they don't get destroyed. Plus, maybe look into why no items get destroyed during the game.

Fun, right? :)

Bugs and Store Stuff

Hey Folks! Hope everyone had a good weekend. Believe it or not, I've had some spare time lately to actually play games. So I decided to finally pick up 7 Days to Die per some players' recommendations. Pretty cool! Kinda feels like Skyrim and Minecraft mushed together in a slightly fantasy-like zombie apocalypse. I'm enjoying it!

Back at work, I tried to tackle a mix of bug fixes and launch tasks for NEO Scavenger mobile. One of the bug fixes was for something I broke while fixing another issue. I was making the decimal output of certain numbers two digits, but ended up breaking one place when fixing another. Those are both fixed now.

I also helped Tiago a bit with narrowing down the cause of a null pointer bug. Could be a case of items getting destroyed during combat turn refresh, but the game still having references to their empty (nullified) husks.

And speaking of Tiago, he thinks he'll be able to resume committing full-time to the game again now that he's (mostly) settled into his new apartment. Which is good! Because I think I've been having trouble mustering the motivation to go this last mile alone. They say the last 10% of gamedev takes 90% of the effort, and man, does it ever feel like it.

Outside of the code, I went through the list of achievements to rewrite them in a style more consistent with other games. E.g. present tense, short title, short description. I also resisted the urge to drop too many pop culture references. (It was hard, believe me.)

And while I was at it, I shored-up the store page text for both Google and Apple pages so they were consistent, and created a 180x120 promo graphic to fill Google's older device requirements.

Plus, a handful of admin tasks like paying bills, replying to vendors, etc. Not a bad day, now that I look at it in review. Not terribly exciting, but I think I nailed at least a few boring-yet-necessary tasks. More to come, for sure :)

Mobile Bug Fixing Resumes, and Docking Track

Hey Folks! Tiago's moved into his new apartment now (mostly), and was able to get back to bug fixing. And momentum is picking up!

First on the list was a null pointer bug after combat when using his memory leak fixes. We tossed around some code this morning, and were able to narrow it down to one or two places. We figured out some ways to avoid the issue, and Tiago's going to additionally make his clean-up code run a bit later to ensure the game has a chance to do whatever it needs before the creature is destroyed from memory.

We also discussed how best to include the new app icon art, as it involves some new source files and icon sizes (particularly for smartphones, now that we have the zoom UI to make playing on them feasible). I re-exported all the necessary sizes for both Android and iOS, then checked those in. Here's a preview:


NEO Scavenger Mobile App Icon

Finally, I reworked the button click noises to be a bit softer. Specifically, I ran some de-click filters on it to cut the high frequencies and expose the low-mid range, which seems more satisfying and crunchy than the older, piercing noises. This is aimed particularly at small device speakers with minimal low-mid range volume.

And over in the world of space music, Josh's latest docking track is a winner. It conjures up that warm, welcoming safety feeling after days or months of tense spaceflight. I've officially approved the track, and the OST is shaping up nicely!

That's all for this week. Have a good weekend, all!

More Icons and Admin

Hey Folks! Not much to show today. I ended up reworking the iOS icon a bit after some test runs. I eventually settled on a border to help segregate the icon from the home screen wallpaper, but getting that border aligned to iOS's rounded corners was a bear. And to make things worse, it appears to change with iOS version.

Fortunately, I was able to nab the svg file iOS uses to mask the icon directly from iTunes Connect, and use that to build my border. And the border itself isn't too hard to make in Illustrator. Then, once I've got the border/background done in vector art, I use Photoshop to add the pixel art character to the icon. Photoshop has some handy tools for scaling a vector layer with antialiasing while the pixel art layer remains nearest neighbor (a.k.a. point sampling). Not a thing one often does in games (it's a bit of a no-no), but for an icon like this, I think it makes sense. The icon looks like any other system icon, but with the subject made out of a pixel art sprite.

I guess we'll see if the world agrees soon :)

That was (unfortunately) the bulk of the day. Seems silly to spend so long on a simple icon, but it is an important asset. Many folks decide whether to click on your app based on that single image!

The little remaining time I had was spent on more admin stuff. However, Josh sent a revision of his latest track my way, so at least I have something to look forward to tomorrow :)

App Icons, Docking Music, and Spam Attack

Hey Folks! Bit of an unusual day today. No real dev work as I spent most of the day on biz admin emails, site spam defenses, and app icon design. Still, I think some good progress was made!

First of all, the spam attack. It looks like the seasonal spam botnet swarmed by early this morning, leaving a trail of senseless Korean spam links. Annoying to wake up to, of course, but thankfully not hard to clean up. Just a few clicks and the users and all their content was zapped. I recorded IPs, usernames, and some other choice info in my "spammer investigation black book" for future reference. And setup a few new content filters in my site's blacklist. Hopefully, they're gone for a while.

Josh's latest track arrived, and I spent some time listening and taking notes for him. This one's for the space game's docking sequence, and is meant to conjure feelings of warmth and safety after a long journey. We both think it's really close, and just needs some sounds cleaned-up for clarity.

There were the usual admin emails, as well. Replying to partnership inquiries (e.g. publishing, PR), web designer updates, some fan emails. That sort of thing.

And finally, app icons for iOS. I have some placeholders already that Tiago and I use for the game, but I'm thinking of upping my icon game a bit. Instead of the game logo, I'm thinking of using the player's sprite with some colorful clothes and equipment. This way, it's a clear pixel art character dressed in post apocalyptic fashion. It's eye-catching, obviously a game, retro, and concise/easy to understand. Users who are into games like that should be interested in looking further, and people who hate pixel art can move on :)

Unfortunately, there's something like 24 different icon sizes for iOS to account for all devices, screens, and OS versions. Sheesh! There are also tools and templates for auto-exporting icons from a single image, but with pixel art, this may not be ideal. Still wrestling with this.

That's all for now. Have a good night, all!

Slow Start, But Back On Track

Hey Folks! Had a bit of a rough start this morning, but I think I'm gaining momentum again.

This morning was a terrible slog through mismatched Haxe libraries, compiler errors, and other issues. As usual, the tinkering I did in my library versions to make one project work (encounter editor) completely broke my other projects.

Incidentally, Tiago setup some batch scripts to help change libraries to and from what the project needs for just this reason. However, in the process of using them, I discovered a bug. Missing files, actually. For some reason, the GitLab repository was missing all .exe files from the Flixel and OpenFL libs. And that caused compile issues on my Windows (i.e. main dev) machine.

Fortunately, once I figured that out, it was an easy fix. I had the .exes in my downloaded lib folders from before, and could just copy them over. But Tiago's going to look into why the GitLab repo just ignored them during his check-in.

With that out of the way, I was back on the path to bug fixing. And I actually knocked off some pretty tricky bugs!

First, I had to (re)fix a bug in our old version of Flixel that caused the console debugger to disable keyboard if the game lost focus. Modern Flixel has this fixed, but we're using an older one for compatibility (and stability) reasons.

I also fixed a bug that caused a null pointer error when trying to use certain items' context menu. Basically, any item with mode changes would crash the game if you opened the context menu for it, closed inventory, and opened it again. Nasty!

And while fixing that, I noticed a related bug. The item popup peek would be empty if the inventory was closed while the popup was open, and then the user tried to view the item peek again. Not too hard to fix, but hard to find.

There's one more peek bug, too. Hardware items and their hidden contents when locked/off. Now that we use sprite atlases, this is broken until we rewrite it for atlases (instead of the old bitmap blitting code). Might be trickier, but at least we know the cause of this bug!

Not a bad day, considering the rough start. Hopefully, tomorrow continues the trend of more progress!

Encounter Editor Dead, But Not Dead Weight

Hey Folks! Hope everyone enjoyed their (long?) weekend. We visited with some friends, Skyped with family, and generally enjoyed decompressing from the normal life stresses for a few days.

Unfortunately, I had to pronounce the NEO Scavenger Encounter Editor dead this morning. At least for now. After a bit more debugging, it seemed to be failing in ways that I could barely see, let alone fix. And what's worse, I discovered that the old (slow-but-functional) Flash binary I was using wasn't actually working, either. It was correctly loading and displaying the network of encounters, but when it saved the data, it corrupted special characters. This, in addition to the extremely slow writing process issue, pretty much killed the editor revamp as a viable task.

However, all was not lost.

As mentioned above, it did still show the data, even if it couldn't save it. And for encounters, that's a big help. Using the standard database editing UI, I could make any changes I needed to and re-export the XMLs for mobile. So in fact, I could still use the editor, just as a viewer.

And that I did! After a few hours of cross-checking and nudging data around, plus a few new marker conditions, I think I've got the first wave of achievements all done. All that remains for those now is to test on iOS and Android.

There is also a second wave of achievements, but they're a bit more tricky to check for. Instead of simple "has condition?" tests, they may require some more involved calculations such as passage of time combined with conditions. And an even harder third wave may require comparing data between games. But, it seems pretty doable, so I'm not worried yet.

Tiago reported in, and he had a pretty low progress week, too. What work he did manage to do was entirely sunk into memory leaks. And worse, he thinks this week may be slim on available time, too. However, we're hoping to both be back on track soon and "firing on all cylinders," as they say.

That's all for now. Have a good one, all!

Editor Fix Progress, New Space Track, and OOO

Hey Folks! Not a lot to show today, unfortunately. Still working on encounter editor issues. But I did get some cool stuff from Josh to review.

One of the tracks Josh has been working on for the space game has been a "take off" track. It fits the bill nicely, with a build-up leading to a vast, empty heartbeat style Great for setting the tone of a potentially deadly flight :)

I'm not sure yet if I will always play this track for take-off, but there are space games I really enjoyed that do take-off and docking tracks. Assuming it doesn't get repeated too frequently, I'll be trying the same. If repetition does become an issue, there are some tricks I'm sure we can try (easiest maybe to have a couple to cycle through). But this gets us rolling for the time being.

For the encounter editor, I decided to take another dive into the code. This time, I made some progress in narrowing down the problem. My encounter nodes are absolutely killing memory. The app thinks it's only using tens of MB, but according to Windows, it's inflating 100s of MB each second, capping at 1.8GB before crashing. So obviously something is amiss.

I've hacked apart my code and tested a bunch of stuff, and I know roughly where to look. But the feedback from the app doesn't help any more. So I've downloaded hxScout to see if I can tell more.

hxScout is a telemetry app for Haxe, and it can report things like framerate, memory use, garbage collection, etc. Useful stuff for memory debugging, and highly pertinent to the mobile memory leak stuff we're now looking at anyway (Tiago's using it as well for his bug). So getting this working was useful in a few regards.

It took some doing to get it running, though. In true Haxe fashion, I had to download and juggle my libs to get the right combo of versions such that every dependency worked. (Now I remember why I switched to Unity for the space game!) However, it's now working. As of this post, I have hxScout showing me useful callstack info, function processing times, and wild numbers of strings and arrays I'm instantiating.

I think I can use this to make a plan of attack for shoring-up the editor.

However, it's the end of the day. And what's more, the end of the week, since tomorrow is Thanksgiving. So I'll be out of the office until Monday. If you're into the turkey thing, enjoy the holiday! And if not, have a good weekend!

Encounters and Achievements

Hey Folks! Been a bit of a slow day today. I actually had a lot of time to work, but seemingly made little progress.

The main issue I'm encountering is that my old NEO Scavenger encounter editor seems to be dying. I was about to start updating some achievement data, which requires changes to some encounters' condition rewards, but ran into issues saving my work.

For one thing, the old encounter editor binary wasn't working anymore, and I couldn't compile a new one.

The editor uses php and mysql to edit the game's database. (I use the database to manage the game's data before packaging it for release.) However, the last time I compiled the editor was before the website was updated, and uses some out of date code. I had to update a boatload of stuff just to get it to run.

And even then, it was just enough to get my old Flash binary hobbling along. It currently takes something 20 minutes to write the game's data back to the database, while in the past, this was more like 20 seconds. I have no idea why there's a sudden jump in execution time.

I briefly tried updating the editor's code to bypass Flash entirely, since Haxe is now getting pretty stable. But I couldn't get it to launch without crashing due to memory limits. It was up to 1.7GB of memory before it gave up, which is pretty ridiculous. I mean, it's a lot of data. NS is data heavy, if nothing else. But it should maybe be half that in memory.

Anyway, it feels like a lot of wasted time there. I'm actually able to edit stuff slowly this way, or "by hand" by using phpMyAdmin (which is like editing text files in a web editor). But I'm thinking this may be a more serious problem. If I need to do any real encounter editing, this is going to be painful.

Despite all that, I did manage to fix a couple of achievements, so they should now trigger at the right times. And I've sunk at least an hour today reacquainting myself with the way the encounters flow and branch. Been a while since I reviewed that nest of wires!

Anyway, probably more of this tomorrow. Maybe I'll take another crack at the editor, and see if I can bring it into the modern times. Might be a waste of a week due to this and the holiday, but then again, who ever gets anything done on Thanksgiving week?