Battle Memory Leaks, and Elusive Leaks

Hey Folks! Work continues on the memory leak patches today.

Tiago found some bugs in the memory leak patch I added that caused battles to have null pointers. He was able to prevent the errors, but it was tricky to sort out which items belonged to the battle vs. regular encounter items. So we plugged away at it a bit, and were finally able to sort them out using the data handler class. Basically, there is one of each battle move in memory that all battles share, so we had to be careful not to delete them prematurely. But also make sure we deleted them when done.

Once that was fixed, I started looking into the remaining items. And things are getting harder to pin down here. Seemingly random items are getting missed on clean-up after a game closes, and no amount of my investigating has been able to find the smoking gun. It almost seems like AI standard loadouts and hex items are to blame, but I can't see why. And I can't prove that it's happening.

I'll probably look at this a bit more tomorrow, but if it's too hard, I might just chalk it up to "good enough" and move on. Tiago says most of the memory leak is plugged after the weekend's changes, so we may be at a point of diminishing returns.

That's all for today. Have a good one, all!

Scavenging Memory Leak Fix, and Business Stuff

Hey Folks! Hope everyone had a good weekend. I used actual tools and lumber this weekend to build a counter stool for the toddler. Despite grumbling through most of it, it feels good to have actually fashioned something with tools! Also, power drills are like the best things ever.

I was all over the place at work today. Paying off State Business and Occupation taxes, renewing my city business license, reviewing corporate credit card offers, and various other paperwork-y things.

And once that was done, I fired-up the iPad and Android phone to see how the latest build was doing. Tiago reports that our memory leak fixes have reduced the leak by 80%! Woohoo! And apart from two null pointer issues that were to be expected, it's looking pretty stable. We now have room in memory for music!

Still the aforementioned null pointers. And Tiago's chasing down sleep/wake crashes, particularly on iOS.

I did manage to solve one of the null pointers in scavenging. Turns out I was destroying all scavenge locales in a hex each encounter turn, instead of just the one that was used. It was tricky to tell one from the other, but after unearthing an old, rarely used variable in my code, I used that to key on the ones that are safe to delete vs. not. Yay for code archeology?

Anyway, I think I have another of those to tackle tomorrow. Combat seems to crash, and I'm betting it's a similar issue. We shall see tomorrow!

Memory Leak Hand-Off

Hey Folks! As Friday evening rolls around, I've taken my memory leak as far as I could. Tiago tends to put some hours in on weekends, so I've checked in my code for him to look over, and see if it's worth continuing.

The big difference I can see in my code is that it stops runaway memory use from encounter items. Each encounter was leaving its items in memory until the app closed, and that adds up. Assuming this was happening during battles, it might explain the leak Tiago was seeing in combat. At the very least, it was happening in cryo and other encounters, even if the user just reloaded the current move over and over. (E.g. trying different choices to see response text change)

So in theory, this might solve the long-term memory leak.

In the shorter term, I didn't see much of a memory drop. In fact, it might be slightly higher. But that might be in exchange for faster data loading, as the game was previously loading and unloading the same items over and over during certain actions (e.g. cycling through a weapon's attack modes, campsites, recipe hints, newspapers). Now, the game leaves these special types loaded for faster reading. So some increase in overall memory, but less reading/processing from storage (which, come to think of it, might save memory that would've been consumed by the data parser).

It still has issues, though. AI are leaking items when the game quits. And we have a lot of catching-up to do testing this new Item code to make sure it's as stable as the old code. Hopefully, Tiago's fresh perspective on my code will highlight anything I've missed. And ultimately, help us decide whether to push forward or revert to the old code and try another solution.

The good news is that his break from memory leaks has given him time to work on the rest of the bugs he's had on hold, instead of spinning wheels on this memory leak.

Probably not the most exciting week for all you readers out there. But a necessary one. Have a good weekend, all!

Memory Leaking Encounters, and Web Admin

Hey Folks! More memory leak patching today. I think I've finally got most of yesterday's leaks sorted out, including the encounters.

I ended up having to postpone the encounter item disposal until a bit later in the turn update code, as there is a point at which discharges and player item removal is finished and new encounter choices are populated. This is where I had to not only clear out old temp items, but also shield any in-game items linked to the temp items so they didn't get discarded out from under the player. Was a bit tricky to figure out, but seems to be working now.

Tiago alerted me to an issue with my item definition fix yesterday, in that he's "lazy loading" them to minimize memory usage and load times. As a result, I had to do a bit more scaffolding around the way item defs are loaded and destroyed to make sure the minimum necessary amount were in memory at any time. Basically, each item def now keeps track of how many items refer to it, and it only gets destroyed if that count goes to 0.

Also, I had to ignore this destroy() call on a few special item types, where all item defs are shared. E.g. newspapers, data files, recipe hints, campsites, etc.

With that out of the way, I'm now looking into hardware and software. It seems these are some of the only remaining items not released from memory when ending a game, for some reason. The hardware that contains software and batteries goes away, but not the contents. And I can't see any reason why this is so. But, I just started this task late in the day, so maybe tomorrow I'll have more luck.

In other news, a chunk of my day was wasted fighting a botnet. This time, it seems like it was trying to brute force its way into user accounts by trying a list of usernames and passwords. And when I figured out how to block that botnet, either it changed tack or another botnet coincidentally started. This time very specifically trying to access any admin-like username it could. So I blocked that, too. I think one or both are still running, as they show up in my logs. But thankfully, they're not doing much with the blocks in place.

Still, it's annoying. And kind of confusing. I should maybe put up a sign somewhere like those "no cash in vault" stickers at convenience stores. This site has literally so little of value to a hacker, I'm not sure why it'd be worth the effort.

Unless they really don't want NEO Scavenger mobile to come out soon? That must be it. They're making me pay for their lost productivity in advance :)

Memory Leaking Camps, Encounters, and Skill Items

Hey Folks! I continued my memory leak investigation today, and I think I found some culprits. It appears campsites, encounter items, and skill selection are behaving badly.

The skill selection is an odd one, as I use a temporary skill item to check if the player has completely filled their skill or trait boxes. And for whatever reason, those test items remain in memory even after quitting. This doesn't have a huge impact, but adds-up if you play multiple new games in a session. This wasn't too hard to fix.

More worrying was the campsite leak. Each hex explored by players or NPCs loads the one or more camps for that hex. And those camps never get removed from memory. Fair enough during a single game, as they all must exist along with their hexes. However, when quitting, they remained in memory. And for longer games, this can mean a lot of unnecessary camps in memory. I think I've got a fix ready for this, too.

Even worse, every encounter item on every encounter page is getting stuck in memory. When you think about how many encounter screens you see in a game, and how many choices come up, this is a lot of items we're talking about. And that's not counting combat. If this includes combat, we're talking dozens of items per-turn! That's fire-hose leaking right there.

And unfortunately, this is harder to fix. Those same encounter items include a mix of temporary items specific to each encounter and more general items from the player's inventory. (Or more accurately, clones of them for encounter use only.) Simply destroying all of them can accidentally destroy important player items (like skills, tools, etc.).

What's worse, the way that encounter screens are resolved, I can't just destroy them each turn without getting null pointers. (Usually from trying to discharge any used tools during the turn advancement.) So I have to be really careful about when I destroy them, not just which ones. Sheesh!

But if this is as bad as it seems, this could mean plugging a major hole in memory management. So there's at least that silver lining. Here's hoping we can find a fix for it!

iPad Testing, Memory Leaks

Hey Folks! Took some time this morning to try out the latest build on iPad, and it's getting pretty close. Almost no new bugs logged, and the worst new one I found was pretty minor. (More annoyance than game-breaking.) For reference, this was a 1.5 hour session, complete with septicaemia. No crashes, no glitches, no problems other than some frustrating item fiddling. And even then, this was during my attempts to use tiny pill bottles without using the zoom feature.

Not bad!

However, one major problem did show up, even if it didn't break the game: memory. By the time I quit, the game was using over 300MB, and that's a bit much for a tablet. (It'd probably crash some older ones.) What's more, returning to the main menu only dropped that to 237MB, which is a far cry from the 71MB it uses on first load.

So the memory problem is pretty real.

Not as bad as before, though. And frankly, 1.5 hours is a pretty long session for a single game on mobile. Coupled with autosave, I probably wouldn't even mind a crash after that long, provided I just reload and continue. But we're going to see what we can do to minimize that chance anyway. Tiago says Apple can be particularly strict about leaks, so I'll take his word for it.

In that vein, I've decided to see what I can do to help. My initial approach was to use hxScout to watch memory and see if I could notice patterns. Unfortunately, I think running hxScout causes some sawtooth memory usage that makes it hard to see in realtime. And my cursory digging-down through the per-frame allocations didn't tell me much.

Trying a different tack, I decided to test spawning and despawning creatures to see if they leave residual memory allocations. And after some trial and error, I was able to spawn and despawn creatures over and over with barely any net memory footprint change.

Still, memory increases over time, whether starting, playing, and quitting over and over, or even during a single game.

I decided next to see if it might be items. So far, I'm tracking the number of items created, initialized, and destroyed. And some interesting data is coming to light. No items are being destroyed in-game, which is one thing. Maybe normal (short games, no lost items), but also maybe items that should be cleaned-up.

More immediately interesting is that 80-120 items are not being cleaned-up after quitting the game. Compared to the hundreds created in a single game, this isn't much. But it might be a place to start. So tomorrow, I'll see if I can get more info on which items these are, and find out why they don't get destroyed. Plus, maybe look into why no items get destroyed during the game.

Fun, right? :)

Bugs and Store Stuff

Hey Folks! Hope everyone had a good weekend. Believe it or not, I've had some spare time lately to actually play games. So I decided to finally pick up 7 Days to Die per some players' recommendations. Pretty cool! Kinda feels like Skyrim and Minecraft mushed together in a slightly fantasy-like zombie apocalypse. I'm enjoying it!

Back at work, I tried to tackle a mix of bug fixes and launch tasks for NEO Scavenger mobile. One of the bug fixes was for something I broke while fixing another issue. I was making the decimal output of certain numbers two digits, but ended up breaking one place when fixing another. Those are both fixed now.

I also helped Tiago a bit with narrowing down the cause of a null pointer bug. Could be a case of items getting destroyed during combat turn refresh, but the game still having references to their empty (nullified) husks.

And speaking of Tiago, he thinks he'll be able to resume committing full-time to the game again now that he's (mostly) settled into his new apartment. Which is good! Because I think I've been having trouble mustering the motivation to go this last mile alone. They say the last 10% of gamedev takes 90% of the effort, and man, does it ever feel like it.

Outside of the code, I went through the list of achievements to rewrite them in a style more consistent with other games. E.g. present tense, short title, short description. I also resisted the urge to drop too many pop culture references. (It was hard, believe me.)

And while I was at it, I shored-up the store page text for both Google and Apple pages so they were consistent, and created a 180x120 promo graphic to fill Google's older device requirements.

Plus, a handful of admin tasks like paying bills, replying to vendors, etc. Not a bad day, now that I look at it in review. Not terribly exciting, but I think I nailed at least a few boring-yet-necessary tasks. More to come, for sure :)

Mobile Bug Fixing Resumes, and Docking Track

Hey Folks! Tiago's moved into his new apartment now (mostly), and was able to get back to bug fixing. And momentum is picking up!

First on the list was a null pointer bug after combat when using his memory leak fixes. We tossed around some code this morning, and were able to narrow it down to one or two places. We figured out some ways to avoid the issue, and Tiago's going to additionally make his clean-up code run a bit later to ensure the game has a chance to do whatever it needs before the creature is destroyed from memory.

We also discussed how best to include the new app icon art, as it involves some new source files and icon sizes (particularly for smartphones, now that we have the zoom UI to make playing on them feasible). I re-exported all the necessary sizes for both Android and iOS, then checked those in. Here's a preview:


NEO Scavenger Mobile App Icon

Finally, I reworked the button click noises to be a bit softer. Specifically, I ran some de-click filters on it to cut the high frequencies and expose the low-mid range, which seems more satisfying and crunchy than the older, piercing noises. This is aimed particularly at small device speakers with minimal low-mid range volume.

And over in the world of space music, Josh's latest docking track is a winner. It conjures up that warm, welcoming safety feeling after days or months of tense spaceflight. I've officially approved the track, and the OST is shaping up nicely!

That's all for this week. Have a good weekend, all!

More Icons and Admin

Hey Folks! Not much to show today. I ended up reworking the iOS icon a bit after some test runs. I eventually settled on a border to help segregate the icon from the home screen wallpaper, but getting that border aligned to iOS's rounded corners was a bear. And to make things worse, it appears to change with iOS version.

Fortunately, I was able to nab the svg file iOS uses to mask the icon directly from iTunes Connect, and use that to build my border. And the border itself isn't too hard to make in Illustrator. Then, once I've got the border/background done in vector art, I use Photoshop to add the pixel art character to the icon. Photoshop has some handy tools for scaling a vector layer with antialiasing while the pixel art layer remains nearest neighbor (a.k.a. point sampling). Not a thing one often does in games (it's a bit of a no-no), but for an icon like this, I think it makes sense. The icon looks like any other system icon, but with the subject made out of a pixel art sprite.

I guess we'll see if the world agrees soon :)

That was (unfortunately) the bulk of the day. Seems silly to spend so long on a simple icon, but it is an important asset. Many folks decide whether to click on your app based on that single image!

The little remaining time I had was spent on more admin stuff. However, Josh sent a revision of his latest track my way, so at least I have something to look forward to tomorrow :)

App Icons, Docking Music, and Spam Attack

Hey Folks! Bit of an unusual day today. No real dev work as I spent most of the day on biz admin emails, site spam defenses, and app icon design. Still, I think some good progress was made!

First of all, the spam attack. It looks like the seasonal spam botnet swarmed by early this morning, leaving a trail of senseless Korean spam links. Annoying to wake up to, of course, but thankfully not hard to clean up. Just a few clicks and the users and all their content was zapped. I recorded IPs, usernames, and some other choice info in my "spammer investigation black book" for future reference. And setup a few new content filters in my site's blacklist. Hopefully, they're gone for a while.

Josh's latest track arrived, and I spent some time listening and taking notes for him. This one's for the space game's docking sequence, and is meant to conjure feelings of warmth and safety after a long journey. We both think it's really close, and just needs some sounds cleaned-up for clarity.

There were the usual admin emails, as well. Replying to partnership inquiries (e.g. publishing, PR), web designer updates, some fan emails. That sort of thing.

And finally, app icons for iOS. I have some placeholders already that Tiago and I use for the game, but I'm thinking of upping my icon game a bit. Instead of the game logo, I'm thinking of using the player's sprite with some colorful clothes and equipment. This way, it's a clear pixel art character dressed in post apocalyptic fashion. It's eye-catching, obviously a game, retro, and concise/easy to understand. Users who are into games like that should be interested in looking further, and people who hate pixel art can move on :)

Unfortunately, there's something like 24 different icon sizes for iOS to account for all devices, screens, and OS versions. Sheesh! There are also tools and templates for auto-exporting icons from a single image, but with pixel art, this may not be ideal. Still wrestling with this.

That's all for now. Have a good night, all!