Triage, and Bug Fixing Resumed

Hey Folks! Just a quick update today as I'm running late.

It's been a while since the last issue triage, so I spent some time this morning reviewing all outstanding issues to update their priorities. Quite a few got shuffled around, and we're getting to the point where we have bugs that are showstoppers vs. bugs we can launch with and patch later. The latter are things that have workarounds and don't corrupt the game, but might be a little annoying. (And frankly, probably not as bad as those I launched on PC with.)

Once that triage was complete, Tiago and I decided to divvy up the top crash bugs and tackle those first. And so far, I may have actually fixed one! Testing seems to bear that out, but only time will tell, as it's a "timebomb" style bug that only appears after playing a bit. I think it was due to creatures destroying campsites' unique camp conditions when they died.

Anyway, it's nice to have things organized again, and a clear path to launch. Even if that path is filled with bumps and curves :)

Revenge of the Crafting Leak

Hey Folks! Hope everyone had a good weekend. Rochelle's licensing exam seems to have gone quite well, and we visited friends to boot. So, successful trip!

Unfortunately, my memory leak fixes from last week were lying in wait for my return. And upon playtesting a mere 3 minutes, snapped shut the jaws of null reference.

It seems my memory leak fixes were overly aggressive, and somehow during crafting of broad spears, I end up trying to clone a component that doesn't exist anymore. The frustrating thing is that I can't figure out whether the mistake is destroying the item or trying to clone it. Only one of those things is correct, but neither seems wrong in the context.

I spent almost the whole day adding trace output and poring over the logs to try and peer into what's going on. From what I can tell, there is a section where consumed ingredients are scheduled to be destroyed, and those ingredients' components are also scheduled separately. In theory, this is redundant, as destroying an item destroys the components, too.

However, this results in a null reference as the game tries to clone the components after they've been destroyed. And if I remove them from scheduled destruction, I get no errors, but they're left over in memory when the game ends.

My best guess so far is that the troublesome components are being detached from their parent somewhere in the code, probably after they are scheduled for destruction. So when it comes time to destroy the parent, they are no longer listed as attached to said parent, and get skipped in clean-up.

Of course, that doesn't deal with the problem of them getting cloned later (the thing resulting in a null reference if they're destroyed). Should they be cloned at all? Am I missing some code to remove these unnecessary components from the code that creates the final, cloned output?

I'll need to look into this tomorrow. The one bit of good news is that in the process of debugging, I cleaned up a minor bit of code that created a lot of redundant looped code and log spam. It's a minor victory, but helps both in performance and debugging just a tiny bit.

But that null/component bug, though...not looking forward to grappling with it again. Worst-case, I suppose I could just back-off a bit and let the memory leak a little more to avoid the null ref, and maybe we can live with it. But I'll try one more day, at least.

Crafting Leak Tamed, OOO Tomorrow

Hey Folks! Early update today as I'm about to head out. I think the crafting leak I mentioned may be mostly under control now.

The fix involved probably over a dozen patches to various places in the code where items were created but later never destroyed. Things like temporary treasures generated while checking a recipe, item components and stack contents during these checks, potential yield items that later were replaced by final copies, etc. It was a huge rat king of interdependent code, and I fear null references in our future.

But, the number of wasted/leaked items in memory is down 90%. From 100+ as a result of a single recipe, to ~10 as a result of multiple recipes. So maybe more like 95% reduced? Anyway, much better. And so far, no nulls. But I've warned Tiago to be on the lookout. We've really disturbed a sleeping dog, here.

With that handed-off to Tiago, I'm heading out of the office. So no news tomorrow. I should be back to normal on Monday, though. Hope everyone has a good weekend!

CombatPairs and Crafting Leak

Hey Folks! Tiago and I are starting to see improvements! Our latest memory leak fixes have the numbers way down, to the degree that short games look pretty safe. But there may still be some long-term effects to fix.

After a quick recap, we decided my next target should be CombatPairs. These are not being destroyed appropriately after combat, and as combat can occur both between player and NPCs and between distant NPCs, this can add-up.

Fortunately, this turned out to be easy to solve. The place where the CombatPairs are no longer needed is quite clear, and I added clean-up code there. All seems well in that department!

Unfortunately, during my testing of this, I discovered a potentially huge leak in crafting. (Which, I suppose, is actually good news if I can fix it before launch.)

It appears that several redundant copies of temporary/interim recipe items are being instantiated during recipe validation. To the degree that when I spawn a player, craft a single sharpened spear, and then quit, I have 100+ spear-related recipe items in memory. Holy cow!

It's taken me near all day to figure out, why, too. I'm now at the point where practically every line has a trace to help me figure out what code branch is causing this. (Damn you, Haxe! Where is your debug-stepper!)

I think I may have found a section I missed when adding clean-up code. So tomorrow, that's where I'll focus first. Hopefully, I can squash this bug before it breeds!

Item Instance Wrangling

Hey Folks! Hope everyone had a good weekend. We're finally on the mend here, it seems. Saturday was like a shroud of ache and lethargy was lifted, and despite a lingering half-cough, I feel like a new man. Wellness is nothing to take for granted!

Also, as most of you probably figured out, yesterday was a stat holiday here. (Martin Luther King Jr. Day, celebrating civil rights and one of its great heroes.)

Back at work, I managed to find and fix a handful of new item leaks. I was feeling a bit discouraged at the beginning, as it felt like my efforts weren't making a dent. But I kept at it, and I'm reaching near-zero levels of items in memory after several turns.

It's a long list of fixes, but they all sound pretty much the same: "added code to explicitly destroy items that ___," where "___" were some temporary items being used in one game system or another. Specifically:

  • Added code to explicitly destroy items cleaned-up from yield pages in CheckRecipe().
  • Added code to explicitly destroy item copies used in TestItemsFitGroundCamp.
  • Added code to explicitly destroy template components before replacing with save data components in ItemInstance.
  • Added code to explicitly destroy template items before replacing with save data items in Creature.
  • Added code to explicitly destroy template items, components, and stack before replacing with clone data in ItemInstance.Clone().
  • Added code to explicitly destroy encounter response validate treasure as it wasn't used for anything.

I also noticed that recipes generated full item instances from a treasure when checking the inputs against the reverse process. Switching this to just a list of ID strings for each item saved a few leaks, too.

All in all, we're now leaking at way lower a rate than before. We still have null reference bugs, though. But if this saves us from crashing after a little while in each game due to memory, that might be worth it. We should be able to find and fix those null bugs eventually.

We're getting there!

Items Fight Back

Hey Folks! The memory leak fixing hit some speed bumps today, as it appears ItemInstances were prepared for my attack and mounted a defense.

They're thinking!

It took most of the morning to even make sense of what I was seeing, for one thing. I could clearly see which items were being leftover, and where they were created and should've been destroyed. But then, when it came time to destroy them, it was as if the game forgot it created them.

A lot of blind searching eventually led me to treasures: they have an option to "supress contents" when generated, such as when one just wants a guaranteed empty bottle. And the way this suppression happened, it would just sweep them under the carpet instead of disposing of them correctly.

Similarly, there turned out to be a bug in disposing of treasure contents that wouldn't fit into the treasure's container, and this needed fixing. This one would've been missed were it not for another bug in the potato chip loot. The bag generates 6 handfuls of chips, but can only fit 2. So the remaining 4 was what tipped me off.

After those were fixed, I finally found a bigger one. Items used to check encounter response outcomes were being left in memory. This, it turned out, was due to the encounter generating a temporary recipe for each encounter response, and then compared player input items to the recipe's needs. And then "oops!" just ignored the recipe when done.

Unfortunately, these recipes often contained multiple items, like torches for scavenging with light. Also unfortunately, those torches contained components. Also also unfortunately, this could happen dozens to times per turn, especially during scavenge encounters. And with dozens or hundreds of scavenges per game, well...let's just say lots of leaking items.

Those are the three I've knocked off today. I'm still seeing hundreds of items leaked. And I've also found a weird null reference bug related to charged items and mode changes, which I almost thought I figured out but then couldn't make it happen again.

I'll certainly have my work cut out for me. For now, though, happy weekend everybody!

Encounters Finished, Items Next. And Music!

Hey Folks! Still on the slow road to recovery here. Probably a notch better, but way more exhausted as I woke up earlier. From what I know of the flu, looks like at least a week or two more of this as it slowly wanes. Ugh.

Yesterday's remaining Encounter leak issue turned out to be well-hidden, but not too hard to solve. The temporary encounter used to update the yellow text when the player makes an encounter choice had some items attached to it (flagged for using/removing in-game). And when destroyed, these caused null reference errors in the game. Since that encounter wasn't using nor removing anything (just generating preview text), I was able to safely detach them before destroying the temp encounter, and all seems well again.

I've checked that code in so Tiago can take a look. First impressions are that it helps a little, but I really need to tackle that ItemInstance leak.

The ItemInstance leak might be a multi-faceted issue, by the looks of it.

One part seems to be hardware-related. Cellphones, laptops, smartphones, and other items with batteries and files seem to regularly leave their contents leaked in memory on exit. The bizarre thing is that it's not consistent. Some hardware has this issue, while others don't. And worse, some items can be leaked and others not, even if they live inside the same hardware container.

Another issue seems to be stacked items inside containers. Potato chips, string, pebbles, etc. A good chunk of the leftover items seem to be in this category.

And even more bizarrely, a dozen or so medium branches, crude lit torches, crude unlit torches, and dirty rags, listed one after the other in sequence, over and over. This looks suspiciously like a recipe or components of a crafted item. So that may be a clue.

In fact, yeah. I'm seeing shoe soles, clumps of rags, clumps of string...this looks like components.

And two cryo encounter items that should've been deleted. That's all useful info, and I think I can start from there tomorrow, and at least figure one or two of these categories out.

Finally, I approved Josh's latest track, which has a nice "time is running out" vibe to it. I think it should do nicely to make players nervous when it starts while they're thinking of what to do next :)

Encounter Memory, and New Music

Hey Folks! Feeling a tad bit better today, thankfully. Slept almost 12 hours to get that benefit, though. Whole family did. What a circus of a week.

I'm making progress on Encounter refactoring, but it's pretty messy.

Basically, the original game would load all encounters, and provide direct references when one was requested. Some local code would then clone encounters as-needed, such as when using battle or scavenge encounters as a template with modified text.

As a result, nowhere in the code does it clean up after itself :)

I've changed the it to always provide clones now, so they can be safely destroyed when needed. I had to make some adjustments to clone to make "deep" clones, so copies couldn't clobber pieces of originals.

There are a ton of places where the game requests whole encounters to check things (validate choices, check conditions, copy text or images, etc.), often many in rapid succession. So these areas had to be patched to clean-up. The down-side is that we have a lot of churn now. Constant cloning/destroying, sometimes redundantly. On the plus side, this happens infrequently. E.g. per-turn, or per user click.

With this and yesterday's changed, the results look promising, in any case. I'm down from ~150 orphaned encounters to ~20 after two turns. ~10 of those are special case that will always be loaded, and the remainder appear to be due to one missed destroy() call. However, when I applied the destroy() call, I started getting null reference issues in seemingly unrelated areas. I'll have to look into that tomorrow.

It's looking like this might result in a net win, but will require a bit more fixing-up. And I haven't really stress-tested it yet (e.g. battle, scavenging, AI encounters).

In other (spaceship gaming) news, Josh got his latest track to me today! I haven't had a chance to listen yet, but I'm excited to. I'll save it for tomorrow so I have something juicy to look forward to :)

Conditions Plugged, Encounters Next

Hey Folks! Memleak work continues today, as conditions get wrapped-up, and we turn to encounters.

For conditions, I finally found out where the extra camp conditions in memory were coming from. AI was cloning hex campsite conditions each time they visited, and never releasing them. As a result, they just piled-up in memory, and for long games with lots of creatures, this could add-up to significant memory. The fix was to just not clone them, as the camp condition should theoretically be the same for all camp visitors. (There is a special case in the code not to destroy a campsite condition when a creature removes it from itself.)

There was also, coincidentally, a bug fix in the way creatures were destroyed. A typo was causing camps and ground to be destroyed when a creature got destroyed, so I've fixed that.

In brief tests, that seemed to solve the remaining condition memory leaks, and cause no other issues. So we'll go with that for now. It's always possible small changes like this have unforeseen effects, though.

Moving on, Encounters are the next big one. After some investigation, it appears each turn, for each creature in an encounter, encounters are getting cloned like crazy. Like 150 encounters in memory after 1-2 turns into the game. (Which are, ostensibly, from either the player or Yezinka.)

When you factor in combat and scavenging (which are also encounters, and in combat's case, involve multiple creatures), this can really get out of hand.

My first idea was to just change the way encounters get requested in the data handler. It's cloned at the source, likely to avoid collisions, and since encounters shouldn't change much from creature-to-creature, I figured it couldn't hurt to not clone them.

But then I remembered Tiago's struggle to get loaded encounters to fit into memory. That's why we load each on-demand from disk.

Plus, battles and scavenge encounters get customized per instance, so that wouldn't work for them the minute more than one creature needed it simultaneously.

So my next idea was to destroy them when the creature is done with them. This turns out to be way harder than expected. Even as the code's author, I can't find a safe (and still reliable) place to do this. Either I delete it too early, or miss my chance before it disappears into memory.

Enter the processed encounter queue. I'm going to see if I can store a reference to each encounter that a creature experiences in a list. And after "some time," that list gets its oldest members destroyed and removed. This way, there's kind of a running destroy queue for each creature. And to avoid figuring out the soonest possible "some time," I just chose 2 turns. In theory, an encounter from two turns ago should be safe to destroy. And when the creature goes bye-bye, the remaining 2 can, too.
I think this approach is sound, if a bit hacky.

The trouble is, it doesn't work. At least, not appreciably. It might've reduced the number of encounters orphaned by the number of encounters creatures have collectively experienced. But for 1-2 turns into the game, this is like 2-5 encounters from a list of ~150. Not good enough.

I have a hunch, though. This may just be the first step. The next step might be to check when each encounter presents its options/choices for the next turn. I bet each outcome is cloning an encounter, and these are just being forgotten each turn. I'll look into those tomorrow.

Whew! Quite a day! And I'm still feeling pretty sick. Almost worse, in fact. Double-whammy? Undead disease? God, I just hope it goes away soon. I'm done with this cough/nose/chills/aches shebang.

Plugging the Memories

Hey Folks! Hope everyone had a good weekend. I decidedly had a bad weekend. Or, at least a bad Sunday. It consisted of fever chills, aching skin, coughing, and swallowing my own mucous for almost 24 hours. Rochelle, too. Our daughter had it Friday, so it was sort of foretold. Still a bit achy and chilly today, and coughing more, but I feel a full human's worth better.

Last week's NEO Scavenger desktop patch seems to be an (almost) success! Kaaven and Lin report that the autosave issues have been relegated to a visual glitch that saving/reloading will fix, and so far, no corrupt saves. Woohoo! Plus, they were able to direct me to repro steps for the glitch, so there's even a chance I can fix that. I plan to give that a shot later. But first: mobile work.

It's been weeks since I've done much more than open the code editor for mobile, and I was starting to feel guilty about not being able to help Tiago. A few of the remaining issues are minor in the grand scheme, but really weird or hard to pin down. Plus, memory leaking is still an issue.

That said, I did (finally!) make some headway on memory today. It turned out that the encounter system was adding one extra creature to the game each time it added a creature to the game. Now, that doesn't actually happen often (think dogman at cryo, or Merga Wraith if amulet is removed), but the cryo encounter automatically adds Yezinka, and possibly a dogman, so that can add-up after multiple play sessions.

Also, it turns out that spawning creatures as a result of scavenging and other random events would sometimes generate creatures that never got used. Primarily, when the faction limit was reached. The game should've been disposing of them if the limit was reached, but instead just ignored them. Again, not a smoking gun, but this all adds-up over time.

I did a bit more testing after these fixes, and I'm still seeing some strange numbers for objects in memory. When a 5-10 minute game quits to the main menu, there were ~1300 item instances and 750 encounters. Considering there are only ~700 types of objects in the game, I'm wondering what these leftover items are. And those encounters? Are they the basic loaded data, or are there copies of that data, too?

Finally, I'm seeing loads of conditions called "Camp benefits" stuck in memory after quitting. This isn't entirely unexpected, since each camp (multiple per explored hex) has a unique camp condition for its stats. But I would expect these to be removed when the game quits, and I'm suspicious more are being generated than used.

Anyway, I'll look more into this tomorrow. But not a bad start to the week!