Hey Folks! Tiago and I are starting to see improvements! Our latest memory leak fixes have the numbers way down, to the degree that short games look pretty safe. But there may still be some long-term effects to fix.
After a quick recap, we decided my next target should be CombatPairs. These are not being destroyed appropriately after combat, and as combat can occur both between player and NPCs and between distant NPCs, this can add-up.
Fortunately, this turned out to be easy to solve. The place where the CombatPairs are no longer needed is quite clear, and I added clean-up code there. All seems well in that department!
Unfortunately, during my testing of this, I discovered a potentially huge leak in crafting. (Which, I suppose, is actually good news if I can fix it before launch.)
It appears that several redundant copies of temporary/interim recipe items are being instantiated during recipe validation. To the degree that when I spawn a player, craft a single sharpened spear, and then quit, I have 100+ spear-related recipe items in memory. Holy cow!
It's taken me near all day to figure out, why, too. I'm now at the point where practically every line has a trace to help me figure out what code branch is causing this. (Damn you, Haxe! Where is your debug-stepper!)
I think I may have found a section I missed when adding clean-up code. So tomorrow, that's where I'll focus first. Hopefully, I can squash this bug before it breeds!
Hey Folks! Hope everyone had a good weekend. We're finally on the mend here, it seems. Saturday was like a shroud of ache and lethargy was lifted, and despite a lingering half-cough, I feel like a new man. Wellness is nothing to take for granted!
Also, as most of you probably figured out, yesterday was a stat holiday here. (Martin Luther King Jr. Day, celebrating civil rights and one of its great heroes.)
Back at work, I managed to find and fix a handful of new item leaks. I was feeling a bit discouraged at the beginning, as it felt like my efforts weren't making a dent. But I kept at it, and I'm reaching near-zero levels of items in memory after several turns.
It's a long list of fixes, but they all sound pretty much the same: "added code to explicitly destroy items that ___," where "___" were some temporary items being used in one game system or another. Specifically:
I also noticed that recipes generated full item instances from a treasure when checking the inputs against the reverse process. Switching this to just a list of ID strings for each item saved a few leaks, too.
All in all, we're now leaking at way lower a rate than before. We still have null reference bugs, though. But if this saves us from crashing after a little while in each game due to memory, that might be worth it. We should be able to find and fix those null bugs eventually.
We're getting there!
Hey Folks! The memory leak fixing hit some speed bumps today, as it appears ItemInstances were prepared for my attack and mounted a defense.
It took most of the morning to even make sense of what I was seeing, for one thing. I could clearly see which items were being leftover, and where they were created and should've been destroyed. But then, when it came time to destroy them, it was as if the game forgot it created them.
A lot of blind searching eventually led me to treasures: they have an option to "supress contents" when generated, such as when one just wants a guaranteed empty bottle. And the way this suppression happened, it would just sweep them under the carpet instead of disposing of them correctly.
Similarly, there turned out to be a bug in disposing of treasure contents that wouldn't fit into the treasure's container, and this needed fixing. This one would've been missed were it not for another bug in the potato chip loot. The bag generates 6 handfuls of chips, but can only fit 2. So the remaining 4 was what tipped me off.
After those were fixed, I finally found a bigger one. Items used to check encounter response outcomes were being left in memory. This, it turned out, was due to the encounter generating a temporary recipe for each encounter response, and then compared player input items to the recipe's needs. And then "oops!" just ignored the recipe when done.
Unfortunately, these recipes often contained multiple items, like torches for scavenging with light. Also unfortunately, those torches contained components. Also also unfortunately, this could happen dozens to times per turn, especially during scavenge encounters. And with dozens or hundreds of scavenges per game, well...let's just say lots of leaking items.
Those are the three I've knocked off today. I'm still seeing hundreds of items leaked. And I've also found a weird null reference bug related to charged items and mode changes, which I almost thought I figured out but then couldn't make it happen again.
I'll certainly have my work cut out for me. For now, though, happy weekend everybody!
Hey Folks! Still on the slow road to recovery here. Probably a notch better, but way more exhausted as I woke up earlier. From what I know of the flu, looks like at least a week or two more of this as it slowly wanes. Ugh.
Yesterday's remaining Encounter leak issue turned out to be well-hidden, but not too hard to solve. The temporary encounter used to update the yellow text when the player makes an encounter choice had some items attached to it (flagged for using/removing in-game). And when destroyed, these caused null reference errors in the game. Since that encounter wasn't using nor removing anything (just generating preview text), I was able to safely detach them before destroying the temp encounter, and all seems well again.
I've checked that code in so Tiago can take a look. First impressions are that it helps a little, but I really need to tackle that ItemInstance leak.
The ItemInstance leak might be a multi-faceted issue, by the looks of it.
One part seems to be hardware-related. Cellphones, laptops, smartphones, and other items with batteries and files seem to regularly leave their contents leaked in memory on exit. The bizarre thing is that it's not consistent. Some hardware has this issue, while others don't. And worse, some items can be leaked and others not, even if they live inside the same hardware container.
Another issue seems to be stacked items inside containers. Potato chips, string, pebbles, etc. A good chunk of the leftover items seem to be in this category.
And even more bizarrely, a dozen or so medium branches, crude lit torches, crude unlit torches, and dirty rags, listed one after the other in sequence, over and over. This looks suspiciously like a recipe or components of a crafted item. So that may be a clue.
In fact, yeah. I'm seeing shoe soles, clumps of rags, clumps of string...this looks like components.
And two cryo encounter items that should've been deleted. That's all useful info, and I think I can start from there tomorrow, and at least figure one or two of these categories out.
Finally, I approved Josh's latest track, which has a nice "time is running out" vibe to it. I think it should do nicely to make players nervous when it starts while they're thinking of what to do next :)
Hey Folks! Feeling a tad bit better today, thankfully. Slept almost 12 hours to get that benefit, though. Whole family did. What a circus of a week.
I'm making progress on Encounter refactoring, but it's pretty messy.
Basically, the original game would load all encounters, and provide direct references when one was requested. Some local code would then clone encounters as-needed, such as when using battle or scavenge encounters as a template with modified text.
As a result, nowhere in the code does it clean up after itself :)
I've changed the it to always provide clones now, so they can be safely destroyed when needed. I had to make some adjustments to clone to make "deep" clones, so copies couldn't clobber pieces of originals.
There are a ton of places where the game requests whole encounters to check things (validate choices, check conditions, copy text or images, etc.), often many in rapid succession. So these areas had to be patched to clean-up. The down-side is that we have a lot of churn now. Constant cloning/destroying, sometimes redundantly. On the plus side, this happens infrequently. E.g. per-turn, or per user click.
With this and yesterday's changed, the results look promising, in any case. I'm down from ~150 orphaned encounters to ~20 after two turns. ~10 of those are special case that will always be loaded, and the remainder appear to be due to one missed destroy() call. However, when I applied the destroy() call, I started getting null reference issues in seemingly unrelated areas. I'll have to look into that tomorrow.
It's looking like this might result in a net win, but will require a bit more fixing-up. And I haven't really stress-tested it yet (e.g. battle, scavenging, AI encounters).
In other (spaceship gaming) news, Josh got his latest track to me today! I haven't had a chance to listen yet, but I'm excited to. I'll save it for tomorrow so I have something juicy to look forward to :)
Hey Folks! Memleak work continues today, as conditions get wrapped-up, and we turn to encounters.
For conditions, I finally found out where the extra camp conditions in memory were coming from. AI was cloning hex campsite conditions each time they visited, and never releasing them. As a result, they just piled-up in memory, and for long games with lots of creatures, this could add-up to significant memory. The fix was to just not clone them, as the camp condition should theoretically be the same for all camp visitors. (There is a special case in the code not to destroy a campsite condition when a creature removes it from itself.)
There was also, coincidentally, a bug fix in the way creatures were destroyed. A typo was causing camps and ground to be destroyed when a creature got destroyed, so I've fixed that.
In brief tests, that seemed to solve the remaining condition memory leaks, and cause no other issues. So we'll go with that for now. It's always possible small changes like this have unforeseen effects, though.
Moving on, Encounters are the next big one. After some investigation, it appears each turn, for each creature in an encounter, encounters are getting cloned like crazy. Like 150 encounters in memory after 1-2 turns into the game. (Which are, ostensibly, from either the player or Yezinka.)
When you factor in combat and scavenging (which are also encounters, and in combat's case, involve multiple creatures), this can really get out of hand.
My first idea was to just change the way encounters get requested in the data handler. It's cloned at the source, likely to avoid collisions, and since encounters shouldn't change much from creature-to-creature, I figured it couldn't hurt to not clone them.
But then I remembered Tiago's struggle to get loaded encounters to fit into memory. That's why we load each on-demand from disk.
Plus, battles and scavenge encounters get customized per instance, so that wouldn't work for them the minute more than one creature needed it simultaneously.
So my next idea was to destroy them when the creature is done with them. This turns out to be way harder than expected. Even as the code's author, I can't find a safe (and still reliable) place to do this. Either I delete it too early, or miss my chance before it disappears into memory.
Enter the processed encounter queue. I'm going to see if I can store a reference to each encounter that a creature experiences in a list. And after "some time," that list gets its oldest members destroyed and removed. This way, there's kind of a running destroy queue for each creature. And to avoid figuring out the soonest possible "some time," I just chose 2 turns. In theory, an encounter from two turns ago should be safe to destroy. And when the creature goes bye-bye, the remaining 2 can, too.
I think this approach is sound, if a bit hacky.
The trouble is, it doesn't work. At least, not appreciably. It might've reduced the number of encounters orphaned by the number of encounters creatures have collectively experienced. But for 1-2 turns into the game, this is like 2-5 encounters from a list of ~150. Not good enough.
I have a hunch, though. This may just be the first step. The next step might be to check when each encounter presents its options/choices for the next turn. I bet each outcome is cloning an encounter, and these are just being forgotten each turn. I'll look into those tomorrow.
Whew! Quite a day! And I'm still feeling pretty sick. Almost worse, in fact. Double-whammy? Undead disease? God, I just hope it goes away soon. I'm done with this cough/nose/chills/aches shebang.
Hey Folks! Hope everyone had a good weekend. I decidedly had a bad weekend. Or, at least a bad Sunday. It consisted of fever chills, aching skin, coughing, and swallowing my own mucous for almost 24 hours. Rochelle, too. Our daughter had it Friday, so it was sort of foretold. Still a bit achy and chilly today, and coughing more, but I feel a full human's worth better.
Last week's NEO Scavenger desktop patch seems to be an (almost) success! Kaaven and Lin report that the autosave issues have been relegated to a visual glitch that saving/reloading will fix, and so far, no corrupt saves. Woohoo! Plus, they were able to direct me to repro steps for the glitch, so there's even a chance I can fix that. I plan to give that a shot later. But first: mobile work.
It's been weeks since I've done much more than open the code editor for mobile, and I was starting to feel guilty about not being able to help Tiago. A few of the remaining issues are minor in the grand scheme, but really weird or hard to pin down. Plus, memory leaking is still an issue.
That said, I did (finally!) make some headway on memory today. It turned out that the encounter system was adding one extra creature to the game each time it added a creature to the game. Now, that doesn't actually happen often (think dogman at cryo, or Merga Wraith if amulet is removed), but the cryo encounter automatically adds Yezinka, and possibly a dogman, so that can add-up after multiple play sessions.
Also, it turns out that spawning creatures as a result of scavenging and other random events would sometimes generate creatures that never got used. Primarily, when the faction limit was reached. The game should've been disposing of them if the limit was reached, but instead just ignored them. Again, not a smoking gun, but this all adds-up over time.
I did a bit more testing after these fixes, and I'm still seeing some strange numbers for objects in memory. When a 5-10 minute game quits to the main menu, there were ~1300 item instances and 750 encounters. Considering there are only ~700 types of objects in the game, I'm wondering what these leftover items are. And those encounters? Are they the basic loaded data, or are there copies of that data, too?
Finally, I'm seeing loads of conditions called "Camp benefits" stuck in memory after quitting. This isn't entirely unexpected, since each camp (multiple per explored hex) has a unique camp condition for its stats. But I would expect these to be removed when the game quits, and I'm suspicious more are being generated than used.
Anyway, I'll look more into this tomorrow. But not a bad start to the week!
Happy New Year everybody! It's been a long hiatus as I work on the mobile version, but one bug fix was important enough to back-port to the old desktop version.
I've just finished uploading new test build 1.15, which includes a fix for the autosave feature.
The build is available to anyone who owns the game at bluebottlegames.com and Steam.
To access the test build on the official site, simply visit the game page, and click any of the download links below the usual Windows, Mac, and Linux buttons (look for a red "Test" button).
Steam users can access the test build by opting into the beta for it.
Updates Included in the Test Build
Test build 1.15 includes only one thing:
As many of you know, autosave had a tendency to cause weirdness when enabled, especially in longer games. One such problem was campsites and items getting misplaced or disappearing. This was due to a bug in the save game code when more than one creature was in the player's hex (and therefore caused havoc during battles).
Fortunately, while porting NEO Scavenger to mobile, I was able to find and fix this issue. And while someday, I'd like to use the new mobile (Haxe-based )engine to replace the current (Flash-based) engine, this should hopefully bridge the gap until then.
Is it now safe to use autosave? Hard to say. This bug might've been causing the other issues, too. It should definitely be safer. I wasn't able to cause any autosave corruption, but I only tested a few hours of play.
The likelihood that this version of NEO Scavenger will work with previous saves is: likely.
As usual, the older the save game version, the less likely it is to work.
As always, let me know what you think of the changes, and if you notice any issues with the new build!
Hey Folks! A little bit of old and new today, as I worked on both the mobile port and Flash version.
Yup, you heard that right. The Flash version.
Our own linibot is a shrewd negotiator, she is. And her way with words convinced me that it might just be worth a micro fix shimmied into the code to bridge the gap until I can start testing the mobile version as a desktop replacement.
You see, autosave has caused no end of headaches for players, as a quick peruse of the forums will indicate. And as part of the mobile port, I think I may have stumbled upon a fix. Or a partial one, anyway. The bit where the campsite becomes inaccessible and/or misplaced in the corner of the screen? That bit might be fixed in mobile. And I think I can fix it in Flash.
However, I was a bit scared about posting a new patch with just this tiny fix in it, especially after almost a year of no new patches. Would folks be annoyed that the only update was a boring bug fix?
Maybe. Then again, some folks might welcome the change. And all will probably at least appreciate the old beast getting some minor grooming. Plus, with mobile launch eluding me over the past few weeks, the desktop port is probably not as close as I imagined.
So...let's do this. I'm going to see what I can fix about that autosave/camp bug. I'm unsure if it'll fix other autosave bugs, such as game save corruption. But if I release it as a test/beta build alongside the normal one, maybe some users will give it a try? We'll see!
In addition to the above, I managed to do a bit of mobile work, too. I integrated the fixes I added before the holidays back into main line, and updated my copy with Tiago's recent fixes and changes. So I think my code is now up-to-date and ready for me to resume fixing PlayerCondition memory.
Feels good to be seeing code again!
Hey Folks! Still catching-up on the inbox here. I managed to get back to a few more people, pay some more taxes, and review some web design updates from The Jibe. But I also had a little bit of time to test some mobile stuff.
The business-y stuff is as dry a topic as always. Business and Occupation taxes for December needed filing/paying, inquiries needed replies, etc. The Jibe stuff was actually more interesting, as it was visual in nature. We're talking about a fan art page for NEO Scavenger, where I can (finally!) post some of the cool pictures folks have sent.
On the mobile game, Tiago's back in action. We reviewed where we left off, and what's next. One thing I could do immediately was to test the latest build to verify some fixes on iPad, and most of those worked. I'm anxious to resume work on the PlayerCondition memory leak fixes, as those were left in flux. I've also learned of a tool for debugging/memory leak fixing called DebugDiag, which I might be able to use on Windows builds of the mobile game. It could help reveal things I'm missing with internal tools.
Will I open the code editor tomorrow? We shall see!