Hey Folks! Last day at the office before taking a week off, so what better way to spend it than working on memory leaks?
Actually, I did also finally pay myself for 2016 this morning. So that's kind of exciting :)
The memory leak situation has regressed. Tiago reports that the memory usage is actually worse after my changes than before. Oof!
This isn't necessarily unheard of. Often, sweeping changes like this miss something that can cause things to be worse, and fixing it pushes it forward to a net gain. So we're both going to look into it a bit to see if that might be the case. Briefly, we also considered that it might be a front-loaded memory cost. I.e. it might use a lot of memory up front, but less as time goes on. (Which would still be better than linearly using more memory each turn.)
Alas, memory crossed the critical high-water point during a fairly brief test, so I think we still have work to do.
So I started looking into why. I managed to stumble across some new bugs in the process, but no smoking guns yet.
One suspicious thing is that I seem to be getting a lot of un-destroyed Creatures, ItemInstances, PlayerConditions, and Encounters. The items and conditions especially are weird, as those are the very things I thought I had finally under control. Did something regress? Are the reports wrong? Is this new memory inflation because of the code we added to monitor things? (E.g. the report keeping a copy of everything, even though the game destroyed it?)
Or maybe the creatures, which host a long list of items and conditions each, are the root? Will destroying those help?
One thing's for sure: I made just about zero progress taming memory today. Bug fixes, a couple. But I can't seem to get ahold of memory right now. I guess that means it's time for a vacation :)
And on that note, Happy Holidays, everyone!
I'll be taking next week off, and therefore pretty quiet on the internet. No work posts each day, etc. But I'll be back on the 3rd, and hopefully, back to business as usual.
Visit loved ones.
Eat and drink yourselves silly, if that's your thing.
Hey Folks! Another split day today, with roughly half coding and half IRS fun.
The coding was wrapping-up the condition memory leaks, and I think that's in a pretty good place now. I found some additional leaks during battles (every move) and in expiring conditions (e.g. poison wasn't destroying itself when finished). Between these and the main changes from earlier this week, the number of conditions resident in memory grows asymptotically. I.e. slower the longer the game is running. And when the game finishes, they all get cleaned up except for the template copies that are used to derive per-creature copies.
Put another way, each new game in the old system generated 150+ new conditions in memory, just by visiting the skill screen and quitting. And battles could inflate that by another 5-15 per turn, permanently.
The new system generates about 80-100 the first time the game reaches the skill screen (30-50% improvement). Battles produce almost nothing each turn (99% improvement). And quitting the game drops it back down to 50-80.
Tiago's got the code now, and hopefully his tests will show an additional decrease in memory bloat per turn and per game!
In the world of taxes, I think I've finally worked out what I need to pay myself in 2016, and importantly, how much tax to withhold. I ended up having to almost file my 2016 taxes to get an estimate, which is annoying. But as usual, I learned a few things about taxes in the process.
The good news is that I think the tax burden now that I'm in the US is lower than when I was in Canada. I guess the question then becomes: is it enough to make up for the drastic increase in cost of living? After all, I'm now paying for my family's health insurance (and copay). And rent is almost 300% my old place. We shall see!
Anyway, I've got the numbers worked out, and I think I'll sleep on it before committing it to the bank account. Never know if I'll think of something at 3am that I missed the first time :)
Still knee-deep in the blood of player conditions here, trying to reduce memory leaks.
Almost half the day was spent restructuring the way conditions are added and removed from creatures. The thrust of this change was to avoid instantiating new conditions where I didn't need to. Often, I only needed to see if a condition existed on a creature, and there was no need to make that creature generate the new condition just to check.
The good news is that the changes seem to be helping. Before the change, the game was generating 152 conditions that never got cleaned up when returning to the main menu. After the change: 46. And even those 46 would be re-used the next time a new game was started (basically, templates from which copies are made later in the game).
The bad news? Quite a lot of code changed. And that could mean stability issues down the line. But, since memory leaks also mean stability issues, it's a rock and a hard place scenario.
I'm still pretty confident in this approach. The wasted memory is down, and I'm putting out the relatively small fires resulting from the changes. Hopefully, it'll be stable enough to check back into the repository for Tiago to work with, and ultimately, we'll see enough of an improvement. (I.e. we'll have enough memory left to load things like music for the game)
Meanwhile, Tiago's still kicking butt sniping other bugs from the list, including a major crash bug resulting from iOS version losing focus (triggered by viewing iOS status bar, exiting app, buying in-app full version unlock, and other sort of important things :)
Hey Folks! Did a bit more work on Josh's procedural music for the space game, as well as returning to more memory leak fixes.
Since the procedural music was having timing issues due to my code, Josh and I agreed he'll continue on the other, more traditional tracks. And in the meantime, I'll try a few more things to see if I can get it to work. No need to make Josh wait for me to fix stuff when he has other tasks cleared for development.
So to help execute that plan, I formalize the official soundtrack specs in a spreadsheet, so we could see the specs and current status of each track. He's returned to working on the in-flight tracks for now.
On the mobile project, Tiago's been blasting through the bug list today. Just one right after the other. Plus, I think he was able to source a new iPad in his hometown after much searching, so can can resume testing locally!
Since he's making all that progress on the task list, I've decided to try and tackle the next largest memory leak: player conditions. These things are getting spawned left and right, and I think much of the time, it's just being done wastefully where I was trying to check things on creatures. I've started reworking the way they're added/removed or checked, and hopefully that can start us down a path to more responsible memory usage.
Oh, and I've decided to try and update the software on my dev MacBook. After months in the closet, the last thing I need is to open it up in an emergency and wait 4 hours for OS and software updates. I think OSX will be done today. Windows tomorrow.
Hey Folks! Hope everyone had a good weekend. We pretty much hibernated here. Though, we did have a nice walk yesterday in sub-freezing weather. Not quite like the picturesque winters up north, but on the other hand, no shoveling!
Today was largely about getting things prepped for the week ahead. Tiago's latest status report looks promising, and we think maybe this time we're really, actually, nearly done :) The last time we thought so, we were interrupted by memory leak hell, and we think we're nearing completion on that. And Tiago's been able to snipe several of the smaller issues over the past week, so we're back to making progress!
Meanwhile, in the space prototype, Josh and I are deciding whether to proceed with this procedural music experiment. The samples I sent him had some issues in the timing. At first, we thought it might be samples getting cut-off before they finished. But the more I listen using simpler test patterns, we might be having trouble with gapless looping. E.g. there may be exactly 1 beat between when a clip is ending and the next starts for a given track, and this is going to cause lag to creep in over time.
It's a weird lag, though, because it's in-time with the beat. Just off by a whole number of beats.
I think there may be a workaround for this, but we may have to fall back to more traditional music. Basically, the way the rest of the game (and NEO Scavenger) works: periodic, randomly-selected tracks.
Finally, tax stuff. I haven't paid myself yet this year since starting the LLC. And I need to do that. And the tricky part about that is that I need to withhold the correct amount of taxes from my paycheck. And in order to do that, I need to know what I'm going to pay myself.
Fortunately, I shouldn't have this conundrum too often. Once I set something up, it should be easier to continue it. It just feels like I'm setting this kind of stuff up every month these days!
Anyway, I'm going to play with taxes for the rest of my day. Have a better night, all!
Hey Folks! Just a quick update today, as I have to run and start making dinner with the fam.
A good chunk of today was spent researching ways to fix my email woes. And current thinking is to offload my email (both personal and website-generated "transactional" emails) from my webhost to a dedicated email service. Something like rackspace.com, GSuite, etc. Still unsure which. But eventually, the new site will be using them anyway, so maybe now's a good time to start moving the email portion over, since my current solution can't reach the Microsoft world.
Second, Josh's latest tracks. I finally got some recordings of the dynamic music-maker with various mixing params, and I want to see what he thinks. His last samples had a sort of Harold Faltermeyer "Sneaking around the mansion" vibe, while this is sort of an Outrun "tropical Ferrari breeze" meets Autechre. And since it's pseudorandom, it sounds slightly different each attempt. I've wrapped-up some videos of it so he can hear for himself, and we can figure out what to do.
Finally, business as usual. Emails, deal offers, and other things that need attention. Some of which were tough calls. And unfortunately, that means I got little actual dev done today.
Oh well. I still think it was a pretty good week. Some good dev work on mobile, website fires (mostly) extinguished, important decisions made, and progress on all fronts.Time for a break! Have a good weekend, all!
Hey Folks! Switched things up a bit today and worked on the site and music playback for the space prototype.
As you can probably tell by the red banner at the top, and possibly by the site maintenance earlier today, it was a day of web-admin work. More code updates and patches. The Hotmail/Live/Outlook issue isn't solved yet, and I've asked my webhost about it. However, there's also a new feature (or rather, a revamp of an old feature that was lost): abuse reporting.
On the old site, users could report a post if it seemed abusive or spammy, and it would alert me so I could take a look. Somewhere along the lines, I accidentally replaced that with a different reporting feature that seemingly did nothing. Seriously, Kaaven had to email me with an "is this thing on?" inquiry the other day. Clicking it didn't even provide feedback to the user that anything happened.
So I installed some new software today that should make it more effective. Now, each post's report "link" flags the post so I can take a look. It'll also email me the first time a post gets flagged, so I don't miss it. It's pretty basic, but it should work better than the thing that was there before (which, I reiterate, did nothing). A low bar to exceed, but there it is :)
I also did a bit of audio coding for the space prototype. Taking a break from mobile dev, I turned my attention to Josh's latest samples. It's a refined set of clips that can be used to dynamically form tracks by layering and playing them at a certain tempo.
There were some subtle differences this time, though. For one thing, the tempo was slower. Also, these are shorter and meant to be played entirely, whereas the old ones were longer loops that had random pieces/segments played. This way is both easier to create content for, and easier for me to manage in code.
So far, I think the code is working. I even have two "phases" for the music: an intro and main phase. This way, the opening measures can sound one way while the main part of the "song" is different. For now, only certain tracks like bass and percussion are likely in the intro, while most other tracks come in later. I think it still needs work, though. But I need more time to articulate why, or what's needed.
Anyway, a bit of a different day today. Possibly more of this tomorrow. Tiago, meanwhile, is still kicking butt on bugs. Several more fixed this morning!
Hey Folks! More memory leak fixing today, plus a bit of fun (not) tax research.
The memory leak had me pretty defeated yesterday. Almost half a day of tracing output and trying experiments, and I still couldn't figure out where these orphaned items were coming from. I had a suspicion they were AI related, but couldn't prove it.
Fast forward one night's sleep, and I was finally able to make progress. Yay sleep!
It turns out AI was abandoning certain items if it could not TakeItem or DropItem, especially during spawning when they had no CurrentHex yet. The result was these items being instantiated then orphaned and stuck in memory even after quitting to main menu.
The code now checks for failed take/drop in some key places, and will destroy the ItemInstance in question. So far, this seems to reduce a significant number of orphaned ItemInstances resulting from AICreatures. (Typically, sticks, bottles, rags, and other low-value things.)
In fact, the ItemInstance debug report I made for listing orphaned items on the main menu seems to be 0 for three consecutive tests! They're short tests, to be fair. But that's a big drop from previous testing! Tiago's going to try testing some more tonight, and see if we're in the clear or not.
In less exciting news, with the end of the year fast approaching, I need to figure out how to pay myself. Now that I run an LLC (with S-Corp election), the way I pay myself has changed a bit from the old Sole Proprietorship I used to have. Namely, I need to pay myself wages, withhold the correct amount in taxes, and report/remit all that to the IRS. And that means setting-up some forms and processes that weren't there before.
So far, I've been putting that off because, well, because I hate learning new tax stuff. And I'm afraid of doing it wrong. I've already paid for consulting with accountants, so I think I know how to proceed. But the first time for a new business/tax thing is always annoying and rocky. Ugh.
Anyway, I've done some research, and I'll chew on that tonight. I think I know what I have to do and what forms to file. For now, time to have a beer and relax!
Hey Folks! Work continues on the memory leak patches today.
Tiago found some bugs in the memory leak patch I added that caused battles to have null pointers. He was able to prevent the errors, but it was tricky to sort out which items belonged to the battle vs. regular encounter items. So we plugged away at it a bit, and were finally able to sort them out using the data handler class. Basically, there is one of each battle move in memory that all battles share, so we had to be careful not to delete them prematurely. But also make sure we deleted them when done.
Once that was fixed, I started looking into the remaining items. And things are getting harder to pin down here. Seemingly random items are getting missed on clean-up after a game closes, and no amount of my investigating has been able to find the smoking gun. It almost seems like AI standard loadouts and hex items are to blame, but I can't see why. And I can't prove that it's happening.
I'll probably look at this a bit more tomorrow, but if it's too hard, I might just chalk it up to "good enough" and move on. Tiago says most of the memory leak is plugged after the weekend's changes, so we may be at a point of diminishing returns.
Hey Folks! Hope everyone had a good weekend. I used actual tools and lumber this weekend to build a counter stool for the toddler. Despite grumbling through most of it, it feels good to have actually fashioned something with tools! Also, power drills are like the best things ever.
I was all over the place at work today. Paying off State Business and Occupation taxes, renewing my city business license, reviewing corporate credit card offers, and various other paperwork-y things.
And once that was done, I fired-up the iPad and Android phone to see how the latest build was doing. Tiago reports that our memory leak fixes have reduced the leak by 80%! Woohoo! And apart from two null pointer issues that were to be expected, it's looking pretty stable. We now have room in memory for music!
Still the aforementioned null pointers. And Tiago's chasing down sleep/wake crashes, particularly on iOS.
I did manage to solve one of the null pointers in scavenging. Turns out I was destroying all scavenge locales in a hex each encounter turn, instead of just the one that was used. It was tricky to tell one from the other, but after unearthing an old, rarely used variable in my code, I used that to key on the ones that are safe to delete vs. not. Yay for code archeology?
Anyway, I think I have another of those to tackle tomorrow. Combat seems to crash, and I'm betting it's a similar issue. We shall see tomorrow!
Hey Folks! As Friday evening rolls around, I've taken my memory leak as far as I could. Tiago tends to put some hours in on weekends, so I've checked in my code for him to look over, and see if it's worth continuing.
The big difference I can see in my code is that it stops runaway memory use from encounter items. Each encounter was leaving its items in memory until the app closed, and that adds up. Assuming this was happening during battles, it might explain the leak Tiago was seeing in combat. At the very least, it was happening in cryo and other encounters, even if the user just reloaded the current move over and over. (E.g. trying different choices to see response text change)
So in theory, this might solve the long-term memory leak.
In the shorter term, I didn't see much of a memory drop. In fact, it might be slightly higher. But that might be in exchange for faster data loading, as the game was previously loading and unloading the same items over and over during certain actions (e.g. cycling through a weapon's attack modes, campsites, recipe hints, newspapers). Now, the game leaves these special types loaded for faster reading. So some increase in overall memory, but less reading/processing from storage (which, come to think of it, might save memory that would've been consumed by the data parser).
It still has issues, though. AI are leaking items when the game quits. And we have a lot of catching-up to do testing this new Item code to make sure it's as stable as the old code. Hopefully, Tiago's fresh perspective on my code will highlight anything I've missed. And ultimately, help us decide whether to push forward or revert to the old code and try another solution.
The good news is that his break from memory leaks has given him time to work on the rest of the bugs he's had on hold, instead of spinning wheels on this memory leak.
Probably not the most exciting week for all you readers out there. But a necessary one. Have a good weekend, all!
Hey Folks! More memory leak patching today. I think I've finally got most of yesterday's leaks sorted out, including the encounters.
I ended up having to postpone the encounter item disposal until a bit later in the turn update code, as there is a point at which discharges and player item removal is finished and new encounter choices are populated. This is where I had to not only clear out old temp items, but also shield any in-game items linked to the temp items so they didn't get discarded out from under the player. Was a bit tricky to figure out, but seems to be working now.
Tiago alerted me to an issue with my item definition fix yesterday, in that he's "lazy loading" them to minimize memory usage and load times. As a result, I had to do a bit more scaffolding around the way item defs are loaded and destroyed to make sure the minimum necessary amount were in memory at any time. Basically, each item def now keeps track of how many items refer to it, and it only gets destroyed if that count goes to 0.
Also, I had to ignore this destroy() call on a few special item types, where all item defs are shared. E.g. newspapers, data files, recipe hints, campsites, etc.
With that out of the way, I'm now looking into hardware and software. It seems these are some of the only remaining items not released from memory when ending a game, for some reason. The hardware that contains software and batteries goes away, but not the contents. And I can't see any reason why this is so. But, I just started this task late in the day, so maybe tomorrow I'll have more luck.
In other news, a chunk of my day was wasted fighting a botnet. This time, it seems like it was trying to brute force its way into user accounts by trying a list of usernames and passwords. And when I figured out how to block that botnet, either it changed tack or another botnet coincidentally started. This time very specifically trying to access any admin-like username it could. So I blocked that, too. I think one or both are still running, as they show up in my logs. But thankfully, they're not doing much with the blocks in place.
Still, it's annoying. And kind of confusing. I should maybe put up a sign somewhere like those "no cash in vault" stickers at convenience stores. This site has literally so little of value to a hacker, I'm not sure why it'd be worth the effort.
Unless they really don't want NEO Scavenger mobile to come out soon? That must be it. They're making me pay for their lost productivity in advance :)
Hey Folks! I continued my memory leak investigation today, and I think I found some culprits. It appears campsites, encounter items, and skill selection are behaving badly.
The skill selection is an odd one, as I use a temporary skill item to check if the player has completely filled their skill or trait boxes. And for whatever reason, those test items remain in memory even after quitting. This doesn't have a huge impact, but adds-up if you play multiple new games in a session. This wasn't too hard to fix.
More worrying was the campsite leak. Each hex explored by players or NPCs loads the one or more camps for that hex. And those camps never get removed from memory. Fair enough during a single game, as they all must exist along with their hexes. However, when quitting, they remained in memory. And for longer games, this can mean a lot of unnecessary camps in memory. I think I've got a fix ready for this, too.
Even worse, every encounter item on every encounter page is getting stuck in memory. When you think about how many encounter screens you see in a game, and how many choices come up, this is a lot of items we're talking about. And that's not counting combat. If this includes combat, we're talking dozens of items per-turn! That's fire-hose leaking right there.
And unfortunately, this is harder to fix. Those same encounter items include a mix of temporary items specific to each encounter and more general items from the player's inventory. (Or more accurately, clones of them for encounter use only.) Simply destroying all of them can accidentally destroy important player items (like skills, tools, etc.).
What's worse, the way that encounter screens are resolved, I can't just destroy them each turn without getting null pointers. (Usually from trying to discharge any used tools during the turn advancement.) So I have to be really careful about when I destroy them, not just which ones. Sheesh!
But if this is as bad as it seems, this could mean plugging a major hole in memory management. So there's at least that silver lining. Here's hoping we can find a fix for it!
Hey Folks! Took some time this morning to try out the latest build on iPad, and it's getting pretty close. Almost no new bugs logged, and the worst new one I found was pretty minor. (More annoyance than game-breaking.) For reference, this was a 1.5 hour session, complete with septicaemia. No crashes, no glitches, no problems other than some frustrating item fiddling. And even then, this was during my attempts to use tiny pill bottles without using the zoom feature.
However, one major problem did show up, even if it didn't break the game: memory. By the time I quit, the game was using over 300MB, and that's a bit much for a tablet. (It'd probably crash some older ones.) What's more, returning to the main menu only dropped that to 237MB, which is a far cry from the 71MB it uses on first load.
So the memory problem is pretty real.
Not as bad as before, though. And frankly, 1.5 hours is a pretty long session for a single game on mobile. Coupled with autosave, I probably wouldn't even mind a crash after that long, provided I just reload and continue. But we're going to see what we can do to minimize that chance anyway. Tiago says Apple can be particularly strict about leaks, so I'll take his word for it.
In that vein, I've decided to see what I can do to help. My initial approach was to use hxScout to watch memory and see if I could notice patterns. Unfortunately, I think running hxScout causes some sawtooth memory usage that makes it hard to see in realtime. And my cursory digging-down through the per-frame allocations didn't tell me much.
Trying a different tack, I decided to test spawning and despawning creatures to see if they leave residual memory allocations. And after some trial and error, I was able to spawn and despawn creatures over and over with barely any net memory footprint change.
Still, memory increases over time, whether starting, playing, and quitting over and over, or even during a single game.
I decided next to see if it might be items. So far, I'm tracking the number of items created, initialized, and destroyed. And some interesting data is coming to light. No items are being destroyed in-game, which is one thing. Maybe normal (short games, no lost items), but also maybe items that should be cleaned-up.
More immediately interesting is that 80-120 items are not being cleaned-up after quitting the game. Compared to the hundreds created in a single game, this isn't much. But it might be a place to start. So tomorrow, I'll see if I can get more info on which items these are, and find out why they don't get destroyed. Plus, maybe look into why no items get destroyed during the game.
Hey Folks! Hope everyone had a good weekend. Believe it or not, I've had some spare time lately to actually play games. So I decided to finally pick up 7 Days to Die per some players' recommendations. Pretty cool! Kinda feels like Skyrim and Minecraft mushed together in a slightly fantasy-like zombie apocalypse. I'm enjoying it!
Back at work, I tried to tackle a mix of bug fixes and launch tasks for NEO Scavenger mobile. One of the bug fixes was for something I broke while fixing another issue. I was making the decimal output of certain numbers two digits, but ended up breaking one place when fixing another. Those are both fixed now.
I also helped Tiago a bit with narrowing down the cause of a null pointer bug. Could be a case of items getting destroyed during combat turn refresh, but the game still having references to their empty (nullified) husks.
And speaking of Tiago, he thinks he'll be able to resume committing full-time to the game again now that he's (mostly) settled into his new apartment. Which is good! Because I think I've been having trouble mustering the motivation to go this last mile alone. They say the last 10% of gamedev takes 90% of the effort, and man, does it ever feel like it.
Outside of the code, I went through the list of achievements to rewrite them in a style more consistent with other games. E.g. present tense, short title, short description. I also resisted the urge to drop too many pop culture references. (It was hard, believe me.)
And while I was at it, I shored-up the store page text for both Google and Apple pages so they were consistent, and created a 180x120 promo graphic to fill Google's older device requirements.
Plus, a handful of admin tasks like paying bills, replying to vendors, etc. Not a bad day, now that I look at it in review. Not terribly exciting, but I think I nailed at least a few boring-yet-necessary tasks. More to come, for sure :)
Hey Folks! Tiago's moved into his new apartment now (mostly), and was able to get back to bug fixing. And momentum is picking up!
First on the list was a null pointer bug after combat when using his memory leak fixes. We tossed around some code this morning, and were able to narrow it down to one or two places. We figured out some ways to avoid the issue, and Tiago's going to additionally make his clean-up code run a bit later to ensure the game has a chance to do whatever it needs before the creature is destroyed from memory.
We also discussed how best to include the new app icon art, as it involves some new source files and icon sizes (particularly for smartphones, now that we have the zoom UI to make playing on them feasible). I re-exported all the necessary sizes for both Android and iOS, then checked those in. Here's a preview:
Finally, I reworked the button click noises to be a bit softer. Specifically, I ran some de-click filters on it to cut the high frequencies and expose the low-mid range, which seems more satisfying and crunchy than the older, piercing noises. This is aimed particularly at small device speakers with minimal low-mid range volume.
And over in the world of space music, Josh's latest docking track is a winner. It conjures up that warm, welcoming safety feeling after days or months of tense spaceflight. I've officially approved the track, and the OST is shaping up nicely!
That's all for this week. Have a good weekend, all!
Hey Folks! Not much to show today. I ended up reworking the iOS icon a bit after some test runs. I eventually settled on a border to help segregate the icon from the home screen wallpaper, but getting that border aligned to iOS's rounded corners was a bear. And to make things worse, it appears to change with iOS version.
Fortunately, I was able to nab the svg file iOS uses to mask the icon directly from iTunes Connect, and use that to build my border. And the border itself isn't too hard to make in Illustrator. Then, once I've got the border/background done in vector art, I use Photoshop to add the pixel art character to the icon. Photoshop has some handy tools for scaling a vector layer with antialiasing while the pixel art layer remains nearest neighbor (a.k.a. point sampling). Not a thing one often does in games (it's a bit of a no-no), but for an icon like this, I think it makes sense. The icon looks like any other system icon, but with the subject made out of a pixel art sprite.
I guess we'll see if the world agrees soon :)
That was (unfortunately) the bulk of the day. Seems silly to spend so long on a simple icon, but it is an important asset. Many folks decide whether to click on your app based on that single image!
The little remaining time I had was spent on more admin stuff. However, Josh sent a revision of his latest track my way, so at least I have something to look forward to tomorrow :)