https://www.youtube.com/watch?v=N7GaWASGcgs windows zip: https://github.com/hydrusnetwork/hydrus/releases/download/v410/Hydrus.Network.410.-.Windows.-.Extract.only.zip exe: https://github.com/hydrusnetwork/hydrus/releases/download/v410/Hydrus.Network.410.-.Windows.-.Installer.exe macOS app: https://github.com/hydrusnetwork/hydrus/releases/download/v410/Hydrus.Network.410.-.macOS.-.App.dmg linux tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v410/Hydrus.Network.410.-.Linux.-.Executable.tar.gz source tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v410.tar.gz I had an ok week. I wasn't as productive as I hoped, but I am happy with the mostly optimisation work. optimisations After some more profiling in IRL situations, and with more helpful info from users, I have done another round of profiling for the new sibling cache, and more besides. A database technique I use for many purposes is now more reliable (fewer lag spikes), and has less CPU overhead. If you found some systems (like the 'related tags' suggestions in manage tags dialog) sometimes took a few seconds to work in the past couple of weeks, they should now be fast again. And you should find many types of file search, particularly those with multiple search predicates, and general tag processing, should be faster than before. My dev machine went from about 3-8k rows/s processing speed in a test environment up to 8-20k rows/s, which is faster than it was before the siblings cache was added. full list - general work: - fixed a bug in the new file service filtering code that was stopping file upload commands to file repositories or ipfs services from sticking - fixed an issue with the export files dialog auto-close-when-done function - I think I fixed a possible bug in the boot file location repair/recovery dialog sometimes not saving corrected paths on unusual file systems - file migration cancel button and shut off timer should work a bit more reliably, more to come here - copying subscription quality csv info to clipboard no longer does nice human numbers (you now get 1234, not csv-breaking 1,234)! - may have fixed a very rare 'or predicate' error when opening a dialog with a 'read' autocomplete input, like export folder or file maintenance jobs dialogs - all pages are better about dealing with missing (i.e. recently deleted) services on load, and autocompletes also - error handling from servers with strange character encodings should be better about dealing with null characters - cleaned up the combined display regen chain code - deleted some obselete db code - . - optimisation review: - after more profiling, and thanks to additional input from users, I have done another round of optimisation for the new caches. using a new technique, more than just mappings are sped up this week - a number of queries that were prone to lag spikes should now have much more reliable speed and also be faster when hammered often - . - join and analyse db optimisations: - these are mostly forcing table join orders, which reduces lag spikes, and reducing some related pre-query analysis overhead, which speeds things up more the faster your drive is (up to double processing speed on an ssd). they will affect different clients to different extents, but if your 'related tags' were taking more than a second to load, it should be sorted this week. systems affected: - archiving files - fetching 'related' suggested tags - tag siblings regen/update in about ten places - all mappings processing - additional mappings processing for add/delete, pend/rescind_pend - importing or deleting files that have tags - loading medias' tags for the first time or on regen - loading any media for the first time - num notes searches - similar files search tree maintenance

[Expand Post]

- many general file hash lookups - many general tag lookups - . - other optimisations: - mappings processing - sibling processing - wildcard tag searches, with and without namespaces, particularly when mixed with other search terms - 'tag as number' searches, with and without namespaces, particularly when mixed with other search terms - searching for tags when mixed with other search terms - has notes/no notes - searching files on 'all known files' with general file metadata system predicates (like size, filetype) - url class, url domain, and url regex file searches, particularly when mixed with other search terms - num tag file searches when mixed with other search terms - has/not has tags file searches when mixed with other search terms - sped up specific display chain regen significantly, with similar separate current/pending optimisations as last week's for combined - converted specific display cache overall regen to use a copy followed by the new chain regen rather than additive file import - sped up combined display chain regen a little bit - the splash window now updates itself with less UI overhead, so spammy updates (like the new tag regen code) use a little less CPU and fewer UI context switches next week I had some IRL fall on my head this week. There's a bit more next week. I still have many small jobs and github to catch up on, and also some final tag siblings lookups to migrate to the new system to eliminate the 'loading tag siblings' step on boot. I'll just keep pushing. I'm scheduled to start the parents cache the week after, so it would be great to have all the siblings changes squared away.