okay, so. i thought this was originally because of my db being broken, but after restoring a db backup from a month ago and it still happens after that, i'm just going to assume it's a bug. whenever i middle-click on a selection tag on a favorite search files page (as in, pages that have favorite searches loaded into them), i get an error that says 'OperationalError: no such table: external_caches.specific_current_mappings_cache_7_5' and nothing shows up in the new page. i looked in the cache db and the current mappings cache only goes up to 4, so i have no clue what's going on here. any ideas? (i should also clarify: this ONLY happens when middle-clicking a tag on a favorite searches page. on other file pages this function works as normal)

I had a great week. The hydrus client can now have a system tray icon–to which it can also minimise or close–framerate and 'number of frames' are now both searchable and sortable, and I was able to clear out a bunch of old bugs, including the issue where tag autocompletes would sometimes not reset back to system predicates properly. The release should be as normal tomorrow. >>13856 Thanks, should be fixed for tomorrow.

>>13863 Sorry for this problem, should be fixed in 391 later today! >>13860 I'm afraid I don't know anything about this problem. If you have a parser for the site that seems to work, could it be a User-Agent issue, as per the discussion here >>13826 ?

>>13824 >Thank you for this report. Is that in the manage tags dialog? Can you say more how that was related to drag and drop import, was this adding tags to files that had come in via drag and drop? Yes on both counts. I dodn't know if that's necessarily a cause, it's just that I didn't do anything else different from usual.

>>13832 Thank you. I have had a look at the code, but there is nothing obviously funny going on. If your other 'normal' dialogs don't sperg out like this, and this seemed like a one-off, please let me know if it happens again, or if you discover a way to reproduce it. These sorts of dialogs generally follow the 'regular_dialog' rules, which by default just try to size the dialog as small as they need to be, and normally off the top-left of their parent. My guess on what might be going on here is if somehow the initialising text was long or had a bunch of newlines, or if something strange like the OS display system changed as it was launching, maybe a system magnifier mode, or ui scale change, or moving window from one monitor to another. This sounds pretty weird to me though, and it looks like that big dialog is maybe trying to size to its parent. If the dialog seems to have that ~20px border all around it so it is a nice centered rectangle smaller than its parent (pic related), that would point to my code momentarily picking the wrong sizing system.

I had a good week mostly cleaning code. I also fixed several bugs, including some tag autocomplete issues and the problem with potential duplicate pairs sometimes being queued up between files that are already set 'not related'. The release should be as normal tomorrow.

>>13780 Thanks m8, keep on pushing. >>13787 Thank you for this report. I am not sure what is going on here, but I will look at it. >>13789 And thank you for this as well. I had a report about something similar to this, particularly related to resetting potential dupes, from another user some time ago, but I just could not reproduce it or see what could be going on. Yes, 'not related' is supposed to be 'false positive, do not ask me about them (or any cross-reference of their dupes) again'. I will give this another look. I'll do as you did and create a fresh simple db to see if I can expose it just on a small collection of files.

I had a great week. I fixed many small bugs, added some quality of life, and am rolling in updated downloaders for e621 and Deviant Art. The release should be as normal tomorrow.

>>13772 >>13773 >>13771 OK. I have it working with PIL sending the raw pixel bytes to rust's smartcrop and then in rust I have multithreading going. Except, there's a problem. Python is way too slow, specifically PIL and I'm positive opencv will be the same. It takes a second or two for a single image just to encode the pixels and then encode the bytes to send to the rust library. Doing a pure python smartcrop won't work, it's too slow. 2-3 seconds per image with pure python. It would take almost a week to generate 100,000 images. PIL/OpenCV passing in bulk to multithreaded rust would be the same. I can get smartcropping to be fast but you've gotta send filepaths, in batches.

(forgive if this reply suddenly spams four times, I am having trouble posting) >>13775 Thank you for these updates. The bottleneck is the human part. It is fairly easy to find a hundred potential duplicates, but having the human user go through them currently takes several seconds per file. My top priority here is to start integrating customisable automatic decision-making rules into dupe processing, so we can quickly clear out easier cases like pixel-perfect png versions of jpegs and then slowly extend that as we add better and more confident tools to do 'yeah, this is definitely a resize' determinations. I don't want to add crop or rotation detection until we have the current queue more under control. I can pass you raw RGB bytes, whatever is simplest on your end. Having a multi-second conversion is a bit slow, yeah. In my experience, OpenCV is about twice as fast as PIL to load an image. It also does pixel conversion stuff all the faster, since it is jumping down to a C++ dll pretty quick, and it sometimes uses OpenCL to do GPU acceleration. I am generally confident I can load, convert, and read out raw RGB values for most images in less than a second (since it happens in hydrus all the time), but perhaps there are additional encoding concerns you need that I do not understand. I am used to scheduling CPU-heavy jobs, however, so if we end up doing this and can get it working overall in reasonable time, say less than a second per file, I can just add this as a new 'file maintenance' job type and have the client do it in the background, one every ten seconds or so, and let clients catch up in a few months of relatively very light work. In this case, I would have no concerns integrating it as an optional thing.

>>13791 >I can pass you raw RGB bytes, whatever is simplest on your end. PIL strictly advises against this and IIRC opencv too, that's the issue. it has to convert from the C/C++ memory into python tuples. opencv and the like are fast because they store the data in memory as C structs, do operations on it, and never have to transfer that data to python and back. Then you get another slowdown converting all these python tuples back into rust or whatever language. Example from the PIL docs: >Accessing individual pixels is fairly slow. If you are looping over all of the pixels in an image, there is likely a faster way using other parts of the Pillow API. A convertion of RGB values into a C struct requires a loop over all the tuples. There might be a way to register a "c" type extension with opencv/pil as a plugin but that sounds really complicated. Unless I'm missing something. I don't image process in python that often. >so if we end up doing this and can get it working overall in reasonable time, say less than a second per file one option I can try is to convert the PIL smartcrop library into opencv to see if there is a sizeable speedup. that *should* lead to a pure python library with less than a second per image. with rust + filepaths I can get that under a second easily and then multiply that by cores, you would be I/O read limited. Maybe I can speed it up with simd/opencl so you don't have to multithread.

>>13776 >newfag doesn't know how to sage all fields

Damn, I just realized I've been using Hydrus for 4½ years already. I actually used to sort images into folders before, haha how quaint. Hydrus woke up the hoarder in me, and now my collection is 168777 files at 356GB. Thank you Hydrus_dev for your hard work and weekly updates.

>>13720 Yeah, I just handle a week at a time, and the time disappears. It sometimes feels like I haven't done much, and then I scroll the big changelog html file and realise I really have been working this thing over and over. It just is what I do now, thankfully I enjoy it. I am glad you like hydrus. A fun thing to do is a bare system:limit=64 search with sort set to 'time imported, oldest first'. This now shows the first n files you imported. Attached is the first image ever imported to hydrus, 8 years and 4 months ago.

I had a great week. As well as some small fixes and cleanup, sorting files is easier, framerate sort is added, and tag lookup has some nice logic improvements. The release should be as normal tomorrow.

Imported to a virgin v385 Hydrus, and it worked just fine. Also, didn't realize you could export them all together when I did it. >>13714 It does. filter_id setting still works with API calls, and I left it at the no filter(56027). If you change it to use your authcode given by Derpi when you make an account, and scrub the filter_id, it should default to your account's current filter. But I haven't tested that part myself. Also, have you tried setting derpi to not expire the login? Does that affect the cookie behavior? I haven't bothered trying to mess with the perpage parameter though, so the 15 gallery size is still there.

>>13716 Thank you friend, this is great. Thank you for the work. I will give these a test my end and roll them in to 387.

>>13725 Not a problem, and thank YOU for the work.

I had an ok week. The mpv player has some fixes and improvements, such as slideshow support, the shortcuts system deals with double clicks better and now handles closing a media viewer or filter, and I fixed a variety of smaller bugs. The release should be as normal tomorrow.

>>13686 Ok, now that endchan is back up, and because its less annoying to post there, i'll be moving back to posting there. however I have an interesting… problem?… and i'm wondering if there is a solution. so right now with hydrus I have a threadwatcher that is 2100 watchers big, and this poses some issues with parsing, some threadwatchers only have 5 images, some have 150+ some have a good name that is easy to follow, some decided a new name scheme each fucking thing is acceptable. some make it to competition and some struggle to get passed the 100 mark. so lets take something like… vore seems to be a good one. there are 3 threads where its the first word, but then there is /vore/ which has 22, then there are different flavors of vore… ect ect. now lets say I want to parse them all at once, im unable to do that because they are not in any way lined up for me to do so. now, I will admit, since the delete tagging has become a thing, I have been going through watcher lists and I probably went through several hundred this week alone parsing every images. going forward what im going to ask for has a far lesser need or impact, but if its easy to implement it would be helpful currently. is there a way to add an exception 'search' to watchers, where if I wrote 'vore' as a search term anywhere in its subject it has 'vore' it would show it and it exclusively? when I remove the term it would show everything again? at the very least for me this would greatly help the parsing process and even if it took a bit to implement, I have a saved session that has a metric fuckton of watchers as well.

>>13687 I agree. I think adding a text filter to watchers, downloaders, and subscription lists is a good idea, and I would already like that tech available for some other parts of the program. I need to do some research first and rejigger my list code to support a filtered view, and then I will try to get this done. I am sure you know, but just in case you do not: you can shift/ctrl-select multiple rows in the watcher list and then right-click->show all importers' new files to do easy one-step en-masse display of imported files.

>>13631 The search complexity doesn't appear to matter, I've tested with a number of single line searches, with wildcard or specifying a specific chapter. The files could definitely have previously had the same chapter tag deleted. The files have not currently any siblings for those tags, however its possible I would have had tried a chapter:* to set:* sibling in the past (didn't work ;)), I may have also tried chapter:x -> set:x. In checking this for you, I've noticed that I now have a bunch of 'unknown tag:'s for these images now - implies maybe some kind of db corruption, and no chapter tag at all. Unfortunately I only have my 378 DB to work off of (easily) to do any testing. * for some of these tests I need to limit the results because of the number of images/tags From 378: Searching:"chapter:*" - finds 2 images, notably these 2 images are actually duplicates and one is from a mislabeled set. Chapters found: 065, 076. Searching:"chapter:076" - 1 image, single chapter found, same as in above search. Searching:"chapter:*76" - 1 image, single chapter found, same as in above search. Searching:"*:*65" - lots*, bunch of chapters found, includes chapter 76, but includes all the images. Probably 100's of chapters - it's matching page:*65 too obviously and it's a bit awkward to count. Searching:"series:*" - lots*, large number of chapters found.

>>13633 ench chan had hardware failure and they are apparently rebuilding from a backup, but as they are small yet had a 250mb upload limit, im sure thats doing them no favors in coming back online. for the watcher just up and dying… I didn't do an extensive test when I found it was fucked, I just noticed that after putting 5 watchers on one and 1 on another a few days later one of the watchers was still unknown, thinking I fucked up a copy paste, I looked into it, and it was still correct, I then noticed that the watchers that weren't dead were not updated at all past a point despite some of the watchers getting to 151 images. im not going to lie, I have a metric fuck ton of watchers, probably something close to 5-6000, with nearly all of them dead and waiting to be parsed. However there are only 38 watchers that are 'active' and among those there were only 7 new ones that were being parsed. this has not been a problem since the watchers were first updated to the new type where I tried dumping several thousand threads at once. as for the file, im on mpc-hc 1.9.1 - just a note if you don't know, apparently one of the original people from it is still doing updates but does not have control over the main site. there have been a few small updates that made some newer encodes play correctly. vlc is 3.0.3, will update that and see pot is 1.7.21126 going a bit more expansive on my video player list, mpc-qt 1808 did play the file, (mpv front end) I have no idea why this will play it if hydrus wasnt and hydrus im assuming is using a newer version of mpv then this given the front end is 9 months old from when I downloaded it and possibly older,

>>13635 Update on this: I updated to 384 and this is the error message "help->about" yields: 2020/02/12 19:36:08: MPV failed to import because:
2020/02/12 19:36:08: Traceback (most recent call last):
File "include/ClientGUIMPV.py", line 21, in <module>
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
File "/home/hydrus/Desktop/hydrus/hydrus/venv/lib/python3.6/site-packages/PyInstaller/loader/pyimod03_importers.py", line 623, in exec_module
File "site-packages/mpv.py", line 48, in <module>
OSError: Cannot find libmpv in the usual places. Depending on your distro, you may try installing an mpv-devel or mpv-libs package. If you have libmpv around but this script can't find it, consult the documentation for ctypes.util.find_library which this script uses to look up the library filename. So, naturally I installed the "libmpv-dev" package and now everything works. In retrospect I should have tried that sooner. For some reason I had it in my head you were shipping your own MPV binary with Hydrus and weren't relying on the system package (as you would with windows). Probably my fault for not paying closer attention.

>>13588 It definitely didn't have a frame. At the time, a few other windows were resized too large (like manage subscriptions) and had their buttons halfway off screen.

You may have noticed Endchan and the /hydrus/ bunker there is currently down. It seems like they have server trouble: https://twitter.com/EndChanXYZ/status/1223397527967289344 If/when they come back up, I'll be posting there again.

I had a good week, but it was a bit light again. I have nicer volume controls for mpv, and I believe I have full mpv support for the Linux build, along with a handful of bugfixes. The release should be as normal tomorrow.

>>13550 >>13549 I've put out a question, hopefully there is a fix I can roll into 382: >>13559

>>13567 Unfortunately, I am told the 'no filter' search strategy no longer works, so I think your best bet now is to use Hydrus Companion to copy your web browser's logged-in derpibooru cookies across to Hydrus, and then it will (presumably?) inherit your account's filtering preferences: https://gitgud.io/prkc/hydrus-companion >>13551 I just did a test download, and I am afraid I get all tags ok–this sounds stupid, but can you double-check your tag import options? If it is still bad, can you do a derpibooru parser test under network->downloader definitions->manage parsers?

I had a good but light week. I have fixed a variety of bugs, including some mpv issues and instability, and some recent UI stuttering. The release should be as normal tomorrow.

Thank you, I am glad you like hydrus! Unfortunately, I am a complete sperg and have great difficulty working with others, so for hydrus I just work on my own and I am afraid I cannot take any PRs. Furthermore, the github is just a mirror of my home environment, so I am not logistically set up to take code that way either. I apologise, I just don't have the kind of brain that can work in any sort of team without exploding in a drama-bomb, so I now strictly abstain. I mostly point othe programmers who would like to help at the Client API now. Great projects like Hydrus Companion are written completely by others: https://hydrusnetwork.github.io/hydrus/help/client_api.html That smartcrop looks neat though! I will check it out a bit more. If you are particularly keen on getting it into hydrus, could you research it and summarise the best way to pull the coodinates of the interesting region of the image, or how to get a pure pixel-perfect crop in memory? I am not super interested in getting a jpeg on disk if that is possible–I'll likely want to do cropping/resizing/quality stuff in my existing pipeline. I can do a PIL image, but normally I work with numpy-compatible OpenCV images internally. Just posting five lines of code would save me a bunch of time. Also as a side note, my code is all WTFPL, so please feel free to do anything you like with it.

>>13544 Unfortunately, I could not find an obvious solution to your problem. In the coming weeks, I will be adding more configuration to the mpv window. One part of that will be the 'vo' (I think 'video output') option. Atm I have it set to gpu-hq, but it has other options for software and opengl rendering iirc. My assumption is my attempt to say 'hardware accelerate this as much as possible' has triggered your GPU into thinking it is vidya. Once that option is available, I think your best bet here is trying some different presets and seeing what changes. I would be very interested in what you see.

>>13556 Smartcrop takes a second to compute, and the python port is even slower than the javascript one at 3s or something. You would need some sort of gpu accelerated code to get fast enough for only the fly thumbnails. >Unfortunately, I am a complete sperg and have great difficulty working with others Yeah, I kinda figured. I'm the same way, so. I have been dabbling with arangodb (a document + graph database with full text support) to see what it would be like to have hydrus in a different database than a relational one. By the way, you are a beast with sql. I checked out some of hydrus dbs in an sql browser and they make my head spin (just a little). Anyways, I was curious what some of the bottlenecks are when trying to scale up so I could figure a way to improve them. Here are some thoughts. 1) Counting tags is extremely slow. Precomputing these is obviously not feasible since even just the combination of say, (5000 choose 2) would be in the millions of records. It's an NP-Complete problem. There's a way around this though and that's to only precompute the supernodes, tags which take up the majority of relationships. So, maybe precomputing the counts of tags for the top 100 tags, as well as their intersections/unions. That would only net you in the hundreds of thousands of records, might be a possibility. Precomputing all tags with no search parameters would probably a good idea regardless. 2) Sending every file with its tags over the wire to then be aggregated on the client is slow. That's probably why it takes so long to load up 10,000 files on hydrus. The solution would be to send just the ids, the bare minimum extra data (like the computed title/series/chapter at the top of the thumbnail) and then compute the counts. When selecting a file you pull the tags for that file and then cache them into the client. You can also start to pull in the tags for only what is in the client viewport. 3) Grouping files by series/chapter/etc can be precomputed and saved, or computed by sqlite. Right now it needs to pull in 10000 files just to group into, say, 100 chapters. That's pretty expensive. It also makes limiting the amount of files you show fucky. 4) full text searching is seriously fast. like really really fast. I haven't checked if you're using it but reverse indexes are the shit. Doing this kinda stuff sucks donkey balls obviously. You need to maintain data integrity constraints, and add more stuff to schedule in the background, you also can tank write speeds if you do don't pay attention, etc etc. I'll keep posting if I think of anything more, don't feel the need to pay it any attention to it and especially not with a sense of urgency.

>>13411 Thank you for all this information. I have wrestled with the idea of title/volume/chapter/page info for a long time. It is in the back of my head to somehow split tags in hydrus into two domains–one for search data like 'character:samus aran' and one for longer one-off descriptions like 'title:samus goes to mars'. It feels to me like descriptive data would benefit from a separate data structure than search data. My thoughts on this are incomplete. Although I played with volume/chapter/page sorting and similar from the start of hydrus, supporting chapters and pleasant narrative viewing has never been great. For the moment, I encourage people to stick with named .cbr files or whatever and use a program like ComicRack. It just does the job better, and for these sorts of nicely named series of files, it is fine to manage them with filenames. Although having 48,900 may be stretching it! I don't think I have time to make hydrus a really good comic reader in any short timescale, but I expect some basic cbr/cbz support will come in the next year or so, and we'll see how the upgrades that come with the media viewer for that go. Having dealt with the awkward hassle of page tags, I am not a huge fan of splitting comics up into individual pages any more, so I'd like to see how chapter/volume files goes. I'd like to generally push in the direction of something like ComicRack, with a couple of bells and whistles like bookmarks and so on.

>>13424 My thoughts are simple, I would love hydrus to handle/manage my comics but pic related on how hydrus handles images, is just not acceptable. now, take comic rack as an example, you have essentially 3 different windows/tabs, the library, the folders, and the pages, Library is a catch all 'I tagged shit I rated shit' amalgamation to get everything. folders, at least for hydrus, would be the hydrus imported comics, because I don't know about anyone else, but some days I go though my folder and just look though it and notice hey, I have aiki, I liked that, let's re read, If this was just looking at hydrus imported files, that's fine too. pages would be where a focused manga gets opened and you can see inside. personally I have files from exhentai that all come in some form of zip, I get files from nhentai that all come in a folder, and I have files that come from other sources that also get made into various formats, the ability to tell hydrus that this zip is a chapter, this zip is a volume, or this folder is a chapter/volume then move those folders to a place thats user parseable would be needed. now the main reason I need it to be user parseable is, see the aiki above, along with my other post, if hydrus shits the bed, gg, you are never getting user parseable files back, that 48k+ files is just the porn section of my manga, now, I honestly don't see 'title:samus goes to mars' as a tag I could ever use. because that would also mean that every single doujin I have would also have titles, and gg on ever finding what I want again without user parsability.

>>13424 >>13435 I think what he wants is just a "collections" feature. it's an ethereal item that is just a collection of other items, and can be represented in sql with a closure table. tv shows/books/manga are really the only scenario where hierarchical data makes much sense, although genres are still a tag so.. but besides that, these are all pretty much UI discussions not data structure ones. you can model everything relationally, hierarchically (if you're insane), with a graph, etc. you can even represent a graph database inside a relational sql one, but that stuff doesn't matter to the end user. tags for everything makes sense from a cataloguing perspective, you just have to make the UI represent the data in a more user friendly way. there's probably no point separating "descriptive data" just because it feels intuitive. the only thing that really needs to be done to solve this is just to have separate "views". a manga "view" would just hide anything that isn't series/chapter/page like you can already do now, but the big difference is that you can present it differently to the user. collections can be used to prevent database bloat, i.e. you don't need to have genre:scifi on every single page only the parent node, but again from the user's perspective it makes no difference as long as it can scale performance-wise.

I had a couple of difficult weeks, but I got some good work done. A prototype for an mpv video player embed is ready for advanced users to test! This new player renders video smoothly and finally adds audio support to hydrus. I have also fixed a heap of bugs, added some new imageboard downloaders, and improved the quality of life of a variety of UI. I still have some things to do, and plenty to test, and messages to catch up on, so the release is likely to be late tomorrow.

>>13523 Thank you. I have just fixed this now, let me know how it works for you. There are some other fixes in this area in today's release as well–please check out the changelog and let me know what else I need to catch up on.

>>13530 >>13527 No worries. I am rolling out 8kun thread url recognition and parsing today, so you should be able to paste 8kun threads into a watcher page and it'll all work, videos and everything, in 380.

I had a great week. I focused on a variety of smaller jobs: better downloader UI, nicer search workflow, m4a audio file support, better select/remove menus, and more file search speed optimisations. I caught the coughing cold that is going around, so my schedule is a bit out. The release will like be quite late tomorrow.

>>13472 Thank you so much, that fixed it! I don't expect to do any clock-changing for the foreseeable future, but I'll follow your advice if that changes.

>>13473 Thank you very much, Hydrus dev! I followed your instructions, cleaning out the old program and reinstalling thru 335 to 377 every 10 or so updates, and now the latest version is running well so far. I encountered 2 minor hurdles in the form of a "missing cache to rebuild" and the migration to the new PTR, both done smoothly. If I encounter any unusual, less-than-optimal events I'll be back with details. Thank you again for your help!

I had a great, simple week. I mostly concentrated on code cleanup and database optimisation. A number of unusually very slow searches and routines (sparse system:known url searches particularly) should now be significantly faster. I also managed to push all the simple file sorts into system:limit queries, so now if you search with sort:filesize:descending and system:limit=256, you'll get the 256 biggest files of the whole sort, not a sample. The release should be as normal tomorrow.