>>24188 >>>24179 >Also check out help->debug->data actions->review threads. I changed some stuff in the past week that actually moved some responsibility away from the system that panel looks at, but nonetheless, do you have a billion things in there bouncing around? Normally you'd have about 12-20 threads and maybe half a dozen 'fast' jobs and 20-30 'slow' jobs. Any chance you have hundreds? Maybe lots of downloader pages? Job Scheduler <class 'hydrus.core.HydrusThreading.JobScheduler'> '3,659 jobs' Job Scheduler <class 'hydrus.core.HydrusThreading.JobScheduler'> '4 jobs' Most of the slow jobs should be DEAD watchers for 4chan and desuarchive threads, most of which were added months ago. >Also,when you say it can't load parents and siblings, what exactly do you mean? Which panel, the 'manage tags dialog', or something else? Do they never ever turn up, or if you wait a crazy amount of time, say five minutes, do they eventually show up? Are you definitely certain that they should be appearing (e.g. an equivalent tag search in the autocomplete dropdown of manage tags shows some parents/siblings info)? The parents editor couldn't load parents, so the Add button didn't activate. I *think* I have left it open for a few hours once and nothing happened. Right now it works, but the "very busy" message is still showing.

>>24191 OK, will do. As a temporary thing I'll just run the db and the media files from that new location with --db-dir and a manual move.

All tags I use are namespaced. Is there any way to block unnamespaced tags from being entered? I'll often make a typo causing the autocomplete to vanish and an unnamespaced tag to be entered. Cleaning this up is easy, but preventing it would be better.

I'm on Hydrus 510, so changes are this was fixed and I'm way behind. But when I imported some videos with music, I noticed that they were getting quieter in the program. But as basic MP4 files in a typical Windows 10 folder outside of Hydrus, the volume is fine. And it's weird because even in my Hydrus importing settings, I made sure to have no limits of how big a file can be when imported.

Recently updated to Debian 13 and I can no longer open the client. >Critical boot error occurred! Details written to crash.log in either your db dir, userdir, or desktop! >Note for hydev: HydrusData did not import; probably a very early import (Qt?) issue! >No module named 'yaml' However yaml is installed and pip says it's fully up to date (6.0.2). Couldn't find the crash.log the error message mentions, and nothing is being written to "client - 2025-09.log".

>>24207 Ah, turns out it was written to desktop, but I just didn't see the file because it put the icon on a different monitor than usual. And to be clear, if I run python in a terminal I can import yaml with no issue. I have no idea why hydrus can't find it.

>>24208 I don't remember if this is the fix, but I *think* I choose these in setup_venv every time I install: (a)dvanced install (q) for PyQt6 (n)ew mpv (t)est OpenCV

>>24207 >>24208 Sorry for the trouble. For whatever reason, yaml has tended to be the first third-party module I import in the program, so when the program complains about yaml being missing, that usually means the venv did not install/activate correctly. https://hydrusnetwork.github.io/hydrus/running_from_source.html#running_it_1 Since your Debian updated, I wonder if your system python updated/mapped to a new location. A venv has multiple absolute path references inside it (including to itself lol), so if you move it, or your system python changes a bunch, you need to rebuild it. If you haven't yet, please try rebuilding your venv with the normal setup_venv script (I assume you are a source user, yes?) and try again. If it still doesn't work, please poke around the advanced section here: https://hydrusnetwork.github.io/hydrus/running_from_source.html#running_it And do "source venv/bin/activate" etc.. to try running it completely manually from terminal. Does your venv activate ok? If you go 'python' from the activated venv, can that import yaml ok? If you go through your setup_venv logs, does it have a huge problem finding 'wheels' and installing them? Let me know how you get on!

>>24214 Rebuilding the venv worked perfectly, thanks.

I had a good week. I fixed the debug thing I had hoped to, overhauled how some potential duplicate discovery works, and fixed a bunch of bugs. The release should be as normal tomorrow. >>24215 Great. Thanks for reporting this. I fixed the error message here for tomorrow--it'll now tell the user to check their venv etc..

>>23956 Did someone build these new coomer downloaders already? I just realized the existing ones broke and the gallery lookups just refuse to work now... i made adjustments to these downloaders before but I remember absolutely nothing so no idea how to fix them

I almost always check "post page can product multiple files", but the default is "unchecked".

What is the best way to extract image urls and seller name from eBay's big piece of JSON? "test parse" only shows the first result of "get all items", so it is hard to test. I guess you have to reach the values of "_type" and "@type" and walk back up ancestors or something.

https://www.youtube.com/watch?v=d4Qt7oFSI2A windows zip: https://github.com/hydrusnetwork/hydrus/releases/download/v639/Hydrus.Network.639.-.Windows.-.Extract.only.zip exe: https://github.com/hydrusnetwork/hydrus/releases/download/v639/Hydrus.Network.639.-.Windows.-.Installer.exe linux tar.zst: https://github.com/hydrusnetwork/hydrus/releases/download/v639/Hydrus.Network.639.-.Linux.-.Executable.tar.zst I had a good week moving several different things forward. v636 was the last release with a macOS App. Please check that release, https://github.com/hydrusnetwork/hydrus/releases/tag/v636 , for information on how to convert a macOS App install to a source install! Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html highlights The system that discovers new potential duplicate pairs (on the 'preparation' tab of a duplicates page) has been completely overhauled. It now runs more efficiently, in smaller, less-laggy blocks, with quicker UI updates. This search no longer works in shutdown time, but you can now tell it to run in 'normal' time. Most users can just leave it alone, but if you are thick in the duplicates system, have a play with it, try turning it on in 'normal' time, and let me know how it goes. It'll act on newly imported files within ten seconds, so let's see how that feels. I fixed the help->debug->profiling->profile mode for Python 3.12+. If you have been waiting to run this for one thing or another, please give it another go. I have split it into four exclusive types now--db, ui, threads, and client_api. The help and stuff has been updated to talk about it, but most users will want 'db'. 'system:number of tags' and 'system:tag as number' have a nicer new namespace selection widget. I think I'll be able to add 'namespaced', i.e. everything except unnamespaced, pretty soon too. Huge multi-column lists now behave better when you add a lot of things or select a lot of things in one go. This particularly affects the subscription dialogs when you move around or paste several hundred things at once. Should just be less laggy all around when lots of things are selected. I fixed an issue where the 'move media files' dialog would say 'all files are in the correct location' in some situations despite you having set it up to move things around. Sorry for the trouble--please give it another go. next week I am not sure. Probably some duplicates auto-resolution to-do.

>>24220 > "test parse" only shows the first result of "get all items" I'm pretty sure test parse shows all items.

>>24222 Happy 639 hydev! Been looking at the new "run in normal time" dedupe mode real quick before I go to sleep, it's pretty neat seeing my PTR petitions start increasing automatically after turning it on as pixel dupes get found. Haven't tried with imports yet, if I encounter any issues I'll let you know. The main reason I wanted to post is this entry in the changelog: >might be laggy, might be cool, might be confusing as files in downloader pages are deleted before your eyes. the super ideal here would be to collapse the whole operation into the single import job and return something like 'file was duplicate' instead of 'already in db' as the import status, but we'll see how this does That is really cool but I just want to provide one suggestion, I think it would be best if the file were to show up even if it was deduped, like have the ideal one show up in its place. Because it would be rather confusing imo to import a file and then have it never show up because it was intelligently deduped before it even had a chance to show up. If that's not possible then maybe show a blurhash or some way to very easily access the king image? Just my 2 cents. Keep working on this lovely software hydev, it's really the best.

>>24176 https://pastebin.com/jr2tUBwA Here's the output from fetching the url

>>24103 >If it is still broken for you, what do you see/parse in the test box of the parser in network->downloader components->manage parsers->kemono-whatever and what do you expect to see? I looked into it more, and it seems like it's actually an issue of encoding from the original files as kemono gets them, so there's actually nothing Hydrus can do. I went back to some old files that were imported from kemono and it turns out that some of them have nonsense filenames as well. I just never noticed until recently. my bad

Has anyone managed to fix the kemono / coomer downloaders and would be so kind as to share them, please?

docs say > Hydrus used to support native downloads of IPFS multihashes, including directories, but it ended up a bit of a mess. I expect to reintroduce it in a nicer format in future. For now, please use the URL downloader. ( https://hydrusnetwork.github.io/hydrus/ipfs.html ) Does it mean I have to use a gateway URL to download or is there a way to pass the CID to the URL downloader? If I build a parser do I also have to emit URLs pointing to my gateway?

when I updated from v637 to v639, the pixiv artist gallery page parser switched to using the outdated style urls for pixiv post pages. I checked the url class links and it looks like Hydrus linked the gallery page class to the parser that adds old style urls instead of new style urls, so I switched the link and it looks like it's back to normal. Why does Hydrus even still have the parser that gives old style post urls in the default list? they just redirect to the new style urls.

>>24192 Thanks. Sorry for the trouble. Yeah, that 3600 jobs looks like it is part of the problem here. it is probably adding a bunch of UI lag. Unfortunately, my downloader engine has some limitations and pain points from the original implementation, and it simply doesn't do parallelisation well yet. I have plans for this, but it'll be a vast overhaul and not something I can slip into normal weekly work. If you can, please try and clear out DEAD/finished watchers and queries. That said, even if there are thousands of things going on every thirty seconds, it shouldn't lock up the parents and siblings fetch. Either the whole client locks up completely, or maybe everything runs a little sluggish, but you don't usually see one thing not coming back. Forgive me if this sounds stupid, but I did rewrite the parents/siblings dialogs a few months ago, I think it was, and they no longer fetch everything when you boot them. Those dialogs now load related pairs once you enter some. So, if you enter 'blue eyes' in the left or right boxes, the dialog should quickly (or, indeed, sluggishly), fetch all the siblings/parents that use 'blue eyes' so you can see what you are editing. This makes the dialogs a lot faster to boot and more focused to work with than the old system that loaded all pairs every time you booted them. Any chance you are just unfamiliar with the new dialog workflow? What happens if you hit the newer 'show all pairs' checkbox--does it lag out for a few seconds and then show like 70,000 pairs? I just overhauled 'profile mode' in the last release, also, so you might like to run through help->debug->profiling and maybe send me a couple profiles. 'db' is probably the one you want, but if you hit 'threads' mode you'll probably get a bunch of spam related to those 3600 jobs. >>24196 I don't think so, not yet. I've been planning, on and off, some 'storage-level' tag filters, a la the view filters you see in tags->manage tag display and search, but I'm not there yet. Since your specific problem is keyboard event related, can you try flipping the 'by default, select the first tag result with actual count in write-autocomplete' under options->tag editing? I added that for a guy who went bananas over how tag results are presented. It'll probably feel jarring to you, but maybe you'll love it. >>24206 I can't promise anything, but check your 'install_dir/db/mpv.conf' file, if v510 has one. There might be some audio normalisation stuff in there. Take it out, restart the client, you might be good.

>>24219 I recommend you only check this for sites like pixiv, where a 'Post URL' really can produce multiple URLs. If hydrus knows that a Post URL can only produce one file, like for most boorus, it can perform efficient 'already in db'/'previously deleted' file import results, without having to hit the URL, when it has seen that URL before. If you set this checkmark too liberally, hydrus will perform less efficiently when it visits sites more than once, and for no benefit since the URL does not produce multiple files. >>24220 >>24223 Sorry for the shitty UI around here. Some of the string processing is half multi-line capable but half still on 'one string at a time', so a few panels around here are bad about providing 'example strings' and such to test dialogs. I do intend to improve this. If you >>24220 are talking about the normal 'test parse' button of a content parser though, yeah it should deliver everything the formula grabs. If the formula is set to grab the 1st item, that's what it will see, but if you have something like (JSON): - get the "posts" item - get the "urls" item - then get all the items It should get everything it sees. However, if Ebay is complicated, with dynamically named fields that you have to look up with a more specific 'type' fetch, then I'm sorry but it might be too complicated for hydrus. I have some support for this sort of thing, but nothing nice like a 'temporary variable' system yet, not really. You can sometimes scrape by by saying 'get the nth item' or getting all items and then using a string processing to filter the outputs to just stuff that fits a regex for 'https...' etc... If you like, post an example of the JSON and we can look at it. >>24224 Thanks. Yeah, I generally agree. It would be lovely to have the client more duplicate-aware in general at the UI level. One really big overhaul that is still missing here, that I keep meaning to do, is to get thumbnails (i.e. their UI-level media objects) aware of duplicate relationships, which is both a storage fetch problem (I need to load all that shit and store it in a nicer manager for each file on search load), and a content update pipeline problem (every time I make a db level update to a duplicate relationship, which can get a little complicated sometimes, I need to update that UI-level manager pretty promptly. Once I have thumbnails able to talk about their duplicates without a db hit, I could potentially do a bunch of this stuff since I could fetch an importer's deleted file and go 'ok, here is the king, provide a menu option to show the user the king' or whatever without having to write a new db access routine for every thing we wanted to do. I could also do fun stuff like sort-grouping files by duplicate/alternate group. Ultimately, I think this is natural. We've got some tech to apply duplicates in powerful new ways, and now the gaps in the UI are showing. I'll keep pushing. >>24225 Sorry that paste is private! I was too late. Did it look like nice JSON, or did it have some weird characters at the start/end? I think you can attach a .json file to an 8chan post? I'm not sure, I forget, but I think you can probably just post it here as a json/zip without too much trouble. >>24233 Ah, thanks for letting me know. Many such cases! >>24238 Yeah for now I think the simplest solution is to figure out a script or a notepad somewhere where you prepend the IPFS multihashes you want to download with your local gateway address, and then spam those URLs into the hydrus url downloader. You might be able to do that automatically with hydrus with some hacked up gallery downloader, but best to be KISS I think. Yeah if you make a parser, try and grab/generate-with-a-prefix a valid URL that will simply produce the file, even if it is localhost. Hydrus can handle an URL no problem, although timeouts can be an issue with IPFS. I would like to grab multihashes directly from the IPFS daemon again, but I don't know when I will get it done. >>24245 Sorry for the trouble. The older a client gets, the more cruft that builds up in the downloader system, and it can cause some crossed wires. Please feel free to delete anything that you don't need any more. I will investigate how the updated parser could have 'jumped' link--if there is an existing link, it isn't supposed to change on update, but I know I made some mistakes, particularly with 'www' domains, in the URL Domain Mask work.

>>24255 The output was a .html file

I might have asked this already but can we get custom checker options? I would like to add "super fast thread" and "glacial thread" personally.

>>24256 Aha, this is interesting. The contents of that file is JSON and it parses ok, pic related. 'result.html' is just the default filename for that debug routine, it looks like the server is working as expected. Going back to your original posts, you are getting 'filetype is not permitted!' error. I hadn't clocked it before, but that error happens when hydrus tries to import a file and it is not a jpeg or webm or whatever. I had thought you were getting this in the 'edit parser' test UI somewhere and thus the problem was that the server was delivering bad content. It sounds like the API URL with 'json=1' isn't being recognised in hydrus as a Post URL (presumably your API redirect Post URL) (so either it is seen as a File URL or there is no match), and thus the 'file log' part of the downloader is just trying to download it as a file, and since it is JSON it then fails. If you place your full URL in the 'edit url classes' dialog, does it get recognised as an URL Class? If it does, is the URL Class a 'Post URL' or 'File URL'? And if you go edit that URL Class and place the URL in the example URL input, is the URL being normalised to something else, or still valid? Is there a parser linked to it all correct in 'manage url class links'? I've updated the error here to recogonise JSON in future. If this happens to an HTML file, you get a nicer error in the file log talking about 'hey did you set up URL Classes correct?' My 'detect file mimetype' routine isn't perfect at detecting JSON, but it should work in this case. >>24258 Great idea. I want custom/favourite/template for all the normal import options, too. I hate how finicky it is to go off the beaten path with this stuff. I'll write this down.

>>23759 >I'll have a quick look at how I do this and maybe figure out a TEST checkbox that just pipes events to mpv if I don't catch them. That would be awesome. I'd be glad to test with different sorts of binds / events and provide feedback.

>>24222 The duplicates sidebar wants to be as wide on the "preparation" tab as on the "auto-resolution tab" (which depends on the table).

>>24222 The 'move media files' dialogue is now working for me. Thank you!

I had a great week. There's downloader improvements, new duplicates processing tools and quality of life, and, from a user, some neat UI features for page and media viewer navigation. The release should be as normal tomorrow. I'll make a new thread for it.

Hey Dev-sama. Checking in to say thanks for the great software, and had a question + bug report(?) What causes the letters at the end of "items" on a gallery downloader or subscription? I tried another gallery download, but it doesn't fix the problem. I assume the letters mean something. Also I switched to Wayland for testing reasons, and the only issue I've noticed is opening a video with MPV fucks up all the windows. Just FYI. I'm going back to Xorg anyways.

>>24271 >letters at the end of "items" that means that those downloads failed. there should be a "retry failed" button to try downloading them again. You can also click "file log" to see which urls failed specifically

>>24272 Thank you anon. Retrying in the file log fixed it perfectly. Dev knows what he's doing to have such good software detail, instead of just spitting out a text list.

I noticed that when comparing duplicate pairs in the auto-resolver's semi automatic mode, the order of the two files is no longer "file one is A and file 2 is B" but instead just the order that the duplicate filter would put them in. That's confusing because it's hard to tell which file the auto resolver is going to mark as the better dupe when I approve the decision. Would you mind switching it back to how it was before, where file one is always A?

>>24260 Thank you for the reminder--I'll try again to get this done. >>24266 Yep, sorry, at the moment my 'notebook' widgets size to the maximum size of their page contents in most cases. I may be able to add clever scrollbars in future, but there's a lot of legacy sludge here I want to clean up before I try anything clever. >>24271 >>24272 >>24273 Just to be clear--'N' is for 'successful/new' (rather than 'already in db'), 'D' is for 'previously deleted', 'Ig' is for 'ignored', which is tag blacklist, parser veto, or 404 etc.., 'S' is for (user-manually) 'skipped', and 'F' is for 'failed'. You can turn off 'N' and 'D' under options->downloading. >>24275 Damn, thank you, I will look at this next week! I thought I had it doing what you said, but some part of the dupe filter must be re-ordering it.

New thread here >>>/t/24277 I'll post v640 to it later today, and this thread should be migrated to >>>/hydrus/ soon. Thanks everyone!