>>24316 >>24317 Hey, I am sorry--are you by any chance running in Python 3.10 or earlier? I do a boot test every week with Python 3.10, but this slipped through. I have just fixed this on master. Please git pull and let me know if you have any more trouble.

>>24305 >>24306 Thanks, good idea. I'll make a 'report mode' for it. I'll try not to make it too spammy. >>24308 Thanks, will do. >>24313 Great idea! This will be a little computationally expensive, but I can figure out a button that fetches the current values and draws a nice bar chart. We'll see what shape curves pop out.

>>24321 That fixed it. Thanks.

>>24321 also yes i use python 3.9 something

So I'm writing a parser for this gallery API I use that has multiple posts on each gallery page. That API gives everything right in the gallery page; title, tags, URL to the content to download, and a link to the original post URL. Because this is declared as a gallery (and not a post) in the downloader's URL class, hydrus never registers the original post URL that I set with "associable/source url", even though the parser sees it when tried manually. Is there a solution for me to register that original URL with the files? Or do I need to get the individual posts' URLs and fetch those instead? Hopefully I'd avoid parsing the individual pages as they're pure HTML (no json API there) and I already have everything I need from the gallery.

>>24284 Can't seem to log in at all, even having disable https global and mixed content in about:config (using librewolf, but couldn't get in with firefox either) Not sure what I'm doing wrong, I can get the api splash page to pop up no problem.

>>24326 Does the url class have this checked?

>>24259 I can't get any of the API stuff to recognize. Also, the parse only seems to work if I *don't* include &json=1, even though Hydrus is saying it doesn't look like HTML or JSON. My guess is it's the way that rule34.xxx redirects, as it'll go from an address that looks like rule34.xxx/index.php...etc. to api.rule34.xxx//index.php... etc. which it isn't recognizing. I'm also just now noticing the double slashes after the domain

>>24328 I have tried it, but it associates the URL of the gallery page and not the URL that's parsed by the downloader, which isn't what I am after. Thanks though!

>>24326 >>24330 I can't exactly model your problem in my head, but I think I get the gist of it. I think what you want is a 'subsidiary page parser', which is a mostly-undocumented way to split a gallery page into separate posts that contain groups of metadata that is to be assigned to each file separately. Check the 'e621 gallery page api parser' and '4chan thread api parser' for examples. I'm not totally sure why the url isn't being associated here. If you are getting the file imported ok, then presumably the URL is 'working', unless we are talking about a gallery hit that includes both the 'Post URL' and the direct file URL. If that's the case, then yeah the subsidiary page parser is the trick. The e621 one is probably what you want to look at. >>24329 If none of the API URLs are recognised in the URL Classes dialog, then that's your problem, it sounds like. You have to get a matching URL Class to get a URL to parse (remember the URL Class-Parser links in 'manage url class links'). If the URL is not recognised, hydrus will keep thinking that URL is a direct file URL that it should download. If you need to match a URL with double leading slashes, try adding a path component that matches anything, and example text the empty string, "", to remind yourself, and make sure the 'DEBUG: remove leading double-slashes...' option at the bottom of options->downloading is unchecked (it should be). If you need to match the 'json=1' URL, then add that as a parameter to the URL Class. Good luck!

I'm on Linux and going to try building from source to fix the mpv issues many have talked about. If I uninstall hydrus will it leave the database alone? And then I can just point it to the database again after I re-install from source? Of course, I'll be sure to backup everything before I start.

>>24332 >subsidiary page parser I should have started there; this is what I am using already. So this API has a `posts` object with a dict inside, and in that dict, each post has a downloadable file (which is the URL that gets associated with the file currently), some tags (those are correctly associated), and a source URL (the one I want associated with the file, and isn't currently). I do use the subsidiary page parser, but I'll review the e621 gallery parser to see what I'm missing. I'll share my downloader here when I get it to work, thank you!

If you have a duplicate filter page open, and it's still searching for duplicates while you click the button to start going through the filter, the searching tab will continue to search while the duplicate filter is loading. Because of this, they both go at half speed since Hydrus is needlessly searching twice. If possible, I think that the duplicate search that the tab does should just be "carried over" to the actual duplicate filter when you start it, so that Hydrus doesn't have to do every search twice. If that's not possible, then can you at least have Hydrus immediately stop searching on the page once you start the filter, so that it doesn't slow down a bunch from searching both at once?

>>24336 oh and you've been making great progress with the auto resolution system :)

I've noticed some old files that I dropped into Hydrus directly from my computer have retroactively gotten downloader tags. Just the creator: namespace though, since I disable the rest and prefer my personal tags. How exactly is this happening? The pages these hash matching files are being found on wouldn't be scraped by any of my subscriptions. Like this pic related for instance. I imported it over three years ago, and now I've noticed it has a pixiv url from a six year old post attached to it. This is the case for about a hundred or so other files out of tens of thousands. When, how, and why is Hydrus accessing these pages to compare hashes? Is it from the PTR? I don't use the PTR, but I did try once when I first got Hydrus before shortly canceling it upon realizing my computer couldn't handle it and that I prefer my own tags anyways. Could it be leftovers from that?

>>24338 >Could it be leftovers from that? If you once synced to the PTR, and never wiped the mappings from your DB after you stopped, then yes that's exactly what's happening

>>24339 Okay, I was a little freaked out and only thought of the PTR halfway though the post. I had thought everything related to it was wiped back then, but apparently not.

>>24293 >it takes 12 goddamn hours to make my fancy deuplicated backup Try vorta for borg backup. You can select what folders to backup where, and it does incremental, compressed, backups. I'd make sure to close Hydrus before backing up though. >>24300 Hellll yeah I love this. A good ML feature would be image/video upscaling, though that is obviously going to more than double storage size and take a long long time to process everything. >>24333 Source fixed my mpv issues. IMO source is misleading, because it's just a portable program with a binary launcher anyways. >uninstall hydrus What do you mean? It runs in a folder. Just download a source version and copy or move your db into the source one.

>>24340 nah, if you just stop syncing, then anything that was already processed stays. It's been years since I stopped myself, but I think to fully remove it, you need to go to the tag migration menu, and wipe the mappings and relationships, then delete the services, then vacuum the database files. Once you do that, it should be fully gone.

Hey hydev I have some borked files that I found when running hydrus video deduplicator. Most don't run at all so just assume every single one is nsfw. I ran a force redownload and only a few got fixed so these are bad on the site's side. https://catbox.moe/c/3uqay7 There's a text file with every url in it.

I was having a lot of trouble finding the place where you set which urls get displayed in the media viewer pop-up frame. I eventually found it, but I wasn't expecting it to be under "network → downloaders", since you're not changing anything about how files are downloaded and technically it's not even about downloading, just what's displayed. I was expecting to find it in the Hydrus options menu, and the first place that I checked was "media viewer hovers". I think it'd be a good idea to duplicate the settings for url display there, or at least have some text to tell users that they're actually in the "downloaders" menu bar item.

I had an ok week working on a mix of things. There's more duplicates work, some crash reporting tech, a new stability test for mpv, and a bunch of smaller stuff. The release should be as normal tomorrow.

Would you mind adding an option (or maybe just make it the way it works if no one minds, but I'm fine either way) to have the queue for auto resolution pairs to be sorted randomly? I'm noticing strings of similar pairs in my semi-automatic rules' queues, and that's not great for making sure that a rule is solid before making it fully automatic. Going through a random sample would give you more confidence that the rule is correct for all kinds of pairs that it'll encounter, rather than just the ones it happened to queue up first. If an actual random sorting isn't possible for whatever reason, then sorting it by sha-256 hash should also be fine. I think sorting that way is basically random.

https://www.youtube.com/watch?v=Y-IUZB3PI90 windows zip: https://github.com/hydrusnetwork/hydrus/releases/download/v642/Hydrus.Network.642.-.Windows.-.Extract.only.zip exe: https://github.com/hydrusnetwork/hydrus/releases/download/v642/Hydrus.Network.642.-.Windows.-.Installer.exe linux tar.zst: https://github.com/hydrusnetwork/hydrus/releases/download/v642/Hydrus.Network.642.-.Linux.-.Executable.tar.zst I had an ok week working on a mix of things. Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html highlights I have tweaked my visual duplicates algorithm with a new tuning suite that generates lots of synthetic data to build performance statistics on. Some false negatives are now 'probably duplicates', and some true negatives now have richer summaries than just 'not duplicates'. I still trust and recommend using the test at 'almost certainly' confidence in auto-resolution rules. I also triaged my remaining auto-resolution work. There are four medium sized jobs and perhaps a dozen smaller ones. I feel confident I can clear it all by the end of the year, and that'll be the system done for a v1.0. I improved the new crash reporter I was working on last week. Unfortunately there are still some false-positive crashes and logspam when the mode is mixed with mpv, so the mode is launching default off. You can turn it on under help->debug->debug modes->use faulthandler to log crashes. If you have had crashes and can test them without using mpv, please give it a go and let me know how you get on. If you are an advanced user and would like to help me test a new mpv thing, please check the changelog. next week I may tweak the new mpv interface a little more, but I mostly want to clear the final auto-resolution work. >>24343 I forgot to say: thank you, I got these. Some worked ok for me, some wouldn't render at all. Seems like some versions of either mpv or ffmpeg could or couldn't handle them. It is useful to have broken files like this, and this is a real mix of weird stuff. A couple of them helped me debug my mpv crasher code this week

How do I follow webcomics with hydrus? I've made a post parser, but now I need something to plug into a subscription or a watcher, right? Any tips / examples?

I dual boot linux and I use Hydrus on both windows and linux, using the same db. Stuff works nicely, but now on linux I have the weirdest issue (just the usual) When playing videos like mp4s... only every second video shows. The rest only play sound. However "every second" means the order of clicking on them >click on video 1 - no image >click on video 2 - image works >click on video 3 - no image >click on video 1 again - image works I have no idea how this shit happens, it's always on linux. I guess I should run hydrus from terminal and toggle mpv reporting or something?

>>24359 I had this issue before. I wish I could remember the fix, have you rebuilt your venv or are you very out of date? Looking back at my posts in a previous thread it looks like it ended up going away after some update. There was a debug mpv command added for this, see my pic.

>>24358 Basically, you need a post parser like you have, but then you need a gallery page parser that returns post urls for the post parser, and also tells Hydrus how to advance to the next page if there are more results. Once you have that, you create a GuG (gallery url generator) which is how you input a query to get the gallery urls. With all 3, you can create a subscription, which uses the GuG for queries, then runs automatically based on your timing settings for the subscription. You also want to make sure that the url classes for those pages are properly liked to the parsers. If you put example urls in the parser definition, then Hydrus will usually do it automatically, but sometimes it gets the linking wrong. So if Hydrus complains about there being nothing to parser, or not finding a parser, check the url class links menu to make sure that the classes and parsers are linked properly. The Hydrus webpage parser is kinda limited (particularly, it can't do Javascript at all) so there are many websites that just can't be used directly by Hydrus, but if you already made a post parser that works, you're probably fine there.

I'm the one who asked for the info on idle mode. Thanks for adding it. It helped me figure out the issue I was having. I had the mouse movement setting enabled for blocking idle mode, but only had it set for 1 minute. I was confused though because I never saw idle mode turn on, even if I haven't touched Hydrus in several minutes. But the idle report mode that you added made it clear that I was misunderstanding what the mouse movement setting was checking for. I assumed it was checking for mouse movement in the Hydrus gui, but it actually watches you mouse for movement anywhere while Hydrus is open, even if it's on a different window, or even a different monitor (or virtual desktop) than the one that has Hydrus. I wanted to use the mouse movement mode to have it block idle mode when I'm moving the mouse in Hydrus, since that means I'm probably using Hydrus, but let it turn on if I haven't moved the mouse in Hydrus for a little bit, since that means I'm probably not using it, so the idle mode lag won't bother me. Since the mouse movement setting looks for the mouse moving anywhere at all, that basically means that idle mode will only turn on when I'm not using my computer at all, which isn't what I want. So I turned it off, and now only have the "user action" one enabled. Anyway, that mode's helpful so thanks!

>>24018 >put your DB in the spot the DB is supposed to go, and everything should work fine. >>24037 >https://hydrusnetwork.github.io/hydrus/database_migration.html#to_new_OS >Install a fresh new hydrus on the new machine and then, using a network share or a USB drive, copy the database folder, files, and thumbnails from the old to the new. What about all my settings? There's stuff I figured out I wanted a certain way ages ago that I really don't want to have to redo and relearn, like color coding and manually ordering the display order for all namespaces.

>>24374 Your DB folder contains everything about your Hydrus install. All the options you have set will still be there if you're using the same DB

>>24375 This is awesome. I heard DB folder I thought it refers primarily to the files themselves and the tags/urls/notes applied to them.

>>24336 Thank you for reporting this. I can't carry over, since the first guy is just counting while the second guy gets richer info, but I will figure out some sort of deferred count or pause or whatever feels good and saves you time. >>24338 >>24339 >>24340 >>24342 Just to add here, if you fully deleted the PTR service under services->manage tags, and you don't see any PTR tab when you open 'manage tags', then these tags are not from the PTR. Tags won't bleed from the PTR to your like 'downloader tags' or 'my tags' etc... If you only paused the PTR sync and it still exists, then these tags could be from the PTR. If you open up 'manage tags' on one of these problem files, under which tag service tab are these tags listed? Is it 'my tags'? If you have local tags appearing on a file as soon as it is imported from hard drive, with no 'add these tags on import' stuff like sidecars or filename tagging going on, then there's two possibilities I can think of: 1) The file was in your client before, you tagged it then, and then deleted it, and then it was re-imported. Hydrus doesn't delete tags from deleted files, so all that survives a delete/re-import cycle. 2) You've done some funky tag migration stuff before, likely with a Hydrus Tag Archive or a PTR migration, using tags->migrate tags. If you set up a tag migration on 'all known files' using sha256 hashes, then hydrus can apply tags to files it hasn't imported yet. Let me know if you discover anything else here! If you hadn't thought of it, I think you can go ['system:number of tags > 0', '-creator:*'] to better find these files. >>24350 Thanks. Yeah, this sucks. I'll put some work in here. >>24354 This is an interesting question. I had intended for auto-resolution rules to have a clever 'examine pairs of lowest quality against the next highest quality first' so I could reduce some false negatives, but when I figured out the sort code for pairs, it just doesn't fit into the auto-resolution system, which I have tuned to have extremely low per-pair overhead. For now, it just reads straight off the db without sorting, and I think due to luck really, and probably legacy structures, that mostly means smaller ids first. We are seeing similar files next to each other because of similar import and tagging times. I think I agree with you--a completely random feed is more useful, and we should choose to do that. I'll have a think about it all. I still want my low overhead, but perhaps I can be clever about the initial insert or something.

>>24358 >>24370 I have some 'how to make a downloader' help here: https://hydrusnetwork.github.io/hydrus/downloader_intro.html It is a little old, but the basics are there. A web comic is tricky because it isn't like a normal booru or imageboard. You don't put a query in to search, and there's no 'gallery' result. TBH I think the KISS answer is you just drag and drop the URL of the web comic into hydrus every time you read it, assuming you check every mondays and fridays or whatever for the page yourself. If you want to set up an automatic thing, you'll want to teach hydrus how to find the newest URL, which means a 'gallery page'. I see your URL Class is in the form domain/date/blah, which is excellent. I also see we have this nice archive page: https://www.buttersafe.com/archive/ I think you should tell hydrus that that archive page is a 'gallery'. Write a parser to grab all the page links, and that lets hydrus queue up all the Post URLs. Hitting that gallery URL a week from now will also get the new URL, and that's how the subscription will want to work. The only problem, where we have to trick hydrus, is that this site doesn't have a 'query text'. You aren't typing 'sexy anime girl' into this site to get the Gallery URL, it just has the one. My downloader engine doesn't have a slot for this yet, so you should make a fake GUG. Check my pic related. You type 'archive' as the query, and the normal URL is created no worries. That should work in a gallery download and a subscription.

>>24359 >>24363 Yeah. Due to crashing and flickering issues, I recycle mpv windows, so when you go from one video to another in the media viewer, I grab an mpv window from the pool and send the old one back to it. If you go through a sleep cycle and an 'active' mpv window doesn't restore correctly because of some GPU driver issue or whatever, you can get this situation where that continually recycled mpv window is fucked in some way, and you are seeing it 'every other time'. That debug command basically seals away all existing mpv windows and generates a fresh pool. It can 'fix' the sleep-restore issue, but if you get a bad mpv window very regularly, during normal Linux work, there might be something else going on. I apologise for how much of a headache this is--my mpv solution is really held together by duct tape. I have a help->debug->report modes->mpv report mode. It'll spam a LOT of stuff to log. If you can turn that on as a broken mpv window is instantiated and then a couple times as you try to load a file to it, maybe we'll learn something. Most of these problems on Linux are solved by: 1) Editing the mpv.conf in your db dir. Usually disabling some audio driver or something that your OS has trouble with 2) Moving from a build version of Linux Hydrus to the source version. Source version nearly always runs mpv better: https://hydrusnetwork.github.io/hydrus/running_from_source.html Let me know what you discover! >>24373 Great, thank you for the feedback! I will make it more clear that's what I am testing. >>24376 Thanks I'll make a note in that section about the db holding everything.

>>24379 >Thanks I'll make a note in that section about the db holding everything. I'm not sure that's necessary, I'm just retarded.