/hydrus/ - Hydrus Network

Archive for bug reports, feature requests, and other discussion for the hydrus network.

Index Catalog Archive Bottom Refresh
Name
Options
Subject
Message

Max message length: 12000

files

Max file size: 32.00 MB

Total max file size: 50.00 MB

Max files: 5

Supported file types: GIF, JPG, PNG, WebM, OGG, and more

E-mail
Password

(used to delete files and posts)
Misc

Remember to follow the Rules

The backup domains are located at 8chan.se and 8chan.cc. TOR access can be found here, or you can access the TOR portal from the clearnet at Redchannit 3.0.

Uncommon Time Winter Stream

Interboard /christmas/ Event has Begun!
Come celebrate Christmas with us here

8chan.moe is a hobby project with no affiliation whatsoever to the administration of any other "8chan" site, past or present.
(10.16 KB 480x360 AwSFD4ZIhf0.jpg)

Version 332 hydrus_dev 11/28/2018 (Wed) 23:05:30 Id: a74b8e No. 10810
https://www.youtube.com/watch?v=AwSFD4ZIhf0 windows zip: https://github.com/hydrusnetwork/hydrus/releases/download/v332/Hydrus.Network.332.-.Windows.-.Extract.only.zip exe: https://github.com/hydrusnetwork/hydrus/releases/download/v332/Hydrus.Network.332.-.Windows.-.Installer.exe os x app: https://github.com/hydrusnetwork/hydrus/releases/download/v332/Hydrus.Network.332.-.OS.X.-.App.dmg tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v332/Hydrus.Network.332.-.OS.X.-.Extract.only.tar.gz linux tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v332/Hydrus.Network.332.-.Linux.-.Executable.tar.gz source tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v332.tar.gz I had a great week catching up on a whole bunch of small jobs. rolling session backups The program now automaticaly makes up to ten rolling backups (i.e. it keeps the ten newest saves) of your sessions! This includes the 'last session' backup used to restore the program on boot, so if you accidentally close something very important and can't get back to it, or if your session breaks for some reason, you can now rewind a bit further. You can access the backups under pages->sessions->append session backup. They are listed by the time they were made. The client also now saves an 'exit session' on a clean client shutdown. This is different to the 'last session', which is updated every few (default 5) minutes, and provides a longer-term backup. This works for the entire in-client serialisation system, so if this works out, it may make sense to apply it to other objects such as the bandwidth tracking system, just to provide an easy additional layer of safety. misc There's a bunch of small stuff this week: Gallery import pages now have a cog button beside the query input that lets you control if new imports start with files and/or search paused! It remembers its values through a session reload. Tag import options that can be 'default' now have a button to specifically load a default, so if you want to do a specific download that has exactly the normal default options but one little thing different, you can just load it up and edit it rather than having to recreate the whole thing from scratch. The new tag id cache is expanded, and I've written a similar cache for file ids. A whole bunch of database operations should be just a bit faster. Export folders can now be told to delete the exported files from the client after they run. (So you can, for instance, tell your client to send all files with 'nsfw' to an import folder for another client.) If you decide to do this, it'll throw up some warnings just to make sure you are ok with it. When running database->check->file integrity, the client can now optionally export a .txt file with all the missing files' known urls! This same list can be pasted right back into a new url downloader in the client to try to get them back! full list - the client serialisation system now supports multiple rolling backups! - client sessions (like the 'last session' that typically loads on boot) are now automatically backed up to ten times in rolling backups! you can review and append the backups if you need to recover from _pages->sessions->append session backup_, where they are listed by their timestamp! - when the client closes, an additional 'exit session' is now saved. this differs to 'last session', which is overwritten every x minutes, and is now available (especially with the new backup) for various error recovery situations - gallery import pages now have a little cog icon to control if new import queues will start with files and/or galleries paused. these states persist through a session reload - tag import options that are 'defaultable' now have a button to let you load in a specific default, so you can easily quickly edit in a one-time slight alteration of the default rather than having to create everything from scratch - under options->downloading, you can now set the 'delay' times on gallery/watcher network error, subscription network error, and subscription other error, now defaulting to 90mins, 12hours, and 36 hours respectively - attempting to launch a client with db version > software version now spawns a blocking messagebox on pre-launch informing the user of the risks and advising task manager force-kill of the process - did a little cleanup on the new tag id database cache and merged some other, older semi-laggy tag-fetching code to use the same system
[Expand Post]- wrote a similar 'file id' database cache for caching file hashes and also merged some old hash-fetching code into it. a variety of file operations are now significantly faster - export folders can now delete files from the client after export. the edit panel will warn you on selecting this and oking the panel. it can't be set if the export type is 'synchronise'. - fixed the edit export folders dialog's old buttons, which were semi-working due to some recent update work - when checking for file integrity, you can now choose to export a .txt file listing all the missing files' known urls, so you can try to recover by feeding them all back into a new url downloader! - the physical file deletion process is now a little simpler and deals with larger jobs in smaller batches, no longer hogging the file read lock the whole time. clearing a large trash should no longer hang other media loads - adding ngugs in the 'export downloaders' dialog now attempts to add the respective gugs as well - fixed an issue where the listctrl would accept certain kinds of duplicate data and hence confuse its indices - fixed all the add buttons on the export downloader panel to exclude all items already in the list when figuring out what additional objects to add - the 'scroll thumbs at this rate per tick' option now _rounds_ the pixel result, rather than always _floor_ing it - wrote a new mass-selection database routine that should reduce memory footprint of autocomplete regeneration. if it works out, I will use it in some other places - improved the errors when a network job that cannot wait fails on an invalid login, and added a separate error when it refers to a hydrus service - fixed a small typo bug when trying to auto-add url classes and parsers - updated an old deprecated checkboxlist call - clarified the integrity section of 'help my db is broke.txt'. - misc fixes next week Next week should be more small stuff like this. Maybe some fun/new for 333 if I can fit it in. I mostly want to tidy up for a 'python 2 final' 334 on the 12th Dec. On the 12th, I will break up for Christmas to convert the program to python 3.
Anonymous 11/29/2018 (Thu) 00:36:05 Id: 50c7ae No. 10813
>>10806 that solved the issue though it also takes away 10 potential button presses, I don't think it matters much as I can change out the shortcuts i'm using. with that said, in the custom shortcuts, would it be possible to have a drop down menu for them and a check mark next to them when selected, kind of like the pause menu stuff, at least this would be a nice in ui way to switch to what you are using without going full screen, loading your short cuts, and needing to do this everything you exit full screen. granted just a note this isn't something I would use immediately, but it would be a nice feature when I finally get to a second phase of rating images into categories.
Anonymous 11/29/2018 (Thu) 00:38:56 Id: 50c7ae No. 10814
I have one issue with saves, a while back you found out they caused a crap ton of hang for me, I have also noticed that when the client goes into what I think is a save, it eats 4-6gb of ram bringing the running ram use of hydrus from 12 to 18 not sure if there is much you can do, but a less ram intensive way to save would be nice.
Anonymous 11/29/2018 (Thu) 02:06:41 Id: 976f7c No. 10815
Any chance you could raise the thumbnail width and height limits? 200x200 is too small for my eyesight.
Anonymous 11/29/2018 (Thu) 03:15:17 Id: 50c7ae No. 10816
(99.83 KB 1117x400 client_2018-11-28_20-59-31.jpg)

>>10815 i'm wondering this too, but not an out right 200x200+ thumbnail, but a scaled up thumbnail where its still 200x200 but I could have it display as 300x300. at least the way I have things set up, I have a very large preview window, nearly 900x900, this serves almost perfectly for my needs but if I just want to sort via thumbnails, being able to scale 50-100% bigger would come in handy, sure at 50% I may only have 5x6 or at 100% i would have 4x5 at current hydrus size, but this would also likely be faster as I don't have to take as much time sorting,
Anonymous 11/29/2018 (Thu) 07:03:06 Id: 50c7ae No. 10817
(10.67 KB 578x408 taskmgr_2018-11-29_01-01-45.png)

here is an image of the save issue where it takes a crap load of ram, at least I believe its saving. for the entirety of that section, the program is locked up.
Anonymous 11/29/2018 (Thu) 11:47:40 Id: 354fcd No. 10818
I never got around to testing the new login scripts for rule34hentai back when it was added to the new login scripts so I'm not sure if this was recently or always been this way but the downloader for that site still doesn't pick up hidden images where you need an account to see. Login works fine I guess, it knows I'm logged in. I'm just not sure if its the downloader or the login script that's causing Hydrus to ignore hidden images when downloading from rule34hentai.
Anonymous 11/30/2018 (Fri) 00:18:09 Id: 50c7ae No. 10824
Ok, while its not as big of a problem as it use to be, tabs are moving around fairly regularly in the program. The biggest problem for me is that all my top level things are page of pages, and those are starting to move around, making it difficult to put the program back in order when things want to move. Is it possible to get tabs to either lock adding new things to it or lock tab positions? Hell, if you have a tab/thing I can put in that just acts as a spacer between tabs, that would help immensely with re organizing because i'm not putting shit into tabs when trying to put everything back in place.
Anonymous 11/30/2018 (Fri) 08:09:21 Id: 65a947 No. 10825
Can you implement a setting on how much the main thumbnail view scrolls with one tick of the scroll wheel? Because the setting I'm using on my mouse now which works well for most other programs scrolls the thumbnails view way too much. One tick scrolls about 2 pages right now. Ideally I'd like one page at most. I just updated to Win10 from Win7 and it wasn't a problem there.
Anonymous 12/01/2018 (Sat) 19:01:32 Id: 190dfa No. 10834
An issue I have now is with the PTR sync. It recently got stuck for some reason. So I decided to reset it, which worked. But it's now so very slow at re-processing the updates It's only working at a bunch of dozen rows per second but especially when it seems to have done one update (say, 880/1621) it just gets stuck at the end for minutes to half an hour - often it seemed like it just wouldn't continue. What's going on there? The database should be stored on a relatively fast HDD on XFS. I recall various Java databases (h2, hqsqldb, cassandra) and other databases (rocksdb and so on) were able to insert k-v or full tuples in the order of quite a few thousands to tens of thousands per second with no weird hangups. >>10814 I was also having some issues with Hydrus' RAM usage. Maybe it was also mainly during saving, though I also couldn't explain why it was using x GB RAM (can you tell why yours is using 12GB RAM?). It doesn't seem to keep all the full sized images and videos loaded in RAM, nor should it have to, right?
Anonymous 12/02/2018 (Sun) 09:21:02 Id: 50c7ae No. 10839
(7.02 KB 401x173 client_2018-12-02_03-18-52.png)

Ok, in making an effort to process my images, I halve the window as in I select half, and I drag and drip them to another page of pages nested inside each other till they are a reasonable sub 1000 size, see image for example. However in doing this I found out that moving one to another has started to just combine the files back into one bundle. its not a pressing bug to me to fix, but its something that I think needs to be addressed because I think this will cause problems in the future.
Anonymous 12/02/2018 (Sun) 17:21:56 Id: e1629a No. 10841
VERSION 333 - SOON TO BE WITH THEE
hydrus_dev Board Owner 12/02/2018 (Sun) 19:57:54 Id: a74b8e No. 10854
>>10813 Sure, this is a good idea. >>10814 That seems pretty huge! I assume this will blow up your client.db size as well, if up to twenty copies (last_session and exit_session) of your main session build up. If the db size becomes a problem for you, I can add an option to change the number of rolling backups kept. This answer is going to suck, but the best way to reduce memory usage is to reduce the size of your session. I do not recommend having more than 50k thumbnails/urls total in your session, so if you have 800k or something, that is obviously going to add up when it comes time to serialise it all and save it to db. I explored a 'what's in memory' debug system a while ago, but didn't have luck getting a stable solution. I am going to try again in python3 in the new year. If I can see what is actually eating that memory, we'll be able to figure out how to reduce it.
hydrus_dev Board Owner 12/02/2018 (Sun) 20:00:20 Id: a74b8e No. 10855
>>10815 >>10816 Due to some technical bullshit, I can't do this by just increasing the number. But I would like to do something here. I am afraid it did not score highly on the 'big things' poll, so we may have to settle for a dirty solution rather than a beautiful one.
hydrus_dev Board Owner 12/02/2018 (Sun) 20:05:11 Id: a74b8e No. 10856
>>10818 I have not used it myself, and I don't think I wrote the shimmie login, so I can't comment cleverly. If it has logged in truly correctly, it should show what your regular browser login shows. Maybe try going manage logins and then hit 'reset login', to force it to try again. If it is still broken then, let me know and I'll dig a little deeper. It might be r34 is returning some specific error and the script isn't picking up on it, thinking everything went ok. You might like to try help->debug->network actions->fetch an url and put in a url you expect to have access to. That should let you save the html, so you can scan it in an editor or load it in your browser and see if it looks right or if it says 'not logged in m8'. You can pastebin the file to me, but be careful if it has private info like your username in it. Let me know what you find out.
hydrus_dev Board Owner 12/02/2018 (Sun) 20:09:42 Id: a74b8e No. 10858
>>10824 >>10839 Thanks. I think the nice answer here is to improve the ui code so it doesn't stutter and hence interpret some clicks as tab drags. But maybe in the meantime I can just have an option to disable them, or yeah, pin them. I like the idea of pinning most, I think. Pin against close and pin against move. If it helps, and you haven't seen, you can right-click those page tabs to get a bunch of different move/collect options. This might be more reliable than DnD for you.
hydrus_dev Board Owner 12/02/2018 (Sun) 20:12:07 Id: a74b8e No. 10859
>>10825 Try options->thumbnails->rate per scroll tick. This is a little experimental. Typically, for every physical scroll, the OS delivers 3 scroll events to hydrus, so a default client tends to scroll three thumb heights per physical scroll. Two pages at once seems high. How high are your pages, approx, in thumb heights?
hydrus_dev Board Owner 12/02/2018 (Sun) 20:19:16 Id: a74b8e No. 10860
>>10834 Check your options->speed and memory to juggle the size of the caches in the client. It usually reserves a few hundred MB for things like image and thumb caches. Most of the size of giganto clients is in huge sessions that have hundreds of thousands of import objects or thumbnails. I recommend clients not have more than 50k thumbnails/urls total in their main session. For your PTR sync, the main bottleneck on HDD clients is random access read latency. You can improve this a bit by increasing the db disk cache under than same options panel. Since your reset tables may be new and un-analyzed, you might like to try database->maintenance->analyze->force as well. Failing that, here's some help to help get me more info: https://hydrusnetwork.github.io/hydrus/help/reducing_lag.html The best solution is to move your db to an SSD though, where processing time typically increases to 20-70k rows/s. Putting the db on SSD and files on HDD works great: https://hydrusnetwork.github.io/hydrus/help/database_migration.html
Anonymous 12/03/2018 (Mon) 20:14:03 Id: 50c7ae No. 10876
>>10854 the number you are looking at is 40k before the client starts to have lag spikes, with everything split into sub 1000 image tabs, its going to be alot easier to just go through it now, but it will take some time. as for db size, its gone from 28gb to 37gb this version. granted this is more an issue when backing up then it is an issue on the nvme >>10855 for me that's in the would be nice category "Add an optional system to record why files are being deleted" Is in the critical one, though its not doing a whole hell of alot better. I have a deadline where this goes from would be really nice to have to shit needs to be there to use the program at all and that's 721gb, so possibly another year and some change before the 3tb drive runs out of space. Like I said a while ago, user notes remembers on delete, if user notes would just be linked to the downloaded files I could at the very least set up some macros for common things and just add them to the files notes, I would see them on re import. any system you make here doesn't need to be a forever system, it just needs to be a system that won't lose its data when the forever system becomes a thing. >>10874 Here's hoping that they don't move to fucking twitter or some other shit and go to an actual porn hosting site like hentai foundry. will definitely miss short stack approved though.
Anonymous 12/03/2018 (Mon) 20:17:01 Id: 295ce3 No. 10877
The tag import copy from defaults does not copy blacklists.
Anonymous 12/03/2018 (Mon) 21:26:24 Id: a55e7b No. 10878
>>10874 SHUT THE FUCK OUT KID DO YOU KNOW HOW MANY DOWNLOADS I STILL HAVE At least I'll have less filter-related dupes after this. >>10876 They're already moving to Twitter like the Japs. Some to Newgrounds too, of all places, because they gave a timely reminder on Twitter. Shit's fucked, all I can say is: at least they're not on DA! I'd wish for Pixiv but they have almost the same policy, just shittier enforcement. At this of all times, I'd put together something and try to get advertisers to pay me bitcoin to avoid payment processors altogether, but I don't know how to get artists to actually use it or how to get exposure. Do you think ad revenue sharing would work enough to get traction?
Anonymous 12/03/2018 (Mon) 21:41:44 Id: 190dfa No. 10879
>>10860 >It usually reserves a few hundred MB for things like image and thumb caches. I'd expect that. Not sure how it gets up to multiple GB of memory used over time > I recommend clients not have more than 50k thumbnails/urls total in their main session. The URLs are a bit surprising… do subscriptions count somehow? > the main bottleneck on HDD clients is random access read latency It's not merely very slow even compared to my experience value for various HDD-hosted databases. It gets stuck for hours. Maybe indefinitely now, nothing happened over 6 hours. > Since your reset tables may be new and un-analyzed, you might like to try database->maintenance->analyze->force No result, unfortunately. > The best solution is to move your db to an SSD though I'm considering it. Not terribly easy to do in this machine, but I guess it's an option if I pay enough to replace an existing SSD with a much bigger one. OTOH again, the HDD storage isn't that slow and it can have a bunch of hundred MB of RAM to do it's job if it must- it worked on other databases with dozens to hundreds of millions of entries before. Never had them lock up weirdly like hydrus' GUI suggests its DB does; they'd generally drop down to a bunch of hundred rows per second on more complex queries that edit data inside tuples.
Anonymous 12/03/2018 (Mon) 21:42:09 Id: fdee05 No. 10880
>>10874 This has to be the one of the most retarded decisions I came across that a company could have made.
Anonymous 12/04/2018 (Tue) 09:32:16 Id: 354fcd No. 10884
>>10883 Crap meant to reply to >>10856
Anonymous 12/04/2018 (Tue) 18:26:10 Id: da63be No. 10885
>>10841 >halfway to version 666
hydrus_dev Board Owner 12/04/2018 (Tue) 20:35:35 Id: a74b8e No. 10886
>>10877 Thank you for this report. This is fixed for tomorrow.
Anonymous 12/05/2018 (Wed) 13:32:16 Id: fdee05 No. 10901
>>10874 Apparently, someone on Danbooru found a way to download raw images from Tumblr that was recently removed. I don't think this can be integrated to Hydrus, especially not before December 17 but it may be a good idea to download all the stuff manually before Tumblr nukes all the artworks, so I decided to let people know. Here's the method: >I think it's still possible to download raw sources from Tumblr, just very inconvenient. I don't think it's intended so it might be fixed, but with the upcoming ban on adult content I figured it's now or never. If you export a blog you run (Blog Settings -> Export) then you get seemingly full size images even from reblogs; I've received 20MB PNGs and JPGs from this export. I last tested this on Nov 24 as all my blogs are 10+GB and I'm on mobile currently. If there are any Tumblr artists who don't post to a proper art sharing site like Pixiv this may be useful.
hydrus_dev Board Owner 12/05/2018 (Wed) 20:39:42 Id: a74b8e No. 10905
>>10876 Thanks. Yeah, if your db is that huge, and grew that much, I guess your session is about 500MB. I recommend you cull as many download queues as you can and put off the processing tabs with 'x to process'-type local tags so you can find those files again later.
hydrus_dev Board Owner 12/05/2018 (Wed) 20:44:07 Id: a74b8e No. 10906
>>10876 >>10878 >>10880 >>10901 I'm still laughing/in disbelief about tumblr's decision. I thought after the uproar they would swiftly reverse the decision, but I guess they don't want to subsidize free porn bandwidth any more. The only conclusion I can make is Verizon just don't have a real business plan for it. No idea why they took it on in the Yahoo purchase. What a way to kill a brand.
Anonymous 12/06/2018 (Thu) 20:53:59 Id: 006ef8 No. 10917
Is their a way to search through Notes or see which images have Notes written on them?
Anonymous 12/08/2018 (Sat) 08:07:30 Id: 65a947 No. 10933
>>10859 Thanks, 0.8 works better, it scrolls a full page per tick now. I have my scroll length set to 5. One page is about 5½ thumbs high. I think this might also be related to the fact that I have Windows 10 set to 125% scale, Hydrus doesn't seem to handle that quite correctly (for example when resizing the width of the tagging panel to the left of the thumbnail view).
hydrus_dev Board Owner 12/08/2018 (Sat) 20:51:22 Id: a74b8e No. 10937
>>10917 Not yet, but I'd like to present that information better in several ways. Adding search would also be useful.
hydrus_dev Board Owner 12/08/2018 (Sat) 20:53:32 Id: a74b8e No. 10938
>>10933 I hope to check out high-dpi stuff in the new year. The new wx is supposed to be able to handle it better, so I'll have a look and see if I can figure out what flag I need to set or whatever. I have a 4k monitor now, so I can test this better.
Index Catalog Archive Top Reply
1
Forms
Delete
Report
Quick Reply