/hydrus/ - Hydrus Network

Archive for bug reports, feature requests, and other discussion for the hydrus network.

Catalog Archive
Name
Options
Subject
Message

Max message length: 12000

files

Max file size: 32.00 MB

Total max file size: 50.00 MB

Max files: 5

Supported file types: GIF, JPG, PNG, WebM, OGG, and more

E-mail
Password

(used to delete files and posts)

Misc

Remember to follow the Rules

The backup domains are located at 8chan.se and 8chan.cc. TOR access can be found here, or you can access the TOR portal from the clearnet at Redchannit 3.0.

US Election Thread

8chan.moe is a hobby project with no affiliation whatsoever to the administration of any other "8chan" site, past or present.

(12.02 KB 480x360 CWYTOOtplY8.jpg)

Version 331 hydrus_dev 11/21/2018 (Wed) 23:12:19 Id: 6850cd No. 10747 [Reply]

Message too long. Click here to view full text.

29 posts and 1 image omitted.
>>10812 What were you trying to download? I sometimes open various(10-20) doujinshis at once and use a full page script to leave them all opening at once to then save them, but it's a short rapid burst. If you're downloading a lot of shit over an extended uninterrupted period of time that could be why.
>>10822 the last one I downloaded was an 8 page comic with everything spaced out 1 minute, I did all my testing on very small downloads and regardless of what I did, 1-1 week long lockouts. It seems they want you to pay them for the ability to download things, as 20$ gets you a weekly 5gb pass on downloads, something I would gladly pay, but I need to link bank accounts and do money transfers for cryptocurrencies, and i believe my bank was one that was one that was punishing people for even trying.
>>10811 >>10808 Thanks. I hadn't heard of the jpg-orig trick. Hydrus currently does the :orig conversion, which works well as long as the artist themselves didn't originally upload a 350x400 80% quality jpg. :^) I know that twitter can provide direct mp4 links for videos, but I haven't figured out a way to parse this info. This service provides it: https://twitter.com/this_vid But takes a couple of minutes to produce an answer, so my guess is they manually request it from twitter using a legacy "I'm an old phone" http header or something and then forward the link. Normal twitter video works with some DASH streaming or something that we can't handle (yet). If you discover a way to get simple mp4 links for twitter vids, please let me know!

>>10732 Thanks. I found this after a bit of searching: https://bugs.archlinux.org/task/60855 https://github.com/opencv/opencv/issues/13202 So I guess those are some places to watch? If the python bindings for cv got messed up in the new version, I should think reverting is the simplest solution for now. OpenCV is critical to Hydrus's drawing, so I'm afraid there isn't much I can easily do on my end.
>>10728 Thank you for reminding me of this. I made a new url class, rolling out tomorrow, that seems to add complete support. (we'll see if after the official split if sfw content/api is still stored on 4cdn domain, which the current watcher goes to)
>>10730 >>10729 learned a new thing since posting that. 4chan lost stripe, you cant buy passes with credit cards, this is the reason for the split.

>>10554 >>10553 >>10555 >>10557 Great, I am glad this worked. In truth, I haven't made many significant db changes in a while due to all this downloader work–it has mostly been smaller object updates, which for me are less subject to bit rot. Let me know if you run into any problems. Things like manage subscriptions are probably in a new location for you and a bit more powerful and complicated. Try dragging and dropping an image tweet URL like this– https://twitter.com/suzybaeily/status/1059168488244342785 –on your client!
is there a way to view your library on an android device via streaming?
>>10734 Not yet. Vote for API here >>10654 .

Version list Anonymous 11/17/2018 (Sat) 23:14:33 Id: 19e6c8 No. 10703 [Reply]
Hey do we have a compiled list of every version and it's features? I haven't updated in AGES and I imagine there are a lot of new things, but I don't want to dig through every individual OP to find out. Can you make a compiled list of versions with major changes, kindof like lists of minecraft updates as an example, where each version will broadly list "bug fixes" or "added desert biome" longer explanations for larger feature updates. Or even, can that happen going foreword somehow?
What version are you using?…
version 310 to 330, but I haven't actually kept up with new features for far longer than that.
Here's the masterlist: https://hydrusnetwork.github.io/hydrus/help/changelog.html You can search by 'version x'. Each changelog is generally formatted in the order: Stuff most users care about (new features) Stuff individual users care about (bug fixes for some dude) Stuff only I care about (code cleanup) So scan the top third or so of the lists and you should get a decent idea of what's happened! v310-> has mostly been downloader and login stuff.

(104.98 KB 1366x768 neat.png)

Anonymous 12/08/2015 (Tue) 10:41:42 Id: 528e0b No. 1553 [Reply] [Last]
Reposting from >>>/tech/457184 Maybe relevant to your interest. Japan did it. Automatic anime images tagging with neural networks. THE FUTURE IS NOW http://illustration2vec.net/ http://illustration2vec.net/papers/illustration2vec-main.pdf
39 posts and 1 image omitted.
>>6428 running pip install peewee seems to have fixed the "tags not being read from the page" problem. Consider all of these bug reports fixed. >>6332 >>6295 >>6180 >>6175 >>6165 >>6157 >>6149 >>6137 >>6130 I'm gonna start using this a lot now, thanks.
Necrobumping for great justice
>3 years ago Oh fuck.

Big things to work on next hydrus_dev 10/31/2018 (Wed) 21:34:04 Id: a13630 No. 10429 [Reply] [Last]
With the download engine and login manager coming to a close, I will need something new to be anxious about and near-overwhelmed by. I will put up a poll in a few weeks for everyone to vote on a big list of possible new features that are too large to fit into my normal weekly work. The poll will allow you to vote on multiple items. I hope to work on the most voted-on item for two to three months before starting the cycle again. This thread is for discussion of the list, which at current looks like this: - Just catch up on small work for a couple of months - Improve tag siblings/parents and tag censorship - Reduce crashes and ui jitter and hanging by improving ui-db async code - Speed up tagging workflow and autocomplete results - Add ways to display files in ways other than thumbnails (like 'details' view in file explorers) - Add text and html support - Add Ugoira support (including optional mp4/webm conversion) - Add CBZ/CBR support (including framework for multi-page format) - Add import any file support (giving it 'unknown' mime) - Improve 'known urls' searching and management - Explore a prototype for neural net auto-tagging - Add support for playing audio for audio and video files - Add OR file search logic

Message too long. Click here to view full text.

102 posts omitted.
(49.88 KB 767x693 jap2.jpg)

>>10611 Nice. >>10429 >Add an interface for waifu2x and other file converters/processors Would it be possible to work with offline versions as well? I installed waifu2x on my machine so that I wouldn't have to rely on an internet connection. t. linox
I am likely to make the poll today, with the release. I may unsticky and lock this thread to move convo over there, but I am not sure. >>10622 That mkv jpg solution looks great! Thank you for figuring out the variable frame rate stuff and putting it a script together. I have copied this to my ugoira notes for when I get to this. >>10623 Let me know if I misunderstand here, but you can probably do this now by running the problem file through a program like HxD to figure out its sha256 hash and then searching in hydrus in 'all known files'/'public tag repo' search domain using system:hash=abcd… . That said, if a file cannot import to hydrus, it likely doesn't have any tags in hydrus–or do you mean like 'what tags it has on the site I meant to get it from'? In either case, I'd be interested in examples of files that look fine but won't import. Please feel free to submit the files themselves or URLs to them! >>10639 Thanks–I put this on my 'see if you can sneak this in' list a little while ago, and it just didn't happen. I am adding it to the list here as "Add an incremental number tagging dialog for thumbnails". >>10648

Message too long. Click here to view full text.

>>10650 The poll is up! Please go >>10654 to vote!

(8.89 KB 480x360 zlp2VEUTTeI.jpg)

Version 329 hydrus_dev 11/07/2018 (Wed) 23:44:06 Id: e517db No. 10573 [Reply]
https://www.youtube.com/watch?v=zlp2VEUTTeI windows zip: https://github.com/hydrusnetwork/hydrus/releases/download/v329/Hydrus.Network.329.-.Windows.-.Extract.only.zip exe: https://github.com/hydrusnetwork/hydrus/releases/download/v329/Hydrus.Network.329.-.Windows.-.Installer.exe os x app: https://github.com/hydrusnetwork/hydrus/releases/download/v329/Hydrus.Network.329.-.OS.X.-.App.dmg tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v329/Hydrus.Network.329.-.OS.X.-.Extract.only.tar.gz linux tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v329/Hydrus.Network.329.-.Linux.-.Executable.tar.gz source tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v329.tar.gz I had a great week. The login manager is now ready for everyone, and I did some neat other stuff as well. login manager I have fixed up how the login manager deals with various errors and am happy for everyone to give it a go. This release also introduces login scripts for Deviant Art, Danbooru, and the Gelbooru sites. >Please check it all out under network->downloaders->READY FOR CAUTIOUS USE: manage logins. This ui needs a bit more work to make it user-friendly, but you basically enter some credentials (username & password) for a domain and then 'activate' the login script for it. Thereafter, the client will try to log in to that site whenever it needs something from it and doesn't think it is logged in. It will make a little popup window while it logs in just to let you how it is doing. If it fails, it will 'block' that domain, and any network jobs will hang on it and any subs will pause on it. Most logins last from 30 days to five years, so after the initial success, you likely won't see it ever again, particularly if you use subscriptions. It seems to work ok!

Message too long. Click here to view full text.

27 posts omitted.
>>10644 > watchers' should probably be watcher's watchers' here stand the for files of the watchers. In English this is correct. > new files What is a new file?
(59.95 KB 1128x698 client_2018-11-14_03-47-15.png)

bit of an issue with how I set the downloaders, I did not anticipate that it would come back with a dead status, does this mean it wont check going forward and needs babysitting, or is this something that will be solved going forward with a subsequent subscription check? If it's going to fail when a range sees no images, but I can't expand the range to much… at least till a derpy login is made which will hopefully have 50 a page… because at 15 images a page it says its has caught up to often, I am at a bit of a loss as to how to proceed. The way this looks to me is it wont retry next check, I could be wrong, but is there a way to force it to never not look at a sub?
>>10642 Your solution in >>10643 looks maybe a bit cumbersome and autistic to me, but if you find it works and doesn't annoy the server you are talking to (I imagine generating those queries is a little CPU expensive on their end), then that could be a solution. I might tune those ranges to be a bit broader, with maybe only five 'bands' of upvotes. The DEAD result in >>10646 is an artifact of that particular band not happening to have files on its first check, which is related to how thin it is. I recommend you wait a couple days and tell that single query to reset/check_now and see if you can initialise it with some files. As long as your check options for this sub are sufficiently broad, like 'only die if less than one new file in six months', I shouldn't think this kind of ratings based feed work ever die (after the first good initialisation) as long as the site is active. >>10644 Do you mean like 'the files that appeared since I looked last'? Unfortunately, watchers don't keep those sorts of records, so they only ever reproduce what their file import options have set in their 'presentation/publish' settings. For the apostrophe, I meant that as the plural of watchers, as >>10645 says, since it works for multi-selection. It doesn't fix itself if you only have one watcher selected. There are a bunch of places around the program where I lazily use the plural label like this even when currently only one thing is selected, like 'do action x to this 1 files', because the finicky text alteration every time for a singular selection is a pain in the ass. I eagerly await the AI takeover that will force all humans to standardise to One Perfect Language that has a single letter for pluralisation and a library I can import to handle it automatically.

Release Tomorrow! hydrus_dev 11/14/2018 (Wed) 00:23:19 Id: 7b04c0 No. 10641 [Reply]
I had a great week. There's some more login manager work, including better ui and login scripts for several new sites, a bunch of bug fixes, some layout improvement and cleanup, and some faster search and tag processing. The release should be as normal tomorrow.
Cool! I thought it was always Tuesdays, but it seems like it's Thursdays. Can't wait for version 333, it's going to be ebin. :DDD

(427.79 KB 1280x1269 WipeOut.jpg)

Source code documenting / project structure cataloging moemon 11/01/2018 (Thu) 00:41:30 Id: 883bd4 No. 10450 [Reply]
For some time now I have been meaning to start looking into the source code of Hydrus to see what it can do and what might be improved upon without bothering Hydrus Dev too much. Problem is, according to my knowledge, the source code is not documented(or commented) at all and the only one who knows what's really going on within the program is Hydrus dev himself, bless his soul. I was wondering if I missed something and there is something like documentation out there and if there isn't I wondered if there were people willing to help document the whole project, or at least try to figure out what's going on exactly. So far I have come up with the following topics if anyone were to write something like a documentation, feel free to add to them: -How to set up your development environment. -How is the program structured, how are objects/modules linked -What are the functions of the objects/modules -How are the objects/modules structured -How does the program as a whole function -How to interact with said objects/modules -How to add/write your own objects/modules to the hydrus client and server I don't know if I nor anyone else has the resources, time or spirit to do this but I wanted to at least ask. Having documentation would help people to inject personal customization into hydrus to make it fit their needs and maybe even spawn a small modding community. It would allow the program to flourish as a whole without forcing everything on a single point/person.
Maybe a nice place to start is for Hydrus dev to enable the public wiki on the hydrus github page such that people can freely read and contribute without having to visit strange unrelated sites and documents, though it might create unwanted work for hydrus dev as he'll need to check yet another official channel and make sure that there won't be pure bullshit written there.
>>10451 Currently requesting wiki from wikidot, but also having an eye for these three http://www.editthis.info/wiki/ https://miraheze.org/ http://www.referata.com/wiki/Main_Page

>>10397 With what I know for coloring, it use to be done all by hand, but then you haver tools that follows objects on screen for special effects, you could apply that to this, where its tracking objects, then adding color on top and it does it automatically. with enough ai in the background you could easily make have 'looks like this, its brown, looks like this its green' for grass, Look into deep fakes, as this is likely the closest we will get to doing that to artists, people who would never make more, super imposed on porn scenes, and its community made so you see real results that don't have big businesses backing it with unlimited funding. as far as tagging goes, I still stand by everything should be human approved. >>10403 definitely looking forward to it
>>10416 >>10414 That's frustrating–it usually relieves it significantly, if not fixes it completely. When I moved over to the new wx last year, some Linux environments got a lot stricter about when I could access ui objects. Mostly it is accessing variables of ui objects from the non ui thread. I cleaned up a whole ton, as many bad accesses as I could find, but there must still be a couple out there. Unfortunately, because I am in python, it is tricky to get good tracebacks or dumps for these errors, and my test environments here don't have the problem any more. I am sorry that I can't offer a good fix here. If you notice that doing one thing tends to cause the error (or, in many cases, sets a state that will cause a crash up to 30 mins later), please let me know, and I'll have an area to focus on to see where I am screwing up. A trick I ran for a while when fixing this the first time was to have people just open the client and leave it for a bit, doing nothing. Then trying again, doing one thing, and then trying again doing another, to logically figure out what dialogs or actions were causing a crash state. If you have the patience and enthusiasm, this would be helpful to me. Can you say which flavour and version of linux you are on, and which window manager?
>>10419 Yeah, for tagging, my idea for a first version here is to add a new suggestions column to manage tags that'll provide rows like '95% confident of "equine_penis"'. So it'll just be a neat tool to make tagging quicker. Because of high error rates with this stuff, I always want human eyes confirming any automatic decision. By the time we can have good systems that don't have any human in the loop, I expect we'll be more concerned with stopping the cloud of flesh-eating nanobots from consuming everything we love.

Release Tomorrow! hydrus_dev 10/30/2018 (Tue) 22:01:00 Id: ebc3f1 No. 10420 [Reply]
I had a good week. The login manager is turned on for advanced users, and I did some neat ui stuff, including custom thumbnail border and margin thicknesses and greatly improved layout for the manage tags dialog. The release should be as normal tomorrow.
2 posts and 1 image omitted.
>>10422 >>>/tech/983134
>>10420 Which websites the login manager will be covering?
>>10424 Anything with a simple login form. It is completely user-configurable and shareable, like the new download system. It can do multi-page scripts, where it will parse some info from the first page and offer that as a temporary variable to another (like a POST form key or CSRF token). Anything needing a captcha is probably too complicated. It determines login status with cookies atm. Cross-domain woo woo like exhentai is probably too complicated, but hydrus also supports importing cookies.txt from your browser so you can just copy across otherwise. Today's release will be pretty empty, but I'll roll out some scripts for the big sites in coming weeks.

(4.03 KB 480x360 3q7GYNfY_Dc.jpg)

Version 326 hydrus_dev 10/17/2018 (Wed) 21:46:27 Id: 2b5234 No. 10273 [Reply] [Last]

Message too long. Click here to view full text.

51 posts and 1 image omitted.
>>10341 Don't worry about it–that number is inflated because I am currently tracking groups with pairs. You get the number of combinations (as in combinations and permutations), which will reduce in size similarly faster than the number you process. There's a bunch of n(n-1) in there, essentially, when the true number is n. At some point I expect to move the system to true group tracking, at which point many of the inflated numbers and unoptimised actions here will work a hell of a lot better.
>>10296 You were right about it defaulting to python 3. What actually happened was, I checked the python version and it displayed the 2.7 version. So my system defaults to 2.7. However as soon as I start venv, it defaults to 3.5. I now installed the packages again using pip2, and tried running it with python2. Now it shows the splash screen and starts initializing stuff, but then fails with this; "ImportError: No module named ordered_dict" I randomly tried installing packages that sounded like they might be related to ordered dicts using pip2: ordereddict persistent-ordered-dict odict But none of them solved the error. Do you have any idea what I'm missing?
>>10377 The new version of requests can't unpickle a session object from the old version of requests. Try v327, it should be fixed (by clearing your old sessions, argh).

(577.00 B 128x128 Pyrus.png)

What kind of dedication and work does it take to create something like this? Anonymous 09/30/2018 (Sun) 04:03:03 Id: 612715 No. 10129 [Reply]
I bet you thought this was a thread made to suck up to Hydrus Dev. HAHA, WRONG! There's so many things about the Hydrus network that irritate the crap out of me, and it's clear that, as they're deliberate design considerations, they're not going to ever change. Therefore, I want to make my own. Problem is, I haven't really developed shit before, so I just want to know what I'm in for. Maybe I'm mistaken, but I'm under the impression that Hydrus Dev was only starting out when he began creating his as well, so maybe he could offer a few pointers, being already so far into his own project. Anyway, here's some notes I've been making on how I might do things differently. I imagine Hydrus Dev has already considered a lot of these ideas before, so being more world-weary, and having lost that old shine in his eyes, maybe he could tell me why they're fucking stupid and wouldn't work at all.
7 posts and 1 image omitted.
They actually did this https://gitgud.io/Dizmal/borehole
>>10197 Not me. Why would I come up with a COOL name like "Pyrus" then abandon it?
>>10244 Because you are a fag who can't actually code or write a proper specification.

(44.30 KB 1024x576 1481168675293.jpg)

The Struggle Is Real Anonymous 02/21/2017 (Tue) 04:34:56 Id: 9850fa No. 5187 [Reply] [Last]
Hey everybody, this is just a thread to talk about our everyday life with Hydrus. How are you coping with your file hoarding addiction now that you're being enabled? Are you remembering to take some time to tag what you download? You ARE making a monthly backup next week, right? What kinds of files are you saving, and what sorts of tags are you committing to the repo? Right now I'm sorting through the downloads from a few of my NSFW subscriptions pre-tagging. They are all about half stuff I want and half gay shit I don't want, so I put "impure" in their subscription tags and I just go through 100 at a time and purify them. It does tend to break about even with me keeping 50 files out of each 100. Pray for me anons. …AND YOU?
118 posts omitted.
I know I've been gone a while, but… Don't forget your backup, anon
(227.10 KB 384x356 ClipboardImage.png)

>the archived stuff isn't even tagged yet >nothing is
(219.56 KB 503x487 ClipboardImage.png)

>>7574 An update on my progress nearly a year later, with 2945 untagged files.

(7.84 KB 480x360 OqEBF2F-4z8.jpg)

Version 325 hydrus_dev 10/03/2018 (Wed) 21:47:37 Id: 923859 No. 10186 [Reply] [Last]
https://www.youtube.com/watch?v=OqEBF2F-4z8 windows zip: https://github.com/hydrusnetwork/hydrus/releases/download/v325/Hydrus.Network.325.-.Windows.-.Extract.only.zip exe: https://github.com/hydrusnetwork/hydrus/releases/download/v325/Hydrus.Network.325.-.Windows.-.Installer.exe os x app: https://github.com/hydrusnetwork/hydrus/releases/download/v325/Hydrus.Network.325.-.OS.X.-.App.dmg tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v325/Hydrus.Network.325.-.OS.X.-.Extract.only.tar.gz linux tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v325/Hydrus.Network.325.-.Linux.-.Executable.tar.gz source tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v325.tar.gz I had a difficult week, but I got some great work done. Save for some final help revisions, the downloader overhaul is complete. final downloader work So, I managed to finish 13 of my 15 final jobs in the downloader overhaul. All that remains is a help pass for subscriptions and a better intro to gallery and watcher downloading, which I will fold into normal work over the coming weeks. This has been a longer journey than I expected, but I feel great to be done. This final work is mostly unusual stuff that got put off.

Message too long. Click here to view full text.

34 posts and 1 image omitted.
>>10225 Right, I just looked into this, and ffmpeg was giving some funny framerates: mp4: 0.77 fps 20tbr webm: 2.5fps 50 tbr tbr is some kind of fps estimate ffmpeg is pulling from the codec or something. iirc, it stands for: T: Standards B: Engineers R: Are fucked in the head m8, big time I am not totally sure, but it looks like there is some variable frame rate going on when I look at them in MPC, so I guess it is even more complicated. In any case, I have rejiggered my parsing code to be less trustworthy of these values when they differ like this, and now the videos' frames are counted manually–I now get 7 and 13 respectively, with correct durations. So these should work better for you in tomorrow's release! As hopefully should the other various low-framerate low-frame-count vids that seem to render all in a rush and then hang on the last frame for ten secs. In v326, turn on help->advanced mode and then right-click these videos and hit reparse files and regen thumbs. They should fix up, at least a bit.

Message too long. Click here to view full text.

This week I started getting this error with pixiv subscriptions in version 325: Exception The subscription Pixiv encountered several errors when downloading files, so it abandoned its sync. Traceback (most recent call last): File "include/ClientImportSubscriptions.py", line 1125, in Sync File "include/ClientImportSubscriptions.py", line 479, in _WorkOnFiles Exception: The subscription Pixiv encountered several errors when downloading files, so it abandoned its sync. It downloads around 10 to 30 pics, then throws that exception. Is there any way to display the post IDs of failed downloads? I retried a few times, with one artist, and it didn't download any new pics on retry, then threw the exception again. So I assume it fails on the same posts again.
>>10275 Pixiv are in the midst of changing some of their layout again. Please try again in v326, which will automatically update you to a hopefully more stable solution.

Release Tomorrow! hydrus_dev 10/17/2018 (Wed) 00:06:01 Id: 9b1eab No. 10270 [Reply]
I had a great couple of weeks. The login manager is ready for advanced users to play with, and I've done a bunch of other work besides, including plenty of fixes and ui additions, some tag autocomplete speedup, layout improvements for small-screen users, and better en-masse downloader & subscription file management. Due to the two-week cycle, there is a lot more than usual to test. The release may be a bit late tomorrow.
>layout improvements for small-screen users Damn, just today I was thinking about this. thanks.

[ 123456789101112131415 ]
Forms
Delete
Report