/hydrus/ - Hydrus Network

Archive for bug reports, feature requests, and other discussion for the hydrus network.

Index Catalog Archive Bottom Refresh
Name
Options
Subject
Message

Max message length: 12000

files

Max file size: 32.00 MB

Total max file size: 50.00 MB

Max files: 5

Supported file types: GIF, JPG, PNG, WebM, OGG, and more

E-mail
Password

(used to delete files and posts)
Misc

Remember to follow the Rules

The backup domains are located at 8chan.se and 8chan.cc. TOR access can be found here, or you can access the TOR portal from the clearnet at Redchannit 3.0.

Uncommon Time Winter Stream

Interboard /christmas/ Event has Begun!
Come celebrate Christmas with us here

8chan.moe is a hobby project with no affiliation whatsoever to the administration of any other "8chan" site, past or present.
(13.11 KB 480x360 U35sSPJI_Bs.jpg)

Version 380 Anonymous 01/16/2020 (Thu) 03:49:05 Id: dda13d No. 13536
https://www.youtube.com/watch?v=U35sSPJI_Bs windows zip: https://github.com/hydrusnetwork/hydrus/releases/download/v380/Hydrus.Network.380.-.Windows.-.Extract.only.zip exe: https://github.com/hydrusnetwork/hydrus/releases/download/v380/Hydrus.Network.380.-.Windows.-.Installer.exe macOS app: https://github.com/hydrusnetwork/hydrus/releases/download/v380/Hydrus.Network.380.-.macOS.-.App.dmg linux tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v380/Hydrus.Network.380.-.Linux.-.Executable.tar.gz source tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v380.tar.gz I had a couple of difficult weeks, with illness and other IRL problems getting in the way, but I got some hopefully pretty neat work done. A new fast video and audio player is available for advanced users to test, and there are a bunch of fixes and ui improvements as well. A user just notified me that the Duplicates page has crazy layout! I apologise. Nothing is broken, it is just sizing wrong, and part of a longer fight I am having to convert my old wx layout code to Qt. I know exactly what happened here, and I will have it fixed for 381. If you discover more UI like this (the system predicate panels have a bit of it), please let me know. mpv This is just for advanced users this week. It is a basic prototype that is not ready for real use. I will improve a bit before turning it on for everyone, hopefully next week. MPV is a good free video player. One of the core benefits of moving hydrus to Qt was the potential of being able to neatly embed it into the media viewer. I am happy to report that these past two weeks have been successful, and it looks like hydrus is finally getting hardware-accelerated video playback and full native audio support! If you are using the windows build in advanced mode, please check out the big ugly filetypes list under options->media. Under all the video and audio filetypes, you can now set 'show using mpv' as a show action. Please forgive how large this list has grown–before I turn mpv on for everyone, I am going to rework this list so you have the option to just say 'for all video: xxxxxx', rather than always having every single filetype. Once you have it set, it should load video almost exactly the same as my native renderer, with the scanbar below for navigation. Audio files will show as a (for-now fixed-size) black box or album art. Be warned: there are no volume/mute controls yet! I have set it at 70% volume for now. MPV has an optional and configurable on-screen display, but for technical reasons I had to stick with my custom controls. I know what features I need to add, so the feedback I am most interested in this week is in where mpv breaks. If it doesn't work for you, or if it fails to load a file, or it has bad performance, or it sizes wrong, or you get a crash after closing the media viewer, anything like that, please let me know. Performance in my tests has been excellent–rendering 4k 60fps no problem as long as you have any sort of branded GPU. I have had some audio jitter on some unusually encoded videos and some crashes (which I think I have fixed). One annoying thing is mpv requires a hefty 57MB dll, making the already bloated hydrus download even bigger. One user suggested perhaps putting out two releases–a 'core', and an 'update', which wouldn't have the big stuff. I am still thinking about it–while I like the overall idea, the hydrus build is held together with duct tape, and having a small non-functional release will increase complexity and inevitably cause ugly syncing difficulties whenever the big stuff does need to be updated. Users running from source will need libmpv and python-mpv to enable this. I haven't pinned down the most convenient and functional way to do this for Linux and macOS yet, so we'll have to figure this out over the coming weeks and get it into the help. Any knowledgeable feedback here would be great. For the near future of mpv, I would like to: figure out the technical side of including mpv in the Linux and macOS builds; write volume/mute UI and shortcuts; add global volume/mute management; test mpv with gifs and apngs; explore custom shaders and configuration files; and generally polish the whole thing. quality of life The tag right-click menu has had a significant overhaul. More commonly used actions are moved up, less common down. Tags with siblings now provide all siblings under the 'copy' menu! System search predicates have better copy and search-action support (like 'exclude' system:archive, which will actually add 'system:inbox'). Right-clicking on the thumbnail grid's 'whitespace' no longer does a 'deselect all' action, so you can now easily right-click in an empty area and go remove->selected! Across the program, all filetypes are now referred to with human language. Instead of 'image/jpg', it is now just 'jpeg'. Instead of 'application/x-7z-compressed', it is now just '7z'. Please let me know where it looks bad, and if you would like to edit these labels. The 'new page chooser' dialog now dismisses itself if you click off it! So, if you accidentally middle-click in some page tab whitespace and it annoyingly pops up, just left-click, and it will go away. Also, this dialog now catches enter/return key, and will hit the 'first' button (if you imagine the nine possible button locations as a number pad, starting at top-left). Hitting enter twice will typically open a new 'my files' search page. A user surprised me recently by asking where to find the 'pause video' shortcut–I hadn't realised there wasn't one! Now this is fixed: 'pause_media' and 'pause_play_media' are added to the 'media_viewer' shortcut set, and 'pause_play_slideshow' is added to 'media_viewer_browser' set. Previously, the slideshow pause/play was hardcoded to space bar–this is now gone, so if you used this, you'll want to set it up yourself. the rest I have added simple support for RealVideo, RealAudio, and TrueAudio files. These are some formats, so if you come across some variable bitrate rmvb file or something that doesn't work, please send it in and I will have a look. MPV couldn't handle a couple of my test .rm files' audio tracks, but the .ttas seemed great. Ever since the CloudFlare-8chan break, I have been telling myself to figure out some new downloader objects for the bunkers everyone scattered to. This job kept being put off, long enough that 8kun came back(!), but I managed to fit in some time in this morning and got 8kun and vch.moe support added for today's release. 8kun was obviously easy to convert from the old 8chan parser, and vch's engine provides an API very similar to 4chan that I was able to duplicate and modify for. I had a proper look at julay, smuglo.li, and endchan, but these use their own API formats (or no API at all), so I will have to write new parsers, which will take longer. I really would like to get them done for 381–please remind me if it looks like this job has slipped away again.
[Expand Post] full list - basic mpv support is added. it comes with the windows build this week, and is a prototype meant for initial testing. the library is optional. users who run from source will want 'python-mpv' added via pip and libmpv available on their PATH, more details in running_from_source help - took an qt-mpv example kindly provided by a user, updated it to work with the hydrus environment, and integrated it into the client as a new choosable view type under audio/video filetypes under options->media for advanced users - reworked how the 'start paused' and 'start with embed button' media viewer options work under options->media. these are now separate checkboxes, not combined with the underlying 'show action'. existing embed/paused show actions should be converted automatically to the correct new values - unfortunately, due to some python/qt/libmpv wrapper mouse interaction issues, mpv's 'on screen controller' overlay is not available - for now, left click pause/plays the mpv window, just like the native mpv window. - preview/next frame shortcuts should work for the mpv window when playing video - no volume/mute controls yet, these will come in the coming weeks, including global mute settings - updated media show and sizing code to account for mpv widgets - reworked my animation scanbar to talk to mpv, and for my mpv window to talk back to it - improved the animation scanbar to be more flexible when frame position and num_frames are not available, both in displaying info and calculating scanbar seek clicks - mpv api version added to help->about - . - new downloader objects: - thanks to a user, updated the 'pixiv artist page' url class to a new object that covers more situations. the defunct 'pixiv artist gallery page' url class is removed - added 8kun and vch.moe download support. I got started on julay, smug, and endchan, but they were a little more tricky and I couldn't finish them in time–fingers crossed, next week - . - menu quality of life: - a right-click on thumbnail whitespace will now not send a 'deselect all' event! feel free to right-click in empty space to do an easy remove->selected - remorked the tag menu layout to move less frequently used actions down: - - moved the discard/require/permit/exclude search predicate actions down - - moved 'open in a new page' below select and copy - - moved copy above select - and some misc menu layout improvement on this menu - fixed some labelling with the discard/require/permit/exclude verbs on negated tags - right-clicking on system search predicates now shows the 'copy' menu correctly - system predicates that offer easy inverse versions (like inbox/archive) should now offer the 'exclude' verb - when right-clicking on a single tag that has siblings, its siblings and those siblings' subtags will now be listed in the copy menu! - copying 'all' tags from a list menu, with or without counts, will now always copy them in the list order - across the program, all menu 'labels' (menu text items that do not have a submenu and have no associated action, like 'imported 3 years 7 months ago') will now copy their text to the clipboard. let's see how it goes - . - other ui quality of life: - across the program's UI, filetypes are now referred to with simpler terms rather than technical mimetypes. instead of 'image/jpg', it is now typically just 'jpeg' - the 'remove selected' buttons on the gallery and watcher pages are now smaller trash icon buttons - the new page chooser will now auto-dismiss if it loses focus–so if you accidentally launch it with a middle-/double-click somewhere, just click again and it'll go away - hitting enter or return on the new page chooser now picks the 'first' button, scanning from the top-left. hitting enter twice now typically opens a new 'my files' search page - added pause_media and pause_play_media shortcuts to the media_viewer shortcut set. new clients will start with space keypress performing pause_play_media - added pause_play_slideshow shortcut to the media_viewer_browser shortcut set. this shortcut is no longer hardcoded by space keypress - the six default shortcut sets now have a small description text on their edit panels - the options->media edit panels now enable/disable widgets better based on current media/preview action - added a checkbox to _options->gui pages_ to set whether middle-clicking a tag in the media viewer or a child tag manager to open a tag search page will switch to the main gui. default is false - mr bones now reports total files, total filesize, and average filesize - mr bones now loads your fate asynchronously - . - the rest: - added tentative and simple realvideo (.rm) and realaudio (.ra) support–seems to work ok, but some weirder variable bit rate formats may not, and I have collapsed the various different extensions just down to .rm or .ra - added trueaudio (.tta) audio support - fixed a bug from the recent search optimisations where a bare inbox search would not cross-reference with the file domain (so some trash could show up in a simple inbox/'my files' query) - fixed an issue with searching for known urls by url class where the class was for a third-or-higher-level domain and was not set to match subdomains (this hit 4chan file urls for a few users) - fixed the issue with 'open externally' button panel not clearing their backgrounds properly - fixed some of the new unusual stretchy layouts in the options dialog - removed overhead from subscriptions' 'separate' operation, which should stop super CPU hang when trying to split a subscription with hundreds of thousands of urls - fixed an issue where the advanced file delete dialog would not show the simple 'permanent delete' option when launched from the media viewer's right-click menu - fixed the select/remove actions for local/remote - fixed 'set_media_focus' from manage tags to correctly activate the underlying media viewer as well as set focus - stopped the 'file lookup script' status control from resizing so wide when it fetches a url - fixed a rare mouse wheel event handling bug in the media viewer - reduced db overhead of the 'loading x/y' results generation routine. this _may_ help some users who had very slow media result loading - cleaned up how the server reports a bootup-action error such as 'cannot shut down server since it is not running'–this is now a simple statement to console, not a full error with trace - improved client shutdown when a system session shutdown call arrives at the same time as a user shutdown request–the core shutdown routine should now only occur once - fixed an issue with thumbnail presentation on collections that have their contents deleted during the thumbnail generation call - misc wx->Qt layout conversion improvements - updated the github readme to reflect some new links and so on - misc code cleanup next week I pushed it a bit hard this week, so I am going to try to take it easy. Hopefully I will be able to get some volume/mute controls going for mpv and then launch it for all users, including on the Linux and macOS builds. Assuming there are no catastrophic problems this week, I will be overwriting user settings to move all 'show as normal' video and audio options to 'show with mpv'. Then these new imageboard parsers and some more Qt cleanup work.
Anonymous 01/16/2020 (Thu) 09:56:07 Id: 098d24 No. 13538
Video playback works fine here, with a few small issues. Geforce Experience thinks the video is a game so it gives me a pop-up about "press alt+z to open in-game overlay" also shows a framerate counter on the video. My normal video player is SMPlayer which also uses mpv and doesn't cause this to happen. Also I noticed the mouse cursor disappears quite fast when you hover over the video, which is fine, but then it doesn't appear when you move the mouse again until it leaves the frame of the video. It's a bit disorienting when you lose your mouse cursor like that, especially if the video is large or full screen. When you zoom out (making video smaller) it doesn't seem to center to the middle of the screen. Loving how fast mpv is, now it's actually viable to scale every video to fit the screen unlike with the native player which was way too slow. 4k60fps scaled to 1440p runs flawless (I have an i7 9700k & RTX 2070 though). Great work, and I'm looking forward to the future improvements you're planning. Now I might actually start using Hydrus for more video than short webm and gifs.
Anonymous 01/16/2020 (Thu) 10:11:53 Id: 098d24 No. 13539
Found another issue… In the thumbnail view, clicking on a normal image, then on a video causes a second long delay, presumably because the mpv container takes that time to load for the preview panel. Thing is, I have the preview panel disabled from the options menu so this shouldn't happen. It's more noticeable if I right-click the video because the menu takes a second to pop up, sometimes stopping at half-transparent.
Anonymous Board Owner 01/18/2020 (Sat) 19:16:57 Id: dda13d No. 13540
Hey lads, if you are on Windows and find MPV doesn't load for you (this seems to affect Win7 users only, but it may also hit Win10 users who launch the program in an unusual way), please try out this test build that I think fixes it: https://mega.nz/#!gnARTKgb!HOV0kJMgX0CrvjBnfNcomfMXiwwbrcBwoHO2ZbhbVNk Let me know how it goes!
Anonymous Board Owner 01/18/2020 (Sat) 21:44:12 Id: dda13d No. 13541
>>13539 >>13538 Thank you for the feedback. Most of the jank like the mouse disappearing and weird single-frame flashes when you switch from mpv-to-not or vice versa I believe I can tighten up this week. I have heard of one crash so far, which I am really pleased by. I agree with the speed, I am really happy and now watching 1080p and 4k vids at fullscreen in the client on my personal machine. When I turn MPV on for all users, I'll reset the default zoom options to scale up to current canvas size. Since we will be doing more edge-touching zooms, I expect I will do something with the seek bar, maybe having it render inside the video frame and pop into view when the mouse in near it, which will give us that neater view when seeing 1080p/4k vids on 16:9 screens. I'll check the zoom coordinate stuff. It not centering may be the same as any other media in the client, but perhaps that can be improved anyway to re-center once the media is below the canvas zoom size. The GeForce thing sounds frustrating. You might be able to disable it for now on your end by playing with specific settings for 'client.exe' in the NVIDIA Settings panel. Everything on my end for mpv is basically 'default' profile, so I'll see if there is any way I can undo that. Atm the only real customisation I have set is 'try to use high quality gpu resources if they are available', so I guess the OpenGL window mpv uses is triggering something in GeForce. The delay shit you got with the preview panel should go away when I tighten up how this works. The disabled preview panel was supposed to not show things in the background, maybe the Qt shift changed the logic there, I will make sure it does nothing as intended.
Anonymous 01/19/2020 (Sun) 00:23:29 Id: e66d00 No. 13542
(6.10 MB 1920x1080 NEVER GIVE UP YOUR WAAAAAAAAAAAAY-tYzMYcUty6s.mp4)

>>13536 after your work on mpv settles down, will you accept PRs? I have some stuff I'm tempted to add. I'm interested in adding smart-crops for thumbnails (which are really nice but CPU intensive) for example. https://github.com/smartcrop/smartcrop.py I have a lot of ideas, can't promise I'll make PRs since I'm busy but I'm interested. also, I appreciate what you do
Anonymous 01/20/2020 (Mon) 17:40:14 Id: 098d24 No. 13544
>>13541 >Since we will be doing more edge-touching zooms, I expect I will do something with the seek bar, maybe having it render inside the video frame and pop into view when the mouse in near it, which will give us that neater view when seeing 1080p/4k vids on 16:9 screens. That sounds great, it has been a small annoyance as well that a video matching your screen size won't fill to the edges because of the seek bar. >The GeForce thing Unfortunately I haven't found a way to block the overlay in specific applications, not in the nvidia control panel nor the geforce experience application. I use other features in the overlay quite often so I'd prefer to not turn it off entirely. If you can find a fix, I'd appreciate it.
Anonymous Board Owner 01/22/2020 (Wed) 06:26:44 Id: 1a43c7 No. 13545
I had a good week with a couple of challenges. The MPV work was difficult, so I could not get nearly as much done as I wanted. I have a basic smooth version working for all Windows users however, and some very simple volume controls. Also a bunch of bug fixes and a significant overhaul of how filetypes are selected and managed in the client. The release should be as normal tomorrow. I am way behind on my messages–I apologise for the delay.
Anonymous 01/25/2020 (Sat) 19:28:57 Id: b79885 No. 13556
Thank you, I am glad you like hydrus! Unfortunately, I am a complete sperg and have great difficulty working with others, so for hydrus I just work on my own and I am afraid I cannot take any PRs. Furthermore, the github is just a mirror of my home environment, so I am not logistically set up to take code that way either. I apologise, I just don't have the kind of brain that can work in any sort of team without exploding in a drama-bomb, so I now strictly abstain. I mostly point othe programmers who would like to help at the Client API now. Great projects like Hydrus Companion are written completely by others: https://hydrusnetwork.github.io/hydrus/help/client_api.html That smartcrop looks neat though! I will check it out a bit more. If you are particularly keen on getting it into hydrus, could you research it and summarise the best way to pull the coodinates of the interesting region of the image, or how to get a pure pixel-perfect crop in memory? I am not super interested in getting a jpeg on disk if that is possible–I'll likely want to do cropping/resizing/quality stuff in my existing pipeline. I can do a PIL image, but normally I work with numpy-compatible OpenCV images internally. Just posting five lines of code would save me a bunch of time. Also as a side note, my code is all WTFPL, so please feel free to do anything you like with it.
Anonymous Board Owner 01/25/2020 (Sat) 19:33:16 Id: b79885 No. 13557
>>13544 Unfortunately, I could not find an obvious solution to your problem. In the coming weeks, I will be adding more configuration to the mpv window. One part of that will be the 'vo' (I think 'video output') option. Atm I have it set to gpu-hq, but it has other options for software and opengl rendering iirc. My assumption is my attempt to say 'hardware accelerate this as much as possible' has triggered your GPU into thinking it is vidya. Once that option is available, I think your best bet here is trying some different presets and seeing what changes. I would be very interested in what you see.
Anonymous 01/25/2020 (Sat) 23:21:03 Id: e66d00 No. 13568
>>13556 Smartcrop takes a second to compute, and the python port is even slower than the javascript one at 3s or something. You would need some sort of gpu accelerated code to get fast enough for only the fly thumbnails. >Unfortunately, I am a complete sperg and have great difficulty working with others Yeah, I kinda figured. I'm the same way, so. I have been dabbling with arangodb (a document + graph database with full text support) to see what it would be like to have hydrus in a different database than a relational one. By the way, you are a beast with sql. I checked out some of hydrus dbs in an sql browser and they make my head spin (just a little). Anyways, I was curious what some of the bottlenecks are when trying to scale up so I could figure a way to improve them. Here are some thoughts. 1) Counting tags is extremely slow. Precomputing these is obviously not feasible since even just the combination of say, (5000 choose 2) would be in the millions of records. It's an NP-Complete problem. There's a way around this though and that's to only precompute the supernodes, tags which take up the majority of relationships. So, maybe precomputing the counts of tags for the top 100 tags, as well as their intersections/unions. That would only net you in the hundreds of thousands of records, might be a possibility. Precomputing all tags with no search parameters would probably a good idea regardless. 2) Sending every file with its tags over the wire to then be aggregated on the client is slow. That's probably why it takes so long to load up 10,000 files on hydrus. The solution would be to send just the ids, the bare minimum extra data (like the computed title/series/chapter at the top of the thumbnail) and then compute the counts. When selecting a file you pull the tags for that file and then cache them into the client. You can also start to pull in the tags for only what is in the client viewport. 3) Grouping files by series/chapter/etc can be precomputed and saved, or computed by sqlite. Right now it needs to pull in 10000 files just to group into, say, 100 chapters. That's pretty expensive. It also makes limiting the amount of files you show fucky. 4) full text searching is seriously fast. like really really fast. I haven't checked if you're using it but reverse indexes are the shit. Doing this kinda stuff sucks donkey balls obviously. You need to maintain data integrity constraints, and add more stuff to schedule in the background, you also can tank write speeds if you do don't pay attention, etc etc. I'll keep posting if I think of anything more, don't feel the need to pay it any attention to it and especially not with a sense of urgency.
Index Catalog Archive Top Reply
1
Forms
Delete
Report
Quick Reply