>(including I think the Linux >0 distance similar files search crash, finally!) Was that at all related to any of my posts: >>11736 >>11758 >>11797 Because I was having an issue on linux crashing. In the process for searching for duplicates an SQL statement similar to SELECT phash_id, phash, radius, inner_id, outer_id FROM shape_perceptual_hashes NATURAL JOIN shape_vptree WHERE phash_id IN (1, 2, 3); would be executed and cause a segfault, not just for hydrus but anything that could execute an sqlite statement. Apparently Sqlite 3.27.1 (and possibly some earlier versions) had a bug where if the size of a list exceeded two items it just crashed. This appears to be fixed in sqlite 3.27.2. I imagine based on the code I looked at having a search distance greater than 0 would make it far less likely to happen if not impossible.

>>11804 There are quite a few places where the client fetches things with lists up to 256 items long, so if you found that just loading some files and tags was ok, I _think_ you weren't getting hit by that exact issue. But yeah, your posts are the bug I think I fixed. Only affected some Linux users, and I could not reproduce it. I worked one on one in the week with a user who had the crash and we ultimately figured it was a problem with SQLite hanging on to the phash while I did a Hamming Distance search on it (which requires some low level C++ struct calls and then some mickey-mouse CPU-level binary counting). Something to do with memory access violation is my best guess. Python usually can't crash by itself. Previously, I was doing hamming searches interleaved with the sqlite call iterating each row, but when I separated it into completed batches, so sqlite was 'done' with the rows, no crash happened. The 0 search distance could do a direct lookup without the hamming check, and that was running fine. It just seemed to be the combination of iterator on sql results and then low-level access on 'bytes' type response from that that caused the crash. Core dumps suggested the crash was in sqlite code, so I guess it was still hanging on to a bytes buffer or something that wasn't cleaned up. Could indeed still be a (different) bug in sqlite, or python's wrapper for it, but I dunno for real. Let me know how 342 works for you!

>>11801 Owarimonogatari >>11803 wholesome

>>11386 with 2 references to a "decentralized booru" https://rbt.asia/g/thread/68261237 https://rbt.asia/g/thread/64493084

>>11387 And maybe this https://www.nayuki.io/page/designing-better-file-organization-around-tags-not-hierarchies

>>11468 You can probably find archives in https://hollaforums.com/ https://powforums.com/ https://supforums.com/ https://veekyforums.com/ https://zigforums.com/

>>11718 the program tends to eat memory when you heavily use it, let me give you a picture of what I mean I save every thread on 4chan that catches my interest, at some point the reposts start to overtake the new images to a point that this is a viable option, thy im getting close to the edge of I need some delete functionality for telling me what the file was deleted for. that aside, right now, over the coarse of 2 months I have created around 6400 watchers, adding somewhere around 150-200k images, almost none of them are displayed, with around 100k displayed. the 100k displayed images takes about 1.5-2gb, I know this from the time I culled the watchers, my current session is 4.3gb and bloats out to 6.5gb on saves. I have absolutely no idea what takes ram, if they were all current watchers, ok, I would believe that, but from experience current watchers for all the threads adds up to around 500mb or less, so for some reason, the program is using an ungodly amount of ram for reasons that are unclear because just storing all of the the data in text form could not possibly add up to over 20mb much less what I estimate at being over 2gb with a doubling in save. I should also note that I come from a session that was hitting 14gb on startup and when a save happened would bloat to 18+gb depending on where hdev is focusing, memory reduction has many different ways to happen.

>>11724 >the program tends to eat memory when you heavily use it I think that goes for most programs. When I had 8GB of RAM my average overall usage was around 3GB, but now that I have 16GB it averages 6 or 7 even though I'm doing the same things.

>>11718 Bare client is about 115MB ram on Windows, so the absolute minimum is pretty low compared to what many users see. I am about 490MB on my IRL laptop that has about 17k items in its main session (mostly thread watchers). The python 3 change lead to some pseudo-memory-leaks, so that was a recent chunk I was able to recover, and I am making an important db-memory change in 341 today. I would appreciate continued feedback here. There isn't a huge amount I can do atm about clients that have hundreds of thousands of pending URLs in their session, but PTR update processing and some other db stuff should be much less traumatic now. There's still more to do, and I still plan to make a memory profiler this year to draw some pie charts or something and squeeze this further.

>>11704 Its possible that its manga and ugoria, but the problem I have is that it says there are 307 new images, but when I tell it to present new, only 21 show up. my concern is that it's filing things that may, in the future, be parseable under an I have it, and when it rechecks it thinks I have it and skips over it. >>11703 for this, sorry, im not getting them from exhentai, im finding new artists, usually in a gallery for an artist they will post where you can find their work directly, and I plug it in to download. effectively, I download a gallery of something in the gallery downloader, and a few weeks later I see an update to it, so I regrab it in the gallery, the best way to put it is these are impulse downloads, not something I want to set a subscription up for. I think this shows off what I would like to avoid fairly well, if there was a way to recheck without adding the same search again, that would be great.

>>11663 >>11636 I did some work on this today and it went well! I have a 'thumbnail experiment mode' in for v341 under help->debug->gui so you can try it out yourself. It loads the fullsize and resizes in memory. There is not a human-noticeable difference in speed. On my dev machine here, which is a few years old, I added some timers and got (once the file data was in memory) approx 500µs to load a resized thumb and 1.8ms to load a fullsize thumb and scale it, which I was impressed by. I have a Nvidia 1030 here to drive my 4k monitor, so perhaps that is accelerating this. I am willing to experiment more here, so I will mention it in my release post and see what you and other users find. After looking at the code, I think that in exchange for simplifying the file system by only having one set of thumbs, I could justify making that single thumb more intelligent in how it swap outs bad-size thumbs when needed (i.e. basically removing the master rather than the resized). So, I think we can get the best of both worlds here, saving space and keeping things fast. I have a job set out for it now, so I'll try to chip away at this in the coming weeks. Thank you for bringing this up.

>>11705 Last bit of info is I currently have about 100-110 gallery download ques (none running), that would send the pending tags through the roof but since Hydrus only displays 1 at at time then I dont think this would do much for thumbnails displayed. It would certainly push up high use of the CPU, disk writing and some ram but only when Hydrus is busy. Despite all the stuff I listed I'm only around 450 MB of Ram.

>>11676 Take care of yourself and get well soon, hydrus dev.

>>11676 Thanks for the update, get well soon!

>>11683 Is this when she dies? I've never seen this in game

I was able to script about 70% of it but ran into an issue. The site has one of those fucking click through screens. other than that it is practically identical to e621. Just with worse tagging.

>>11470 >The site has one of those fucking click through screens. You mean, the confirm age one? That may be solved by connecting to the "yes, I am 18" URL directly.

>>11521 it blocks that. you need A verified cookie.

>>11599 Thank you, it's working now. I almost was about to post that it still wasn't working but the client crashed when I was about to take a screenshot and it started to work after the restart.

>>11599 Oh, and to add to your question: I indeed have migrated from Windows long time ago. I just tried a fresh install and it defaults to that setting, so the problem was just outdated defaults. If I wasn't pantsu-on-head retarded, I would've checked if it was working on a fresh install first. Sorry for wasting time.

>>11603 I reinstalled 339 and havent noticed the memory spike as bad as before but it is locking up and stops responding pretty often so i still end up having to restart anyways for any downloading. the program still works for main archival and search though.

Read >>10203 then write up a proper proposal.

Hi, having a small problem with the Manage Import Folders option.
TypeError
CallBlockingToWX() missing 1 required positional argument: 'func'
Traceback (most recent call last):
File "include\HydrusThreading.py", line 342, in run
callable( *args, **kwargs )
File "include\ClientGUI.py", line 2608, in THREAD_do_it
controller.CallBlockingToWX( wx_do_it )
TypeError: CallBlockingToWX() missing 1 required positional argument: 'func'
Not a problem with export folders.

>>11532 Thank you for this report. I made a couple of stupid mistakes last week–this was one. I am sorry for the problem. It will be fixed in tomorrow's release.

This thread is getting laggy, so: N E W E W T H R E A D H R E A D >>11542 >>11542 >>11542 >>11542

>>11533 Thanks, I will.

>>11536 >Do you ever do right-click->repare and regen thumbs on your movies? Yes, in fact I did this on all my videos because I'm the guy who had Hydrus randomly generate broken green thumbnails for videos (which seems to be fixed now by the way, if not by you then perhaps because I got a new computer and switched from Windows 7 to 10). An auto-fix would be nice because it has happened to a lot more than just that one video. Is webm and mkv very close in how the files are structured? Because most of these were downloaded from gelbooru which lists them as webm. But I guess it doesn't really matter if the hash stays the same.

>>11540 Yeah, iirc, a webm is just an mkv with VP8/9 video and vorbis audio, I think. All webms are mkvs–they are a subset. I think the idea was to guarantee a limited mkv format that would work on all devices for HTML5 streaming etc… I could also have some of my ffmpeg parsing wrong here, or had it wrong when you did this reparse. I used to use an external library to figure this out, and now I compare ffmpeg output specifically. If you run the reparse again, maybe it'll flip back to webm! Anyway, should be fixed for tomorrow. Let me know if you still have any trouble.

>>11507 this is a copypasta that gets autobanned over on cuckchan. people use it to stop chinese threads/ chinese shilling in their tracks because what's listed in the copypasta is all censored by the chinese government.

晚上好。我正在学汉语。

Don't you go to jail for anime titty in China

It's not working for me, even on a clean install. I'm using the zip version for windows, but it just starts and ends the process instantly, there's no error log or anything. I'm guessing it's some python 3 fuckery since my last install was v334, which is weird since my main python is python 3.6. I guess I'll try 336 then 335 to see if either of those work.

>>11490 Same thing happens with 336 and 335, so I'm guessing it's a problem with python 3. When I open python via the terminal it goes straight to 3.6 though, so it can't be a problem with the path. No crash reports for either of those as well, it just opens and instantly closes without even showing the startup window. v334 is the last version that works.

>>11492 Are you on Windows 7? You may need some OS updates and possibly VCRedist 2015, as here: >>11260 >>11495 https://www.microsoft.com/en-us/download/details.aspx?id=48145

Eric S Raymond's Commands 0. Be fearful of your god, that worship without self-sacrifice is a sin 1. You shall complete basic programming training, he who does not is a heretic 2. As per daily rituals, you must code everyday, to seek the road to ascension 3. The hackerspace shall be your temple, and the members shall be your brethren 4. At the end of your journey, you will present your Magnum Opus as a final offering a. The Magnum Opus will act as a reflection of what you have not learned b. The Magnum Opus will be a project that you have full responsibility over c. The Magnum Opus will be on public display, and all critics will be your judge Taken from http://esr.ibiblio.org/?p=4817 and a message to Hydrus Dev

>>11452 Hey, I strongly recommend you let a new client sync at the default rate. It is good for the database to sync in pieces, so it can run maintenance as it goes. There are many years of tags in the PTR now, hundreds of millions. I recommend new users not try to get them all at once. Give it time, and you'll see tags start to appear after a few processing cycles.

>>11435 Any ideas of designing OR search? https://danbooru.donmai.us/wiki_pages/43049 https://gelbooru.com/index.php?page=help&topic=cheatsheet https://yande.re/wiki/show?title=cheat_sheet_extended https://e621.net/help/show/cheatsheet https://hypnohub.net/help/cheatsheet (MyImouto) https://derpibooru.org/search/syntax (Best syntax)

>>11390 when it comes to anti virus, they have 2 methods of working 1) knowing the behavior of the program, if its doing something iffy, they may catch it 2) knowing the exact exe other then this, anti viruses are kind of worthless. you are honestly better off working in a vm, and becaues these things dont typically catch new things, only old ones, by the time antiviruses catch and patch for things, your os is usually already patched, or will be patched shortly, and for the few things that would still hit you that you don't ask for, it would be better to have a hardware based firewall.

https://i.4cdn.org/gif/1548042014339.webm float division by zero… (Copy note to see full error) Traceback (most recent call last): File "include\ClientImportFileSeeds.py", line 1211, in WorkOnURL self.DownloadAndImportRawFile( file_url, file_import_options, network_job_factory, network_job_presentation_context_factory ) File "include\ClientImportFileSeeds.py", line 544, in DownloadAndImportRawFile self.Import( temp_path, file_import_options ) File "include\ClientImportFileSeeds.py", line 763, in Import ( status, hash, note ) = HG.client_controller.client_files_manager.ImportFile( file_import_job ) File "include\ClientCaches.py", line 1088, in ImportFile file_import_job.GenerateInfo() File "include\ClientImportFileSeeds.py", line 289, in GenerateInfo self._thumbnail = HydrusFileHandling.GenerateThumbnail( self._temp_path, mime, percentage_in = percentage_in ) File "include\HydrusFileHandling.py", line 113, in GenerateThumbnail renderer = HydrusVideoHandling.VideoRendererFFMPEG( path, mime, duration, num_frames, cropped_dimensions )

>>11422 Thank you for this report. Someone actually sent me a similar unusual-frame-count music vid the other day, so this is fixed for tomorrow's release already! Let me know if you still have any trouble in v337.