I setup hydrus-dd and api-thingy. How can it help me tag my files, do I just batch the client_files into it and it just werks? lel

Is there an existing/planned support for symlinks/hardlinks instead of "copy," particularly on winblows? Primarily to save space for large synchronized folders.

Why Hydrus doesn't support *.jp2 (jpg200)? It's so much better compared to jpg in terms of compression.

god damn the public repository is a clusterfuck is there a way to hide certain tags but not actually block the images with them? is there a way to look at all the tags in the public repository? i was hoping i'd just drop it in and autotag all my shit but i guess not

Was the in-client backup option removed?

can i import/export the list of previously deleted files?

>>14662 Hmm, I am not sure. It sounds like you have it set up correct, so perhaps the way nitter is delivering files is unusual. Is there any chance the files are coming in with a different 'sort' than usual (i.e. not 'newest first'), or it is just a normal username search? If you go into the subscription and drill down to the precise query, going even further into its edit window, there should be the 'file import status' button which will show you the actual URLs it has been working on, and also the 'gallery log'. These will have the full data, and may shed light on whether it is getting things in the wrong order, or maybe how it is making decisions on fetching new gallery pages. Are the URLs in ascending id (10000, 10001, 10002) order, or are older ids appearing later, after that initial sync? Is the gallery search doing more than one page per subsequent check (it should have info about how it worked in the 'notes' column of the gallery log)? Normally the gallery will be like 'found 5 new urls, not fetching any more pages'. Normally 100 initial files, and 100 per-check files works well for most sites, nitter included. Does this artist have a lot of retweets of other artists? Do their tweets tend to be single images, or multi-image galleries?

>>14663 I am not sure how much it is in total, but it will be a decent multiple of the total end database size, which I think is now about 30-40GB. It makes regular checkpoints to stay safe and not lose data, which involves saving to disk. Catching up on the PTR is essentially doing about eight years of processing work at once. I generally recommend to leave it work in idle time over a few weeks, but if it is set to do lots of work in one go, it'll add up. If you are completely synced up now, it should settle down to a little bit every day or so. Depending on how that data is exactly counted in your Windows version, that number may be slightly misleading. I know some versions count 'reading output from an executable' as disk I/O, so when hydrus reads file metadata and generates thumnails from ffmpeg during file import, it can add a casual 400MB when it is really just reading from memory. So if you were doing file imports, or using something like my old video renderer, it may be slightly off. I don't know if those read bytes are legit or include pulling from the disk cache in your ram. In any case, the PTR involves periods of heavy CPU and disk activity. It is something like 880 million mappings now, and a mechanical HDD can't keep up any more.

>>14668 I believe there are several ways to use it. I think the simple method for already-imported files is: https://gitgud.io/koto/hydrus-dd#lookup-script under network->downloader components->SEMI-LEGACY: file lookup scripts, and then turn on 'file lookup scripts' under options->tag suggestions. Make sure the dd server is running. Then, your manage tags dialog for single files will have a new 'file lookup' column with the 'dd' entry, which lets you query the dd server from hydrus with the file and get tag suggestions back. With the Client API connection, things are more complicated. If you have an external script that can generate your imported files' hash names, I think you can make that script call dd to tag through the client api, like so: https://gitgud.io/koto/hydrus-dd#for-api-intergration I don't have personal experience with it. Koto is in the discord, if you are comfortable with that, and some other users who do have experience with it.

>>14671 I've played with it a bit, mostly Linux experiments, and some stuff to help IPFS access the same files. I am not completely opposed to the idea, but I will first have to extend my file storage system to deal with more error states. At the moment, it assumes that all files are there and actually there. My next plan for file storage is to allow multiple potential storage locations for a single file and to have graceful 'this file storage isn't available atm' mode, which will include multiple 'missing/weird location' error handling. I understand you can hardlink folder locations now and the file storage system is generally ok with it. Some lads have to do that to get some NAS systems working right, I think. I'll caveat all this with my thoughts on preserving original folder structures, though: https://hydrusnetwork.github.io/hydrus/help/faq.html#external_files If a media collection is large enough that space is a worry, it may not be worth trying to manage it anywhere with a friendly folder structure. I recommend 2GB movies keep their nice filenames and are kept out of hydrus, but all the tens of thousands of webms are just too difficult to keep track of. If you want a small collection of favourite files somewhere easily accessible, then an 'export folder' in hydrus may suit you well.

>>14673 Is that JPEG2000? Do you have any example files I can look at and try to get working my end? >>14676 I don't think so. If you are ok with a bit of SQL, you can pull it right out of the database. Would something like a list of hex sha256 hashes work? Do you want to migrate this list from one client to another?

>>14682 >Do you want to migrate this list from one client to another? pretty much

>>14651 That would make it easier for sure! It would be great to have all of those features, but beggars can't be choosers!

Is there a way to change the default tab names? For example if I create a new gallery tab I would like for the tab name to simply be G. Files tab would be F. Etc. I often have enough tabs open that I need to rename them to more condensed names. Otherwise they take up so much space that the navigation arrows appear and the Qt glitch that causes the current tab to be on the far right makes it hell to navigate through the tabs. I am aware of the page of pages workaround but dislike that way of doing things. Alternatively is there a way to enter the next/previous tab using keyboard shortcuts? I looked in the shortcut command options but couldn't find a command that would do that.

Would it be possible to integrate YouTube-DL into Hydrus?

I have tried various tags and sources, but the gallery downloader does not seem to produce anything. Have I overlooked something obvious? Thanks in advance.

I wanted to be able to select multiple pics and use a shortcut to set them as alternates, but I could only do it through the custom shortcuts which don't work on the main view. Did I do something wrong or is there no way to do that currently?

>>14695 Nevermind, I reinstalled it and it works now, but I have no idea what I did differently though.

>>14671 Bumping because it got mixed with the wrong replies: >Is there an existing/planned support for symlinks/hardlinks instead of "copy," particularly on winblows? >Primarily to save space for large synchronized folders.

Any chance of multiple local file domains in the same installation happening within the next few years or it still a pipe dream?

I've been using hydrus for a while now, and it's been wonderful for my image collection. Right now, I'm trying to do something similar for the zip archives I have (10k+ archives mostly from sadpanda). Is there a tag repository for things like this? I've tried the PTR, but it doesn't seem like the tags for archives is well developed. Another problem I have is the thumbnails. Without having to do intensive stuff like change the code to read into the zip file to get the thumbnail, is it possible to change the thumbnail or find the thumbnail in the filesystem and replace it with the desired thumbnail?

>>14709 >is it possible to change the thumbnail or find the thumbnail in the filesystem and replace it with the desired thumbnail? Yes, but you'll have to do it yourself. If the file is under db/t[number]/[fille].something then the thumbnail is under db/f[number]/[file].thumbnail

>>14712 I don't think this is true for archive files, from what I've seen archive files always uses a static zip image for all files. I've tried testing this by importing a zip file. It does generate a regular db/f[number] file, but it doesn't generate a db/t[number].thumbnail file like it does with images. Placing a dummy thumbnail image in db/t[number].thumbnail (I just took an existing thumbnail image and renamed it to the zips hash) to "trick" it, didn't work, it still uses the standard zip image. So the client must see its a zip file, and only use the zip image. But, if you try retrieving the thumbnail through the api (http://127.0.0.1:45869/get_files/thumbnail?hash=[number]) it does return that fake thumbnail. So I suppose, the api must have a different logic for retrieving thumbnails. Also, don't do what I just did with "tricking" hydrus by creating a thumbnail for the zip file, and then successfully retrieving it via the api. Once you retrieve it via the api, hydrus stops recognizing that zip file and no longer be searchable. I'm mostly trying to see if I can create a gallery within hydrus, where the you can preview the archive, by having it show the cover.

Tried posting this once but I don't think it worked becaues captcha I can create a new database on a Samba share but I can't move an existing database to the Samba share through any method. I'm on macOS, so I've tried both the built in database migration tool and manually copying ~/Library/Hydrus. Nothing worked. If I manually copy ~/Library/Hydrus to a USB drive, I can start Hydrus from that database successfully. The error and client.log just say "serious error" and dump out a call stack trace. Here is the trace:
2020/09/11 12:28:11: Traceback (most recent call last):
File "Hydrus/hydrus/client/ClientController.py", line 1798, in THREADBootEverything
File "Hydrus/hydrus/client/ClientController.py", line 803, in InitModel
File "Hydrus/hydrus/core/HydrusController.py", line 498, in InitModel
File "Hydrus/hydrus/client/ClientController.py", line 189, in _InitDB
File "Hydrus/hydrus/client/ClientDB.py", line 299, in __init__
File "Hydrus/hydrus/core/HydrusDB.py", line 185, in __init__
File "Hydrus/hydrus/core/HydrusDB.py", line 454, in _InitDB
File "Hydrus/hydrus/core/HydrusDB.py", line 494, in _InitDBCursor
(don't know if code tags work here, never posted on 8kun) and the offending line:
def _InitDBCursor( self ):

self._CloseDBCursor()

db_path = os.path.join( self._db_dir, self._db_filenames[ 'main' ] )

db_just_created = not os.path.exists( db_path )

self._db = sqlite3.connect( db_path, isolation_level = None, detect_types = sqlite3.PARSE_DECLTYPES )

self._connection_timestamp = HydrusData.GetNow()

self._c = self._db.cursor()

if HG.no_db_temp_files:

self._c.execute( 'PRAGMA temp_store = 2;' ) # use memory for temp store exclusively


self._c.execute( 'ATTACH ":memory:" AS mem;' ) # ERROR HERE
I can't figure out what the problem is. My first thought was some kind of permission problem, but the file permissions are the same whether Hydrus creates a DB on the Samba share or whether I manually copy or use the built in migration to move an existing DB there. The permissions match the ones of a DB on my local machine (rw-rr on files, rwxr–r-xr-x on directories), and they appear as owned by me and I can read/write/delete them no problem. I'm hoping someone with experience with Hydrus internals can at least point me to things to try, because I am lost.

hi this is my first time posting here I use manjaro and I downloaded Hydrus and wanted to get a server running, but when I launch the server executable it doesn't seem to do anything. I then tried running it by using "python 3 server.py and got this error in the hydrus_crash.log
Traceback (most recent call last):
File "hydrus/server.py", line 120, in <module>
File "hydrus/hydrus/server/ServerController.py", line 32, in ProcessStartingAction
EOFError: EOF when reading a line


I don't seem to be scraping translations on gelbooru. Is there some parser that will grab these ("notes" / page_notes on gelbooru)?

>>14715 Found my problem. I just ran a text diff on client.db created on my local machine vs one created on the Samba, and a few bytes different in the header led to me discovering that the database had journal_mode=WAL on my local machine but the "legacy" journal mode when created on the Samba share. SQlite docs say WAL-mode does not work with DBs on network storage. I also found the Hydrus option no_wal. Running Hydrus normally, creating a new WAL DB, then running Hydrus no_wal to change it to legacy journaling, and then finally copying it over to Samba and running Hydrus again worked. Going to try it with my actual DB now. I don't have SQL experience so please let me know if what I am doing will lead to breaking things in the future. It seems to work for now.

>>14684 I am sorry I have been procrastinating on this question. When I looked at the SQL, it is actually a pain in the ass to insert back in manually. Since this has been asked a couple times before, I will just write a way in the client to do import/export, and then it will be easily sorted.

>>14693 There's no keyboard shortcut yet, but I want this easy navigation. I will think about adding options to edit the default names. Although I can see your problem, I also encourage slender gui sessions as much as possible. If a client has dozens and dozens of pages open, a better solution is often to figure out a neater workflow, perhaps with temporary processing tags on 'my tags' and saved searches to quickly bring those files back. I will think about it though. I collpased my mess of static 'sort this shit out' pages down to a single 'processing' page when I added favourite/saved searches, and it has been a godsend.

>>14694 I would like to. My current plan is to write a 'external executables that (optionally) eat files and spit out files' system that we can enter any exe into, and then plug that into the downloader or general file conversion. youtube-dl, ffmpeg, imagemagick, waifu2x, anything like that.

>>14695 >>14701 I am sorry for the trouble. I am not sure what was going on there, it looks like the downloader jobs did not start. Please let me know if you get this again.

>>14700 Do you have help->advanced mode turned on? I think I hide some of those clever commands behind that.

>>14702 Ah, do you mean for 'export folders'? I experimented with this with manual export (I think you need help->advanced mode turned on), and I did have a plan to try this out for export folders. I'll bump this up and take a proper look.

>>14703 It is not a pipe dream, but it is such a big job that it'll have to win a 'big job' poll. I expect to get through three or four big jobs a year once I am back to normal schedule, and the next poll should be up by Christmas. Let's see how it does there, and we'll see when we can expect it.

>>14709 >>14712 >>14713 I plan to have nice support for traditional zip/cbz/cbr files in future. It is sufficiently popular that I expect it to win a 'big job' poll pretty soon. I'll have proper thumbs, browsing of internal pages, and support for conversion back and forth between individual files and compiled 'chapters' or 'volumes' or whatever. This will be a part of my attempt to solve the problem of handling awkward 'page' tags.

>>14715 >>14721 Thank you for this information. I will attempt to detect WAL and networking gubbins in future. I generally do not recommend running the db itself across a network share due to latency issues, but I hope I can at least detect the situation and throw up a warning, and ideally recover and set WAL off. NEW THREAD >>14731 >>14731 >>14731 >>14731