/hydrus/ - Hydrus Network

Archive for bug reports, feature requests, and other discussion for the hydrus network.

Index Catalog Archive Bottom Refresh
Name
Options
Subject
Message

Max message length: 12000

files

Max file size: 32.00 MB

Total max file size: 50.00 MB

Max files: 5

Supported file types: GIF, JPG, PNG, WebM, OGG, and more

E-mail
Password

(used to delete files and posts)

Misc

Remember to follow the Rules

The backup domains are located at 8chan.se and 8chan.cc. TOR access can be found here, or you can access the TOR portal from the clearnet at Redchannit 3.0.

Uncommon Time Winter Stream

Interboard /christmas/ Event has Begun!
Come celebrate Christmas with us here


8chan.moe is a hobby project with no affiliation whatsoever to the administration of any other "8chan" site, past or present.

(35.02 KB 583x263 hydrus-encryption.jpg)

Database Encrypted Sync Anonymous 09/09/2017 (Sat) 13:42:39 Id: 32c621 No. 6712
I haven't seen this discussed so I thought I'd make a thread on it. We are all aware that Hydrus runs fine inside a Truecrypt container, but having one huge file can be a problem. For instance if you backup your 500GB database inside a truecrypt container and then download even just 1 image you basically need to backup the whole 500GB all over again if you run an automated backup to an external HDD or a NAS. Another problem with Truecrypt/Veracrypt volumes is you have to set the container size at the time of making it, leaving it too big or too small all too easily. Well I've been experimenting with different crypto programs that feature encrypting directories while keeping files separate. The filenames and contents are unreadable, but they are still their own files so syncing and backup programs know exactly what to copy and what is the same. The problem is that Hydrus doesn't want to run at all within these encrypted directories, but luckily we can have our installed client separate to the db folder. >install an encryption program (I used cppcryptfs but there are a bunch for various platforms) >create a new encrypted folder on any drive >mount it as a Drive letter (you will need to use the same Drive letter each time) >Move your Hydrus db folder to that mounted drive (or you can make a new one by ignoring this step) >create a shortcut to your client.exe adding -d="path to db folder on virtual drive" (e.g "C:\Hydrus Network\client.exe" -d="Z:\db") >use shortcut it should find your database and start like normal, with any new files being encrypted as their own files that can be synced individually. I haven't tried it but theoretically you could even mount a cloud provider as a virtual drive and store your entire database that way, probably stupid but I might try it for fun.
Is there anything especially good about cppcryptfs besides being able to run on Windows and providing encryption of loose files and filenames? I would be worried about using something that's clearly marked "experimental" and only used by a few people. Have you tried cryptomator, encfs4win, encfsmp, or mounting a secret rclone provider?
Question is, why do you feel the need to have your imageboard memes encrypted?
(26.34 KB 600x263 xi.jpg)

>>6714 I tried Cryptomator, didn't like it's implementation. Tried encfsmp, it works fine but apparently encfs got tore a new one by a security audit and "the developers have no idea what they're doing, it's worse than nothing". I'm sure it's fine for personal use but all your cool crypto friends might bully you for it. Rclone is a good alternative, I just didn't have the patience to get a local provider working the way I wanted. cppcryptfs is a windows version of gocryptfs which thus far seems okay. >>6715 What if I needed to store illegal memes like pic related? Honestly this is just password protected porn for paranoid fools who thinks anyone cares they like the anime boobies.
I don't know anything about cppcryptfs or any other advanced systems, but if you want to backup a truecrypt-like container efficiently, you can do it efficiently by mounting the backup volume and copying internal contents. i.e.: Create 500GB container on your main comp, put hydrus on it Create 500GB container on your backup location Keep main one mounted to do your normal hydrus stuff. To make a backup: Shut down hydrus Mount backup container using your encryption program. Backup the contents from main mounted location to backup. Dismount backup. You aren't cloning the whole 500GB whack every time, only the internal changes. I know one user who stores his client_files directory on the cloud through some Windows drive mounting thing that gives hydrus a real path to look at. It seems to work ok, although it can obviously be laggy in certain maintenance situations and you are losing your privacy to some Silicon Valley nascent AI, wew lad. Some other users have also reported good success storing their media over a network share. I recommend thumbnails are always stored locally, since access latency affects these most. The db files should also preferably be on the local machine as their transactional db stuff is only reliable on local hard drives where the OS can guarantee honest exclusive locks and so on.
>>6712 Use LUKS Encrypt the whole volume.
>>6718 >losing your privacy to some Silicon Valley nascent AI, wew lad. Does encrypting the whole db file work for backup only if you don't plan to use it actively? Also, on this "incremental backup" where only the actual changes and not the entire things are copied over, how does encryption play in this? I add a single image to my db, then encrpyt it again. The added image is just a few mb, but will I have to update every GB of the entire database if a single thing changes and it's re-encrypted? (I am using veracrypt)
>>10556 >>6718 Encrypted ZFS could be a solution
>>10556 To make it simple, backup the contents of one encrypted container to another. Do not backup encrypted images, as you'll be doing 500GB every time. Instead just mount the two images and backup their internal folder structures as normal, as if you were backing up an unencrypted location. Afaik, there's no super easy way to do a minimal backup where you only update the sectors of the image/files that have changed, although there is probably some specialised software out there that does that by tracking sector hashes or something (although this would presumably have to rescan the whole contents of the source anyway to spot differences, or be part of the fundamental file system like >>10564 suggests). It is simpler just to use FreeFileSync and copy-overwrite the files with newer modification timestamps to the backup. db files can be pretty big–maybe on the order of 10GB for hydrus, but 10GB is ultimately not a big deal for local backups once a day/week/month. The media files are just jpgs and whatever and copy in reasonable time as they would in any other situation. I do a weekly backup of a 1 million file hydrus db to a usb drive, and it takes about five minutes to scan and five minutes to action. Easy peasy. I don't know enough about veracrypt to comment cleverly about it. If you are putting files in an encrypted container, it is easy, but if it does some mickey-mouse on-the-fly per-file encryption, then your situation may be different.
>encfs >files are same size as unencrypted and not merged into one file >can write a tiny script to decrypt when your run Hydrus and encrypt back when you exit >don't have to have whole disk encryped I've never tried it with the cloud meme but it may work.
>>10578 This is very late, but I want to thank you for your reply, Dev. If it'll run with the entire hydrus directory in an encrypted container, then it's all dandy. I'lll just try. Read only if time is there: I will probably put the entire thing into an encrypted container, I don't know what I was thinking, murmuring about some sort of container only for the db folder. Plus, it's not like excluding the program files has any benefits, they're tiny in comparison. I just have to live with mounting and dismounting each time I backup. What am I babbling, obfuscation is the primary objective for encryption, so of course I can't backup unmounted and expect the program to not see it as the entire thing having changed. Inconvenience or security is the dichotomy I guess.
>>6712 >Hydrus runs fine inside a Truecrypt container Pretty mediocre at best. LUKS runs better. > leaving it too big or too small all too easily Use Linux' LVM, then you can grow it (shrinking is not directly possible in most situations, but you can create a new smaller LV, copy the data, delete the old one).
>>10578 >>11116 (cont'd) One more thing, if you use Borgbackup for backups, that can actually make backups relatively efficient - even for the Sqlite databases you should save like 70%, and for the actual files it will be next to 100% of what was in the older backup. So on a re-run your 500GB backup will likely be just the new / changed data plus maybe 1-2GB or something on that fast method of [backup-side] deduplication not being entirely perfect. This is basically a "super easy way" to do a relatively minimal backup. Well, maybe it's only "easy" rather than super easy if you're currently on Windows and need to use the Win 10 Linux Subsystem, a network share or a live USB or a VM or something to run Borgbackup on that directory, but it's not going to be terribly hard either.
>>10578 That is not going to be convenient unless we can make it simple (especially when someone has TBs of data on a RAID-enabled NAS)
>>11118 Borgbackup is your ticket to make it simple. >>11123 > https://github.com/bup/bup (Python, slow) Actually this one is a fine CLI tool too, but unlike borg it's not considered stable by the developers, and you likely do want some borg features like the ability to prune older database backups and staggered retention for the other files. Slow is extremely relative. Yea, python is not the fastest language around even if you use pypy (which however definitely makes most software that crunches some data quite a bit faster over standard python). But even that may not matter much if you're mainly waiting for IO coming from your HDD, which is usually what happens if you try to get data about your 40TB of image files (or whatever you might have on your array). > See https://github.com/gilbertchen/benchmarking for info A very limited test without the various configurables and apparently without any attempts to isolate how much time was spent waiting for IO or whatever, done only on the weirdly scheduled OS that is OSX? And I don't even see what interpreters they used for python (was the "python" symlink pypy or mainline python and not a link?) and such. Never mind there are actually a LOT of configurables in these tools, and for some weird reason they didn't even try to pick the same compression algorithms even when it was possible (why not run the same settings on the same zlib or lz4 or zstd compression on all the tools so they become more comparable?). Anyhow, use Duplicacy or restic over borg if you prefer, but I wouldn't pay too much attention to that benchmark.
>>11123 I'm using duplicati myself, it's still slow because OS chokes on the amount of hard drive seeks, takes around 30 minutes to back up a 100k set for me, probably going to be even slower when you're backing up from an encrypted volume. It doesn't even matter if there were any changes to the collection, just the act of checking 100k files on a hard drive is a very slow process.
>>11123 I haven't used any of those, so I can't comment cleverly. I expect they do the job well. I use FreeFileSync once a week on my laptop's IRL 1.2M-file hydrus db, backing up to a WD passport. It takes several minutes to compare all the millions of files and thumbs and then a few more to sync the backup. Unless I have a gonzo week, I'd estimate it is usually less than 20min total. Since I have it going in the background as I put the hydrus build together on my dev pc, it is no trouble at all.


Forms
Delete
Report
Quick Reply