Setting up Syncthing.

A very common problem one has if one has enough files stacked up in one place, is whether or not those files have been copied to another system already. Have they already been copied off? To where on the other system were they copied? Sure, you can deduplicate them through various means but that tends to be kind of a sledgehammer thing to do, especially when one of the things with files is a mobile device. You could always upload the files to a provider's cloud¹ storage, like Google Drive or iCloud or something.

But what if you don't want to? What if your Google account doesn't have enough space and you don't want to buy more for whatever reason? Maybe you like self-hosting things you use. Maybe you use several different platforms (Android phone, Apple laptop, Linux server) and want to readily and easily have access to those files from any of them? What if the storage capacity on your mobile device is limited and you can't pop a microSD card in (which is a direction many mobile manufacturers seem to be going in)? What if you want to automatically move those files off as they're saved to keep the amount of usable space at more or less the same volume in your constrained environment? On the flip side of the coin, what if you want to easily copy stuff up to one of your devices, which isn't necessarily an easy thing to do these days?

Enter Syncthing

Syncthing is a tool for keeping one or more configured directories synchronized on multiple devices. If a file is created in a shared directory on one device it gets replicated onto all the other devices that the directory is shared with. Conversely, if a file gets deleted or changed on one device the deletion (or a new, edited copy) gets propagated to all of the others within seconds. You don't need to key in or remember IP addresses or anything like that. If the device is on the Net it's potentially syncable. I've found that Syncthing is remarkably good at figuring out how to punch holes through multiple layers of firewalls, which I had not expected. It takes some tinkering to set up Syncthing the first two or three times (or at least I had to tinker with it for a while) but once it's up and running it's pretty solid.

Unlike commercial storage providers, only your devices ever see those files. The synched files are encrypted when they leave the device, and are decrypted when they arrive on the other devices. Additionally, only devices that you've specifically authorized can have files shared to them. Syncthing is also granular in what and how you share files: If you install Syncthing on some of your devices you can share whatever files you want with any combination of those devices. It is not all or nothing.

Syncthing is available for pretty much every platform you're likely to use and even a few that you probably aren't. I installed Syncthing for Android out of the Google Play store, but if you're feeling saucy it's also in the F-Droid software collection. There is also an implementation of Syncthing for Apple iOS, but I haven't tried it. For Linux users Syncthing is available in the package repositories of Arch Linux, Ubuntu, and Debian (all of which I'm using), and probably others. If you want to play around with the source code or compile it yourself you can always find it on Github.

Let's start with installing Syncthing on two devices, a Raspberry Pi running Raspbian sitting on my desk and an Android phone I've added to my collection of things to test on. Let's assume for the moment that you're SSHing into said Linux box from a laptop or desktop machine. I recommend that, when you do so, you do it like this; I'll explain why in a minute:

{18:47:49 @ Sat Jan 28}[drwho @ windbringer ~]() $ ssh -L 10000:localhost:8384 pi@setec-astronomy

The stuff after -L means, roughly, "Open port 10000 on my computer when you open the SSH connection. Connect it to port 8384 on the computer I just SSH'd into." This is called SSH port forwarding, which you need in this case because there is no desktop app; the control panel is a web page. If you want, you can make port 8384 available to your entire home network later so you don't have to mess with this, but for the moment it is what it is.

The mobile apps, however, do have their own control panel, so don't worry about that.

The Raspbian install is about a straightforward as it gets (and, of course, it works like this on pretty much any Debian-derived install):

{18:25:26 @ Sat Jan 28}[pi @ setec-astronomy ~](2) $ sudo apt-get install syncthing

{18:26:18 @ Sat Jan 28}[pi @ setec-astronomy ~](2) $ systemctl --user enable syncthing.service
Created symlink /home/pi/.config/systemd/user/default.target.wants/syncthing.service → /usr/lib/systemd/user/syncthing.service.

{18:26:25 @ Sat Jan 28}[pi @ setec-astronomy ~](2) $ systemctl --user start syncthing.service

Note: I'm doing this inside of a Screen session. If you do and you get a "Failed to connect to bus" error, it means that your shell doesn't have the XDG_RUNTIME_DIR variable set. To fix this, temporarily disconnect from your Screen session and run the following command:

{18:31:51 @ Sat Jan 28}[pi @ setec-astronomy ~]() $ export| grep XDG_RUNTIME | sed 's/^declare -x //'>> .bashrc

When you resume your Screen session, run the command source ~/.bashrc so the variable will take effect and you won't get that error anymore.

Now, the reason I had you set up that SSH port forward is because we're going to configure Syncthing on your Linux box. Plug the URL http://localhost:10000/ into your web browser. You will see Syncthing asking you if you want to turn on anonymous reporting. I just click "no" and move on with my life, and I suggest that you do as well. Once that's sorted you'll see the Syncthing control panel. "Default Folder" will just be a gray bar on the screen; click it to unroll it into what you see in the screenshot. By default you'll have a directory called Sync/ which is pretty much what it says on the tin. That is the default directory that Syncthing sets up for you and creates it if the directory doesn't exist already. You can get rid of this by hitting the Edit button and then the Remove button, but I advise against it. In general, you want to tweak as few settings as possible to get things running smoothly, and from experience I've found that Syncthing is no different in this regard. For our purposes things are fine as they are.

The second part of our proof of concept is installing Syncthing on a cellphone or tablet. All you need to do is search your appstore of choice for 'Syncthing' and install it. I would also recommend installing a file manager on Android devices because it makes a few later steps much easier to perform. I recommend Material Files because it's easy to use and lets you set bookmarks for some directories on your phone.

Now let's do something simple: We're going to keep a directory on our mobile device synched up with the shared directory on the Linux box. Let's throw a couple of random files in there so we know if it worked:

{16:11:29 @ Sun Jan 29}[pi @ setec-astronomy ~](2) $ ls -alF Sync/
total 52
drwxr-xr-x  3 pi pi  4096 Jan 2916:11 ./
drwxr-xr-x 17 pi pi  4096 Jan 2916:09 ../
-rw-r--r--  1 pi pi  5947 Jan 2820:59 laid-off.md
-rw-r--r--  1 pi pi   503 Jan 2820:59 neologism-thousand-meter-stare.md
-rw-r--r--  1 pi pi  6325 Jan 2820:59 setting-up-syncthing.md
drwxr-xr-x  2 pi pi  4096 Jan 2818:26 .stfolder/
-rw-r--r--  1 pi pi 16610 Jan 2820:59 tools-on-the-bench.md

Start up Syncthing on your mobile device. Tap through until you see the "Storage Permission" screen; this means that Syncthing is asking your permission to read and create files on your device, and it won't work without it. Tap "Grant Permission" and when you're asked, tap "Allow access to manage all files." Then hit the back arrow. Continue. Then you'll see a screen asking you for "Location Permission." What Syncthing is asking is whether or not it's allowed to run only on wireless networks that you give it permission to. You can tap continue and not really cause yourself any trouble. Then Syncthing will churn for a while and then you'll get a popup that says "Battery Optimization." If you tap "Turn off for Syncthing" your phone's power cell will be used up a little faster than usual. You can also tap "Later" to skip this, but you can come back to it later. For our purposes, hit "Later."

Now to connect the two devices with Syncthing. Go to the Syncthing control panel you have open in your web browser and hit the "Actions" button in the top right corner, then hit Show ID. You will be presented with a popup window that says Device Identification and has a long string of capital letters and numbers and a large QR code that contains the same information. We want the QR code because that means not having to retype that device ID which never gets keyed in correctly the first time.

On your mobile device tap on "Devices" in Syncthing. You'll see "No devices found" because you haven't configured any yet. We're about to change this - tap the plus sign in the top-right corner of Syncthing to open the Add Device panel. Tap on the little QR code looking icon next to "Device ID." A window will pop up asking you if you want to allow Syncthing to take pictures and record video; tap "while using the app" because Syncthing needs to photograph the QR code in your web brower. Point the camera at your screen and it'll catch the QR code and translate it into the system ID. Tap on the "Name" field and type in a name you'll remember, then tap the checkmark in the top right corner. You don't need to set anything else here. In your desktop web browser, tap the escape key to close the "Add Remote Device" window.

Now the somewhat annoying part: Telling Syncthing on your Linux box about your mobile device. If you click the "Add Remote Device" button in your web browser you will be shown the Add Device popup. If you go back to your phone and in Syncthing tap the three-bars menu icon in the top left and then Show device ID, you will see the long alphanumeric string and QR code that identifies your mobile. Glance back at your web browser and, if you're lucky you'll see your mobile device's Device ID in the list of nearby devices. If you're not you'll have to retype your phone's Device ID in that field, or find a way to copy it off of your phone so you can just copy-and-paste it. If you tap the sideways V shaped icon on your phone you'll see a scrollable list of ways that you can get that ID off of your phone. I e-mailed it to myself (see the topmost icon), then copy-and-pasted it into the "Device ID" field. I also recommend putting text in the "Device Name" field so that you know what device it is.² It should look something like this.

The connection's set up, so now let's share the default Sync/ folder with your phone. In your web browser on your desktop, click on the bar that says Default Folder, which will show you the vital statistics for that share (note "Folder Path" which shows you the directory on the Linux box it corresponds to; in this demonstration it's /home/pi/Sync). Click the Default Folder's Edit button and then click on the Sharing tab. You will see a list of Syncthing devices that you can connect the shared directory to. Check the box next to the name of your mobile device (in this example, Crappy Android Burner Phone (affiliate link - it's a nice little phone to keep in your emergency kit, plus you don't have to activate it if you're only using wifi)) and then click the Save button.

Now, back on your mobile phone, look at your notifications (Android: Pull the notification bar down; iOS: I don't know, do whatever it is that works) and you should see a little message saying "Device wants to share folder..." Tap on it, tap on "Default Folder" and give it a name that makes sense to you (for the purposes of this writeup, call it Sync). Don't touch any of the other settings, then tap the checkmark in the top-right corner of the screen. Tap on the "Directory" field because you have to tell it where on your mobile the synched files will live. To make life much easier I'm going to tap Documents in the location picker that comes up because it already exists and it's right there. You can, of course, get more creative later but as I said earlier, keep it as simple as possible to make it work. Tap "Use this folder." When the popup on your phone asks "Allow Syncthing to access files in Documents?" tap the Allow button. Then tap the checkmark icon in the top right corner to save your settings.

On your mobile drop back to the app screen and tap the Material Files icon (or use whatever file manager you prefer; substitute as appropriate for this paragraph). When you're asked if you want to allow Material Files to manage all files, tap the toggle to turn it on and then the back arrow (because otherwise it can't do its job, which is navigate the files on your phone). Tap on the Documents folder that you connected Syncthing to and you should see the files you copied into the Sync directory on your phone.

And now you're done, as annoying as it was. The nice thing is that you only have to do it once - get your devices to know about each other and you're done. Share a directory once and then you never have to worry about it again. Syncthing has the nice property of being "set it and forget it," as they say. Copy files into the Sync directory on one device and they'll be copied automatically over to the other one. Conversely, delete one of the files (or move it out of the shared directory) and it'll be erased from the other devices.

A thing I learned while writing this: if the copies of Syncthing you're using have very different version numbers you will run into problems, usually the devices won't be able to see each other. On the RasPi I used for this demonstration I wasn't seeing the files moving in either direction, and Syncthing on my Android test phone kept throwing a server selected unsupported protocol version 303 error. What I did was follow the procedure here to add the official Syncthing package repo to my RasPi and then did a sudo apt-get update to install the latest stable version of Syncthing. That fixed the problem without needing to do anything else.

Happy hacking!