After getting the excellent TeslaMate up and running, I wanted to make sure I didn’t lose any data by having an automated backup. Rather than me forget to back it up and copy files over, I thought I’d see if I could re-use the rclone to Google Drive setup I used with TeslaCam on a Pi 0 W (Guide here).
EDIT: 19/01/2021 – I’ve had to update the guide, as Google looks to have started 403’ing some of the requests which is causing backups to fail. This looks to be due to the lack of Client ID and Client Secret, meaning that all the rclone requests are getting lumped together and then rate limited.
If you are new to this guide and haven’t set this up yet, then carry on below. If you’ve set this up in the past, complete the ‘Get a Google Client ID and Client Secret’ then scroll down to the bottom and follow ‘How to edit your existing rclone config to add in your Google Client ID and Client Secret’ (a catchy heading I’m sure you’ll agree!).
Get a Google Client ID and Client Secret
Log into the Google API console (Google Cloud Platform) with your Google account.
- Select an existing project, or create a new one.
- Under "ENABLE APIS AND SERVICES" search for "Drive", and enable the "Google Drive API".
- Click "Credentials" in the left-side panel (not "Create credentials", which opens the wizard), then "Create credentials"
- Click on "CONFIGURE CONSENT SCREEN" button (near the top right corner of the right panel), then select "External" and click on "CREATE"; on the next screen, enter an "Application name" ("rclone" is OK), complete the support and developer email addresses (just your normal email) then click on "Save & Continue" Click "Save & Continue" again on the Scopes screen, click "Save & Continue" again on the Test Users screen - It should then show you a summary of what you have just setup.
- Click again on "Credentials" on the left panel to go back to the "Credentials" screen.
- Click on the "+ CREATE CREDENTIALS" button at the top of the screen, then select "OAuth client ID".
- Choose an application type of "Desktop app" if you using a Google account or "Other" if you using a Google Workspace account and click "Create". (the default name is fine - feel free to enter TeslaMate if you wish)
- Click on the OAuth Consent Screen and Click PUBLISH APP and confirm you wish to publish.
It will now give you your Client ID and Secret, store these somewhere (or leave the page open) as we’ll need them shortly.
Installing and configuring rclone
Get to the shell or ssh into your Pi
First step, start the session as root: sudo -i
Next we install rclone: curl https://rclone.org/install.sh | sudo bash
Now to configure: rclone config
These are the options to choose to setup Google Drive
n for new remotes
gdrive for name
13 for Google Drive
Paste in your Client ID (hit enter)
Paste in your Client Secret (hit enter)
Option 1 (some people have said option 3 works just as well)
Leave root_folder_id and service_account_file blank (hit enter to leave blank on both)
n to skip advanced config
n for autoconfig
Copy the link to a browser to create a token, then approve in the Google authentication screen (click Advanced, then click Go to TeslaMate (or whatever you called the service)) then click Allow on the popup, then Allow again for the Confirm your choices screen.
Copy the verification code and paste into your SSH session.
n for team drive
y to confirm
q then enter to quit config
Now run: export RCLONE_DRIVE="gdrive"
Let’s now verify that gdrive has been created by running: rclone listremotes
You should now see gdrive, if not you’ll need to start the rclone config process again as something has gone wrong.
Run this: rclone lsd "$RCLONE_DRIVE": there should be no files showing.
We now need to choose a folder to copy the backup files to, I’ve assumed TeslaMate, feel free to swap for whatever works better for you, run this command: export RCLONE_PATH="TeslaMate"
Now we will make the folder in Google Drive with that folder name: rclone mkdir "$RCLONE_DRIVE:TeslaMate"
Run this command again to check if the folder has now been created: rclone lsd "$RCLONE_DRIVE":
This step might not be needed for our purposes, but I ran it just in case: export ARCHIVE_SYSTEM=rclone
TeslaMate backup and upload script
Staying in the sudo -i session
Run this to make your backup folder on your Pi: mkdir /home/pi/tmbackup You can name your folder what you like, but ensure you adjust it for the rest of the steps from here on.
Navigate to that folder: cd /home/pi/tmbackup
Run: echo $PATH copy the output of that, we’ll need it for our script shortly
We now need to create our script in that folder, using your favourite editor (I’m using nano): nano tmbackup.sh
This is the contents of my script:
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
now=`date +"%A"`
cd /home/pi/tmbackup
sudo /usr/local/bin/docker-compose exec -T database pg_dump -U teslamate teslamate > /home/pi/tmbackup/teslamate.bck_${now}
rclone copy --max-age 24h /home/pi/tmbackup --include 'teslamate.*' gdrive:TeslaMate
If your output from echo $PATH above was different, replace after the PATH=
If your database is called db or something else, you’ll need to adjust your script to suit.
I’ve setup the filename to be teslamate.bck_Dayoftheweek that way you’ll only keep 7 days worth both on your Pi and on your Google Drive folder, if you want to keep all of them (*shrugs*) then feel free to change the date format.
The script will copy over any files in our backup drive that start with teslamate. that are less than 24 hours old.
Quit your editor and save your file.
You’ll now need to change permission on your script to make it executable, run: chmod +x tmbackup.sh
You can now run your script to test it works! To do so, run the following: ./tmbackup.sh
Once the command prompt returns, run: ls -l this should show you all the filenames in your directory including file sizes.
Adding to the cron
Now it’s time to get the TeslaMate backup automated.
Run the following (whilst still in the root session): crontab -e
I’ve set my backup job to run at 03:00 every day, as I hope both me and the car will be asleep by then! To do that, I use this entry:
0 3 * * * /home/pi/tmbackup/tmbackup.sh
This means 0 minutes past 3am every day
Quit the editor, save the crontab and you’re good to go! – I would suggest testing this first, before going to bed and keeping your fingers crossed, to do that, trying the cron entry above, but with X * * * *, with X being the next minute you’re about to hit, let it run, then change it back to something sensible like the above.
All done! Your TeslaMate backup is now automated 🙂
How to edit your existing rclone config to add in your Google Client ID and Client Secret
If you’ve set this up in the past, but are having backup issues, this is how to update your existing config:
SSH in
On a Pi run: sudo -irclone config
These are the options to update rclone with your new Client ID and Client Secret:
e (edit existing remote) 1 (assuming that's your gdrive one) Y - to edit Client ID Paste in Client ID - Enter Y - to edit Client Secret - Enter Paste in the Client Secret - Enter N - for Value "scope" N - for root folder ID N - for service account file N - for advanced config Y - This is OK (default) Y - already have a token, refresh N - to auto config Y - To refresh your token
Then follow the steps to re-authenticate for rclone accessing google drive with the google account you created the ID and Secret with.
You will see a warning screen where it says this app hasn’t been verified, click advanced and then ‘Go to TeslaMate (unsafe) <- the name will depend on what you set during the OAuth setup).
After that it’ll ask you to confirm the access, so click allow, then allow again. You are then given the code to paste back into your SSH session, do this and press enter.
N - to configure this as a team drive Q - to quit
Run a test backup from the /tmbackup directory by running ./tmbackup.sh (assuming the directory and script names are as per the original guide).
Have you also seen..
How to bulk import Supercharger location data
How to bulk import Destination charger data