How to implement Duplicity with Azure - Linux Tutorial

CARI.net Blog

November 15, 2018

🕒 Reading Time: 4 minutes

As the first step make sure that the OS is updated

yum update -y
yum upgrade -y

Install Duplicity

yum install ncftp python-paramiko python-pycryptopp lftp python-boto python-dev librsync-dev

1. We need to get all the required packages

2. After this we will download Duplicity from source

cd /root
wget http://code.launchpad.net/duplicity/0.6-series/0.6.22/+download/duplicity-0.6.22.tar.gz

tar xzvf duplicity*
cd duplicity*\

python setup.py install

Create SSH and GPG Keys

Create SSH Keys

ssh-keygen -t rsa

We will generate a key pair with the following command:

Press Enter at the prompts to create a password-less SSH key with the default settings

Transfer it to the system that will host your backups with this command:

ssh-copy-id root@backupHost

Answer yes to accept the unverified host, and then enter the root password of the remote system to transfer your public key.

Test that you can now log in without a password from your duplicity droplet by issuing:

ssh root@backupHost

While you are logged in through SSH, create the directory structure that will house our backup files:

mkdir -p /remotebackup/duplicityDroplet

You can name the directory anything you’d like, but remember the value so that you can specify it later.

When you are finished, exit back out into your duplicity droplet:

exit

Create GPG Keys

We will be using GPG for extra security and encryption. The commands will store our keys in a hidden directory at /root/.gnupg/:

gpg --gen-key

You will be asked a series of questions that will configure the parameters of the key pair.

Please select what kind of key you want:
   (1) RSA and RSA (default)
   (2) DSA and Elgamal
   (3) DSA (sign only)
   (4) RSA (sign only)
Your selection? 
RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048) 
Requested keysize is 2048 bits
Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0) 
Key does not expire at all
Is this correct? (y/N) y

Press enter to accept the default “RSA and RSA” keys. Press enter twice again to accept the default keysize and no expiration date.

Type y to confirm your parameters.

You need a user ID to identify your key; the software constructs the user ID
from the Real Name, Comment and Email Address in this form:
    "Your Name (Der Name) <name@yourname.na>"

Real name: Your Name
Email address: your_email@example.com
Comment: 
You selected this USER-ID:
    "Your Name <your_email@example.com>"

Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? o

Enter the name, email address, and optionally, a comment that will be associated with this key. Type O to confirm your information.

Next, you will be setting up a passphrase to use with GPG. Do Not Enter any Passphrase leave it blank by hitting “Enter”

At this point, you will be asked to generate entropy. Entropy is basically a word that describes how much unpredictability is in a system. Your VPS needs entropy to create a key that is actually random.

We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.

Not enough random bytes available.  Please do some other work to give
the OS a chance to collect more entropy! (Need 280 more bytes)

To make this process faster use Haveged to generate Entropy, run the following commands in another SSH window

yum install haveged
chkconfig haveged on

When you’ve generated enough random pieces of information, your key will be created:

gpg: /root/.gnupg/trustdb.gpg: trustdb created
gpg: key 05AB3DF5 marked as ultimately trusted
public and secret key created and signed.

gpg: checking the trustdb
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
pub   2048R/05AB3DF5 2013-09-19
      Key fingerprint = AF21 2669 07F7 ADDE 4ECF  2A33 A57F 6998 05AB 3DF5
uid                  Your Name 
sub   2048R/32866E3B 2013-09-19

pub 2048R/015AB3DF5 2013-09-19

The part highlighted above is your public key ID. You will need this later to encrypt the data you will be transferring.

If you forget to write down your public key ID, you can get it again by querying the gpg keyring:

gpg --list-keys

/root/.gnupg/pubring.gpg
------------------------
pub   2048R/015AB3DF5 2013-09-19
uid                  Your Name <your_email@example.com>
sub   2048R/312866E3B 2013-09-19

We now have all of the necessary components in place to securely backup using duplicity.

Setup Azure Environment

In this point you only need to get an Storage Account in Azure. and get the Azure Account Key:

Duplicity to Azure Blob

First we need to get Azure installed on the server by running these commands:

sudo yum install python-pip
sudo pip install azure

(If necessary update the version)

After the installation is completed go edit this file /etc/enviroment and input the following information:

AZURE_ACCOUNT_NAME=storageaccountname
AZURE_ACCOUNT_KEY=storageaccountkey
PASSPHRASE=passphrase_for_GPG_encryption

(When creating the Azure container you should get the Azure Account Name and the Azure Accoun Key)

Initial Testing

We will create a test directory to back up and we will fill the directory files numbered 1-100 by running the following commands:

cd ~
mkdir test
touch test/file{1..100}

To create a backup and send it to Azure

duplicity --no-encryption /path/on/theserver azure://blobname

A succesfull backup gives this result:

Local and Remote metadata are synchronized, no sync needed.
Last full backup date: none
No signatures found, switching to full backup.
--------------[ Backup Statistics ]--------------
StartTime 1541963735.41 (Sun Nov 11 11:15:35 2018)
EndTime 1541963958.35 (Sun Nov 11 11:19:18 2018)
ElapsedTime 222.94 (3 minutes 42.94 seconds)
SourceFiles 81032
SourceFileSize 2330593566 (2.17 GB)
NewFiles 81032
NewFileSize 2330593566 (2.17 GB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 81032
RawDeltaSize 2297167355 (2.14 GB)
TotalDestinationSizeChange 1060502883 (1011 MB)
Errors 0

You can check if the backup was actually created with this commands:

ssh root@nameofthehost
cd /nameofthehost/duplicityDroplet
ls

Restore Backup from Azure

duplicity azure://blobname /path/on/theserver

Exclude folders from Duplicity Backup

duplicity --no-encryption --exclude /path1excluded --exclude /path2excluded --exclude /path3excluded / azure://duplicitydemo

In case you see an error related to Paramiko when trying to run a backup just install this:

sudo yum install duply duplicity python-paramiko python-gobject-2

The last step is: Create a CronJob to setup Backup frequency

Contact Info