In a previous article, I'd talked about DigitalOcean's new offering called "Spaces", which allows you to store unstructured info like files, blobs, videos etc for a really cheap price. While you can use the GUI web interface to add and remove files, the real power comes when we integrate it with the command line and are able to automate tasks like file updation, backups, etc. In this tutorial, I'll show you how to set up command line access - not just to DigitalOcean Spaces, but for any storage service that uses the S3 protocol.
Step 1: Setting up a Test Space in DigitalOcean
Based on the steps in the previous article, I've set up a test space that contains two files:
- Piano Songs.txt
You can see them here:
Step 2: Installing the s3cmd Command Line Tool on Linux
The tool we're going to use to access spaces is called s3cmd. While Linux flavors like Ubuntu and CentOS/RHEL come with default s3cmd packages, they're all rather out of date. What we want is at least version 2.0.0. But CentOS for example, comes with version 1.6 by default.
A great place to search for new RPMs is Rpmfind. For example, you can get a list of s3cmd rpms at this URL here. You can scroll to find the rpm appropriate to your distribution. In this example, I choose the package for Enterprise Linux 7 for my CentOS 7 distro. Once I have the package, I type the following commands into my Linux server to download and install the RPM:
wget ftp://22.214.171.124/linux/epel/testing/7/x86_64/s/s3cmd-2.0.0-2.el7.noarch.rpm yum localinstall s3cmd-2.0.0-2.el7.noarch.rpm
This allows you to install s3cmd in a formal way instead of messily building from source, or even just downloading the tool. Once you find the right RPM and extract it, you can check the version using the following command:
And get the following output:
Step 3: Set up the Access Keys on DigitalOcean
To connect with spaces with s3cmd, we need two things:
- An access key
- A secret key
We can get both of them by logging into DigitalOcean and clicking the "API" tab and scrolling down to the section called "Spaces access keys". Then click "Generate New Key". You'll be asked to give a name for the key as shown here:
Once the key has been generated, you'll see two lines for the new key. The top line is the access key. The bottom line is the secret key:
It's important to note that the "secret key" will display only once! So make sure you copy it and store it in a place where you can retrieve it if necessary. The access key and secret key are what we need to connect to spaces.
Step 4: Configure s3cmd to Connect to DigitalOcean Spaces
We need to provide s3cmd with the connection and authentication information. To do this, type the following into your Linux server:
This will initiate the configuration process and will ask you a series of questions. Here are the important ones with their corresponding values:
- Access key - Enter your access key
- Secret key - Enter your secret key
- S3 Endpoint - nyc3.digitaloceanspaces.com
- Bucket Template - %(bucket)s.nyc3.digitaloceanspaces.com
For the remaining questions, you can just press "Enter" and just accept the defaults. Once the settings have been entered, the tool will ask you to test the connection and save your settings like this:
As you can see here, the connection is successful and the configuration has been saved!
Accessing the Space using s3cmd
Now that the configuration is complete, we can access our list of spaces on DigitalOcean using the command:
For our example, it shows us that we have a space called s3://test-spaces-demo .
And we can get the details of any individual space using the command:
s3cmd ls [space-name]
You can see in the screenshot above, that our two files have been listed as expected. You can get a much fuller list of s3cmd commands using this link and it'll show you how to add new files, update them, delete objects etc. So now you have full CLI control over the objects in your DigitalOcean spaces.