CloudFiles curl example
I wrote the following script to simplify accessing CloudFiles container’s by utilizing cURL. This example will show you how to list your containers as well as their contents, via the command line without having to login to your control panel. This can also be a useful example to provide you with a sample code base to implement within your own infrastructure.
CloudFiles uses a a “ReST” API which allow’s you to interact with it in a variety of ways by using curl, PHP, Python, and Ruby. This particular script was written in bash so that I could utilize it from a shell, however the same principals can be adapted to the other technologies as the API will interact with them in the same manner. So how does it work?
First the API expects a username and a password to provide initial authentication. In this case you will pass your username and API key provided by the rackspace control panel located HERE. Within the script you will see the following code which performs this action:
${CURL} "X-Auth-User: ${USER}" -H "X-Auth-Key: ${APIKEY}" https://auth.api.rackspacecloud.com/v1.0 2>&1 \ | grep Storage | awk -F'<' '{print $2}'
After passing your username and API key to the ReST API it will return you a Storage Token, and a Storage URL. These will be used to access your containers. Keep in mind that the Storage Token should not be saved within a database as the token does expire and is required for any operation which lists, uploads or deletes data. Once we have established authentication and we have our storage token we can then list and modify contents. In this example we are only listing contents, however modifying data is not much different. Here are a few examples on how to list container’s and their contents.
List all containers:
curl -s -H "X-Auth-Token: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXX" <container URL>
List a containers contents:
curl -s -H "X-Auth-Token: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXX" <container URL>/<container>
Additional information on how to manipulate data can be found in the CloudFiles developers guide located on the rackspace website at the following URL:
http://docs.rackspacecloud.com/files/api/cf-devguide-latest.pdf
You can also find various bindings for PHP, Python, Ruby, Java etc.. here:
http://www.rackspacecloud.com/cloud_hosting_products/files/api
Now that we have covered the basics on how the script works, here are some example usage scenarios:
Basic container listing:
$./cf-list.sh USERNAME APIKEYContainer listing with verbosity(outputs to a log file):
$./cf-list.sh USERNAME APIKEY 1
#!/bin/bash # # File: cf-list.sh # # Written by: Tim Galyean # Date: 8/4/2010 # ################################################## USER="${1}" APIKEY="${2}" DEBUG="${3}" LOG="cflist.log" function f_exec() { if [ "${DEBUG}" = "1" ]; then # setting debug to 1 allows you to see full HTTP headers # using this option will redirect the output to cflist.log echo "redirecting verbose output to ${LOG}" CURL="curl -v -H" f_grabauthtoken 2>&1 > ${LOG} f_list 1>>${LOG} 2>&1 # this url can be used to parse your containers and their contents # Example: curl -s -H "X-Auth-Token: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXX" <container URL>/<container> echo "curl -s -H \"X-Auth-Token: ${STORAGETOKEN}\" $STORAGEURL" else # without using debug this script will just print your containers to stdout CURL="curl -s -H" f_grabauthtoken f_list fi } # Grab API Authentication Token, and Storage URL function f_grabauthtoken() { # Establish initial authentication ${CURL} "X-Auth-User: ${USER}" -H "X-Auth-Key: ${APIKEY}" https://auth.api.rackspacecloud.com/v1.0 2>&1 | grep Storage | awk -F'<' '{print $2}' # Parse storage token to be used as an authentication key STORAGETOKEN=`curl -v -H "X-Auth-User: ${USER}" -H "X-Auth-Key: ${APIKEY}" https://auth.api.rackspacecloud.com/v1.0 2>&1 | \ grep Storage | awk -F'<' '{print $2}' | grep Token | awk '{print $2}' | tr -d '\r'` # Parse the storage URL which will be the location of your containers STORAGEURL=`curl -v -H "X-Auth-User: ${USER}" -H "X-Auth-Key: ${APIKEY}" https://auth.api.rackspacecloud.com/v1.0 2>&1 | \ grep Storage | awk -F'<' '{print $2}' | grep Url | awk '{print $2}'` } # List Containers function f_list() { if [ -n ${STORAGETOKEN} ]; then # This sends the API your token and storage url # the API will return your container listing ${CURL} "X-Auth-Token: ${STORAGETOKEN}" $STORAGEURL else echo "Please authenticate first" fi } # Usage instructions function f_verify() { # basic usage instructions # to enable debugging use: ./cf-list.sh <username> <apikey> 1 if [ -z "${USER}" ] || [ -z "${APIKEY}" ]; then echo "Usage: ./cf-list.sh <username> <apikey>" else f_exec fi } f_verify
Download it here: http://www.timgalyean.com/cf-list.sh
Hey man,
Script looks pretty good. Would it be too exposing for rackspace if you showed us what kind of output you get, or what kinds of practical use you could put it to?
I only have one minor suggestion for the script.
“if [ "${DEBUG}" = "1" ]; then”
Since you are using an integer as the 3rd argument, you could use -eq instead of “=”, like:
“if [ "${DEBUG}" -eq "1" ]; then”
This wouldn’t give you any different or better results or anything. Just nit-picking
You could also do something like we did with the old managed-installer.sh script, where you would not even provide a 3rd-argument debug mode, but rather before you run the script, just run “export DEBUG=1″ to enter debugging mode, and “unset DEBUG” to return to normal.
Cheers
Ryan,
Thanks for the feedback. You’re questions made me realize that I had left out some crucial information so I decided to go ahead and write another article describing what CloudFiles is, how it works and what it can be used for. You can find the article here:
http://www.timgalyean.com/2010/09/what-is-cloudfiles/
As for the bash edits – Good point. I just might have to do some updates today