Amazon Web Services Tips

  


[an error occurred while processing this directive]

No Title

[an error occurred while processing this directive]

Reference: Cisco: Internetworking Basics

Contents

Introduction

This guide describes how to mount an Amazon S3 bucket as a virtual drive to a local file system on Linux by using s3fs and FUSE.

The example provided in this guide will mount an S3 bucket named idevelopment-software to /mnt/s3/idevelopment-software on an EC2 instance running CentOS 6.3 (x86_64).

Requirements

Perform the following prerequisites on the target machine as root.

  1. Install the following prerequisite packages.


    # yum -y install gcc # yum -y install libstdc++-devel # yum -y install gcc-c++ # yum -y install curl-devel # yum -y install libxml2-devel # yum -y install openssl-devel # yum -y install mailcap # yum -y install make # yum -y install svn

  2. Download, compile, and install the FUSE (Filesystem in Userspace) module.

    Note: s3fs requires FUSE version 2.8.4 when running CentOS 5. Using a version of FUSE higher than version 2.8.4 on CentOS 5 will result in the following error when attempting to mount the S3 bucket:


    # s3fs idevelopment-software /mnt/s3/idevelopment-software /bin/mount: unrecognized option `--no-canonicalize'

    CentOS 6 works with the latest FUSE version which at the time of this writing is release 2.9.1.

    CentOS 6


    # yum remove fuse fuse* fuse-devel # cd /usr/local/src # wget http://sourceforge.net/projects/fuse/files/fuse-2.X/2.9.1/fuse-2.9.1.tar.gz # tar -xzvf fuse-2.9.1.tar.gz # cd fuse-2.9.1 # ./configure --prefix=/usr # make # make install # ldconfig # export PKG_CONFIG_PATH=/usr/lib/pkgconfig:/usr/lib64/pkgconfig/ # pkg-config --modversion fuse 2.9.1 # ldconfig -v | grep fuse libfuse.so.2 -> libfuse.so.2.9.1

    CentOS 5


    # yum remove fuse fuse* fuse-devel # cd /usr/local/src # wget http://sourceforge.net/projects/fuse/files/fuse-2.X/2.8.4/fuse-2.8.4.tar.gz # tar -xzvf fuse-2.8.4.tar.gz # cd fuse-2.8.4 # ./configure --prefix=/usr # make # make install # ldconfig # export PKG_CONFIG_PATH=/usr/lib/pkgconfig:/usr/lib64/pkgconfig/ # pkg-config --modversion fuse 2.8.4 # ldconfig -v | grep fuse libfuse.so.2 -> libfuse.so.2.8.4

    Manually load the FUSE module for the first time.


    # modprobe fuse

  3. Download, compile, and install the latest version of s3fs.


    # cd /usr/local/src # svn checkout http://s3fs.googlecode.com/svn/trunk/ s3fs # cd s3fs # autoreconf --install # ./configure --prefix=/usr # make # make install # ldconfig

    Verify the s3fs installation.


    # s3fs --version Amazon Simple Storage Service File System 1.61 Copyright (C) 2010 Randy Rizun License GPL2: GNU GPL version 2 This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.

  4. Create an s3fs password file for storing your AWS Access Key ID and Secret Access Key.

    The default location for the s3fs password file can be created:

    The s3fs password file has the following format. Use this format if you have only one set of credentials. Specify your AWS Access Key ID and Secret Access Key separated by a colon with no spaces between.


    AccessKeyId:SecretAccessKey

    If have more than one set of credentials, then you can have default credentials as specified above, but the format includes the bucket name and the default S3 credentials, all separated by a colon with no spaces between.


    my-s3-bucket:AccessKeyId:SecretAccessKey

    where my-s3-bucket is the S3 bucket name to mount, AccessKeyId is the Access Key ID, and SecretAccessKey is the Secret Access Key.

    The AWS Access Key ID and Secret Access Key serve the purpose of ID and Password to access Amazon S3. Navigate to Security Credentials, click on the Access Keys tab under Access Credentials to create or view your Access Key ID and Secret Access Key.

    Note: The AWS Access Key ID and Secret Access Key displayed below are not my actual S3 security credentials. It is nothing more than a set of random characters and is only being shown to display example output. It is not valid. However, if you have a lot of time on your hands, give it a try!


    # touch /root/.passwd-s3fs # chmod 600 /root/.passwd-s3fs # vi /root/.passwd-s3fs idevelopment-software:B9W1IF9E1H0CQ17V1LQW:U1DfhLaQ1bhQ1B7/DcNai2C9qmaI8wpqDJnn3Qas

Mount S3 Bucket

Create a mount point directory for the S3 bucket and use the s3fs command to mount it.

  1. Create a directory for mounting the S3 bucket.


    # mkdir -p /mnt/s3/idevelopment-software

  2. Mount the S3 bucket using the correct permissions and any other options.


    # s3fs -o allow_other -o use_cache=/tmp idevelopment-software /mnt/s3/idevelopment-software

  3. Verify the s3fs mounted file system.


    # grep s3fs /etc/mtab s3fs /mnt/s3/idevelopment-software fuse.s3fs rw,nosuid,nodev,allow_other 0 0 # df -Th /mnt/s3/idevelopment-software Filesystem Type Size Used Avail Use% Mounted on s3fs fuse.s3fs 256T 0 256T 0% /mnt/s3/idevelopment-software

  4. Manually unmount the virtual drive using the umount command:


    # umount /mnt/s3/idevelopment-software

  5. Automatically mount the S3 bucket when the server boots by adding an entry to /etc/fstab using the following syntax:


    s3fs#bucketname /mnt/mount_folder fuse option1,option2 0 0

    For example:


    s3fs#idevelopment-software /mnt/s3/idevelopment-software fuse allow_other,use_cache=/tmp 0 0

s3fs Limitations

About the Author

Jeffrey Hunter is an Oracle Certified Professional, Java Development Certified Professional, Author, and an Oracle ACE. Jeff currently works as a Senior Database Administrator for The DBA Zone, Inc. located in Pittsburgh, Pennsylvania. His work includes advanced performance tuning, Java and PL/SQL programming, developing high availability solutions, capacity planning, database security, and physical / logical database design in a UNIX / Linux server environment. Jeff's other interests include mathematical encryption theory, tutoring advanced mathematics, programming language processors (compilers and interpreters) in Java and C, LDAP, writing web-based database administration tools, and of course Linux. He has been a Sr. Database Administrator and Software Engineer for over 20 years and maintains his own website site at: http://www.iDevelopment.info. Jeff graduated from Stanislaus State University in Turlock, California, with a Bachelor's degree in Computer Science and Mathematics.



Copyright (c) 1998-2017 Jeffrey M. Hunter. All rights reserved.

All articles, scripts and material located at the Internet address of http://www.idevelopment.info is the copyright of Jeffrey M. Hunter and is protected under copyright laws of the United States. This document may not be hosted on any other site without my express, prior, written permission. Application to host any of the material elsewhere can be made by contacting me at jhunter@idevelopment.info.

I have made every effort and taken great care in making sure that the material included on my web site is technically accurate, but I disclaim any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on it. I will in no case be liable for any monetary damages arising from such loss, damage or destruction.

Last modified on
Wednesday, 21-Nov-2012 18:21:07 EST
Page Count: 572902