One place for hosting & domains

      Storage

      Use Cases for Linode Object Storage


      Updated by Linode

      Contributed by

      Linode

      What is Object Storage?

      Object Storage is a method of storing data that differs in a number of ways from Block Storage. Block Storage splits files into small blocks of data. Minimal file metadata is stored alongside this data and, in general, descriptive metadata must be stored in a separate file or database. In order to use a Block Storage volume it must be attached to a host server, where it acts like a hard drive.

      In contrast, Object Storage stores data, called objects, in containers, called buckets, and each object is given a unique identifier with which it is accessed. In this way, the physical location of the object does not need to be known. The objects are stored alongside rich, configurable metadata that can be used to describe any number of arbitrary properties about the object. Each object has its own URL, so accessing the data is often as simple as issuing an HTTP request, either by visiting the object in a browser or retrieving it through the command line.

      Benefits and Limitations

      Object Storage scales easily because all the objects are stored in a flat, scalable name space. Object Storage does not require a host server in order to be used, meaning many different clients can read from it or write to it.

      With that said, there are limitations to Object Storage. Objects in Object Storage cannot be modified at the block level, as with Block Storage, and must be rewritten in their entirety every time a change is made. This makes any scenario with many successive read/write operations – such as the needs of databases or transactional data – a poor choice for Object Storage. Additionally, Object Storage traffic runs over HTTP, so it does not benefit from the I/O speeds of a mounted Block Storage volume. As a rule of thumb, Object Storage shines when files do not need to be updated frequently.

      Below are some of the more popular use cases for Object Storage.

      Use Cases

      Static Site Hosting

      Because Object Storage buckets provide HTTP access to objects, it’s easy to set up a bucket to serve static websites. A static website is a website that does not require a server-side processing language like PHP to render content. And because a static site does not require each page to be processed with every request, they are usually very quick to load. For more information on setting up a static site on Object Storage, read our Host a Static Site on Linode Object Storage guide. For more on static site generators, visit our How to Choose a Static Site Generator guide.

      Website Files

      If you don’t want to host your entire site on Object Storage (for example: you plan to use a CMS like WordPress), you can still choose to host some of your site’s assets, like images and downloads, with Object Storage. This will save disk space on your server and can help reduce your costs.

      Software Storage and Downloads

      Similar to hosting website files, hosting software applications on Object Storage is a great use case for developers looking to give quick access to their products. Simply upload the file to a bucket and share its URL.

      Unstructured Data

      Unstructured data is any data that does not fit into a traditional database. Object Storage excels at storing large amounts of unstructured data. With the ability to configure custom metadata for each piece of unstructured data, it is easy to extrapolate useful information from each object and to retrieve objects with similar metadata. Examples of unstructured data include images, video, audio, documents, and Big Data.

      Images, Video, Audio, and Documents

      Multimedia assets like images, videos, audio files, and documents are a perfect match for Object Storage. In general these types of files do not change frequently, so there is no need to store them on Block Storage volumes. Because each file has its own URL, streaming the content of these files or embedding them in another program or website is simple and does not require the use of a server.

      Big Data

      Big Data typically describes data sets that are so large and so diverse that it takes specialized tooling to analyze them. In many cases the data that comprises Big Data is considered unstructured and does not fit neatly into a database, making it a great candidate for Object Storage.

      Artifact Storage

      As more and more of the development life cycle becomes automated and tested, more and more artifacts are generated in the process. Object Storage is a great solution for developers looking to store these artifacts, such as the bulk collection of logs. Sharing stored artifacts is as simple as sharing a URL. And if you’d rather your artifacts stay private, you can distribute an access key.

      Cold Storage

      Object Storage is, in the majority of cases, significantly cheaper than Block Storage. While Object Storage can incur a cost when retrieving data, the cost benefit for infrequently accessed data can provide you with an overall cost reduction when compared to similar methods.

      Similarly, Object Storage has benefits over tape storage. Tape storage is frequently used for archival purposes, but the read times that come with tape storage are many times more than what you’ll find with Object Storage. Special considerations have to be made when transferring tape drive data, such as the ability to ship drives safely across long distances. With Object Storage, this data is available through HTTP from anywhere in the world.

      Note

      The outbound data transfer for Linode Object Storage is part of your Linode account’s total transfer pool, which will reduce or completely eliminate transfer costs for Object Storage if you are also running Linode instances. If you expend your allotted transfer pool, you will be billed at a rate of $0.02 per GB for outbound transfers.

      Backups

      Databases and other critical data can be backed up to Object Storage with little effort using a command line client for easier automation. Objects within Object Storage are normally replicated three times, providing resiliency should an error occur with the underlying hardware. Additionally, buckets can be versioned so you never lose access to older backups.

      Private File Storage

      Objects can be made private and only accessible with a key. By default, all new objects in a bucket are set to private, so they are inaccessible by normal HTTP requests (though it’s easy to set public permissions on objects if you’d like). This makes it easy to store secure data.

      Next Steps

      If you’re curious about how to use Object Storage, you can read our guide on How to Use Linode Object Storage for detailed instructions on creating buckets and uploading objects. Read our Host a Static Site using Linode Object Storage to get started with hosting your own site on Object Storage.

      Find answers, ask questions, and help others.

      This guide is published under a CC BY-ND 4.0 license.



      Source link

      Use Cases for Block Storage


      Updated by Linode

      Contributed by

      Linode

      What is Block Storage

      Block Storage is a type of persistent cloud data storage that is similar to a traditional block device, like the hard drive in a PC. With Block Storage, your data is divided into blocks, which are the small, discrete units that Block Storage can read from and write to. These blocks are assigned unique identifiers, but these are generally not human-readable, so a filesystem is usually installed which maps your files to the underlying blocks they correspond to. This relationship is also analogous to your PC’s filesystem and hard drive.

      A Block Storage Volume houses these blocks of data. Volumes can be attached to a cloud computing instance, which makes its data and filesystem available to the instance. If your instance is running Linux, then mounting a Volume’s filesystem is just like mounting any other filesystem.

      Volumes are stored separately from your cloud instances, but inside the same data center, and they are attached via the data center’s private networking. A Volume can be detached from a cloud instance and its data will persist, even if the cloud instance is deleted. The Volume can also be re-attached to a different instance (though only one attachment at a time is possible). Volumes can also be increased in size at any time, independent of an instance’s built-in storage.

      Benefits and Limitations

      A Block Storage Volume augments the raw storage capacity of a cloud instance, which can be useful if your storage needs are greater than your computing demands. Because a Volume is scalable, it can adapt as your data grows in size. Additionally, all data stored with Linode Block Storage is replicated three times, so your Volumes are highly available and fault tolerant.

      Note

      While the health and uptime of Linode Block Storage is closely monitored by Linode Support, we still recommend making separate backups of your Volumes.

      Because Volumes are directly connected to an instance, their I/O speeds are much faster than those of an alternative storage solution like Object Storage. As well, the nature of Block Storage allows you to read and write small parts of your data, which means that you can incrementally update your files. This is in contrast to Object Storage, which requires a full re-upload of a file to update it.

      Some aspects of Block Storage lead to natural limitations. In particular, a Volume needs to be attached to a cloud instance for its data to be accessible. In comparison, a file stored in Object Storage can be downloaded by any internet connected client at any time.

      Below you will find some of the more popular use cases for Block Storage.

      Use Cases

      Databases

      Databases require quick read/write operations, and Block Storage Volumes are mounted directly to a cloud instance’s filesystem, so there is a minimal delay in writing and retrieving data. Growing businesses often have increasing data storage demands, so it’s also important for a database to be scalable. A Volume that stores your database can be resized to meet your storage needs.

      Persistent Storage

      Block Storage Volumes can be detached from a cloud instance just as easily as they are attached, meaning that it’s possible to create hot-swappable drives with Block Storage. This is useful if you need to perform the same kind of tasks across a fleet of instances with the same data.

      Caution

      While the Block Storage service has full support for hot swapping, it is important to follow the detachment instructions outlined in our Using Block Storage guide. If a Volume is not safely detached, there is a risk of data loss for the Volume.

      Container Storage

      Containers, like those created with Docker or inside Kubernetes Pods, can benefit from having some type of persistent storage. This helps to keep a container’s size down and makes it easy to maintain data outside of the normal lifecycle of the container or Pod.

      If you are using Docker, you can use the Docker Volume Driver for Linode to create a Docker volume from a Block Storage Volume. Similarly, if you are using Kubernetes you can use the Container Storage Interface (CSI) Driver for Linode Block Storage to create a Persistent Volume Claim that’s backed by a Block Storage Volume.

      Running Cloud Software

      In a climate where ownership over one’s data is an important need for many individuals and organizations, hosting your own cloud software is a great use case for Block Storage. Create and mount a Block Storage Volume, install an application like OwnCloud, and point its data folder to a location on your Volume. If you ever run out of space on your Volume you can always increase its size.

      Storage for Media Library Applications

      There are a few media library applications, most notably Plex, that offer media streaming functionality to internet enabled devices. The media libraries these applications serve can quickly grow in size, depending on the number of movie and audio files they contain. Using a Block Storage Volume can provide you with storage capable of growing with the needs of your library.

      Note

      Ephemeral Storage

      Various stages of the software development lifecycle can create large amounts of temporary data, such as buffers, builds, and cache and session data. While this data might only exist for a short period of time, it requires and utilizes storage space. Creating a Block Storage Volume just for ephemeral data is a good use case for times when the storage supplied with your instance is not enough, or for when you need extra space for a short period of time.

      Data Backups

      Having backups of your data is always a good idea, and Block Storage Volumes make for scalable and quickly accessible backup mediums. Store anything that you might need to quickly transfer to another instance, or anything that you might need at a moment’s notice.

      Boot Disks

      You can boot from disk images installed to a Block Storage Volume. This provides a cost effective means of maintaining an image that can be attached to a new Linode. For example, you could save money by creating and removing on-demand Linode instances that boot from a Volume. As well, you can boot from a Volume to access and recover an instance whose normal operating system may not be running as expected.

      Note

      Linode provides a built-in Rescue Mode feature, but maintaining your own rescue Volume can allow you to include the recovery tools you prefer to use.

      Next Steps

      For more information on how to use Block Storage, consult our How to Use Block Storage with Your Linode guide.

      Find answers, ask questions, and help others.

      This guide is published under a CC BY-ND 4.0 license.



      Source link

      Host a Static Site using Linode Object Storage


      Updated by Linode

      Contributed by

      Linode

      Note

      Object Storage is currently in a closed early access Beta, and you may not have access to Object Storage through the Cloud Manager or other tools. To gain access to the Early Access Program (EAP), open up a Customer Support ticket noting that you’d like to be included in the program, or e-mail [email protected] – beta access is completely free.

      Additionally, because Object Storage is in Beta, there may be breaking changes to how you access and manage Object Storage. This guide will be updated to reflect these changes if and when they occur.

      Why Host a Static Site on Object Storage?

      Static site generators are a popular solution for creating simple, fast, flexible, and attractive websites that are easy to update. You can contribute new pages and content to a static site in two steps:

      1. First, write the content for your site’s new page using Markdown, an easy-to-learn and light-weight markup language.

      2. Then, tell your static site generator to compile your Markdown (along with other relevant assets, like CSS styling, images, and JavaScript) into static HTML files.

      The second compilation step only needs to happen once for each time that you update your content. This is in contrast with a dynamic website framework like WordPress or Drupal, which will reference a relational database and compile your HTML every time a visitor loads your site.

      Benefits of Hosting on Object Storage

      Traditionally, these static HTML files would be served by a web server (like NGINX or Apache) running on a Linode. Using Object Storage to host your static site files means you do not have to worry about maintaining your site’s infrastructure. It is no longer necessary to perform typical server maintenance tasks, like software upgrades, web server configuration, and security upkeep.

      Object Storage provides an HTTP REST gateway to objects, which means a unique URL over HTTP is available for every object. Once your static site is built, making it available publicly over the Internet is as easy uploading files to an Object Storage bucket.

      Object Storage Hosting Workflow

      At a high-level, the required steps to host a static site using Object Storage are:

      1. Install the static site generator of your choice to your local computer.

      2. Create the desired content and build the site (using your static site generator).

      3. Upload the static files to your Object Storage bucket to make the content publicly available over the Internet.

      This guide will use Hugo to demonstrate how to create a static site and host it on Linode Object Storage. However, there are many other static site generators to choose from–Jekyll and Gatsby are popular choices, and the general steps outlined in this guide could be adapted to them. For more information on choosing a static site generator, see the How to Choose a Static Site Generator guide.

      Before You Begin

      1. Read the How to Use Linode Object Storage guide to familiarize yourself with Object Storage on Linode. Specifically, be sure that you have:

        • Created your Object Storage access and secret keys.
        • Installed and configure the s3cmd tool.
      2. Install and configure Git on your local computer.

      Install the Hugo Static Site Generator

      Hugo is written in Go and is known for being extremely fast to compile sites, even very large ones. It is well-supported, well-documented, and has an active community. Some useful Hugo features include shortcodes, which are an easy way to include predefined templates inside of your Markdown, and built-in LiveReload web server, which allows you to preview your site changes locally as you make them.

      1. Install Hugo on your computer:

        macOS:

        Linux/Ubuntu:

        • Determine your Linux kernel’s architecture:

          uname -r
          

          Your output will resemble the following:

            
          4.9.0-8-amd64
          
          
        • Navigate to Hugo’s GitHub releases page and download the appropriate version for your platform. This example command downloads version 0.55, but a newer release may be available:

          wget https://github.com/gohugoio/hugo/releases/download/v0.55.0/hugo_0.55.0_Linux-64bit.deb
          
        • Install the package using dpkg:

          sudo dpkg -i hugo*.deb
          
      2. Verify that Hugo is installed. You should see output indicating your installed Hugo’s version number:

        hugo version
        

      Create a Hugo Site

      In this section, you will use the Hugo CLI (command line interface) to create your Hugo site, initialize a Hugo theme, and add content to your site. Hugo’s CLI provides several useful commands for common tasks needed to build, configure, and interact with your Hugo site.

      1. Create a new Hugo site on your local computer. This command will create a folder named example-site and scaffold Hugo’s directory structure inside it:

        hugo new site example-site
        
      2. Move into your Hugo site’s root directory:

        cd example-site
        

        Note

        All commands in this section of the guide should be issued from your site’s root directory.

      3. You will use Git to add a theme to your Hugo site’s directory. Initialize your Hugo site’s directory as a Git repository:

        git init
        
      4. Install the Ananke theme as a submodule of your Hugo site’s Git repository. Git submodules allow one Git repository to be stored as a subdirectory of another Git repository, while still being able to maintain each repository’s version control information separately. The Ananke theme’s repository will be located in the ~/example-site/themes/ananke directory of your Hugo site.

        git submodule add https://github.com/budparr/gohugo-theme-ananke.git themes/ananke
        

        Note

        Hugo has many available themes that can be installed as a submodule of your Hugo site’s directory.
      5. Add the theme to your Hugo site’s configuration file. The configuration file (config.toml) is located at the root of your Hugo site’s directory.

        echo 'theme = "ananke"' >> config.toml
        
      6. Create a new content file for your site. This command will generate a Markdown file with an auto-populated date and title:

        hugo new posts/my-first-post.md
        
      7. You should see a similar output. Note that the file is located in the content/posts/ directory of your Hugo site:

          
        /home/username/example-site/content/posts/my-first-post.md created
        
        
      8. Open the Markdown file in the text editor of your choice to begin modifying its content; you can copy and paste the example snippet into your file, which contains an updated front matter section at the top and some example Markdown body text.

        Set your desired value for title. Then, set the draft state to false and add your content below the --- in Markdown syntax, if desired:

        /home/username/example-site/content/posts/my-first-post.md
         1
         2
         3
         4
         5
         6
         7
         8
         9
        10
        11
        12
        13
        14
        
        ---
        title: "My First Post"
        date: 2019-04-11T11:25:11-04:00
        draft: false
        ---
        
        # Host a Static Site on Linode Object Storage
        
        There are many benefits to using a static site generator. Here is a list of a few of them:
        
        - Run your own website without having to manage a Linode.
        - You don't need to worry about running a web server like Apache or NGINX.
        - Static website performance is typically very fast.
        - Use Git to version control your static website's content.


        About front matter

        Front matter is a collection of metadata about your content, and it is embedded at the top of your file within opening and closing --- delimiters.

        Front matter is a powerful Hugo feature that provides a mechanism for passing data that is attached to a specific piece of content to Hugo’s rendering engine. Hugo accepts front matter in TOML, YAML, and JSON formats. In the example snippet, there is YAML front matter for the title, date, and draft state of the Markdown file. These variables will be referenced and displayed by your Hugo theme.

      9. Once you have added your content, you can preview your changes by building and serving the site using Hugo’s built-in webserver:

        hugo server
        
      10. You will see a similar output:

          
        &nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp| EN
        +------------------+----+
          Pages&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp| 11
          Paginator pages&nbsp&nbsp&nbsp&nbsp|  0
          Non-page files&nbsp&nbsp&nbsp&nbsp&nbsp|  0
          Static files&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp|  3
          Processed images&nbsp&nbsp&nbsp|  0
          Aliases&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp|  1
          Sitemaps&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp|  1
          Cleaned&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp&nbsp|  0
        
        Total in 7 ms
        Watching for changes in /home/username/example-site/{content,data,layouts,static,themes}
        Watching for config changes in /home/username/example-site/config.toml
        Serving pages from memory
        Running in Fast Render Mode. For full rebuilds on change: hugo server --disableFastRender
        Web Server is available at http://localhost:1313/ (bind address 127.0.0.1)
        Press Ctrl+C to stop
        
        
      11. The output will provide a URL to preview your site. Copy and paste the URL into a browser to access the site. In the above example Hugo’s web server URL is http://localhost:1313/.

      12. When you are happy with your site’s content you can build your site:

        hugo -v
        

        Hugo will generate your site’s static HTML files and store them in a public directory that it will create inside your project. The static files that are generated by Hugo are the files that you will upload to your Object Storage bucket to make your site accessible via the Internet.

      13. View the contents of your site’s public directory:

        ls public
        

        Your output should resemble the following example. When you built the site, the Markdown file you created and edited in steps 6 and 7 was used to generate its corresponding static HTML file in the public/posts/my-first-post/index.html directory.

          
          404.html    categories  dist        images      index.html  index.xml   posts       sitemap.xml tags
            
        


        Track your Static Site Files with Git

        It’s not necessary to version control your site files in order to host them on Object Storage, but we still recommended that you do so:

        1. Display the state of your current working directory (root of your Hugo site):

          git status
          
        2. Stage all your files to be committed:

          git add -A
          
        3. Commit all your changes and add a meaningful commit message:

          git commit -m 'Add my first post.'
          

        Once you have used Git to track your local Hugo site files, you can easily push them to a remote Git repository, like GitHub or GitLab. Storing your static site files on a remote Git repository opens up many possibilities for collaboration and automating your static site’s deployment to Linode Object Storage. To learn more about Git, see the Getting Started with Git guide.

      Upload your Static Site to Linode Object Storage

      Before proceeding with this section ensure that you have already created your Object Storage access and secret keys and have installed the s3cmd tool.

      1. Create a new Object Storage bucket; prepend s3:// to the beginning of the bucket’s name:

        s3cmd mb s3://my-bucket
        

        Note

        Buckets names must be unique within the Object Storage cluster. You might find the bucket name my-bucket is already in use by another Linode customer, in which case you will need to choose a new bucket name.

      2. Initialize your Object Storage bucket as a website. You must tell your bucket which files to serve as the index page and the error page for your static site. This is done with the --ws-index and --ws-error options:

        s3cmd ws-create --ws-index=index.html --ws-error=404.html s3://my-bucket
        

        In our Hugo example, the site’s index file is index.html and the error file is 404.html. Whenever a user visits your static site’s URL, the Object Storage service will serve the index.html page. If a site visitor tries to access an invalid path, they will be presented with the 404.html page.

      3. The command will return the following message:

          
            Bucket 's3://my-bucket/': website configuration created.
              
        
      4. Display information about your Object Storage’s website configuration to obtain your site’s URL:

        s3cmd ws-info s3://my-bucket
        
      5. You should see a similar output. Be sure to take note of your Object Storage bucket’s URL:

          
              Bucket s3://my-bucket/: Website configuration
        Website endpoint: http://website-us-east-1.linodeobjects.com/
        Index document:   index.html
        Error document:   404.html
            
        

        Note

        The Linode Object Storage early access Beta provides SSL enabled by default. This means you can access your Object Storage bucket using https, as well.

      6. Use s3cmd’s sync command to upload the contents of your static site’s public directory to your Object Storage bucket. This step will make your site available publicly on the Internet. Ensure you are in your site’s root directory on your computer (e.g. /home/username/example-site):

        s3cmd --no-mime-magic --acl-public --delete-removed --delete-after sync public/ s3://my-bucket
        
        Option                         Description
        no-mime-magicTells Object Storage not to use file signatures when guessing the object’s MIME-type.
        acl-publicSets the access level control of the objects to public.
        delete-removedDeletes any destination objects with no corresponding source file.
        delete-afterDeletes destination files that are no longer found at the source after all files are uploaded to the bucket.
      7. Use a browser to navigate to your Object Storage bucket’s URL to view your Hugo site:

        Hugo Index Page

        Note

        It may take a minute or two after your s3cmd sync completes for the page to appear at your bucket’s website URL.

      8. If needed, you can continue to update your static site locally and upload any changes using s3cmd’s sync command from step 3 of this section.

      (Optional) Next Steps

      After uploading your static site to Linode Object Storage, you may want to use a custom domain for your site. To do this, you can add a CNAME entry to your domain’s DNS records that aliases it to your Object Storage bucket’s website URL. To learn about managing DNS records on Linode, see the DNS Manager and DNS Records: An Introduction guides.

      As noted before, it’s possible to trigger automatic deployments to the Object Storage service when you push new content updates to GitHub or GitLab. This is done by leveraging a CI/CD (continuous integration/continuous delivery) tool like Travis CI. Essentially, you would build your Hugo site within the Travis environment and then run the s3cmd sync command from it to your bucket.

      More Information

      You may wish to consult the following resources for additional information on this topic. While these are provided in the hope that they will be useful, please note that we cannot vouch for the accuracy or timeliness of externally hosted materials.

      Find answers, ask questions, and help others.

      This guide is published under a CC BY-ND 4.0 license.



      Source link