Object Store Expansion

Document created by Sathish Narayanan Employee on Jul 28, 2015
Version 1Show Document
  • View in full screen mode

Overview

 

The purpose of this document is to provide the steps needed to expand an existing Object store cluster to meet the growing storage requirements. In the example used in this document, the steps are shown for adding two new nodes to an existing small cluster of 8 nodes. The two new nodes will be referred to as OS-9 & OS-10 in this document.

 

Preparing the new nodes

 

Installing the Ubuntu Operating System

Object Store functions as a layered application running on the following version of Ubuntu Server operating system Ubuntu 12.04 LTS 64-bit ("Precise Pangolin")

Follow the instructions in the Object Store Kickstart Guide https://control.akamai.com/dl/NetworkOperator/Object%20Store/3.1R1.0/Object_Store_Kickstart_Guide_R3.1R1.0.pdfto download the kickstart image from the Akamai Aura Support online portal. The kickstart image contains this version of Ubuntu. Do not download/install Ubuntu from http://releases.ubuntu.com/ as this could cause Object Store package incompatibilities.

The Kickstart image also includes the Object Store image so that all required software is installed on each node as part of kickstarting a new server.

 

Prerequisite-1:

Upload all json files to one of the nodes in the existing cluster under a specific folder (e.g. /home/os/json). All json updates need to be happen from this node and preferably from the context of this folder location.

 

Prerequisite-2:

 

Run "cat /etc/network/interfaces and make sure the routes to the NTP servers exist on OS nodes 1 through 10.

 

 

Note: We should also confirm that the routes have been added using "netstat –rn".

 

1. Fix NTP on the existing cluster if needed (NTP should be working for registration of new nodes to complete)

 

object-store list-ntp-servers

 

object-store set-ntp-servers ntp_servers.json

 

object-store list-ntp-servers

 

 

Note: We should only move on to step-2 after confirming that all nodes in the cluster have synchronized with the NTP source by using "ntpq –p" on each node and looking for a line starting with an "*".

 

2. Join OS-9 to the cluster

 

object-store list-nodes (on both the existing cluster & OS-9)

 

register-object-store-node 10.0.0.1 (from OS-9)

 

Note that the registration may sit there for a while (up to 10 minutes) before displaying any output.  This is because it retrieves the NTP configuration from the cluster and will not proceed until it has synchronized with the NTP source.

 

object-store list-nodes (on both the existing cluster & OS-9)

 

Note: Move to step-3 if OS-9 is successfully registered with the cluster

 

3. Join OS-10 to the cluster

 

object-store list-nodes (on both the existing cluster & OS-10)

 

register-object-store-node 10.0.0.1 (from OS-10)

 

Note that the registration may sit there for a while (up to 10 minutes) before displaying any output.  This is because it retrieves the NTP configuration from the cluster and will not proceed until it has synchronized with the NTP source.

 

object-store list-nodes (on both the existing cluster & OS-10)

 

Note: Move to step-4 if OS-10 is successfully registered with the cluster

 

4. Update ram-buffers

 

object-store list-ram-buffers

 

object-store update-ram-buffers ram_buffers.json

 

object-store list-ram-buffers

 

Note: Move to step-5 if the ram-buffers update is successful.

 

 

5. Update devices

 

object-store list-devices

 

object-store update-devices update_devices.json

 

object-store list-devices


Note:  The update-devices and update-ram-buffers can be executed from any node that is a member of the cluster.  The JSON files have the information for the devices that need to be added to (or removed from) the cluster, but they do not indicate where the processing occurs


6a.  Verificaiton (Registration process)


object-store list-nodes

{

  "nodes": [

    {

      "hostname": "os10.foo.bar.example.com",

      "ipaddress": "10.0.0.10"

    },

    {

      "hostname": "os9.foo.bar.example.com",

      "ipaddress": "10.0.0.9"

    },

    {

      "hostname": "os8.foo.bar.example.com",

      "ipaddress": "10.0.0.8"

    },

    {

      "hostname": "os5.foo.bar.example.com",

      "ipaddress": "10.0.0.5"

    },

    {

      "hostname": "os4.foo.bar.example.com",

      "ipaddress": "10.0.0.4"

    },

    {

      "hostname": "os7.foo.bar.example.com",

      "ipaddress": "10.0.0.7"

    },

    {

      "hostname": "os2.foo.bar.example.com",

      "ipaddress": "10.0.0.6"

    },

    {

      "hostname": "os1.foo.bar.example.com",

      "ipaddress": "10.0.0.1"

    },

    {

      "hostname": "os3.foo.bar.example.com",

      "ipaddress": "10.0.0.3"

    },

    {

      "hostname": "os2.foo.bar.example.com",

      "ipaddress": "10.0.0.2"

    }

  ]

}

 

6b.  Verificaiton (ram-buffers)

object-store list-ram-buffers

{

  "ram-buffers": [

    {

      "status": "in-cluster",

      "ipaddress": "10.0.0.10",

      "redundancy-group": "5"

    },

    {

      "status": "in-cluster",

      "ipaddress": "10.0.0.9",

      "redundancy-group": "5"

    },

    {

      "status": "in-cluster",

      "ipaddress": "10.0.0.8",

      "redundancy-group": "4"

    },

    {

      "status": "in-cluster",

      "ipaddress": "10.0.0.5",

      "redundancy-group": "1"

    },

    {

      "status": "in-cluster",

      "ipaddress": "10.0.0.4",

      "redundancy-group": "4"

    },

    {

      "status": "in-cluster",

      "ipaddress": "10.0.0.7",

      "redundancy-group": "3"

    },

    {

      "status": "in-cluster",

      "ipaddress": "10.0.0.6",

      "redundancy-group": "2"

    },

    {

      "status": "in-cluster",

      "ipaddress": "10.0.0.1",

      "redundancy-group": "1"

    },

    {

      "status": "in-cluster",

      "ipaddress": "10.0.0.3",

      "redundancy-group": "3"

    },

    {

      "status": "in-cluster",

      "ipaddress": "10.0.0.2",

      "redundancy-group": "2"

    }

  ]

}


 



Attachments

    Outcomes