Storage

Storage

About Storage

Storage plans allow an account to control where data related to their account is stored. This can be critical when compliance with regulations is required.

A storage plan has three main components:

  1. attachments: configuration for where to store attachments (binary data typically, like voicemails or faxes)
  2. connections: connection information to various third-party storage sites
  3. plan: a description of the storage plan(s) for the account.

Attachments

Rather than storing binary data like voicemails, received faxes, and call recordings, in the Kazoo databases, it can be convenient to store them in third-party storage services like Amazon S3, Google Drive, etc. This keeps the binary data in a place that an account or user maintains control over. Kazoo keeps a pointer to the location when it needs to fetch the binary data (such as when you call into your voicemail box).

The attachments object configures storage back-ends - for instance, if you want to store to S3, you’ll add your AWS secret and key and your S3 bucket information here.

Connections

Connections can be used to point to an alternative CouchDB instance for storing the JSON documents or attachments (for instance: putting CDRs in their own cluster). These can be specified in the storage plans.

Plans

Plans determine what to do with certain classes of databases:

  1. account: where to store account data
  2. modb: where to store temporal data, like CDRs or voicemails
  3. system: where to store system data, like prompts

Within the database classification, you can define things like the connection to use when reading/writing, what types of documents should be stored/retrieved, etc.

Enabling the storage endpoint

Crossbar must have the storage endpoint enabled first:

sup crossbar_maintenance start_module cb_storage

Schema

KeyDescriptionTypeDefaultRequiredSupport Level
^_Ignores CouchDB fields prefixed by underscores`boolean()integer()object()string()`
^pvt_Ignores Kazoo private fields prefixed by pvt_`boolean()integer()string()`
attachmentsDefines where and how to store attachments. Keys are 32-character identifiers to be used in storage plans#/definitions/storage.attachmentsfalse
connectionsDescribes alternative connections to use (such as alternative CouchDB instances#/definitions/storage.connectionsfalse
idID of the storage documentstring()false
planDescribes how to store documents depending on the database or document type#/definitions/storage.planfalse

storage.attachment.aws

schema for AWS attachment entry

KeyDescriptionTypeDefaultRequiredSupport Level
handlerWhat AWS service to usestring('s3')true
settings.bucketBucket name to store data tostring(6..63)true
settings.bucket_access_methodhow to access the host.`string(‘auto’‘vhost’‘path’)`
settings.bucket_after_hostuse bucket after host as part of urlboolean()false
settings.hostthe s3 host, leave empty for defaultstring(1..)false
settings.keyAWS Key to usestring(1..128)false
settings.portport to useinteger()false
settings.regionthe region where the bucket is locatedstring(1..)false
settings.schemescheme to use to access host`string(‘http’‘https’)`false
settings.secretAWS Secret to usestring(1..128)false
settings.tags_key_path.[]string()false
settings.tags_key_pathThe key path to an array of values in the document to use as S3 object tagsarray(string())false
settingsAWS API settingsobject()true

storage.attachment.azure

schema for azure attachment entry

KeyDescriptionTypeDefaultRequiredSupport Level
handlerWhat handler module to usestring('azure')true
settings.accountthe azure account namestring()true
settings.containerthe azure container where the files should be savedstring()true
settings.keythe azure api keystring()true
settingsSettings for the Azure accountobject()true

storage.attachment.dropbox

schema for dropbox attachment entry

KeyDescriptionTypeDefaultRequiredSupport Level
handlerWhat handler module to usestring('dropbox')true
settings.oauth_doc_idDoc ID in the system ‘auth’ databasestring(1..)true
settingsSettings for the Dropbox accountobject()true

storage.attachment.google_drive

schema for google drive attachment entry

KeyDescriptionTypeDefaultRequiredSupport Level
handlerWhat handler module to usestring('google_drive')true
settings.folder_idFolder ID in which to store the file, if anystring()false
settings.oauth_doc_idDoc ID in the system ‘auth’ databasestring(1..)true
settingsSettings for the Google Drive accountobject()true

storage.attachment.google_storage

schema for google storage attachment entry

KeyDescriptionTypeDefaultRequiredSupport Level
handlerWhat handler module to usestring('google_storage')true
settingsSettings for the Google Storage accountobject()true

storage.attachment.http

schema for HTTP(s) attachment entry

KeyDescriptionTypeDefaultRequiredSupport Level
handlerThe handler interface to usestring('http')true
settings.base64_encode_dataToggles whether to base64-encode the attachment databoolean()falsefalse
settings.send_multipartToggle whether to send multipart payload when storing attachment - will include metadata JSON if trueboolean()false
settings.urlThe base HTTP(s) URL to use when creating the requeststring(7..)true
settings.verbThe HTTP verb to use when sending the data`string(‘post’‘put’)`putfalse
settingsHTTP server settingsobject()true

storage.attachment.onedrive

schema for OneDrive attachment entry

KeyDescriptionTypeDefaultRequiredSupport Level
handlerWhat handler module to usestring('onedrive')true
settings.oauth_doc_idDoc ID in the system ‘auth’ databasestring(1..)true
settingsSettings for the OneDrive accountobject()true

storage.attachments

Defines where and how to store attachments. Keys are 32-character identifiers to be used in storage plans

KeyDescriptionTypeDefaultRequiredSupport Level
^[a-z0-9]{32}$.nameFriendly name for this configurationstring()false
^[a-z0-9]{32}$.settings.field_listlist of fields to compose destination urlarray([#/definitions/storage.attachments.field](#storageattachments.field))false
^[a-z0-9]{32}$.settings.field_separatortoplevel, field separator to compose destination urlstring()false
^[a-z0-9]{32}$.settings.folder_base_pathbase folder pathstring()false
^[a-z0-9]{32}$.settingsSettings all handlers implementobject()false
^[a-z0-9]{32}$Configuration for the supported storage backendsobject()false

storage.attachments.field

field used when composing destination url

KeyDescriptionTypeDefaultRequiredSupport Level

storage.connection.couchdb

schema for CouchDB connection entry

KeyDescriptionTypeDefaultRequiredSupport Level
driverstring('kazoo_couch')true
namestring()false
settings.connect_options.keepaliveboolean()false
settings.connect_optionsobject()false
settings.connect_timeoutinteger()false
settings.credentials.passwordinteger()true
settings.credentials.usernamestring()true
settings.credentialsobject()false
settings.ipstring()true
settings.max_pipeline_sizeinteger()false
settings.max_sessionsinteger()false
settings.pool.namestring()true
settings.pool.sizeinteger()true
settings.poolobject()false
settings.portinteger()true
settingsobject()true

storage.connections

Describes alternative connections to use (such as alternative CouchDB instances

KeyDescriptionTypeDefaultRequiredSupport Level
^([a-z,0-9]){32}$#/definitions/storage.connection.couchdbfalse
localobject()false

storage.plan

Describes how to store documents depending on the database or document type

KeyDescriptionTypeDefaultRequiredSupport Level
accountschema for database storage plan#/definitions/storage.plan.databasefalse
aggregateschema for database storage plan#/definitions/storage.plan.databasefalse
modbschema for database storage plan#/definitions/storage.plan.databasefalse
systemschema for database storage plan#/definitions/storage.plan.databasefalse

storage.plan.database

schema for database storage plan

KeyDescriptionTypeDefaultRequiredSupport Level
attachmentsDescribes what attachment types to store using this plan#/definitions/storage.plan.database.attachmentfalse
connectionWhich connection UUID to use when storing to this database typestring()false
database.create_optionsobject()false
database.names.[]string()false
database.namesList of database names to match (non-matching names won’t use this plan)array(string())false
databaseobject()false
rangefalse
regexfalse
types./[a-z_]+/schema for document type storage plan#/definitions/storage.plan.database.documentfalse
typesThe document types to store with this planobject()false

storage.plan.database.attachment

schema for attachment ref type storage plan

KeyDescriptionTypeDefaultRequiredSupport Level
handlerstring()false
params.folder_pathfolder pathstring()false
paramsobject()false
stubboolean()false

storage.plan.database.document

schema for document type storage plan

KeyDescriptionTypeDefaultRequiredSupport Level
attachmentsschema for attachment ref type storage plan#/definitions/storage.plan.database.attachmentfalse
connectionstring()false
rangefalse
regexfalse

Fetch

GET /v2/accounts/{ACCOUNT_ID}/storage

curl -v -X GET \
    -H "X-Auth-Token: {AUTH_TOKEN}" \
    http://{SERVER}:8000/v2/accounts/{ACCOUNT_ID}/storage
{
    "auth_token": "{AUTH_TOKEN}",
    "data": {
        "attachments": {
            "{UUID}": {
                "handler": "s3",
                "name": "S3 Storage",
                "settings": {
                    "bucket": "{S3_BUCKET}",
                    "key": "{AWS_ACCESS_KEY}",
                    "secret": "{AWS_SECRET_KEY}"
                }
            }
        },
        "id": "{ACCOUNT_ID}",
        "plan": {
            "modb": {
                "types": {
                    "mailbox_message": {
                        "attachments": {
                            "handler": "{UUID}"
                        }
                    }
                }
            }
        }
    },
    "request_id": "{REQUEST_ID}",
    "revision": "{REVISION}",
    "status": "success"
}

Create

PUT /v2/accounts/{ACCOUNT_ID}/storage

curl -v -X PUT \
    -H "X-Auth-Token: {AUTH_TOKEN}" \
    http://{SERVER}:8000/v2/accounts/{ACCOUNT_ID}/storage

For instance, setting up your HTTP server to receive new voicemails for the account:

{
  "data": {
    "attachments": {
      "{UUID}": {
        "handler": "http",
        "name": "My HTTP server",
        "settings": {
          "url": "http://my.http.server:37635/some_prefix",
          "verb": "post"
        }
      }
    },
    "plan": {
      "modb": {
        "types": {
          "mailbox_message": {
            "attachments": {
              "handler": "{UUID}"
            }
          }
        }
      }
    }
  }
}

Change

POST /v2/accounts/{ACCOUNT_ID}/storage

curl -v -X POST \
    -H "X-Auth-Token: {AUTH_TOKEN}" \
    http://{SERVER}:8000/v2/accounts/{ACCOUNT_ID}/storage

Patch

PATCH /v2/accounts/{ACCOUNT_ID}/storage

curl -v -X PATCH \
    -H "X-Auth-Token: {AUTH_TOKEN}" \
    http://{SERVER}:8000/v2/accounts/{ACCOUNT_ID}/storage

Remove

DELETE /v2/accounts/{ACCOUNT_ID}/storage

curl -v -X DELETE \
    -H "X-Auth-Token: {AUTH_TOKEN}" \
    http://{SERVER}:8000/v2/accounts/{ACCOUNT_ID}/storage

Fetch

GET /v2/accounts/{ACCOUNT_ID}/storage/plans

curl -v -X GET \
    -H "X-Auth-Token: {AUTH_TOKEN}" \
    http://{SERVER}:8000/v2/accounts/{ACCOUNT_ID}/storage/plans

Create

PUT /v2/accounts/{ACCOUNT_ID}/storage/plans

curl -v -X PUT \
    -H "X-Auth-Token: {AUTH_TOKEN}" \
    http://{SERVER}:8000/v2/accounts/{ACCOUNT_ID}/storage/plans

Fetch

GET /v2/accounts/{ACCOUNT_ID}/storage/plans/{STORAGE_PLAN_ID}

curl -v -X GET \
    -H "X-Auth-Token: {AUTH_TOKEN}" \
    http://{SERVER}:8000/v2/accounts/{ACCOUNT_ID}/storage/plans/{STORAGE_PLAN_ID}

Change

POST /v2/accounts/{ACCOUNT_ID}/storage/plans/{STORAGE_PLAN_ID}

curl -v -X POST \
    -H "X-Auth-Token: {AUTH_TOKEN}" \
    http://{SERVER}:8000/v2/accounts/{ACCOUNT_ID}/storage/plans/{STORAGE_PLAN_ID}

Patch

PATCH /v2/accounts/{ACCOUNT_ID}/storage/plans/{STORAGE_PLAN_ID}

curl -v -X PATCH \
    -H "X-Auth-Token: {AUTH_TOKEN}" \
    http://{SERVER}:8000/v2/accounts/{ACCOUNT_ID}/storage/plans/{STORAGE_PLAN_ID}

Remove

DELETE /v2/accounts/{ACCOUNT_ID}/storage/plans/{STORAGE_PLAN_ID}

curl -v -X DELETE \
    -H "X-Auth-Token: {AUTH_TOKEN}" \
    http://{SERVER}:8000/v2/accounts/{ACCOUNT_ID}/storage/plans/{STORAGE_PLAN_ID}

Skipping attachment settings validation

When a storage plan is PUT/POSTed to Crossbar, KAZOO will attempt to use the attachments’ settings and store a small text file to verify that files can be stored remotely. KAZOO will then issue a GET request to read the file back to test retrieval.

For “dumb” storage backends this is typically a non-issue as storing/retrieving files is what the backend does!

For “smart” backends, where a custom handler (like an HTTP web app) is accepting the files, adding code to handle this test file’s storage/retrieval could place an unnecessary burden on the backend or be redundant after the first test if using the same destination for all accounts. As such, a request parameter can be included to skip this portion of the validation:

curl -v -X PUT \
    -H "X-Auth-Token: {AUTH_TOKEN}" \
    -H "Content-Type: application/json" \
    -d '{"data":{...}}' \
  http://{SERVER}:8000/v2/accounts/{ACCOUNT_ID}/storage?validate_settings=false

!!! danger If the storage backend is unable to process the storage request, you could lose the data attempting to be stored.

Enabling This Feature

By default, Kazoo will not allow clients to skip settings validation. Clients that include the validate_settings request parameter on these systems will receive a 400 validation error indicating attachment storage settings must be tested.

Sysadmins can allow clients by setting a system_config flag: sup kzs_plan allow_validation_overrides

Disabling it later is similar: sup kzs_plan disallow_validation_overrides

URL formatting

It is possible to craft the URLs used by the handler based on the JSON document and attachment being saved by specifying a field_list array of objects that will help KAZOO map values to the generated URL:

{UUID}:{
    "handler":"{HANDLER}",
    "settings":{
        "field_list":[
            {"arg":"account_id"}
            ,{"arg":"id"}
            ,{"arg":"attachment"}
        ],
        "url":"http://base.your.domain/"
    }
}

In this case (the default for the HTTP handler) the URL provided in the handler’s settings will be appended with /{ACCOUNT_ID}/{DOC_ID}/{ATTACHMENT_NAME}.

Field Options

list of fields to compose destination url

KeyDescriptionTypeDefaultRequiredSupport Level
arga argument passed to the handler`string(‘account_id’‘db’‘id’‘attachment’)`
consta constant value added to the stringstring()false
fielda field from the metadata documentstring()false
groupgroup the inner fields definitions with an empty separatorarray()false

Examples

Given a base URL of http://my_server.com/storage, an attachment call.mp3 being stored in the account000 account on the abc123 doc defined below:

{"_id":"abc123"
 ,"foo":"bar"
 ,"bing":"bang"
}

We can create the following generated URLs:

| URL | field_list | | | http://my_server.com/storage/account000/abc123/call.mp3 | [{"arg":"account_id"}, {"arg":"id"}, {"arg":"attachment"}] | | | http://my_server.com/storage?path=/account000/abc123/call.mp3 | [{"const":"?path="}, {"arg":"account_id"}, {"arg":"id"}, {"arg":"attachment"}] | | | http://my_server.com/storage/bar/call.mp3 | [{"field":"foo"}, {"arg":"attachment"}] | | | http://my_server.com/storage/account001_call.mp3 | [{"group":[{"arg":"account_id"}, {"const":"_"}, {"arg":"attachment"}]}] | |

Global Storage settings

If you are the super-duper admin you can setup storage policy across your cluster by using the URL /v2/storage instead of per-account.

The validation request will arrive with /system_data/{DOC_ID}/{ANAME} on the end of the URL.

If you need to revert back to the standard settings you can run sup kzs_plan reset_system_dataplan; and changes will be removed and the initial version reinstalled.