OpenWare OPEX

rk_logo

OPEX installation and administration

This document aims to cover the OpenWare Platform EXchange (OPEX) installation and administration.

Installation

Prerequisites

Please install the following tools and read the documentations before starting.

Terraform

Please make sure to use Terraform 0.11 due to breaking changes in version 0.12

AWS Command Line Interface

Kubernetes

Helm

Ruby

  • apt (Debian or Ubuntu)
$ sudo apt-get install ruby-full
  • yum (CentOS, Fedora, or RHEL)
$ sudo yum install ruby

Create a base platform repository

Install the Kite DevOps tool

Make sure you have a fresh Ruby version installed and run: $ gem install kite

Create your Infrastructure as Code base repository

Create a new kite project using:

$ kite new PROJECT_NAME

Change configuration for cloud provider

Make sure to delete GCP block if you will be using AWS or vice versa

Please note that regions and zones differ for AWS and GCP default: &default # Note that region formats differ between AWS, GCP and AliCloud region: "europe-west4" # always change this for AWS zone: "europe-west4-c" # always change this for AWS gcp: # delete this block if you're using AWS project: "helios-devel" service_account: "~/safe/helios-devel.json" # if you want to use OPEX workspace this file should be located in the root of the kite project bucket_name: "kite-state-helios-devel" aws: # delete this block if you're using GCP access_key: "********************" # make sure to put credentials here secret_key: "********************" # make sure to put credentials here bucket_name: "kite-state-helios-devel" ~

Generate an environment(e.g. development/test/production)

Kite environments are separated workspaces with their own credentials, variables and modules.

Generate an environment

$ kite generate environment *env_name* --provider=aws|gcp

If you want to change the credentials for an environment, edit config/cloud.yml and regenerate environment with the same command.

Now the environment should be generated at config/environments/*env_name*

Add a module to your environment

To add a Kite module to your environment, you should first fetch them:

  kite mod fetch https://url.for/your/module --env *env_name*

This should clone module's source files into modules/*module_name*.

Then you need to render modules:

  kite mod vars https://url.for/your/module --env *env_name*

This will create a vars.*module_name*.yml file with all variables needed by the module. If you add kite-opex modules your default configuration will look similar to this:

---
# Infrastructure configuration(ignore when using baremetal)
terraform:

  node_type:     "m4.large"
  cluster_name:  "test"
  max_nodes:     "5"
  vpc_name:      "peatio-platform"
  subnet_name:   "peatio-rds-subnet"
  vpc_cidr:      "10.0.0.0/16"
  subnet_cidr_1: "10.0.10.0/26"
  subnet_cidr_2: "10.0.20.0/26"
  az_1_name:          "eu-central-1a" # make sure that availability zones are within the region
  az_2_name:          "eu-central-1b" # make sure that availability zones are within the region
  registry_name: "test"

  rbac: true
profile: standard # small, standard and large available
email: support@heliostech.fr
infrastructure:
  load_balancer_ip: ''
# Sendgrid configuration
smtp:
  sender_name: "Kayen Admin"
  host: "smtp.sendgrid.net"
  port: "2525"
  user: "apikey"
  password: "*smtp_password*"
# Bitgo proxy configuration
bitgo:
  enabled: true
  environment: test
# Change tags if you want to use different imgaes
dependencies:
  redis: 1.0.2
  rabbitmq: 0.8.1
  gcloud-sqlproxy: 0.3.3
  kube-lego: 0.4.0
  nginx-ingress: 0.14.1
  peatio: 0.1.1
  barong: 0.2.3
  vault: 0.12.0
  enterprise-front: 0.1.1
tags:
  barong: 2.0.4-alpha
  coinhub: 0.2.4
  peatio: 2.0.2-alpha
  vault: 0.10.1
peatio:
  titleMarkets: KAYEN
  host: api.2-0-alpha.kayen.org
  daemons:
    blockchain: 1
    global_state: 1
    k: 1
    payment_transaction: 1
    withdraw_audit: 1
  amqpDaemons:
    deposit_collection: 1
    deposit_collection_fees: 1
    deposit_coin_address: 1
    slave_book: 1
    market_ticker: 1
    matching: 1
    order_processor: 1
    pusher_market: 1
    pusher_member: 1
    trade_executor: 1
    withdraw_coin: 1
  env:
    admin: admin@peatio.tech,lbellet@heliostech.fr,dkoval@heliostech.fr,cmeulien@heliostech.fr,vshatravenko@heliostech.fr,ashanaakh@heliostech.fr,ysavchuk@heliostech.fr,mkuvshynnikov@heliostech.fr,ncaiserman@heliostech.fr,emasiuk@heliostech.fr,dhawkins@heliostech.fr,tbykova@heliostech.fr,pkucherenko@heliostech.fr,mboginska@heliostech.fr
# Here you can configure withdraw limits and fees
  fiats:
  - code: usd
    symbol: "$"
    min_deposit_amount: 0
    min_collection_amount: 0
    withdraw_limit_24h: 100
    withdraw_limit_72h: 200
    deposit_fee: 0
    withdraw_fee: 0
  coins:
  - code: btc
    symbol: "฿"
    base_factor: 100000000
    blockchain_key: btc-testnet
    precision: 8
    min_deposit_amount: 0
    min_collection_amount: 0
    withdraw_limit_24h: 100
    withdraw_limit_72h: 200
    deposit_fee: 0
    withdraw_fee: 0
  markets:
  - base:
      code: btc
      precision: 8
    quote:
      code: usd
      precision: 8
# Change this to use custom frontend
front:
  enabled: true
  host: www.kayen.io
  repository: gcr.io/front/frontend
  tag: latest
  authUrl: https://api.test.kayen.io/auth/barong
  blockchainInfoUrl: https://testnet.blockchain.info/tx/
  tradePageUrl: https://api.test.kayen.io/trading/btcusd
# Change this to use custom trading-ui
frontend:
  enabled: false
  host: trading.kayen.io
  repository: gcr.io/openware-production/frontend
  tag: latest
# Change this to configure twilio
twilio:
  account_sid: ACd944e3b2e1a4c5e37bfa8d9f11a245b6
  auth_token: 694e60aff87c5d1a66c272a62f931aab
  phone_number: "+18556312248"
# Change this to configure cryptonodes deployment
cryptonodes:
  # THIS CONFIGURATION IS USED BOTH FOR PACKER AND K8S CRYPTONODES DEPLOYMENT
  ethereum:
    enabled:         true      # Set to false if you don't want it to be deployed
    client:          "geth"    # Available options: geth, parity.
    network:         "rinkeby" # Available options: rinkeby, kovan, mainnet.
    disk_size:       "30"
    hostname:        "eth.helioscloud.com"
    rpcport:         "8545"
    port:            "30303"
    whitelisted_ips: ["0.0.0.0/0"] # Set this to empty array [] to whitelist just cluster IP

  bitcoind:
    enabled:         true      # Set to false if you don't want it to be deployed
    network:         "testnet"
    disk_size:       "30"
    hostname:        "btc.helioscloud.com"
    rpcuser:         "admin"
    rpcpassword:     "changeme"
    rpcport:         "18332"
    port:            "18333"
    whitelisted_ips: ["0.0.0.0/0"] # Set this to empty array [] to whitelist just cluster IP

  bitcoincash:
    enabled:         true      # Set to false if you don't want it to be deployed
    network:         "testnet"
    disk_size:       "30"
    hostname:        "bch.helioscloud.com"
    rpcuser:         "admin"
    rpcpassword:     "changeme"
    rpcport:         "18332"
    port:            "18333"
    whitelisted_ips: ["0.0.0.0/0"] # Set this to empty array [] to whitelist just cluster IP

  litecoind:
    enabled:         true      # Set to false if you don't want it to be deployed
    network:         "testnet"
    disk_size:       "30"
    hostname:        "ltc.helioscloud.com"
    rpcuser:         "admin"
    rpcpassword:     "changeme"
    rpcport:         "19332"
    port:            "19335"
    whitelisted_ips: ["0.0.0.0/0"]

  dashd:
    enabled:         true      # Set to false if you don't want it to be deployed
    network:         "testnet" # set to mainnet, testnet or regtest
    disk_size:       "30"
    hostname:        "dash.helioscloud.com"
    rpcuser:         "admin"
    rpcpassword:     "changeme"
    rpcport:         "19998"
    port:            "19999"
    whitelisted_ips: ["0.0.0.0/0"] # Set this to empty array [] to whitelist just cluster IP

# Change this to configure scout
scout:
  enabled: false
  key: changeme
  app: 2-0-alpha.kayen.org
  logLevel: warn
  env: production

Fill in vars.*module_name*.yml with correct values and render the module:

Note: the profile field determines the scale of your deployment, you can choose from small, standard and large depending on your load expectations

  kite mod render modules/*module_name* --env *env_name*

[Optional] Build and run the OPEX Workspace

OPEX Workspace is a Docker image built from your local kite platform and tailor-made for all the deployment operations. It contains all the necessary tools and is ready to use out of the box.

To bring the workspace up and build it if needed, simply run the bin/workspace_up.sh script.

WARNING Please make sure that you service account json file is located in the root of your kite project, before building the docker image. Also in config/cloud.yml the path should be considering kite project root folder. e.g. google-sa.json, instead of ~/safe/google-sa.json

The built image an be accessed by the opex-workspace:*env_name*-*version* tag, which is generated using the VERSION file in the root of the platform.

Apply Terraform configuration from the environment

Note: if you'd like to use a baremetal environment as your infrastructure, follow this document

Set your default gcloud credentials using

  aws configure
  kite terraform apply --env *env_name*

Configure and apply the kite-opex module

Step-By-Step Deployment

0. Enable shell autocompletion

To enable shell autocompletion for the OPEX Ruby binary, simply add:

source <($PWD/bin/opex complete *bash or zsh*)

to your bashrc or zshrc

1. Infrastructure

Apply Terraform configuration from the environment first

Set your default gcloud credentials using sh gcloud auth application-default login

To bring up infrastructure with Helm, Ingress and Kube Lego

./bin/opex infra install all

You can also install components separately, to see available options, type:

./bin/opex infra install

If you want to use an existing platform created by your own you need to dump infrastructure information in the file config/environments/tf-outputs.yml according to those specifications:

Key Description
cluster_name Name of the Kubernetes cluster to use for the deployment
generated_barong_user_password Password for the peatio DB user
generated_peatio_user_password Password for the barong DB user
instance_connection_name Name of the CloudSQL connection
instance_name Name of the CloudSQL instance
project GCP project ID
region GCP region
zone GCP zone
service_account_b64 GCP Service Account key used for initial provisioning encoded in Base64
storage_sa_key_b64 GCP Service Account key used for accessing GCS in Base64
cloudsql_sa_key_b64 GCP Service Account key used for Cloud SQL interactions encoded in Base64

Here is an example of output format:

cluster_name: "test"
instance_connection_name: "helios-devel:europe-west4:master-1337"
...
2. App Dependencies

Next you should deploy all the Peatio and Barong dependencies (Redis, RabbitMQ, SQL proxy, BitGo proxy, Bitcoind, Parity) Most components have their respective configs files located at config/environments/*env_name*/

You can either edit vars.*module_name*.yml file or edit a each file for a corresponding component respectively

Then install all components with the following command:

./bin/opex deps install all # This doesn't include Bitcoind and Parity

To check if everything is installed properly, run:

$ kubectl -n <deployment_id>-backend get pods
NAME                                      READY     STATUS    RESTARTS   AGE
gcloud-sqlproxy-xxxxx-xxx                 1/1       Running   0          1m
kube-lego-xxxxx-xxx                       1/1       Running   0          1m
nginx-ingress-controller-xxxxx-xxx        1/1       Running   0          1m
nginx-ingress-default-backend-xxxxx-xxx   1/1       Running   0          1m
rabbitmq-xxxxx-xxx                        1/1       Running   0          1m
redis-xxxxx-xxx                           1/1       Running   0          1m

Main dependency of Barong is Vault, so first check if the Vault configuration in config/environments/*env_name*/vault.yml is correct.

For a more detailed view on Vault usage and deployment , check its dedicated documentation.

Main dependency of Peatio is Cryptonodes, so first check if Cryptonodes configuration in config/environments/*env_name*/(bitcoind,parity).yml is correct. A more detailed instruction on cryptonodes deployment can be found here

3.x Application Deployments

In case you'd like to deploy the whole application stack(i.e. initial OPEX deployment), simply run ./bin/opex app install all

The following chapters will cover each application's configs, deployment and administration.

3. Barong Server

Configure your Barong deployment in config/environments/*env_name*/barong.yml:

  • Twilio configuration yaml twilio: accountSid: "ACd944e3b2e1a4c5e37bfa8d9f11a245b6" authToken: "694e60aff87c5d1a66c272a62f931aab" phoneNumber: "+18556312248"

  • Recaptcha configuration yaml APP_NAME: "Kayen" CAPTCHA_ENABLED: true RECAPTCHA_SITE_KEY: "changeme" RECAPTCHA_SECRET_KEY: "changeme"

  • S3 configuration ```yaml storage: provider: "AWS" bucket: "1337-bucket-barong-docs" accessKey: "*********" secretKey: "*********" region: "eu-central-1" # make sure that the right region is used




- Change seeds and configure management api

You can add seeds in this block
```yaml
seeds_yml:
    users:
      - email: "admin@kayen.org"
        password: "tdj8Zwrr"
        role: "admin"
        state: "active"
        level: 3

    levels:
      - key: email
        value: verified
        description: "User clicked on the confirmation link"
      - key: phone
        value: verified
        description: "User entered a valid code from sms"
      - key: document
        value: verified
        description: "User personal documents have been verified"

Documentation for management api is available here ```yaml management_api_yml: keychain: applogic: algorithm: RS256 value: key scopes: otp_sign: mandatory_signers: - applogic permitted_signers: - applogic

Next, run:

```bash
./bin/opex app install barong

You can also grant an existing user admin privilliges (Barong admin will not be Peatio admin by default):

# Retrieve a barong pod name
$ kubectl get pods

# Run the rails console on Barong
$ kubectl exec -it BARONG_POD -- rails console

# Find the user
irb(main):001:0> admin = User.find_by(email: "email@gmail.com")

# Grant the user admin
irb(main):002:0> admin.update(role: 'admin')

# Validate email manually
irb(main):003:0> admin.update(confirmed_at: Time.now)
irb(main):004:0> admin.level_set(:mail)
4. Peatio deployment

Each cryptocurrency needs to either use a BitGo wallet or an RPC node. To use a BitGo wallet in the system, follow these steps: - Create a wallet - Get your cluster nodes' IPs by running kubectl get nodes -o template --template='{% raw %}{{range.items}}{{range.status.addresses}}{{if eq .type "ExternalIP"}}{{.address}}, {{end}}{{end}}{{end}}{% raw %}' - Create a BitGo API access token for each cryptocurrency, whitelisting the nodes' IPs and setting a spending limit for only one currency at a time- Enter the wallet ID, address and access token for each currency in config/environments/*env_name*/wallets.yml yaml - name: Ethereum Deposit Wallet blockchain_key: eth-testnet currency_id: eth address: *wallet address* kind: deposit nsig: 1 max_balance: 0.0 status: active gateway: bitgo # For cryptonode it would be an RPC client, ex. geth settings: bitgo_test_net: on bitgo_wallet_id: *wallet id* bitgo_wallet_passphrase: *wallet passphrase* bitgo_rest_api_root: http://coinhub-coinhub.*deployment_id*-backend/api/v2 bitgo_rest_api_access_token: *api token* To use an RPC node follow these steps:

  • Deploy the cryptocurrency node

  • Generate the wallets as listed here

  • Configure config/environments/*env_name*/wallets.yml the following way

    - name:             Ethereum Hot Wallet
      blockchain_key:   eth-rinkeby
      currency_id:      eth
      address:          '0x9999999'
      kind:             hot
      nsig:             1
      max_balance:      100.0
      status:           active
      gateway:          geth  # It can be any other supported RPC client
      settings:
      uri:            'http://node_addr:8545'
      secret:         'secret'
    

All the other Peatio config files are located at config/environments/*env_name*/peatio:

  • application.yml - general settings including API CORS configuration, base customization(logo, color etc.), metadata and hostname

You can also configure the timeout for peatio endpoints and /admin tab, to do so change the following block in application.yml

  service:
    timeout_ms: "10000" # This value is specified in milliseconds
    idle_timeout_ms:  "500000"
    connect_timeout_ms: "4000"
  • blockchain.yml - blockchain configuration for each cryptonode

  • wallets.yml - cryptocurrency wallet configuration

  • currencies.yml - configs for initially provisioned cryptocurrencies

  • markets.yml - configs for initially provisioned markets/trading pairs

  • management_api_v1.yml - configs for the Peatio Management API.Documentation for Management API is available here

  • plugins.yml - a list of Peatio plugins to enable(the respective source code must be bundled into the Docker image for it to work)

To finally deploy Peatio, run:

./bin/opex app install peatio
5. Cryptonodes deployment

For a more detailed view on Cryptonodes usage and deployment , check dedicated documentation.

6. Frontend deployment

To deploy the custom frontend component, see the instructions for it.

7. Tower deployment

To deploy Tower, check if all the values from config/environments/*env_name*/tower.yml are correct, the default ones should be valid, then run the deployment script:

./bin/opex app install tower
Maintenance

If you want to do some maintenance on k8s nodes or cryptonodes that you've deployed, you can do so via a bastion node. To be able to ssh to these VM's you have to follow this guide:

1.From your local machine, start ssh-agent with the following command to manage your keys for you:

  me@local:~$ eval `ssh-agent`

2.Call ssh-add to load the gcloud compute public keys from your local computer into the agent, and use them for all SSH commands for authentication:

  me@local:~$ ssh-add ~/.ssh/google_compute_engine

3.Log into an instance with an external IP(use private IP for k8s nodes) address while supplying the -A argument to enable authentication agent forwarding.

  gcloud compute ssh --ssh-flag="-A" INSTANCE
Troubleshooting
  • If you are getting "wrong certificate" errors in the browser, you probably should simply wait a few minutes before Let's Encrypt issue the certificate.

Appendix

Benchmarking the deployment with Toolbox

To benchmark your deployment, fill in config/environments/*env_name*/toolbox.yml with correct values and then simply run ./bin/opex toolbox benchmark. The benchmark report would be saved in config/environments/*env_name*/reports afterwards.

Debugging the deployment with Toolbox console

If you'd like to debug your deployment, feel free to launch the Toolbox console by simply running ./bin/opex toolbox console. This will create a Kubernetes Pod with all the necessary debug tools already installed. To connect to the pod, run kubectl exec -it toolbox-*id* bash

Using a private Docker registry

Store the credentials to login to the registry as kubernetes secret:

kubectl create secret docker-registry regcred --docker-server=<your-registry-server> --docker-username=<your-name> --docker-password=<your-pword> --docker-email=<your-email>

Then configure the components values with the secret name and your private registry:

image:
  repository: <your-registry-server>
  tag: 1.2.3
  pullPolicy: Always
  pullSecret: regcred