Install All-In-One (Binary)

This page introduces how to install BladePipe On-Premise Binary version in the Linux operating system.

On-prem deployment involves various components of BladePipe. It is recommended to first learn about BladePipe architecture to facilicate deployment process.

Resource Preparation

Resource List

Resource	System/Software	Recommendation	Number	Description
Machine to deploy BladePipe	Centos 8.5 / RHEL / Cloud Linux	4 core, 16 GiB Mem, 100 GiB Disk	1	Run the Console, Workers, DataJobs and monitoring services
Metadata database	MySQL 8	2 core, 4 GiB Mem, 100 GiB Disk	1	Store metadata such as data sources, DataJobs, Workers, etc.
Alert Notification	Slack / Discord		1	Slack / Discord bot or Webhook

Ports occupied: 8111, 8084, 8085, 7007, 8083, 9090

Environment Preparation

Log in to the Machine

Set up the Environment

Run the following comand to install JDK (OpenJDK).
```
yum -y install java-17-openjdk-devel.x86_64
```

Add a user.

useradd -d /home/bladepipe -m bladepipe

passwd bladepipe

Grant sudo privileges to the user.

echo "bladepipe ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers

Create a catalog.

mkdir -p /home/bladepipe/{logs,backup,tar_gz}

chown bladepipe:bladepipe /home/bladepipe/*

(Optional) Disable the system firewall and set selinux=disabled.

firewall-cmd --state

systemctl stop firewalld.service

systemctl disable firewalld.service

firewall-cmd --state

Configure core parameters.

vim /etc/security/limits.conf

# nofile - maximal opened files, * for all users

*           soft       nofile     65535
*           hard       nofile     65535

# optional file is 20-nproc.conf
vim /etc/security/limits.d/90-nproc.conf

# modify bladepipe's maximal opened processes
*             soft       nproc      1024 
bladepipe     soft       nproc      131072

Install Metadata Database

Install the MySQL command-line tool and database. If you use an existing database, just skip this step.

rpm -ivh https://repo.mysql.com/mysql80-community-release-el8-9.noarch.rpm

yum -y install mysql-server.x86_64

# start the mysql
systemctl start mysqld

Create a metadata database.

CREATE DATABASE  `bladepipe_console` DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;

CREATE DATABASE  `bladepipe_rdp` DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;

Create a username and password and grant privileges.

create user 'bladepipe'@'%' identified by 'Bladepipe#2024'; 

grant all on bladepipe_console.* to 'bladepipe'@'%'; 

grant all on bladepipe_rdp.* to 'bladepipe'@'%'; 

flush privileges;

exit;

Install BladePipe Console

Go to BladePipe. Click Enterprise Edition > Download (compatible with X86/ARM64) to get the download link.

Download the package to the specified path.

cd /home/bladepipe/tar_gz

wget -c "{paste the download link got from BladePipe website}" -Obladepipe.tgz

Modify the privilege to the package.

chown -R bladepipe:bladepipe /home/bladepipe/

Switch to the user bladepipe, and enter the home directory.
```
su - bladepipe

cd /home/bladepipe/tar_gz
```

Unzip the package.

tar -xaf bladepipe.tgz

tar -zxvf bladepipe-console.tar.gz

mv /home/bladepipe/tar_gz/bladepipe /home/bladepipe

Modify the parameters in the configuration file.

cd ~/bladepipe/console/conf

vi business-north_america.properties

Configure the following parameters.

Parameter	Description
spring.datasource-cc.url	URL of the metadata database bladepipe_console. Modify host:port, database name and time zone(if in different time zone)
spring.datasource-cc.username	Username of the metadata database bladepipe_console
spring.datasource-cc.password	Password of the metadata database bladepipe_console
spring.datasource-rdp.url	URL of the metadata database bladepipe_rdp. Modify host:port, database name and time zone(if in different time zone)
spring.datasource-rdp.username	Username of the metadata database bladepipe_rdp
spring.datasource-rdp.password	Password of the metadata database bladepipe_rdp
jwt.secret	The secret key of the system login authentication algorithm. It can be a 64-bit random code.
console.rsocket.dns	Private network host of the machine (without port)
prometheus.host	Private network host of the machine (without port)

Start the Console.

cd ~/bladepipe/console/bin

sh ./startConsoleAndUpdDb.sh

Install Prometheus

Download from Prometheus official site

su - bladepipe

cd ~/tar_gz

wget -c "https://github.com/prometheus/prometheus/releases/download/v2.53.4/prometheus-2.53.4.linux-amd64.tar.gz" -Oprometheus.tar.gz

tar -zxvf prometheus.tar.gz -C /home/bladepipe

Edit prometheus.yml.

cd /home/bladepipe/prometheus

vi prometheus.yml

Add config to scrape_configs section in file.

- job_name: 'bladepipe-worker'
  scrape_interval: 10s
  metrics_path: '/metrics'
  static_configs:
  - targets: ['localhost:8084']
- job_name: 'bladepipe-console'
  scrape_interval: 10s
  metrics_path: '/console/metrics/prometheus'
  static_configs:
  - targets: ['localhost:8111']

Start up prometheus using the following scripts.

./prometheus --log.level=debug 1>>stdout.log 2>&1 &

Log in to the Console

Console Address (Please visit it in Chrome): http://${your_bladepipe_console_ip}:8111
Default username: test@clougence.com
Default password: clougence2021
Default verification code: 777777
info
It is recommended to change the credentials according to the following docs:

Activate BladePipe

Please refer to License to get the activation code.

Install a Worker

Please refer to Install a Worker.

Resource Preparation​

Environment Preparation​

Log in to the Machine​

Set up the Environment​

Install Metadata Database​

Install BladePipe Console​

Install Prometheus​

Log in to the Console​

Activate BladePipe​

Install a Worker​