1.4 The TiDB platform architecture and landscape

  |   0 评论   |   0 浏览

1.4 The TiDB platform architecture and landscape（TiDB 平台架构和全景图）

主讲人：

Liqi Geng(耿立琪)

R&D Engineer of PingCAP TP-Storage Team

Multi-Raft/Distributed Transaction

Part I: TiDB Core System

TiDB Core System

计算、存储分离的架构：

  • 存储层 -- TiKV(行存), TiFlash(列存)
  • 计算层 -- TiDB, TiSpark 分别处理 TSQL 和 SparkSQL
  • PD（Placement Driver） -- 负责整个集群的管理

TiDB (SQL Layer)

  • Stateless SQL Layer
    • Client can connect to any existing tidb-server instance
  • Full-featured SQL Layer
    • Speak MySQL wire protocol
    • Cost-based optimization
    • Secondary index support
    • Online DDL

SQL Layer

TiKV/TiFlash(Storage Layer)

  • TiKV: row-based storage engine
  • TiFlash: columar storage engine

Storage Layer

TiKV 是行式存储，更适合事务，TiFlash 是列式存储，更加适合分析处理。数据根据范围切分，同个范围的数据有多个副本，副本之间通过 Raft 协议进行同步。另外，TiFlash 上的副本固定由 Raft Leaner 进行同步。

事务类的查询走 TiKV，分析类的查询走 TiFlash。

PD(Placement Driver)

  1. Store the metadata of the cluster such as the region location of a specific key.
  2. Schedule and load balance regions in the TiKV cluster, including but not limited to data migration and Raft group leader transfer.
  3. Allocate the transaction ID that is globally unique and monotonically increasing.

TiSpark(SQL Layer)

SQL Layer

Part II: TiDB Ecosystem Tools

  • TiUP
  • Lightning, Dumpling
  • Backup & REstore(BR)
  • Change data capture(CDC)
  • DM(Data Migration)
  • Dashboard
  • ...

TiUP

  • A new development and component management tool introduced in TiDB Platform 4.0
  • Support local deployment, cluster deployment, component versioning and distribution
  • https://tiup.io

组件管理工具，单机部署、集群部署

Lightning

  • Import large amounts of data into a TiDB cluster
  • Support reading SQL dump exported via Mydumper or CSV data source

全量导入工具

Dumpling

  • Export data stored in TiDB as SQL or CSB data files

全量导出工具

Fast Backup & Restore(BR)

  • For large clusters
  • Distributed backup/restore
  • ~150MB/s per TiKV instance for backup/restore(with tunable backup speed)
  • Support external storage, such as NFS and S3
  • Multiple backup methods: Full/DB/Table
  • Support CA/SSL security certification

Fast Backup & Restore

分布式备份和恢复工具

TiDC

Change Data Capture Tool

  • Highly Available
  • Distributed, scale to any TiKV cluster size
  • REplicate KV changes between TiKV cluster in milliseconds
  • Transaction Restore
  • ETL workload

TiDC

TiCDC 是一款通过拉取 TiKV 变更日志实现的 TiDB 增量数据同步工具，具有将数据还原到与上游任意 TSO 一致状态的能力，同时提供开放数据协议 (TiCDC Open Protocol)，支持其他系统订阅数据变更。

LandScape

LandScape