To answer this question you need to look at independently audited
standardized benchmarks. The benchmarks provided by the Transaction Processing
Performance Council are my favorite, although there are others that are worth
considering. If a DBMS vendor does not provide
such a benchmark then then I would be reluctant to agree with any of their
claims of superior performance. At one
time a tech account whispered to me they had secretly done the benchmark but
discovered their performance was so bad they decided to never publish it. Even some of the correlated SQL commands within the benchmark
refused to execute. Other vendors have indicated the same.
When you look at the results of the tests today you will likely be
surprised by the results. Many of the
most popular DBMS vendors no longer top the lists and winners may be some you may have never heard of.
The benchmarks I like to look at are:
The winner of TPC-C for is the Chinese Alibaba Cloud’sOceanBase yielding 707,351,007 transactions per minute! It leaves Oracle & DB2 in the dust, and
SQL Server is almost nowhere to be found (for locking reasons).
The winner of TPC-H for analytic processing EXASOL’sEXASolution yielding 11,612,395 queries per hour! It also has the best Price/QphH.
A special mention should be given for Alibaba Cloud’sAnalyticDB. Like General Electric’s Predix IoT cloud DBMS they are both built on Pivotal’s Greenplum open source DBMS,
which is a MPP version of PostgreSQL for petabyte scale databases. Special mention is given because they both
integrate the MADLib MPP statistical utility which features machine learning algorithms
at scale. There is a lot of ROI that can
be gained by doing A.I. at terabyte volume. (Most ML has to be done with
datasets measure in kilobytes.) This
integrates ML right into the SQL command and generates cost-optimized queryplans that considers the cost of data movement.
There is another reason to study TPC benchmarks. Each benchmark comes with a “Full Disclosure”
document telling you what hacks they had to do to get the posted performance—and
they all do hacks; no one runs the tests on a default system that has not been tweaked
to the max. So you study their
disclosure so you can try the same tricks on your database! A wealth of insight can be gained this way. For example, no one runs a DBMS on top of a
SAN. They go JBOD and carefully place their data files & partitions.
