commjilo.blogg.se - Mongodb performance vs postgres

#Mongodb performance vs postgres drivers#
#Mongodb performance vs postgres update#
#Mongodb performance vs postgres driver#
#Mongodb performance vs postgres full#

#Mongodb performance vs postgres drivers#

Sysbench was, according to OnGres, the only option for OLTP performance testing, but our experts found in their benchmark repository that they had run YCSB tests with production drivers on all sides. MongoDB Mistake #2: Misunderstanding YCSB MongoDB has neither interest in nor intellectual obligation to respond to the OnGres material with code, and we’re certainly not wasting our time submitting a pull request on a repo designed to cheat against us. Instead, as stated by their Head of Engineering Comms in a Twitter thread where several people were asking for ways to reproduce their numbers:

#Mongodb performance vs postgres update#

More interestingly: did MongoDB bother to patch the sysbench benchmark, update the Lua driver, add connection pooling or replace it with another driver, run the benchmark again and publish the results? Our published source code and automation would have allowed them to do this quite easily.

#Mongodb performance vs postgres driver#

Moreover, it tells the world how much MongoDB actually cares about Lua users, specially if this driver (which was created by MongoDB) remains unmaintained, and no other official Lua driver has been created. I disagree, because this means that this is the performance that Lua users get. Because of this, MongoDB concludes that “ any reasonable tester would have looked for an alternative benchmark”.

The reasons we choose to use connection pooling are deeply discussed (#26-29).ĭriver quality. Results for both cases were detailed in the report (pages 32, 37) and in the S3 bucket that stores all the results. We test PostgreSQL with and also without connection pooling (the latter results were unrecognized by MongoDB in their response). Its also interesting to note they also inaccurately implied that we wrote the sysbench implementation, when it was in fact developed by Percona (see page 25 of the whitepaper).Ĭonnection pooling.

Worst of all, MongoDB kept mentioning the “experimental driver” throughout their article we used for all three benchmarks when it was used only in one: the OLTP benchmark. They make this mistake several times in their article. In response to the quote above, we observed that MongoDB actually began “ with the transactions test” but then switched to discuss about the OLTP test. They decided to use an experimental, unsupported, non-production Lua driver for the sysbench implementation they created to perform one of their three tests. This makes OnGres' selection of a MongoDB driver particularly strange. Production MongoDB drivers have connection pooling. MongoDB Mistake #1: Mixing Benchmark Data This posts analyzes and responds to the most significant mistakes, and proposes a collaborative and iterative approach to benchmarking, with transparency as key driver.

#Mongodb performance vs postgres full#

And worse, it mixes which benchmark is which, quotes phrases out of context, ignores things that were detailed on the whitepaper and is full of mistakes. But unfortunately we found the tone of MongoDB’s post arrogant and unprofessional. We love criticism, when it’s constructive, correct and leads to a conversation that enriches database users.

All the raw results are public, stored on a several GB S3 bucket.

A benchmark was written to test the ACID transactions support.

All this code is open source and can be used to reproduce our results or make changes and run your own benchmarks.

We created a 100% automated platform to run the benchmarks, record monitoring information, and process the results.

We adhered to a strict benchmark ethics policy, which is detailed in the whitepaper (page #4) and reproduced in the benchmark presentation.

We wrote a very comprehensive 50-page whitepaper explaining in great detail everything we did, why we did it and especially how we did it.

Here is what OnGres did to maintain transparency: Well, if anything, now we will definitely do more benchmarks comparing PostgreSQL and MongoDB, and continue to do so with transparency. They went on to lecture OnGres that we shouldn’t be doing benchmarks. Ironically, they advocated in their blog response that we should have made our “ benchmarks reproducible and published our results in full”. Then, MongoDB produced new performance results, refuting ours, without clarifying how they achieved those numbers nor publishing any source code. In a nutshell, MongoDB claimed that we at OnGres are PostgreSQL experts but we made “ a range of basic errors in the use of MongoDB”. For a quick overview, check our presentation summarizing the benchmark and the results. While a long read at close to 50 pages, we encourage you to at least read the executive summary (2 pages) and any other relevant section, to have the right context. Which they wrote as a response to the whitepaper “ Performance Benchmark: PostgreSQL / MongoDB”, published and sponsored by EnterpriseDB and performed by OnGres. This post is a reply to MongoDB’s “ Benchmarking: Do it right or don’t do it at all” post.