I think what we're seeing with every database company providing new full-text search capabilities is an example of AI coding productivity showing up in the real world.
AFAIK all of these are implementations of the BM25 algorithm. You can just tell an agent to read about BM25 and implement it in your system of choice. Cool to see. Seems like there's still a lot of juice to be squeezed out of how it's architected and integrated into each system, but you can't help but wonder if this will lead to aggressive commodification
Becoming more the norm for them, Neki is the same.
Immediately rules out ever using them (though I don't currently have any problems that would benefit from that level of scale currently, have in the past though).
Postgres's license allows this but for me (personally) it leaves a bad taste.
I struggle with FTS inside SQL (SQLite and MSSQL). There is often a fairly significant impedance mismatch between the relational concerns and how the documents need to be stored.
I've always preferred to use SQL as the system of record and then build/maintain an external Lucene index. Do we think these integral FTS capabilities are at the point where a hybrid architecture doesn't make sense anymore? How much customization exists in this provider?
in my experience it’s pretty common to find big inverted indexes for text directly in the database - not necessarily large docs but certainly free text records in volume. using bm25 and unicode’s breakiterator is a very good way to build it. like putting lucene in the database basically - makes a lot of sense when the database is already large. places that bend over backwards to move search out of the db are usually trying to avoid having a very large db (and often end up with one anyway, getting the worst of both worlds)
I’m one of TIN’s developers and if you google my username you’ll see I’ve been in this space for a long time.
The answer to your first question is simply: yes
As far as your second question, what customization do you need that you believe TIN or PlanetScale doesn’t provide? These are things we can do, with alacrity.
It started with paradeDB and pg_search https://www.paradedb.com/blog/introducing-search
Timescale has pg_textsearch https://github.com/timescale/pg_textsearch
Neon and Databricks have Lakebase Search https://docs.databricks.com/aws/en/oltp/projects/lakebase-se...
Now PlanetScale.
AFAIK all of these are implementations of the BM25 algorithm. You can just tell an agent to read about BM25 and implement it in your system of choice. Cool to see. Seems like there's still a lot of juice to be squeezed out of how it's architected and integrated into each system, but you can't help but wonder if this will lead to aggressive commodification
They're not providing a local extension with the same performance at the time - it's only offered on their cloud services.
The local version https://github.com/planetscale/lead is mainly just for testing the syntax, it doesn't have the same perf characteristics.
Immediately rules out ever using them (though I don't currently have any problems that would benefit from that level of scale currently, have in the past though).
Postgres's license allows this but for me (personally) it leaves a bad taste.
I've always preferred to use SQL as the system of record and then build/maintain an external Lucene index. Do we think these integral FTS capabilities are at the point where a hybrid architecture doesn't make sense anymore? How much customization exists in this provider?
The answer to your first question is simply: yes
As far as your second question, what customization do you need that you believe TIN or PlanetScale doesn’t provide? These are things we can do, with alacrity.