How Can We Create Custom Field Types in Payload?

[flowingkhaos]

Hi there!

I am using a postgres adapter to use supabase as DB, and i am working on a RAG system. Which means that i want to store fields of type vector and tsvector into supabase postgres DB. I can't figure out how to accomplish that since it isn't clear how to declare custom field types.

For instance,

  {
  
  //collections/Article.ts
  
  import { CollectionConfig } from 'payload'
  import type { Field } from 'payload'
  

    export const Articles: CollectionConfig = {
     slug: 'articles',
        fields: [
        {
          name: 'embedding',
          type: 'json', // <- i need to change this to a vector type
          required: true,
         },
        ]  as Field[],
  }

In this case, in order to store embeddings in the database articles table, the field type in the payload Article.ts collection file AND the column type in the DB table NEED to match. However as we all know there is a limited selection of types native to payloads framework:

array
blocks
checkbox
json
code
collapsible
date
email
group
number
point
radio
relationship
rich-text
select
tabs
text
textarea
ui
upload

So here I ask: Does anyone know how to declare custom field types?

[DanRibbens]

Hey @flowingkhaos what you're working on sounds very interesting!

@r1tsuu has your answer from this other discussion: #8544 (comment)

You should be able to iterate from that and make the custom db type that way.

[flowingkhaos]

Nice, thanks for the direction!

[flowingkhaos]

My situation was very specific, but not necessarily uncommon. I am using supabase through the postgres adapter. I wanted to create collections with an ' embeddings ' field, so that I can perform RAG on these tables using my own technology.

In that specific instance, supabase expects ' vector ' and ' tsvector ' column types for search functions to work as intended. These types aren't supported in payload. No matter how i flipped it, I wasn't able to create a type match, between the backend and supabase(postgres) for embeddings.

If your field typing issue in payload, is related to storing vectors(embeddings) in collections, here is my solution.

Supabase gives you two project slots by default. Therefore, I encourage AI developers/engineers, that want to use payload as a cms to create content, generate embeddings for it and store it, to do as follows:

• Create a second project in your account, to store copies of each entry for the collection of your choice, with their embeddings added.
• Check the payload-types.ts file, or the collection table in your original supabase project, in order to create its parallel table in the second project. Leave out the fields/columns that aren't necessary to transfer over.
• Create a ' lib ' folder, where you will paste supabase's ' App frameworks ' boiler plate code, in order to configure a second supabase project in your application. (don't forget to set your .env variables)
• Create an ' actions ' folder, where you will write your server action or helper function to generate embeddings for a given field.
• Create a ' hooks ' folder where you will write a payload hook that triggers afterChange. This is where you will implement embedding and storing logic for each entry that is created or saved from the admin ui.

It is not as messy and complicated as it sounds.. The concept is to use two parallel tables that each reside in different supabase projects, to avoid conflict. One for storing cms entries, and one for storing copies with their embeddings. We use payload's hooks to generate an embedding and store the entry in the parallel table every time an entry is created, updated or duplicated. The parallel table will always be in the second supabse project, the one that has copies of your entries with embeddings. don't forget to use a field that is unique, in order to avoid duplicate content in the tables that contains embeddings, since embedding costs money. Slug is commonly used for that purpose. It is far from the most efficient solution, but it fixes the problem, while keeping your structure scalable and somewhat flexible.

Hope that was insightful..