notifications
News
Products
For Enterprises
Embeddings
World-class multimodal multilingual embeddings for search/RAG/agents.
Reranker
World-class neural retriever for maximizing search relevancy.
Reader
Read URLs or search the web, get better grounding for LLMs.
Segmenter
Segment long text into chunks and do tokenization.
For Power Users
PromptPerfect
Premier tool for prompt engineering
More power user tools
SceneXplain
Leading AI solution for image captions and video summaries
BestBanner
Blog to banner, without the prompts!
JinaChat
More modality, longer memory, less cost
Rationale
Ultimate AI decision-making tools
For Developers
DocArray
The data structure for multimodal data
Jina
Build multimodal AI applications on the cloud
More developer tools
CLIP-as-service
Embed images and sentences into fixed-length vectors with CLIP
Finetuner
Fine-tune embeddings on domain specific data for better search quality
JCloud
Deploy a local project as a cloud service. Radically easy, no nasty surprises.
LangChain-Serve
Langchain apps on production with Jina & FastAPI
VectorDB
A Python vector database you just need - no more, no less
DALL-E Flow
A human-in-the-Loop workflow for creating HD images from text
DiscoArt
Create compelling Disco Diffusion artworks in one line of code
ThinkGPT
Agent techniques to augment your LLM and push it beyond its limits
DevGPT
Your virtual development team
RunGPT
An open-source cloud-native of large multimodal models serving framework
Jerboa
An experimental finetuner for open-source LLMs
Company
About us
Contact sales
Intern program
Join us
open_in_new
Download logo
open_in_new

Segmenter

Free API for segmenting long text into chunks and tokenization.

What is a Segmenter?

A segmenter is a crucial component that converts text into tokens or chunks, which are the basic units of data that an embedding/reranker model or LLM processes. Tokens can represent whole words, parts of words, or even individual characters.

Chunking long documents, lightning fast!

You can also use Segmenter API to cut long documents into smaller chunks, making it easier to process them in embeddings or rerankers. We leverage common structural cues and build a set of rules and heuristics which perform well across diverse types of content, e.g. Markdown, HTML, LaTeX and CJK languages.
Maximum length of each chunk: 1000
Maximum number of characters in each chunk. In practice the chunk length can be smaller than this value, if there is a good boundary in the text.
0 chunks in total
Segmenter API is free!
By providing your API key, you can access a higher rate limit, and your key won't be charged.
Rate Limit
ProductAPI EndpointDescription Allowed RequestWithout API Key (RPM)With API Key (RPM)With Premium API Key (RPM)Average Latency (s)Token Usage Counting
Segmenter APIhttps://segment.jina.aiTokenize and segment long textGET/POST2020010000.3Token is not counted as usage.
Reader APIhttps://r.jina.aiConvert URL to LLM-friendly textGET/POST2020010001.6Count the number of tokens in the output response.
Reader APIhttps://s.jina.aiSearch the web and convert results to LLM-friendly textGET/POST5401007.7Count the number of tokens in the output response.
Embedding APIhttps://api.jina.ai/v1/embeddingsConvert text/images to fixed-length vectorsPOSTblock60keyboard_double_arrow_up300keyboard_double_arrow_up
bolt
depends on the input size
Count the number of tokens in the input request.
Reranker APIhttps://api.jina.ai/v1/rerankTokenize and segment long textPOSTblock60keyboard_double_arrow_up300keyboard_double_arrow_up
bolt
depends on the input size
Count the number of tokens in the input request.
keyGet your API key
handshakeContact sales
Segmenter API
Our Segmenter API is crucial for helping LLMs manage input within context limits, and optimizing model performance. It allows developers to count tokens and extract relevant text segments, ensuring efficient data processing and cost management.
report_problemWe cannot generate an API key because we couldn't verify if you are human. If you believe this is an error, please contact us. 
Contact
key
API Key & Billing
code
Usage
more_horiz
More
  
home
Auto preview
help_outlinefaq
  Status
Use GET request to count tokens
Use POST request for more features
upload
Request
curl -X POST 'https://segment.jina.ai/' \
	 -H "Content-Type: application/json" \
	 -d '{
    	"content": "
  Jina AI: Your Search Foundation, Supercharged! 🚀
  Ihrer Suchgrundlage, aufgeladen! 🚀
  您的搜索底座，从此不同！🚀
  検索ベース,もう二度と同じことはありません！🚀
"}'



FAQ
Rate Limit
ProductAPI EndpointDescription Allowed RequestWithout API Key (RPM)With API Key (RPM)With Premium API Key (RPM)Average Latency (s)Token Usage Counting
Segmenter APIhttps://segment.jina.aiTokenize and segment long textGET/POST2020010000.3Token is not counted as usage.
Reader APIhttps://r.jina.aiConvert URL to LLM-friendly textGET/POST2020010001.6Count the number of tokens in the output response.
Reader APIhttps://s.jina.aiSearch the web and convert results to LLM-friendly textGET/POST5401007.7Count the number of tokens in the output response.
Embedding APIhttps://api.jina.ai/v1/embeddingsConvert text/images to fixed-length vectorsPOSTblock60keyboard_double_arrow_up300keyboard_double_arrow_up
bolt
depends on the input size
Count the number of tokens in the input request.
Reranker APIhttps://api.jina.ai/v1/rerankTokenize and segment long textPOSTblock60keyboard_double_arrow_up300keyboard_double_arrow_up
bolt
depends on the input size
Count the number of tokens in the input request.
Segmenter-related common questions
How much does the Segmenter API cost?
 
The Segmenter API is free to use. By providing your API key, you can access a higher rate limit, and your key won't be charged.
If I don't provide an API key, what is the rate limit?
 
Without an API key, you can access the Segmenter API at a rate limit of 20 RPM.
If I provide an API key, what is the rate limit?
 
With an API key, you can access the Segmenter API at a rate limit of 200 RPM. For premium paid users, the rate limit is 1000 RPM.
Will you charge the tokens from my API key?
 
No, your API key is only used to access a higher rate limit.
Does the Segmenter API support multiple languages?
 
Yes, the Segmenter API is multilingual and supports over 100 languages.
What is the difference between GET and POST requests?
 
GET requests are solely used to count the number of tokens in a text, allows you easily integrate it as a counter in your application. POST requests supports more parameters and features, such as returning the first/last N tokens.
What is the maximum length I can tokenize per request?
 
You can send up to 64k characters per request.
How does the chunking feature work? Is it semantic chunking?
 
The chunking feature segments long documents into smaller chunks based on common structural cues, ensuring accurate segmentation of text into meaningful chunks. Essentially it is a (big!) regex pattern that segments text based on certain syntactical features that often align with semantic boundaries, such as sentence endings, paragraph breaks, punctuation, and certain conjunctions. It is not semantic chunking. This (big) regex is as powerful as it can be within the limitations of regular expressions. It balances complexity and performance. While true semantic understanding isn't possible with regex, it well-approximates context by common structural cues.
How do you handle special tokens such as 'endoftext' in the Segmenter API?
 
If the input contains special tokens, our Segmenter API will put them in the field 'special_tokens'. This allows you to easily identify them and handle them accordingly for your downstream tasks, e.g. removing them before feeding the text into an LLM to prevent injection attacks.
Does chunking support other languages than English?
 
Besides western languages, chunking also works well with Chinese, Japanese, and Korean.
API-related common questions
code
Can I use the same API key for embedding, reranking, reader, fine-tuning APIs?
 
Yes, the same API key is valid for all search foundation products from Jina AI. This includes the embedding, reranking, reader and fine-tuning APIs, with tokens shared between the all services.
code
Can I monitor the token usage of my API key?
 
Yes, token usage can be monitored in the 'Buy tokens' tab by entering your API key, allowing you to view the usage history and remaining tokens.
code
What should I do if I forget my API key?
 
If you have misplaced a topped-up key and wish to retrieve it, please contact support AT jina.ai with your registered email for assistance.
Contact
code
Do API keys expire?
 
No, our API keys do not have an expiration date. However, if you suspect your key has been compromised and wish to retire it or transfer its tokens to a new key, please contact our support team for assistance.
Contact
code
Why is the first request for some models slow?
 
This is because our serverless architecture offloads certain models during periods of low usage. The initial request activates or 'warms up' the model, which may take a few seconds. After this initial activation, subsequent requests process much more quickly.
code
Is user input data used for training your models?
 
We adhere to a strict privacy policy and do not use user input data for training our models.
Billing-related common questions
attach_money
Is billing based on the number of sentences or requests?
 
Our pricing model is based on the total number of tokens processed, allowing users the flexibility to allocate these tokens across any number of sentences, offering a cost-effective solution for diverse text analysis requirements.
attach_money
Is there a free trial available for new users?
 
We offer a welcoming free trial to new users, which includes one million tokens for use with any of our models, facilitated by an auto-generated API key. Once the free token limit is reached, users can easily purchase additional tokens for their API keys via the 'Buy tokens' tab.
attach_money
Are tokens charged for failed requests?
 
No, tokens are not deducted for failed requests.
attach_money
What payment methods are accepted?
 
Payments are processed through Stripe, supporting a variety of payment methods including credit cards, Google Pay, and PayPal for your convenience.
attach_money
Is invoicing available for token purchases?
 
Yes, an invoice will be issued to the email address associated with your Stripe account upon the purchase of tokens.
Offices
location_on
Berlin, Germany (HQ)
Prinzessinnenstraße 19-20, 10969 Berlin, Germany
Geschäftsanschrift: Leipzigerstr. 96, 10117 Berlin, Germany
location_on
Beijing, China
Level 5, Building 6, No.48 Haidian West St. Beijing Haidian, China
location_on
Shenzhen, China
402, Floor 4, Fu'an Technology Building, Shenzhen Nanshan, China
Search Foundation
Embeddings
Reranker
Reader
Segmenter
Get Jina AI API key
API Status
Company
About us
Contact sales
Newsroom
Intern program
Join us
open_in_new
Download logo
open_in_new
Terms
Terms & Conditions
Privacy
Manage Cookies
    email
 Jina AI GmbH © 2020-2024.