Tokenization Explained with Examples

 Tokenization Explained with Examples


Tokenization is the process of breaking down text into smaller units called tokens.

These tokens can be words, characters, or subwords, depending on the approach. Tokenization is one of the first steps in Natural Language Processing (NLP) because computers need structured input to understand human language.


Why is Tokenization Important?


Computers don’t understand sentences the way humans do.


Tokenization helps convert text into manageable pieces for tasks like translation, sentiment analysis, chatbots, and search engines.


Types of Tokenization

1. Word Tokenization


Splitting a sentence into words.


👉 Example:

Sentence: "I love quantum computing."

Tokens: ["I", "love", "quantum", "computing"]


This is useful for tasks where meaning is based on words.


2. Character Tokenization


Splitting text into individual characters.


👉 Example:

Sentence: "AI"

Tokens: ["A", "I"]


This is often used in languages like Chinese, or in tasks like spelling correction.


3. Subword Tokenization


Breaking text into smaller meaningful parts (subwords).

This helps handle rare or unknown words by splitting them into pieces.


👉 Example (using Byte-Pair Encoding or WordPiece):

Word: "unhappiness"

Tokens: ["un", "happi", "ness"]


This way, even if the model has never seen “unhappiness”, it can still understand it from known subwords.


4. Sentence Tokenization


Splitting a paragraph into individual sentences.


👉 Example:

Paragraph: "Quantum computing is fascinating. It has great potential."

Tokens:

["Quantum computing is fascinating.", "It has great potential."]


Real-Life Example of Tokenization in Action


In Google Translate, text is tokenized into smaller chunks so the system can translate word by word or phrase by phrase.


In chatbots, tokenization helps the model break down user queries into understandable parts.


✅ In short:


Tokenization = splitting text into smaller parts.


Types = word, character, subword, and sentence tokenization.


It’s a crucial first step in making human language understandable for machines.

Learn Artificial Intelligence Course in Hyderabad

Read More

What Is NLP and Why Is It Important?

🤖 Natural Language Processing (NLP)

Fine-Tuning Pre-trained Models for Custom Tasks

Transfer Learning: Train Faster with Less Data


Comments

Post a Comment

Popular posts from this blog

Handling Frames and Iframes Using Playwright

Handling Frames and Iframes Using Playwright When automating modern web applications, you’ll often come across frames or iframes—HTML elements used to embed another HTML document within the current page. If you're using Playwright for browser automation, knowing how to handle frames properly is crucial for interacting with elements inside them. In this blog, we’ll walk through how to identify, access, and interact with frames and iframes using Playwright with code examples in JavaScript and Python. 🧾 What Are Frames and Iframes? <frame>: Rarely used now; part of the outdated <frameset> HTML structure. <iframe> (inline frame): Embeds another HTML page inside the current one. Common for third-party content like ads, maps, or payment gateways. 🎯 Why Frames Matter in Automation Elements inside an iframe are not part of the main DOM, so trying to interact with them directly will result in errors unless you switch the context to the frame. 🔍 Locating an Iframe in Pla...
Read more

Cybersecurity Internship Opportunities in Hyderabad for Freshers

 🚀 1. AICTE Cyber Security Internship (Jul–Aug 2025) Organizer: AICTE via its internship portal—free, fully online Duration: ~4–6 weeks starting July 2025 Perks: Hands-on labs (Kali, Nmap, Wireshark), capstone project, AICTE certificate Cost/Stipend: Free, no stipend Apply by: July 15, 2025 via AICTE portal  en.wikipedia.org +15 govtinternship.com +15 joincomputerit.blogspot.com +15 Perfect for freshers seeking practical exposure with certified recognition. 🏢 2. Gainsight – Application Security Intern (On-site) Role: SOC monitoring, SIEM dashboards, incident response Location: Hyderabad office Requirements: Bachelor’s in CS/related field; enthusiastic freshers welcome Stipend: Not specified  hyderabadtechbytes.blogspot.com placementdrive.in Ideal if you want corporate cybersecurity experience using enterprise tools. 🌱 3. Internshala – Cyber Security Internships Listings: Multiple internship openings across Hyderabad (both remote and on-site) Fields: Ethical hacking, se...
Read more

Tosca for API Testing: A Step-by-Step Tutorial

Tosca for API Testing: A Step-by-Step Tutorial Tricentis Tosca is a powerful and user-friendly test automation tool that supports a wide range of testing types — including API testing. With Tosca, you can easily design, automate, and execute API tests without writing any code. It supports REST, SOAP, and other service types, making it a great choice for end-to-end API validation. 🧰 What is Tosca API Testing? Tosca’s API testing allows testers to: Send requests (GET, POST, PUT, DELETE, etc.) Validate responses (status codes, body, headers) Chain requests and share data between them Perform security, load, and functional tests on APIs ✅ Step-by-Step Guide: Tosca API Testing 🔹 Step 1: Open Tosca and Create a Workspace Launch Tosca Commander Create a new workspace (or open an existing one) Choose the workspace type as Multi-user or Single-user based on your setup 🔹 Step 2: Add a New API Test Case Navigate to the Modules section. Right-click and choose Scan > API Scan. The API Scan wi...
Read more