The Ultimate Guide to SMS Segmentation and Text to Unicode Conversion

Introduction

If you have ever integrated an SMS gateway like Twilio or AWS SNS into your application, you have likely encountered unexpected message segmentation. You might send what looks like a short text message, only to be billed for two or three separate SMS segments.

This is due to the invisible complexities of SMS encoding. In this article, we'll dive deep into how GSM-7 and UCS-2 encodings work, why your 160-character limit sometimes drops to 70 characters, and how our newly updated Text to Unicode & SMS Analysis tool can help you debug these issues instantly.

The 160 Character Limit: GSM-7 Encoding

Standard SMS messages are sent using the GSM-7 encoding format. GSM-7 is a highly efficient 7-bit alphabet that can pack up to 160 characters into a single standard 140-byte SMS payload.

However, GSM-7 is extremely limited. It primarily supports:

• Basic Latin letters (A-Z, a-z)
• Numbers (0-9)
• Common punctuation marks
• A handful of Greek letters

The GSM-7 Extended Set

Some characters, like the Euro symbol (€) or square brackets ([ and ]), are part of the "GSM-7 Extended" character set. These characters require an escape sequence, meaning they actually consume 2 characters of your 160-character limit.

The 70 Character Limit: UCS-2 Encoding

The moment you include even one character that falls outside the GSM-7 character set—such as Thai characters, Japanese Kanji, smart quotes (often copy-pasted from Microsoft Word), or an Emoji—the telecommunications network is forced to switch the entire message encoding to UCS-2 (a precursor to UTF-16 Unicode).

Because UCS-2 requires 16 bits (2 bytes) per character instead of 7 bits, the maximum number of characters you can fit in a standard SMS payload drops drastically from 160 to 70.

What Happens When You Exceed the Limit?

If your message exceeds the limit (160 for GSM-7, or 70 for UCS-2), the carrier must split your message into multiple segments. To ensure the receiving phone can reassemble them in the correct order, a User Data Header (UDH) is added to each segment.

This UDH takes up space, slightly reducing the capacity of each segmented part:

• Multipart GSM-7: Drops from 160 to 153 characters per segment.
• Multipart UCS-2: Drops from 70 to 67 characters per segment.

Examples of Character Calculations

To make this concrete, let's look at how different types of characters are counted against your SMS segment limits:

1. English (GSM-7)

Standard English text (e.g., Hello World!) uses the basic GSM-7 character set.

• Cost: 1 character = 1 count.
• Limit: 160 characters per message.
• Example: "Hello" = 5 counts.

2. Thai and Other Non-Latin Languages (UCS-2)

Languages like Thai, Arabic, Japanese, or Chinese are not supported by GSM-7. Including even one character from these languages forces the entire message into UCS-2 encoding.

• Cost: 1 character = 1 count (but against a smaller limit).
• Limit: 70 characters per message.
• Example: "สวัสดี" = 6 counts (out of the 70 max).

3. Emojis (UCS-2)

Emojis are a special case. Because they exist in the higher planes of Unicode, they are encoded as "surrogate pairs" in UTF-16/UCS-2.

• Cost: 1 emoji = 2 counts.
• Limit: 70 characters per message (UCS-2 mode is activated).
• Example: "Hello 😊" = 5 (English) + 1 (Space) + 2 (Emoji) = 8 counts (out of the 70 max).

4. GSM-7 Extended Symbols

Certain symbols (like [, ], {, }, ^, ~, |, €, \) are technically supported in GSM-7 but belong to the extended table. They require an escape character to be sent.

• Cost: 1 symbol = 2 counts.
• Limit: 160 characters per message (Stays in GSM-7 mode).
• Example: "Total: 100€" = 6 (English) + 1 (Space) + 3 (Numbers) + 2 (Euro symbol) = 12 counts (out of the 160 max).

How to Optimize and Debug Your SMS

To help developers avoid unexpected billing costs and garbled text, we have launched the Text to Unicode & SMS Analysis tool on GoodParse.

Our tool provides a real-time playground for your text:

1. Live Encoding Detection: Instantly see whether your text is being processed as GSM-7 or UCS-2.
2. Hidden Character Highlighting: If your text trips the UCS-2 limit, the tool pinpoints exactly which non-GSM characters are responsible (e.g., catching a hidden smart quote).
3. SMS Parts Preview: See exactly how the carrier will chop up your message into "Part 1", "Part 2", etc., based on the strict 153/67 character multipart rules.

Decoding Mojibake

In addition to SMS analysis, the tool acts as a robust Unicode converter. If you are experiencing "mojibake" (garbled text rendering on a website), you can easily extract the raw \uXXXX or U+XXXX code points of your string to see exactly what bytes are being transmitted under the hood.

Conclusion

Understanding the difference between GSM-7 and UCS-2 is essential for any developer working with SMS automation. By diligently checking your message templates for hidden Unicode characters, you can effectively double your SMS limits and halve your gateway costs.

Try out our Text to Unicode Tool today to optimize your next SMS campaign!