Encoding Explained Simply

A comprehensive guide from the Telegraph to Unicode.

"What hath God wrought!" —*Samuel Morse, first telegraph message (1844)*

This blog will help you understand all about character encodings, which has dramatically evolved. Explores the full spectrum of encoding schemes that have shaped digital communication.

Introduction

What is encoding anyway?

Well, encoding is information in one form and converting it into another form so it can be stored, transmitted, or processed by a system. When we talk about the character encoding its a technique, where we are instructing the machine that how to treat numbers, letters, and symbol, how to turn them into numbers that machine understands.

Long story short Character encoding = a set of rules that map characters (human-readable text) to numbers (machine-readable binary).

Example :- →you have a message in english (original form) → you translate into morse code/ binary code (encoded form) → later, decoding in original form ( english ). Character encoding: "A" → 65 (in ASCII) → 01000001 (binary).

Overview of history

Morse Code (1837-1844)

Morse code was the first widely used electrical character encoding system, developed by Samuel Morse and Alfred Vail. It used frequency analysis of English letters to assign shorter codes to more common letters.

Type: Variable-length binary encoding using dots and dashes.
What it does: Encodes letters, numbers, and punctuation using short signals (dots) and long signals (dashes).
Where used: Telegraph communication, amateur radio, early electrical communication systems
Limitation: Requires manual interpretation; not suitable for direct computer processing.

Baudot Code (1870)

Developed by Émile Baudot in 1870, this was one of the first true binary character encodings. Used shift characters to 2X the available characters and space.

Type: 5-bit fixed-length encoding with 32 possible char.
What it does: Encodes uppercase letters, numbers, and punctuation using five-bit combinations
Where used: Telegraph systems, early teleprinter networks
Limitation: Limited character set requiring FIGS/LTRS shift codes to represent both letters and numbers. (here FIGS is for figures/numbers/symbols, LTRS is used for letters).

This is it for the historical overview, we’re focusing more on what we’re using today and and how it important it is in digital communication.

Early Computer Era (1960 - 1980)

ASCII (1963-1968)

Type: 7-bit encoding (128 characters, values 0–127).
What it does: Encodes English letters (upper/lowercase), digits, punctuation, and control codes.
Where used: Early computers, text files, internet protocols, email headers.
Limitation: Only supports basic English; cannot represent non-English characters like é, ñ, or 漢字.

This was the major limitation of ASCII that it could only support 127 characters or values. Because of which various extensions were released later.

After ASCII, there was extended ASCII Era → ISO 8859-1 : it was of 8-bit (256 characters). Limited to Western European languages, no support for Eastern European, Asian, or other scripts.

and yess Windows is also in this game, they built Windows-1252 (CP-1252), which was built on ISO 8859-1 and used in Microsoft Windows, legacy documents, older web apps. Then there was a race for language based encodings like, Big-5, GB2312, Shift JIS, KOI8-R.

Coming to Present

Unicode Era (1990s-Present)

Unicode, introduced in 1991, unified global characters into a single set. Post-1990 encodings focus on efficient Unicode representation

How UTF changed the entire shape of communication and encoding, and became dominating while others faded?

→ to put it in a simple way, just understand that older encodings like ASCII, ISO-8859, and Shift-JIS were region-specific, meaning text could only be correctly displayed within a limited language set, which led to incompatibility and garbled characters across systems. Unicode solved this by defining a universal set of characters, and UTF-8 became dominant because it’s backward-compatible with ASCII, efficient for English (1 byte per character), supports all world languages and symbols, avoids endianness issues, and was adopted as the web and programming standard. Today, languages like Go natively store strings as UTF-8, making it seamless to handle both simple ASCII text and complex multilingual or emoji-rich data.

Use case : like building REST APIs that returns JSON responses: UTF-8 ensures that "message": "Hello 🌍" is transmitted correctly and understood by any client worldwide.

UCS-2 and UCS-4 (Early 1990s)

Type: UCS-2: Fixed 2-byte; UCS-4: Fixed 4-byte.
What it does: Directly map Unicode code points in fixed widths (UCS-2 for basic plane only)
Where used: Early Windows NT, legacy Java strings.
Limitation: UCS-2 can't handle emojis or rare scripts, both waste space.

UTF-8 (1992) → Major Breakthrough

Invented in 1992, UTF-8 exploded in the 2000s and now dominates globally.

Type: Variable-length (1-4 bytes per character)
What it does: Encodes all Unicode characters efficiently, ASCII-compatible for English.
Where used: 98.7% of websites, modern apps, databases, file systems.
Limitation: More bytes for Asian scripts than UTF-16, but overall most versatile.

Other types UTF-16, UTF-32. It’s niche, and rarely used.

It’s a simple hello world program in GO, where hello → english, and chinese letter is used for World. This is because Go supports UTF-8 character encoding.

Some modern programming languages which defaults to UTF-8:

• Go, Julia, Rust, and Swift 5 assume UTF-8 encoding.

• Python is transitioning to UTF-8 for internal string storage.

• Microsoft now recommends UTF-8 for Windows API applications.

Conclusion

As we have walked through historical foundations of ASCII from 1960s to its UTF variants since early 2000s, you can get the one idea that, there are no more encoding issues, we’re in an era where whole world is connected through internet and character encoding has transformed from a regional puzzle into a global standard that powers our connected world. We can experience seamless text-handling across languages and devices. Slowly we’re moving towards AI driven encodings or VR symbol systems maybe, idk anything about it, you can do research anyway.

KEEP BUILDING, See yaa!