1. Home
  2. Lessons
published
15 November 2024
by
Ray Morgan
updated
17 January 2025

Character Sets and Unicode

Understanding character sets is Unicode are fundamental to enabling your application to support multiple languages and writing systems. Properly handling character encoding ensures that your application can correctly display and process text in any language. If the character encoding is not handled correctly, users may see garbled text or question marks instead of the intended characters.

Key Concepts

Character Sets

A character set is a collection of characters that a computer can recognize and use for displaying text. Each character in the set is mapped to a specific numerical value called a code point. Early character sets like ASCII (American Standard Code for Information Interchange) were limited in scope, primarily covering English characters (plus some control codes). Beyond ASCII, some other common character sets include ISO-8859-1 (for Western Europe), Windows-1256 (for Arabic), KS X 1001 (for Korean), and Unicode, which has become a de-facto standard for the web and will be our primary focus.

Unicode

Unicode is a universal character encoding standard for representing and organizing all characters used in written languages, including complex scripts and rare characters, symbols, and even emojis.

Unicode assigns each character a unique number, called a code point. Code points are written in the format U+xxxx, where xxxx is a hexadecimal number. Example: The code point for the letter 'A' is U+0041, and for the Euro symbol '€' it is U+20AC.

The Unicode Standard defines a maximum of 1,114,112 valid code points (U+0000 to U+10FFFF), with 17 planes of 65,536 code points each. As of version 16.0, released in September 2024, the Unicode Standard defines 154,998 characters across 168 scripts. These characters encompass a wide range of symbols, including letters, digits, punctuation marks, and various symbols used in writing systems worldwide.

The Unicode Consortium regularly updates the standard to include new characters and scripts, reflecting the evolving nature of written communication. For the most current statistics and detailed information on character assignments, refer to the Unicode Consortium's official statistics page

Common Pitfalls

Incorrect Character Encoding

Applications that fail to correctly configure character encoding can produce garbled or unreadable text, often displayed as question marks (?) or other unexpected symbols. This typically occurs when there is a mismatch between the encoding used to store or transmit data and the encoding expected by the application or database. For instance, if a database stores text as Latin-1 (ISO-8859-1) but the application expects UTF-8, non-ASCII characters like accented letters (é) or symbols (€) may not render correctly.

Impact: User frustration, reduced credibility and usability, especially if users cannot read critical information in their own language (or at all).

Prevention: Ensure consistent use of Unicode encoding like UTF-8 (more on UTF-8 in the next lesson) across all parts of the application, including source code, databases, APIs, and file storage.

Relying on Outdated or Limited Character Sets

Using older character sets like ASCII or ISO-8859-1 limits your application's ability to handle global languages. ASCII, for example, supports only English letters and a small set of symbols, making it entirely unsuitable for other languages. ISO-8859-1 expands the range slightly to cover Western European languages but still excludes many characters used in Slavic, Greek, Asian, and African scripts.

Impact: Exclusion of non-Western audiences, inability to support multilingual text or special characters (e.g., emojis), and increased technical debt when migrating to Unicode later.

Prevention: Adopt Unicode as the encoding standard from the start to avoid these limitations and future migration costs.

Handling Complex Scripts Incorrectly

Some languages, such as Arabic, Hindi, and Thai, use scripts that require complex rendering, such as contextual shaping, ligatures, or bidirectional text support. (More on all of these in later lessons.) Failure to account for these complexities can result in improperly rendered text, such as broken or disjointed characters. Additionally, text alignment for right-to-left (RTL) languages like Arabic or Hebrew can be disrupted if the application's layout is not designed to accommodate bidirectional text.

Impact: A poor user experience for speakers of these languages, making the application seem incomplete and unprofessional.

Prevention: Use libraries or frameworks that support complex text rendering and implement proper bidirectional text handling in UI design.

Inconsistent Updates to the Unicode Standard

The Unicode Standard is periodically updated to include new scripts, symbols, and emojis. If an application or system fails to adopt these updates, users may encounter missing or unsupported characters, especially for newer additions like emojis or less common languages. For example, an emoji introduced in Unicode 15.0 may not display correctly in systems that only support Unicode 14.0.

Impact: Missing symbols or placeholder characters (e.g., a rectangle or square) can confuse users or give the impression that the application is outdated.

Prevention: Stay informed about Unicode updates and regularly update your system’s libraries, fonts, and character encoding support.

Assuming Fixed Byte Sizes for Characters

Some developers mistakenly assume that all characters are of fixed byte sizes, such as one byte per character in ASCII. However, in UTF-8 encoding, characters can occupy between one and four bytes, depending on their complexity. This can lead to buffer overflows, string truncation, or errors in character counting when working with non-ASCII text.

Impact: Data loss, application crashes, or improperly displayed content.

Prevention: Use functions and libraries designed to handle variable-length encodings and Unicode-aware string operations.

Workflow and Functional Requirements

  • Features:
    • Support for Unicode encoding (e.g., UTF-8) across all input, storage, and output layers of the application.
    • Compatibility with languages requiring complex scripts or bidirectional text rendering.
  • Completion Criteria:
    • Text input, storage, and display function correctly for multiple languages.
    • No garbled or placeholder characters are visible in the UI.
  • Test Cases:
    • Verify display of various Unicode characters, including complex scripts and emojis.
    • Test input and retrieval of multilingual text in the database.

Demos

Implementation

  • Configure everything in your development environment and application stack to use UTF-8 encoding by default, including code editors, databases (and tables), files, and APIs.
  • Validate all input from all sources — users, APIs, and other data sources — to ensure that it is encoded as UTF-8 text. Have workflows and tools (preferably automated) to convert text from other encodings to UTF-8.
  • Regularly update your system to align with the latest Unicode standard.

Design Rationales

  • Using Unicode ensures extensibility and future-proofing as new scripts and characters are added.
  • UTF-8 is the most efficient encoding for web applications, balancing compatibility and storage efficiency. The advantages of UTF-8 over other Unicode encodings will be covered in detail in the following lessons.)
← Back Next →
✕