method

unpack_graphemes

v5.1.7 - Show latest stable - 0 notes - Class: ActiveSupport::Multibyte::Unicode

1.0.0
1.1.6
1.2.6
2.0.3
2.1.0
2.2.1
2.3.8
3.0.0
3.0.9
3.1.0
3.2.1
3.2.8
3.2.13
4.0.2 (0)
4.1.8 (0)
4.2.1 (0)
4.2.7 (0)
4.2.9 (0)
5.0.0.1 (0)
5.1.7 (0)
5.2.3 (0)
6.0.0 (0)
6.1.3.1
6.1.7.7
7.0.0
7.1.3.2
7.1.3.4
What's this?

unpack_graphemes(string) public

Unpack the string at grapheme boundaries. Returns a list of character lists.

Unicode.unpack_graphemes('à¤à¥à¤·à¤¿') # => [[2325, 2381], [2359], [2367]]
Unicode.unpack_graphemes('CafÃ©') # => [[67], [97], [102], [233]]

Show source

# File activesupport/lib/active_support/multibyte/unicode.rb, line 49
      def unpack_graphemes(string)
        codepoints = string.codepoints.to_a
        unpacked = []
        pos = 0
        marker = 0
        eoc = codepoints.length
        while (pos < eoc)
          pos += 1
          previous = codepoints[pos - 1]
          current = codepoints[pos]

          # See http://unicode.org/reports/tr29/#Grapheme_Cluster_Boundary_Rules
          should_break =
            if pos == eoc
              true
            # GB3. CR X LF
            elsif previous == database.boundary[:cr] && current == database.boundary[:lf]
              false
            # GB4. (Control|CR|LF) Ã·
            elsif previous && in_char_class?(previous, [:control, :cr, :lf])
              true
            # GB5. Ã· (Control|CR|LF)
            elsif in_char_class?(current, [:control, :cr, :lf])
              true
            # GB6. L X (L|V|LV|LVT)
            elsif database.boundary[:l] === previous && in_char_class?(current, [:l, :v, :lv, :lvt])
              false
            # GB7. (LV|V) X (V|T)
            elsif in_char_class?(previous, [:lv, :v]) && in_char_class?(current, [:v, :t])
              false
            # GB8. (LVT|T) X (T)
            elsif in_char_class?(previous, [:lvt, :t]) && database.boundary[:t] === current
              false
            # GB9. X (Extend | ZWJ)
            elsif in_char_class?(current, [:extend, :zwj])
              false
            # GB9a. X SpacingMark
            elsif database.boundary[:spacingmark] === current
              false
            # GB9b. Prepend X
            elsif database.boundary[:prepend] === previous
              false
            # GB10. (E_Base | EBG) Extend* X E_Modifier
            elsif (marker...pos).any? { |i| in_char_class?(codepoints[i], [:e_base, :e_base_gaz]) && codepoints[i + 1...pos].all? { |c| database.boundary[:extend] === c } } && database.boundary[:e_modifier] === current
              false
            # GB11. ZWJ X (Glue_After_Zwj | EBG)
            elsif database.boundary[:zwj] === previous && in_char_class?(current, [:glue_after_zwj, :e_base_gaz])
              false
            # GB12. ^ (RI RI)* RI X RI
            # GB13. [^RI] (RI RI)* RI X RI
            elsif codepoints[marker..pos].all? { |c| database.boundary[:regional_indicator] === c } && codepoints[marker..pos].count { |c| database.boundary[:regional_indicator] === c }.even?
              false
            # GB999. Any Ã· Any
            else
              true
            end

          if should_break
            unpacked << codepoints[marker..pos - 1]
            marker = pos
          end
        end
        unpacked
      end

unpack_graphemes

Related methods