From: Eli Zaretskii Date: Wed, 15 Oct 2014 14:11:25 +0000 (+0300) Subject: Update the bidirectional reordering engine for Unicode 6.3 and 7.0. X-Git-Tag: emacs-25.0.90~2635^2~679^2~50 X-Git-Url: http://git.eshelyaron.com/gitweb/?a=commitdiff_plain;h=ed7ebd933a5294e0f27cb49268e7b6376de36648;p=emacs.git Update the bidirectional reordering engine for Unicode 6.3 and 7.0. src/bidi.c (bidi_ignore_explicit_marks_for_paragraph_level): Remove variable. (bidi_get_type): Return the isolate initiators and terminator types. (bidi_isolate_fmt_char, bidi_paired_bracket_type) (bidi_fetch_char_skip_isolates, find_first_strong_char) (bidi_find_bracket_pairs, bidi_resolve_brackets): New functions. (bidi_set_sos_type): Renamed from bidi_set_sor_type and updated for the new features. (bidi_push_embedding_level, bidi_pop_embedding_level): Update to push and pop correctly for isolates. (bidi_remember_char): Modified to accept an additional argument and record the bidi type according to its value. (bidi_cache_iterator_state): Accept an additional argument to only update an existing state. Handle the new members of struct bidi_it. (bidi_cache_find): Arguments changed: no lnger accepts a level, instead accepts a flag telling it whether it is okay to return unresolved neutrals. (bidi_initialize): Initiate and staticpro the bracket-type uniprop table. Initialize new isolate-related members. (bidi_paragraph_init): Some code factored out into find_first_strong_char. (bidi_resolve_explicit_1): Function deleted, its code incorporated into bidi_resolve_explicit. (bidi_resolve_explicit): Support the isolate initiators and terminator. Fix handling of embeddings and overrides according to new UBA requirements. Record information about previously seen characters here (moved from bidi_level_of_next_char). (bidi_resolve_weak): Adapt to changes in struct members. (FLAG_EMBEDDING_INSIDE, FLAG_OPPOSITE_INSIDE, MAX_BPA_STACK) (STORE_BRACKET_CHARPOS, PUSH_BPA_STACK): New macros. (bidi_resolve_neutral): Call bidi_resolve_brackets to handle the paired bracket resolution. Handle isolate initiators and terminator. (bidi_type_of_next_char): Remove unneeded code for BN limit. (bidi_level_of_next_char): Move the code that records information about previous characters to bidi_resolve_explicit. Fix logic of resolving neutrals and make sure their cache entries are updated. Remove now unneeded special handling of PDF level. src/dispextern.h (struct glyph): Enlarge the width of resolved_level. (BIDI_MAXDEPTH): New macro, renamed from BIDI_MAXLEVEL and enlarged per Unicode 6.3. (enum bidi_bracket_type_t): New data type. (struct bidi_saved_info): Leave only 2 type members out of 4. Remove bytepos. (struct bidi_stack): Add members necessary to support isolating sequences. (struct bidi_it): Add new members necessary to support isolating sequences and bracket pair resolution. src/xdisp.c (Fbidi_resolved_levels): New function. (syms_of_xdisp): Defsubr it. (append_glyph, append_composite_glyph, produce_image_glyph) (append_stretch_glyph, append_glyphless_glyph): Convert aborts to assertions. (syms_of_xdisp) : New variable. src/term.c (append_glyph, append_composite_glyph) (append_glyphless_glyph): Convert aborts to assertions. src/.gdbinit (pgx): Display the character codepoint, resolved level, and bidi type also for glyphless glyphs. lisp/simple.el (what-cursor-position): Update to support the new bidi characters. lisp/descr-text.el (describe-char): Update to support the new bidi characters. admin/unidata/unidata-gen.el (unidata-prop-alist): New properties 'paired-bracket' and 'bracket-type', in support of the UBA 6.3. (unidata-gen-table): Support PROP-IDX being a function. (unidata-describe-bidi-bracket-type, unidata-gen-brackets-list) (unidata-gen-bracket-type-list): New functions. (unidata-check): Support checking the 'bracket-type' attribute. (unidata-gen-files): Don't create backups for uni-*.el files. admin/unidata/Makefile.in (${unidir}/charprop.el): Depend on BidiMirroring.txt and BidiBrackets.txt. admin/unidata/BidiBrackets.txt: New file, from Unicode. etc/NEWS: Mention the UBA implementation update. etc/HELLO: Remove now unneeded directional control characters. doc/lispref/nonascii.texi (Character Properties): Document the new properties 'bracket-type' and 'paired-bracket'. doc/lisprefdisplay.texi (Bidirectional Display): Update the version of the UBA to which we are conforming. test/BidiCharacterTest.txt: New file, from Unicode. test/biditest.el: New file. --- ed7ebd933a5294e0f27cb49268e7b6376de36648 diff --cc admin/ChangeLog index f3d6d048e21,f3d6d048e21..e9fc535454b --- a/admin/ChangeLog +++ b/admin/ChangeLog @@@ -1,3 -1,3 +1,18 @@@ ++2014-10-15 Eli Zaretskii ++ ++ * unidata/unidata-gen.el (unidata-prop-alist): New properties ++ 'paired-bracket' and 'bracket-type', in support of the UBA 6.3. ++ (unidata-gen-table): Support PROP-IDX being a function. ++ (unidata-describe-bidi-bracket-type, unidata-gen-brackets-list) ++ (unidata-gen-bracket-type-list): New functions. ++ (unidata-check): Support checking the 'bracket-type' attribute. ++ (unidata-gen-files): Don't create backups for uni-*.el files. ++ ++ * unidata/Makefile.in (${unidir}/charprop.el): Depend on ++ BidiMirroring.txt and BidiBrackets.txt. ++ ++ * unidata/BidiBrackets.txt: New file, from Unicode. ++ 2014-10-13 Glenn Morris * authors.el (authors-aliases, authors-fixed-case) diff --cc doc/lispref/ChangeLog index 3955bd09b38,b8f17c0fae5..ad1f6a6d45a --- a/doc/lispref/ChangeLog +++ b/doc/lispref/ChangeLog @@@ -1,3 -1,3 +1,11 @@@ ++2014-10-15 Eli Zaretskii ++ ++ * nonascii.texi (Character Properties): Document the new ++ properties 'bracket-type' and 'paired-bracket'. ++ ++ * display.texi (Bidirectional Display): Update the version of the ++ UBA to which we are conforming. ++ 2014-10-13 Glenn Morris * Makefile.in (dist): Update for new output variables. diff --cc etc/ChangeLog index 711208e0e32,c74f86b1fb5..2d13bc4060a --- a/etc/ChangeLog +++ b/etc/ChangeLog @@@ -1,3 -1,3 +1,9 @@@ ++2014-10-15 Eli Zaretskii ++ ++ * NEWS: Mention the UBA implementation update. ++ ++ * HELLO: Remove now unneeded directional control characters. ++ 2014-10-13 Jan Djärv * NEWS: Move and clarify OSX >= 10.6. diff --cc lisp/ChangeLog index 80157c67704,80157c67704..f2e2d8b96f2 --- a/lisp/ChangeLog +++ b/lisp/ChangeLog @@@ -1,5 -1,5 +1,11 @@@ 2014-10-15 Eli Zaretskii ++ * simple.el (what-cursor-position): Update to support the new bidi ++ characters. ++ ++ * descr-text.el (describe-char): Update to support the new bidi ++ characters. ++ * emacs-lisp/tabulated-list.el (tabulated-list-mode): Force bidi-paragraph-direction to 'left-to-right'. This fixes buffer-menu display when the first buffer happens to start with diff --cc src/ChangeLog index 6823f6d3127,c1b7c6cc8c3..22fd7891df5 --- a/src/ChangeLog +++ b/src/ChangeLog @@@ -1,12 -1,3 +1,79 @@@ ++2014-10-15 Eli Zaretskii ++ ++ Update the bidirectional reordering engine for Unicode 6.3 and 7.0. ++ * bidi.c (bidi_ignore_explicit_marks_for_paragraph_level): Remove ++ variable. ++ (bidi_get_type): Return the isolate initiators and terminator ++ types. ++ (bidi_isolate_fmt_char, bidi_paired_bracket_type) ++ (bidi_fetch_char_skip_isolates, find_first_strong_char) ++ (bidi_find_bracket_pairs, bidi_resolve_brackets): New functions. ++ (bidi_set_sos_type): Renamed from bidi_set_sor_type and updated ++ for the new features. ++ (bidi_push_embedding_level, bidi_pop_embedding_level): Update to ++ push and pop correctly for isolates. ++ (bidi_remember_char): Modified to accept an additional argument ++ and record the bidi type according to its value. ++ (bidi_cache_iterator_state): Accept an additional argument to only ++ update an existing state. Handle the new members of struct bidi_it. ++ (bidi_cache_find): Arguments changed: no lnger accepts a level, ++ instead accepts a flag telling it whether it is okay to return ++ unresolved neutrals. ++ (bidi_initialize): Initiate and staticpro the bracket-type uniprop ++ table. Initialize new isolate-related members. ++ (bidi_paragraph_init): Some code factored out into ++ find_first_strong_char. ++ (bidi_resolve_explicit_1): Function deleted, its code incorporated ++ into bidi_resolve_explicit. ++ (bidi_resolve_explicit): Support the isolate initiators and ++ terminator. Fix handling of embeddings and overrides according to ++ new UBA requirements. Record information about previously seen ++ characters here (moved from bidi_level_of_next_char). ++ (bidi_resolve_weak): Adapt to changes in struct members. ++ (FLAG_EMBEDDING_INSIDE, FLAG_OPPOSITE_INSIDE, MAX_BPA_STACK) ++ (STORE_BRACKET_CHARPOS, PUSH_BPA_STACK): New macros. ++ (bidi_resolve_neutral): Call bidi_resolve_brackets to handle the ++ paired bracket resolution. Handle isolate initiators and ++ terminator. ++ (bidi_type_of_next_char): Remove unneeded code for BN limit. ++ (bidi_level_of_next_char): Move the code that records information ++ about previous characters to bidi_resolve_explicit. Fix logic of ++ resolving neutrals and make sure their cache entries are updated. ++ Remove now unneeded special handling of PDF level. ++ ++ * dispextern.h (struct glyph): Enlarge the width of resolved_level. ++ (BIDI_MAXDEPTH): New macro, renamed from BIDI_MAXLEVEL and ++ enlarged per Unicode 6.3. ++ (enum bidi_bracket_type_t): New data type. ++ (struct bidi_saved_info): Leave only 2 type members out of 4. ++ Remove bytepos. ++ (struct bidi_stack): Add members necessary to support isolating ++ sequences. ++ (struct bidi_it): Add new members necessary to support isolating ++ sequences and bracket pair resolution. ++ ++ * xdisp.c (Fbidi_resolved_levels): New function. ++ (syms_of_xdisp): Defsubr it. ++ (append_glyph, append_composite_glyph, produce_image_glyph) ++ (append_stretch_glyph, append_glyphless_glyph): Convert aborts to ++ assertions. ++ (syms_of_xdisp) : New variable. ++ ++ * term.c (append_glyph, append_composite_glyph) ++ (append_glyphless_glyph): Convert aborts to assertions. ++ ++ * .gdbinit (pgx): Display the character codepoint, resolved level, ++ and bidi type also for glyphless glyphs. ++ +2014-10-15 Dmitry Antipov + + Avoid unwanted point motion in Fline_beginning_position. + * lisp.h (scan_newline_from_point): Add prototype. + * search.c (scan_newline_from_point): New function, refactored from... + * cmds.c (Fforward_line): ...adjusted user. + * editfns.c (Fline_beginning_position): Use scan_newline_from_point + and simplify the former since the latter doesn't move point. + 2014-10-14 Dmitry Antipov Cleanup terminal handling code. diff --cc test/ChangeLog index 1ef5ee99775,1ef5ee99775..ea11f9429f0 --- a/test/ChangeLog +++ b/test/ChangeLog @@@ -1,3 -1,3 +1,9 @@@ ++2014-10-15 Eli Zaretskii ++ ++ * BidiCharacterTest.txt: New file, from Unicode. ++ ++ * biditest.el: New file. ++ 2014-10-08 Leo Liu * automated/print-tests.el: New file.