little boxes made of words

1. On Wed, Jan 30 at 11am,
   Jens Ayton said —

   Wait… 16-bit? Why would you limit your programming language to the BMP? Treating Unicode as an array of 16-bit values is a bug, not a feature.
2. On Wed, Jan 30 at 11am,
   Michael Tsai - Blog - Arc 0 said —

   […] Update: Jens Alfke is “flummoxed” by Graham’s comments about character sets. […]
3. On Wed, Jan 30 at 11am,
   Neil said —

   Why would you only use 16-bit characters, rather than just making the default string encoding UTF8? That would be a more compact representation for the common case of Latin characters, and still let you represent all of Unicode.
4. On Wed, Jan 30 at 12pm,
   Mork said —

   Please don’t confuse Lisp with Arc. Common Lisp and Scheme users have very ideas about what a good programming language is. In fact, the implementation that Arc uses — MzScheme, a Scheme implementation — already supports utf-8 (why he didn’t just use that as a basis for utf-8 support eludes me).
   Arc’s idea of good design is _not_ representative of any more established Lisp variant. (Arc is a very disappointing take on what was claimed to become a ‘100-year language’. But hey, it only took 6 years of design (see also http://xach.livejournal.com/156104.html).)
5. On Wed, Jan 30 at 12pm,
   Mork said —

   That should read ‘have very different ideas’, not ‘have very ideas’.
   That’s what I get for not proofreading.
6. On Wed, Jan 30 at 1pm,
   michael said —

   RE: your comment about i18n at JavaSoft in 1996.

   By 1998 and 1999, JavaSoft had to spend a tremendous amount of time and effort making Java2D play nice in the i18n world and handle all those “advanced typographic features” of Unicode.

   And don’t get me started on how they initially forgot to create a printing API for Java2D. 1998 was a very busy year.
7. On Wed, Jan 30 at 1pm,
   Anonymous said —

   It should be obvious that unicode isn’t that required: your 16-bit broken unicode is just fine- but does it really offer any advantages over 8bit? Does 32-bit broken unicode offer enough advantages over your 16bit to warrant using that?

   Making strings of bytes instead of strings of characters is a good thing- and good enough for any web app anyone would ever want to write (think: � is a perfectly good encoding for unicode on the web)- and a lot more portable.

   On the other hand, building characters into the language causes no end in problems for languages like Python which generate coding errors unexpectedly, and CL which has no standard for external formats. Punting the issue makes it easier to get your program working faster, and I can’t possibly consider why that would be a bad thing… Especially considering that nobody complaining about this can offer a single reason why having bignum characters for all strings is a good thing.
8. On Wed, Jan 30 at 1pm,
   Dande said —

   I think what Paul Graham meant to say is, “There is so much to do in designing a language, and things like character sets that require a lot of detail and planning are going to have to wait. The kind of people who don’t understand that I am going to focus and develop how the language feels before I worry about that kind of stuff probably won’t like Arc.”
9. On Wed, Jan 30 at 1pm,
   chanson (LJ) said —

   A lot of what I’ve read about Arc since Graham published his tutorial has been worrying. Why would I even bother looking at Arc instead of Scheme? It’s not even as interesting a language as either classical prefix Dylan or even infix Dylan — and the classical prefix Dylan was essentially just Scheme re-imagined with a CLOS-style object system underneath (and exposed to the developer).
10. On Wed, Jan 30 at 1pm,
    Joe said —

    I don’t know what’s worse, PG not understanding unicode, or all the people whining about it not understanding it either. Using 16 bit characters doesn’t solve anything, as it still won’t hold tons of code points that are above 16 bit, so you still need to use a variable length encoding. At which point you might as well just use utf-8.
11. On Wed, Jan 30 at 2pm,
    Anonymous said —

    Joe: I think people mean lots of different things by “using unicode”.

    Some are talking about string-representation, and string-access time. Others are talking about automatic io-coding. Both of these things have further questions: what to do for invalid codings, or how to handle un-normalized code points.

    But even before you get into what you think the answers to these questions are, start with what exactly you need the language to help with. What part of unicode is least pleasant to deal with directly- so much so that language-support could help? Then how would you keep your new feature from pissing off the people who that doesn’t bother?

    I don’t mean to suggest there aren’t answers: just that they aren’t obvious; this isn’t a “simple” thing by a long shot.
12. On Wed, Jan 30 at 2pm,
    notbrainsurgery (LJ) said —

    This was my sentiment exactly, after reading Paul’s announcement.

    Also his arguments seems little backwards. It goes like this:

    1. Python was not decided with right character representation in the beginning and it took a year to change that

    2. I would not want to do that, so I will use character representation which is unlikely to satisfy most of the world population and is pretty much obsolete so I would not have the problem.

    Vadim
13. On Wed, Jan 30 at 7pm,
    tonetheman said —

    my snarky note is that he is right.. ascii was good enough for a long time and in the length of time you have to work on a language you are much better off spending your time implementing something that matters. Later you can go back and fix character issues. Getting the language is more important in the scheme of things.

    as for HTML… CSS sucks. period. If you think otherwise you have not had to write it and account for all the variations of browsers. CSS was meant to be written by tools not humans… that said CSS sadly is becoming the black magic of choice for some reason…
14. On Thu, Jan 31 at 3pm,
    newlisp said —

    #!/usr/bin/env newlisp

    (constant (global ‘☼) MAIN)

    (context ‘☺)

    (define (☻ ✄ ☁ ⍾)
    (print ✄ ☁ ⍾))

    (define (‽)
    (println {‽}))

    (context ☼)

    (set ‘℥ “what ” ‘ᴥ “the ” ‘ᴒ “dickens”)

    (☺:☻ ℥ ᴥ ᴒ)

    (☺:‽)
    (exit)
15. On Thu, Jan 31 at 10pm,
    ndanger.organism :: blog :: Arc: the blogosphere rages said —

    […] 96 Characters Ought To Be Enough For Anyone: Jens’ Unicode retort […]
16. On Fri, Feb 1 at 1am,
    cimota.com » Blog Archive » Remove your assumptions said —

    […] Jens Alfke’s latest blog post rambles about a couple of things but finishes on something that I really empathised with: Apple engineer: …and the layout needs to take into account ligatures and contextual forms, where adjacent letters change glyphs depending on neighboring characters, or even merge into a single glyph. […]
17. On Fri, Feb 1 at 8am,
    Bob K. said —

    Real Unicode support is pretty hard if your language has a “character” object. One nice bit of fun is that Unicode encodings can represent the same character once it actually needs to be exchanged with another implementation (which could be a API, a network protocol, or application data format). First, you need to choose an encoding scheme, like UTF-7, UTF-8, UTF-16, and then, within some of those schemes, there are multiple choices like: is “á” one code point (as in ISOLatin1, which is also the first block of Unicode), or is a + accent, or accent + a ? Look at this if you dare. So any language implementation is going to have to deal with these issues once it promises “Unicode support.” Also, if looks like if you’re willing to use IBM’s ICU, there is help

    I’d probably choose a canonical representation of 32 bits for a Unicode character, but perhaps that’s hopeless from a performance point of view.
18. On Fri, Feb 1 at 3pm,
    Andrew Thompson said —

    Is there some reason not to just make characters 16 bits wide, specify they are in fact UTF-16, which is indistinguishable from UCS-2 in all the common cases and provide library support for the ugliness that occurs if people actually use something outside the basic multilingual plane (which means China and Japan, probably).

    Unicode normalization is a real pain, but that usually only matters when doing I/O and is a library issue.
19. On Fri, Feb 1 at 3pm,
    Jens Alfke said —

    @Andrew Thompson — You took the words out of my mouth. The mainstream Unicode APIs I’m aware of (on Mac OS X, Windows, Java) basically do this. 16-bit Unicode isn’t perfect, but people seem to have settled on it as the sweet spot where you get almost all the necessary functionality without so much pain. (There are APIs in CoreFoundation to deal with the edge cases like normalization, but I’ve never had to use them, fortunately.)

    On the other hand, 8-bit encodings are troublesome. There actually is a reason why Python and Ruby are putting in all the effort to get off of 8-bit strings. UTF-8 isn’t generally too horrible, but there are too many not-uncommon text manipulations where you operate on characters and character positions, and then it becomes a pain in the ass. Worst of all, if you’re an English-speaking developer, you don’t hit most of those edge cases very soon, unless you’re very careful about I18N testing. Then your app barfs in weird ways when you get users or installations in the rest of the world.

    (And using HTML-style escaping as an encoding for working with 8-bit strings in memory, as “Anonymous” suggested, is just nuts. HTML-escaped text is nasty to deal with — even simple stuff like string comparisons becomes difficult and expensive, because there are so many ways to represent any one string.)
20. Name:
    Email:
    Website:
    Name and email are required. Thanks!
    
    [Or log in]    [Or register]

    Or use your OpenID:
    LiveJournal username DeadJournal username Other OpenID URL

    HTML: You may use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>

    Notify me of followup comments via e-mail

Design: Hemingway, modified by Chris Gerlach, further modified by me, Jens.

Entries feed   Comments feed