New XYENC 1/13/09 release

[wbass@xxxxxxxx]

I have just updated the XYENC and XYDEC programs available via my web
site. To get them, go to

 http://basspad.com/xywrite/

and download xyenc094.zip.

I was recovering from a virus and in something of a fog during the last
several days where I did work on this release, so please don't be
surprised if there are some bonehead errors in the release.

I found Harry Binswanger's 15 Jan 2009 12:12 post discouraging -- either
Harry's not paying much attention, or I'm doing a lousy job of getting
across what the XYENC work is about.

I'm not concerned that much about the "binarys" and "code" in this release
being broken as much I am about the documentation (which is usually the
thing that is hardest and done most poorly for most software, including
XyWrite). So I went ahead and did the release, with the expectation that
I'll probably take another look at the documentation (which may take
awhile).

It's not that I think the documentation doesn't explain enough. It's just
that it is too hard to explain what the problem is that is being solved --
since it's not really one simple problem but a set of fairly fuzzy ones --
in a way where most folks don't fuzz out and stop reading before they
understand.

Clue to Harry: it's not about inventing a "better" language. XPL is what
WE HAVE to work with, and what things like U2 are written in. XyWrite
doesn't support XyBasic, and U2 isn't (I don't think) written in XyBasic.
XYENC is a tool to make XPL easier to work with and maintain, not to
replace XPL with the "perfect language."

Anyway, Carl mostly "gets it," even if Harry doesn't, so maybe it's not
all my fault.

A question/request for Carl:

QDF2.PM in this release is a macro mostly of the form

BC ch /this/that/XC
BC ch /thus/them/XC
BC ch /thud/thunk/XC

I often follow a practice of commenting such things thusly

BC ch /this/that/XC BC this is a comment#
BC ch /thus/them/XC BC this is another#
BC ch /thud/thunk/XC BC and so is this#

where the "#" at the end of each line is a 3 byte 0FFh character. The BCs
after the XCs causes the comments to be placed harmlessly on the command
line, where they are promptly overwritten by the BC on the next line. But
a 3 byte 0FFh at the end of each line is essential to keep the CR at line
end from doing its usual "execute" function, and trying to "execute" the
comments.

Anyway, Carl, I wanted to be sure that this technique is still valid in
XyWrite IV. Please let me know if it's not, and I will stop this practice.

Here's the readme from the release:
=================================================
This is a readme for the 1/13/2009 zip distribution for the XYENC program
and associated programs. You can get the zip file at

 http://basspad.com/xywrite/xyenc094.zip

XYENC.COM is a program for encoding XyWrite files, especially XyWrite
files containing XPL program content. There are 5 variants of the program,
which differ slightly as described in XYENCDEC.TXT. They differ mostly in
their assumption about the capabilities of the target environment -- for
example, some variants will produce output that can be handled on machines
that only support only 7 bit (rather than 8 bit) characters. When less
capable (e.g. 7 bit) environments are targetted, encoding is somewhat less
efficient (but the less efficient encodings can also be more readable,
particularly if you haven't memorized the displayable graphics for
characters in the 80h to 0FEh range).

XYDEC.COM is a companion program which "decodes" the encodings produced by
any of the variants of the XYENC.COM programs, to exactly regenerate the
original file. XYENC encoding, followed by a XYDEC decoding of ANY file
should produce the original.

QDF1.PM, QDF2.PM, and QDF3.PM are three XPL programs which can be used to
reformat XYENC encoded files for better readability. The transformations
done by QDF1, QDF2, and QDF3 do not affect one's ability to regenerate an
exact copy of the original file with XYDEC.

Native XyWrite XPL program can be relatively difficult to work with,
discuss, and manage, particularly in a modern world with mixed computing
environments. They can be viewed only with XyWrite (and even then, often
only with the appropriate XyWrite version). XPL programs are "executed" by
XyWrite very much like "machine code" is executed by a computer, with the
result that very, very little (if any) in the way of "extra" characters
can be added to an XPL program for purposes of visually "formatting" the
program for human readability -- so XPL programs tend to be nearly
unreadable in their native form. Even XyWrite hides most of an XPL
program's content (in "formatting" triangles) unless the programs are
viewed in "expanded display mode." This deprives people working with XPL
programs of what XyWrite normally does best -- to sensibly format data for
readability.

XYENC is a program which helps deal with these shortcomings, by encoding
(or recoding) XPL programs

1. so that the file can be handled and maniupulated in
  environments other than under XyWrite. XyWrite itself
  encodes much of its content -- especially XPL programs --
  with "3 byte encodings" of character-like entities, where
  each three byte encoding starts with a 0FFh or 0FEh byte.
  These encodings can cause havoc, appear as jibberish, or
  actually be garbled (with data loss) by environments such
  as email systems or any similar editing environment other
  than XyWrite. XYENC translates XyWrite encodings such
  that the programs can sensibly be transmitted, viewed,
  edited, and generally managed in nearly any computing
  environment.

2. so that the file can be sensibly formatted for display,
  whether it be under XyWrite or some other editing
  context. XYENC doesn't itself do any significant visual
  reformatting (such as indenting), but the encoding (a) is
  such that whitespace (spaces, tabs, CRLFs) can be added
  without effecting the XYDEC decode, so that formatting
  *can* be done, and (b) is such that encoded file would
  normally be viewed in XyWrite in "normal" (e.g., not
  "expanded") view, when XyWrite is used as the viewer.

QDF1.PM (Quick and Dirty Formatter) is itself an XPL program that takes
advantage of the fact that whitespace can be added to an XYENC encoding
without affecting the decode, and it "formats" a XYENC encoded file with
indentaton and similar whitespace artifacts to enhance reading and
understanding the XPL program code. QDF1 formatting does not affect the
ability to decode the file back to the original, using XYDEC.

QDF2.PM and QDF3.PM are additional XPL programs to do additional
formatting enhancements to XYENC encoded content, for additional
readability. But, unlike QDF1.PM, they do so with the assumption that the
target viewing environment is XyWrite only, rather than an ongoing,
any-editor-will-do, mixed computing environment. QDF2 and QDF3 formatting
also does not affect the ability to decode the file back to the original,
using XYDEC.

I think these programs are useful for people who create and work with XPL
programs.

In this zip collection, I have "lifted" two help frame programs out of the
XYWWWEB.U2 collection from users.datarealm.com/xywwweb, to use as samples. A
have named these file:

 ENCODE.PM
 ATTRIB.PM

To get an idea of how these programs might be useful, I suggest the
following:

First, please look at ENCODE.PM with XyWrite IV. Notice that you must use
expanded mode to look at the program. Assume that program isn't working
correctly, and that your goal is to understand it, so that you can fix it.
I think understanding such programs, by other than the author, is a
considerable effort.

Then look at ENCODE.ENC with XyWrite IV (or XyWrite III+). Note that you
should NOT be using expanded mode. Consider whether you think that this
would help you more quickly understand the workings of ENCODE.PM. I think
the ENCODE.ENC representation makes understanding considerably easier.

Then keep in mind that ENCODE.ENC has been generated from ENCODE.PM by
four quick steps (XYENCLH, QDF1, QDF2, QDF3), plus a small amount of
"touch up". (The main additional "touch up" was done on very long lines --
some line splits in the header, the initial SV01, the later SV15, and one
long IF clause).

With a little experience, programs such as ENCODE can also be updated (or
even created) in their XYENC encoded form, at which point

 XYDEC ENCODE.ENC [filename]

will reproduce a correspoding updated copy of ENCODE.PM.

PLEASE NOTE: this is a change from what was distributed with version 0.93.
In version 0.93, the "reformatting" in the encode.enc file (e.g., the
indenting) was done entirely "by hand" (and largely incorrectly, I might
add). In the current distribution, the ENCODE.ENC file is as formatted by
QDF1, QDF2, and QDF3, with only a small amount of "touch up."

ATTRIB.ENC has been created from ATTRIB.PM in the same way.

ENCODE.TXT is ENCODE.PM run simply run though the XYENC program. This
produces an ASCII file with line length <= 63, for emailing XY XPL files
or other XyWrite files.

Files in this zip include:

 0readme.txt - this file
 0changes.txt - list of changes/additions
 xyenc.com  - the encoding program, line length 60-63
 xyencnb.com - the encoding program, no line breaks
 xyencl.com  - a variant of the encoding program
 xyench.com  - another variant of the encoding program
 xyenclh.com - another variant of the encoding program
 xydec.com  - the decoding program
 xyencdec.txt - detailed technical documenation for the
         above programs, XyWrite III+ format
 qdf1.pm   - an XPL program for formatting XYENC output.
         See the 0changes.txt document for more
         description.
 qdf2.pm   - an additional XPL program for formatting
 qdf3.pm   - and another
 xyenc.lst  - a partial assembly listing of the XYENC
         program.
 xydec.lst  - a partial assembly listing of the XYDEC
         program
 inout.lst  - the portion of the assembly listing that
         has been removed from both of the above
         listings. This part is is from an "include"
         file, and is identical for both of the
         above, so it has been factored out. This
         part is the parm parsing and the file
         input/output portion of the programs, and
         generally isn't necessary if your interest
         is the tranlation algorithms.

         You can regenerate the include file and
         both source files from the above, by
         editing the above and deleting a 32
         character wide rectangle on the left. But,
         before doing that, there are a few lines of
         macro expansion, marked by a '1' in column
         30, near the bottom of XYENC.LST and
         XYDEC.LST, that you should delete first,
         since the macro ('hookcode') will
         regenerate those lines.
 qdf1.src   - my archive/reference "source" file for
         QDF1.PM, used to produce QDF1.PM via XYDEC.
         It will help me figure out what I did in
         QDF1, three years from now. It will also
         help you understand QDF1's operation, if
         you care.
 encode.pm  - a sample program from xywwweb. Discussed
         above
 encode.enc  - a XYENCLH encoding of the above, as
         formatted with QDF1, QDF2, and QDF3
 encode.txt  - an XYENC encoding of ENCODE.PM for email
 attrib.pm  - a sample program from xywwweb
 attrib.enc  - a XYENCLH encoding of the above, as
         formatted with QDF1, QDF2, and QDF3

Wally Bass