Interrim XYENC/XYDEC v0.95 release

[wbass@xxxxxxxx]

I've put up an interim release of the v0.95 XYENC/XYDEC modules at

 basspad.com/xywrite/xye094x.zip

This is v0.95 of the two executables. I haven't finished the documentation
or other work that I want to do for the 0.95 release, so only the two
executables and a version of this note are in the zip. Testing at this
point has been limited. For other files in the package, refer to the v0.94
release at this time.

Main differences:

1. The bug with short files uncovered by Harry Binswanger is
  fixed. I'm still annoyed at myself for my initial rather
  arrogant assumption that the problem was probably a
  cockpit error on Harry's part.

2. There is now only 1 XYENC executable. AFTER the two
  filename parms, it takes zero or more of several
  parms/switches:

   "nn" : makes it chop output lines at "nn" chars max (20
      to 99)
   "L" : makes it do what the old XYENCL module did
   "H" : makes it do what the old XYENCH module did
   "B" : makes it do Both (as does "L" and "H")
   "C" : makes XYENC include a "checksum" token in its
      output (discussed below).

3. XYENC line splitting is slightly improved. XYENC now
  knows how to split after an exclaim token. The actual
  variance in line length of the old module was supposed to
  be 3 max, but was actually 5 (because of the "!hh'AA"
  output of length 6 for an "improperly" encoded
  primitive). This gets it back down to 3.

4. The primitive name table has been extended to encompass
  the 41 new primitives in NB 8.0i (last one at
  0FFh,83h,49h), in addition to the 380 in XyWrite IV.

  NB has changed the XyWrite IV "MN" primitive (at
  82h,0ADh) to "M<" and added a new "MN" primitive (at
  83h,3Dh). To preserve compatability w/XY IV (which is my
  first priority), I swapped those, and left "MN" at the XY
  IV 82h,0ADh location while putting "M<" at 83h,3Dh. Also,
  I changed the NB primitive names that start with "à"
  (0E0h) to start with a '"' (double quote) rather than an
  "à", since generating a name with an "à" in it breaks
  XYENC's "all ASCII output" "promise." The NB primitive
  "-D" is shown by XYENC as "D-". The XYENC/XYDEC
  encode/decode "cycle" is not affected by any this, since
  XYDEC "expects" what XYENC does.

If you specify the 'C' parameter to XYENC, XYENC includes a "checksum"
token at the beginning of its output file. A checksum token is the same
length (20) as a date/time stamp token and looks like:

'0 checksum 472AB93E

where the last 8 chars are hex digits which represent a CRC32 "CRC check"
on the original file that was encoded. CRC32 is the same check algorithm
used by ZIP programs, and either program should give you the same check
digits for any given file. The chances of a CRC32 check NOT detecting a
data corruption error when one occurs are about 1 in about 4 trillion.

If XYDEC finds a checksum token in its input, then it does CRC32 checking
on the output that it generates, and makes sure that the generated CRC
matches the one specified in the checksum token. The checksum token must
be near the top of the XYDEC input (before any tokens that actually
generate output, but you could, for example, put comments before the
checksum token) or it will not be recognized. You can delete the checksum
token at any time to prevent XYDEC from doing the CRC check.

There are at least a couple of uses for the checksum.

Firstly, for something encoded for purposes of email tranmission and
delivery, it gives an end-to-end check of data validity, and will, for
example, detect errors introduced by the email system.

Secondly, the CRC token gives a user a quick way to insure that
modifications that he does to a XYENC encoded file -- adding comments,
formatting, whatever -- won't or didn't introduce any errors in the
ultimately decoded output from XYDEC. Doing "XYDEC infile . >nul" does a
quick check which produces an error message (requiring a keystroke
response to clear) if a change was accidentally made that will corrupt
XYDEC's output.

The checksum code has undergone some changes since its introduction. Both
XYENC and XYDEC are "1 pass" programs. XYENC therefore "wants" to (and
originally did) put the checksum token at the END of its output file,
since the program doesn't know the checksum value until then. But XYDEC
wants the checksum at the beginning of its input file, because CRC
generation is compute intensive, and XYDEC doesn't want to compute CRC
check data if/when there is no checksum token and therefore no CRC check
is being requested.

So XYENC was updated for XYDEC's benefit, and it now puts the checksum
token at the beginning of the output file. To accomplist that, XYENC puts
in a checksum "placeholder" initially, and then uses the random access
capabities of the file system to go back and fill in the CRC value at the
end of processing, before closing the file. When the 'C' switch is
specified, therefore, XYENC is no longer a simple "sequential" or "stream"
type program that is suitable for redirected or "piped" output. The 'C'
parameter should not be used in conjunction sending XYENC's output to
"standard out" or to a "pipe" -- if it is used in that setting, XYENC will
not update the checksum "placeholder", and the checksum token will not be
valid.

Suggestions/bug reports/comments welcome.

Wally Bass