README: Installation instruction for bibparse-1.04

Table of contents

• Jump start
• Introduction
• Installation
• Bug reports
• UNIX Systems
• IBM PC DOS
• Test suite
• Sample build output for UNIX
• Implementation Notes

Jump start

As with most GNUware, you can build, test, and install this program on most UNIX systems by these simple steps

csh et amici:
	setenv CC ...your favorite C or C++ compiler...
	./configure && make all check install

sh et amici:
	CC=...your favorite C or C++ compiler...
	export CC
	./configure && make all check install

Or in one line, if you have env (most modern UNIX systems do):

	env CC=... ./configure && make all check install

If you don't set the CC environment variable, then gcc (or cc, if gcc is not available) will be assumed.

If you wish to undo a make install, just do make uninstall; this will remove any files in system directories put there by make install.

See below for further details, and for instructions for non-UNIX systems.

Introduction

This directory contains bibparse, biblex, bibunlex, and bibdup, a suite of programs for support of BibTeX bibliography files. They are described in the following paper:

@String{TUGboat = "TUGboat"}

@Article{Beebe:TB14-4-395,
author =       "Nelson Beebe",
title =        "{{Bibliography prettyprinting and syntax checking}}",
journal =      TUGboat,
year =         "1993",
volume =       "14",
number =       "4",
pages =        "395--419",
month =        Dec,
}

The complete text of the TUGboat article is included with bibclean 2.08 and later versions.

You can find distributions of bibclean in a variety of formats in the same place that you found the bibparse (et al) distribution. The master location is ftp://ftp.math.utah.edu/pub/tex/bib. Via electronic mail, a message with the body

	help
	send index from tex/bib

to tuglib@math.utah.edu will get you started.

Three files, biblex.c, bibparse.c, and bibunlex.c, are generated by lex (or flex), and yacc (or bison). Copies of these files from the author's site are preserved in the save subdirectory for manual bootstrapping, but on UNIX systems, they should be generated automatically as part of the installation process.

Installation

Starting with version 1.03, bibparse has been adapted to use the GNU autoconf automatic configuration system for UNIX installations.

GNU autoconf is run at the author's site to produce the configure script from configure.in.

The configure script is run at each installer's UNIX site to produce Makefile from Makefile.in , bibdup.sh from bibdup.sin, and config.h from config.hin. The configure script is a large (2900+ lines) automatically-generated Bourne shell program that investigates various aspects of the local C implementation, and records its conclusions in config.h.

Should you do a make maintainer-clean [not recommended, except at the author's site], the configure script will be deleted, and you will need recent versions of both GNU m4 and autoconf correctly installed to reconstruct things, which can be done this way:

	autoheader	# Regenerate config.hin
	autoconf	# Regenerate unedited configure
	make configure.time-stamp	# To fix configure script

For convenience and safety, the distribution includes a subdirectory named save that contains read-only copies of the files Makefile, config.h, and configure created by autoconf and make configure, plus the files biblex.c and bibunlex.c generated by lex, and the file bibparse.c generated by yacc. This will allow recovery from a lost or damaged configure file, and support manual bootstrapping on systems that lack lex/flex and yacc/bison support, or have broken implementations of those programs.

Suitable hand-crafted config.h files are provided for non-UNIX systems, and in the unlikely event of a failure of the configure script on a UNIX system, config.h can be manually produced from a copy of config.hin with a few seconds of editing work. If you do this, remember to save a copy of your config.h under a different name, because running configure will destroy it. If you have GNU autoconf installed (the installation is very simple and source code is available from ftp://prep.ai.mit.edu/pub/gnu/autoconf/autoconf-x.y.tar.gz), (x.y = 2.13 at the time of writing this), you might try augmenting configure.in instead, then run autoheader, autoconf, and configure.

Thus, on UNIX, installation normally consists of just two steps (assuming a csh -compatible shell):

	setenv CC ...your favorite C or C++ compiler...
	./configure && make all check install

If you like, add OPT='your favorite optimization flags' to the make command; by default, no optimization flags are set.

The GNU standard installation directories /usr/local/bin for binaries, and /usr/local/man/man1 for manual pages are assumed. The prefix /usr/local can be overridden by providing an alternate definition for the make variable, prefix, on the command line:

	make prefix=/some/other/path install

After installation, you can do

	make distclean

to restore the directories to their distribution state. You should also do this between builds for different architectures from the same source tree; neglecting to do so will almost certainly lead to failure, because the config.cache file created by configure will lead to an incorrect config.h and Makefile for the next build.

Bug reports

Please report all problems, suggestions, and comments to the author:

UNIX Systems

The code can be compiled with either C (K&R or ISO/ANSI Standard C) or C++ compilers, although whether C++ can be used for the lex/flex- and yacc/bison-generated code depends on how modern their versions are. With some C++ compilers, it may be necessary to supply additional switches for force the compiler to stay in C++ mode, rather than reverting to C mode (e.g., on DEC Alpha OSF/1, you must do setenv CC "cxx -x cxx").

On UNIX systems, the only variables that you are likely to need to override on the make command line are the Makefile settings of CC and OPT, and if you wish to do make install, the setting of prefix.

If you are installing bibparse on a new system, you should definitely run make check before installing it on your system. Sample output of make check from a UNIX system is given below.

These programs have been built, tested, and validated with two independent lexers (flex and lex), and three independent parser generators (bison, byacc, and yacc), using all available C and C++ compilers on these 11 UNIX systems for the 1.04 release: 413 successful builds, and 49 failures. Because some compilers are stored in two different directories, or under two separate names, the actual number of independent builds is somewhat less.

While the large number of successful builds is gratifying, the failures are potentially even more important if they can be traced to deficiencies in my code. Close examination of the build logs shows that they are not: they are entirely due to one of the following:

• Code from lex/flex and/or bison/byacc/yacc which is not C++-compatible; there is no excuse for such vendor sloppiness, but it exists. C has had an ANSI/ISO Standard since December 1989, and the draft standard was stable at least two years before that, so the target language has been publicly defined for about a dozen years. All C-Standard-conforming code should compile without problems under C++ (barring use of C++ reserved words).
• Lack of compiler support for the non-standard stack memory allocator, alloca(), used in bison-generated code.
• Internal compiler errors (e.g., lcc on NeXT and SGI IRIX 6.x systems).
• Broken vendor-provided yacc (SGI IRIX 5.3).
• Broken vendor-provided header file, alloca.h (pgCC). A bug report about this to the vendor, The Portland Group (http://www.pgroup.com), produced a prompt response with the following patch:

  #ifdef __cplusplus
  extern "C" char * __builtin_alloca (int);
  #endif
  
  *** /usr/pgi/linux86/include/alloca.h   Thu Apr 22 18:57:51 1999
  --- ./alloca.h  Tue May 11 11:52:01 1999
  ***************
  *** 36,41 ****
  --- 36,44 ----
    
    # define __alloca(size)       __builtin_alloca (size)
    # define alloca(size) __alloca (size)
  + #ifdef __cplusplus
  + extern "C" char * __builtin_alloca (int);
  + #endif
    
    __END_DECLS
  

Successes

flex + bison (70 compilers)

flex + byacc (75 compilers)

flex + yacc (66 compilers)

lex + bison (67 compilers)

lex + byacc (72 compilers)

lex + yacc (63 compilers)

Failures

flex + bison (7 compilers)

flex + byacc (2 compilers)

flex + yacc (11 compilers)

lex + bison (10 compilers)

lex + byacc (5 compilers)

lex + yacc (14 compilers)

IBM PC DOS

Of all the compilers that I have available on the IBM PC under Microsoft DOS and Windows (two versions of Turbo C, four of Microsoft C, and two of Watcom C), none support 32-bit int data. This prevents the use of the yacc version of bibparse.c, because it requires an integer type that can hold values as large as -10000000. Had all such data been given a type defined by a typedef, a one-line change from int to long would permit the transfer of the code to an environment with 32-bit long support. Sadly, such is not the case. Perhaps the commercial Mortice-Kern MKS Tools package of UNIX tools on DOS addresses this issue, but I do not yet have it.

Another problem is that the parser tables are too big for the obnoxious 64KB-segment limit of the Intel x86 architecture, and none of the compilers, except for the Watcom 386 compiler, will compile the code.

Fortunately, the solution is simply to use bison instead of yacc. It has been designed to work with integers as small as 16 bits, and its parser tables are more compact.

The three programs can be built standalone in this environment, without the need for lex, flex, or yacc libraries, provided that the symbol HAVE_IBMPC is defined in the config.h file, and the symbol HAVE_FILENO is undefined there. This will enable a simple version of the yywrap() function, and also reduce the memory requirements to fit in the 64KB-segment limit.

The ibmpc/dos/README file contains details of the builds and tests of bibparse under several IBM PC DOS C and C++ compilers, and instructions for building and testing bibparse with other compilers.

Since bibparse uses no floating-point arithmetic, and PC DOS has no shared libraries, I expect that the executables will run on any version of DOS greater than 4.0. They may also run on earlier versions. At the time of writing, MS-DOS 6.22 is current, and the bibparse executables work fine on it. They also work on Caldera OpenDOS 7.01, the version supplied by Sun under Solaris 2.6 for the SunPC emulator, and on the Windows version of DOS under Windows 95.

Test suite

The bibparse distribution includes a comprehensive test suite which is run by

	make check

For use on other operating systems, the ibmpc/dos/check.bat file for PC DOS may serve as a useful starting point for preparing a script to run the tests.

Please do run the validation suite at your site before installing the program. Compilers are complex software systems that also have bugs, so the fact that the program runs correctly somewhere else does not mean that it will do so on a different system.

Sample build output for UNIX

Here is a log of a successful build on Sun Solaris 2.6 using the native C++ compiler, CC:

env CC=/opt/SUNWspro/bin/CC ./configure && make 
creating cache ./config.cache
checking for gcc... /opt/SUNWspro/bin/CC
checking whether the C compiler (/opt/SUNWspro/bin/CC   ) works... yes
checking whether the C compiler (/opt/SUNWspro/bin/CC   ) is a cross-compiler... no
checking whether we are using GNU C... no
checking whether /opt/SUNWspro/bin/CC  accepts -g... yes
checking for c++... c++
checking whether the C++ compiler (c++  ) works... yes
checking whether the C++ compiler (c++  ) is a cross-compiler... no
checking whether we are using GNU C++... yes
checking whether c++ accepts -g... yes
checking for compiler language... C++
checking how to run the C preprocessor... /opt/SUNWspro/bin/CC  -E
checking for alloca.h... yes
checking for stdlib.h... yes
checking for nawk... nawk
checking for chmod... chmod
checking for gzip... /usr/local/bin/gzip
checking for scp... scp -p
checking for rcp... (cached) scp -p
checking for cp... (cached) scp -p
checking for deroff... deroff
checking for diff... diff
checking for distill... /usr/local/bin/distill
checking for ditroff... no
checking for groff... groff
checking for etags... etags
checking for expand... expand
checking for jar... jar
checking for lex... lex
checking for flex... (cached) lex
checking for ln... ln
checking for ls... ls
checking for man2html... /usr/local/bin/man2html
checking for mkdir... mkdir
checking for mv... mv
checking for nroff... nroff
checking for rm... rm
checking for rmdir... rmdir
checking for sed... sed
checking for shar... shar
checking for spell... spell
checking for tar... tar
checking for touch... touch
checking for unzip... unzip
checking for yacc... yacc
checking for zip... zip
checking for zoo... zoo
checking for fileno... no
checking for alloca in -lPW... no
checking for lex/flex library... -ll
checking for yacc library... -ly
checking for bison flags...
checking for preprocessor symbols needed on Sun Solaris...  -D__EXTERN_C__
updating cache ./config.cache
creating ./config.status
creating Makefile
creating bibdup.sh
creating config.h
yacc -v  bibparse.y

conflicts: 6 shift/reduce
sed -e 's/char *yytext *\[YYLMAX\]/char yytext[40960]/' -e '/static void __yy__unused() { main(); }/d' -e 's/#[         ]*define[       ]*YYLMAX.*$/#define YYLMAX BIBYYLMAX/' -e '/extern[     ]*char[         ]*[*]getenv[    ]*[(][  ]*[)]/d' < y.tab.c > bibparse.c
rm -f y.tab.c
/opt/SUNWspro/bin/CC  -D__EXTERN_C__ -DBIBYYLMAX=40960 -I.  -c bibparse.c
lex -t biblex.l | sed -e 's/char *yytext *\[YYLMAX\]/char yytext[40960]/' -e '/static void __yy__unused() { main(); }/d' -e 's/#[       ]*define[       ]*YYLMAX.*$/#define YYLMAX BIBYYLMAX/' -e '/extern[     ]*char[         ]*[*]getenv[    ]*[(][  ]*[)]/d' > biblex.c
136/1000 nodes(%e), 311/2500 positions(%p), 50/500 (%n), 3095 transitions,
188/10000 packed char classes(%k),  230/3000 packed transitions(%a),  265/6000 output slots(%o)
/opt/SUNWspro/bin/CC  -D__EXTERN_C__ -DBIBYYLMAX=40960 -I.  -Dyylex=xxlex -o bibparse bibparse.o biblex.c  -ll -ly
/opt/SUNWspro/bin/CC  -D__EXTERN_C__ -DBIBYYLMAX=40960 -I.  -o biblex lexmain.c biblex.c  -ll -ly
lexmain.c:
biblex.c:
lex -t bibunlex.l | sed -e 's/char *yytext *\[YYLMAX\]/char yytext[40960]/' -e '/static void __yy__unused() { main(); }/d' -e 's/#[     ]*define[       ]*YYLMAX.*$/#define YYLMAX BIBYYLMAX/' -e '/extern[     ]*char[         ]*[*]getenv[    ]*[(][  ]*[)]/d' > bibunlex.c
/opt/SUNWspro/bin/CC  -D__EXTERN_C__ -DBIBYYLMAX=40960 -I.  -o bibunlex bibunlex.c  -ll -ly

The following checks should produce no output, other than their names
lex001
lex002
lex003
lex004
lex005

The following checks should produce no output, other than their names
parse001
parse002
parse003
parse004
parse005
parse006
parse007
parse008
parse009
parse010
parse011
parse012
parse013
parse014
parse015

The following checks should produce no output, other than their names
unlex001
unlex002
unlex003
unlex004
unlex005

Implementation Notes

While the GNU autoconfigure system is a great boon for end users, it is a painful system for developers to implement for their code. Despite already having adapted several other programs to use this system, I spent two long and tedious days at bibparse 1.03 experimenting with settings in the configure.in file before the tests succeeded on a large enough number of systems that I can have some confidence that the installations will go smoothly at most end-user sites.

At version 1.04, when I ported bibparse to even more systems, and a new parser generator, byacc, it took another four days of tuning the configure.in file to handle system peculiarities, notably on GNU/Linux.

A major problem with autoconfigure is that it is not sufficiently C++-aware. The simple test files that it creates for probing features of the host implementation lack correct prototypes, and compilation failures then lead to incorrect setting of values in config.h and Makefile.

On most of the test systems, the vendor-provided lex is not C++-aware either, so that compilation failures from missing prototypes, or inconsistent or old-style function declarations, are common.

One vendor, HP, generates a reference to an non-standard external header file, yacc.h, in yacc output, but the location of that header file is unknown to g++. In any event, that file is wrong anyway, because it declares yyparse() to have C linkage, but the version in the generated C file will be compiled with C++ linkage. This was solved by created a temporary empty yacc.h file in the rules for bibparse.o in Makefile.in.

The same vendor, HP, generates a call to main() in lex output which then fails to match the prototype, and causes compilation failure. This was solved by an edit step in Makefile.in to removed the offending line.

Another problem for C++ use is that, on some systems, the normal lex functions input() and output() must be renamed to avoid conflicts with uses of those names in C++ header files. On Sun Solaris 2.x, they become lex_input() and lex_output(), so bibyydcl.h has to handle this case specially. I don't see any good way to do this with configure, since it is very awkward to produce tests in languages other than C.

The GNU/Linux (Caldera Light 2.0.30) system on which I tested bibparse 1.03 had yacc, but not its library, -ly, so a more complex test had to be generated in configure.in to handle that irregularity.

The RedHat GNU/Linux systems on Intel Pentium and Sun SPARC hardware on which I tested bibparse 1.04 have lex, but not its library, -ll, and worse, that lex is really flex, which does not behave compatibly, unless it is invoked with the compatibility-mode flag, -l. However, that flag cannot be used with ordinary lex: on HP systems, it produces a warning; on IBM systems, it produces garbage in the output C file; on SGI systems, it causes lex to hang in an apparent I/O wait. It took several tries on various systems before I settled on the code finally used in configure.in to handle these variations.

On SGI IRIX 5.3 systems, there is a bug in yacc which results in a bogus complaint

UX:yacc: ERROR: Redeclaration of precedence of TOKEN_EQUALS, line 210

Thus, on that system, either bison or byacc (Berkeley yacc) must be used, and neither are provided by the vendor.

You can get bison at the Free Software Foundation archive (or one of its several mirrors): ftp://prep.ai.mit.edu/gnu/bison/bison-1.27.tar.gz,

The most recent version of Berkeley byacc that I could find is at the Cygnus FTP site: ftp://ftp.cygnus.com/pub/gdb/infrastructure/byacc-940324.tar.gz That version has been make to run with an early version of GNU autoconfig, and has bug fixes beyond the byacc version 1.9 of 22-Feb-1993 that I found at the Berkeley BSD development site: ftp://ftp.cs.berkeley.edu/pub/4bsd/byacc.1.9.tar.Z Even though the Cygnus version is newer (23-Mar-1994), it produces code that did not compile under C++ until I made this patch:

===================================================================
RCS file: byacc-940324/byacc/RCS/skeleton.c,v
retrieving revision 1.1
diff -r1.1 byacc-940324/byacc/skeleton.c
98a99,102
>     "#ifdef __cplusplus",
>     "extern \"C\"",
>     "#endif",
>     "    extern char *getenv();",
105,108d108
<     "#ifdef __cplusplus",
<     "extern \"C\"",
<     "#endif",
<     "    extern char *getenv();",
===================================================================

On Sun Solaris 2.x, the symbol __EXTERN_C__ must be defined in order to expose certain function prototypes in the code generated by lex and yacc. configure.in has a special test for this system to generate the needed definition.

The linkers on some UNIX systems do not search /usr/local/lib by default, even though this has long been a standard location for locally-installed libraries. Consequently, the configure.in rules add this directory to the library search path when the flex library -lfl is required. If your site has a non-standard location for this library, where your linker cannot find it without help, then you'll have to override the LDFLAGS value when you run make, e.g.,

make LDFLAGS='-L/my/private/lib'