gEDA: The "scriptEDA manifesto"

[Ales Hvezda <ahvezda-at-NOSPAMseul_dot_org>]

[ Ales here, I'm reposting this since majordomo didn't recognize the
  e-mail as being subscribed to the geda-dev mailinglist. ]

-- Cut here --
From: "R. Timothy Edwards" <tim@stravinsky.jhuapl.edu>

Hello everyone,

   I've had the opportunity recently to play around a bit with the idea of
"scriptEDA", which is one of those ideas which is amazingly simple provided
that the resources exist to make it happen.  In this case, the person to
recognize that the resources are in place to make possible the integration
of many or all of our disparate open-source EDA tools is Pinhong Chen at
UC Berkeley.  If you have not heard of "scriptEDA", I urge you to visit
his website at http://www-cad.eecs.berkeley.edu/~pinhong/scriptEDA/, and
read the several papers you can download from there.

   The following text is some thoughts I had on an implementation of
scriptEDA which would include the major open-source EDA tools that I know
of and/or am involved in the development of.  Please keep in mind that
this is just a collection of first impressions, and the main thing I would
like to get out of it, at the moment, is feedback and opinions.  I would
like to see the open-source EDA movement become an integrated system to
rival the big EDA companies like Cadence and Mentor Graphics; but it should
be the Bazaar, not the Cathedral, as Eric Raymond would say, as befits the
open-source software movement in general.  I think the scriptEDA idea may
be the right path to take to get there.

						Tim Edwards
						January 15, 2002

The scriptEDA "manifesto" v0.1
-----------------------------------------

"ScriptEDA" is a general-purpose method for integrating various CAD
tools, envisioned by Pinhong Chen, U.C. Berkeley.  The idea is that
many EDA tools have their own command-line interfaces, often very
spare, usually with a unique syntax, and rarely offering capability 
beyond executing internal commands and changing the application's
internal state.  If all of these programs can be "wrapped" by an
interface to a well-known interpretive scripting language, then they
can have both a consistent interface, and a method for sharing data
with other EDA programs.  The interface generator known as "SWIG" is
capable of generating wrappers for the major scripting languages,
including Tcl, Perl, and Python  (I have played around with SWIG,
and I have found that the "stable" version 1.1p5 chokes up on
X11-based programs, but that the development version 1.3.10 breezed
through the entire source of XCircuit, generating a 1MB+ source file
which compiled (eventually, after I climbed the learning curve) without
a hitch).

Visit the SWIG website (www.swig.org) for details on the implementation
of SWIG, but in a nutshell, what SWIG does is to take a standalone
program and a ".i" file which describes which routines should be
accessible when called from an interpretive scripting language, and
writes the bits of C code necessary to interface between the syntax
of the scripting language and the indicated program subroutines.  The
program's main() routine is ignored, and the rest of the program
becomes a shared object (.so) library file, a module for a scripting
language like Tcl, Perl, or Python.  Instead of being run from a
shell, the program is run from the scripting language interpreter.
This does not slow the program down in any way, as the only change
from the program's standpoint is that its entry point has changed.
There is some impact on memory usage, as the interpreter itself takes
memory, and so does the wrapper code.  However, these are what I
consider to be necessary tradeoffs for suddenly having an easy way
to integrate all programs which run under the same interpreter.
The interpreter has direct access to the internal data structures
of the program through the selected subroutines made visible to it.
This makes the program extendible through scripts, and provides a
simple way for programs to share data with one another.  It also
provides a way for users to access low-level functions in a program
without messing with the source code.  And finally, it allows users
to create extensions to a tool in whichever scripting language they
like best, so they don't have to stumble along, trying to learn
the vagaries of the syntax of the scripting language that happens
to be the favorite of the tool's developer (like Python for me, or
Scheme for Rajit. . .).  Although some of the resulting code will
probably run much slower than a user would want it to, it's much
easier and more practical to start with a working script, then
code the equivalent directly into the source to make it run fast,
than it is to try to code that function directly into the source
to begin with (particularly if the script communicates between
two different tools).

Insofar as many EDA programs are command-line driven, the wrapping
method is quite simple.  However, a number of critical EDA programs
are GUI-driven, and may or may not have a command-line interface in
addition to menu functions and direct GUI input (mouse or keyboard;
e.g., X11 events).

What I would like to have is a document concerning the implementation
of all EDA programs, including those without existing command-line
interfaces, under a "scriptEDA"-type environment.  It need consider
only open-source programs, that is, those which can be readily modified
to fit into the scriptEDA system.  Of major significance will be:

	Magic (VLSI layout),
	PCB (printed circuit board layout),
	SPICE (analog simulation),
	rsim (digital simulation),
	gemini (LVS),
	xcircuit (schematic capture), and the various components of
	gEDA (gschem, acs, etc.).

What's written below is not specific enough to be considered an API.  It
is more of a overview of the things that an API is going to need to consider
in detail.

The overall intention is to create a system with the following properties:

1) Every component of the system can be run independently (that is, without
   being launched from within the scripting environment), or as part of
   scriptEDA.  At compile time, each program will generate one or more shared
   libraries of its internal subroutines.  The standalone executable will only
   define the main() routine and any routines defining a command-line interface
   which are not used under the scriptEDA system.  Preferably, the shared
   library itself should be divided into several parts, one core part used
   by the standalone version and all of the interpreters, and one shared
   library for each interpreter compiled (Tcl, Perl, Python).

2) In conjunction with part (1), at compile-time, each component will allow
   compilation for any of the interpreters which are declared as part of the
   API, as well as (optionally) compilation of the standalone program.
   Supported interpreters will probably be limited to Tcl, Perl, and Python,
   as the major interpretive scripting languages used today for EDA work.
   Because SWIG is defined independently of all interpreters, compilation
   can be done for any interpreter supported by SWIG, but the "glue" scripts
   which hold the system together will be developed and supported only for
   the major languages defined by the API.  This, of course, can be changed
   at any time, as warranted.  The reasoning behind this flexibility is that
   the long-term existance of any one interpreter cannot be guaranteed, but
   as all of them are capable of the same essential functions, translation
   of scripts between different languages is fairly easy to maintain.

3) Each component (application) is free to be developed with minimal
   constraints as to how it must act within the scriptEDA environment.
   The scriptEDA system may not force the reimplementation of a program's
   GUI or internal data organization.  However, recommendations may be
   made which suggest development directions which enable better
   compatibility with other programs in the scriptEDA system.  An
   application may cede its GUI to the interpreter if requested, but is
   not forced to do so.  The reason that this is not made a requirement
   in the API is that many applications have GUI operations so tightly
   integrated into the program that separating them out would not be
   feasible except maybe as a long-term project (e.g., xcircuit and PCB).
   The degree to which a tool is willing to cede its non-time-critical
   functions to the interpreter will likely have a direct impact on its
   success within the scriptEDA system.

4) The scriptEDA system may not *require* a particular component.  For
   example, both gschem and xcircuit are viable schematic capture
   programs;  a designer may find gschem better for digital and PCB
   design, and may find xcircuit better for analog and VLSI design.
   One tool should be able to appropriately replace the other within
   the scriptEDA system.

-------------------------------------------------------------------------
Operation

The "top level" of scriptEDA consists of starting the interpreter.  The
interpreter, in turn, generates a thread for each application program,
and manages data transfers between all of them.  Creation of threads may
be from the interpreter command line (e.g., a call to "magic" with
arguments), or it may run a GUI as part of the master thread (e.g., a
windowed environment with file load/save dialog boxes and such, and
perhaps tabs to switch between application windows).  The application
program might relinquish all of its callback operations to the controlling
GUI if requested.  The application will want to keep time-critical
operations to itself.  It will be necessary to have a naming convention
for calling tools from the (each) interpreter.

Interactive programs will require some method to return to the calling
interpreter.  Running each application as a separate thread gives a bit
more flexibility in choosing the communication method between the
interpreter and the applications.  Programs may communicate data directly.
The master thread may make calls to query the internal data structure of
any program at any time.  This means that every application program will
need to acquire and release mutex locks when altering data which may be
visible to another thread.  Another method would be to have each
application program accept a particular signal (interrupt or event such
as an X11 event) which will cause it to suspend execution until receiving
another signal, during which time another thread could access its data.

An example of this is provided by a recent demonstration of ng-spice
inside an interpreter environment.  Rather than run SPICE as a "batch
job" and require ".print" statements on all nodes to be queried at
run-time, generating huge output files, the calling interpreter
simultaneously runs ng-spice and a graphics plotting program.  There
are several possibilities here.  SPICE can run as usual, with a ".print"
statement, but with the interpreter redirecting SPICE's stdout to itself,
reading the output, and passing it as input to the graphics plotting
program.  Alternatively, SPICE could send one value (time, most likely,
or just a boolean value handshaking flag) and stall while the interpreter
reads its data structures directly and passes the information to the
plotting program.  If speed is of the essence, then a third possibility
assumes that the plotting program is written specifically to parse SPICE
output, and the interpreter is then responsible for linking the output
of SPICE to the input of the plotting program via a pipe, and interrupting
SPICE whenever a change is needed in the simulation parameters.

To take this one step further, we imagine an example in which we launch
four application programs:  magic, for VLSI layout; SPICE, for simulation;
the plotting program, for visualizing the simulation output; and ext2spice,
which generates SPICE decks from magic layout.  The controlling interpreter
is running a GUI which has a tab interface, from which we choose "magic" as
the active input window.  The GUI also has menu, which offers a selection
"simulation" with items "start", "stop", "reset", "add node", and "remove
node".  On choosing "reset", the interpreter sends an "extract all" message
to magic, and then launches "ext2spice", which generates a SPICE deck, which
is passed to the simulator as input (or, better, ext2spice can be rewritten so
that it acts as a filter function between magic and SPICE, avoiding any file
creation at all).  In magic, we select a node using the usual magic selection
mechanism.  Then we choose menu item "add node", which sends a "getnode"
command to magic, grabs the output, then sends a "print" instruction to
SPICE.  On choosing "start", SPICE begins running, and passes values for
the specified node to the plotting program for output.  Conversely, we
can activate the plotting program window, highlight one of the traces
being plotted, and execute a "go to node" function in magic which will
move to and highlight the selected node.  The voltage value of the node
at the current simulation timestep could be printed in magic as it
executes.  In a way, this functionality is similar to "irsim" in Magic,
except that components such as graphical output can be added without
altering the source code of magic, and the entire simulator can be changed
(repacing "ext2sim" with "ext2spice" and "irsim" with "SPICE", or ACS, or
analog/diglog, or whatever).  In addition, the interface is smoother
("ext2sim" or "ext2spice" run automatically, and the tech file(s) needed
by irsim or the models needed by SPICE can be determined from the internal
state (technology data structures and path specifiers) of magic) than the
existing "rsim" interface in magic.

Finally, an added bonus to this method is that the "rsim" interface can be
completely removed from magic, reducing some of its excessive `code bloat.'
The same is true for basically ANY feature of an EDA tool which is better
developed apart from the core program.  This applies (in my opinion) to
all of the following tools in magic:  The routers (all of them!), the
wire tool, irsim/rsim and the rsim tool, the netlist tool and window, and
the color window.

-------------------------------------------------------------------------
Development

Implementation of scriptEDA around programs which are inherently command-line-
interface oriented and which do not have a GUI is straightforward and follows
the recommendations outlined by Pinhong Chen.  Entire command sets are
wrapped such that they are translated directly into scripting language
equivalents.  Program output is captured from stdout where required.
Programs with more complicated interfaces probably need to be considered on
a case-by-case basis.  For each program, it will be necessary to determine
which functions need to be accessed by the interpreter, and which may be
considered "private" functions.  If SWIG is told to process every single
routine, it will generate a wrapper for every single routine, and the
resulting C code source file can be larger than the rest of the program
combined.  By choosing judiciously which functions should be accessed, the
interpreter wrapper interface can be kept reasonably small and efficient.

Magic:
Magic already has an embedded "scheme" interpreter.  Since "scheme" can be
chosen at runtime, it is easy enough to disable it when compiling for
scriptEDA.  The main problem with magic is its incredibly complicated
I/O interface, using a forked process to handle X11 calls, such that it
gets inputs from dual sources: the calling shell, and the X11 event queue.
To further complicate things, a choice of graphics interface can be compiled
in and selected at runtime, so the X11 interface may be bypassed altogether.
Probably the best thing to do with magic is to leave its X11 interface in
place and replace the interface to the terminal with a connection to the
interpreter.  So the function which normally blocks on a "select" function
waiting to get input will be rewritten for scriptEDA to simply return to
the interpreter.  The interpreter, not TxDispatch(), will call magic
functions.  In fact, the entire "textio" interface will probably be missing
from scriptEDA magic, with the TxDispatch() calls handled as specified in
Pinhong Chen's scriptEDA "Case Study".  It is probably not wise to tinker
with the XHelper7 process, namely because X11 is not thread-safe, and if
operated as a thread, it likes to rearrange X11 calls until it confuses
itself to death.  I am not sure yet how to deal with command line entry
from key events grabbed through the XHelper7 process.

XCircuit:
XCircuit is another program with an embedded interpreter but fortunately,
like Magic, it can be compiled without it.  However, "without it" means
that XCircuit runs its own pitiful scripting language.  The current
development direction for XCircuit is to get rid of the embedded
interpreter in favor of the scriptEDA extended interpreter, which would
then do a lot of file handling itself, including handling of netlist
formats and disk I/O.  Because XCircuit is inherently GUI-oriented, and
not command-line oriented, it will need to add a method to return back
to the interpreter for command execution and scripting-language file
execution.  Because the widget operations are horribly intertwined with
the rest of the code, getting XCircuit into a state where it can
relinquish control of the GUI to the interpreter, when requested, will
require a LOT of work.  Meanwhile, its major task within scriptEDA will
be for netlisting and LVS in conjunction with magic and PCB.  When LVS
is confirmed, it can receive simulation feedback in the form of voltage/
current/digital signal values printed by specified nodes as the simulation
progresses, and highlighting a node can simultaneously highlight the
equivalent node in the layout and the trace for that node in the
simulation.  If LVS is not confirmed, separate simulations can be run
on the schematic and the layout and compared.  LVS itself should provide
feedback directly into the layout and schematic programs.

Gemini:
Gemini is an LVS program.  When LVS is required, it will take input from
the layout and schematic capture programs, and attempt a match.  Because
text output from LVS is inscrutable at best, it will be preferred for
the interpreter to have direct access to the internal state of the LVS
program, from which it can extract the node name information necessary
to highlight those parts of a circuit which are believed to be matched.
Feedback can be used to resolve automorphisms (assuming anyone cares),
and selection of a node in either the layout or schematic will highlight
the equivalent node in the other.  The main reason for direct access to
data structures is that while net names can be requested and shown in
layout and schematic applications, device names (in VLSI; that is,
transistors, resistors, capacitors, etc.) are generated by the LVS
program on the fly.  Gemini documentation suggests that at least one
version of gemini knows how to write feedback layer information in the
form of a magic ".mag" file.  I have never seen this part of the code,
but it should be recovered or rewritten.

SPICE:
ng-spice has already been modified to run in a scriptEDA environment.
It is probably necessary to do some more work to give the calling
interpreter greater control over internal routines, such that
experimenting with ways to overcome convergence difficulties could
be tried as scripts.  Too much scripting control is probably not
advisable, because SPICE needs to run as fast as possible.  The nice
thing about scriptEDA is that it has the potential to SPEED UP 
simulations by bypassing all the disk accesses required by lengthy
disk operations.

PCB:
PCB would benefit from transferring all of its rats-nest and netlist
capabilities to the controlling interpreter.  This would allow easier
integration with a schematic capture program.  PCB does not have a
command-line interface, so it will need a way to return to the
calling interpreter or else allow parallel access to its data
structures using handshaking.  A command-line interface will be
more-or-less built in the interpreter, with occasional code changes
needed to support the interface and/or make it consistent and clean.

rsim/irsim:
I have not looked much at the irsim code, but since it is run
entirely by command-line interface, wrapping it into scriptEDA
should be straightforward and follow the procedure outlined in the
scriptEDA papers, with little or no modification to the source.
As mentioned above, though, it is its interface to magic, written
into the magic source, which should be removed and transferred to
the interpreter.

Mixed-mode simulation:
An interesting question is whether the scriptEDA system could actually
break up a circuit into digital and analog parts, and run each part
separately as a SPICE or irsim simulation, with each part passing its
outputs to the others at each timestep, and the interface controlling
the D/A and A/D conversions and the matchup of timesteps.  It's an
interesting possibility.  I've always been annoyed that some mixed-signal
simulations have to run excruciatingly slowly because 95% of the circuit
is simple digital gates but has to be simulated in analog along with the
5% of the circuit that's truly analog.  If done correctly with threads,
the result should run efficiently in parallel on a multi-processor system.

gEDA:
I know very little about the impelentation details of the gEDA
application programs, so I will need Ales Hvezda and others involved
to make some comments about fitting them into the scriptEDA scheme of
things.  Please feel free to add comments and suggestions for individual
gEDA tools here.

-------------------------------------------------------------------------