Post-mortem for Mnemonic, by Kasper Peeters ----------------------------------------------------------------------- These are some notes describing the mistakes made when we were designing and writing Mnemonic, plus some ideas on how to do it better (some of those ideas have meanwhile been implemented elsewhere, see the links). 1. We went overboard with the idea of designing a message-based infrastructure. We were aiming for something that was both type-safe and extremely flexible, ending up with a very clever but extremely non-standard method, namely to use C++ templates to get the job done. Because of this, the build process was unfamiliar to many people and the core code was too difficult to understand for most developers. With hindsight, we should have accepted a less type-safe mechanism. I am now of the opinion that we should have stayed with the standard method that made Unix so succesful: ascii streams through pipes. See http://www.damtp.cam.ac.uk/user/kp229/modglue/ for more on this idea. In particular, using ascii streams makes it dead-easy to manipulate the data flowing between various modules, something which is always going to be a mess when using a binary interface (corba, dcop, com, you name it). 2. The layout library had an internal structure based on TeX's idea of boxes and glue (mainly since I intended the typesetter to be able to typeset complicated maths). Since the w3c box model differs substantially from the one of TeX, this made it hard to get robust rendering (although I am still reasonably happy with the way it came out eventually, it was definitely pretty snappy for a first attempt). For the same reason, people have always had problems designing html typesetters with TeX. There are simply too many situations where kludges are needed (TeX's mechanism to break paragraphs into lines is very good, but it constantly interferes with the much w3c idea of how things should be done). Based on this, I concluded that one should have layout trees which reflect the layout specifications, preferably one-to-one. Use an internal TeX-like structure if you want to typset TeX, and use a hierarchical box tree if you want to typeset XHTML/XML+CSS. 3. We believed too much in the idea that we had to write a full "Netscape and co." replacement. We were constantly trying to catch up with the w3c specs (which, by the way, suffer from a serious problem: they get published without a reference implementation that shows that it is actually feasible to implement them. Some of the requirements in the CSS standard are completely idiotic, for instance the ability to have combined scrolling and fixed-position elements together with transparency; nobody can implement that completely without graphics hardware support). Having to play catch-up with the big browsers also meant that we had to understand their oddities. Try to figure out how the table algorithm of the old netscape works, it's not funny (and definitely not standard either). My goal has always been to write a browser which I can use for my daily work, which means accessing and reading the papers at the Los Alamos archives http://arXiv.org/. I was on the team because I was looking for a tool that allowed seamless integration between browsing the HTML part and browsing the TeX part of that site, including hyperlinks in the TeX part; see also the hyperTeX pages at http://arXiv.org/hypertex/ (if you want to know more about what I have in mind, contact me privately). We should have focussed on that, instead of twisting our minds trying to see how e.g. a javascript implementation could be put on top of our typesetting library. It is never good to write a replacement for something that already exists. Think of some completely new approach or a new requirement that cannot possible be met by the existing tools, and implement that instead. Having said that, the four years of browser development (since the start of the Mnemonic project) have still not made the above "scientific browsing" possible, despite MathML and vastly more powerful computers. The big browsers essentially didn't change much at all, they just got bigger and more flashy (who ordered "skins"?) but the actual work that you can get done with them has not changed much at all. The PDF viewers popping up to read papers from arXiv.org on todays computers are actually at least an order of magnitude slower than the DVI viewers on four year old machines. 4. Although we achieved separation between internal logic and GUI, this was done (to some extent) through the use of the "factory" paradigm (ie. a "layout factory" class existed which you could ask to create a "text box"; depending on the implementation this would be a gtk-based object or an ncurses one). This effectively just increased code size and made things more difficult to understand. The real way to do this, as I see it now, is to have a complete separation between the display part and the layout part, to the extent that they communicate through a page description language written in ascii, like Postscript or PDF (although they are too bulky for this purpose, something lighter like DVI is needed). I have meanwhile drafted such a human-readable, human-editable, yet compact page description language, and started work on a viewer for it. As a bonus, such a viewer can be used for other projects as well. 5. Which brings me to the last point: we were trying to do everything in one go: design the typesetter, the browsing gui, the xml parser, the graphics loaders, and the networking library. For several of these, perfectly good standard tools exist (for instance, fetching URLs really can be done with wget without any problems if you accept the unix stream idea; there is no need to re-invent the wheel here; see also the modglue pages referred to above). There is a general tendency to write huge programs that are intended to do everything at once, when perfectly fine command line tools already exist that do the job (and are usually much more stable and powerful). Mozilla is an example of this, but so is KDE (it is a mistake to think that it is a good thing to rewrite standard Unix tools in the form of "components" which can then only be used within the chosen component framework. Unix is already a component framework). Some of the code that was originally in Mnemonic has meanwhile been taken out, cleaned up, and put on the net in a form that can be used separately from the rest: oil-xmlhandler: Now available in the form of the wwtxt library, http://www.damtp.cam.ac.uk/user/kp229/wwtxt/ containing just the xml and css parsers and nothing else, in the form of a C++ library. Provides a tree structure like in lib-dom. lib-url: Available as liburl, http://www.damtp.cam.ac.uk/user/kp229/liburl/ lib-dom: See above. The wwtxt library makes use of the tree.hh templated tree class, which was inspired by lib-dom, http://www.damtp.cam.ac.uk/user/kp229/tree/ lib-layout and co: The layout routines will be available at some point in the form of a XML+CSS -> page description language converter. A viewer for this language is in preparation as well. Contact me if interested. oil-csshandler: The code that deals with dimensionful numbers (e.g. "5 cm", "3 pt" and that sort of stuff) has been split off completely and is now available in a general purpose C++ library called dimnum, http://www.damtp.cam.ac.uk/user/kp229/dimnum/ lib-core: The scripts used to create C++ shared libraries, as well as the timer and threads code, has gone into my proj++ library, http://www.damtp.cam.ac.uk/user/kp229/proj++/ Altogether, we spent quite a bit of time on this, and a lot of code is now ready for the trashcan. But it was still fun! ;-) Kasper Peeters 10th March 2002