The price of a LaTeX rewrite would be even higher: incompatibility. If one tries to rewrite TeX, the problem gets much much worse, since TeX is, for all intents and purposes, bug-free. A new implementation will certainly not be. Joel Spolsky said it very well
http://www.joelonsoftware.com/articles/fog0000000069.html
I disagree that the current situation is LaTeX developers' fault. TeX and LaTeX are complex pieces of software, that are developed over time. They were extended as the capabilities of their platforms increased, to take advantage of those capabilities. This requires full access to the OS utilities, and naturally LaTeX environment does that. If anything, this is Apple's fault. For whatever reason, they cannot allow applications to use the existing capabilities of the underlying OS. This goes against the Unix mindset; of course there will be unpleasant consequences, but one cannot hold Unix mindset responsible for these.
Unless I'm forgetting something from the article, the author never said it was the LaTeX developer's fault -- he simply said Apple was right for not allowing it in its current state. Not all situations are someone's "fault", and thus just because Apple is not at fault doesn't mean the developers are. It's just as you declared it, an unfortunate result of many decisions which were right at the time.
That being said, I also think you have a low bar for definitions of "bugs". Terrible performance is very much a "bug" in my book. Even the things that aren't "bugs" in the strictest sense can still make for bad software. The executable described here is 4GB, that's 1/4 thte total space on a low end iPad! Also, if tomorrow someone does discover a bug, it sounds like this would be a mess to fix. The jumble of source simply won't be attracting many new developers (whether it be to fix existing bugs, add new features, or port to new platforms).
The LaTeX executable isn't 4GB, it's 3MB or so (pdflatex/pdftex) or 16MB in the case of XeLaTeX. The 4GB consists of billions of extension/modules/packages for typesetting documents.
Yes I noticed that in the comments before I posted which is why I was careful to specifically say "The executable described here is 4GB" so as to refer to whatever it is he was trying to work with. I suppose whether all of the 4GB is needed for what he wanted, whether it could easily be taken apart, or could now be remade in a much smaller package, are all valid questions.
4G is a bit over the top. My install (TeXLive 2010 on OpenBSD) is ~2G, and that includes gobs of rarely-used packages and documentation.
Really, borrowing MikTeX's install-on-first-use for packages and making documentation optional should result in a much more reasonable size. If you want to get fancy, teach LaTeX how to decompress packages on demand.
Here comes the problem: Apple requires it to be a single binary, distributed once. You can't install packages the LaTeX ways should you want to, so you have to bundle most (if not all) packages into the single binary.
You could use a virtual machine library (if anything suitable exists) and run everything inside one executable - it would probably have horrid performance but you get rid of the single binary restriction. You can put on a nice front-end and tell the user how much longer he has to wait until the operation is complete.
Virtual machines are generally prohibited by the guidelines as well. (Exceptions exist for things like script interpreters, which need to be present for some games to run, e.g. Lua.)
I think the difference here is between interpreting scripts that come with the binary, and downloading new scripts from the internet (or other untrusted locations). The latter is prohibited.
To get around it, you'd have to make it In App Purchases, because these also come from a the App Store.
Apple wants to restrict the platform options available to preserve their user lock-in. They were sort of forced to make an exception for games because they had no alternative to offer.
It seems the cult of Apple is touchy today and can't detect a joke. I'm sorry if I have inadvertently hurt your feelings - I will be sure to clearly state my humorous intentions next time. </humour>
For the record, I do not own devices based on iOS or Android and I don't care for your little Apple-Google feud with which you seem to be so concerned with. So please, carry on with those market statistics about Android, I am dying to know more.
Well you obviously weren't careful enough. Nowhere in the article does he talk about a 4 GB binary or binaries. To quote him:"A 4GB Tex distribution dependant on over a 100 binaries" He is talking about the _whole_ distribution not a single or several binaries. So even though you were careful, you still misquoted the article.
"The App Store guidelines insist that any iPad app be a single executable. Jailbreaking would ease this restriction, but in this case Apple’s gatekeepers are right. A 4GB TeX distribution dependant on over 100 binaries is not acceptable on the iPad."
From that quote ("A 4GB TeX distribution dependant on over 100 binaries is not acceptable on the iPad."), it is clear that he is musing about a scenario where he ships all 4GB. As stated many times (and also in that same quote), on the iPhone you are forced to ship something as one executable. Thus, the theoretical distribution he is referring to would in fact be a single executable of about 4GB. In fact I think that's the whole point of those two sentences. So it appears that I was indeed quite careful.
> As stated many times (and also in that same quote), on the iPhone you are forced to ship something as one executable.
On the other hand, yes the executable must be "one". On the other hand you can download stuff (either directly or via DLC). The issue is that while Apple has relaxed their restrictions on bundling interpreters in applications (you can do that), they have not relaxed their restrictions on downloading interpretable code: it's not allowed. So downloading even styles would be verboten, since styles are TeX, and TeX is executable code.
I think you are missing the point. The apple guidelines do not state that a program can only consist of an executable. That would not make any sense. Rather they state that any application can only consist of a single executable in addition to whatever resource files are included. In the case of the Latex distribution in question one would have to convert all 100 executables into one executable, my guess would be perhaps 100 megabytes. In addition one would have to include lots of resource files, mainly sty-files bringing the whole bundle of an executable + resource files up to 4GB.
AFAIK Apple doesn't restrict you from bundling files in your application. That simply wouldn't make any sense.
The point here is that he is talking about an application consisting of a single runnable binary of size X with tons of bundled .sty-files that brings the whole application up to 4GB.
Previously, this would not be allowed, since .sty-files are interpreted (which in turn would make it impossible to implement LaTeX on the iPhone), but that really wasn't the point here. The point is simply that this wouldn't be a single 4GB executable, but a an executable of perhaps 100MB bundled with a whole bunch of .sty (and other) files to a total size of 4 GB.
Edit: Fixed typos. Sorry. HN doesn't work very well on my phone.
> For whatever reason, they cannot allow applications to use the existing capabilities of the underlying OS. This goes against the Unix mindset; of course there will be unpleasant consequences, but one cannot hold Unix mindset responsible for these.
Using the native capabilities of the OS in the sense you mean - allowing TeX to use its ancient, hideous bitmap font system, goes against:
- Unix philisophy - do one thing well, link against or pipe to other apps/libraries that do their thing well
- Simplicity. The user should not be forced to make an unnecessary choice where one option is almost always superor to the other.
Knuth's concept of 'bug free' is one of an academic computer scientist. TeX has many, many issues: unnecessary dependencies, a horrible build process, and many UX issues that stopped TeX from reaching TeX's original goal of 'allowing anyone to create typeset documents'. This is fine: Knuth is a mathematically focused computer scientist from a period where everyone who had access to a computer was a programmer. It was valid in this era.
TeX's original goal of 'allowing anyone to create typeset documents'
Actually, according to this interview[1], speaking of the very first version of TeX, "I should mention that I had no idea that I was designing a typesetting system that anybody else in the world would ever use, besides me and my secretary. My idea was that I would teach Phyllis, who was my secretary for almost 20 years… So Phyllis and I were going to be the users of this TeX system. … I tried to design it just good enough to do my books and no other books."
Even with the released version, I don't think TeX was ever intended to be the last word on typesetting; it was simply his best attempt at a decent markup. He then froze the features so that previously typeset books could be corrected and reprinted using the same source. He left it up to future programs to innovate.
The big problem on tablet devices is constraints in memory. Yes the unix way is to have various small utilities work together through pipes and interprocess communications. The big challenge at that point is how to do you manage memory? What if a single process in that huge chain uses up all of the memory and another process gets killed by the OS which was part of that chain (do note, no swap on mobile devices)... How is the calling program supposed to handle one of its subprocesses suddenly dying?
Do note that the same is true on Google's Android OS. You can run as many threads as you want, but no calling other executables to do work in the background ...
A current tablet device, even a cheap one, has about 256Mb of main memory. Android (at least the cyanogen mod kernel) has also the ability to use a part of the main memory to store compressed swap partition data. That leaves about 80 Megabytes to userspace programs (on Android). About the amount a normal desktop computer had in 1998.
LaTex dates back to the beginning of the 80s. Given that it has the ability to create large documents, I doubt that the required amount of main memory scales linearly with the size of the document.
Also, when you use subprocesses, besides checking their return value, you have some sort of interprocess communication which should detect when one process failed to finish his job.
It looks more like the smartphone runtimes and their sandboxes just don't support it at all, because modern GUI applications rarely build upon multiple executables in the background.
Android does have IPC mechanisms (Intents and remote methods) that can be used to have another process do work in the background. The difference is that even if you use them, you're still not likely to wind up with something that looks like a set of cooperating command-line utilities.
In any case, I absolutely agree that resource constraints aren't keeping LaTeX off of our mobile devices.
I have TeX Live available in the Ubuntu chroot on my HP TouchPad, so you're right that it's not a resource thing. Admittedly in this case webOS provides a pretty standard Linux environment under the hood, so it falls pretty firmly outside the Android model (much less iOS).
I'm curious where you get the idea that LaTeX is bug free. In my experience it is rare to find two packages that work together bug free, yet alone the collections of them needed to form even a respectable document.
I assume he meant that core TeX (the part written ages ago by Knuth) is in effect bug-free. However you do have a good point that one effect of core TeX being ancient and not really updated is that, in practice, what people actually use is TeX plus giant macro packages whose size is bigger than TeX itself, and those are certainly not bug-free. It's at least plausible that a rewrite wouldn't lead to more bugs total, if the rewrite made it easier to write less-buggy macro packages that interacted more nicely (the TeX macro/module situation is not great for that).
Nobody uses (Knuth's) TeX anymore. They use PDFTeX, XeTeX and LuaTeX. None of these are as stable as TeX, but in practice a whole lot better than most other software I use.
All of these versions of TeX are just extensions of the original TeX. The core Knuth's code is still used in each one of them. The stability comes from the core, not just from the work of pdftex, xetex and luatex developers.
The stability comes from the core, but the problems arise from the extensions. I have pushed LuaTeX to some limits and seen lots of segfaults and other errors.
It's 'bug-free' in the same sense that qmail is bug free: you take a piece of software, design it very carefully to do only the minimum amount required to make it fulfill its purpose, spend a lot of time on making sure there are no 'bugs' under a very strict technocratic definition of 'bugs' (i.e., few features, slow running speed, idiosyncratic workings aren't 'bugs') and then make a career out of marketing yourself as being able to write 'bug free' code.
Of course to make the software useful in the real world, a lot of things have to be added to it - make sure to not include any of it in the 'core' otherwise the 'bug free' claim will go out the window really quickly.
So yeah, in that sense I guess it's technically (which I use as a euphemism for 'autistically') correct to call TeX bug free (the bounty program for bugs found in it is quite famous, and part of fore-mentioned marketing).
You think Donald Knuth has made a career for himself out of being able to right 'bug free code'?
Really?
He's made a career for himself out of being an extremely creative computer scientist and writing outstandingly scholarly books about programming. Far from marketing himself as producing bug-free code, he has published articles entirely devoted to the history of bugs found in his software.
(As for the bits that turn TeX into LaTeX, no one is making a career out of claiming that they are bug free -- for the simple reason that no one thinks they are.)
Of course not, it's hyperbole to illustrate the point. In the specific cases of Qmail and TeX, the 'bug free' aspect of it however is overblown and a marketing gimmick. It's not that hard (given enough time) to make something bug free, it's a whole different thing when the software needs to evolve at the same time to meet the (changing and expanding) requirements of actual users.
Actually, the reason for not evolving/rewriting TeX is so that old documents can be typeset and produce the same results. I find that an appealing goal.
TeX wasn't born in a night and suddenly appear in binary form. Knuth worked a hell lot to make it clean, make it work and make it fast. At least, for when he wrote it. Go and read The TeXbook
It is also worth pointing out Joel says "don't rewrite" specifically in the context of a software company, where I strongly believe it is largely correct. In the context of a decades-old open source project the decision may be quite different for legitimate reasons.
I find it likely that you'd be better off performing an algorithm extraction and doing a true from-scratch rewrite than trying to incrementally translate TeX and LaTeX.
The only think i 'd keep from old tex is part of the mathematical notation. Otherwise, it's bloated, quirky and doesn't handle international text well.
"""The price of a LaTeX rewrite would be even higher: incompatibility."""
Yeah. The only benefit is a codebase that reflects current needs and can grow in new directions and follow current practices, that programmers of today are familiar with. And also be faster, and more flexible. But what are those compared to compatibility right?
(Of course people interested in pure compatibility could just continue to use the old version, until the new is compatible enough, or they don't need compatibility anymore).
"""If one tries to rewrite TeX, the problem gets much much worse, since TeX is, for all intents and purposes, bug-free."""
And your wrong assumption is a) that a problem has to be bug free to be useful or b) that a new effort can't produce a relatively bug free codebase?
And your wrong assumption is a) that a problem has to be bug free to be useful or b) that a new effort can't produce a relatively bug free codebase?
How are either of those requirements wrong? I certainly expect my programs to be bug free. LaTeX is a tool that's older than many GNU utilities. It's been reviewed and tested numerous times over by the harshest test suite of all: the real world.
Yes, in theory, you could rewrite TeX to be cleaner, while maintaining backwards compatibility and lack of bugs and regressions. In practice, this is very very hard. It's so hard that no one, no individual or company has managed to pull it off.
But hey, who knows? You could be the first. The LaTeX codebase is open-source. If you think you can pull off what the most talented software engineers in the world haven't been able to do, go right ahead.
"""How are either of those requirements wrong? I certainly expect my programs to be bug free."""
Good luck with that. What I said, though, was that "a problem has to be bug free to be useful" is a wrong assumption. We all use programs that have bugs, and still found them immensely useful.
"""Yes, in theory, you could rewrite TeX to be cleaner, while maintaining backwards compatibility and lack of bugs and regressions. In practice, this is very very hard. It's so hard that no one, no individual or company has managed to pull it off."""
What are you talking about? Tons of people have written TeX versions of their own, even in Lua, and most of those are widely used.
"""But hey, who knows? You could be the first."""
Hardly. There is LuaTex, XeTeX (even more widely used) and others. Are you just talking out of your ass? (Not to mention several other modern typesetting engines that are not TeX based. You make it sound like TeX is an impossible achievement).
Now, what I said I'd like to see was not a rewriting of TeX itself, but a modern version of a TeX like engine, with or without compatibility. Programs like XeTeX and Omega for example provide Unicode support and TTF/OT support to TeX, but I'd like to see something more modular, and with a more streamlined markup than TeX+LaTeX. While we're at it, even scriptable.
There are also some improvements found since in typesetting, like those employed in the InDesign layout engine concerning edge text and hyphenation.
Both LuaTeX and XeTeX have the original TeX code at their core, so they are not bottom-up rewrites.
The purpose and benefit of backward compatibility for TeX is that a 30 year old manuscript written in TeX can be fed to the latest version and produce identical output to what was produced 30 years ago. (Of course, you'd also have to archive the macro packages and font files you were using, but those were never machine dependent).
Thanks. A have given lout a look in the past, but I though it wasn't actively developed anymore. I also liked Prince, but it is proprietary unfortunately.
Don't you think intelligent people have better things to do than to rewrite old code, just because Apple's platform lacks important features that have been well-known for several decades?
The notion that every time we have a new language/platform, we must REWRITE THE WORLD!! is absurd and a huge waste of resources. We should stop making excuses for it.
"""Don't you think intelligent people have better things to do than to rewrite old code, just because Apple's platform lacks important features that have been well-known for several decades?"""
Yeah, you probably missed the whole point of the comment, TFA, and several other stuff besides.
It's not Apple's platform that's the problem, is TeX/LaTeX that is lacking important features. In EVERY platform.
People have been hacking them on to it, but having a better codebase to start off, without the assumptions TeX/LaTeX make would really make it so much easier.
"""The notion that every time we have a new language/platform, we must REWRITE THE WORLD!! is absurd and a huge waste of resources."""
Yeah, good thing that nobody suggested anything like that in the thread.
Btw, the notion that you can add your worthless reply without first understanding what the parent poster says is absurd. You should stop making excuses for it.
I disagree that the current situation is LaTeX developers' fault. TeX and LaTeX are complex pieces of software, that are developed over time. They were extended as the capabilities of their platforms increased, to take advantage of those capabilities. This requires full access to the OS utilities, and naturally LaTeX environment does that. If anything, this is Apple's fault. For whatever reason, they cannot allow applications to use the existing capabilities of the underlying OS. This goes against the Unix mindset; of course there will be unpleasant consequences, but one cannot hold Unix mindset responsible for these.