Bill McCoy: Books 2.0

my take on Apple iBooks Author 1.0 and openness

2012-01-23T22:41:00.000-08:00

iBooks Author 1.0 is the topic du jour, and Apple is taking some heat for restrictive licensing terms and for not using pure EPUB 3 as the format. Ed Bott goes so far as to suggest Apple resign their membership in IDPF.

I think this is a step too far, for three reasons:

1. IDPF does not expect its members every action to be in support of open standards. Many members are large, complex organizations that can and do pursue open approaches and closed proprietary solutions simultaneously. Some might say that for most for-profit corporations "Plan A" would almost always to establish a proprietary architectural franchise with lock-in, and helping create an open, level playing field would be the (not as attractive but far more likely) "Plan B" - mainly to help prevent your competitors from so doing. And concretely, it seems Apple is positioning Author 1.0 as just one way to author for iBooks 2.0, and in general most publishers and authors will be submitting EPUB content which is still fully supported (with more leading-edge EPUB 3 features than any other commercial reading system).

2. Timing issues make Apple's 1.0 approach understandable. Development of iBooks Author 1.0 was done while EPUB 3 was being standardized and was likely in final testing by the approval of the final standard less than 3 months ago. So while much of the functionality there could have been done in EPUB 3, none was possible in EPUB 2 and given the timing issue a "1.0" solution that uses proprietary markup isn't so bad. Apple has been an active and collaborative participant in EPUB 3 process, and in further standards work beyond EPUB 3, and while Apple doesn't ever make forward-looking announcements I would be highly surprised if they don't in due course officially support EPUB 3 in iBooks. As mentioned above they've already delivered more EPUB 3 feature support incrementally (and, compatibly) than anyone else. Meanwhile their restrictive licensing of iBooks Author created content arguably helps keep their 1.0 proprietary format from getting "out in the wild" so in that light could even be viewed (OK, somewhat optimistically) as a pro-standards move. It sure seems crazy evil to me now, but could well end up being viewed warmly in hindsight, if Apple loosens up the licensing terms as they move to standard EPUB format.

3. Innovation generally front-runs standardization. More bluntly, standards bodies are not where innovation happens, it's where previous innovation is "cleaned up and made presentable". For example, most new features in HTML5 were created by a browser vendor unilaterally implementing their own ideas, which were only later hammered into interoperable standard form. Several were done by Apple Safari team, inc. Canvas. This can create legacies so isn't always the cleanest path but then standards done de nova by Working Groups are problematic for different reasons, and in general I'd rather have a standard feature whose utility has already been proven as a working innovation vs. being invented in a theoretical fashion.

Overall while I can't say I'm at all thrilled that iBooks Author 1.0 team chose to create a EPUB-based ".ibooks" format variant, I can completely understand it. We should IMO be encouraging Apple and others to continue down the path of interoperable standards - of bringing digital publishing to the Open Web and visa-versa - not suggesting that this is an either/or vs. proprietary innovation. It's where Apple goes in the future with EPUB 3.0 support and beyond that counts, not what they delivered this month in a canonical "Version 1.0". For example in Safari they now support W3C standard HTML5 Canvas features, and have made that available for all including Google Chrome via the shared open source WebKit library. The good news is that, especially for Apple, a 1.0 product is just a first step, not a final destination. So buckle up, it's going to be quite a ride!

try it harder

2011-11-12T12:10:00.000-08:00

I've been very excited to see some attention being paid to publishing industry associations, due to a series of posts by industry consultants Brian O'Leary and Don Linn (here, here, and here). Two of their larger points have already triggered substantial agreement and work on next steps within and between several of the organizations he references, including the IDPF: we are underfunded, and we need to do more to coordinate industry-wide discussion and action on a number of key issues (Brian references Jane McGonigal in terming this "superstructures" for "super-threats"). A third point, with which I also agree strongly, is that one of the key issues is to ensure that our collective efforts appropriately take into account the impact and trajectory of the largest player in digital books, Amazon, rather than treating them as an outlier.

En route they serve up some plain words in three areas regarding shortcomings of existing industry associations. A number of these points I don't buy, certainly not with respect to the group I'm leading, the International Digital Publishing Forum (IDPF). But they are valuable feedback and I hope lead to some further discussion.

Existing trade associations including IDPF are moving at a "glacial pace" accompanied by "bureaucratic wrangling". I feel this was an entirely unwarranted criticism as far as IDPF is concerned. Since these aspersions were unaccompanied by supporting evidence, and follow-up private communications failed to elucidate any such, they could perhaps just as well be ignored as flame-bait. But since Brian and Don collectively have a lot of years of industry experience, and my respect, I chose to take their statements as motivation to grade IDPF on our work of late on EPUB 3. Judge for yourself here.

There are too many publishing industry associations and some should be consolidating / sunsetting. I completely agree that organizations have life-cycles and that, particularly in disruptive times, status-quo and business=as-usual attitudes make no sense, much less attitudes of entitlement. Nonprofits can be prone to all of these sins, as their success metrics are by definition less crisp and their governance can become self-perpetuating. But the only example cited is IDPF & BISG, with slim supporting argument from Brian that "It makes no sense to have two industry standards bodies, one physical, the other digital". This argument strikes me as a non sequitur. Why would anyone think there's no room for having a global standards organization as well as a U.S. research organization? Whether or not the former focuses only on digital format standardization and the latter focused on both physical and digital supply-chain issues seems entirely orthogonal to their having fundamentally different missions and roles. As stated clearly by their website, the "Book Industry Study Group (BISG) is a national, not-for-profit U.S. book trade association". While they do some standards development work they are better known for doing so in partnership with other organizations (like EDItEUR) and their core work is in research and development of best practices. The IDPF, by contrast, is a global organization, whose work is focused on developing the open standard format for publication interchange and delivery (EPUB). And, the global nature of IDPF is accelerating rapidly. North America is now a minority of our membership, and even more so at the margin with new memberships, and Asia has displaced Europe as the number two region. IDPF members come from STMS publishing, education, magazine publishing, multimedia publishing, certainly not just trade publishing.

If IDPF is to converge someday with any organization, it would IMO likely be W3C, who look after the core Web Standards for content. EPUB 3 is now simply a reliable packaging of HTML5, so we are on a technical convergence path. That being said, there's no way EPUB 3 could have be done in the foreseeable future within W3C. Their membership is very horizontally drawn from the broader IT industry, and have in recent years given documents much less attention than applications. Having IDPF separately looking after EPUB, the portable document packaging of Web content, driven by the business priorities of our our publishing-industry membership, and coordinating with W3C to build on and advanced broader Web standards, seems entirely appropriate right now.

And since Don and Brian don't get more specific, i'm not sure where their "consolidation" recommendations would lead. It's not just IDPF & BISG. It's EDItEUR, BIC, DAISY, NISO, IDEAlliance, and more. It's a global world but not all issues (particularly around supply chains) are going to be global or pan-publishing-segment. Certainly it's a reality that not all of these organizations can continue to exist indefinitely with precisely all their existing missions. And at some point "digital" vs. "print" distinctions will become irrelevant. But, from where I sit, I can't see any consolidation that makes sense now, even as I see lots of actionable steps on improving coordination. And overall I'm a believer that all organizations - whether it be a venture-backed startup, operating business, or a non-profit association - should focus, not seek to be all things to all people.

Trade association boards should be smaller not larger. Again the only example cited was IDPF. Mea culpa for originating the recently approved proposal to expand the IDPF Board from 9 to 14. I've served on and worked with boards of nonprofits, startups, and large public corporations. Certainly board size is a judgment call about which different people can reasonably differ, but best practices seem to point to non-profit boards being larger than 9 to ensure a diversity of representation - and average board size of U.S. nonprofits is 16. For a global organization that is custodian of a standard with impact across the spectrum from trade publishing to education to government reports to more - and represents all parts of the value chain - it seems ridiculous to expect 9 people to adequately represent the breadth of membership interests. And with a Board of 9, you may well only have a literal handful at any given meeting which is getting into smoke-filled-room territory. We have 21 great candidates running for 7 seats on the IDPF Board and I wish we could add them all to the Board! And the last thing I'd want is that the 7 of them who win would displace all the great board members I've already got. Peter Brantley of Internet Archive (@naypinya) opined on a private list that "bigger boards are diluting, and can dissolve the director's ability to lead if there is heterogeneity in board vision, if the board is activist or prone to intervention". Well, in my book no trade association board, composed of part-timers with full-time day jobs and saddled with the vested interests of their employers, should be activist or prone to intervention. The proper job of a board is to provide strategic guidance to, and oversight of (ultimately, the hiring and firing of), management. I see a board of 14 as helping IDPF in this regard. We'll see, of course - "be careful what you ask for" is a wise proverb and I just asked to double the numer of my bosses - but I just don't see that Brian, Don, or Peter have any compelling arguments here (and, the IDPF membership overwhelmingly voted in favor of expanding the board, so it seems clear they are in the minority).

Again, I don't want to come off as defensive, and I solicit feedback on all of the above. I do also want to echo my support for Brian and Don's broader aspirational themes. We need to look hard look at what's working and what isn't, I just hope we can do it with positive spirit not carping and negativity.

EPUB 3: preliminary report card

2011-11-11T14:49:00.000-08:00

Is the IDPF moving at a "glacial pace" thanks to "bureaucratic rambling"? Are we helping to create a "tragedy of the commons" by looking after our "individual interests to the detriment of the industry as a whole"? To the point of being substantially to blame for Amazon's present domination of U.S. e-book sales? So claim publishing consultants Brian O'Leary and Don Linn (here, here, and here).

I agree with their meta points (that our rapidly disrupting industry needs to do more to reinvent itself, increase both funding and coordination of collective efforts, and lose any remnants of business-as-usual attitudes). But, what about their hard words for IDPF? Given that the organization is embarking on strategic and operational planning for 2012 and beyond, guided by a new contingent of Board members, it seems timely to make their aspersions as impetus to take stock. Since the work of the organization over the past year has been, by design, focused on EPUB 3, which was approved just last month, it seems appropriate to look back and ask: How'd we do on EPUB 3?

This of course begs the question of whether focusing the IDPF mission on establishing the global standard interchange and distribution format (EPUB) was the right strategic decision. But criticisms about pace and process really speak to execution not strategy. So it seems reasonable to drill down on how on-target these criticisms are with respect to EPUB 3. I posted separately here on the larger themes of their posts.

Admittedly, as the ED, a former Board member, and long-time proponent of EPUB as an open standard I'm not exactly a neutral party in grading IDPF on execution. But neither am I thin-skinned or shy about looking for ways to improve. So to any members reading this post - really, to anyone in the digital publishing community - it'd be great if you could skim my self-grading below, and let me know where you agree or otherwise about how we've done and how we can do better.

Getting off the dime: B-. Many, myself included, feel that the time lag from EPUB 2 to EPUB 3 was too long. Of course there were external factors that contributed to this (timing of EPUB 2 adoption, iPad & iBooks, and above all HTML5/CSS3 readiness). But it seems clear that some of the EPUB 3 charter development that took place in early 2010 could have been going on in mid to late 2009. But now work has kicked off on key enhancements (advanced adaptive layout, fixed-layout metadata, dictionaries and indexes). So things are looking up.
Execution speed: A. EPUB 3 went from charter to complete proposed specification in 12 months (May 2010 to May 2011), and in 5 more months got the standard approved (Oct. 2011). It's hard for me to see how 17 months total elapsed time could be viewed as a glacial pace even for a trivial standards effort, which EPUB 3 was not. W3C folks by and large view IDPF as moving with reckless abandon! And, it's not just spec-ware: major pieces of EPUB 3 have already shipped in Reading Systems, not just what's in HTML5 and CSS3 modules but for example Media Overlays in iBooks. Unlike any prior EPUB release, we had a validator beta at the time the members were asked to approve the spec.
Execution Quality: A-. Overall, I feel the inherent tension between specification quality and timeliness was well balanced by the EPUB 3 WG. We have a standard that actually specifies a content model, provides conformance requirements, has a normative schema, etc. In other words, something that supports interoperability, not just a "list of supported HTML5 tags and CSS elements... though it is not exhaustive".. Upgrading the WG Editor en route was also a smart move - for more from Matt Garrish see his excellent introduction What is EPUB 3? (free from O'Reilly).
Decision making: A. Is EPUB 3.0 perfect, and did we get everything we wanted to done in 3.0? No, and no. But that's part and parcel of being lean/agile, and as noted things that missed the train are being taken up now with rapidity. Clearly the decision to fully adopt HTML5 was the most critical. This (and corollary decisions to align with modern CSS) created big headaches since HTML5 and many CSS modules aren't yet fully baked, but I feel we did exactly the right thing vs.forking or going down the road of a custom schema a la DocBook/DITA. With all due respect to Joseph Pearson's minimalist bias, I also think we hit the right balance with JavaScript support, MathML support and above all in making EPUB 3 a complete accessibility solution.The Open Web is the universal platform and EPUB 3 is well-positioned as the reliable, accessible portable document packaging of Web content. There's nothing in EPUB 3 that I feel in hindsight should have been left out, and nothing that didn't get in that I feel should have delayed the train.
Plays Well With Others (aka External collaboration): B+. We needed to develop a strong liaison w/ W3C and help to advance several languishing CSS specs particularly for global language support (vertical writing etc.). Meanwhile one of the items that didn't make the EPUB 3 train, page templates for advanced adaptive layout, spawned W3C CSS Regions which is already shipping in Chrome and in the beta of IE 10 / Windows 8. We should certainly do more in liaison and alignment with W3C but we made a lot of progress on that during the EPUB 3 cycle. We also established collaborations with other groups (EDItEUR, BISG, IDEAlliance, NISO, et. al.) that are moving forward well, while avoiding "boil the ocean" scope creep.
Gym Class Survival (aka process): A. The accusation here was "bureaucratic wrangling". But, again, I feel Don's aspersions don't match up with facts on the ground, and give us highest marks here as well. I think the EPUB WG clearly deserves high marks, certainly not bashing (no credit to me personally - this was due to WG chair Markus Gylling's deft leadership and the amazing collaboration of the whole 100+ people involved). The IDPF EPUB WG moved quite rapidly and yet was able to stay with a primarily consensus-based process. Was everything always harmonious? No, but creating a universal, global format among parties whose business interests are not always aligned is no cakewalk. And building on Web Standards that themselves are buffeted by such divergent interests can make it more like walking a tightrope. For example that we ended up no further along on video codec specification than a non-normative suggestion to use either H.264 or VP8 is not my idea of a perfect outcome, and some wrangling certainly happened en route. But we still got a lot more concrete than HTML5 itself, and without derailing the EPUB 3 train. And that was the worst case of wrangling during the EPUB 3.0 cycle, during which time many dozens of similarly weighty decisions were smoothly made. Of course "smoothly" is relative and standards groups are inherently a bit awkward. Some might suggest doing away with any formal process. But, while I'm all for crisp decisions, but we know from open source (Linux) what happens when even an extremely benevolent dictator (Linus) is doing this - forking. Which, especially for a content format, is the mortal enemy of interoperability. I personally would rather see an open transparent process, warts and all. Overall, characterizing the IDPF's process as "bureaucratic wrangling" seems to me un-called-for and disrespectful of the efforts on the part of the many IDPF members who pitched in and contributed their time and energy over the past year+.
Overall grade (aka business impact): I (INCOMPLETE). At the end of the day, a standard is either a business accelerant, or it's worthless. EPUB 3 is freshly minted and it's too soon to pass judgment on its impact. Which means, I can't yet suggest any overall grade. But, the early returns certainly seem promising. Even with the hardest nut - Amazon - it's clear that EPUB 3 has already put the nail in the coffin of Amazon's proprietary Mobi format, KF8 being essentially an admission the EPUB3/HTML5 has won.

Net-net, my take is that IDPF is advancing EPUB at a brisk pace, and the results are helping both in stymieing Amazon's attempt to promulgate a closed silo and in enabling competitors to increase their own market share. Do we need to do even more, and push to deliver meaningful results even faster? Absolutely! But my take is that, we are on the right track with EPUB 3, and doing the right stuff. We need acceleration, not a major course correction. In other words, Brian and Don's bitching, with respect to IDPF's execution over the past year, was totally off target.

Are we home free with EPUB 3 guaranteed to become the universal global standard? No way - there is still risk of forking or divergence, and the transition to interactive HTML5-based content presents innumerable opportunities for vendors to try to lead publishers down proprietary paths. But that's exactly why we all need to pull together. If you're not part of the solution, then you're part of the problem! But, besides my distaste for negativity, Monday-morning quarterbacking and sweeping criticisms without any accompanying factual support, perhaps part of the disconnect I'm having with Brian and Don can be attributed to principled disagreement about the proper role and scope of IDPF and other publishing trade associations. This relates to the broader points they make, about which I ruminated in my other post.

But, again, I don't want to be defensive. Maybe I've got the EPUB 3 scorecard all wrong. I welcome your feedback.

Korea Going All-In For Digital Textbooks, Global Standards

2011-07-09T19:46:00.000-07:00

Last week's announcement that Korea plans to offer all textbooks in digital form by 2015 was widely publicized, and thanks to the Internet news echo chamber the nut graf, even from reputable outlets like CNN, soon became Korea textbooks paperless by 2015. I was in Seoul when this came out, at a workshop on EPUB 3 organized by the Korean Education and Standards Ministries, so I can say with some confidence that there will still be a lot of paper textbooks used in Korea in 2015, i.e. the follow up coverage was hyping a fictitious timeline. But it is nevertheless eminently clear that the Korean government is seriously committed to pursuing a digital-centric model for textbooks and other learning materials. They are putting significant thought leadership and financial backing behind the initiative, and considering all the critical issues, such as accessibility.

I'm also extremely happy to report that the key drivers in the Korean government ministries seem to have a keen appreciation for the imperative to adopt global standards. In the build-out of mobile infrastructure and other areas of IT, Korea frequently adopted unique domestic solutions that generally were not successful outside Korea and ultimately only impeded the domestic market development while putting local companies an unnecessary disadvantage on the global stage. There was for example Neomtel SIS, which was widely adopted within Korea and pushed as a Flash Player competitor (Ever heard of it? I didn't think so). SIS was entirely unsuccessful at expanding globally, and was eventually even pushed aside by Flash within Korea. And of course Flash itself is now on the way towards being superseded by HTML5 and associated Web Standards. So, Korean stakeholders understand very well that adopting global standards rather than proprietary solutions is the pathway to global success. And with companies like Samsung and LG becoming top global brands I detect no defensiveness: the thought process is all about building bridges, not erecting walls. That's good news for Korea, as well as for EPUB 3 and HTML5.

I can't say anything more specific now, but I expect to have much more to share about Korea and EPUB in the very near future...

Once More, with Feeling

2011-07-09T19:30:00.000-07:00

I'm blogging here again. Earlier this year I put my early stage startup effort on the back burner to become Executive Director of the International Digital Publishing Forum (IDPF). Things got pretty crazy pretty fast, what with finishing up work on EPUB 3 and putting on a digital publishing conference for over 900 people. But, there are things I want to say about these exciting times in digital publishing that don't belong as official statements on the website of a global trade and standards organization and that I can't fit into 140 characters, so I'll be publishing them here. I'm finishing up an Asia trip just now, so my next triad of posts will look at the digital publishing markets in Korea, China, and Japan. The name "Books 2.0" remains a bit too narrow for my (and IDPF's) scope of interest which includes magazines and other kinds of digital publications and documents, learning materials, transmedia content - essentially anything with some text centricity and/or evolutionary heritage stemming from print publishing. But, "Books 2.0" is a bit snappier than "Publications and Documents 2.0" so I'm sticking with it.

From Books 2.0 to WebPaper: blog move

2010-05-23T09:01:00.001-07:00

I haven't been posting here for the last several months, mainly because I've been working to get a new startup off the ground. Along the way I've realized that my passion for digital publishing is about more than the future of books. Books are certainly being radically transformed by digital reading and the Web, but so are other types of documents. Books represent a small fraction of the many millions of documents distributed annually by individuals and organizations (reports, presentations, white papers, case studies, brochures, data sheets, etc.).

As we move from the desktop publishing era (where physical paper was still the end product) to the digital publishing era (where on-screen consumption is increasingly the norm), I believe that we will continue to see horizontal enabling technologies winning out over vertical solutions. For desktop publishing, PostScript and its close cousin PDF became the core platform, addressing needs of commercial publishers as well as organizations and individuals. For digital publishing, it's increasingly clear that the Web and its HTML format represent the core platform (in an even broader sense this is what Tim O'Reilly terms the Internet Operating System ).

With printing no longer the raison d'etre, Web-published documents can be liberated from the restrictive requirement to emulate paper. Documents can adapt to reader preferences and mobile device capabilities, and include rich media, interactivity, and nonlinear navigation without being shackled to an architecture based on a linear sequence of fixed pages. Yet, this creates new challenges for Web publishers, especially in a world where print is still important, and where webpage-oriented browsers don't yet provide good reading experiences for long-form texts or complex layouts.

Our startup is working to solve these challenges and help enable an easier and better way to publish documents to the Web and mobile devices, delivering the fundamental benefits of portable documents built on open standard Web technologies. I think of what we're doing as helping to create not just "Books 2.0" but "Documents 2.0", or, since "document" is kind of an amorphous term, "Paper 2.0". Hence our name: WebPaper.

That all may sound a bit vague, which is no accident since WebPaper is still under wraps. We're inviting sign-ups for our forthcoming Beta, and I'll be blogging over there.

Q&A: What PDF version for eBooks?

2010-01-25T07:16:00.000-08:00

I often receive queries like:

I am converting the Word file of my book to a PDF (a bit later in epub). To be read on as many devices as possible, is PDF or PDF/A better? PDF/A-1a or PDF/A-1b?

This may seem like a rather nit-picky question, and the bottom-line answer is straightforward: stick to PDF/A to maximize portability, and the lower conformance level "b" is fine. But some interesting strategic points are illustrated by the details underlying this answer. The rest of this post delves into some of these details, reaching the possibly surprising conclusion that, even though Adobe invented the PDF format and supply the PDF software embedded in many devices, the bar of de facto PDF compatibility is now being set by Apple, not Adobe. A future post will look at how the situation with varying levels of PDF format support exemplifies control issues and other problems with open standards that evolve from proprietary origins.

On a side note, it's encouraging that the author posing this question was already savvy to the fact that he should ideally be offering epub, the open standard for reflow-centric digital publications. EPUB, built on Web standards including HTML and XML, was architected to enable optimized reading experiences on different-size screens.

But despite the rapid proliferation of epub support on devices, there are many reasons why it might be practical to publish a PDF format edition of an eBook. The reason hinted at by the "a bit later in epub" is expediency. PDF is a pre-paginated "typeset" format that is almost universally produced in the process of creating print (or print-on-demand) books, and tools for creating and assembling PDFs are widespread. So as a self-publishing author you probably already have, or be able to instantly create, a PDF of your book. By contrast, epub is a newer standard and tools are still emerging, and not yet quite push-button in nature, especially if your book has complex page layouts. You might perhaps want to put content out through a distribution channel that requires PDF (although self-publishing services ranging from Scribd to Lulu have recently added epub support, it is not yet as universally supported as PDF).

OK, on to PDF format details. Adobe supplies the PDF software on the Amazon Kindle, Sony Reader, Barnes&Noble Nook, and many other devices, via their Reader Mobile SDK. The description of Adobe's RMSDK is rather vague about PDF support specifics, even in its FAQ. The FAQ for Adobe Digital Editions, Adobe's desktop eBook reading application (which utilizes the RMSDK engine), is a bit more detailed about PDF support, although it still hedges a bit:

Digital Editions supports a superset of ISO standard 19005-1 (PDF/A). PDF/A is designed to support more secure, long-term information archiving; it is based on a subset of PDF 1.4 (the version of PDF supported by Acrobat 5.0). Additional PDF capabilities in Digital Editions beyond PDF/A include basic encryption, DRM-based encryption, JBIG2 image compression, transparency, and compressed object streams. The intention is to support in Digital Editions those PDF features reasonably needed by eBooks and other commercially published content, balancing 100% coverage with a focus on small size and high performance...

Both FAQs note that various enterprise-oriented features, such as interactive forms, security of the Livecycle PolicyServer variety, JavaScript, and digital signatures, are not supported.

So although there is some wiggle room, especially around future capabilities, this is relatively clear, and provides clear guidance that PDF/A (aka PDF 1.4 aka "Save As Acrobat 5 Compatible") can be safely employed.

The next question is what additional implementations of PDF might need to be considered beyond RMSDK, given the goal of publishing content that can be read on as many devices as possible. PDF has always been an openly published format, so there are many implementations of PDF support. Undoubtedly the most widely used beyond Adobe's is Apple Preview, the image and document viewing application built-in to OS/X on Macs and iPhones. While Adobe's enterprise-featured free Adobe Reader and its even heftier sibling Acrobat software run on Macs, many users prefer the nimbler, streamlined, and multi-format supporting Preview. As PDF support is built-in (including creation) to OS/X, many Mac users don't ever both to install Adobe Reader or Acrobat... and iPhone users don't even have these options. Given that iPhone is arguably the most popular device for eBook reading, and that any content being consumed on desktops will certainly end up being opened in Preview, support by Preview is almost certainly the most important consideration. And this is before the advent of the Apple Tablet.

Unfortunately, Apple apparently does not appear to document anywhere exactly what level of PDF is supported by Preview. The latest version of Preview is 5.0, which shipped with Snow Leopard (OS/X 10.6.0), although most users are likely to have an older version, and it's even less clear what is supported by Preview on iPhones. So what's a harried self-publishing author to do? Especially one who might use Windows desktops, and may be iPhone-less?

The answer is simple: look to Adobe's mobile PDF implementation, since it's forced to follow Apple's lead!. That is, there are millions of PDF files out in the wild that have been created by, and/or viewed on, OS/X, so any PDF language feature supported by Preview is likely one that publishers and end users will expect to be supported by Adobe's RMSDK. Thus, while Adobe's full software clearly defines the high end of PDF compatibility requirements, Apple has in effect set the bar for the low end requirements, a bar that even Adobe needs to meet. As evidence, consider that with the exception of DRM (which I will touch on momentarily), all the added features beyond PDF/A listed in Adobe's FAQ, such as basic encryption and transparency, are supported in Apple Preview. And after Apple Preview 5.0 shipped with support for JPEG 2000 images, several months later Adobe followed suit with the latest RMSDK 9.1 release. Coincidence? Unlikely.

So, if your file views properly on Adobe Digital Editions, you should be in good shape. Ideally you should test this, but if you save as PDF/A, that should be good enough. The features beyond PDF/A that are supported in this lower-bar of compatibility are in most cases not going to be critical, although they may leave your eBook a bit larger than it could be if it took full advantage of what one might consider to be the "Apple Preview de facto standard" for proliferated PDF format support.

As far as the difference between PDF/A-1a and PDF/A-1b, these represent two different conformance levels, PDF/A-1a indicating support for "Tagged PDF" data structures that provide for accessibility. Some reading software supports read-out-loud features for accessibility, and may also support the option of "reflowing" a PDF on a smaller screen. However, most PDF creation software - even some from Adobe - does not include "Tagged PDF" data structures. This is in part because the PDF creation often flows out of a printing process which does not have access to the high-level data model of the authoring application. And when these data structures are included, they are often incomplete or even inaccurate. As a result, PDF reading software that support accessibility and/or reflow views tend to use various heuristics to reconstruct the "reading order" of text - heuristics that in some cases may not even utilize, and almost certainly would not require, Tagged PDF. And, regardless, a PDF eBook is most likely not going to deliver a fantastic auditory rendition, since PDF is a "typeset at the factory" paper-replica format.

Regarding DRM, the goal of having content readable on as many devices as possible precludes DRM support, since Adobe's proprietary ACS4 DRM is not presently supported by Apple on Macs or iPhones. In any case, for almost any self-publishing author (and arguably major publishers) increased exposure to, and maximum convenience for, end users should outweigh any increase in piracy resulting from foregoing copy-limiting technology. And most pirated editions originate from scanned print books, and no DRM technology can protect against that.

This post has touched on only a few of the considerations around PDF format level compatibility. My next post will go over some additional issues, including the ISO-32000 standard, PDF Portfolios, and other bleeding-edge features, and draw some additional conclusions.

Apple Cooks Up Rich Interactive eBooks With PastryKit

2009-12-18T15:37:00.000-08:00

There was a lot of discussion earlier this week about the discovery (via Daring Fireball) of a new Apple JavaScript framework, used to implement the iPhone User Guide (link for iPhone users). The PastryKit framework enables HTML-based content with a more iPhone-native UI: full screen support, fixed-position toolbars, and native-feel scrolling. Much of the speculation focused on implications for mobile app development. But looking at PastryKit as just an "iPhone Web app library" misses a key point: the iPhone User Guide is not an application but a rich interactive eBook. Thus PastryKit (and the related TuneKit framework for iTunes LP) are very likely tip-offs to the format underpinnings of Apple's widely rumored plans to "reboot publishing" in conjunction with the pervasively rumored "iPad tablet" device.

It's significant that this approach is, at its core, not Apple-proprietary but rather Web standards-based, and conceptually consistent with the epub eBook format standard. As such it may end up setting direction for the industry as a whole. It's clear that rich media and interactivity will, over time, be integral to eBooks and other digital publications. I.e., digital books will not just be digital equivalents of paper books, but will evolve into new kinds of mixed media. Timo Hannay recently wrote a clever self-referential essay (also in HTML and JavaScript) showcasing a range of new design elements that will be called for.

To date, the epub standard has focused on eliminating the most basic barrier to immersive digital reading: enabling "reflow" so that content is adaptable to different size screens (vs. PDF which is "typeset at the factory" for a particular page size). Interactivity and rich media have not been mainstream requirements for eBooks, in part because today's E Ink based screens are slow and monochromatic, and also because traditional publishers have first had to worry about digitizing their existing print-oriented content. So how eBooks will support rich media and interactivity in a first-class manner is still an open question. Early experiments, like Random House UK's "Books and Beyond" enhanced editions, have utilized epub's embedding mechanism to add Flash SWF content, which is however currently supported only in Adobe Digital Editions on PCs and Macs.

The alternative approach Apple seems to be taking - leveraging the browser stack and JavaScript - could be a harbinger of the ultimate standard solution. Webkit is fast becoming pervasive across smartphones and other devices, so this approach is not something that would be inherently iPhone-only (as evidenced by cross-platform smartphone Web app frameworks like PhoneGap). And, this direction is consistent with the spirit of the epub format, which is being rapidly adopted for eBooks. While some details would surely require harmonization to achieve complete compatibility, the shared approach is to ZIP-package (X)HTML with associated assets - images, CSS, fonts, etc. - together with a manifest and other metadata.

Stepping back, this approach implies a more unified approach for eBooks and the Web. Rather than thinking of packaged digital publications and the Web as separate silos, this approach foreshadows a converged architecture, with epub support potentially even being built-in to future browsers. After all, Webkit has already been used as the core of epub rendering solutions such as Stanza and Bookworm, and has incorporated key capabilities required for full epub support, such as SVG. Although there is presently no direct support in Webkit for epub's OCF container format, nor for true pagination, these are things that Apple, Google, and/or the community at large could surely address.

There would certainly be some gotchas to be worked out. One of the strengths of epub is that it is a well-defined validatable XML schema, using XHTML and DAISY for content flows and thus avoiding the chaos of "tag soup" HTML. Mixing in JavaScript with XHTML content willy-nilly could compromise archivability and accessibility. One solution to this problem would be to be more restrictive on where and how JavaScript could be used within a publication. For example, "islands" of full HTML+JS could be supported, using IFRAME or another embedding mechanism. JavaScript could also be restricted to well-defined contexts, such as event handlers.

Digital Books - Into The Future

2009-11-14T06:19:00.000-08:00

This blog is in some respects a continuation of my corporate blog at my soon-to-be former employer Adobe Systems. I'm excited about reinventing books in the digital world, and much of my work at Adobe these last several years has been driven by this passion, and the broader goals that I share with colleagues such as Bill Hill and Tim O'Reilly.

That being said this blog is strictly personal, and will not be driven by any corporate priorities or agendas.

As well, my work at Adobe these last three years has been centered around a relatively prosaic objective: establishing open standards that enable multi-channel/cross-device distribution of eBooks. For all intents and purposes, this work is done: epub is now firmly established as the industry standard for reflow-centric eBooks. That took a considerable effort, on the part of many people, and I'm really proud that we did it. But... that was the easy part: essentially migration of print to digital. epub does take portable documents to the next level - breaking past beyond PDF's paper-replica model. But that's only the beginning of the fundamental reinvention of the book that digital content and the Web will enable. In other words: now it's really going to get interesting. I expect my future work, and this blog, to focus on this transformation.