Jul 01 2018

markdown to docx

I was looking for a way to convert swagger2markup markdown output to docx using Java (as opposed to Pandoc).

I found flexmark-java which describes itself as:

Java Markdown parser with source level AST. CommonMark 0.28, emulation of: pegdown, kramdown, markdown.pl, MultiMarkdown. With HTML to MD, MD to PDF, MD to DOCX conversion modules.

Its MD to DOCX is in flexmark-docx-converter module, which happily, uses docx4j under the covers.

I tried it, and found it works nicely, the exception being table rendering if you open the resulting docx in LibreOffice (Word automatically sets the column widths, but in LibreOffice 5.3 Writer, the column widths are bad and painful to fix).   The underlying issue is that markdown doesn’t specify column widths, and in docx4j, we don’t provide an algorithm to help the user set sensible values.

Development of flexmark-docx-converter was  sponsored by Johner Institut GmbH (medical device documentation)

The documentation for flexmark-docx-converter is at https://github.com/vsch/flexmark-java/wiki/Docx-Renderer-Extension

That page says:

Word does not handle inserted HTML very well.

It would be quite straightforward to use docx4j-ImportHTML to work around that :-)

 

Apr 28 2018

From VariableReplace to OpenDoPE data binding

This blog post is a walkthrough of how to easily move from variable replacement to OpenDoPE content control data binding.

Introduction

Variable replacement is quite a popular way to get started generating Word documents.

I guess that’s because developers expect this sort of approach to be available, and its easy: all you have to do is add your variables to the document, then bang, you replace them.

But its not all a bed of roses, there’s some thorns in there too:

  1. the so-called “split run” problem, in which Word has split your variable name across more than one XML element, due to formatting, spelling/grammar etc
  2. variable replacement is great if you just want to replace text, but what if you want to replace images, conditionally include/exclude content, or repeat table rows or list entries?

Content control data binding is a great solution to these problems.

Your data (provided in XML format) is bound to content controls using XPath, and with the OpenDoPE conventions, this approach offers:

Some users create very complex contracts and reports this way.

Automated Migration

The good news is that docx4j contains code to automatically migrate a document which has variables on its surface, to one which contains OpenDoPE content controls.

The code is in FromVariableReplacement.java

Have a look at the main method to see how to use it.

There have been some fixes recently, so you should use docx4j-nightly-20180428.jar (or 3.3.8 when released) or later.

OK, let’s assume you now have a docx file with content controls in it.  You may want to further develop your template.  For this you need an OpenDoPE Authoring tool.

OpenDoPE Authoring

The friendliest OpenDoPE authoring tool is the “No-XML” Word AddIn.

With this it is very easy to add conditions, but the limitation is that it assumes a fixed XML format.   If you want to use your own XML format (or to bind escaped XHTML I suspect), you’ll need to use one of the older add ins.

Here we’ll walk through adding a condition with “No XML” add in.

For this example, we’ll use the Commonwealth of Australia’s model Confidentiality Agreement, available at http://www.business.gov.au/IPToolkit

Here’s what the first few blanks look like, represented as content controls with the “No XML” AddIn’s ribbon showing in Word:

NoXMLAddIn1

I had pressed the “Show tags” button to be able to see the content controls in orange above.

Further down, there’s an optional Indemnity clause.

Since its optional, let’s wrap that in a conditional content control.  First, we need a question “Do you want the Indemnity”.  It works this way because this AddIn is aimed primarily at the interactive use case, that is, a user can answer questions in their web browser to generate an instance document.

But the resulting template can be used just as easily for the more common non-interactive / entirely programmatic case.

So click the “Insert Q/A” button.  I did this with my cursor somewhere in the middle of the Indemnity clause.

Fill in the form (for Multiple Choice choose yes):

qa_mcq

click next, then on the next page, choose type boolean (true/false), then ok.

You’ll see a content control inserted where your cursor was.  We don’t really want that, so its a bit annoying (you can/should delete it).  You’ll see why we did this just below.

Now select the clause heading and body, and click “Wrap with Condition”.  You’ll see something like:

condition1

In the condition builder, define the following condition:

indemnity_condition

then click OK. (Now you can see why we needed to set up that question first)

Our resulting conditional clause appears as follows:

indemnity_result

That’s all you need to do.  We can now try generating an instance document from this template.

Document generation runtime

To generate a document, use docx4j code to populate an Answers object, then call Docx4J.bind. For example:

WordprocessingMLPackage wordMLPackage = WordprocessingMLPackage
.load(new java.io.File(inputfilepath));

answers = new Answers();

addAnswer("Sponsor_name_ACNABN_oW", "CSIRO of Some St, Sydney")
addAnswer("want__Indemnity_clause_K8", "true"); // or false
// etc

Docx4J.bind(wordMLPackage, answers, Docx4J.FLAG_BIND_INSERT_XML & Docx4J.FLAG_BIND_BIND_XML);
Docx4J.save(wordMLPackage, new File(outputfilepath), Docx4J.FLAG_NONE);

where addAnswer is just:

private void addAnswer(String key, String value) {
Answer a = new Answer();
a.setId(key);
a.setValue(value);
answers.getAnswerOrRepeat().add(a);
}

How do you know what key to use?  Look in the answers part in the docx and use the corresponding ID (yes, you should be able to see this in the AddIn, but the reason you can’t is that for the interactive use-case, you never need to know), or you can just invoke Docx4J.bind with debug level logging enabled for org.docx4j.model.datastorage, and it will print out the relevant part.

answersPart

That’s about it.  If you have questions, they are probably best posted in the relevant docx4j forum or on StackOverflow.

 

 

 

 

 

 

Mar 27 2018

Docx4j and WebSphere 2018

TLDR

Current 3.3.x Docx4j works with WebSphere versions 8.5.5.9 and 9.0.0.5 in WebSphere’s default configuration (tested with IBM Java 8, which is not the default in WebSphere 8.5.5.9).

docx4j 3.3.7 contains an important fix for errorsCount where XLXP2 is in use with fallback JAXBContext of Sun/Oracle or reference implementation (see below for context).

Scope/Assumptions

Our testing was based around the following assumptions:

  • IBM JDK (not Sun/Oracle)
  • IBM JAXB (see below)
  • Xalan is available for use via System.setProperty(“javax.xml.transform.TransformerFactory”, org.apache.xalan.transformer.TransformerImpl)

Out of Scope of testing: OSGi. Others have done some work on OSGi in the past though; see https://github.com/uncleit/docx4j-osgi/blob/master/pom.xml or https://github.com/kimios

JAXB Background

IBM has their own proprietary JAXB implementation. By default, WebSphere uses com.ibm.xml.xlxp2.jaxb, which has the concept of fallback/ MarshallerProxy. The actual implementation it uses is in com.ibm.jaxb.tools.jar.

It is possible to configure WebSphere to instead use the JAXB implementation in the Sun/Oracle JRE, but usually you would not do this if you are using the IBM JDK.  Alternatively, your application could use MOXy JAXB (by including the relevant jars).

Here we tested with WebSphere’s default, namely:

Primary JAXBContext:
bundleresource://138.fwk797973828/com/ibm/xml/xlxp2/jaxb/JAXBContextImpl.class,
Version: 1.6.2-jaxb,
Fallback JAXBContext:
bundleresource://11.fwk797973828/com/ibm/jtc/jax/xml/bind/v2/runtime/JAXBContextImpl.class Build-Id: null

For more information, see https://stackoverflow.com/questions/48700004/does-webspheres-jaxb-marshallerproxy-use-the-reference-implementation

WebSphere has property: com.ibm.xml.xlxp.jaxb.opti.level (see https://www.ibm.com/support/knowledgecenter/en/SSAW57_8.0.0/com.ibm.websphere.nd.doc/info/ae/ae/xrun_jvm.html#com.ibm.xml.xlxp.jaxb.opti.level ):

  • At level=0, optimization methods are not enabled;
  • At level=3 (default), both unmarshalling and marshalling optimization methods are enabled.

In our testing, we used values 0 and 3 (or not set).

WebSphere has several other JAXB related properties which we left at their default settings.

ErrorsCount

Docx4j contains a class JaxbValidationEventHandler, which is responsible for handling unexpected content (both mc:AlternateContent which is common, and certain other errors in an incoming docx).

In the JAXB reference implementation, there is a method shouldErrorBeReported(); see https://github.com/javaee/jaxb-v2/blob/master/jaxb-ri/runtime/impl/src/main/java/com/sun/xml/bind/v2/runtime/unmarshaller/UnmarshallingContext.java#L1350

Previously errors (ie unexpected content) were not ignored if UnmarshallingContext.getInstance().parent.hasEventHandler()

Some time around 2015, JAXB was changed so that after unexpected content has been encountered 10 times (ie in 10 docx parts), the error won’t be reported (ie docx4j’s JaxbValidationEventHandler won’t be invoked, so docx4j doesn’t have the opportunity to deal with the content error, with the result that content is silently dropped).

Recent versions of docx4j work around this, by resetting the error counter, and docx4j 3.3.7 builds on this with an important fix for errorsCount where XLXP2 is in use with fallback JAXBContext of Sun/Oracle or reference implementation

Test Results

With environment WebSphere 9.0.0.4, current docx4j/Plutext releases work well.

With environment: WebSphere 8.5.5.13 (WebSphere 8.5.5.9 upgraded in order to run IBM Java 8),  current docx4j/Plutext releases work well.

(Older Java should also be ok, but was outside the scope of testing)

Methodology Notes

In testing, there are several things to be aware of:

  1. WebSphere might re-use a jar in multiple webapps. In case of unexpected results, ensure you don’t have different versions of the same jar in other webapps, stop the server, clearClassCache, and restart.
  2. If you are looking for JaxbValidationEventHandler log entries but cannot see them, double check that your jar files do not contain another log4j.xml.

Java 2 Security

If you have Java 2 Security enabled in WebSphere, you will need certain permissions enabled in policy settings.

Mar 15 2018

PDF Converter task sizing and auto scaling

With FarGate, you have to specify a task size:

task-sizing

Load testing with JMeter, I have found that 2 vCPU works well for the Task CPU setting.  The minimum Task Memory you can set for 2 vCPU’s is 4GB.  (The PDF Converter doesn’t use that much RAM, so it would be good to be able to specify just 1GB, particularly since FarGate pricing includes a cost per GB)

For my load testing (32 parallel conversions), served by 2 tasks:

JMeter_2-tasks

So, an average of 9.8 sec per conversion (based on a range of documents, some short/quick, others long/slow).

With FarGate, you can set a service to auto-scale, under CPU load or based on incoming requests.

So let’s improve on those response times, by auto-scaling the number of tasks available for processing the incoming PDFs.

How to do this? FarGate tells me my CPU utilization was:

UtilizationCPU

So let’s “update” the service to set auto-scaling to happen at 40%:

auto-scaling-cpu40

Re-running the load test, here are the results:

JMeter_autoscaled

You can see the response time better than halved, and throughput doubled.

At the end of the test, I can see that it auto-scaled to 10 tasks:

tasks-status-scaled-cpu40

Looking at the load balancer target group, you can see it went from 2 tasks to 5 tasks to 10:

healthy hosts

(the test sarted at 23:13 and finished at 11:28; scaling in occurred some time after the test concluded).

You can see from the graph below that the average response times drop as these extra tasks become available.

response-times-over-time

Running the load test one last time, with 8 tasks in place from the start:

JMeter_10-tasks

we have an average response time of 2.2 seconds, and we’re converting 12.48 documents per second.

In summary, configuring the cluster so that each task has 2 vCPUs, and auto-scaling when CPU utilization hits 40%, looks like a good place to start tweaking your own instance.

Mar 12 2018

Using HTTPS on FarGate

This is the second post in a series on scaling the PDF Converter using Amazon’s FarGate service.

In the first post, we got the PDF Converter running across 2 instances behind a load balancer, in under 20 minutes.

Now, we want to use HTTPS.  The Amazon documentation is at https://docs.aws.amazon.com/elasticloadbalancing/latest/application/create-https-listener.html

First, go to the AWS Certificate Manager (ACM): https://console.aws.amazon.com/acm/home?region=us-east-1#/firstrun/ to request a certificate (for your domain).

Now go to your load balancer, and choose “create listener“. Choose HTTPS.  You should see something like:

alb-https-listener

(here I’ve used plutext.com, but obviously you’ll have substituted your own domain).

If/when you click “create”, you’ll probably get a warning saying your security group doesn’t allow HTTPS, so click on the security group to allow traffic on port 443.

We’re not quite there yet.  If you try converting using your load balancer endpoint (something like https://EC2Co-EcsEl-1GY7BNHSDU1HTH-1150934046.us-east-1.elb.amazonaws.com:443), you’ll get an error saying the certificate subject name does not match target host name.

To overcome this, you need to update your DNS records so you have a host with the right name resolving to the load balancer.

The recommended way to do this is to use Amazon’s Route53 DNS.

But just to prove what we’ve done so far works, its enough to put an entry in your /etc/hosts file mapping a host covered by your certificate, to the load balancer’s IP address.  Then:

$ curl -v -X POST --data-binary @HelloWorld.docx -o out.pdf https://fargate.plutext.com:443/v1/00
000000-0000-0000-0000-000000000000/convert
Note: Unnecessary use of -X or --request, POST is already inferred.
% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
Dload  Upload   Total   Spent    Left  Speed
0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 54.89.45.53...
* Connected to fargate.plutext.com (54.89.45.53) port 443 (#0)
* found 148 certificates in /etc/ssl/certs/ca-certificates.crt
* found 597 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
*        server certificate verification OK
*        server certificate status verification SKIPPED
*        common name: *.plutext.com (matched)
*        server certificate expiration date OK
*        server certificate activation date OK
*        certificate public key: RSA
*        certificate version: #3
*        subject: CN=*.plutext.com
*        start date: Mon, 12 Mar 2018 00:00:00 GMT
*        expire date: Fri, 12 Apr 2019 12:00:00 GMT
*        issuer: C=US,O=Amazon,OU=Server CA 1B,CN=Amazon
*        compression: NULL
* ALPN, server accepted to use http/1.1
> POST /v1/00000000-0000-0000-0000-000000000000/convert HTTP/1.1
> Host: fargate.plutext.com
> User-Agent: curl/7.47.0
> Accept: */*
> Content-Length: 4082
> Content-Type: application/x-www-form-urlencoded
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
0  4082    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0} [4082 bytes data]
* We are completely uploaded and fine
< HTTP/1.1 200 OK
< Date: Mon, 12 Mar 2018 05:01:52 GMT
< Content-Type: application/pdf
< Content-Length: 38507
< Connection: keep-alive
< access-control-allow-origin: *
<
{ [16384 bytes data]
100 42589  100 38507  100  4082  16903   1791  0:00:02  0:00:02 --:--:-- 16903
* Connection #0 to host fargate.plutext.com left intact

Now we know it works, you can add a CNAME record at your DNS provider, mapping your chosen host name to the load balancer’s host name.

Remove the entry we added to /etc/hosts, give your CNAME entry time to propogate, then verify the curl command works.

Mar 12 2018

Scaling the PDF Converter with AWS Fargate

This is a walkthrough of deploying the PDF Converter on Amazon’s FarGate.

What is Fargate?  New since November 2017,  its an easy way of deploying containers on AWS ECS.  You don’t have to manage the underlying EC2 instances, and the wizard takes care of the setup, so you can be up and running in less than 20 mins!

With FarGate, you make a “cluster” which you can easily size to suit a known conversion volume, or have it auto-scale with load.  Largely thanks to Docker!

This walkthrough assumes you already have an AWS login.

To getting things working:

  1. there’s 4 steps in Amazon’s firstRun wizard: https://console.aws.amazon.com/ecs/home?region=us-east-1#/firstRun
  2. then you configure the health check path

But first, check things are configured correctly for ECS in your Amazon account.  Since FarGate currently only works in N.Virginia, visit https://console.aws.amazon.com/ecs/home?region=us-east-1#/getStarted

ECS FirstRun Wizard

If you don’t already see the “Getting Started” wizard pictured below, click https://console.aws.amazon.com/ecs/home?region=us-east-1#/firstRun (this is easier than “create new cluster” at https://console.aws.amazon.com/ecs/home?region=us-east-1#/clusters/create/new since it also creates a Service and Task, but more importantly, your load balancer).

fargate-firstrun-step1

In the “Container definition” section, click the “configure” button on the “custom” image.

Type the following in image: plutext/plutext-document-services:2.1-0, and set the other values as per the image below:

 

container-settings-dockerhub

Next, in “Task definition”, edit the task definition name, to say: pds-task-definition

Click next.

Service

On the “service” screen, click “edit” to set the number of tasks to 2, and choose “Application Load Balancer”.service

Click next.

Cluster

On this screen, just change the cluster name to: plutext-document-services

When you click next, the review screen should show:

review

Click “Create”.  The wizard will perform various tasks; it might take 3 or 4 mins.

When it is done, you should see:

preparing-service

Click the “view service” button.

Health Check

You need to set the health check path in your load balancer.  (Unfortunately, FarGate currently doesn’t populate this from the HEALTHCHECK statement in your Dockerfile)

So in your cluster, click your service, where you’ll see the load balancer target group:

cluster-service

 

Click that.

Now, you’re in your load balancer, where you can click “edit health check” and enter path:  /v1/00000000-0000-0000-0000-000000000000/ping

Result should be:

health-check

Before you go back to your service, click on the load balancer itself, and make a note of its DNS name.   You’ll see the host name there in the basic configuration:

alb-hostname

 

Now if you go back to your service, on the “tasks” tab, you should see:

tasks-status

ie “RUNNING”

Try it out!

To convert a document, you need the DNS host name of the load balancer you made a note of above.  Now you can test with something like:

curl -v -X POST –data-binary @HelloWorld.docx -o out.pdf http://EC2Co-EcsEl-1N1ULP12K5TGG-2127307716.us-east-1.elb.amazonaws.com:80/v1/00000000-0000-0000-0000-000000000000/convert

Check for “200 OK” and try opening out.pdf.

Next steps

In our next post, we’ll configure HTTPS, and in the one after that, we’ll add a license key.

Jun 30 2017

JAXB RI new home on the web

https://github.com/javaee/jaxb-v2 is JAXB’s new home (jaxb.java.net now redirects there).

Issue tracker is at https://github.com/javaee/jaxb-v2/issues (though it seems some existing issues didn’t get migrated over, for example https://github.com/gf-metro/jaxb/issues/22)

I’m not sure where you are supposed to get official binary releases from; Maven I guess?

http://search.maven.org/#search%7Cgav%7C1%7Cg%3A%22com.sun.xml.bind%22%20AND%20a%3A%22jaxb-ri%22

http://search.maven.org/#search%7Cgav%7C1%7Cg%3A%22javax.xml.bind%22%20AND%20a%3A%22jaxb-api%22

Discussion group is apparently at https://javaee.groups.io/g/metro, but first you need to join the parent group https://javaee.groups.io/g/javaee

https://javaee.github.io/ has details of other the projects migrated from Java.net to GitHub

Jul 25 2016

Tool review: Merging Word documents on the desktop

Sometimes Microsoft Word users need to join several Word documents into a single file, without loss of formatting.

An example would be a cover letter, a quote, and a contract.

Or a proposal or contract, plus appendices.

(In legal industry parlance the finalised collection of documents is called a “closing binder”, “electronic bundle” or less commonly, a deal bible.  And its usually in PDF…)

Word itself doesn’t do this for you.  So this blog post is a review of tools you might download/install to try to get the job done.

TLDR: I don’t want to be negative, but bottom line is they won’t help … the 5 tools I found and tested each do a poor job at this.  If you, dear reader, know of a better tool, please share in a comment!   Most people seem to convert their documents to PDF first, then merge the PDFs – for good reason!

By way of background/disclosure, our Docx4j Enterprise product is good at this (if I may say so myself), but we don’t sell it to end users.  This background allowed me to make a couple of very simple documents to test the products.

Without further ado, the 5 products I tested were:

My first test was to merge 2 documents which define Heading 1 differently.  Does the merged document keep the distinct appearance of the 2 headings?

The only product which was able to pass this basic test was Icestand’s.

Unfortunately Icestand failed my second test: it lost section formatting (page orientation, headers/footers).

And my third test: list numbering.  If doc 1 contains “1,2,3” and so does doc 2, in the merged output, do you get “1,2,3  1,2,3” or “1,2,3 4,5,6”?

Since all the others failed test 1, I didn’t subject them to tests 2 or 3.  But I found:

  • most/all of these programs require Word to be installed.  That’s probably OK for a desktop utility.
  • with the exception of kutools, which appears in Word’s ribbon, each presents as standalone/free-standing software with its own UI.
  • you’d expect to be able to arrange your input documents in the order you want, but 3steps lacked even that!

Since these products all lose formatting, unless there’s a better product out there somewhere, you’d get better results by converting to PDF, then merging the PDF files.   In this respect, bundledocsCaseLines and others look interesting; they do the PDF conversion as well, removing a step from the process.

Using PDF might be fine, if the deal is done.  And interestingly, in some cases, PDF might even be a requirement.  For example, the UK Supreme Court “recommends” it.

But if the documents are still being finalised, Word is preferable, since its an editing format. Anyone tried converting the resulting merged PDF file back to Word?!

 

 

May 11 2016

Docx4j Helper Word AddIn: new version v3.3.0

We’ve just published a new version of our Helper AddIn for Word; you can download it from  http://www.plutext.com/dn/downloads/1472868282152/Docx4j_Helper_3-3-1-03.exe (link updated 3 Sept 2016).

Here’s what it looks like:

helper-ribbon

Its new features:

  • code generation from your selection (no need to use the webapp interface)
  • generate PDF output using Plutext’s commercial PDF Converter (either a locally installed instance, or http://converter-eval.plutext.com)
  • document sanitisation/anonymisation so you can safely publish it for support purposes
  • easy docDefaults manipulation

I’ll run through these one by one.

code generation from your selection

With the old version, you launched a local version of the docx4j webapp to generate code.

You can still do that (click the Load Helper button, then Parts List); its a useful way to see all the parts in your docx.

With this version, if you click the “Inspect selection” button, you’ll see the corresponding XML:

editor-hello-world

What’s new is the “Java” button.  If you click that, it’ll generate corresponding code, and display that in your web browser:

code-part1

:

code-part2

 

generate PDF output

The “PDF” button generates a PDF.  Either of your whole document, or your selection.

It uses our commercial PDF Converter so its probably most useful:

  1. if you want to evaluate that, or
  2. have found something which doesn’t convert correctly, and want technical support

It can use either a locally installed instance, or http://converter-eval.plutext.com

You configure that with the “Config” button.

The generated PDF will open in whatever Windows opens PDFs with for you.

document sanitisation/anonymisation

The idea of the “anonymise” button is to make it easy for you to email/publish a docx (or PDF) for support purposes, without giving away sensitive info.

Again, if nothing is selected, it’ll do your whole docx.  Otherwise, just your selection.

The results will be saved to a temporary local docx (so your source docx is unaffected), then opened.

If there is anything potentially sensitive the code can’t remove, it’ll let you know.

The code which does this is at https://github.com/plutext/docx4j/tree/master/src/main/java/org/docx4j/anon

easy docDefaults manipulation

Its sometimes useful to see/edit your doc defaults.

docdefaults

Your changes will open in a new temp docx.

For example, try changing font size (w:sz) from 22 to 42.

You can also make changes in Word’s Paragraph and Font dialogs by pressing the “Set as Default” button, then look here to see how that translates into XML.  Without this, its hard to be sure whether you’ve changed your default styles, or docDefaults!

Feedback and Comments

 

Any feedback, comments, requests for new features etc, please post in http://www.docx4java.org/forums/docx4jhelper-addin-f30/   Alternatively, Plutext customers can email support.

Apr 13 2016

PDF/A-2b compliant Word to PDF

Plutext’s commercial PDF Word/docx Converter now produces fully PDA/A-2b compliant PDF output.

We say this having tested its output using http://verapdf.org/ “a purpose-built, open source, file-format validator covering all PDF/A parts and conformance levels.”

You can try our PDF Converter now, at http://converter-eval.plutext.com/