building javadoc with modules

[Justin Lee]

I'm trying to run javadoc against the java9 src.zip and I have no idea how
to handle the modules. Even if I go with the classic "javadoc @sources"
approach I get an error about too many modules defined. I've tried a
number of variations but I can't quite seem to crack that nut.

My latest attempt looks like this:

javadoc -d /tmp/javadoc9 -html5 --module \
java.management.rmi \
jdk.packager.services \
jdk.scripting.nashorn.shell \
<more module names here...> \
-Xmaxerrs 1000 \
@source.files



Does anyone have an example of running the javadoc tool from the command
line and generating docs for modularized code like this? Thanks.

[Jonathan Gibbons]

Justin,

This ought to be simple, but as you discovered, it isn't.

I tried to generate the docs using the JDK 10 src.zip, available here:
https://download.java.net/openjdk/jdk10/ri/openjdk-10_src.zip

After ou unzip the sources, the command ought to be as simple as
something like the following:

/opt/jdk/10/bin/javadoc \
--module-source-path 'openjdk/src/*/{linux,unix,share}/classes' \
-d api \
--expand-requires transitive \
--module java.se

Here's what those lines mean:

/opt/jdk/10/bin/javadoc \# path to javadoc
--module-source-path 'openjdk/src/*/{linux,unix,share}/classes' \#
specify "pattern" for location of sources
-d api \ # output directory
--expand-requires transitive \ #saves typing lots of module names
--module java.se # the root module name (plus its transitive
dependencies)

Note: the combination of --module java.se --expand-requires transitive
will just give you the java.* modules (because java.se does not depend
on any jdk.* modules); if you want the JDK modules as well, you'll have
to list them separately,

So what's the problem?

If you run the command, it gives lots of "cannot find symbol" messages
for a comparatively small number of types. The most frequent one that
showed up for me was java.nio.ByteBuffer ... and therein lies the hint
of the problem. There is no source for java.nio.ByteBuffer in the
src.zip file! It turns out that for a number of classes, including
{Byte,Char}Buffer, Charset{De,En}coder, the source is generated as part
of the full JDK build.

There are two possible workarounds:

1. Run the full build first, then figure out the path to the gensrc
directory. For my system, it is something like
build/linux-x86_64-normal-server-release/support/gensrc/
meaning that the --module-source-path option has to be something like:

-module-source-path
'openjdk/src/*/{linux,unix,share}/classes:build/linux-x86_64-normal-server-release/support/gensrc/*'

If you're looking to generate full docs for all of Java SE, this is
the recommended solution, but if you're going to build JDK, then you
might as well use the JDK makefiles to build the docs as well, with just
"make docs" or "make docs-jdk-api" or something like that.

2. The other option is much more verbose, and not recommended if you're
trying to generate full documentation for Java SE.
It relies on the fact that the definitions for the missing symbols do
exist in JDK itself, for the right version of JDK. So you can use the
--patch-module option to "patch" every module you want to document with
its source code. If you're looking to modify one or a few modules, and
generate updated docs, this may be a reasonable approach, but if you're
looking to generate docs for all 72 JDK modules, that's a long command
line! You will still need the basic --module-source-path option, to keep
javadoc happy and to tell it that you're working in "multi-module mode",
but you won't need to provide the gensrc directory.

The alternative is to find/use a consolidated src.zip file that (just)
contains all the source for your platform, and nothing else, in a single
simple exploded module hierarchy.

-- Jon

I'm trying to runjavadocagainst the java9 src.zip and I have no idea
how to handle the modules. Even if I go with the classic
defined. I've tried a number of variations but I can't quite seem to
crack that nut.
javadoc -d /tmp/javadoc9 -html5 --module \
java.management.rmi \
jdk.packager.services \
jdk.scripting.nashorn.shell \
<more module names here...> \
-Xmaxerrs 1000 \
@source.files
Does anyone have an example of running thejavadoctool from the command
line and generating docs for modularized code like this? Thanks.
--
http://antwerkz.com <http://antwerkz.com/> http://antwerkz.com/+
http://antwerkz.com/twitter http://antwerkz.com/github

[Jonathan Gibbons]

Justin,

It seems there are two forms of src.zip out in the wild. My previous
response was based on using a src.zip that is essentially just a zip of
the contents of appropriate OpenJDK repo. As such, it has top level
directories like "make", "src", "test", and the src/ directory does not
contain any of the generated sources.

This response is based on the other form of src.zip, which is
platform-specific and just contains all the source files for a build.
The entries in this zip are of the form <module>/<package>/<class>.java

The basic form of the javadoc command is this:

/opt/jdk/1.9.0/bin/javadoc \
--module-source-path 'play/javadoc-demo/src/*' \
-d play/javadoc-demo/api \
-quiet \
-Xmaxerrs 10000 \
-Xmaxwarns 10000 \
--expand-requires transitive \
--module java.se

Compared to my notes from yesterday, this has a simplified
`--module-source path`, and has additional options `-quiet`,
`-Xmaxerrs`, `-Xmaxwarns`.
The argument to module-source-path should be the path to the expanded
zip file, followed by '/*' (or `\*` on Windows.) You probably need to
quote the argument to prevent your shell expanding the '*': that is
important.

This command should run to completion, although it generates (for me)
1704 errors and 918 warnings.

The warnings are embarrassing, and are mostly about bad javadoc comments.

The errors are all about custom tags being used in the documentation
comments. You can provide definitions for most of these on the javadoc
command line, like this:

/opt/jdk/1.9.0/bin/javadoc \
--module-source-path 'play/javadoc-demo/src/*' \
-d play/javadoc-demo/api \
-quiet \
-Xmaxerrs 10000 \
-Xmaxwarns 10000 \
--expand-requires transitive \
--module java.se \
-tag beaninfo:X \
-tag revised:X \
-tag since.unbundled:X \
-tag spec:X \
-tag specdefault:X \
-tag Note:X \
-tag ToDo:X \
-tag 'apiNote:a:API Note:' \
-tag 'implSpec:a:Implementation Requirements:' \
-tag 'implNote:a:Implementation Note:' \
-tag param \
-tag return \
-tag throws \
-tag since \
-tag serialData \
-tag factory \
-tag see \
-tag 'jvms:a:See <cite>The Java&trade; Virtual Machine
Specification</cite>:' \
-tag 'jls:a:See <cite>The Java&trade; Language Specification</cite>:'

The list is taken from the JDK makefiles. That omits definitions for 3
tags, for {@extLink}, {@incubating} and {@moduleGraph} which are handled
by taglets (i.e. compiled source code) contained in the JDK build. With
that extended command, the counts go down to 79 errors, 918 warnings,
which is likely as far as we'll get without using the taglets and/or
fixing javadoc comments.

Note that the actual JDK API documentation is generated with additional
options, to set header and footer text containing license and other
information.

I hope this helps.

-- Jon

Post by Jonathan Gibbons
Justin,
This ought to be simple, but as you discovered, it isn't.
https://download.java.net/openjdk/jdk10/ri/openjdk-10_src.zip
After ou unzip the sources, the command ought to be as simple as
/opt/jdk/10/bin/javadoc \
--module-source-path 'openjdk/src/*/{linux,unix,share}/classes' \
-d api \
--expand-requires transitive \
--module java.se
/opt/jdk/10/bin/javadoc \# path to javadoc
--module-source-path 'openjdk/src/*/{linux,unix,share}/classes' \#
specify "pattern" for location of sources
-d api \ # output directory
--expand-requires transitive \ #saves typing lots of module names
--module java.se # the root module name (plus its transitive
dependencies)
Note: the combination of --module java.se --expand-requires transitive
will just give you the java.* modules (because java.se does not depend
on any jdk.* modules); if you want the JDK modules as well, you'll
have to list them separately,
So what's the problem?
If you run the command, it gives lots of "cannot find symbol" messages
for a comparatively small number of types. The most frequent one that
showed up for me was java.nio.ByteBuffer ... and therein lies the hint
of the problem. There is no source for java.nio.ByteBuffer in the
src.zip file! It turns out that for a number of classes, including
{Byte,Char}Buffer, Charset{De,En}coder, the source is generated as
part of the full JDK build.
1. Run the full build first, then figure out the path to the gensrc
directory. For my system, it is something like
build/linux-x86_64-normal-server-release/support/gensrc/
-module-source-path
'openjdk/src/*/{linux,unix,share}/classes:build/linux-x86_64-normal-server-release/support/gensrc/*'
If you're looking to generate full docs for all of Java SE, this
is the recommended solution, but if you're going to build JDK, then
you might as well use the JDK makefiles to build the docs as well,
with just "make docs" or "make docs-jdk-api" or something like that.
2. The other option is much more verbose, and not recommended if
you're trying to generate full documentation for Java SE.
It relies on the fact that the definitions for the missing symbols do
exist in JDK itself, for the right version of JDK. So you can use the
--patch-module option to "patch" every module you want to document
with its source code. If you're looking to modify one or a few
modules, and generate updated docs, this may be a reasonable approach,
but if you're looking to generate docs for all 72 JDK modules, that's
a long command line! You will still need the basic
--module-source-path option, to keep javadoc happy and to tell it that
you're working in "multi-module mode", but you won't need to provide
the gensrc directory.
The alternative is to find/use a consolidated src.zip file that (just)
contains all the source for your platform, and nothing else, in a
single simple exploded module hierarchy.
-- Jon

I'm trying to runjavadocagainst the java9 src.zip and I have no idea
how to handle the modules. Even if I go with the classic
defined. I've tried a number of variations but I can't quite seem to
crack that nut.
javadoc -d /tmp/javadoc9 -html5 --module \
java.management.rmi \
jdk.packager.services \
jdk.scripting.nashorn.shell \
<more module names here...> \
-Xmaxerrs 1000 \
@source.files
Does anyone have an example of running thejavadoctool from the
command line and generating docs for modularized code like this? Thanks.
--
http://antwerkz.com <http://antwerkz.com/> http://antwerkz.com/+
http://antwerkz.com/twitter http://antwerkz.com/github

[Jonathan Gibbons]

Justin,

Thanks for the extra info.

There is one improvement coming soon for --module-source-path.

The current spec embodies an assumption about the naming of directories
contain source code.  This may not hold for some existing projects that
may be converted to use modules.  The improvement will address that
shortcoming.

Enhancement: https://bugs.openjdk.java.net/browse/JDK-8208608
CSR: https://bugs.openjdk.java.net/browse/JDK-8208609

-- Jon

I've been using the src.zip that comes with the JDK install on my
mac.  In general, I'm going to be applying this process to arbitrary
source jars for various maven dependencies and so i'm just trying to
work out how all the parameters now work in a modularized world.  I'll
play with what you've shown here.  It looks really promising.  Thanks
for all that feedback.  It was definitely more than I was hoping for. 
Here's hoping it all starts to make sense for me now.
On Thu, Aug 30, 2018 at 5:04 PM, Jonathan Gibbons
Justin,
It seems there are two forms of src.zip out in the wild. My
previous response was based on using a src.zip that is essentially
just a zip of the contents of appropriate OpenJDK repo.  As such,
it has top level directories like "make", "src", "test", and the
src/ directory does not contain any of the generated sources.
This response is based on the other form of src.zip, which is
platform-specific and just contains all the source files for a
build. The entries in this zip are of the form
<module>/<package>/<class>.java
/opt/jdk/1.9.0/bin/javadoc \
    --module-source-path 'play/javadoc-demo/src/*' \
    -d play/javadoc-demo/api \
    -quiet \
    -Xmaxerrs 10000 \
    -Xmaxwarns 10000 \
    --expand-requires transitive \
    --module java.se <http://java.se>
Compared to my notes from yesterday, this has a simplified 
`--module-source path`, and has additional options `-quiet`,
`-Xmaxerrs`, `-Xmaxwarns`.
The argument to module-source-path should be the path to the
expanded zip file, followed by '/*' (or `\*` on Windows.) You
probably need to quote the argument to prevent your shell
expanding the '*': that is important.
This command should run to completion, although it generates (for
me) 1704 errors and 918 warnings.
The warnings are embarrassing, and are mostly about bad javadoc comments.
The errors are all about custom tags being used in the
documentation comments.  You can provide definitions for most of
/opt/jdk/1.9.0/bin/javadoc \
    --module-source-path 'play/javadoc-demo/src/*' \
    -d play/javadoc-demo/api \
    -quiet \
    -Xmaxerrs 10000 \
    -Xmaxwarns 10000 \
    --expand-requires transitive \
    --module java.se <http://java.se> \
    -tag beaninfo:X \
    -tag revised:X \
    -tag since.unbundled:X \
    -tag spec:X \
    -tag specdefault:X \
    -tag Note:X \
    -tag ToDo:X \
    -tag 'apiNote:a:API Note:' \
    -tag 'implSpec:a:Implementation Requirements:' \
    -tag 'implNote:a:Implementation Note:' \
    -tag param \
    -tag return \
    -tag throws \
    -tag since \
    -tag serialData \
    -tag factory \
    -tag see \
    -tag 'jvms:a:See <cite>The Java&trade; Virtual Machine
Specification</cite>:' \
    -tag 'jls:a:See <cite>The Java&trade; Language
Specification</cite>:'
The list is taken from the JDK makefiles. That omits definitions
are handled by taglets (i.e. compiled source code) contained in
the JDK build. With that extended command, the counts go down to
79 errors, 918 warnings, which is likely as far as we'll get
without using the taglets and/or fixing javadoc comments.
Note that the actual JDK API documentation is generated with
additional options, to set header and footer text containing
license and other information.
I hope this helps.
-- Jon

Post by Jonathan Gibbons
Justin,
This ought to be simple, but as you discovered, it isn't.
https://download.java.net/openjdk/jdk10/ri/openjdk-10_src.zip
<https://download.java.net/openjdk/jdk10/ri/openjdk-10_src.zip>
After ou unzip the sources, the command ought to be as simple as
/opt/jdk/10/bin/javadoc \
    --module-source-path 'openjdk/src/*/{linux,unix,share}/classes' \
    -d api \
    --expand-requires transitive \
    --module java.se <http://java.se>
/opt/jdk/10/bin/javadoc \# path to javadoc
    --module-source-path
'openjdk/src/*/{linux,unix,share}/classes' \# specify "pattern"
for location of sources
    -d api \ # output directory
    --expand-requires transitive \ #saves typing lots of module names
    --module java.se <http://java.se> # the root module name
(plus its transitive dependencies)
Note: the combination of --module java.se <http://java.se>
--expand-requires transitive will just give you the java.*
modules (because java.se <http://java.se> does not depend on any
jdk.* modules); if you want the JDK modules as well, you'll have
to list them separately,
So what's the problem?
If you run the command, it gives lots of "cannot find symbol"
messages for a comparatively small number of types. The most
frequent one that showed up for me was java.nio.ByteBuffer ...
and therein lies the hint of the problem. There is no source for
java.nio.ByteBuffer in the src.zip file! It turns out that for a
number of classes, including {Byte,Char}Buffer,
Charset{De,En}coder, the source is generated as part of the full
JDK build.
1. Run the full build first, then figure out the path to the
gensrc directory. For my system, it is something like
        build/linux-x86_64-normal-server-release/support/gensrc/
    meaning that the --module-source-path option has to be
-module-source-path
'openjdk/src/*/{linux,unix,share}/classes:build/linux-x86_64-normal-server-release/support/gensrc/*'
    If you're looking to generate full docs for all of Java SE,
this is the recommended solution, but if you're going to build
JDK, then you might as well use the JDK makefiles to build the
docs as well, with just "make docs" or "make docs-jdk-api" or
something like that.
2. The other option is much more verbose, and not recommended if
you're trying to generate full documentation for Java SE.
It relies on the fact that the definitions for the missing
symbols do exist in JDK itself, for the right version of JDK. So
you can use the --patch-module option to "patch" every module you
want to document with its source code.  If you're looking to
modify one or a few modules, and generate updated docs, this may
be a reasonable approach, but if you're looking to generate docs
for all 72 JDK modules, that's a long command line! You will
still need the basic --module-source-path option, to keep javadoc
happy and to tell it that you're working in "multi-module mode",
but you won't need to provide the gensrc directory.
The alternative is to find/use a consolidated src.zip file that
(just) contains all the source for your platform, and nothing
else, in a single simple exploded module hierarchy.
-- Jon

I'm trying to runjavadocagainst the java9 src.zip and I have no
idea how to handle the modules.  Even if I go with the classic
defined.  I've tried a number of variations but I can't quite
seem to crack that nut.
javadoc -d /tmp/javadoc9 -html5 --module \
 java.management.rmi \
 jdk.packager.services \
 jdk.scripting.nashorn.shell \
 <more module names here...> \
 -Xmaxerrs 1000 \
Does anyone have an example of running thejavadoctool from the
command line and generating docs for modularized code like
this?  Thanks.
--
http://antwerkz.com <http://antwerkz.com/> http://antwerkz.com/+
http://antwerkz.com/twitter http://antwerkz.com/github

--
http://antwerkz.com <http://antwerkz.com/> http://antwerkz.com/+
http://antwerkz.com/twitter http://antwerkz.com/github

[Jonathan Gibbons]

Justin,

To follow up on this some more, can you tell me where you are getting
"the java9 src.zip" that you refer to?
Or, if you can't recall, can you check the layout of the contents?

Is it like this:

openjdk/ADDITIONAL_LICENSE_INFO
openjdk/ASSEMBLY_EXCEPTION
openjdk/bin
openjdk/configure
openjdk/doc
openjdk/LICENSE
openjdk/make
openjdk/Makefile
openjdk/README
openjdk/src
openjdk/test

Or more like this:

java.base/com
java.base/java
java.base/javax
java.base/jdk
java.base/module-info.java
java.base/sun
java.compiler/javax
java.compiler/module-info.java
java.datatransfer/java
java.datatransfer/module-info.java
java.datatransfer/sun
java.desktop/com
java.desktop/java
java.desktop/javax
java.desktop/module-info.java
etc

The first form is essentially a zip of the repo, that I found here, for
JDK 10:
https://download.java.net/openjdk/jdk10/ri/openjdk-10_src.zip
There is an equivalent one for JDK 9, here:
https://download.java.net/openjdk/jdk9/ri/openjdk-9_src.zip
This is the form that I was using in my response yesterday.

The second form is "just" the sources, in a simple layout, and is
specific to a single platform.
It does contain the generated sources. This zip file is generated
automatically during the
build on a specific platform.


-- Jon

I'm trying to runjavadocagainst the java9 src.zip and I have no idea
how to handle the modules. Even if I go with the classic
defined. I've tried a number of variations but I can't quite seem to
crack that nut.
javadoc -d /tmp/javadoc9 -html5 --module \
java.management.rmi \
jdk.packager.services \
jdk.scripting.nashorn.shell \
<more module names here...> \
-Xmaxerrs 1000 \
@source.files
Does anyone have an example of running thejavadoctool from the command
line and generating docs for modularized code like this? Thanks.
--
http://antwerkz.com <http://antwerkz.com/> http://antwerkz.com/+
http://antwerkz.com/twitter http://antwerkz.com/github