I wish I didn’t have to: Decompiling Java

“I wish I didn’t have to” sounds like a category on its own, sort of like a complement to The Daily WTF. But, alas, I do — I need to understand what the differences are between two JARs, and I haven’t looked into Java decompilers since JAD (goes to show… something). But after some brief googling — Procyon rules! So here’s a script to do JAR diffs.

P.S. The humility of the author in designating a pretty good product as a 0 version (a propos — this is silly) led me to dig further into this project and it is cool stuff. Look forward to digging some more.

Java-to-Python converter

“Anything that can be done, could be done ‘meta'” (© Charles Simonyi) is right up there with “Laziness, impatience and hubris” (© Larry Wall) as pithy description of my development philosophy. Also, unfortunately, there’s another one: “Once it’s clear how toproceed, why bother to proceed” (or something like that). So, with that in mind…

I wanted a Python client library for GData (thankfully, they released
one last week
, so this is moot — good!), so I thought of automagically converting the Java library to Python. I tried Java2Python, but it’s based on ANTLR grammar for Java 1.4, and the library, of course, is in Java 5. As I was relearning ANTLR and writing all these actions by hand (the pain!), I took a break and found
Java 1.5 parser with AST generation and visitor suport by Julio Gesser (no relation,
I presume?) and Sreenivasa Viswanadha, based on JavaCC. Aha! Much easier… But then, of course, Google releases the Python version of the library I needed in the first place, so I don’t bother wrapping this project up… Here it is for whoever wants it: http://code.google.com/p/j2p/.

Dbdb – a JPDA-based single-stack debugger for mixed-language programming

Dbdb project is officially up for adoption, because I have no plans for working on it (I am sick of it).

Dbdb is a proof-of-concept of a JPDA-based single-stack debugger for mixed-language programming, done as an Eclipse plugin (but doesn’t have to be). It is based on Java 6 (“Mustang”). The proof-of-concept is allowing a developer to debug Java code that calls a PL/SQL stored procedure. The debugging session in Java proceeds normally, nothing to write home about. When a Statement.execute() (or similar) statement is executed, however, the debugger connects to the Oracle’s VM and shows a combined call stack, from Java down into PL/SQL. (See screenshot). The idea, of course, that it can be done with other combinations, but Java-into-Oracle-stored-proc is a very common scenario.

P.S. This is a rehash of an older post. I am trying to see what Blogger is like vs. LJ (for instance, LJ breaks javablogs feeds).

That’s it, done…

That’s it, done!

Bassem (Max) Jamaleddine

 
Prof.Madden finally approved the latest version of Dbdb write-up, and so I am all set for my 10+-years-overdue degree. With that, I’ve updated the sourceforge project
with all the latest stuff from my workspace, including the docs on the page, Javadoc, code (and aforementioned docs also) in CVS, etc (even a screenshot).
Dbdb project is officially open for adoption, because I have no plans for working on it (I am sick of it). Fly, baby, fly…
P.S.

  • I have to see whether Pat and Spencer actually decided to use this one for the IDEA Plugin Contest… There’s still time…
  • Maybe I do want to augment it for use with GWT, so it automagically inserts a debugger; statement as the first
    line any native Javascript method… Just for kicks… Nah, it would be too slow…

Tips

  • Cygwin productivity tips
  • It is possible to do ASP/JavaScript debugging without using the Visual Inter Dev by using IIS script debugger
  • If you ever see an error like “Java Plug-in for Netscape Navigator Should Not Be Used with Microsoft Microsoft Internet Explorer. Please Use Java Plug-in for Microsoft Internet Explorer”, see http://home.att.net/~cherokee67/ffjavaerror.html
  • Creating Windows EXE from Java (don’t ask):I found NativeJ the best for my task. I also tried JexePack, but it didn’t seem to do well with IBM JVM (it says that unregistered version will just show a modal dialog box on startup, and it did on my machine, but when I tried running it on the WebSphere box, it seemed to create an endless loop with those modal dialogs. Finally, Excelsior JET is good, but too complex for a little task like this — creating an executable involves too many steps skipping over things I don’t need. The interface is not the best either — too much like a wizard, too little like an IDE that it really is. But maybe it’s good for bigger, more involved projects.

Some notes

  1. One can argue that a step from one VM into another should not, from debugger’s point of view, be a step within one thread, but a new thread should be open. But this involves too much interfacing with the debugger (thinking about how to tell it that it needs to register a ThreadStartEvent to open a new thread when it didn’t request it, e.g.), so I scrap the idea, but feel it worthy of noting.
  2. %#@#$%@#$%@^@@^%

    java.lang.ClassCastException: org.hrum.dbdb.DelegatingThreadReference cannot be cast to com.sun.tools.jdi.ThreadReferenceImpl

    Did I mention how this pisses me off?

  3. Ok, even though Dbdb needs Java 6, the proof-of-concept is not supporting
    new JDBC driver
    loading
    … Just a note to self…

ReflectiveVisitor

Inigo Surguy presents< an implementation of Visitor pattern based on reflection. I found useful a slight modification of it, that allows for visiting not only the narrowest type, but every type possible , based on the visitAll flag: ReflectiveVisitor.java. This functionality comes in quite handy for tasks such as, for instance, validation. For example, if one has a complex class hierarchy (especially with multiple< inheritance) and different types have different requirements for it to be valid, and a concrete instance may implement a number of those, these validations can be well separated out.

See also:

  1. Visiting Collection elements
  2. Depth-first polymorphism

What’s going on here?

Consider the following code:

MethodEntryEvent evt;
ObjectReference con;
...

Class evtClass = evt.getClass();
System.out.println("Class of evt: " + evtClass);
System.out.println("Methods of evt: " +
                Arrays.asList(evtClass.getMethods()));
try {
  Value v = evt.returnValue();
  System.out.println(v);
} catch (Throwable ex) {
  ex.printStackTrace();
}

try {
  java.lang.reflect.Method retvalMethod =
          evtClass.getMethod("returnValue", null);
  retvalMethod.setAccessible(true);
  con = (ObjectReference)retvalMethod.invoke(evt, (Object[])null);
} catch (Throwable t) {
  t.printStackTrace();
}
System.out.println("Returned: " + con);

When running, this code prints the following:

 Class of evt: class com.sun.tools.jdi.EventSetImpl$MethodExitEventImpl Methods of evt: [ public com.sun.jdi.Value com.sun.tools.jdi.EventSetImpl$MethodExitEventImpl.returnValue(),  public java.lang.String com.sun.tools.jdi.EventSetImpl$LocatableEventImpl.toString(),  public com.sun.jdi.Method com.sun.tools.jdi.EventSetImpl$LocatableEventImpl.method(),  public com.sun.jdi.Location com.sun.tools.jdi.EventSetImpl$LocatableEventImpl.location(),  public com.sun.jdi.ThreadReference com.sun.tools.jdi.EventSetImpl$ThreadedEventImpl.thread(),  public int com.sun.tools.jdi.EventSetImpl$EventImpl.hashCode(),  public boolean com.sun.tools.jdi.EventSetImpl$EventImpl.equals(java.lang.Object),  public com.sun.jdi.request.EventRequest com.sun.tools.jdi.EventSetImpl$EventImpl.request(), public com.sun.jdi.VirtualMachine com.sun.tools.jdi.MirrorImpl.virtualMachine(),  public final native java.lang.Class java.lang.Object.getClass(),  public final void java.lang.Object.wait(long,int) throws java.lang.InterruptedException,  public final void java.lang.Object.wait() throws java.lang.InterruptedException,  public final native void java.lang.Object.wait(long) throws java.lang.InterruptedException,  public final native void java.lang.Object.notify(),  public final native void java.lang.Object.notifyAll()] java.lang.NoSuchMethodError: com.sun.jdi.event.MethodExitEvent.returnValue()Lcom/sun/jdi/Value; at org.hrum.dbdb.DriverManagerMethodExitEventListener.process(DriverManagerMethodExitEventListener.java:99) at org.hrum.dbdb.DbdbEventQueue.removeDebug(DbdbEventQueue.java:168) at org.hrum.dbdb.DbdbEventQueue.remove(DbdbEventQueue.java:47) at org.eclipse.jdt.internal.debug.core.EventDispatcher.run(EventDispatcher.java:226) at java.lang.Thread.run(Thread.java:619) Returned: instance of oracle.jdbc.driver.T4CConnection(id=435)(class com.sun.tools.jdi.ObjectReferenceImpl) 

Now, I will run this in debug mode and set a breakpoint at the red line above. When the breakpoint is hit, evaluation of evt.returnValue() returns an instance of com.sun.tools.jdi.ObjectReferenceImpl. However, when the execution is resumed, the result is as above (that is, evt.returnValue() results in a NoSuchMethodError).

Further, if we remove the green line (retvalMethod.setAccessible(true);), we will get an IllegalAccessException on the invocation:

Class org.hrum.dbdb.DriverManagerMethodExitEventListener can not access a member of class com.sun.tools.jdi.EventSetImpl$MethodExitEventImpl with modifiers “public”

What is going on?

I’d say it’s left as an exercise for the reader, but honestly, at the moment, I don’t feel like looking for an answer at all. I will perhaps let Bob and Dr. Heinz Max Kabutz (did I mention how much I enjoy referring to Dr.Heinz Max Kabutz?) to do the detective work…


ENVIRONMENT: This code is part of a plug-in project I am running in Eclipse 3.2RC3, with Mustang.

(no subject)

Nice to know that Sun’s implementation when manipulating ostensibly
insterfaces, in fact, expects them to be implementation classes inside.
For example, com.sun.tools.jdi.MirrorImpl.validateMirrorOrNull()
inside casts things to com.sun.tools.jdi.MirrorImpl rather than
com.sun.jdi.Mirror.

I love it when…

…I spend time working on something under an [reasonable] assumption
that I can do X, spend some more time realizing that I actually cannot,
lots more on cranking out convoluted code for working around that limitation, and
then find
out that this X has in fact been implemented in a later release than
the one I have…

Here, the feature X is being able, upon an exit from
the method, get the value it returned. This feature is there in
JDK
1.6
. I don’t need it anymore for now though… Maybe I will…

ProxyPlus?

I am backdating this entry, so I don’t remember quite what this was about…

E-mailed Shane:

Hi,

I am working on a project that necessitates precisely what you describe — Proxies of classes, not just interfaces. So thank you, that I, apparently, don’t have to delve into BCEL (I will let you know about the project when I have a bit more to show than I do now).

However, I’m a bit confused. In a description at http://www.gnufoo.org/java/java.html you point to your
junit.extensions.jfunc.util.ProxyPlus class, but it does not exist in the jcfe.jar I got from http://www.gnufoo.org/jcfe/, nor does it exist in the jfunc project. Where should I look for this functionality?

Thanks.

Got a bounce. So, for now, will do with explicit org.hrum.dbdb.JDBCDriverFactory‘s mechanism of going through drivers loaded, deregistering them and registering, in their stead, org.hrum.dbdb.DbdbDriver‘s that delegate to the registered drivers.

P.S. This is what P6Spy does

Monkey business


After reading <a
href=http://discuss.fogcreek.com/joelonsoftware/default.asp?cmd=show&ixPost=75607&ixReplies=51>this, I thought I’ll put in my couple of bucks… This is a RANT!

It seems that the orientation in business is towards “monkey”
programmers — those who do not think, but do as they are
told. This is because management, apparently (and justifiably),
believes that at any given time it is easier to hire a hundred
monkeys (those are trained ones, that do not type randomly,
and so less than a million and less than infinite time will suffice,
but this is not a good analogy anyway), than a Shakespeare – or even Dumas
(with his own monkeys, so that’s another bad analogy, woe is me!)

As a result, there are (the list is by no means exhaustive; Java is
the language unless otherwise specified — I think Java has produced
more monkeys who think they are software engineers than anything
else — at least VB does not lend one an air of superiority):

  • …monkeys who would rather sharpen the
    carpal-syndrome-inducing skills of cutting and pasting the same
    thing over and over again, rather than learn something like sed or Perl or a
    similar tool —
    or, indeed, spend some effort finding out about the existence
    of such tools and their availability on the monkey platform of
    choice (read: Windows) — or even finding out what plugins are
    available for their lovely IDE.

    IBM, for example, provides a framework called
    EAD4J, Enterprise
    Application Development for Java (it is only available with
    purchase of IBM IGS consulting services). It includes components
    similar to Struts, log4j, etc.
    The framework is well designed, but here is a catch — because
    of its design, adding or changing a service requires changes to
    about 8 files. There are abstract processes, process factories,
    interfaces, factories, XML files with queries, files containing constants to
    look up these queries, etc., etc. It would really be nice if there was
    a simple way to manage it, plugging in your logic where
    some IDE plugin or script do the, well, monkey
    job. Otherwise it’s overdesigned.

    Now, there are simple plugins for the current IDE of choice, WSAD, that at
    least allow generating these
    standard files (if not managing them, which is also important —
    change one signature, and you have to change several
    files). These plugins are provided by IGS
    But nooo, the monkeys here prefer to create all of this by hand. It’s
    a painful sight.

  • …macaques who cannot fathom how one
    could write a client-server application that does not communicate through
    XML requests embedded in HTTP, but – o, horror! – actually has its own
    application layer protocol.

  • …baboons who think that
    patterns
    are not merely possible (albeit very good) approaches to problems
    (and indeed are generalizations of good approaches to common
    problems that have arisen). In fact, they
    are the only way to solve problems, and that they must be copied from of
    the book, or else it wouldn’t work. They wouldn’t know a pattern they
    haven’t read about if it bit them on that place their head is
    forever hidden in. If GoF didn’t write about it, it ain’t a pattern.

  • Ok, I am tired of enumerating primate species. I’ll
    just give an anecdote.

    I wrote a module used by several teams. Because of the ever-changing
    requirements, some methods and classes became
    useless. I gave a fair warning by email, then I gave a second one by
    marking them deprecated in the code. I notice that the
    deprecated
    tags were periodically removed. I send mail about this, and mark them
    deprecated again. And again. And again.

    A monkey who was the team leader of another team came complaining that
    I should remove it, because he cannot perform a build. Everyone else
    can,
    but he can’t, and so I should remove the single tag (that is probably
    more useful to the whole project than anything he’s ever
    produced). He cannot be bothered to find out how to make
    it work? Why can everyone else make it work? Oh, he’s using some Ant
    scripts? What? That’s an excuse? What the hell does that
    have to do with anything? Oh, he didn’t write those
    scripts. Well, write your own, or take them from those
    people for whom they work. Oh, you don’t have time? Well,
    I don’t have time to keep giving you warnings you just
    ignore, you twit.

    Screw you, I finally thought, the warning has been there for some
    time. I’ll just remove this stuff altogether.
    His build promptly crashed. “Not my problem – we talked about this
    over 5 weeks ago!”, I gloated, producing the emails from my
    appropriately named CYA folder.

    As Butch said, “that’s what he gets for fucking up my sport.”

In short, they are not
Joel’s kind of programmers,
to put it mildly. Monkeys see and monkeys do. They do not think. They
have been taught a way to do things, and it is beyond them to figure
out that there could be another way. I honestly do not think they
understand what a boolean is (I submit that in their mind there is an
if statement, and then there’s a boolean type)
when they write:


if (thingie.isOk()) {

    return true;

} else {

    return false;

}

Then someone they blindly trust (it must be an established authority,
like a book/magazine — only that approved by an already established
authority, because monkeys do not further their education on their
own, — a manager, instructor at a paid course) tells
them about a ternary operator. Now they write:


return thingie.isOk() ? true : false;

The above two examples are from an actual production code.

Further, because monkeys do not think, they often reinvent the
wheel, badly. Which is also ironic, because they have been imbued with
all the right (and wrong) buzzwords, including “reuse”. I hesitate
to hazard a guess as to whether there is some meaning in their heads
they associate with this word, or is it just something they cry out
when playing free associations with their shrinks (“OO –
Encapsulation! Polymorphism! Reuse!”).

Here are some more anecdotes.

  • One programmer on a project wrote his own utilities to convert things
    from/to hex numbers, for crying out loud. Here is Java, the only thing
    he knows at all, and he can’t be bothered to think that maybe,
    just maybe, such a thing is a part of standard API.

  • This same monkey took several weeks to write a
    parser (for a very simple
    grammar, containing only certain expressions and operators such as
    ANDs and ORs). When I asked him why he didn’t use a
    parser generator (such as ANTLR, CUPS or JavaCC), he replied that
    he didn’t know any of them. Now, it is not a crime not to know a
    particular technology, but surely a programmer must be a) aware
    that there are such things as parser generators, and b) be
    able to learn how to use one. Whether he lacked the understanding or the
    desire to learn, is this the kind of developer you want?

  • Background: We needed to create some scripts doing export from the
    database. The export was to be done under some specific
    conditions, which were to be specified in the queries
    (that is, only export dependent tables if their parent
    tables are eligible to be exported, etc.) The logic was
    only in SQL queries, the rest were just scripts passing
    these queries to DB2 command-line, logging everything.
    All of those were written by hand, 80% time spent copying
    and pasting things, and then looking for places where the
    pasted things needed to be changed a bit (for example,
    some things are exported several times into different IXF
    files, because they are dependent on different
    things. These files need to be numbered sequentially, so
    next one does not overwrite the other. What do monkeys do?
    Number them by hand. Great.)

    When I suggested automating things, in fact, automating
    from the first step – even before writing our own queries,
    using the metadata to generate the
    queries themselves, I was looked at as if I just escaped
    from the mental asylum.

    Monkey But you cannot just rely on metadata, there are also
    functional links which are not foreign keys.

    Me Why are they not foreign keys in the first place?

    Monkey Because they are functional.

    Me Stop using that word. Tell me why are they not foreign keys?

    Monkey Because they are nullable.

    Me A foreign key can be nullable! Why is it not a foreign key?
    OK, whatever, that’s our DBA’s problem… But there’s a convention for
    functional keys anyway (we know they all start with
    SFK_, by convention). I’ll use that.

    Two days pass. My script works. A week later, they have problems
    with their original scripts. My approach works,
    demonstrably. But ok, they want to keep doing it their
    way, fine. They ask for help with their way – those scripts, wrapping
    hand-made SQL queries (which are already being automatically
    generated, but I’ll hold on that for now…)

    Monkey What are you doing?

    Me Writing a Perl script.

    Monkey But there is no Perl on Windows.

    Me See, I am sitting at a Windows machine and I have Perl.

    Monkey What is it for? I thought Perl was only for the Web?

    Me I am writing a script to generate your silly scripts
    from the small set of user input. The resulting files, which you
    are now doing BY HAND, are cluttered with repetitive stuff, such
    as error-handling code and file numbering, and it’s error-prone to do
    search and
    replace manually. So we’ll generate all these scripts using my script.

    Monkey But they don’t have Perl on their Windows.

    Me Who are “they”?

    Monkey The client?

    Me First of all, this is for the AIX machine. Second, this is
    not for them, we will just deliver the generated shell scripts, the
    Perl script is for us only.

    It takes several iterations for Monkey to get it.

    A day passes…

    Me Hey, where’s my Perl script I wrote to generate the import
    scripts?

    Monkey We have to have only shell scripts.

    Me Yes, I used that one to create those shell scripts, dammit!

    Monkey (sits writing these shell scripts again by hand. At the
    moment, manually replacing some upper-case strings into lower-case)
    I
    removed it from CVS. They only want shell scripts on their machine.

    Me It wasn’t going on their machine! It’s only for us!!!

    Monkey Here, I changed these files already, you change the
    rest.

    Me (giving up) OK.

    Monkey Oh, and they have to be K-shell. Change them all to
    .ksh.

    Me Why do they have to be ksh? What’s wrong with sh? They are
    all very simple anyway, just call db2 import, check error status,
    that’s it.

    Monkey They have to be K shell. That’s what the DBA said.

    Me What the hell does the DBA have to do with it?

    Monkey He wants to be able change them, and he doesn’t know sh,
    only ksh.

    Me Ok, fine. I suppose you’re right, echo is
    different in K-shell.

    Monkey misses the seething sarcasm. Of course.

    Monkey Right here.

    Me I don’t see them. What is this OAD_0035.ksh? Is that it?

    Monkey Yes.

    Me What does this mean? What do these numbers mean?

    Monkey That’s what they said they should be called.

    Me Who are “they”???

    Silence.

    Me OK, you have a script called OAD_0035.ksh calling
    OAD_0038.ksh, which in turn calls OAD_0038_1.ksh, OAD_0039_2.ksh, etc.
    Why are they called this? It’s hard to remember which one is which.

    Monkey Why do you want to know what it means?

    Me Because if I don’t know what it means, it’s much harder for
    me to look at the file and see what is supposed to be inside. Ah, I
    see
    you added the insightful comment inside each file with its meaningful
    name. Ah, I see also, you use that stupid name inside of it over and
    over again, to write to the logs, instead of just using $0. (deep
    breath). I’ll just create some symbolic links to them with meaningful
    names, so I know what’s going on…

    An hour later

    Me Where are my links?

    Monkey They only wanted files there that are named like
    OAD_0035, etc.
    Me What the hell do these numbers mean???

    Monkey I don’t know. For security.

    Me (pause) Who told you to do this?

    Monkey The client.

    Me The client is a company. Who have you met from the company?

    Monkey I don’t know. They said the client wants this.

    Me Who said? Where? When?

    Now I’m really curious. I turn with this question to
    others. Finally I come to the last monkey who knows.

    Monkey 5 Uh, the client has guidelines on what they are
    supposed to be called. It should start with OAD, and
    then underscore, then four characters.

    Me Why four characters?

    Monkey 5 For normalization.

    Me What normalization?! What can you possibly
    change in a
    luminous egg
    , I mean, normalize in a 20-line shell script?

    Monkey 5 So they can keep them consistent and do some things to
    all of them, regardless of what they are called.

    Me What can they possibly want to do with shell scripts? Rename
    them to some other numeric pattern? There isn’t even any method as to
    how they are named, it’s not like a certain number
    pattern means it’s dependent on the other. You just
    named them randomly…

    Curtain

    But hey, fire one, and the replacement is easy to find. That’s true.
    I suppose Henry Ford would be proud, but isn’t this a backward
    approach? You don’t need monkeys at all, most of this work can
    be automated.

    Maybe I need another line of work. 🙂