Hi,
Well, the serialization fiasco has mostly been figured out.
It turns out that yes, my serializer does work. Moreover, it works perfectly, and everything would be good . . . except that the deserializer does not. As it happens, Java's deserialization is, put simply, broken. There's nothing for it. In real life (C++) I'd use a copy constructor, and that would be the end of it, but Java apparently can't handle that.
Unfortunately, the deserializer doesn't, and can't work, unless the GWT designers had thought of it, which they didn't. We'd be stuck, except Joe Collard put into words what I was coming to the realization was necessary (on same post by Nialls mentioned previously, here): "see if you can write a wrapper that contains all of the things you will want saved".
Essentially, the idea boils down to: write your own serializer. It's bloody messy, but Java's (de)serialization is broken and reflection isn't supported by GWT. So, we have to go with it.
No private/protected fields can, of course, be serialized. Hopefully this shouldn't be an issue. It appears to have worked for Joe's group, so hopefully it will work for ours.
Ian
Saturday, March 24, 2012
Serialization Update
Hi,
So, when running the serializer, GWT will produce errors:
[ERROR] [web_final] - Errors in 'file:/C:/dev/Java/CS%20460%20Project/Web_Final/src/cs460/webdnd/client/utilities/serialization/serializer.java'
[ERROR] [web_final] - Line 24: No source code is available for type java.io.ByteArrayOutputStream; did you forget to inherit a required module?
[ERROR] [web_final] - Line 25: No source code is available for type java.io.ObjectOutputStream; did you forget to inherit a required module?
[ERROR] [web_final] - Line 39: No source code is available for type java.io.InputStream; did you forget to inherit a required module?
[ERROR] [web_final] - Line 39: No source code is available for type java.io.ByteArrayInputStream; did you forget to inherit a required module?
[ERROR] [web_final] - Line 40: No source code is available for type java.io.ObjectInputStream; did you forget to inherit a required module?
[ERROR] [web_final] - Line 47: No source code is available for type java.lang.ClassNotFoundException; did you forget to inherit a required module?
In response to this, Nialls took over serialization/deserialization using code he found online. This code produces tons of errors, as he writes here: http://niallsc.blogspot.com/2012/03/canvas-serialization-status-failed.html.
However, this misses something--my serializer worked. It produces this output (if you add a print statement):
Now, this looks to me like taking a canvas and serializing it (and I can get a canvas back, deserializing it). I'm not sure what the problem is with Nialls's code, but mine seemed to work.
I feel somewhat accused, but I submit the above as evidence that the code I wrote is working properly. There were limited tests with more complex canvases (nested, with content, etc.) that were conducted some time ago, successfully. It is possible that there is some new problem (like maybe a certain component can't be serialized) which can be solved in the same way as the canvas's serialization was.
Now, I don't know what to say. Maybe there's something subtle going on that I'm missing. Maybe I don't know what I'm talking about. Maybe there's a fundamental flaw in this that really will somehow prevent this from working. I welcome explanations.
Regardless, I think it's a good idea to meet Sunday as Nialls suggested to work this out. For the good of the project, it needs to be solved sooner rather than later--and I had already thought it was solved.
Thanks,
Ian
So, when running the serializer, GWT will produce errors:
[ERROR] [web_final] - Errors in 'file:/C:/dev/Java/CS%20460%20Project/Web_Final/src/cs460/webdnd/client/utilities/serialization/serializer.java'
[ERROR] [web_final] - Line 24: No source code is available for type java.io.ByteArrayOutputStream; did you forget to inherit a required module?
[ERROR] [web_final] - Line 25: No source code is available for type java.io.ObjectOutputStream; did you forget to inherit a required module?
[ERROR] [web_final] - Line 39: No source code is available for type java.io.InputStream; did you forget to inherit a required module?
[ERROR] [web_final] - Line 39: No source code is available for type java.io.ByteArrayInputStream; did you forget to inherit a required module?
[ERROR] [web_final] - Line 40: No source code is available for type java.io.ObjectInputStream; did you forget to inherit a required module?
[ERROR] [web_final] - Line 47: No source code is available for type java.lang.ClassNotFoundException; did you forget to inherit a required module?
In response to this, Nialls took over serialization/deserialization using code he found online. This code produces tons of errors, as he writes here: http://niallsc.blogspot.com/2012/03/canvas-serialization-status-failed.html.
However, this misses something--my serializer worked. It produces this output (if you add a print statement):
Now, this looks to me like taking a canvas and serializing it (and I can get a canvas back, deserializing it). I'm not sure what the problem is with Nialls's code, but mine seemed to work.
I feel somewhat accused, but I submit the above as evidence that the code I wrote is working properly. There were limited tests with more complex canvases (nested, with content, etc.) that were conducted some time ago, successfully. It is possible that there is some new problem (like maybe a certain component can't be serialized) which can be solved in the same way as the canvas's serialization was.
Now, I don't know what to say. Maybe there's something subtle going on that I'm missing. Maybe I don't know what I'm talking about. Maybe there's a fundamental flaw in this that really will somehow prevent this from working. I welcome explanations.
Regardless, I think it's a good idea to meet Sunday as Nialls suggested to work this out. For the good of the project, it needs to be solved sooner rather than later--and I had already thought it was solved.
Thanks,
Ian
Serializer/Deserializer
Hi,
So, since last time, I have added a new class "CanvasSerializable", which is a direct subclass of "Canvas", except it adds support for serialization. If every canvas is serializable, then making the serializer work properly is much easier, and allows for better design.
The changes affected a lot of the project, so I changed every single reference to Canvas to CanvasSerializable, and then converted things back that just flat out couldn't be (e.g., GWT callbacks return Canvas[], so I can't upcast them to CanvasSerializable).
This seems to have had an effect already. The serializer appears to be working properly. There's still some problems on the server-side (I think) so serializing doesn't quite work. Because there's no serialized data, naturally, the deserializer doesn't work either.
Hopefully, over the next few posts, the serializer module may be completed!
Ian
So, since last time, I have added a new class "CanvasSerializable", which is a direct subclass of "Canvas", except it adds support for serialization. If every canvas is serializable, then making the serializer work properly is much easier, and allows for better design.
The changes affected a lot of the project, so I changed every single reference to Canvas to CanvasSerializable, and then converted things back that just flat out couldn't be (e.g., GWT callbacks return Canvas[], so I can't upcast them to CanvasSerializable).
This seems to have had an effect already. The serializer appears to be working properly. There's still some problems on the server-side (I think) so serializing doesn't quite work. Because there's no serialized data, naturally, the deserializer doesn't work either.
Hopefully, over the next few posts, the serializer module may be completed!
Ian
RSA Cryptography
Hi,
I wrote a little cryptography module. It implemented a (theoretically) fully secure RSA cryptosystem.
This turned out to be harder than anticipated. In addition to jogging my memory of RSA, implementing it in Java is a nightmare (exercise of actual real programming).
It turns out that there are two different APIs for this. The one most directly implemented by Java has a bug that prevents it from working except when writing to a file. The second is also too buggy to use.
So, I decided to write my own implementation. It turns out that there are a number of (buggy) Java implementations. These fail in a number of cases, (usually when a message is too long). Obviously, real RSA doesn't crash. I ended up basing my solution on an existing RSA implementation and then rewriting half of it to actually work.
In the process, I also tried compressing the RSA output stream. It turns out that the Java "Inflate" class has a bug that causes it to hang on the general kind of data that can be produced by compressing an RSA stream.
I can't really do anything about that, because if I were to try to fix all the bugs in the JRE, I should at least get paid--and anyway, at this time, there are more pressing matters, such as serialization, updates concerning which should appear on the next post.
Ian
I wrote a little cryptography module. It implemented a (theoretically) fully secure RSA cryptosystem.
This turned out to be harder than anticipated. In addition to jogging my memory of RSA, implementing it in Java is a nightmare (exercise of actual real programming).
It turns out that there are two different APIs for this. The one most directly implemented by Java has a bug that prevents it from working except when writing to a file. The second is also too buggy to use.
So, I decided to write my own implementation. It turns out that there are a number of (buggy) Java implementations. These fail in a number of cases, (usually when a message is too long). Obviously, real RSA doesn't crash. I ended up basing my solution on an existing RSA implementation and then rewriting half of it to actually work.
In the process, I also tried compressing the RSA output stream. It turns out that the Java "Inflate" class has a bug that causes it to hang on the general kind of data that can be produced by compressing an RSA stream.
I can't really do anything about that, because if I were to try to fix all the bugs in the JRE, I should at least get paid--and anyway, at this time, there are more pressing matters, such as serialization, updates concerning which should appear on the next post.
Ian
Thursday, March 22, 2012
Compression
Hi,
Well, I took a little break from doing parsing and serialization whilst I wait for others to get set up to use the serializer/deserializer.
Some, especially Alex, expressed concern that serialized data or the generated HTML would be very large and difficult to send. So, I wrote a compression wrapper around Java's Inflate/Deflate APIs. It turns out that Java's String.getBytes(...) method is broken, which made this difficult. After some false starts, I finally just gave up trying to work around their bugs and just wrote out what the methods were supposed to do myself.
The algorithm is the standard ZLIB compression algorithm, which should be able to achieve very good compression rates.
I may try writing an encryption algorithm as well.
Ian
Well, I took a little break from doing parsing and serialization whilst I wait for others to get set up to use the serializer/deserializer.
Some, especially Alex, expressed concern that serialized data or the generated HTML would be very large and difficult to send. So, I wrote a compression wrapper around Java's Inflate/Deflate APIs. It turns out that Java's String.getBytes(...) method is broken, which made this difficult. After some false starts, I finally just gave up trying to work around their bugs and just wrote out what the methods were supposed to do myself.
The algorithm is the standard ZLIB compression algorithm, which should be able to achieve very good compression rates.
I may try writing an encryption algorithm as well.
Ian
Regular Expression for Tag Processing
Hi,
Last time I mentioned that regular expressions might be useful for breaking a tag into elements. My regular expression is:
([^\s\"=]+\s*=\s*([^\s\"=]+|\"[^\s^\"]+\"))|([^\s\"=]+)
[Edit: Updating for single quotes and other improvements: ([^\s\"\'=]+\s*=\s*([^\s\"\'=]+|[\"\'][^\s\"\']+[\"\']))|([^\s\"\'=]+)]
Notice that it is awesome. I tested it on the input data:
div class=hijkjh width=100 height = 6 src= "hello.png" why=bec6=ause src="hello.png" src="hello.png?=yo?" src="hello.png" src="hello.png" src="hello.png" sup
. . . which is deliberately malformed in some cases. You can test the regex at http://regexpal.com/. Notice that the regex matches valid elements. All I have to do to determine if a tag is well formed is check whether anything other than whitespace is unmatched.
I have merged these changes into the converter itself, so the new tag system should work much better.
Ian
Last time I mentioned that regular expressions might be useful for breaking a tag into elements. My regular expression is:
([^\s\"=]+\s*=\s*([^\s\"=]+|\"[^\s^\"]+\"))|([^\s\"=]+)
[Edit: Updating for single quotes and other improvements: ([^\s\"\'=]+\s*=\s*([^\s\"\'=]+|[\"\'][^\s\"\']+[\"\']))|([^\s\"\'=]+)]
Notice that it is awesome. I tested it on the input data:
div class=hijkjh width=100 height = 6 src= "hello.png" why=bec6=ause src="hello.png" src="hello.png?=yo?" src="hello.png" src="hello.png" src="hello.png" sup
. . . which is deliberately malformed in some cases. You can test the regex at http://regexpal.com/. Notice that the regex matches valid elements. All I have to do to determine if a tag is well formed is check whether anything other than whitespace is unmatched.
I have merged these changes into the converter itself, so the new tag system should work much better.
Ian
Revised Parser
Hi,
I revised the parser. It can parse much better. I fixed a few problems, cleaned up the code some, and in the process actually was able to make the necessary algorithms shorter.
I ended up not using regular expressions, because they were basically overkill. All that was really necessary was to split the input stream into tags and content, instead of merely tags. I kept most of the rest essentially the same (although I refactored the main algorithm into separate functions for cleanliness).
So, progress was made. However, a new issue has arisen. The parser can't properly parse something like <img src="imgs/srv?=cheese.png"> because of the equals in the quoted string. In practice, the parser will spit out a warning that the HTML is malformed, and just output the tag as-is. So, the parser is not fundamentally broken; it's just that it is properly formed and the parser just doesn't understand it.
When I was creating the parser, it never occurred to me that such situations could arise, so I deliberately attempt to have the parser look for equals. In this case, the parser finds two equals inside what it considers one element: (e.g., for <div class="hello=freedom">, it thinks it has gotten a <div> tag with a single element of the form A=B=C, which is malformed). The quotes don't matter.
To fix this, I may use regular expressions. Though, as trends are going now, I'll probably think I will and then end up not.
Finally, from here-on out, the parser is now being called a converter. The converter comprises the renderer, which converts GWT into HTML, a parser, which breaks that HTML up into lexical chunks, a formatter, which cleans up the HTML and formats everything, and a synthesizer, which tidies up and combines everything.
Ian
I revised the parser. It can parse much better. I fixed a few problems, cleaned up the code some, and in the process actually was able to make the necessary algorithms shorter.
I ended up not using regular expressions, because they were basically overkill. All that was really necessary was to split the input stream into tags and content, instead of merely tags. I kept most of the rest essentially the same (although I refactored the main algorithm into separate functions for cleanliness).
So, progress was made. However, a new issue has arisen. The parser can't properly parse something like <img src="imgs/srv?=cheese.png"> because of the equals in the quoted string. In practice, the parser will spit out a warning that the HTML is malformed, and just output the tag as-is. So, the parser is not fundamentally broken; it's just that it is properly formed and the parser just doesn't understand it.
When I was creating the parser, it never occurred to me that such situations could arise, so I deliberately attempt to have the parser look for equals. In this case, the parser finds two equals inside what it considers one element: (e.g., for <div class="hello=freedom">, it thinks it has gotten a <div> tag with a single element of the form A=B=C, which is malformed). The quotes don't matter.
To fix this, I may use regular expressions. Though, as trends are going now, I'll probably think I will and then end up not.
Finally, from here-on out, the parser is now being called a converter. The converter comprises the renderer, which converts GWT into HTML, a parser, which breaks that HTML up into lexical chunks, a formatter, which cleans up the HTML and formats everything, and a synthesizer, which tidies up and combines everything.
Ian
Subscribe to:
Posts (Atom)