Hi,
Well, actually, the meeting went better than I had previously expected. However, there was a LOT that could have gone better.
Notably, the serializer and deserializer worked pretty well, with issues.
Somehow, the parser didn't work at all. This is problematic, particularly since I didn't change the code since last night, when it did. This means that someone changed the code so that it's passing in garbage to the parser, so it either gives a warning and crashes, or produces erroneous output. I was very disappointed in this. It should have worked perfectly, and I was looking forward to showing it off.
That said, at least internally, we have a proof of concept. As a group, we know all of the technology in play will be fundamentally valid. It's just a matter of getting it to work properly.
Let's do this.
Ian
Wednesday, March 28, 2012
Tuesday, March 27, 2012
Updates
Hi,
So, the serializer and renderer are not completed. Those were my areas of responsibility. While I think that the fact that they are not yet completed is a product of factors that aren't all of my doing, I personally said I would get these done, so it's ultimately my fault. If worst comes to worst, I'll take the blame for it.
Now. Not all is lost, of course. The deserializer is coming along. That will need to be finished. The serializer *should* be done already.
As far as rendering, I am pleased to announce that the key link between GWT's retarded API and my HTML parser has been discovered (albeit through some particularly nasty recursion)! As a result, the renderer can now take a canvas and spit out HTML that corresponds to it. It's not perfect, but I think it will satisfy for tomorrow (next client meeting). As proof, see this HTML file from the generator: http://cs.unm.edu/~imallett/other/460/Untitled-2.html.
Thanks,
Ian
So, the serializer and renderer are not completed. Those were my areas of responsibility. While I think that the fact that they are not yet completed is a product of factors that aren't all of my doing, I personally said I would get these done, so it's ultimately my fault. If worst comes to worst, I'll take the blame for it.
Now. Not all is lost, of course. The deserializer is coming along. That will need to be finished. The serializer *should* be done already.
As far as rendering, I am pleased to announce that the key link between GWT's retarded API and my HTML parser has been discovered (albeit through some particularly nasty recursion)! As a result, the renderer can now take a canvas and spit out HTML that corresponds to it. It's not perfect, but I think it will satisfy for tomorrow (next client meeting). As proof, see this HTML file from the generator: http://cs.unm.edu/~imallett/other/460/Untitled-2.html.
Thanks,
Ian
Monday, March 26, 2012
Serializer and Converter
Hi,
I finished the basics of the serializer. It serializes all children. A couple of fields are still incorrect, but they're not as important. I subsequently released control of the module to the group to improve.
Now, as far as the HTML converter is going, it's not. Don't get me wrong; the parser, formatter, and synthesizer described here: http://iancs460.blogspot.com/2012/03/revised-parser.html are all working perfectly! The renderer just isn't. Right now, all that's happening is the renderer is calling "canvas.getElement().getInnerHTML()". This produces the final output:
<html>
<head>
<title>
[PAGE TITLE HERE]
</title>
</head>
<body>
<div id="isc_7P" eventproxy="isc_WebPage_0" class="normal" style position: relative; z-index: 201872; visibility: hidden; background-image: url(http://1.bp.blogspot.com/-h-aleanaggc/tcbtz8mgaai/aaaaaaaaalq/hofsrycnkle/s320/checker.png); overflow-x: visible; overflow-y: visible; left: 5px; top: 5px; width: 1373px; height: 936px; background-repeat: repeat repeat; onscroll return isc_webpage_0.$nd()>
<div id="isc_7Q" eventproxy="isc_WebPage_0" style="POSITION:relative;VISIBILITY:inherit;Z-INDEX:201872;CURSOR:default;">
<div id="isc_7R" eventproxy="isc_Canvas_17" class="normal" style="POSITION:absolute;LEFT:0px;TOP:0px;WIDTH:1373px;HEIGHT:902px;Z-INDEX:201890;OVERFLOW:visible;" onscroll return isc_canvas_17.$nd()>
<div id="isc_7S" eventproxy="isc_Canvas_17" style="POSITION:relative;VISIBILITY:inherit;Z-INDEX:201890;CURSOR:default;">
</div>
</div>
<div id="isc_7T" eventproxy="isc_Canvas_16" class="normal" style position: absolute; z-index: 201908; overflow-x: visible; overflow-y: visible; left: 0px; top: 0px; width: 1373px; height: 936px; onscroll return isc_canvas_16.$nd()>
<div id="isc_7U" eventproxy="isc_Canvas_16" style="POSITION:relative;VISIBILITY:inherit;Z-INDEX:201908;CURSOR:default;">
<div id="isc_7V" eventproxy="isc_WebPageCanvas_3" class="header" style position: relative; z-index: 201926; overflow-x: hidden; overflow-y: hidden; left: 0px; top: 0px; width: 1373px; height: 135px; onscroll return isc_webpagecanvas_3.$nd()>
<div id="isc_7W" eventproxy="isc_WebPageCanvas_3" style="POSITION:relative;VISIBILITY:inherit;Z-INDEX:201926;CURSOR:default;">
</div>
</div>
<div id="isc_7X" eventproxy="isc_WebPageCanvas_4" class="body" style position: relative; z-index: 201944; overflow-x: hidden; overflow-y: hidden; left: 0px; top: 10px; width: 1373px; height: 722px; onscroll return isc_webpagecanvas_4.$nd()>
<div id="isc_7Y" eventproxy="isc_WebPageCanvas_4" style="POSITION:relative;VISIBILITY:inherit;Z-INDEX:201944;CURSOR:default;">
</div>
</div>
<div id="isc_7Z" eventproxy="isc_WebPageCanvas_5" class="footer" style position: relative; z-index: 201962; overflow-x: hidden; overflow-y: hidden; left: 0px; top: 20px; width: 1373px; height: 45px; onscroll return isc_webpagecanvas_5.$nd()>
<div id="isc_80" eventproxy="isc_WebPageCanvas_5" style="POSITION:relative;VISIBILITY:inherit;Z-INDEX:201962;CURSOR:default;">
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>
This looks pretty good, but it doesn't render anything else. I suspect this may be due to it's not calling the same on its children? In which case, the fix might be simple. If not . . .
See also: http://stackoverflow.com/questions/9855735/get-smartgwt-canvas-as-html
Ian
I finished the basics of the serializer. It serializes all children. A couple of fields are still incorrect, but they're not as important. I subsequently released control of the module to the group to improve.
Now, as far as the HTML converter is going, it's not. Don't get me wrong; the parser, formatter, and synthesizer described here: http://iancs460.blogspot.com/2012/03/revised-parser.html are all working perfectly! The renderer just isn't. Right now, all that's happening is the renderer is calling "canvas.getElement().getInnerHTML()". This produces the final output:
<html>
<head>
<title>
[PAGE TITLE HERE]
</title>
</head>
<body>
<div id="isc_7P" eventproxy="isc_WebPage_0" class="normal" style position: relative; z-index: 201872; visibility: hidden; background-image: url(http://1.bp.blogspot.com/-h-aleanaggc/tcbtz8mgaai/aaaaaaaaalq/hofsrycnkle/s320/checker.png); overflow-x: visible; overflow-y: visible; left: 5px; top: 5px; width: 1373px; height: 936px; background-repeat: repeat repeat; onscroll return isc_webpage_0.$nd()>
<div id="isc_7Q" eventproxy="isc_WebPage_0" style="POSITION:relative;VISIBILITY:inherit;Z-INDEX:201872;CURSOR:default;">
<div id="isc_7R" eventproxy="isc_Canvas_17" class="normal" style="POSITION:absolute;LEFT:0px;TOP:0px;WIDTH:1373px;HEIGHT:902px;Z-INDEX:201890;OVERFLOW:visible;" onscroll return isc_canvas_17.$nd()>
<div id="isc_7S" eventproxy="isc_Canvas_17" style="POSITION:relative;VISIBILITY:inherit;Z-INDEX:201890;CURSOR:default;">
</div>
</div>
<div id="isc_7T" eventproxy="isc_Canvas_16" class="normal" style position: absolute; z-index: 201908; overflow-x: visible; overflow-y: visible; left: 0px; top: 0px; width: 1373px; height: 936px; onscroll return isc_canvas_16.$nd()>
<div id="isc_7U" eventproxy="isc_Canvas_16" style="POSITION:relative;VISIBILITY:inherit;Z-INDEX:201908;CURSOR:default;">
<div id="isc_7V" eventproxy="isc_WebPageCanvas_3" class="header" style position: relative; z-index: 201926; overflow-x: hidden; overflow-y: hidden; left: 0px; top: 0px; width: 1373px; height: 135px; onscroll return isc_webpagecanvas_3.$nd()>
<div id="isc_7W" eventproxy="isc_WebPageCanvas_3" style="POSITION:relative;VISIBILITY:inherit;Z-INDEX:201926;CURSOR:default;">
</div>
</div>
<div id="isc_7X" eventproxy="isc_WebPageCanvas_4" class="body" style position: relative; z-index: 201944; overflow-x: hidden; overflow-y: hidden; left: 0px; top: 10px; width: 1373px; height: 722px; onscroll return isc_webpagecanvas_4.$nd()>
<div id="isc_7Y" eventproxy="isc_WebPageCanvas_4" style="POSITION:relative;VISIBILITY:inherit;Z-INDEX:201944;CURSOR:default;">
</div>
</div>
<div id="isc_7Z" eventproxy="isc_WebPageCanvas_5" class="footer" style position: relative; z-index: 201962; overflow-x: hidden; overflow-y: hidden; left: 0px; top: 20px; width: 1373px; height: 45px; onscroll return isc_webpagecanvas_5.$nd()>
<div id="isc_80" eventproxy="isc_WebPageCanvas_5" style="POSITION:relative;VISIBILITY:inherit;Z-INDEX:201962;CURSOR:default;">
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>
This looks pretty good, but it doesn't render anything else. I suspect this may be due to it's not calling the same on its children? In which case, the fix might be simple. If not . . .
See also: http://stackoverflow.com/questions/9855735/get-smartgwt-canvas-as-html
Ian
Sunday, March 25, 2012
Progress
Hi,
Well, we're making progress. On the serializer, the correspondence between the input and the output is nearly one-to-one in the ways that matter. The problems are the "left" data field is not being set, and the children are not being serialized.
I don't know why the first is the way it is, but the second is due to me simply not implementing it yet because it's hard. However, I'm about to go do that now. Hopefully by tonight or tomorrow evening the "serializer"/"deserializer" will do everything it needs.
I'd also like to note that Nialls has done a nice job improving our home page. There are now joke testimonials and a cleaner layout. I think I might like to work on some web design soon, seeing as the algorithms section of this project is nearly wrapped up.
Ian
Well, we're making progress. On the serializer, the correspondence between the input and the output is nearly one-to-one in the ways that matter. The problems are the "left" data field is not being set, and the children are not being serialized.
I don't know why the first is the way it is, but the second is due to me simply not implementing it yet because it's hard. However, I'm about to go do that now. Hopefully by tonight or tomorrow evening the "serializer"/"deserializer" will do everything it needs.
I'd also like to note that Nialls has done a nice job improving our home page. There are now joke testimonials and a cleaner layout. I think I might like to work on some web design soon, seeing as the algorithms section of this project is nearly wrapped up.
Ian
"Serializer" Hacks
Hi,
Well, GWT is broken and buggy as ever. For instance, the methods .getEdgeSize() and .getExtraSpace() both fail on a newly created canvas with a NullPointerException. Turns out that typing too fast (e.g., my auto log-on script) can crash GWT, as well as certain properties. GWT at least catches errors and gives you nice feedback if you do some things (like call some methods after construction). Though, I'm not sure how you call a message before construction? And yes, GWT still crashes every now and again, which is . . . disconcerting.
The API is inconsistent between the getters and setters, which made matching everything up a huge ordeal. Not to mention that the API has (literally!) about 200 getters and setters. Not all of them are necessary, probably--but how am I to know? The code is horribly messy (notably because Java doesn't have function pointers, nor macros) too.
It does kinda remind me, though, of how slow the whole thing is. GWT takes forever to start, then the website has to load up. Then you have to click login, wait for the window to come up, enter your username/password, click send, wait for that to go through, open a new project, and *then* you can click "save", and actually run a single test! Our code isn't exactly slow per-se; the process to do something is just soooo long!
In any case, after much complaining and whatnot, the serializer/deserializer kinda works. A canvas can be taken in, serialized, and then deserialized. I don't know if the correspondence is one-to-one in the important ways (except for what I know isn't, like recursively evaluating children), but it's a strong start.
Tomorrow (today) I'll finish it up. I DO have other classes too, so I'll need to work on that as well. It's just about 1:45AM, so, knocking off for the night. See you bright and early.
Ian
Well, GWT is broken and buggy as ever. For instance, the methods .getEdgeSize() and .getExtraSpace() both fail on a newly created canvas with a NullPointerException. Turns out that typing too fast (e.g., my auto log-on script) can crash GWT, as well as certain properties. GWT at least catches errors and gives you nice feedback if you do some things (like call some methods after construction). Though, I'm not sure how you call a message before construction? And yes, GWT still crashes every now and again, which is . . . disconcerting.
The API is inconsistent between the getters and setters, which made matching everything up a huge ordeal. Not to mention that the API has (literally!) about 200 getters and setters. Not all of them are necessary, probably--but how am I to know? The code is horribly messy (notably because Java doesn't have function pointers, nor macros) too.
It does kinda remind me, though, of how slow the whole thing is. GWT takes forever to start, then the website has to load up. Then you have to click login, wait for the window to come up, enter your username/password, click send, wait for that to go through, open a new project, and *then* you can click "save", and actually run a single test! Our code isn't exactly slow per-se; the process to do something is just soooo long!
In any case, after much complaining and whatnot, the serializer/deserializer kinda works. A canvas can be taken in, serialized, and then deserialized. I don't know if the correspondence is one-to-one in the important ways (except for what I know isn't, like recursively evaluating children), but it's a strong start.
Tomorrow (today) I'll finish it up. I DO have other classes too, so I'll need to work on that as well. It's just about 1:45AM, so, knocking off for the night. See you bright and early.
Ian
Saturday, March 24, 2012
Serialization Update 2
Hi,
Well, the serialization fiasco has mostly been figured out.
It turns out that yes, my serializer does work. Moreover, it works perfectly, and everything would be good . . . except that the deserializer does not. As it happens, Java's deserialization is, put simply, broken. There's nothing for it. In real life (C++) I'd use a copy constructor, and that would be the end of it, but Java apparently can't handle that.
Unfortunately, the deserializer doesn't, and can't work, unless the GWT designers had thought of it, which they didn't. We'd be stuck, except Joe Collard put into words what I was coming to the realization was necessary (on same post by Nialls mentioned previously, here): "see if you can write a wrapper that contains all of the things you will want saved".
Essentially, the idea boils down to: write your own serializer. It's bloody messy, but Java's (de)serialization is broken and reflection isn't supported by GWT. So, we have to go with it.
No private/protected fields can, of course, be serialized. Hopefully this shouldn't be an issue. It appears to have worked for Joe's group, so hopefully it will work for ours.
Ian
Well, the serialization fiasco has mostly been figured out.
It turns out that yes, my serializer does work. Moreover, it works perfectly, and everything would be good . . . except that the deserializer does not. As it happens, Java's deserialization is, put simply, broken. There's nothing for it. In real life (C++) I'd use a copy constructor, and that would be the end of it, but Java apparently can't handle that.
Unfortunately, the deserializer doesn't, and can't work, unless the GWT designers had thought of it, which they didn't. We'd be stuck, except Joe Collard put into words what I was coming to the realization was necessary (on same post by Nialls mentioned previously, here): "see if you can write a wrapper that contains all of the things you will want saved".
Essentially, the idea boils down to: write your own serializer. It's bloody messy, but Java's (de)serialization is broken and reflection isn't supported by GWT. So, we have to go with it.
No private/protected fields can, of course, be serialized. Hopefully this shouldn't be an issue. It appears to have worked for Joe's group, so hopefully it will work for ours.
Ian
Serialization Update
Hi,
So, when running the serializer, GWT will produce errors:
[ERROR] [web_final] - Errors in 'file:/C:/dev/Java/CS%20460%20Project/Web_Final/src/cs460/webdnd/client/utilities/serialization/serializer.java'
[ERROR] [web_final] - Line 24: No source code is available for type java.io.ByteArrayOutputStream; did you forget to inherit a required module?
[ERROR] [web_final] - Line 25: No source code is available for type java.io.ObjectOutputStream; did you forget to inherit a required module?
[ERROR] [web_final] - Line 39: No source code is available for type java.io.InputStream; did you forget to inherit a required module?
[ERROR] [web_final] - Line 39: No source code is available for type java.io.ByteArrayInputStream; did you forget to inherit a required module?
[ERROR] [web_final] - Line 40: No source code is available for type java.io.ObjectInputStream; did you forget to inherit a required module?
[ERROR] [web_final] - Line 47: No source code is available for type java.lang.ClassNotFoundException; did you forget to inherit a required module?
In response to this, Nialls took over serialization/deserialization using code he found online. This code produces tons of errors, as he writes here: http://niallsc.blogspot.com/2012/03/canvas-serialization-status-failed.html.
However, this misses something--my serializer worked. It produces this output (if you add a print statement):
Now, this looks to me like taking a canvas and serializing it (and I can get a canvas back, deserializing it). I'm not sure what the problem is with Nialls's code, but mine seemed to work.
I feel somewhat accused, but I submit the above as evidence that the code I wrote is working properly. There were limited tests with more complex canvases (nested, with content, etc.) that were conducted some time ago, successfully. It is possible that there is some new problem (like maybe a certain component can't be serialized) which can be solved in the same way as the canvas's serialization was.
Now, I don't know what to say. Maybe there's something subtle going on that I'm missing. Maybe I don't know what I'm talking about. Maybe there's a fundamental flaw in this that really will somehow prevent this from working. I welcome explanations.
Regardless, I think it's a good idea to meet Sunday as Nialls suggested to work this out. For the good of the project, it needs to be solved sooner rather than later--and I had already thought it was solved.
Thanks,
Ian
So, when running the serializer, GWT will produce errors:
[ERROR] [web_final] - Errors in 'file:/C:/dev/Java/CS%20460%20Project/Web_Final/src/cs460/webdnd/client/utilities/serialization/serializer.java'
[ERROR] [web_final] - Line 24: No source code is available for type java.io.ByteArrayOutputStream; did you forget to inherit a required module?
[ERROR] [web_final] - Line 25: No source code is available for type java.io.ObjectOutputStream; did you forget to inherit a required module?
[ERROR] [web_final] - Line 39: No source code is available for type java.io.InputStream; did you forget to inherit a required module?
[ERROR] [web_final] - Line 39: No source code is available for type java.io.ByteArrayInputStream; did you forget to inherit a required module?
[ERROR] [web_final] - Line 40: No source code is available for type java.io.ObjectInputStream; did you forget to inherit a required module?
[ERROR] [web_final] - Line 47: No source code is available for type java.lang.ClassNotFoundException; did you forget to inherit a required module?
In response to this, Nialls took over serialization/deserialization using code he found online. This code produces tons of errors, as he writes here: http://niallsc.blogspot.com/2012/03/canvas-serialization-status-failed.html.
However, this misses something--my serializer worked. It produces this output (if you add a print statement):
Now, this looks to me like taking a canvas and serializing it (and I can get a canvas back, deserializing it). I'm not sure what the problem is with Nialls's code, but mine seemed to work.
I feel somewhat accused, but I submit the above as evidence that the code I wrote is working properly. There were limited tests with more complex canvases (nested, with content, etc.) that were conducted some time ago, successfully. It is possible that there is some new problem (like maybe a certain component can't be serialized) which can be solved in the same way as the canvas's serialization was.
Now, I don't know what to say. Maybe there's something subtle going on that I'm missing. Maybe I don't know what I'm talking about. Maybe there's a fundamental flaw in this that really will somehow prevent this from working. I welcome explanations.
Regardless, I think it's a good idea to meet Sunday as Nialls suggested to work this out. For the good of the project, it needs to be solved sooner rather than later--and I had already thought it was solved.
Thanks,
Ian
Serializer/Deserializer
Hi,
So, since last time, I have added a new class "CanvasSerializable", which is a direct subclass of "Canvas", except it adds support for serialization. If every canvas is serializable, then making the serializer work properly is much easier, and allows for better design.
The changes affected a lot of the project, so I changed every single reference to Canvas to CanvasSerializable, and then converted things back that just flat out couldn't be (e.g., GWT callbacks return Canvas[], so I can't upcast them to CanvasSerializable).
This seems to have had an effect already. The serializer appears to be working properly. There's still some problems on the server-side (I think) so serializing doesn't quite work. Because there's no serialized data, naturally, the deserializer doesn't work either.
Hopefully, over the next few posts, the serializer module may be completed!
Ian
So, since last time, I have added a new class "CanvasSerializable", which is a direct subclass of "Canvas", except it adds support for serialization. If every canvas is serializable, then making the serializer work properly is much easier, and allows for better design.
The changes affected a lot of the project, so I changed every single reference to Canvas to CanvasSerializable, and then converted things back that just flat out couldn't be (e.g., GWT callbacks return Canvas[], so I can't upcast them to CanvasSerializable).
This seems to have had an effect already. The serializer appears to be working properly. There's still some problems on the server-side (I think) so serializing doesn't quite work. Because there's no serialized data, naturally, the deserializer doesn't work either.
Hopefully, over the next few posts, the serializer module may be completed!
Ian
RSA Cryptography
Hi,
I wrote a little cryptography module. It implemented a (theoretically) fully secure RSA cryptosystem.
This turned out to be harder than anticipated. In addition to jogging my memory of RSA, implementing it in Java is a nightmare (exercise of actual real programming).
It turns out that there are two different APIs for this. The one most directly implemented by Java has a bug that prevents it from working except when writing to a file. The second is also too buggy to use.
So, I decided to write my own implementation. It turns out that there are a number of (buggy) Java implementations. These fail in a number of cases, (usually when a message is too long). Obviously, real RSA doesn't crash. I ended up basing my solution on an existing RSA implementation and then rewriting half of it to actually work.
In the process, I also tried compressing the RSA output stream. It turns out that the Java "Inflate" class has a bug that causes it to hang on the general kind of data that can be produced by compressing an RSA stream.
I can't really do anything about that, because if I were to try to fix all the bugs in the JRE, I should at least get paid--and anyway, at this time, there are more pressing matters, such as serialization, updates concerning which should appear on the next post.
Ian
I wrote a little cryptography module. It implemented a (theoretically) fully secure RSA cryptosystem.
This turned out to be harder than anticipated. In addition to jogging my memory of RSA, implementing it in Java is a nightmare (exercise of actual real programming).
It turns out that there are two different APIs for this. The one most directly implemented by Java has a bug that prevents it from working except when writing to a file. The second is also too buggy to use.
So, I decided to write my own implementation. It turns out that there are a number of (buggy) Java implementations. These fail in a number of cases, (usually when a message is too long). Obviously, real RSA doesn't crash. I ended up basing my solution on an existing RSA implementation and then rewriting half of it to actually work.
In the process, I also tried compressing the RSA output stream. It turns out that the Java "Inflate" class has a bug that causes it to hang on the general kind of data that can be produced by compressing an RSA stream.
I can't really do anything about that, because if I were to try to fix all the bugs in the JRE, I should at least get paid--and anyway, at this time, there are more pressing matters, such as serialization, updates concerning which should appear on the next post.
Ian
Thursday, March 22, 2012
Compression
Hi,
Well, I took a little break from doing parsing and serialization whilst I wait for others to get set up to use the serializer/deserializer.
Some, especially Alex, expressed concern that serialized data or the generated HTML would be very large and difficult to send. So, I wrote a compression wrapper around Java's Inflate/Deflate APIs. It turns out that Java's String.getBytes(...) method is broken, which made this difficult. After some false starts, I finally just gave up trying to work around their bugs and just wrote out what the methods were supposed to do myself.
The algorithm is the standard ZLIB compression algorithm, which should be able to achieve very good compression rates.
I may try writing an encryption algorithm as well.
Ian
Well, I took a little break from doing parsing and serialization whilst I wait for others to get set up to use the serializer/deserializer.
Some, especially Alex, expressed concern that serialized data or the generated HTML would be very large and difficult to send. So, I wrote a compression wrapper around Java's Inflate/Deflate APIs. It turns out that Java's String.getBytes(...) method is broken, which made this difficult. After some false starts, I finally just gave up trying to work around their bugs and just wrote out what the methods were supposed to do myself.
The algorithm is the standard ZLIB compression algorithm, which should be able to achieve very good compression rates.
I may try writing an encryption algorithm as well.
Ian
Regular Expression for Tag Processing
Hi,
Last time I mentioned that regular expressions might be useful for breaking a tag into elements. My regular expression is:
([^\s\"=]+\s*=\s*([^\s\"=]+|\"[^\s^\"]+\"))|([^\s\"=]+)
[Edit: Updating for single quotes and other improvements: ([^\s\"\'=]+\s*=\s*([^\s\"\'=]+|[\"\'][^\s\"\']+[\"\']))|([^\s\"\'=]+)]
Notice that it is awesome. I tested it on the input data:
div class=hijkjh width=100 height = 6 src= "hello.png" why=bec6=ause src="hello.png" src="hello.png?=yo?" src="hello.png" src="hello.png" src="hello.png" sup
. . . which is deliberately malformed in some cases. You can test the regex at http://regexpal.com/. Notice that the regex matches valid elements. All I have to do to determine if a tag is well formed is check whether anything other than whitespace is unmatched.
I have merged these changes into the converter itself, so the new tag system should work much better.
Ian
Last time I mentioned that regular expressions might be useful for breaking a tag into elements. My regular expression is:
([^\s\"=]+\s*=\s*([^\s\"=]+|\"[^\s^\"]+\"))|([^\s\"=]+)
[Edit: Updating for single quotes and other improvements: ([^\s\"\'=]+\s*=\s*([^\s\"\'=]+|[\"\'][^\s\"\']+[\"\']))|([^\s\"\'=]+)]
Notice that it is awesome. I tested it on the input data:
div class=hijkjh width=100 height = 6 src= "hello.png" why=bec6=ause src="hello.png" src="hello.png?=yo?" src="hello.png" src="hello.png" src="hello.png" sup
. . . which is deliberately malformed in some cases. You can test the regex at http://regexpal.com/. Notice that the regex matches valid elements. All I have to do to determine if a tag is well formed is check whether anything other than whitespace is unmatched.
I have merged these changes into the converter itself, so the new tag system should work much better.
Ian
Revised Parser
Hi,
I revised the parser. It can parse much better. I fixed a few problems, cleaned up the code some, and in the process actually was able to make the necessary algorithms shorter.
I ended up not using regular expressions, because they were basically overkill. All that was really necessary was to split the input stream into tags and content, instead of merely tags. I kept most of the rest essentially the same (although I refactored the main algorithm into separate functions for cleanliness).
So, progress was made. However, a new issue has arisen. The parser can't properly parse something like <img src="imgs/srv?=cheese.png"> because of the equals in the quoted string. In practice, the parser will spit out a warning that the HTML is malformed, and just output the tag as-is. So, the parser is not fundamentally broken; it's just that it is properly formed and the parser just doesn't understand it.
When I was creating the parser, it never occurred to me that such situations could arise, so I deliberately attempt to have the parser look for equals. In this case, the parser finds two equals inside what it considers one element: (e.g., for <div class="hello=freedom">, it thinks it has gotten a <div> tag with a single element of the form A=B=C, which is malformed). The quotes don't matter.
To fix this, I may use regular expressions. Though, as trends are going now, I'll probably think I will and then end up not.
Finally, from here-on out, the parser is now being called a converter. The converter comprises the renderer, which converts GWT into HTML, a parser, which breaks that HTML up into lexical chunks, a formatter, which cleans up the HTML and formats everything, and a synthesizer, which tidies up and combines everything.
Ian
I revised the parser. It can parse much better. I fixed a few problems, cleaned up the code some, and in the process actually was able to make the necessary algorithms shorter.
I ended up not using regular expressions, because they were basically overkill. All that was really necessary was to split the input stream into tags and content, instead of merely tags. I kept most of the rest essentially the same (although I refactored the main algorithm into separate functions for cleanliness).
So, progress was made. However, a new issue has arisen. The parser can't properly parse something like <img src="imgs/srv?=cheese.png"> because of the equals in the quoted string. In practice, the parser will spit out a warning that the HTML is malformed, and just output the tag as-is. So, the parser is not fundamentally broken; it's just that it is properly formed and the parser just doesn't understand it.
When I was creating the parser, it never occurred to me that such situations could arise, so I deliberately attempt to have the parser look for equals. In this case, the parser finds two equals inside what it considers one element: (e.g., for <div class="hello=freedom">, it thinks it has gotten a <div> tag with a single element of the form A=B=C, which is malformed). The quotes don't matter.
To fix this, I may use regular expressions. Though, as trends are going now, I'll probably think I will and then end up not.
Finally, from here-on out, the parser is now being called a converter. The converter comprises the renderer, which converts GWT into HTML, a parser, which breaks that HTML up into lexical chunks, a formatter, which cleans up the HTML and formats everything, and a synthesizer, which tidies up and combines everything.
Ian
Wednesday, March 21, 2012
Fourth Client Meeting, Updates, and Serialization/Deserialization
Hi,
As catchup from last time:
-With the parser complete, serialization and deserialization of raw Canvas-es is next. This is necessary for saving and loading.
-The parser may not actually be complete; more formatting may be required and there are other issues.
-Had a client meeting. It went poorly.
The serialization and deserialization is pretty straightforward so far. It uses a slightly modified version of the standard Java serialization routines. It's not fully functional (it doesn't reconstruct everything properly yet), but it's in-place and a good start. Nialls will pass it the actual canvases, and then the real fun will begin: getting it working with "real" data.
The parser is annoying me. It turns out I actually wrote pure XML parser instead of an HTML parser. So I'll need to make some *significant* changes. I'm thinking of resurrecting regular expressions. I'm dedicating this afternoon/evening to doing this.
We also had our fourth client meeting. Honestly, I think it went the worst out of all client meetings thus far.
I think a lot of this had to do with the group not being very productive over Spring Break. I freely admit that I definitely slacked, and I feel bad about it, particularly since my modules were least integrated out of everyone's.
I feel like I didn't get credit for making the serializer/deserializer from Ackley. Yes, it wasn't fully integrated, but *it worked*. I also feel like Spring Break ought to be a break. I know that the reality of it is that work happens over Spring Break and that I'm expected to do things. But I still think it's problematic to call it a "break" if it really isn't to our society. Same applies to Christmas Break, for year-long classes. In the extreme, Summer, for some summer assignments.
So, for next time, the serializer/deserializer, and a revised parser.
Ian
As catchup from last time:
-With the parser complete, serialization and deserialization of raw Canvas-es is next. This is necessary for saving and loading.
-The parser may not actually be complete; more formatting may be required and there are other issues.
-Had a client meeting. It went poorly.
The serialization and deserialization is pretty straightforward so far. It uses a slightly modified version of the standard Java serialization routines. It's not fully functional (it doesn't reconstruct everything properly yet), but it's in-place and a good start. Nialls will pass it the actual canvases, and then the real fun will begin: getting it working with "real" data.
The parser is annoying me. It turns out I actually wrote pure XML parser instead of an HTML parser. So I'll need to make some *significant* changes. I'm thinking of resurrecting regular expressions. I'm dedicating this afternoon/evening to doing this.
We also had our fourth client meeting. Honestly, I think it went the worst out of all client meetings thus far.
I think a lot of this had to do with the group not being very productive over Spring Break. I freely admit that I definitely slacked, and I feel bad about it, particularly since my modules were least integrated out of everyone's.
I feel like I didn't get credit for making the serializer/deserializer from Ackley. Yes, it wasn't fully integrated, but *it worked*. I also feel like Spring Break ought to be a break. I know that the reality of it is that work happens over Spring Break and that I'm expected to do things. But I still think it's problematic to call it a "break" if it really isn't to our society. Same applies to Christmas Break, for year-long classes. In the extreme, Summer, for some summer assignments.
So, for next time, the serializer/deserializer, and a revised parser.
Ian
Friday, March 16, 2012
Parser Complete
Hey,
I'm pleased to announce the alpha version of the parser! The parser can take any GWT canvas and convert it into HTML, the formatting of which can be adjusted to as desired. Additionally, it can take HTML and make inferences to make it valid HTML (this is important because GWT outputs non-standards-compliant HTML).
Currently, the parser can be customized to output/not output newlines and pipe characters (for viewing alignment) and whether tags are capitalized/not. I didn't really know what other options to add, but it should be trivial to add them.
The thing is pretty robust, in that it will take any HTML you could possibly throw at it. It will even detect some problems (e.g., "<div class=a=b>") and let you know about them. I can't yet give it my (imaginary) gold seal of approval, but it's definitely solid enough for alpha.
Anyway, this concludes my original goal of getting the parser working. The parser is fully integrated into Web_Final, and is ready for use. All changes have been committed.
TODO:
1. parser does not attempt to find redundant or useless code (e.g., "<div></div>" . . . I (think?) is useless).
2. Do we, or do we not want HTML to GWT conversion? Depending on the answer to 1., this may be as simple as removing the useless code and then feeding the generated HTML back into a canvas object. But I honestly haven't looked into it.
Thanks,
Ian
I'm pleased to announce the alpha version of the parser! The parser can take any GWT canvas and convert it into HTML, the formatting of which can be adjusted to as desired. Additionally, it can take HTML and make inferences to make it valid HTML (this is important because GWT outputs non-standards-compliant HTML).
Currently, the parser can be customized to output/not output newlines and pipe characters (for viewing alignment) and whether tags are capitalized/not. I didn't really know what other options to add, but it should be trivial to add them.
The thing is pretty robust, in that it will take any HTML you could possibly throw at it. It will even detect some problems (e.g., "<div class=a=b>") and let you know about them. I can't yet give it my (imaginary) gold seal of approval, but it's definitely solid enough for alpha.
Anyway, this concludes my original goal of getting the parser working. The parser is fully integrated into Web_Final, and is ready for use. All changes have been committed.
TODO:
1. parser does not attempt to find redundant or useless code (e.g., "<div></div>" . . . I (think?) is useless).
2. Do we, or do we not want HTML to GWT conversion? Depending on the answer to 1., this may be as simple as removing the useless code and then feeding the generated HTML back into a canvas object. But I honestly haven't looked into it.
Thanks,
Ian
Sunday, March 11, 2012
Project Mid-Semester Self-Evaluation
Hi,
I wrote my self-evaluation.
I honestly only see a marginal purpose in it: I understand the rationale for a self-evaluation, but I somewhat object to the notion of it being the programmer doing it (I was under the impression that evaluation was a function of a manager. Am I doing a manager's work?). Putting it in writing is also annoying, although I do recognize the force of having something in that medium to refer back to later as a concrete reference.
In any case I gave it a go, and, as near as I can tell, executed the assignment well. If the client has any problems with what I did, I'd appreciate knowing about it in advance so that I may fix them (although I understand if that's unreasonable).
Thanks,
Ian
I wrote my self-evaluation.
I honestly only see a marginal purpose in it: I understand the rationale for a self-evaluation, but I somewhat object to the notion of it being the programmer doing it (I was under the impression that evaluation was a function of a manager. Am I doing a manager's work?). Putting it in writing is also annoying, although I do recognize the force of having something in that medium to refer back to later as a concrete reference.
In any case I gave it a go, and, as near as I can tell, executed the assignment well. If the client has any problems with what I did, I'd appreciate knowing about it in advance so that I may fix them (although I understand if that's unreasonable).
Thanks,
Ian
Scratch Regular Expressions
Hi,
We had another group meeting this Sunday, and we got everyone on the same page for the project. In particular, we fixed a lot of issues pertaining to project setup that were preventing most people from getting anything working. Apart from a delay due to DST, the meeting went well.
I have been working on the regular expression matcher--or, more precisely, I was. I decided that regular expressions are messy and overkill for the application (purpose being merely to reformat the GWT output).
Instead, the new plan is to use a simple algorithm (string search) to find and format tags. This is simpler and cleaner. In addition, it has the advantage of being strongly linear time (which is only the best case for regular expressions). I could throw in some flak about Java not supporting pointer arithmetic, but at this point, I'm just glad that I'm writing algorithms :-)
Additionally, there will need to be a formatter API for the parser. This is implemented as a simple class that can (optionally) be passed to .generate(...). The class is analogous to a "struct", in that the point is to set various data fields, with others being set to defaults, and then pass it to a function.
I'm not entirely satisfied with this implementation, because the "struct" needs a public method ".apply()", which applies the struct's data values to an input string. It's paradoxical because this method must be public so that the parser can call it, but only the parser needs it. So, I'm violating encapsulation. Yet, I can't just move the .apply() method to be a private method of the parser because that violates encapsulation (the method really should be a method of the "struct"). In C++, I'd make parser a friend class, but in Java . . .
Final notes:
-I'm creating a new project wherein I can test my parser separate from the other modules of the program. Normally, I wouldn't care, but GWT is too unreliable and slow to fit my mandate of obscene amounts of runtime testing. It's a code quality thing.
-I'm working on the project mid semester self-evaluation (http://cs.unm.edu/~Ackley/460/selfEvaluation.html). I'm . . . not psyched.
Ian
We had another group meeting this Sunday, and we got everyone on the same page for the project. In particular, we fixed a lot of issues pertaining to project setup that were preventing most people from getting anything working. Apart from a delay due to DST, the meeting went well.
I have been working on the regular expression matcher--or, more precisely, I was. I decided that regular expressions are messy and overkill for the application (purpose being merely to reformat the GWT output).
Instead, the new plan is to use a simple algorithm (string search) to find and format tags. This is simpler and cleaner. In addition, it has the advantage of being strongly linear time (which is only the best case for regular expressions). I could throw in some flak about Java not supporting pointer arithmetic, but at this point, I'm just glad that I'm writing algorithms :-)
Additionally, there will need to be a formatter API for the parser. This is implemented as a simple class that can (optionally) be passed to .generate(...). The class is analogous to a "struct", in that the point is to set various data fields, with others being set to defaults, and then pass it to a function.
I'm not entirely satisfied with this implementation, because the "struct" needs a public method ".apply()", which applies the struct's data values to an input string. It's paradoxical because this method must be public so that the parser can call it, but only the parser needs it. So, I'm violating encapsulation. Yet, I can't just move the .apply() method to be a private method of the parser because that violates encapsulation (the method really should be a method of the "struct"). In C++, I'd make parser a friend class, but in Java . . .
Final notes:
-I'm creating a new project wherein I can test my parser separate from the other modules of the program. Normally, I wouldn't care, but GWT is too unreliable and slow to fit my mandate of obscene amounts of runtime testing. It's a code quality thing.
-I'm working on the project mid semester self-evaluation (http://cs.unm.edu/~Ackley/460/selfEvaluation.html). I'm . . . not psyched.
Ian
Wednesday, March 7, 2012
Third Client Meeting
Hi,
Well, the third client meeting has come and gone. Honestly, once again, I think it went pretty well. I think we've got the presentation format down fairly solidly: a brief summary of last week's accomplishments, and focus for future work in the coming time ahead, followed by a demo with specific questions deferred to those who know best.
As for me, the parser is functionally complete. It parses all GWT canvases into HTML flawlessly, as far as I can tell. The only drawback is that the generated HTML is horrible. I decided to forgo recursive descent parsing in favor of letting GWT do the heavy lifting. Ultimately, this should (maybe? I don't know, given GWT's robustness record) be more stable and resilient to improvements group members add that take advantage of more features of GWT. However, in classic GWT bloatedness, the HTML has only limited customization options--and nothing is able to resolve redundant and useless code (e.g. "<div></div>").
My solution? Write a regular expression meta parser/converter for the HTML parser's output! This should at least be interesting, and will have the benefit that we will have beautiful control over the generated HTML. In contrast to GWT, I will ensure that there is a clean API for this.
By next time, I hope to have at least a start on doing this.
Other notes on the meeting:
-User stories appear to be wanted.
-I speculated that perhaps we could serialize and deserialize Canvases to save them and restore them. This may not be possible, although I don't know why you can't just subclass Canvas and add "implements Serializable". Maybe some internal thing?
Ian
Well, the third client meeting has come and gone. Honestly, once again, I think it went pretty well. I think we've got the presentation format down fairly solidly: a brief summary of last week's accomplishments, and focus for future work in the coming time ahead, followed by a demo with specific questions deferred to those who know best.
As for me, the parser is functionally complete. It parses all GWT canvases into HTML flawlessly, as far as I can tell. The only drawback is that the generated HTML is horrible. I decided to forgo recursive descent parsing in favor of letting GWT do the heavy lifting. Ultimately, this should (maybe? I don't know, given GWT's robustness record) be more stable and resilient to improvements group members add that take advantage of more features of GWT. However, in classic GWT bloatedness, the HTML has only limited customization options--and nothing is able to resolve redundant and useless code (e.g. "<div></div>").
My solution? Write a regular expression meta parser/converter for the HTML parser's output! This should at least be interesting, and will have the benefit that we will have beautiful control over the generated HTML. In contrast to GWT, I will ensure that there is a clean API for this.
By next time, I hope to have at least a start on doing this.
Other notes on the meeting:
-User stories appear to be wanted.
-I speculated that perhaps we could serialize and deserialize Canvases to save them and restore them. This may not be possible, although I don't know why you can't just subclass Canvas and add "implements Serializable". Maybe some internal thing?
Ian
Monday, March 5, 2012
Mid-Parser and the Importance of Software Engineering
Hi,
The parser proceeds well. I'm currently refactoring the code to be more modular and robust, as well as cleaner. I have my own subpackage in the solution, and I'm going to fill it with as many files as I need to make the architecture pretty.
In terms of functionality, I wasn't able to accomplish anything over the weekend. If you must know, I instead wrote a GPU raytracer for rendering 3D fractal geometry, complete with distance estimators, Phong shading, and frustum clipping. Woe that this isn't a graphics project.
However, I had planned on doing that; the parser will still be advanced in time for the next client meeting on Wednesday. Positioning will be in place. In fact, I anticipate, the way things are going, for the parser to be just about completed, with minor functionality to be added as new features are made accessible by the GUI.
Meanwhile, GWT is being snarky again. Actually, strictly speaking, not "snarky". That word is closer to "sassy" or "cheeky", and at least implies some level of sophisticated defiance. No, GWT is being outright stupid. My primary complaint right now (because I'm not so directly working with it) is that it's not robust. It takes like ten seconds to start up, and when it does, half the time there's some internal error. I've found that restarting Eclipse usually helps. But, I am frustrated that something that's supposedly super-awesome can't even get out the starting gate when you try to use it. And, I'm immediately suspicious of something that runs differently each time. It brings to mind thread contention issues and broken dependencies. And I'd be worried about that except GWT is necessarily single-threaded. . . . which still leaves the question: why can't GWT work perfectly every time? Is that too much to ask?
It brings up an interesting point about software engineering. I consider myself much more a computer scientist and programmer, instead of a software engineer. The difference is subtle, but I talk about it a bit here.
Nevertheless, I continually get the feeling that for some reason my code is inherently better than code produced in the "real world". It is endlessly annoying to have to work around someone else's bug in a library, or to write wrappers around broken APIs so that they're usable. Yet somehow, for some reason, my code always manages to be fast, flexible, and, above all, robust. I frequently (weekly or so) receive emails from satisfied users of my libraries praising their "beautiful API[s]" and robustness. In one coder's words:
The point is not to demonstrate that I'm a better person or something--the issue is one of design. Better-designed code tends to work better. Do I just pay more attention to that than other people? These questions go through my head rather more often than I'd normally care to admit, and they are of particular relevance as I rework the parser. Group members: you may expect robust, powerful, and well-designed code.
But it all brings me back to GWT. I'm not going to speculate on why it can't reliably start, but it suffices to say that it makes debugging extremely annoying, so long as "extremely" is VERY strongly emphasized. I think I may start a separate project, and run the parser on GWT objects that have been saved out and serialized. I wager it would be faster--though, the way things are going, I wouldn't be (too) surprised if GWT has some meaningless constraint that prevents you from doing this.
So, project-wise and in sum: the parser will be next to done by Wednesday. Furthermore, it will live up to my standards of design. GWT is annoying. We'll see how it plays out.
Ian
The parser proceeds well. I'm currently refactoring the code to be more modular and robust, as well as cleaner. I have my own subpackage in the solution, and I'm going to fill it with as many files as I need to make the architecture pretty.
In terms of functionality, I wasn't able to accomplish anything over the weekend. If you must know, I instead wrote a GPU raytracer for rendering 3D fractal geometry, complete with distance estimators, Phong shading, and frustum clipping. Woe that this isn't a graphics project.
However, I had planned on doing that; the parser will still be advanced in time for the next client meeting on Wednesday. Positioning will be in place. In fact, I anticipate, the way things are going, for the parser to be just about completed, with minor functionality to be added as new features are made accessible by the GUI.
Meanwhile, GWT is being snarky again. Actually, strictly speaking, not "snarky". That word is closer to "sassy" or "cheeky", and at least implies some level of sophisticated defiance. No, GWT is being outright stupid. My primary complaint right now (because I'm not so directly working with it) is that it's not robust. It takes like ten seconds to start up, and when it does, half the time there's some internal error. I've found that restarting Eclipse usually helps. But, I am frustrated that something that's supposedly super-awesome can't even get out the starting gate when you try to use it. And, I'm immediately suspicious of something that runs differently each time. It brings to mind thread contention issues and broken dependencies. And I'd be worried about that except GWT is necessarily single-threaded. . . . which still leaves the question: why can't GWT work perfectly every time? Is that too much to ask?
It brings up an interesting point about software engineering. I consider myself much more a computer scientist and programmer, instead of a software engineer. The difference is subtle, but I talk about it a bit here.
Nevertheless, I continually get the feeling that for some reason my code is inherently better than code produced in the "real world". It is endlessly annoying to have to work around someone else's bug in a library, or to write wrappers around broken APIs so that they're usable. Yet somehow, for some reason, my code always manages to be fast, flexible, and, above all, robust. I frequently (weekly or so) receive emails from satisfied users of my libraries praising their "beautiful API[s]" and robustness. In one coder's words:
". . . the API is cleasn[sic], crisp[,] and compact--yet still powerful. Your code is astoundingly bug-free and my only problem[s] come from me being completely retarded. Using your code is like breathing freely."I myself hardly ever have the opportunity to send such mails. I typically send bug reports.
The point is not to demonstrate that I'm a better person or something--the issue is one of design. Better-designed code tends to work better. Do I just pay more attention to that than other people? These questions go through my head rather more often than I'd normally care to admit, and they are of particular relevance as I rework the parser. Group members: you may expect robust, powerful, and well-designed code.
But it all brings me back to GWT. I'm not going to speculate on why it can't reliably start, but it suffices to say that it makes debugging extremely annoying, so long as "extremely" is VERY strongly emphasized. I think I may start a separate project, and run the parser on GWT objects that have been saved out and serialized. I wager it would be faster--though, the way things are going, I wouldn't be (too) surprised if GWT has some meaningless constraint that prevents you from doing this.
I also found a curious little method Canvas.getInnerHTML(), which appears to return a whole lot of nonsense. It looks like some of it might be useful, but it's clearly no substitute for real parsing. Which reminds me, back to that design thing: one good design principle is to avoid name collisions. I solve this with namespaces, and by naming my functions and classes intelligently. At the very least, prefixing with some easy to recognize identifier. In Java, the evidently accepted solution is to use different capitalization, because apparently that's so much cleaner. On right, what Eclipse has to say about the unrecognized symbol "HTML". Terse and uniform APIs? Mmmmm no. |  |
So, project-wise and in sum: the parser will be next to done by Wednesday. Furthermore, it will live up to my standards of design. GWT is annoying. We'll see how it plays out.
Ian
Friday, March 2, 2012
Second Client Meeting
Hi,
Well, in contrast to last time, our presentation this time went very well. We were much better organized and we covered the material linearly and logically.
However, I also think that there were some issues. The most noticeable was our use of time. We were cut somewhat short on time. I was under the impression that we had 15 minutes. Apparently, we have 10.
In any case, Ackley apparently doesn't care about PowerPoints. That's fine; we can work with that.
Solution: future presentations will be all demo, and as we show various sections, group members will discuss their contributions. This should be more entertaining and make better use of time.
Ian
Well, in contrast to last time, our presentation this time went very well. We were much better organized and we covered the material linearly and logically.
However, I also think that there were some issues. The most noticeable was our use of time. We were cut somewhat short on time. I was under the impression that we had 15 minutes. Apparently, we have 10.
In any case, Ackley apparently doesn't care about PowerPoints. That's fine; we can work with that.
Solution: future presentations will be all demo, and as we show various sections, group members will discuss their contributions. This should be more entertaining and make better use of time.
Ian
Subscribe to:
Posts (Atom)