I created V2.01, but unfortunately I can't get to it right now. As soon as I have retrieved it and have run it against a batch of test files without any problems, I'll release it.
fred01 said
Would it be also possible to add a logging option so that the output is redirected to a file?
Yes, it's possible, but why not add something like ">>c:\output.log" to the command line?
fred01 said
I also run into an
Edited at
fred01 at
Re: DeflOpt V2.01
Zardalu said
fred01 said
Would it be also possible to add a logging option so that the output is redirected to a file?
Yes, it's possible, but why not add something like ">>c:\output.log" to the command line?
I tried to put this it in zipmax config sometime ago, but it would not work, so used a batch file as a temporary solution for now, but was hoping to have a more sophisticated way :-)
Thank you for the new version, will try it as soon as I'm back (writing from a cybercafe now).
Zardalu at
DeflOpt thread
I released V2.01. I also added a "silent" option.
http://www.walbeehm.com/download/
Ben Jos.
Zardalu at
V2.02 released
V2.02 released. Significantly faster than V2.01. Same performance in optimisation.
Ben Jos.
Thundik81 at
DeflOpt thread
Thanks a lot, Ben
Quick comparison on png files (screenshots compressed with pngoutwin)
CPU : Bi-Xeon @ 3000
Timer for calculating execution time (in addition of native cycles counting)
3 successive runs for each version
DeflOpt V2.00
Number of files processed : 59
Number of files rewritten : 59
Total number of bytes saved: 7,081
39,272,848,147 cycles.
Kernel Time = 0.109 = 00:00:00.109 = 0%
User Time = 12.640 = 00:00:12.640 = 95%
Process Time = 12.750 = 00:00:12.750 = 96%
Global Time = 13.172 = 00:00:13.172 = 100%
38,863,666,680 cycles.
Kernel Time = 0.109 = 00:00:00.109 = 0%
User Time = 12.609 = 00:00:12.609 = 96%
Process Time = 12.718 = 00:00:12.718 = 97%
Global Time = 13.015 = 00:00:13.015 = 100%
38,891,161,350 cycles.
Kernel Time = 0.156 = 00:00:00.156 = 1%
User Time = 12.656 = 00:00:12.656 = 97%
Process Time = 12.812 = 00:00:12.812 = 98%
Global Time = 13.015 = 00:00:13.015 = 100%
DeflOpt V2.02
Number of files processed : 59
Number of files rewritten : 59
Total number of bytes saved: 7,081
14,848,200,277 cycles.
Kernel Time = 0.140 = 00:00:00.140 = 2%
User Time = 4.671 = 00:00:04.671 = 94%
Process Time = 4.812 = 00:00:04.812 = 96%
Global Time = 4.969 = 00:00:04.969 = 100%
15,374,020,950 cycles.
Kernel Time = 0.109 = 00:00:00.109 = 2%
User Time = 4.734 = 00:00:04.734 = 91%
Process Time = 4.843 = 00:00:04.843 = 93%
Global Time = 5.156 = 00:00:05.156 = 100%
15,477,956,955 cycles.
Kernel Time = 0.156 = 00:00:00.156 = 3%
User Time = 4.703 = 00:00:04.703 = 90%
Process Time = 4.859 = 00:00:04.859 = 93%
Global Time = 5.203 = 00:00:05.203 = 100%
ace_dent at
Ben,
thanks for this useful tool to knock a few more bytes from my pngs. I'm going to be adding it to my PNG Optimization Guide soon and wondered if / what / when you had any updates planned. I remember a while ago you said there might be a few further enhancements down the road...
Thanks again,
Andrew
Zardalu at
ace_dent said
wondered if / what / when you had any updates planned. I remember a while ago you said there might be a few further enhancements down the road...
I have some ideas for looking at more trees, but I haven't yet come up with a way to efficiently implement it. I might try to do it in a slow way first to see if it is worth it. In addition, it turns out that a tree that looks better sometimes produces worse results. I'm thinking of looking at that too.
Either way, I don't know whether it will be tomorrow or next week or next month...
Ben Jos.
ace_dent at
Output from silent mode
Ben,
thanks for the feedback. I just incorporated DeflOpt into my script using the new silent switch. Although this makes for a less verbose output, the program 'header' (*** DeflOpt ... *** , etc.) is still produced. Ideally no output would be generated in silent mode. If you would consider this change / enhancement, that'd be cool.
Many thanks,
Andrew
secretpng at
Looking for latest version of BJWFlate
Zardalu, any chance you could post it at your download site: http://www.walbeehm.com/download/ ?
I have recently tested DEFLOPT for the first time. It works great and was looking to test BJWFlate also.
Thank You,
Zardalu at
DeflOpt thread
Andrew: I'm considering it... and, better yet, I'll probably do it in the next version, but I've been a bit busy lately... real life and such... there are some things I like to do more in my free time than programming, although there are not many. Don't know what kind of smiley to end this with...
secretpng: Try Roman Scherzer's site: http://www.clrmame.com/bjwflate.htm... it's very close to the latest... I don't think I've ever released 1.54x (an EVEN slower version that might give slightly better results) and 1.55 (which sometimes did better than 1.54 and sometimes worse), but, well, BJWFlate is OLD. It was decent enough when I released it, but the latest versions of Ken's KZIP beat it 99% of the time, while also being a lot faster. There MIGHT also be a very rare bug in BJWFlate... I'm not sure... someone reported some file that BJWFlate allegedly created a corrupt archive for, but I never was interested enough anymore to even check. Sorry. I don't think it's worth it for me to develop BJWFlate any longer. Ken's KZIP is so much better and faster in almost every case. Maybe at some point, I'll get interested again, but then I would write something completely new instead of making BJWFlate better at the cost of even more speed.
Ben Jos.
fred01 at
DeflOpt(ing) OpenOffice.org files
I tried DeflOpt (v2.02) on an OpenOffice.org files, but it failed with:
ERROR: CRC in local fileheader differs from the one in the central directory.
The only other program to fail on testing such files is AdvanceCOMPzip (v1.15), with:
Invalid crc on data descriptor
Other zip utilities, including 7zip and PKZIP (DOS and windows), test and extract these files correctly. But, peculiarly, PKZIPFIX (or -fix) would
Zardalu at
DeflOpt thread
The ZIP format has two directories. Local and glocal. Both should contain the same information. But I found that MP3s and lots of other files in these days of file sharing don't contain the last few bytes. PKZIPFIX only tries to reconstruct the global directory from the local ones. Please remember that PKZIPFIX is OLD.
Having said that, tell me where I can find the files you stated failed. Or where I can download the failures. I think DeflOpt isn't the problem, but, rather, it's a case of bad downloads. Still... if I can fix "easy" errors, I might consider adding them... .
Ben Jos.
Edited at
fred01 at
Thank you for the reply.
Zardalu said
The ZIP format has two directories. Local and glocal. Both should contain the same information. But I found that MP3s and lots of other files in these days of file sharing don't containt the last few bytes.
Yes, I suspected that it had something to do with local/global header discrepancy, but I'm not familiar enough with their binary content to see where. Is there an "intelligent" hex viewer that would show all the different parts of a ZIP header, like date, size, CRC, etc.?
I was not aware that data in MP3s were also compressed with ZIP.
Zardalu said
PKFIX only tries to reconstruct the global directory from the local ones. Please remember that PKZIPFIX is OLD.
I also tried pkzipc -fix and it produced binary identical file with pkzipfix's. And it seems that the global directory is good, but
Awesoken at
I was not aware that data in MP3s were also compressed with ZIP.
There is no ZIP in MP3. You read Zardalu's words wrong. He was referring to a bug with file sharing services not always sending the whole file. MP3, being a popular file type, is often where the problem is noticed.
fred01 at
Awesoken said
There is no ZIP in MP3. You read Zardalu's words wrong. He was referring to a bug with file sharing services not always sending the whole file. MP3, being a popular file type, is often where the problem is noticed.
Ok, I get it. Thank you for clarification.
Zardalu at
Thanks, Ken, for clarifying what I meant. Yes, I should have stated that more clearly. A long time ago, I wrote a utility to "fix" MP3s. I found that the last frame (if indeed it WAS the last to start with) often had a few bytes missing. Most players didn't care and played the files correctly. Correctly in the sense that the person listening to them wouldn't be able to tell. But I, being as anal as I am, wanted my MP3s to not be corrupt. So I wrote my utility and it told me when an MP3 was corrupt and it fixed it by making the corrupt last frame correct (filling it with "nothing"... after all, you can't hear the corruption 99% of the time it anyway).
What I meant, really, was that the glocal directory of a ZIP file is all the way at the end of that ZIP file. The local entries are not. Now apart from the case of a "streaming ZIP file" (and my ZIP file reading code handles those correctly), where the CRCs are not known until afterwards, the entries in the global directory at the end of a ZIP file should be identical to the local entries earlier on in the ZIP file. So if you get a ZIP file with some bytes at the end missing, then there will be discrepancies between global and local, because global is incomplete. All PKZIPFIX does is that it reads the local entries and constructs a new global directory from those.
In pretty much all my programming, I rather am safe than sorry. So when something is wrong, I don't make assumptions that have the risk of losing things. That's why DeflOpt bitches and quits. I really don't want to make DeflOpt less reliable, so the alternative would be for me to write something similar to my MP3 utility and similar to PKZIPFIX. But I'll have a look first at those files to see exactly what the problem is.
Ben Jos.
Edited at
Zardalu at
Awesoken said
I was not aware that data in MP3s were also compressed with ZIP.
There is no ZIP in MP3. You read Zardalu's words wrong. He was referring to a bug with file sharing services not always sending the whole file. MP3, being a popular file type, is often where the problem is noticed.
I'm sure that in some cases, it's a bug, but I'm also sure that in other cases, it's a situation of someone uploading a file and being kind enough to wait until it finishes uploading, but then still disconnecting a few seconds early because it seemed to him that the file had finished uploading, but the last few bytes still never arrived. And in the case of file sharing services, a lot of files (including corrupt ones) multiply like rabbits...
Ben Jos.
Zardalu at
DeflOpt 2.03
I am going to release version 2.03 immediately after posting this message. 2.03 only fixes 2 problems. No change in speed or optimisation performance.
fred01: Thanks again for the files. I examined them and found that they WERE correct after all. So DeflOpt did it wrong. Fixed in 2.03.
fred01: Sorry, I added some code a while ago to add a logging option, but I have not yet finished it. So logging is not in 2.03.
Andrew: Sorry, but I have not yet made the "silent" switch even more silent.
The way things look now, 2.04 will be released in December and it will include the two things mentioned above that did not make it into 2.03.
Go to http://www.walbeehm.com/download/ to download it and either e-mail me (read DeflOpt.txt) or post something in this thread for comments, problems, and so on. Note that e-mail is faster because I don't check here very often and I have notifications turned off.
Ben Jos.
fred01 at
Re: DeflOpt 2.03
Zardalu said
fred01: Thanks again for the files. I examined them and found that they WERE correct after all. So DeflOpt did it wrong. Fixed in 2.03.
I tried, but now get: "ERROR: Zero length data." with the file. I double checked by redownloading it and comparing with a local copy I forgot to erase :roll: and they are binary identical, so there were no corruption during the transfer.
Could you try to look at it again? (or try with your own file if you have OpenOffice installed. I have this error with all my files).
Zardalu said
fred01: Sorry, I added some code a while ago to add a logging option, but I have not yet finished it. So logging is not in 2.03.
Thank you for thinking about it, and take you time with this one, as it is a "convenience option".
Zardalu at
fred01 said
I tried, but now get: "ERROR: Zero length data." with the file.
There is something strange with the file "current.xml" inside that archive. It is a zero-length file, but its compressed size is given as 2. Normally, DeflOpt does not have a problem with zero-length files, but for some reason, this one causes problems. I'll have a closer look.
Ben Jos.
Zardalu at
DeflOpt 2.04
fred01 said
I tried, but now get: "ERROR: Zero length data."
This is fixed in version 2.04. Get it from http://www.walbeehm.com/download/. I have also updated the other tools there since they were affected too.
Ben Jos.
fred01 at
Re: DeflOpt 2.04
Zardalu said at
This is fixed in version 2.04. Get it from http://www.walbeehm.com/download/. I have also updated the other tools there since they were affected too.
Works fine now, thank you. (Sorry for the delay, somehow I missed your last reply in the thread. :-[ )
jojo4u at
Re: DeflOpt thread
Hello Ben Jos,
your download site http://www.walbeehm.com/download is down since some days. And I couldn't find a mirror. Could somebody please point me to a copy? Version 2.02 apparently creates a BJWxxxx.tmp file in the %TMP% directory when it's not able to compress the file. I once had 19 000 files in my temp directory after some extensive testing ...
Zardalu at
jojo4u said at
your download site http://www.walbeehm.com/download is down since some days.
It's a DNS problem. The IP has changed recently and DNS servers take a while to catch up. I find that I can't access walbeehm.com myself from home, but I can from my job... .
If you can't wait, send me an e-mail to walbeehm.com with the part before the "@" also "walbeehm". And put DeflOpt in the subject.
Ben Jos.
Zardalu at
Alternatively, in the mean time, http://web12.topchoice.com/~walbeehm/download should still work.
Ben Jos.
jojo4u at
Thanks, both pages are working now.
In 2.04, there is still the problem with the temp file. Thanks to this, I got 65000+ files at one point in my temp dir. This slowed down DeflOpt significantly! Basically, it creates a BJWxxxx.tmp file in the %TMP% directory when it's not able to compress the file further. Command line is: deflopt.exe /k /s file > nul
Zardalu at
DeflOpt 2.05 released
jojo4u said at
In 2.04, there is still the problem with the temp file. [...] Basically, it creates a BJWxxxx.tmp file in the %TMP% directory
Thanks! Fixed in 2.05. Available now from http://www.walbeehm.com/download.
Ben Jos.
Zardalu at
Shameless plug
OK. I know this is a shameless plug, but it also gives some information on the programs I have made available for download. I have actually made a "real page" out of my download page: http://www.walbeehm.com/download/.
Ben Jos.
krick at
Re: Shameless plug
Zardalu said at
OK. I know this is a shameless plug, but it also gives some information on the programs I have made available for download. I have actually made a "real page" out of my download page: http://www.walbeehm.com/download/.
Ben Jos.
I'm having trouble downloading files from your page. It looks like your web server is sending a content type of "html" for "7z" files. When I click on one of the download links, IE shows me the contents of the file instead of prompting me with the download dialog. It's possible that there's a problem with IE on my end but I just wanted to check if anyone else is seeing the same thing.
Sined at
Re: DeflOpt thread
I have the same “problem”, you just need a right click, and “save to”.
fred01 at
Re: Shameless plug
krick said at
I'm having trouble downloading files from your page. It looks like your web server is sending a content type of "html" for "7z" files. When I click on one of the download links, IE shows me the contents of the file instead of prompting me with the download dialog. It's possible that there's a problem with IE on my end but I just wanted to check if anyone else is seeing the same thing.
It also confuses my Firefox v2. May be this is a limitation of the server that uses html for unknown content types?
Zardalu at
Display instead of download
krick said at
When I click on one of the download links, IE shows me the contents of the file instead of prompting me with the download dialog...
Thanks. Fixed.
Ben Jos.
Martin at
Re: DeflOpt thread
I think it’s better to use application/x-7z-compressed instead of unknown/nothing.
Zardalu at
Martin said at
I think it’s better to use application/x-7z-compressed instead of unknown/nothing.
Thank you! Changed it.
Ben Jos.
counting_pine at
Hi Ben Jos, Do you have any idea why DeflOpt throws an error on this image? http://img405.imageshack.us/my.php?image=colorspaceg6r5b5tf3.png
I get the message: "ERROR: End of input without encountering EOB!?!?!?."
Zardalu at
counting_pine said at
Hi Ben Jos, Do you have any idea why DeflOpt throws an error on this image? http://img405.imageshack.us/my.php?image=colorspaceg6r5b5tf3.png
I get the message: "ERROR: End of input without encountering EOB!?!?!?."
Hmm... I'll look into it. Meanwhile, if you want to make it smaller, you still can:
*** DeflOpt V2.05 *** *** Built on Sat Mar 10 10:03:30 2007 *** *** Copyright (C) 2003-2007 by Ben Jos Walbeehm ***
"C:/INTERNET/DOWNLOAD/colorspaceg6r5b5tf3.png" Number of bytes saved: 1 (672 --> 671) File rewritten.
Number of files processed : 1 Number of files rewritten : 1 Total number of bytes saved: 1 19,501,792 cycles.
Thanks for letting me know! I'll post more when I find out more.
Ben Jos.
Zardalu at
DeflOpt 2.06 released
counting_pine said at
Hi Ben Jos, Do you have any idea why DeflOpt throws an error on this image? http://img405.imageshack.us/my.php?image=colorspaceg6r5b5tf3.png
I get the message: "ERROR: End of input without encountering EOB!?!?!?."
This is fixed in version 2.06, which I have just released. Thanks for reporting!
Ben Jos.
counting_pine at
Re: DeflOpt thread
Thanks, it's working now. Hope it wasn't too hard to track down. Do you mind if I ask why it was happening? Or would I need intimate knowledge of the program's inner workings to understand?
Zardalu at
counting_pine said at
Thanks, it's working now. Hope it wasn't too hard to track down. Do you mind if I ask why it was happening? Or would I need intimate knowledge of the program's inner workings to understand?
Deflated data consists of codes of varying lengths. These lengths can be anywhere from 1 to 15 bits. When parsing deflated data, you could do it by getting one bit at a time from the input until you have a valid code, then decode it, process it, and then repeat this for the next code and so on. This works nicely, but is very slow. So instead of getting one bit at a time, my parser actually looks at the next 15 bits and determines what the next code is from that. This can be implemented in a way that is much faster than the one-bit-at-a-time method.
My code has a buffer that contains bits that the parser has read, but not used yet. Whenever there are not enough bits in this buffer, it keeps reading one byte (8 bits) at a time from the input and adding those 8 bits to the buffer until there are enough bits in the buffer. An example: At the very start the buffer is empty and let's say the parser wants to look at the next 15 bits. So it reads one byte from the input and puts it in the buffer. Then there are 8 bits in the buffer. Still not enough, so it reads another byte and adds those bits to the buffer as well. Now there are 16 bits in the buffer. The parser looks at the first 15 bits and let's say it finds that the first code is 4 bits long. It would then process that code and discard those 4 bits from the buffer. So then the buffer contains 12 unused bits.
This works nicely until it gets to the end of the input. When it reaches the end of the input and wants to look at the next 15 bits, the input might not contain that many bits anymore. So when it cannot read a byte from the input anymore to be added to the buffer, it adds a dummy zero-byte to the buffer (so 8 zero-bits). It adds this dummy byte only once because otherwise, in case the deflated data is corrupt, keeping adding dummy bytes could result in an infinite loop. In the case of this particular PNG, this went wrong. I'll simplify the explanation of what actually happened here. What actually happened was a lot more rare but more complicated to explain. The simplified explanation still is very close to what actually happened:
Towards the end of parsing the deflated data, at some point, there were 6 unused bits left in the buffer and no more bytes in the input. The parser wanted to look at the next 15 bits, and since there were not enough bits in the buffer, and since it had just read the last byte of the input, it added a dummy zero-byte. So then there were 14 bits in the buffer. But it needed 15. It had already added a dummy zero-byte, so it would not add more. It then ended the parsing loop, saw that it had not encountered the final "End Of Block" code and exited with an error. In fact, the 6 unused bits that were initially left in the buffer exactly constituted that EOB code (so the deflated data was not corrupt), but the parser could never see it.
The fix was easy: Instead of adding ONLY ONE dummy zero-byte (8 zero-bits), it now adds ONLY ONE dummy zero-word (16 zero-bits) when it has reached the end of the input.
Ben Jos.
counting_pine at
Thanks, that makes a lot of sense. I've written some inflate code before and I'm aware of the problem, but my code was PNG specific, and I had 32 bits worth of checksum to parse at the end, so it wasn't something I had to work around.
I'm still intrigued about why this case was so rare. Is it much more complicated?
Its`about at
This programme is great, not only in that it can achieve close to what kzip does with -r in a fraction of the time (that is, after using kzip without -r), but that it works on gz too.
A feature request, if I may: Matroska container level support. Even after using Matroska, I still manage to compress it by another 2-4MB with kzip, which is annoying, in that I can't do that at the container level.
Another feature request would be native Linux support but, I think that'd be pushing my luck ;)
Zardalu at
Its`about said at
This programme is great, not only in that it can achieve close to what kzip does with -r in a fraction of the time (that is, after using kzip without -r), but that it works on gz too.
A feature request, if I may: Matroska container level support. Even after using Matroska, I still manage to compress it by another 2-4MB with kzip, which is annoying, in that I can't do that at the container level.
Another feature request would be native Linux support but, I think that'd be pushing my luck ;)
I think you misunderstood what DeflOpt does and does not do. DeflOpt is NOT an archiver like kzip. It can NOT do compression on its own. Rather, it grew from an archiver I wrote (called BJWFlate) many years ago. I found that there were a few optimisation algorithms that I used in BJWFlate that no other archivers used.
The easiest way to explain DeflOpt, maybe, is this: (1) Take one or more files. (2) Compress them with your favourite utility or set of utilities. (3) Run DeflOpt.
No matter which utility you use, DeflOpt can usually use the "hard work" that archivers did and improve on that. So you can use archiver A and compress it down to 20 kB. DeflOpt will usually be able to make it a few bytes smaller. Use archiver B (let's say it's Ken's kzip) and compress it down to 10 kB. DeflOpt will usually be able to make it a few bytes smaller.
So: DeflOpt is NOT an archiver. It simply takes the hard work done by other archivers and tries to improve on that. If you think that when utility A compresses something to 20 kB and kzip can do it in 10 kB and DeflOpt can make kzip's result 10 bytes smaller, yes that's true, but DeflOpt will NOT be able to take utility A's compression and make it even compete with what kzip can do "natively".
If you understood my explanation, you will now also understand why DeflOpt will never be able to help you on your Matroska request.
Your 3rd request: Yes, I could probably make DeflOpt versions for Linux, Mac, and other platforms. Or have someone else more knowledgeable with those platforms port my code. But only 1 or 2 people are interested. Not enough for me.
It reminds me of when I wrote a uudecoder back in early 90s (UU.COM). I got literally thousands of e-mails telling me that mine was the best and so on and that they would gladly pay for it, but it was free. Many people asking me for ports for Linux. I made it shareware for 5 dollars, shortly after that. All the thousands of people, where were they then? I think I got around 50 people who registered.
I'm not in this game to become rich and famous. On the other hand, my experience with UU.COM and one or two things before and after that have made me very reluctant to share my code.
Ben Jos.
Zardalu at
counting_pine said at
I'm still intrigued about why this case was so rare. Is it much more complicated?
Well, I don't always read 15 bits ahead, but rather, the number of bits of the longest code. In lots of cases, this will be 15 and in lots of cases, this number is also (obviously!) the length of the EOB code. In this particular case, that number was 6. No problem there. But it had two codes ahead of it both of length 1. That's what made it rare.
Ben Jos.
Its`about at
Zardalu said at
Its`about said at
This programme is great, not only in that it can achieve close to what kzip does with -r in a fraction of the time (that is, after using kzip without -r), but that it works on gz too.
A feature request, if I may: Matroska container level support. Even after using Matroska, I still manage to compress it by another 2-4MB with kzip, which is annoying, in that I can't do that at the container level.
Another feature request would be native Linux support but, I think that'd be pushing my luck ;)
I think you misunderstood what DeflOpt does and does not do. DeflOpt is NOT an archiver like kzip. It can NOT do compression on its own. Rather, it grew from an archiver I wrote (called BJWFlate) many years ago. I found that there were a few optimisation algorithms that I used in BJWFlate that no other archivers used.
The easiest way to explain DeflOpt, maybe, is this: (1) Take one or more files. (2) Compress them with your favourite utility or set of utilities. (3) Run DeflOpt.
No matter which utility you use, DeflOpt can usually use the "hard work" that archivers did and improve on that. So you can use archiver A and compress it down to 20 kB. DeflOpt will usually be able to make it a few bytes smaller. Use archiver B (let's say it's Ken's kzip) and compress it down to 10 kB. DeflOpt will usually be able to make it a few bytes smaller.
So: DeflOpt is NOT an archiver. It simply takes the hard work done by other archivers and tries to improve on that. If you think that when utility A compresses something to 20 kB and kzip can do it in 10 kB and DeflOpt can make kzip's result 10 bytes smaller, yes that's true, but DeflOpt will NOT be able to take utility A's compression and make it even compete with what kzip can do "natively".
If you understood my explanation, you will now also understand why DeflOpt will never be able to help you on your Matroska request.
Ben Jos.
I think you've misunderstood yourself ;) I'm asking you to add support for optimising container level compression. Mkvtoolsnix uses zlib to compress matroska files at the container level (Matroska supports Deflate, LZO and bzip2, but players can only read Deflate at the moment), but the compression isn't exactly very good. I'm saying that I can draw this conclusion from having used kzip to further compress the files at the archive level. I want greater compression at the container level.
Edit: As for it only achieving a few bytes, I regularly get 1-2KiB shaved off PNGs with DeflOpt.
Edited by Its`about at
Zardalu at
Ah, yes. I did some reading on Matroska and I indeed misunderstood. I guess I didn't understand the use of the word "container". In fact, a PNG file would be a container then too. It contains several chunks ("elements") and some of those may or may not be Deflated. In the case of PNG files, the chunks/elements that could (but not necessarily do) contain Deflated data are iCCP, iDAT, iTXt, and zTXt. And, on that same abstraction level, ZIP files are containers as well.
But I think that my original statement still holds. Yes, even if I extended DeflOpt to do .mkv (or .mks and .mka) files as well, I would ONLY be trying to improve upon zlib... not upon the several superior Deflate programs.
OK... here are my thoughts right now: (1) I HATE to write parsers. Parsers are just programs to follow a specification. There really is not much "innovation/challenge/talent" involved. "Everyone" can parse a file. Even more so since there are so many libraries available. They're like "cook book recipes". Which brings me to point 2: (2) If you take a look at Ken's code for just about ANY program he has written, you'll see that he pretty much never uses 3rd party libraries; he writes everything from scratch. I am very much the same way. I do not want to include the Matroska library or zlib or any library in my code. (If you want to really nitpick, you could say that both Ken and I still rely on the Microsoft Run-Time library...).
I can see how DeflOpt could make Matroska files smaller, but I would have to study the Matroska file format specification first, write a parser for it. After having dealt with ZIP and PNG files, I'd rather write something more generic, something I have outlined some ideas about before: A tool that could convert any file using the Deflate format into the format most used (ZIP), then use tools such as ZipMax and zipmix to optimise them (with DeflOpt as a final step), then use the "inverse" of the tool to convert that file back into the original format. A tool like that could be written such that it would only rely on "script language" to indicate which "meta-fields" need to be modified. So a plug-in architechure.
Just think about it: Having a tool like that, converting the Deflated parts of Matroska files to regular ZIP format is a LOT better because then you could use ZipMax/zipmix to use hundreds of settings/programs to optimise the compression.
I guess the bottom line is that, yes, I could add Matroska support to DeflOpt, but it requires more work at this point than I am willing to put my very limited free time into...
Ben Jos.
Edited by Zardalu at
Its`about at
Zardalu said at
If you take a look at Ken's code for just about ANY program he has written,
Truth be told I wouldn't mind looking at kzip ;)
I appreciate your response. I use DeflOpt a lot, though I don't like having to type wine before every command ;) I hope that someday you manage to find a way of making it work with limited time, via your suggested method or otherwise.
Edit: I misread what you said, removed part of my post that didn't make sense.
Edited by Its`about at
Zardalu at
I could probably port DeflOpt to Linux pretty easily. But I'm not sure if I'm ready to make its source code publicly available. I'm NOT looking for someone to SELL DeflOpt, mind you. I'm just a bit paranoid about releasing my source code. I have had way too many bad experiences in the past. People stealing my code (not only programs, but also web pages) and releasing my code as if THEY had written it. I HATE thieves. My code is my own. Good or bad. I do NOT borrow or steal from anyone else. I think Ken can attest to that.
Ben Jos.
TSamuel at
I would like to request returning the old behavior of DeflOpt 1.x (via a switch or whatever) to at least consider files of any extension. I often have files that are renamed from .gz, .zip, or .png to something else (mostly proprietary extensions) that I would like to optimize via script w/o either having to rename & then remember the old extension or making temporary hardlinks (if supported).
Thanks for a great utility, TPS =)
Zardalu at
TPS said at
I would like to request returning the old behavior of DeflOpt 1.x (via a switch or whatever) to at least consider files of any extension.
The reason I removed that is because 1.x dealt with ZIP files only. 2.x added PNG and GZIP support. In order to add the old behaviour, I would have to add code that tries to recognise the file format...
Still, I'll keep your request in mind.
Ben Jos.
Zardalu at
TPS said at
I would like to request returning the old behavior of DeflOpt 1.x (via a switch or whatever) to at least consider files of any extension.
I'm considering again...
I just found out that .kmz files are simply .kml (Keyhole Markup Language) files that have been zipped (PKZIP-compatible). While it is a lot easier for me to add code that treats a .kmz extension the same way as DeflOpt treats a .zip extension, the .kmz thing has made me realise that making DeflOpt smarter (i.e. making it try to determine the file format by examining the actual file instead of simply looking at the extension) will make DeflOpt be able to deal with future extensions that are really just existing formats a lot better. So the "any extension request" has now gotten a higher priority for me. I'm still worried, though... . People could specify "*" in the root directory with the recursive switch on, and DeflOpt would scan/examine/parse every single file on the entire drive...
Meanwhile, does anyone know of any more extensions that are simply new extensions for old formats?
Also, I have received a request by email to have DeflOpt be able to keep the original date/time while rewriting files. This will definitely be a feature of the next version.
Ben Jos.
fred01 at
Zardalu said at
Meanwhile, does anyone know of any more extensions that are simply new extensions for old formats?
J/R/WAR (Java); ODT/S/P/G/F/B/M (OpenOffice.org documents), OTG/P/S/T (OpenOffice.org templates), SXD/W (Sun XML Draw/Writer) also BAU, SOB, STW, for various resources and DAT but this will be probably too generic; CRF(Thief1/2 resources); PK3/4 (Quake3/4); KPZ (Komodo Project Zipped); XPI (Mozilla/Chrome theme/extensions); WMZ (Windows Media Player Skins);
Zardalu said at
I'm still worried, though... . People could specify "*" in the root directory with the recursive switch on, and DeflOpt would scan/examine/parse every single file on the entire drive...
How about showing a warning if the list contains more than 100(1000) items, similar to the way it is used in bash when autocompleting?
Edited by fred01 at
ace_dent at
Compatability problem with DeflOpt and Paint Shop Pro...
Howdy!
It's great to see DeflOpt continue to develop. I have stumbled across a problem with large images (generally photos), where the resulting png cannot be displayed in Paint Shop Pro X (PSP)... it just appears as 100% black. While I'm happy to accept this might be poor compliance on the part of PSP, I thought it might be worth further investigation in case there is some bug lurking there! I have isolated a minimal test case that can be downloaded here. Details:
0. BMP 300x1 px, 8bpp grayscale (although only 17 unique colors), 1,378 bytes. 1. -> OptiPNG, PNG : 8bpp grayscale, 214 bytes. (IDAT 157bytes). Views OK. 2. -> DeflOpt: 8bpp grayscale, 213 bytes. (IDAT 156bytes). Will *NOT* display in PSP...
Meanwhile, does anyone know of any more extensions that are simply new extensions for old formats?
.svgz (gzipped SVG)
Zardalu at
fred01 said at
How about showing a warning if the list contains more than 100(1000) items, similar to the way it is used in bash when autocompleting?
Sounds good, but it would cause problems when people use scripts that are meant to run unattendedly. And, sure, I could add another switch to not display warnings, but I think I'll just assume the user knows what he is doing. After all, DeflOpt is a command line program and people who know how to use a command line generally are not newbies. ; ) Reminds me of a quote saying "Make a program fool-proof and only a fool will want to use it." And I don't want to spend my valuable time writing code that protects people who don't know what they're doing against themselves. If I did that, I would never get around to writing really useful things. : P
ace_dent said at
It's great to see DeflOpt continue to develop. I have stumbled across a problem with large images (generally photos), where the resulting png cannot be displayed in Paint Shop Pro X (PSP)... it just appears as 100% black. While I'm happy to accept this might be poor compliance on the part of PSP, I thought it might be worth further investigation in case there is some bug lurking there! I have isolated a minimal test case that can be downloaded here. Details:
Thanks, Andrew. Always happy to see feedback from you. I downloaded the file and I'll look into it.
OK... you guys have convinced me... . I've received a lot of requests both here and through email to reinstate 1.x behaviour. Meaning that DeflOpt will scan files regardless of extension. And seeing how many other extensions there are that really are just GZIP, PNG, or ZIP files, and realising that many more could be created every day, I'll add a switch to have DeflOpt be "smart" and scan files first to see if they are in a supported format and then treat them accordingly. Using this switch of course will make DeflOpt slower, but I'll just assume the user knows what he's doing when specifying "/a". New switches will be "/a" and "/d". I hope to find the time to code those changes soon.
Ben Jos.
Zardalu at
Re: Compatibility problem with DeflOpt and Paint Shop Pro...
ace_dent said at
I have stumbled across a problem with large images (generally photos), where the resulting png cannot be displayed in Paint Shop Pro X (PSP)... it just appears as 100% black. While I'm happy to accept this might be poor compliance on the part of PSP, I thought it might be worth further investigation in case there is some bug lurking there!
I have analysed the files you made available. The only thing I could find that might cause PSP to have a problem is the following:
The compressed data does not use any distance codes. In TestCase1.png, two distance codes, distance code 0 and distance code 1, are defined and both are given a length of 1, but neither one is actually used. In TestCase2.png, since the deflate specification requires at least one distance code to be defined, DeflOpt defines one distance code, distance code 0, and gives it a length of 0. According to the deflate specification, this is perfectly valid. The specification at http://www.gzip.org/zlib/rfc-deflate.html states (bold added by me):
A code length of 0 indicates that the corresponding symbol in the literal/length or distance alphabet will not occur in the block, and should not participate in the Huffman code construction algorithm given earlier. If only one distance code is used, it is encoded using one bit, not zero bits; in this case there is a single code length of one, with one unused code. One distance code of zero bits means that there are no distance codes used at all (the data is all literals).
While I don't have PSP, I have tried a handful of viewers and editors and none of those had any problem with TestCase2.png. This of course does not prove there's nothing wrong with that PNG, but all this makes me think that that PNG is perfectly valid and that PSP has a bug related to this.
If you run DeflOpt in verbose mode (/v), for TestCase1.png, it will show:
I am wondering if the PNG files you have problems with in PSP all have "D-codes: 1".
Ben Jos.
Zardalu at
DeflOpt 2.07 released
I have just released DeflOpt 2.07. Get it from http://www.walbeehm.com/download/. The changes: - /a option added ("scan all files") - /d option added ("preserve date and time")
See the included DeflOpt.txt for more information.
Ben Jos.
Awesoken at
Re: DeflOpt thread
DeflOpt defines one distance code, distance code 0, and gives it a length of 0. According to the deflate specification, this is perfectly valid.
Damn, I could have saved you some time. Shame on me for being lazy and silent. I went through exactly the same process with PNGOUT. The /mincodes option of PNGOUT adds a distance code if none exist. The source of the problem is in Zlib 1.2.1, which had a bug where it would report an error if the stream had no distance codes - even if it was legal according to the official spec. A few months later, somebody noticed PNGOUT was generating images that didn't work on certain mobile phones. Apparently, making it store at least 2 distance codes seemed to fix it. Apparently, these phones are even buggier than Zlib 1.2.1. You are welcome to copy my /mincodes option name.. as I'm sure it'll make it easier for everyone.
Zardalu at
Awesoken said at
Damn, I could have saved you some time. Shame on me for being lazy and silent.
I could have saved myself some time too. I read about the "cell phone problem" here, but only skimmed through those posts. If I had paid more attention, I would have known what your "mincodes" option meant. I don't know if I am going to add something to DeflOpt to deal with it, though. I am reluctant because (1) Zlib 1.2.1 was buggy and those cell phones are buggy, in the sense that they do not conform to the specification. I feel it's up to "them" to fix the problem and up to users to upgrade to versions that don't have those bugs (and hoping the problem won't even exist anymore after that), and (2) because, similar to what you mentioned about your PNGOUT, my DeflOpt was meant to optimise size. Why make it do something sub-optimally just because someone else did not conform to the specification?
I think I'm a bit different from you, Ken, in that I am mainly programming tools that I find useful myself and as long as they work for me, it will take a lot for me to make them useful for others. I think you already know that about me, Ken, but I just thought I'd make it explicit. I'm not programming to become famous or anything. I just program things that I find useful myself and if others find my code useful too, great. If not, hmm... well... maybe... ; )
Most programmers are like me, I guess. They program for their own use. Ken is an exception in at least two ways: (1) He is the best programmer I have ever encountered. (2) He puts in a lot of effort to make his programs work for everyone (including me!).
Ben Jos.
Zardalu at
Linux port
I have had some requests for a Linux version of DeflOpt, both here and through email. I thought a Linux port would be relatively straightforward, but then I tried to compile DeflOpt using gcc (or, rather, g++) under Cygwin and saw that there were a lot more problems than I thought.
I hate source code that has #ifdef statements all over the place (because the more portable you make something, the less readable it tends to become), so I rewrote some things so that there are only very few #ifdef statements. I'm not entirely finished yet, but most of the work is done now.
I am looking for someone who has a Linux box and is willing to test a native Linux version of DeflOpt. Of course, that person would get the source code of DeflOpt and my Makefile for it to compile it with g++. Preferably someone that has some knowledge of makefiles and programming under Linux. That person could then host the executable (but not the source) or I could host the executable myself.
Since I do not check this forum very often, please reply to me by email if interested. And read DeflOpt.txt first to see how to email me, because it's very likely if you don't do that, I will never see your email. ; )
Thanks! Ben Jos.
Zardalu at
Re: Linux port
Oh... one more thing... in lots of places, the DeflOpt code assumes "little-endian-ness", so a Linux executable compiled for a big-endian CPU would not work.
Ben Jos.
Zardalu at
Native Linux executable
Thanks to JonoF and Ken, I now have access to a Linux box. I compiled a native Linux version of DeflOpt. It was compiled in 32-bit mode on a 64-bit Gentoo Linux box using "g++" 4.1.2. Note that this executable will only run on i386 compatible Linux boxes. I am looking for people who want to beta test DeflOpt on a Linux box. Contact me by email if you are interested in beta-testing. Instructions on how to email me can be found in DeflOpt.txt, part of the DeflOpt.7z package on the download page on my website (http://www.walbeehm.com/download/). Make sure to put the word "DeflOpt" in the subject line because your email will most probably be deleted automatically otherwise.
Note that I put "g++" within quotes because it seems I can't post this message without the quotes around that?!?!?!