Thanks.

Do GOG actually pack or re-pack all games?
It would be nice to know, but there might be a bunch of variables to that answer.

As you say, all games should have an integrity check regardless of size.
If a small size though, I guess using my GOGPlus Download Checker program will check fairly quickly, but that of course relies on internal checksums existing to give a true verification. Failing that, you are left with the download checksum verification, which currently requires Galaxy or gogrepo.py.

P.S. I guess it is possible, that the program (i.e. InnoSetup) used by GOG to create the installer, always provides the internal checksums, but they may not always be checked when installing. In that case, my GOGPlus Download Checker program would certainly be worthwhile using. In any case, it would show you on use, what it found.

Checksums are actually the elephant in the room.

Clearly they exist, as both Galaxy and gogrepo.py can access and use them, as presumably did the old GOG Downloader.

Things wouldn't be half so bad, for those of us not wanting to use Galaxy, if there was a checksum alongside every browser download, that we could use with or after downloading with either our browser or some preferred third party downloader.

This is especially so, as many of us have a right-click option in Windows, to verify a checksum with the selected file. Or as I mentioned earlier, I could whip up a simple checker using the free 'fsum.exe'. So it could be as simple as paste plus drag & drop.

That's it, you've put it better than I initially did when I was talking about the packing (wrong term) as it's indeed the installer used (still talking about offline ones). I could be wrong here but I doubt very much there is a silent verification when there is no option to verify prior to install. It wouldn't make sense to have the option on some games but not others and to it anyway.... But at this point, what doesn't make sense to me could make perfect one for deciders.

Agreed on the rest.

I am now ready to do some serious testing, starting with UPDATE ALL, to build a new manifest from scratch. I need to do this rather than just use my existing one, to see if the forked version does anything that might impact my extraction code.

If you recall, all up for the 978 games in my GOG library, it took 5 hours and 50 minutes to get all the required detail via UPDATE ALL to build the manifest file on my PC. That was for both Windows and Linux plus all game extras. I now have 1,010 games, so it will take a little longer just due to that, plus the forked version of 'gogrepo.py' seemingly gets more detail than the original version, so potentially the full update could take several hours ... and providing nothing goes wrong, I should never need to do it again.

Doing a full update, if you have a lot of games, is the biggest imposition from using 'gogrepo.py', but well worth the benefit. However, the program can also update based on game title, and so you could conceivably build the manifest up slowly for just those games you don't yet have.

How big the imposition is, depends on the power and speed of your PC, plus state of your web connection, plus the number of games in your GOG library. For some, it will be barely any imposition at all to create a full manifest for the first time. For others it could take several hours, or potentially even longer.

I do note though, that with the forked version of 'gogrepo.py', there is a Resume option, so I guess you could do the Update process in stages if you are savvy enough with that.

Whatever the case may be, you cannot really use 'gogrepo.py' without that manifest file, so using UPDATE is essential.

P.S. Doing many of the tests is time consuming, so more often than not, it is that and not just my coding that holds up the show. Then there is fine tuning and bug fixing. As you might appreciate, I don't test absolutely everything, before making a usable version available. If I did, you would most likely be waiting until Xmas or beyond for a release.

Alas I had to postpone doing the BIG UPDATE tonight ... ran out of time.

However, I did do a couple of little ones, checking on an issue for someone - see here

And from that I learnt some things.

1. Manifest created with original 'gogrepo.py' is not compatible with the forked version.
2. Trying to update an entry in that manifest with the forked version causes errors.
3. The manifest needs to be created from scratch with the forked version.

I also discovered I had changed something in my code during an earlier reworking, that prevented me adding games one by one, just by their title using UPDATE ONE. I've now fixed that, and added two games that way to an empty manifest, as mentioned at the link above.

I also found out that the Log file for UPDATE is not a game folder based one, but a file in the main program folder called - gogrepo.log, with new entries just tacked on the end. Not what I was hoping for, but good enough I guess.

The main program window has been slightly modified with the resizing of the LOG button to accommodate a checkbox that enables opening the new 'gogrepo.log' file, instead of the program 'Record.log' file. You only see these changes when the forked version of 'gogrepo.py' is in use, as evident in the following screenshot.

Main Window

Essentially this mean you have quick easy access to the 'gogrepo.log' file to check on Verify etc results.

Still not found time to do the full update from scratch, but meanwhile I have been testing and adding new features.

The List has two more right-click options.

Remove Selected Game - You get a dialog prompt with three options.
(1) Remove game from manifest and list.
(2) Just remove from manifest.
(3) Abort.

Delete Manifest - This deletes the manifest file (via a prompt) and empties the List and title list file.

Main Window + Menu screenshot

I am now working on a new feature for the GOGRepo GUI.

You can see the current changes to the Update window to illustrate that in the following screenshot.

Update Window

Basically it is a way to do a full Update in stages.

For some of us, with a slow PC and many games, it can potentially take several hours to do an Update from scratch, which occurs when you first use 'gogrepo.py' to create your manifest file for the first time, plus it is also something you should do regularly (every month or two), so I have been told, to ensure you get everything, as sometimes GOG fails to add an updated tag/flag.

Doing a full update in stages, means you don't need to come up with a lot of time in consecutive hours, but can rather do it with the odd hour or portion of time, here and there. And as I am sure, many may be put off by the time needed to create their first manifest file, this should be greatly helpful.

I see 'Update ALL In Stages' as being done in two ways.
(1) For no manifest.
(2) For an existing manifest.
NOTE - You are prompted for which method to use. With method (2) you also potentially have the ability to convert your manifest from the one used by the original 'gogrepo.py', to that used by the forked version.

As you may notice in the screenshot, there is something called 'Blocks' and something called 'Titles', that can be adjusted. Basically a block is the specified number of titles (games). So in the screenshot, you see 3 blocks and 20 games per block. That means 60 games would be updated in one pass, with 1 block being a process, so 3. Currently the maximums are 9 blocks and 30 titles (270 games).

You enable the Stages checkbox to gain access to this new feature.
Then you click the BEGIN button to start. This either runs a full update for a blank manifest until your total count of games and titles have been saved to the resume manifest, or it just works its way through your existing list of games, removing and then replacing the normal manifest entries based on the number specified.
Once the process has been started, you can move on to the next lot of blocks, using the CONTINUE button.
You repeat with the CONTINUE button until all games have been newly added to the normal manifest file.

The program keeps tabs on what has and hasn't been done.
Using the BEGIN button again, starts over.
If you meanwhile acquire new games to your GOG library, then that will just require using the regular Update usage for that, and won't impact method (1). With method (2) you may want to wait until after all stages have completed or they could get done twice, unnecessarily.

I've now coded a working version of the first way (method), as mentioned last post, and it works great.
That said, it also works with an existing manifest, by just deleting it so you can start over from scratch. Most of the same code will be used for the second way (method).

And during my testing I came up with a better method (more reliable etc) for closing the 'gogrepo'py' process, so when I get the time, I will replace my other currently existing versions in GOGRepo GUI.

Small change to the Update Window, with the addition of an Info button.

Onward and Upward!

------------------------------------------------------

LET GOG KNOW HOW YOU FEEL BY VOTING
https://www.gog.com/wishlist/site/keep_the_gog_downloader_and_keep_it_up_to_date

We are up to 1709 votes now. :)

I've now finally rebuilt my manifest, using the Kalanyr fork of gogrepo.py along with my 'Update In Stages' option (method 1), fine tuning the process as I went, and it worked well.

I've started work on method 2, which uses an existing manifest, and removes and updates game titles as it goes, using the specified number of games to be done at a time. In other words a game entry is completely removed from the manifest and then recreated in it, hopefully meaning all detail is picked up, whether GOG remember to set the update tag or not.

It has occurred to me, that I could use the opportunity of this method, to do some comparison checking, seeing as this may well be the regular updating method of choice, perhaps once a month etc ... dependent of course, on however long it takes the user to work through their game list.

Comparison checking, would mean a before and after comparison, to see if anything has changed, and adding that game to a Changed list if it has. At this stage, I don't know exactly how complex I might go with this ... whether I would try and detect what has changed and create a record of that, or just let the user investigate. The most important bit, is knowing something has changed, especially as I have learnt to both my horror and annoyance, that GOG cannot be trusted to even notify you of an update, let alone provide any detail in a Changelog entry if they do. Not that my comparison reporting would make you any wiser in regard to the Changelog if GOG have failed with that, but at least you will know something has changed, and if you wish, investigate further.

Due to all the above, I have not even tested Downloading and Verifying yet, with the forked version of gogrepo.py, so it will be a while before I do a proper release of the updated version of GOGRepo GUI ... though for those who are super keen, an executable can be found at GitHub on the files list, providing of course that it is remembered the program is incomplete and some stuff untested. The absolute latest version, can be found in the 'Latest_Incomplete' folder, which can be seen as a kind of nightly build.

There has been some minor GUI changes, with the removal of the MOVE button for now, and probably unlikely to be returned. The cover image field now also displays some working statuses, so has been renamed ... currently only used for the 'Update In Stages' process, but that may change and be used to replace all popup Splashes.

Main Window

Update Window

A checkbox has been added to the Update window to the right of the CONTINUE button. It changes that button into a CLEANUP one, which may come in handy after a crash or manual cancel. It is also a three state checkbox, so can be used (block) to skip the first check that CONTINUE does, which should mostly be unnecessary ... it is an automated check of a crash/manual cancel, which can take a few minutes of time.

I've now had occasion, due to the latest two demos at GOG, to test both the Download and Verify processes with my adapted code for the forked version of gogrepo.py. I also tested the normal type of UPDATE ALL, which just revealed those two new games.

Verify worked as expected.
Download however didn't, and had some issues, though the Windows files downloaded okay.
One of the demos had a Linux version and that was being skipped, despite my choice of settings.
Upon investigation, I came to realize that unlike with the original gogrepo.py, you need to specify OS and Language for downloading. The original just takes it from your Update settings.
So I coded a fix for that, but during further testing, discovered you cannot use the -os and -skipos parameters together, but only one or the other, so I have now disabled both the -skipos and -skiplang parameter options. In a GUI scenario having both is not really needed, as you can easily click to select what you want, with no typing to save on.
I also realized I needed to do a similar bugfix to the Floating Progress window code, so that has been done too.
All seems to be working well now after further testing.

As well as having done most of the code now for the second method (way) for 'Update All In Stages', I decided to add location code for the other windows, so that their last position gets remembered. I coded that for all except the Queue window, which takes its position from the main program window which it overlays.

So progress, aside from that little download hiccup, now solved, is coming along quite nicely. :)

IMPORTANT maybe, but I have come to realize, that the Linux SH installer files are only verified to exist (and probably a size check), not any MD5 or internal content checking ... at least from the few I have tested. That may be different if you are using gogrepo.py on Linux. Anyway, I then just used my GOGPlus Download Checker program to verify the SH file integrity.

P.S. Something users need to be aware of, and that I believe is now catered for in the developer version of gogrepo.py, is that you cannot specify to just download a DLC on its own ... perhaps the same for Patch files too, unless they come under shared, which I suspect just refers to any BIN files. So until I get a hold of the developer version and it is readily available to users, my GUI isn't able to do that kind of targeted downloading. That will only come with a future adaption and update.

Today I decided to add a new related option for Update, which is also a quick way to test similar code I am working on.
In short, for UPDATE ONE, you can now enable a checkbox to do a Replace & Compare, for just the one selected game title. I've done it and it works well. You can see the new checkbox option in the following screenshot.

Update Window

If changes are detected the title is recorded to Compared.txt, and you have two text files to compare.

----------------------------------------------

During the testing of downloading with the forked version of gogrepo.py, something has come up that needs further investigation and could be disturbing ... at least for me.

Basically, I have not been able to download above 1.3 Mbs (often less) in the last few days.
Usually, with the original gogrepo.py and the old GOG Downloader and Galaxy and Free Download Manager 5, I get around 4.5 to 5 Mbs.

As a comparison during these last few days, I have been getting my usual 4.5 to 5 Mbs download speed from GOG with Free Download Manager 5, except for Friday evening, where it was down to 3.5 Mbs, which is not surprising as slower web is usually experienced then.

I am only using 1 download thread as opposed to the default of 4 that both versions of gogrepo.py use, but that never impacted me with the original of the two. So I need to test the forked version to see if threads are done differently.

Or it could be something GOG SDK related, as I am reminded that the old GOG Downloader often experienced slow download speeds, especially during the height of a sale or just after. I suspect deliberate tailoring of the speed, especially for game extras files. Free Download Manager 5 with the browser links, never seems to suffer much speed wise, other than when congestion makes it run slow. I also seem to recall complaints about Galaxy running slow with downloads lately.

Okay, I've had a chance to experiment a bit more with download speeds, and increasing the number of download threads in the forked version of gogrepo.py from 1 to 2, 3 or 4 appears to increase download speed from 1.3 Mb/s to 1.8 Mb/s.

That's not enough though, as Free Download Manager 5 is averaging just over 3 Mb/s at around the same period of time.

It seems obvious that there is some congestion right now at the GOG web server end, and anything using the GOG SDK is having its speed tailored to a lower speed, no doubt to reduce the impact on bandwidth for everyone.

So right now it is much more encouraging to use Free Download Manager 5 instead of gogrepo.py, if you want to get your downloads done in a more timely fashion.

The other day, I did just that, then moved the downloads into the same destination folder that gogrepo.py was using, and then used the Verify option of gogrepo.py to check the files downloaded with Free Download Manager 5. That worked well, and is much quicker than the integrity check used by my other program - GOGPlus Download Checker.

Overall though, it is disappointing to miss out on the one click download all files for a game, that gogrepo.py gives you.
That said, with a few clicks you can populate the queue in Free Download Manager 5.

If you care about other customers though, you may want to tolerate what speed the GOG SDK seems to be providing, so you can gain the other benefits of gogrepo.py.

NOTE - While playing around with the number of download threads to be used, I sometimes could not get downloading with gogrepo.py to work ... it would not resume or replace. This meant I first needed to delete content already downloaded to the game title folder, plus what was in the !downloading folder for resuming, before I could then start over.

--------------------------------------------------

I've also tweaked GOGRepo GUI a bit, so that the REMOVE ALL button is now available after a manual STOP where you have left the QUEUE window and then returned to it. The REMOVE ALL button now clears all other download settings, meaning you no longer need to restart the program to get a query for that.

A few more tweaks - fine tuning and improvements.

The UPDATE NOW button on the Update window, has been reduced to UPDATE, and a button added to view the Changed.txt file, that lists game title updates where a change has occurred.

Update Window screenshot

------------------------------------------

Changing the OS option on the main program window, now shows a prompt requiring an OK to make the change permanent, otherwise the change is only temporary and lost after the program closes.

I have now seemingly completed (untested) the BEGIN portion of code for 'Update In Stages' method 2.
The number of game Titles in a block now includes 1, 2, 5, 15 and 25, as well as the original 10, 20 and 30.

EDIT - Now tested and fine tuned.

-------------------------------------------

Interesting thing to note about download speed yesterday, and related.
It was too slow for me with gogrepo.py for the 19 Gb game I wanted to download, so I switched to using Free Download Manager 5 instead, to have it complete in a timely fashion. That was for one EXE and five BIN files.
By about the second or third BIN file, I was getting over 5 Mb/s, which was my usual kind of speed when the web isn't too congested.
It was too late at that point to check how the speed might have improved for the forked version of gogrepo.py, because it is not file specific only game specific, which would mean starting over.
I believe the current dev forked version of gogrepo.py, allows you to be more file specific using wild cards, which would be great, and give you the missing true flexibility that is needed.

Earlier today, I finished the remaining code for the second method of 'Update In Stages', but I still have testing to do before I upload.

Meanwhile I got sidetracked with a related project I have been keen to do for a while - GOGRepo Simple GUI.

I've got the basics done and have tested it a few times and it works well with EXE and BIN files. It should also currently support SH files if the checksum for each exists in the manifest.

GOGRepo Simple GUI screenshot

So this is a third program now, and the simplistic nature of this one, might appeal to more people.

You still need 'gogrepo.py', the original or forked version.
You need Python to be installed along with the required libraries.
You need 'fsum.exe' for the MD5 checking.
You need 7-Zip for zip files and 7z files.
You need UnRAR for RAR files.

I've still got to do the code for ZIP, 7Z and RAR files, which will be copied pretty much from my GOGPlus Download Checker program, but modified to suit. I also need to do the SETUP code for the SETUP button, which does the Python libraries and cookie file in a similar manner to that option with GOGRepo GUI.

The purpose of this alternate GUI, is to support downloading with your browser or other programs. The program itself doesn't do any downloading, and while it does some Updating to the manifest, it does it on-the-fly, so doesn't require creating a manifest file first. It doesn't use 'gogrepo.py' for verifying file integrity, instead it uses 'fsum.exe' and 7-Zip etc. Because of that, your files can reside anywhere and don't require a folder structure supported by 'gogrepo.py'.

The program requires a web connection to get the list of games in your GOG library, and it does that fairly quickly.
It also requires a web connection for part of the verify process, where it populates the manifest for the game you are checking. At the moment, both MD5 and File Size are checked against what is listed in the manifest for the dropped file.

The program doesn't check for changes at GOG, but does grab them during verify. The idea is you keep tabs yourself on what has been updated via your library web page and or changelog. If you want more data, then you are better off using my GOGRepo GUI program.

Use of this new program in a nutshell, is as follows.
(1) Download your game files via browser links, either with your browser or some third party program.
(2) Use GOGRepo Simple GUI to verify the integrity of your downloads, using drag & drop with each file.

P.S. This is much quicker than my GOGPlus Download Checker program, because it does a different kind of MD5 etc checking ... at least for EXE, BIN and SH files. The MD5 checking I imagine, would be the same as what Galaxy does.