v2024.11.06

documentation/!README-mad-v2024.09.24-migration-checklist.txt

@@ -0,0 +1,51 @@
+1.) Read and understand the changes -- see ./documentation/FolderStructure-Default (AIO) or 
+    ./documentation/FolderStructure-WorkDirOverride to visualize the changes.
+    
+Summary:
+--------
+ * Renamed _debug, _temp, _flocks to .debug, .temp, .flocks (so you can hide them if desired)
+ * Renamed 'downloads' folder (transferring files) to .inflight
+ * Renamed 'completed' folder to downloads (makes more sense as there are uploads / downloads (& .inflight)
+ * Moved "plugins" and "hosts" folders to ScriptDir in code (so they must reside in the same folder as
+   mad.sh (and curl_impersonate, mad.config if you choose to use them)
+ * Added ./data folder in WorkDir (the downloads_completed, uploads_completed reside here). It is intended
+   to be a more long-term data storage (things that plugins will / do use.
+ * Moved results.txt to ./downloads for downloads and ./uploads for uploads. This file is the verbose
+   processing of lines (OK, SKIP, FAIL, RETRY, NOCDN, PASSWORD, RENAME, etc). It is helpful to see an outline
+   of all the terminal processing.
+ * A summary of all the uploaded download links is created in ./uploads/results-links.txt (only successes)
+ * Upload completed tickets are put in the ./uploads/_tickets folder to keep it less cluttered.
+ * If uploads are done manual mode (through the UI ./mad.sh upload), then a file named
+   ./uploads/temp_upload_handler.txt is created and used to ensure multiple terminals do not reprocess
+   completed / failed upload files. If using uploads.txt, this is not created and not necessary as it is
+   handled directly in the uploads.txt
+    
+2.) End any mad.sh processes
+3.) Delete your old hosts & plugins folders (save off any plugins you are working on or wish to save). If you
+    have plugins that reference any of the old completed.txt, results.txt, or results_uploads.txt, then will need
+    to be referenced in their new locations: $WorkDir/data/downloads_completed.txt, 
+    $WorkDir/downloads/results.txt, $WorkDir/uploads/results.txt
+4.) Delete _debug, _temp, _flocks in your old folder
+5.) Rename 'downloads' folder to '.inflight' (or delete if empty)
+6.) Rename 'completed' folder to 'downloads'
+5.) Copy all the files in this bundle and move to your MAD locations (overwrite anything there)
+6.) Rename completed.txt to downloads_completed.txt and move to ./data
+7.) If you have LoadPlugins="SkipUrlsInCompletedTxt.sh" in your mad.sh or mad.config, the name has changed to
+    SkipUrlsInDownloadsCompletedTxt.sh, so update it in the LoadPlugins="" line.
+
+Why?
+ * Less clutter everywhere (especially for those who use the WorkDirOverride)
+   - Scripts are in ScriptDir (mad.sh, mad.config, curl_impersonate, plugins, hosts)
+   - Downloads (complete) and all their result data is in downloads folder (except urls.txt, can be anywhere)
+     - Downloads can work in batches.. once done and extracted moved.. the contents can be purged
+   - Downloads that are being transferred / resuming, are in the .inflight folder.
+     - Once batch of downloads is complete.. this can be purged. There shouldn't be anything in it unless
+       something is processing or failed while processing and is waiting for resume.
+   - Uploads (files and all the results data / result-links.txt are in uploads (except uploads.txt);
+     - Uploads can work in batches.. once done and Download Links handled, this can be purged for next batch
+     - Upload tickets are saved in ./uploads/_tickets
+   - All .folders can be hidden, they are also generally short-term (really for processing a batch)
+     - Once processing is complete (no uploads or downloads active, these can be purged)
+   - New data folder is for longer term storage of statdata such as completed uploads / downloads for plugins
+
+* That's it for now..