kittykat/mad
1
2
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

v2024.11.06

Browse source
This commit is contained in:
kittykat 2024-11-08 09:32:54 +00:00
parent 1f87d598a4
commit 808d64768b
118 changed files with 28958 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

84
plugins/!README-plugins.txt Executable file
View file

@ -0,0 +1,84 @@
mad.sh / mad.config (Settings)
------------------------------
# Plugin System: This allows writing a script file that will be loaded / overlaid upon launch, allowing users to
# override functions already in mad.sh with additional functionality
# (ie. unzip after successful download, skip urls that are already in completed.txt)
## Additional Hook Functions added for easy of overriding:
# * OnLoad(): Occurs after load mad.config / load plugins (prior to processing).
# * BeginProcessing(): Occurs immediately after beginning processin of urls.txt (loops with Catnaps).
# * PreProcessUrl(): occurs immediately after reading in an unprocessed url (^http) line to process.
# * PostSuccessfulDownload(): occurs after a download success (is marked #OK# in the urls.txt).
# * PostFailedDownload(): occurs after a download fails (is marked #FAIL# in the urls.txt).
# * PostFailRetryDownload(): occurs after a download fails with a retry (is marked #RETRY# in the urls.txt).
# * DoneProcessingAllUrls(): occurs after all the urls have finished processing (no flocks or other terms downloading).
# * PostSuccessfulUpload(): occurs after an upload success (after upload completed ticket is created in ./completed/).
# * PostFailedUpload(): occurs after an upload fails definitively -- #FAIL# in the uploads_processed.txt
# * PostFailRetryUpload(): occurs after an upload fails with a retry (network drop, unexpected result)
# * DoneProcessingAllUploads: occurs after alll the files have finished processing
# @Default= <blank> (load no plugins)
LoadPlugins=""
* This takes a comma separated list of filenames in the plugins folder if you wish to load / use them (case-sensitive).
ie. LoadPlugins="SkipOkUrlsInResultsTxt.sh,CatnapUntilCtrlC.sh"
Folder Location:
----------------
/plugins folder should reside in the ScriptDir (ame directory as mad.sh script)
Plugin Info:
-------------
Plugins are just scripts that contain functions that you wish to use to add functionality or to
override (code to do other stuff or more stuff). I have added 10 "hook" functions in optimal
places that allows performing additional actions during the processing of urls in mad.sh.
The easist way to start making a plugin to do something you want is to copy and past the current function in a script,
and then code in your additions. These will not be overwritten by a mad.sh script update.
Notes:
------
* Overriding one of the 5 main hooks will always be safe.
- v1 Hooks (using the mad.sh function name) works, as I intend to keep functionality updates elsewhere,
though this is no longer the recommended way as it will not allow other plugins to use the hook.
ie. OnLoad() {}
- v2 Hooks do not care as they will all be loaded and ran in the hook.
ie. OnLoad_<MyUniqueFilename>() {}
* If you override a function in mad.sh, you may need to re-implement its core functionality in your override (v1 hook).
* Prefer using local variables ie. (local datestamp=) instead of variables, and/or name them uniquely as plugin functions
have access to all mad.sh vars and functions, and can overwrite variables (no bueno). I recommend prepending your
pluginname, filename, or an abreviation of (ie. local ppunzipUrlName=${remote_url}, or anything of the likes).
Unless your intent is to overwrite a mad.sh variable (ie. WorkDir -- then you could hook OnLoad_<myfilename>() {}
and set it (WorkDir="/some/path/")
!!----------- Important -----------!!
* If you use a v1 Hook (override a function name in mad.sh), no other plugins will have access to that hook.
! Feel free to share any helpful plugins you make with the group. I will review anything on the thread to verify it,
and possibly add it to the official bundle with plugins if approved.
=== Some ideas to get you started ===
---------------------------------------
* Unzip files after successfully downloaded (PostSuccessfulDownload or PostProcessUrl or DoneProcessingAllUrls if
you wish to perform unzipping at the end of processing everything.
* Auto ./mad.sh reset any #RETRY# failed downloads after the entire urls.txt is processed and run again.
* Shutdown computer after completion of downloads (or beep, or whatever) -- DoneProcessingAllUrls.
* Create your own log of completions or urls and status to a DB
* Start another process on load of script OnLoad() hook (will likely want to check process isn't already running,
as multi terminal mad.sh script will each run the OnLoad() hook -- $isRunning=$(ps -ef|grep 'procname')
- This could auto start ./mad.sh clipmon to monitor for copied urls.
* Keep track of every url you have already downloaded and skip processing again
* Add special url fixing to bad urls copied / pasted -- ie. http://1fichier.com/?1234567890abcdefg?download
* Rename / move files out to collection based off of keywords or foldernames
* Sleep indefinitely after processing urls.txt -- waiting for urls.txt to be updated with more urls
* Warn if drive space < specific amount
* Process unzipped downloads and rename folder with (#p, #v)
* Process unzipped downloads and auto create thumbs
* ... you get the just of it

93
plugins/AutoResetAndRetryDownloads.sh Executable file
View file

@ -0,0 +1,93 @@
#! Name: AutoResetAndRetryDownloads.sh
#! Author: kittykat
#! Version: 2024.09.27
#! Desc: Runs mad.sh reset after processing all urls and then relaunches MAD
#! Usage: Edit LoadPlugin="" line in mad.sh or mad.config
#! LoadPlugin="AutoResetAndRetryDownloads.sh"
#!
#! Plugin System: Additional Hook Functions added for easy of overriding:
#!
#! To Hook a Function, your function must begin with the hookable function name, and be unique
#! Best practice: HooKFuncName_<yourfilename>
#! ie.
#! PostSuccessfulDownload_myfilename() {
#! }
#!
#! Available Hook Functions:
#! -------------------------
#! * OnLoad(): Occurs after load mad.config / load plugins (prior to processing).
#! * BeginProcessing(): Occurs immediately after beginning processin of urls.txt (loops with Catnaps).
#! * PreProcessUrl(): occurs immediately after reading in an unprocessed url (^http) line to process.
#! * PostSuccessfulDownload(): occurs after a download success (is marked #OK# in the urls.txt).
#! * PostFailedDownload(): occurs after a download fails (is marked #FAIL# in the urls.txt).
#! * PostFailRetryDownload(): occurs after a download fails with a retry (is marked #RETRY# in the urls.txt).
#! * DoneProcessingAllUrls(): occurs after all the urls have finished processing (no flocks or other terms downloading).
#! * PostSuccessfulUpload(): occurs after an upload success (after upload completed ticket is created in ./downloads/).
#! * PostFailedUpload(): occurs after an upload fails definitively -- #FAIL# in the temp_upload_handler.txt
#! * PostFailRetryUpload(): occurs after an upload fails with a retry (network drop, unexpected result)
#! * DoneProcessingAllUploads: occurs after alll the files have finished processing
#!
DoneProcessingAllUrls_AutoResetAndRetryDownloads() {
local plugName='AutoResetAndRetryDownloads'
local plugFunc='DoneProcessingAllUrls_AutoResetAndRetryDownloads'
if [ "${DebugPluginsEnabled}" == "true" ]; then
echo -e "[${PINK}DEBUG${NC}]: Running ${PINK}$plugFunc${NC} in ${BLUE}$plugName${NC} ...${NC}"
fi
local pInputFile="$1"
arard_ticketactivetimeout=60
arard_ticketexpirytimeout=540
if [ "$LoopThroughFileUntilComplete" == "true" ] && [ ! -z $CatnapDuration ] && ((CatnapDuration > 0)); then
arard_ticketactivetimeout=$(( arard_ticketactivetimeout + (CatnapDuration * 60) ))
arard_ticketexpirytimeout=$(( arard_ticketexpirytimeout + (CatnapDuration * 60) ))
fi
if grep -Eqi '^#(http.*#RETRY#|direct=http.*#RETRY#).*$' "$pInputFile" ; then
if [ ! -f "${WorkDir}/.temp/AutoResetAndRetryDownloadsarard_startTime.txt" ]; then
echo $(date +%s) > "${WorkDir}/.temp/AutoResetAndRetryDownloadsarard_startTime.txt"
else
arard_startTime=$(cat "${WorkDir}/.temp/AutoResetAndRetryDownloadsarard_startTime.txt")
arard_currTime=$(date +%s)
arard_durationseconds=$((arard_currTime - arard_startTime))
if ((arard_durationseconds > arard_ticketexpirytimeout)); then
if [ "${DebugPluginsEnabled}" == "true" ]; then
echo -e "[${PINK}DEBUG${NC}]: Ticket expiry occurred ($arard_durationseconds > $arard_ticketexpirytimeout)..${NC}"
fi
echo $(date +%s) > "${WorkDir}/.temp/AutoResetAndRetryDownloadsarard_startTime.txt"
arard_startTime=$(cat "${WorkDir}/.temp/AutoResetAndRetryDownloadsarard_startTime.txt")
arard_currTime=$(date +%s)
arard_durationseconds=$((arard_currTime - arard_startTime))
elif ((arard_durationseconds > arard_ticketactivetimeout)); then
if [ "${DebugPluginsEnabled}" == "true" ]; then
echo -e "[${PINK}DEBUG${NC}]: Ticket inactive ($arard_durationseconds > $arard_ticketactivetimeout)..${NC}"
fi
for ((a=1; a<=2; a++)); do
printf "\033[1A\r"
printf "\33[2K\r"
done
echo -e "${NC}"
echo -e "${YELLOW}Reset&Retry${NC}: Done processing... ${BLUE}$((arard_durationseconds/60))${NC} min(s). Expiry ${PINK}$(( (arard_ticketexpirytimeout - arard_durationseconds)/60 ))${NC} min(s).${NC}"
return 0
fi
if [ "${DebugPluginsEnabled}" == "true" ]; then
echo -e "[${PINK}DEBUG${NC}]: Ticket active ($arard_durationseconds)..${NC}"
fi
for ((a=1; a<=2; a++)); do
printf "\033[1A\r"
printf "\33[2K\r"
done
echo -e "${NC}"
echo -e "${YELLOW}Reset&Retry${NC}: Dozing ${BLUE}1${NC} min. Processing for ${BLUE}$((arard_durationseconds/60))${NC} of ${BLUE}$((arard_ticketactivetimeout/60))${NC} min(s)${NC}. Expiry ${PINK}$(( (arard_ticketexpirytimeout - arard_durationseconds)/60 ))${NC} min(s).${NC}"
sleep 1m
fi
if [ ! -f "${WorkDir}/.temp/AutoResetAndRetryDownloadsCnt.lk" ]; then
touch "${WorkDir}/.temp/AutoResetAndRetryDownloadsCnt.lk"
trap "rm -f ${WorkDir}/.temp/AutoResetAndRetryDownloadsCnt.lk; echo ""; tput cnorm; exit" 0 1 2 3 6 15
madReset "$pInputFile" "true" "false" "false" "true"
rm -f "${WorkDir}/.temp/AutoResetAndRetryDownloadsCnt.lk"
fi
qChkLineCount=$(grep -Ei '^(http|direct=http)' "${pInputFile}" | wc -l | awk '{ print $1 }')
if ((qChkLineCount > 0)) ; then
ReloadScript "$@"
fi
fi
return 0
}

57
plugins/CatnapCtrlC.sh Executable file
View file

@ -0,0 +1,57 @@
#! Name: CatnapCtrlC.sh
#! Author: kittykat
#! Version: 2024.09.21
#! Desc: Keeps mad.sh running until Ctrl-C, waiting for urls.txt updates
#! Usage: Edit LoadPlugin="" line in mad.sh or mad.config
#! LoadPlugin="CatnapCtrlC.sh"
#! Notes:
#! * Return 0 (true), continue script (exit)
#! * Return 1 (false), return to processing the beginning of urls.txt -- would be used with a sleep
#! to continue to run indefinitely until Ctrl-C is pressed, waiting for more urls
#!
#! Plugin System: Additional Hook Functions added for easy of overriding:
#!
#! To Hook a Function, your function must begin with the hookable function name, and be unique
#! Best practice: HooKFuncName_<yourfilename>
#! ie.
#! PostSuccessfulDownload_myfilename() {
#! }
#!
#! Available Hook Functions:
#! -------------------------
#! * OnLoad(): Occurs after load mad.config / load plugins (prior to processing).
#! * BeginProcessing(): Occurs immediately after beginning processin of urls.txt (loops with Catnaps).
#! * PreProcessUrl(): occurs immediately after reading in an unprocessed url (^http) line to process.
#! * PostSuccessfulDownload(): occurs after a download success (is marked #OK# in the urls.txt).
#! * PostFailedDownload(): occurs after a download fails (is marked #FAIL# in the urls.txt).
#! * PostFailRetryDownload(): occurs after a download fails with a retry (is marked #RETRY# in the urls.txt).
#! * DoneProcessingAllUrls(): occurs after all the urls have finished processing (no flocks or other terms downloading).
#! * PostSuccessfulUpload(): occurs after an upload success (after upload completed ticket is created in ./downloads/).
#! * PostFailedUpload(): occurs after an upload fails definitively -- #FAIL# in the temp_upload_handler.txt
#! * PostFailRetryUpload(): occurs after an upload fails with a retry (network drop, unexpected result)
#! * DoneProcessingAllUploads: occurs after alll the files have finished processing
#!
DoneProcessingAllUrls_CatnapCtrlC() {
local plugName='CatnapCtrlC'
local plugFunc='DoneProcessingAllUrls_CatnapCtrlC'
if [ "${DebugPluginsEnabled}" == "true" ]; then
echo -e "[${PINK}DEBUG${NC}]: Running ${PINK}$plugFunc${NC} in ${BLUE}$plugName${NC} ...${NC}"
fi
local pInputFile="$1"
CatnapCount=$((CatnapCount + 1))
if ((CatnapCount > 2)); then
for ((a=1; a<=2; a++)); do
printf "\033[1A\r"
printf "\33[2K\r"
done
elif ((CatnapCount > 1)); then
for ((a=1; a<=5; a++)); do
printf "\033[1A\r"
printf "\33[2K\r"
done
fi
echo -e "${NC}"
echo -e "${YELLOW}Ctrl-C Catnapping${NC} for ${CatnapDuration} mins and waiting for urls in ${YELLOW}${pInputFile}${NC}... ${YELLOW}zZzZzZ${NC} ${BLUE}x$CatnapCount${NC}"
sleep ${CatnapDuration}m
return 1
}

90
plugins/SkipUrlsInDownloadsCompletedTxt.sh Executable file
View file

@ -0,0 +1,90 @@
#! Name: SkipUrlsInDownloadsCompletedTxt.sh
#! Author: kittykat / beautfar
#! Version: 2024.11.06
#! Desc: Skips urls that already exist in downloads_completed.txt
#! This is similar to the SkipOkUrlsInResultsTxt.sh script, except it is better --
#! - Creates "./data/downloads_completed.txt with only completed urls (less noise / clutter).
#! - Stores more detail including the date
#! Usage: Edit LoadPlugin="" line in mad.sh or mad.config
#! LoadPlugin="SkipUrlsInDownloadsCompletedTxt.sh"
#!
#! Notes:
#! * Return 0 (true), continue processing this url
#! * Return 1 (false), stop processing this url and move to next
#!
#! Plugin System: Additional Hook Functions added for easy of overriding:
#!
#! To Hook a Function, your function must begin with the hookable function name, and be unique
#! Best practice: HooKFuncName_<yourfilename>
#! ie.
#! PostSuccessfulDownload_myfilename() {
#! }
#!
#! Available Hook Functions:
#! -------------------------
#! * OnLoad(): Occurs after load mad.config / load plugins (prior to processing).
#! * BeginProcessing(): Occurs immediately after beginning processin of urls.txt (loops with Catnaps).
#! * PreProcessUrl(): occurs immediately after reading in an unprocessed url (^http) line to process.
#! * PostSuccessfulDownload(): occurs after a download success (is marked #OK# in the urls.txt).
#! * PostFailedDownload(): occurs after a download fails (is marked #FAIL# in the urls.txt).
#! * PostFailRetryDownload(): occurs after a download fails with a retry (is marked #RETRY# in the urls.txt).
#! * DoneProcessingAllUrls(): occurs after all the urls have finished processing (no flocks or other terms downloading).
#! * PostSuccessfulUpload(): occurs after an upload success (after upload completed ticket is created in ./downloads/).
#! * PostFailedUpload(): occurs after an upload fails definitively -- #FAIL# in the temp_upload_handler.txt
#! * PostFailRetryUpload(): occurs after an upload fails with a retry (network drop, unexpected result)
#! * DoneProcessingAllUploads: occurs after alll the files have finished processing
#!
PreProcessUrl_SkipUrlsInDownloadsCompletedTxt() {
local plugName='SkipUrlsInDownloadsCompletedTxt'
local plugFunc='PreProcessUrl_SkipUrlsInDownloadsCompletedTxt'
if [ "${DebugPluginsEnabled}" == "true" ]; then
echo -e "[${PINK}DEBUG${NC}]: Running ${PINK}$plugFunc${NC} in ${BLUE}$plugName${NC} ...${NC}"
fi
local pFullLine="$1"
local pUrlOnly="${1%%\ *}"
if grep -Eqi '|' <<< "${pUrlOnly}" ; then
pUrlOnly="${pUrlOnly%%\|*}"
fi
if grep -Eqi '^direct=' <<< "${pUrlOnly}" ; then
pUrlOnly=${pUrlOnly/direct=/}
fi
if [ -z "$pFullLine" ]; then
return 0
fi
local tUrl="${pUrlOnly##*\:\/\/}"
if [ "${DebugPluginsEnabled}" == "true" ]; then
echo -e "[${PINK}DEBUG${NC}]: pFullLine: $pFullLine${NC}"
echo -e "[${PINK}DEBUG${NC}]: pUrlOnly: $pUrlOnly${NC}"
echo -e "[${PINK}DEBUG${NC}]: tUrl: $tUrl${NC}"
fi
if [ ! -z "$tUrl" ]; then
linematch=""
if [ -f "${WorkDir}/data/downloads_completed.txt" ]; then
linematch=$(grep -Eni "[OK].*url:.*${tUrl}.*\$" "${WorkDir}/data/downloads_completed.txt")
else
if [ "${DebugPluginsEnabled}" == "true" ]; then
echo -e "[${PINK}DEBUG${NC}]: File not found: ${BLUE}${WorkDir}/data/downloads_completed.txt${NC}"
fi
fi
if [ ! -z "$linematch" ] ; then
echo -e ""
echo -e "${GREEN}$pUrlOnly${NC} has already been downloaded in ${GREEN}downloads_completed.txt${NC}"
echo -e "${BLUE}line${NC}:${NC}"
echo -e "${linematch//, /\\n}"
successDownloadExists "$pUrlOnly" "URL found in ./data/downloads_completed.txt (line#: ${linematch%%:*})"
return 1
fi
linematch=""
if [ -f "${WorkDir}/data/downloads_completed.txt" ]; then
linematch=$(grep -Eni "[REMOVED].*url:.*${tUrl}.*\$" "${WorkDir}/data/downloads_completed.txt")
fi
if [ ! -z "$linematch" ] ; then
echo -e ""
echo -e "${RED}$pUrlOnly${NC} already mareked removed in ${GREEN}downloads_completed.txt${NC}"
echo -e "${BLUE}line${NC}: ${linematch//, /\\n}"
removedDownload "$pUrlOnly" "URL found in ./data/downloads_completed.txt (line#: ${linematch%%:*})"
return 1
fi
fi
return 0
}

67
plugins/examples/ExampleAddNewFuncAndCallOnSuccessfulDownload.sh Executable file
View file

@ -0,0 +1,67 @@
#! Name: ExampleAddNewFuncAndCallOnSuccessfulDownload.sh
#! Author: kittykat
#! Version: 2024.09.21
#! Desc: Add a new function to MAD and call it whenever a file is successfully downloaded
#! Usage: Edit LoadPlugin="" line in mad.sh or mad.config
#! LoadPlugin="ExampleAddNewFuncAndCallOnSuccessfulDownload.sh"
#!
#! Notes: Not that this would be used as such (performing a sha256 hash of file would be added directly to the
#! desired hook -- PostSuccessfulDownload(). This is just an example to show that creating your own
#! function and then calling it is possible.
#!
#! * This could be used to add new host urls, and process them:
#! PreProcessUrl_mysha256() : parse the line and determine if it is a supported host, call new functions
#!
#! Plugin System: Additional Hook Functions added for easy of overriding:
#!
#! To Hook a Function, your function must begin with the hookable function name, and be unique
#! Best practice: HooKFuncName_<yourfilename>
#! ie.
#! PostSuccessfulDownload_myfilename() {
#! }
#!
#! Available Hook Functions:
#! -------------------------
#! * OnLoad(): Occurs after load mad.config / load plugins (prior to processing).
#! * BeginProcessing(): Occurs immediately after beginning processin of urls.txt (loops with Catnaps).
#! * PreProcessUrl(): occurs immediately after reading in an unprocessed url (^http) line to process.
#! * PostSuccessfulDownload(): occurs after a download success (is marked #OK# in the urls.txt).
#! * PostFailedDownload(): occurs after a download fails (is marked #FAIL# in the urls.txt).
#! * PostFailRetryDownload(): occurs after a download fails with a retry (is marked #RETRY# in the urls.txt).
#! * DoneProcessingAllUrls(): occurs after all the urls have finished processing (no flocks or other terms downloading).
#! * PostSuccessfulUpload(): occurs after an upload success (after upload completed ticket is created in ./downloads/).
#! * PostFailedUpload(): occurs after an upload fails definitively -- #FAIL# in the temp_upload_handler.txt
#! * PostFailRetryUpload(): occurs after an upload fails with a retry (network drop, unexpected result)
#! * DoneProcessingAllUploads: occurs after alll the files have finished processing
#!
#! ----------- Hooks ----------------------------
#!
#! Hook PostSuccessfulDownload() function
PostSuccessfulDownload_mysha256() {
local plugName='ExampleAddNewFuncAndCallOnSuccessfulDownload'
local plugFunc='PostSuccessfulDownload_mysha256'
if [ "${DebugAllEnabled}" == "true" ]; then
echo -e "[${PINK}DEBUG${NC}]: Running ${PINK}$plugFunc${NC} in ${BLUE}$plugName${NC} ...${NC}"
fi
local pRemoteUrl="$1"
local pFilePath="$2"
local pFileName="$3"
local pFolderName="$4"
local pFileSize="$5"
local fileHash=$(hookHashFile "$pFilePath")
echo -e "${BLUE}File${NC}: $pFileName"
echo -e "${BLUE}sha256${NC}: $fileHash"
return 0
}
#! ---------- New User Defined Functions --------------------
#! Define a new function to hash a file
#! This will be loaded in the OnLoad() hook in mad.sh, and will become an available function to use after OnLoad()
hookHashFile() {
local plugName='ExampleAddNewFuncAndCallOnSuccessfulDownload'
local plugFunc='hookHashFile'
if [ "${DebugPluginsEnabled}" == "true" ]; then
echo -e "[${PINK}DEBUG${NC}]: Running ${PINK}$plugFunc${NC} in ${BLUE}$plugName${NC} ...${NC}"
fi
local pFilePath="$1"
echo $(sha256sum "$pFilePath" | awk '{ print $1 }')
}

163
plugins/examples/ExamplesMainHooks.sh Executable file
View file

@ -0,0 +1,163 @@
#! Name: ExamplesMainHooks.sh
#! Author: kittykat
#! Version: 2024.09.21
#! Desc: Just some examples on easy plugins...
#! Usage: Edit LoadPlugin="" line in mad.sh or mad.config
#! LoadPlugin="ExamplesMainHooks.sh"
#!
#!
#! Plugin System: Additional Hook Functions added for easy of overriding:
#!
#! To Hook a Function, your function must begin with the hookable function name, and be unique
#! Best practice: HooKFuncName_<yourfilename>
#! ie.
#! PostSuccessfulDownload_myfilename() {
#! }
#!
#! Available Hook Functions:
#! -------------------------
#! * OnLoad(): Occurs after load mad.config / load plugins (prior to processing).
#! * BeginProcessing(): Occurs immediately after beginning processin of urls.txt (loops with Catnaps).
#! * PreProcessUrl(): occurs immediately after reading in an unprocessed url (^http) line to process.
#! * PostSuccessfulDownload(): occurs after a download success (is marked #OK# in the urls.txt).
#! * PostFailedDownload(): occurs after a download fails (is marked #FAIL# in the urls.txt).
#! * PostFailRetryDownload(): occurs after a download fails with a retry (is marked #RETRY# in the urls.txt).
#! * DoneProcessingAllUrls(): occurs after all the urls have finished processing (no flocks or other terms downloading).
#! * PostSuccessfulUpload(): occurs after an upload success (after upload completed ticket is created in ./downloads/).
#! * PostFailedUpload(): occurs after an upload fails definitively (host not responding, missing required response)
#! * PostFailRetryUpload(): occurs after an upload fails with a retry (network drop, unexpected result)
#! * DoneProcessingAllUploads: occurs after alll the files have finished processing
#!
#! ----------- Hooks ----------------------------
#!
OnLoad_ExamplesMainHooks() {
local plugName='ExamplesMainHooks'
local plugFunc='OnLoad_ExamplesMainHooks'
if [ "${DebugPluginsEnabled}" == "true" ]; then
echo -e "[${PINK}DEBUG${NC}]: Running ${PINK}$plugFunc${NC} in ${BLUE}$plugName${NC} ...${NC}"
fi
local pScriptSourceDir="$1"
local pScriptArguments="$2"
return 0
}
BeginProcessing_ExamplesMainHooks() {
local plugName='ExamplesMainHooks'
local plugFunc='BeginProcessing_ExamplesMainHooks'
if [ "${DebugPluginsEnabled}" == "true" ]; then
echo -e "[${PINK}DEBUG${NC}]: Running ${PINK}$plugFunc${NC} in ${BLUE}$plugName${NC} ...${NC}"
fi
local pInputFile="$1"
return 0
}
PreProcessUrl_ExamplesMainHooks() {
local plugName='ExamplesMainHooks'
local plugFunc='PreProcessUrl_ExamplesMainHooks'
if [ "${DebugPluginsEnabled}" == "true" ]; then
echo -e "[${PINK}DEBUG${NC}]: Running ${PINK}$plugFunc${NC} in ${BLUE}$plugName${NC} ...${NC}"
fi
local pFullLine="$1"
local pUrlOnly="${1%%\ *}"
if grep -Eqi '|' <<< "${pUrlOnly}" ; then
pUrlOnly="${pUrlOnly%%\|*}"
fi
if grep -Eqi '^direct=' <<< "${pUrlOnly}" ; then
pUrlOnly="${pUrlOnly%%\=*}"
fi
return 0
return 1
}
PostSuccessfulDownload_ExamplesMainHooks() {
local plugName='ExamplesMainHooks'
local plugFunc='PostSuccessfulDownload_ExamplesMainHooks'
if [ "${DebugPluginsEnabled}" == "true" ]; then
echo -e "[${PINK}DEBUG${NC}]: Running ${PINK}$plugFunc${NC} in ${BLUE}$plugName${NC} ...${NC}"
fi
local pRemoteUrl="$1"
local pFilePath="$2"
local pFileName="$3"
local pFolderName="$4"
local pFileSize="$5"
return 0
}
PostFailedDownload_ExamplesMainHooks() {
local plugName='ExamplesMainHooks'
local plugFunc='PostFailedDownload_ExamplesMainHooks'
if [ "${DebugPluginsEnabled}" == "true" ]; then
echo -e "[${PINK}DEBUG${NC}]: Running ${PINK}$plugFunc${NC} in ${BLUE}$plugName${NC} ...${NC}"
fi
local pRemoteUrl="$1"
local pMessage1="$2"
local pMessage2="$3"
return 0
}
PostFailRetryDownload_ExamplesMainHooks() {
local plugName='ExamplesMainHooks'
local plugFunc='PostFailRetryDownload_ExamplesMainHooks'
if [ "${DebugPluginsEnabled}" == "true" ]; then
echo -e "[${PINK}DEBUG${NC}]: Running ${PINK}$plugFunc${NC} in ${BLUE}$plugName${NC} ...${NC}"
fi
local pRemoteUrl="$1"
local pMessage1="$2"
local pMessage2="$3"
return 0
}
DoneProcessingAllUrls_ExamplesMainHooks() {
local plugName='ExamplesMainHooks'
local plugFunc='DoneProcessingAllUrls_ExamplesMainHooks'
if [ "${DebugPluginsEnabled}" == "true" ]; then
echo -e "[${PINK}DEBUG${NC}]: Running ${PINK}$plugFunc${NC} in ${BLUE}$plugName${NC} ...${NC}"
fi
local pInputFile="$1"
madReset "$pInputFile" "true" "false" "false" "true"
qChkLineCount=$(grep -Ei '^(http|direct=http)' "${pInputFile}" | wc -l | awk '{ print $1 }')
if ((qChkLineCount > 0)) ; then
ReloadScript "$@"
fi
return 0
}
PostSuccessfulUpload_ExamplesMainHooks() {
local plugName='ExamplesMainHooks'
local plugFunc='PostSuccessfulUpload_ExamplesMainHooks'
if [ "${DebugPluginsEnabled}" == "true" ]; then
echo -e "[${PINK}DEBUG${NC}]: Running ${PINK}$plugFunc${NC} in ${BLUE}$plugName${NC} ...${NC}"
fi
local pFilePath="$1"
local pHostCode="$2"
local pFileName="$3"
local pFileSize="$4"
local pDownloadLink="$5"
return 0
}
PostFailedUpload_ExamplesMainHooks() {
local plugName='ExamplesMainHooks'
local plugFunc='PostFailedUpload_ExamplesMainHooks'
if [ "${DebugPluginsEnabled}" == "true" ]; then
echo -e "[${PINK}DEBUG${NC}]: Running ${PINK}$plugFunc${NC} in ${BLUE}$plugName${NC} ...${NC}"
fi
local pFilePath="$1"
local pHostCode="$2"
local pMessage1="$3"
local pMessage2="$4"
return 0
}
PostFailRetryUpload_ExamplesMainHooks() {
local plugName='ExamplesMainHooks'
local plugFunc='PostFailRetryUpload_ExamplesMainHooks'
if [ "${DebugPluginsEnabled}" == "true" ]; then
echo -e "[${PINK}DEBUG${NC}]: Running ${PINK}$plugFunc${NC} in ${BLUE}$plugName${NC} ...${NC}"
fi
local pFilePath="$1"
local pHostCode="$2"
local pMessage1="$3"
local pMessage2="$4"
return 0
}
DoneProcessingAllUploads_ExamplesMainHooks() {
local plugName='ExamplesMainHooks'
local plugFunc='DoneProcessingAllUploads_ExamplesMainHooks'
if [ "${DebugPluginsEnabled}" == "true" ]; then
echo -e "[${PINK}DEBUG${NC}]: Running ${PINK}$plugFunc${NC} in ${BLUE}$plugName${NC} ...${NC}"
fi
local pInputFile="$1"
return 0
}

7
plugins/ocr/tessdata/HowTo - tesseract traineddata.txt Executable file
View file

@ -0,0 +1,7 @@
How to setup tesseract-ocr traineddata:
* The eng_best.traineddata can be downloaded from:
https://github.com/tesseract-ocr/tessdata_best/raw/main/eng.traineddata
(SHA256: 8280AED0782FE27257A68EA10FE7EF324CA0F8D85BD2FD145D1C2B560BCB66BA)
* And then extracted to ./plugins/ocr/tessdata/ folder (15,400,601 bytes)

127
plugins/ocr_captcha.sh Executable file
View file

@ -0,0 +1,127 @@
#! Name: ocr_captcha.sh
#! Author: kittykat
#! Version: 2024.10.13
#! Desc: Script to extract captcha from image using tesseract-ocr and imagemagick
#! Usage: Edit LoadPlugin="" line in mad.sh or mad.config
#! LoadPlugin="ocr_captcha.sh"
#!
#! Dependencies
#! * imagemagick: resize / enhance / convert image
#! (sudo apt-get install imagemagick)
#! * tesseract-ocr: Extract text from image
#! (sudo apt-get install tesseract-ocr)
#! - The eng_best.traineddata is included in ./plugins/ocr/tessdata/ folder (15,400,601 bytes)
#! - It can otherwise be downloaded and placed in the folder from:
#! https://github.com/tesseract-ocr/tessdata_best/raw/main/eng.traineddata
#! (SHA256: 8280AED0782FE27257A68EA10FE7EF324CA0F8D85BD2FD145D1C2B560BCB66BA)
#!
#! Notes:
#! * Return 0 (true), successfully processed image
#! * Return 1 (false), failed to process image
#!
#! Available Hook Functions:
#! -------------------------
#! * OnLoad(): Occurs after load mad.config / load plugins (prior to processing).
#! * BeginProcessing(): Occurs immediately after beginning processin of urls.txt (loops with Catnaps).
#! * PreProcessUrl(): occurs immediately after reading in an unprocessed url (^http) line to process.
#! * PostSuccessfulDownload(): occurs after a download success (is marked #OK# in the urls.txt).
#! * PostFailedDownload(): occurs after a download fails (is marked #FAIL# in the urls.txt).
#! * PostFailRetryDownload(): occurs after a download fails with a retry (is marked #RETRY# in the urls.txt).
#! * DoneProcessingAllUrls(): occurs after all the urls have finished processing (no flocks or other terms downloading).
#! * PostSuccessfulUpload(): occurs after an upload success (after upload completed ticket is created in ./downloads/).
#! * PostFailedUpload(): occurs after an upload fails definitively -- #FAIL# in the temp_upload_handler.txt
#! * PostFailRetryUpload(): occurs after an upload fails with a retry (network drop, unexpected result)
#! * DoneProcessingAllUploads: occurs after alll the files have finished processing
#!
CaptchaOcrImage() {
local plugName='ocr_captcha'
local plugFunc='CaptchaOcrImage'
if [ "${DebugPluginsEnabled}" == "true" ]; then
echo -e "[${PINK}DEBUG${NC}]: Running ${PINK}$plugFunc${NC} in ${BLUE}$plugName${NC} ...${NC}"
fi
DEPENDENCIES=(tesseract convert)
for DEPENDENCY in ${DEPENDENCIES[@]} ; do
if [ -z $(which $DEPENDENCY) ] ; then
if [ "$DEPENDENCY" == "convert" ]; then
echo "imagemagick not installed. Aborting"
else
echo "$DEPENDENCY not installed. Aborting"
fi
return 1
fi
done
TESSERACT_CMD=$(which tesseract)
export TESSDATA_PREFIX="${ScriptDir}/plugins/ocr/tessdata"
captcha_image_filepath="$1"
tessdata_type="$2"
imagemagick_extra_params="$3"
local captcha_image_filename="${captcha_image_filepath##*/}"
if [ ! -f "$captcha_image_filepath" ]; then
echo -e "Image not found."
return 1
fi
mkdir -p "$WorkDir/.temp"
IMGtemp="$WorkDir/.temp/`date +%y%m%d-%H%M%S`_$captcha_image_filename.tif"
convert "$captcha_image_filepath" -units PixelsPerInch -respect-parenthesis \( -compress LZW -resample 900 -density 300 -bordercolor black -border 1 -trim +repage -fill white -draw "color 0,0 floodfill" -alpha off -shave 1x2 \) \( -bordercolor black -border 2 -fill white -draw "color 0,0 floodfill" -alpha off -shave 0x1 +repage \) -antialias -sharpen 0x3 "$IMGtemp"
if grep -Eqi "Deskew" <<< "$imagemagick_extra_params" ; then
convert "$IMGtemp" -deskew 10 "IMGtemp"
fi
if grep -Eqi "ContrastStretch_1x60" <<< "$imagemagick_extra_params" ; then
convert "$IMGtemp" -contrast-stretch 1%x60% "$IMGtemp"
elif grep -Eqi "ContrastStretch_1x70" <<< "$imagemagick_extra_params" ; then
convert "$IMGtemp" -contrast-stretch 1%x70% "$IMGtemp"
elif grep -Eqi "ContrastStretch_1x75" <<< "$imagemagick_extra_params" ; then
convert "$IMGtemp" -contrast-stretch 1%x75% "$IMGtemp"
elif grep -Eqi "ContrastStretch_1x80" <<< "$imagemagick_extra_params" ; then
convert "$IMGtemp" -contrast-stretch 1%x80% "$IMGtemp"
elif grep -Eqi "ContrastStretch_1x85" <<< "$imagemagick_extra_params" ; then
convert "$IMGtemp" -contrast-stretch 1%x85% "$IMGtemp"
elif grep -Eqi "ContrastStretch_1x90" <<< "$imagemagick_extra_params" ; then
convert "$IMGtemp" -contrast-stretch 1%x90% "$IMGtemp"
elif grep -Eqi "ContrastStretch_1x95" <<< "$imagemagick_extra_params" ; then
convert "$IMGtemp" -contrast-stretch 1%x95% "$IMGtemp"
elif grep -Eqi "ContrastStretch_5x80" <<< "$imagemagick_extra_params" ; then
convert "$IMGtemp" -contrast-stretch 5%x80% "$IMGtemp"
elif grep -Eqi "ContrastStretch_5x85" <<< "$imagemagick_extra_params" ; then
convert "$IMGtemp" -contrast-stretch 5%x85% "$IMGtemp"
elif grep -Eqi "ContrastStretch_5x90" <<< "$imagemagick_extra_params" ; then
convert "$IMGtemp" -contrast-stretch 5%x90% "$IMGtemp"
elif grep -Eqi "ContrastStretch_5x95" <<< "$imagemagick_extra_params" ; then
convert "$IMGtemp" -contrast-stretch 5%x95% "$IMGtemp"
elif grep -Eqi "ContrastStretch_10x90" <<< "$imagemagick_extra_params" ; then
convert "$IMGtemp" -contrast-stretch 10%x90% "$IMGtemp"
elif grep -Eqi "ContrastStretch_10x95" <<< "$imagemagick_extra_params" ; then
convert "$IMGtemp" -contrast-stretch 10%x95% "$IMGtemp"
fi
if grep -Eqi "Brightness_120" <<< "$imagemagick_extra_params" ; then
convert "$IMGtemp" -modulate 120 "$IMGtemp"
elif grep -Eqi "Brightness_130" <<< "$imagemagick_extra_params" ; then
convert "$IMGtemp" -modulate 130 "$IMGtemp"
elif grep -Eqi "Brightness_135" <<< "$imagemagick_extra_params" ; then
convert "$IMGtemp" -modulate 135 "$IMGtemp"
elif grep -Eqi "Brightness_140" <<< "$imagemagick_extra_params" ; then
convert "$IMGtemp" -modulate 150 "$IMGtemp"
elif grep -Eqi "Brightness_145" <<< "$imagemagick_extra_params" ; then
convert "$IMGtemp" -modulate 145 "$IMGtemp"
elif grep -Eqi "Brightness_150" <<< "$imagemagick_extra_params" ; then
convert "$IMGtemp" -modulate 150 "$IMGtemp"
elif grep -Eqi "Brightness_155" <<< "$imagemagick_extra_params" ; then
convert "$IMGtemp" -modulate 155 "$IMGtemp"
elif grep -Eqi "Brightness_160" <<< "$imagemagick_extra_params" ; then
convert "$IMGtemp" -modulate 160 "$IMGtemp"
fi
if grep -Eqi "NUMBERONLY" <<< "$tessdata_type" ; then
captcha=$($TESSERACT_CMD --psm 8 --oem 1 -l eng_best --dpi 70 -c tessedit_char_whitelist=0123456789 "$IMGtemp" stdout | tr -d " " | xargs)
captcha=${captcha//[!0-9]/}
elif grep -Eqi "ALPHAONLY" <<< "$tessdata_type" ; then
captcha=$($TESSERACT_CMD --psm 8 --oem 1 -l eng_best --dpi 70 -c tessedit_char_whitelist=abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ "$IMGtemp" stdout | tr -d " " | xargs)
captcha=${captcha//[!a-zA-Z]/}
elif grep -Eqi "ALPHANUMERIC" <<< "$tessdata_type" ; then
captcha=$($TESSERACT_CMD --psm 8 --oem 1 -l eng_best --dpi 70 -c tessedit_char_whitelist=0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ "$IMGtemp" stdout | tr -d " " | xargs)
captcha=${captcha//[!0-9a-zA-Z]/}
else
captcha=$($TESSERACT_CMD --psm 8 --oem 1 -l eng_best --dpi 70 "$IMGtemp" stdout | tr -d " " | xargs)
fi
rm -f "$IMGtemp"
echo -e "[CAPTCHA_CODE:${captcha}]"
}

74
plugins/pjscloud.sh Executable file
View file

@ -0,0 +1,74 @@
#! Name: pjscloud.sh
#! Author: kittykat
#! Version: 2024.10.14
#! Desc: Wrapper to use PhantomJSCloud to retrieve url response
#! Usage: Edit LoadPlugin="" line in mad.sh or mad.config
#! LoadPlugin="pjscloud.sh"
#! * Optional: Setup additional / personal PJSCloud accounts
#!
#! Notes:
#! * Returns http query response
#!
#! Available Hook Functions:
#! -------------------------
#! * OnLoad(): Occurs after load mad.config / load plugins (prior to processing).
#! * BeginProcessing(): Occurs immediately after beginning processin of urls.txt (loops with Catnaps).
#! * PreProcessUrl(): occurs immediately after reading in an unprocessed url (^http) line to process.
#! * PostSuccessfulDownload(): occurs after a download success (is marked #OK# in the urls.txt).
#! * PostFailedDownload(): occurs after a download fails (is marked #FAIL# in the urls.txt).
#! * PostFailRetryDownload(): occurs after a download fails with a retry (is marked #RETRY# in the urls.txt).
#! * DoneProcessingAllUrls(): occurs after all the urls have finished processing (no flocks or other terms downloading).
#! * PostSuccessfulUpload(): occurs after an upload success (after upload completed ticket is created in ./downloads/).
#! * PostFailedUpload(): occurs after an upload fails definitively -- #FAIL# in the temp_upload_handler.txt
#! * PostFailRetryUpload(): occurs after an upload fails with a retry (network drop, unexpected result)
#! * DoneProcessingAllUploads: occurs after alll the files have finished processing
#!
#!
pjscloud_tor_request() {
local plugName='pjscloud'
local plugFunc='pjscloud_tor_request'
if [ "${DebugPluginsEnabled}" == "true" ]; then
echo -e "[${PINK}DEBUG${NC}]: Running ${PINK}$plugFunc${NC} in ${BLUE}$plugName${NC} ...${NC}"
fi
pjs_targeturl="$1"
pjs_data="$2"
if [ -z "$pjs_targeturl" ]; then
echo "Aborting: No target url specified."
fi
GetRandomPjsKey
if [ ! -z "$pjs_targeturl" ] && [ ! -z "$pjs_data" ] ; then
pjs_urldata='{"url":"'"$pjs_targeturl"'", urlSettings:{"operation":"POST", "data":"'"$pjs_data"'"}, "renderType":"plainText"}'
else
pjs_urldata='{"url":"'"$pjs_targeturl"'", "renderType":"plainText"}'
fi
if [ "${UseTorCurlImpersonate}" == "true" ]; then
response=$("${curl_impersonate[@]}" --proxy "socks5h://${tor_identity}@${TorIp}:${torPort}" \
--connect-timeout ${ConnectTimeout} \
--insecure -k -s \
-H "Content-Type: application/json" \
-H "Expect:" -X POST \
-d "$pjs_urldata" \
"https://PhantomJScloud.com/api/browser/v2/$RandomPjsKey/" & sleep 8s; kill -HUP $! 2>/dev/null)
else
response=$(curl --proxy "socks5h://${tor_identity}@${TorIp}:${torPort}" \
--connect-timeout ${ConnectTimeout} \
--insecure -k -s \
-H "Content-Type: application/json" \
-H "Expect:" -X POST \
-d "$pjs_urldata" \
"https://PhantomJScloud.com/api/browser/v2/$RandomPjsKey/" & sleep 8s; kill -HUP $! 2>/dev/null)
fi
echo -e "[PJSC_RESPONSE_BEGIN]"
echo -e "${response}"
echo -e "[PJSC_RESPONSE_END]"
}
#!
#! --------------- Extra Functions ------------------- #
#!
GetRandomPjsKey() {
arrSize=${#ar_pgsKey[@]}
index=$(($RANDOM % $arrSize))
RandomPjsKey=${ar_pgsKey[$index]}
}
#! Initialize PJSCloud and get a random key
GetRandomPjsKey