This post is for anyone whose scrapers on kodi headless aren't working and is having difficulty setting up scrapers on kodi-headless.
For the benefit of anyone searching this is how I setup scraping on my headless kodi installation as for me it took a few steps and a little bit of time, I'm using non default scrapers which were not installed by default in the headless instal and I encountered a learning curve with various things during the migration. I didn't find much about the scraping side in a headless server when I searched. Naturally nearly everything online refers to the Kodi GUI. I'm using the armhf Kodi container built by
linuxserver.io. If you're using default scrapers (tvdb, moviedb) in your old install just copying sources.xml will work for most people.
I wanted to migrate to a headless kodi server from a working Kodi/MySQL install and then set up file scraping. You can setup scraping on a fresh install (you're not migrating) easily too and scrape everything from scratch, I've included those details at the bottom of the post. I'm going to assume you have the Docker container setup (not sure where Kodi is at running headlessly otherwise?) and your MySQL/phpMyAdmin is setup and working correctly (+= and your old databases have been imported). Also assuming your headless Kodi uses the same version as your old box.
As always, before doing anything whatsoever with your Kodi database (or nothing, backup anyway) make sure you have everything backed up using phpMyAdmin or mysqldump and verify the integrity of your backups. Accidentally cleaning your db instead of updating before fixing your setup could destroy everything.
I'm not an expert in Kodi, these are merely the steps I took to go from point A (scraping not working) to point B (scraping for movies and TV working) so if anyone who is could review this post for any issues I'd appreciate it. These steps may be unnecessary, particularly if you're using the default scrapers, if so copying sources.xml should be enough to get scraping working but these steps should get you past any issues regardless.
1 - I set up the headless Kodi Docker container and using the -v switch in the docker create command added the paths to my files exactly as they are on my old server. No path substitution or messing with the SQL database is needed. It's very simple when creating a container in Docker to recreate the paths exactly as they are on your old server.
2 - Copy your old .kodi/userdata/sources.xml to your new headless installation. Check the paths listed in sources.xml are correct for your new headless server. At this point you can start the headless container, then see if scraping is working, use `kodi-send --host=KODIPADDRESS --port=9777 -a "UpdateLibrary(video)"` or `kodi-send --host=KODIPADDRESS --port=9777 -a "UpdateLibrary(video), '/a/particular/path/you/want/to/update'"`. Install kodi-send from install kodi-eventclients-kodi-send package on Ubuntu. Check .kodi/temp/kodi.log or see if new stuff appears in your Kodi client. Enable debug logging via advancedsettings.xml to assist.
3 - If it's not working. Open up phpMyAdmin or use mysql, go to MyVideos107, go to the "path" table. Run the command `SELECT * FROM `path` WHERE strPath='/your/path'`. If scraping was working on your old install and your sources are the same on your headless server you will find the information you need. Check the SQL command. Run this command for each of the sources in sources.xml that you want to scrape. Note the scraper listed in the "strScraper" column for the path. For each path and each scraper listed for it check that particular scraper is installed in your .kodi/addons folder. Also, for each scraper you copy ensure any dependencies are copied and enabled.
4 - If it does not exist copy the addon from your old kodi install or unzip it to the addons folder from a direct download and restart Kodi. If the addon does not exist and you copy it across or unzip you will also have to install sqlite3, open the Addons27.db (again create a backup before doing this) file from the .kodi/userdata/Databases folder, go to the "installed" table, and go to the "enabled" column for the scraper you've just copied over and ensure it's set to 1. Restart Kodi again.
5 - Also check guisettings.xml in your .kodi/userdata folder. Make sure the default scraper is the scraper you've setup in the previous steps.
If all that has been done your scraper of choice should be working in your headless install.
?:[a-i]|\.[1-9])(?![0-9]))?)([^\\/]*)$]
arseAndCorrectUrl() adding custom header option 'accept-language: en'