So the latest update to the TVDB scraper has been a little "bumpy", to say the least. I sincerely, absolutely apologise for that.
I thought it important to answer some of the more obvious questions regarding the new scraper, and what it means going forward.
Why have you done this?
The TVDB has updated it's API, requiring us to update our scraper to continue to access their database.
Currently, both the old v1 API and the new v2 API work, but this will change soon. In fact, the original sunset date for the v1 API was 1 October. We were already living on borrowed time.
From Kodi v18 (Leia), Python scraping will be possible, and a Python TVDB scraper using the v2 API is already in the works, but for versions below 18 a new scraper using the current regex-based methods was needed.
Why was it so broken?
I wrote the majority of the scraper a long while ago, and misremembered and misjudged how complete it was when I submitted it.
A few rounds of real world testing would have turned up many bugs. And in fact, in a sense, that is what happened.
Thank you to everyone who has pointed out bugs and errors. Please continue to do so, I will endeavour to fix them.
I'm just sorry you didn't get to volunteer for that duty.
Do I need to refresh my library?
Short answer, YES. Everyone will.
Even if you haven't noticed any issue after the scraper update, when the v1 API is taken down new episodes won't get added to any show that has not been refreshed.
This will happen whether you are using the new scraper or have reverted back to the old scraper. The old scraper of course, will cease to work completely.
The reason for this is because Kodi stores an episodeguide URL for each show, which it uses when adding new episodes.
This stored URL is now wrong, but because the v1 API still exists it's possible for the scraper to use the old URL and translate it to the new one on the fly.
However, once the v1 API goes offline, that function won't run, because the old URL will return an error (e.g. 404, 410, 418, or whatever).
You should use this grace period to find a convenient time to refresh your library to replace the old URL with the new one. When time runs out you will be forced to refresh regardless.
How do I refresh?
If you only have a few ongoing series, just refresh them individually. Complete series that will have no more episodes added are not affected, only series with new episodes being added are affected.
To refresh, press 'i' to bring up the info screen and then select the Refresh option.
Only the show details need updating, you don't need to update every episode if you don't want.
![Image]( "Full sized picture")
To refresh the entire library, you will need to go to your source folder(s), bring up the context menu ('c' or long-press enter) and select Change content.
First change the content type to 'None' and agree to remove the contents from the library when prompted. You will not lose watched/unwatched status on the files!
Repeat, but this time setting the content back to TV Shows. Make sure the information provider is The TVDB, and check the settings!
If you are not using English as your default language you may need to turn on the new Fallback language setting.
![Image]( "Full sized picture")
What if I'm using NFO files?
If you're just using URL NFO files (that is, they only contain the TVDB website URL of the TV show), you're fine. You still need to refresh - you don't get out of that - but you'll be fine.
The new scraper can parse those just as well as the old. (Better, in fact.)
If you're using an external media manager to generate full NFO files for the TV Shows AND for every episode BEFORE you add them to the library, you're double-fine.
All your information is coming from those files anyway, you'll barely even notice the change under the hood. You shouldn't even need to refresh.
(So long as your media manager has also made the switch to the v2 API.)
If you have a full tvshow.nfo file but not full episode NFO files, and so rely on the scraper to pull the information for new episodes, you're in trouble.
(This might be the case if you've exported your library to single files previously. Episodes with exported NFO files will appear fine, but newer episodes without won't.)
This is because that tvshow.nfo file contains the wrong episodeguide URL, even when you refresh the show, if you don't choose to ignore local information that wrong URL will just be reused.
If it's just a previously exported nfo file, just choose to ignore it and/or delete it. You can always re-export once you've confirmed everything is working again.
If you need to keep the NFO file because you've manually edited it, you can convert it to a mixed NFO file by simply adding the TVDB website URL for that show to it somewhere outside the xml (e.g. below everything), and then also removing the incorrect episodeguide URL (including the <episodeguide> tags).
When you refresh, the scraper will use the URL to fetch all the information from the TVDB as normal, but will then overwrite that with the information from xml portion of the nfo file (but not the episodeguide URL, because you removed that, see?).
You should then have the new episodeguide URL and still have all of your edited data.
Will my artwork change?
If you don't have locally saved artwork, then refreshing may indeed pull different artwork from the TVDB.
This has to do with the fact that the scraper pulls the art with the highest voted-on average. Averages can change as more people vote; what was popular when you originally added the show to your library may no longer be so.
This would be just as true if you refreshed with the old scraper, and has little to do with the change in API.
If you manually chose a different artwork than the one pulled by the scraper then refreshing WILL undo that.
It is true that fewer artwork will be returned by the new scraper than the old due to the API only returning artwork in the languages specified (Default/Fallback/English - English art is always pulled).
If you manually changed to an artwork from another language, you may find it no longer available.
What if I really don't want to refresh my library?
Warning: This is not a recommended procedure for the novice user. Undertaking this procedure is at your own risk.
It may be possible to manually update the database to directly replace the old episodeguide URLs with the new.
The old URLs look like:
The new URLs look like:
These are located in the C10 column of the tvshow table of MyVideosXXX.db (where XXX is the highest number in your database directory).
This will require a SQLite browser/editor (or MySQL) and knowledge of SQL. It can't be done with a text editor.
It may also be possible to do this with a Python script via the JSON-RPC API, although someone with more knowledge on that would have to comment.
Caution (and back-ups) should be taken before attempting any direct manipulation of the database.
So are there any new features in this new scraper?
YES!!
If you never saw or tested my previous XSLT version, then please read the first post of that thread.
ALL of those features and fixes are in this scraper (... Except Trakt.tv ratings).
Why does it not work on OpenELEC?
By intent or accident OpenELEC does not parse the XSLT used in the new scraper.
This is not a bug in the scraper. XSLT has been available to scrapers since Kodi v13 (Gotham). This is an issue with OpenELEC, and should be brought up with it's maintainers.
LibreELEC by all reports works just fine.
This is too much work! Where can I get the v1.8.5 scraper?
Please understand that the v1.8.5 scraper for The TVDB will soon become obsolete and non-functional.
To continue scraping with The TVDB after the v1 API goes offline you will need to update to v2.0.x. There is no workaround for this.
You are strongly urged to make adjustments now, during this grace period.
I thought it important to answer some of the more obvious questions regarding the new scraper, and what it means going forward.
Why have you done this?
The TVDB has updated it's API, requiring us to update our scraper to continue to access their database.
Currently, both the old v1 API and the new v2 API work, but this will change soon. In fact, the original sunset date for the v1 API was 1 October. We were already living on borrowed time.
From Kodi v18 (Leia), Python scraping will be possible, and a Python TVDB scraper using the v2 API is already in the works, but for versions below 18 a new scraper using the current regex-based methods was needed.
Why was it so broken?
I wrote the majority of the scraper a long while ago, and misremembered and misjudged how complete it was when I submitted it.
A few rounds of real world testing would have turned up many bugs. And in fact, in a sense, that is what happened.
Thank you to everyone who has pointed out bugs and errors. Please continue to do so, I will endeavour to fix them.
I'm just sorry you didn't get to volunteer for that duty.
Do I need to refresh my library?
Short answer, YES. Everyone will.
Even if you haven't noticed any issue after the scraper update, when the v1 API is taken down new episodes won't get added to any show that has not been refreshed.
This will happen whether you are using the new scraper or have reverted back to the old scraper. The old scraper of course, will cease to work completely.
The reason for this is because Kodi stores an episodeguide URL for each show, which it uses when adding new episodes.
This stored URL is now wrong, but because the v1 API still exists it's possible for the scraper to use the old URL and translate it to the new one on the fly.
However, once the v1 API goes offline, that function won't run, because the old URL will return an error (e.g. 404, 410, 418, or whatever).
You should use this grace period to find a convenient time to refresh your library to replace the old URL with the new one. When time runs out you will be forced to refresh regardless.
How do I refresh?
If you only have a few ongoing series, just refresh them individually. Complete series that will have no more episodes added are not affected, only series with new episodes being added are affected.
To refresh, press 'i' to bring up the info screen and then select the Refresh option.
Only the show details need updating, you don't need to update every episode if you don't want.
To refresh the entire library, you will need to go to your source folder(s), bring up the context menu ('c' or long-press enter) and select Change content.
First change the content type to 'None' and agree to remove the contents from the library when prompted. You will not lose watched/unwatched status on the files!
Repeat, but this time setting the content back to TV Shows. Make sure the information provider is The TVDB, and check the settings!
If you are not using English as your default language you may need to turn on the new Fallback language setting.
What if I'm using NFO files?
If you're just using URL NFO files (that is, they only contain the TVDB website URL of the TV show), you're fine. You still need to refresh - you don't get out of that - but you'll be fine.
The new scraper can parse those just as well as the old. (Better, in fact.)
If you're using an external media manager to generate full NFO files for the TV Shows AND for every episode BEFORE you add them to the library, you're double-fine.
All your information is coming from those files anyway, you'll barely even notice the change under the hood. You shouldn't even need to refresh.
(So long as your media manager has also made the switch to the v2 API.)
If you have a full tvshow.nfo file but not full episode NFO files, and so rely on the scraper to pull the information for new episodes, you're in trouble.
(This might be the case if you've exported your library to single files previously. Episodes with exported NFO files will appear fine, but newer episodes without won't.)
This is because that tvshow.nfo file contains the wrong episodeguide URL, even when you refresh the show, if you don't choose to ignore local information that wrong URL will just be reused.
If it's just a previously exported nfo file, just choose to ignore it and/or delete it. You can always re-export once you've confirmed everything is working again.
If you need to keep the NFO file because you've manually edited it, you can convert it to a mixed NFO file by simply adding the TVDB website URL for that show to it somewhere outside the xml (e.g. below everything), and then also removing the incorrect episodeguide URL (including the <episodeguide> tags).
When you refresh, the scraper will use the URL to fetch all the information from the TVDB as normal, but will then overwrite that with the information from xml portion of the nfo file (but not the episodeguide URL, because you removed that, see?).
You should then have the new episodeguide URL and still have all of your edited data.
Will my artwork change?
If you don't have locally saved artwork, then refreshing may indeed pull different artwork from the TVDB.
This has to do with the fact that the scraper pulls the art with the highest voted-on average. Averages can change as more people vote; what was popular when you originally added the show to your library may no longer be so.
This would be just as true if you refreshed with the old scraper, and has little to do with the change in API.
If you manually chose a different artwork than the one pulled by the scraper then refreshing WILL undo that.
It is true that fewer artwork will be returned by the new scraper than the old due to the API only returning artwork in the languages specified (Default/Fallback/English - English art is always pulled).
If you manually changed to an artwork from another language, you may find it no longer available.
What if I really don't want to refresh my library?
Warning: This is not a recommended procedure for the novice user. Undertaking this procedure is at your own risk.
It may be possible to manually update the database to directly replace the old episodeguide URLs with the new.
The old URLs look like:
Code:
<episodeguide><url cache="((ID)).xml">http://www.thetvdb.com/api/((API-KEY))/series/((ID))/all/en.zip</url></episodeguide>The new URLs look like:
Code:
<episodeguide><url post="yes" cache="auth.json">https://api.thetvdb.com/login?{"apikey":"((API-KEY))","id":((ID))}|Content-Type=application/json</url></episodeguide>These are located in the C10 column of the tvshow table of MyVideosXXX.db (where XXX is the highest number in your database directory).
This will require a SQLite browser/editor (or MySQL) and knowledge of SQL. It can't be done with a text editor.
It may also be possible to do this with a Python script via the JSON-RPC API, although someone with more knowledge on that would have to comment.
Caution (and back-ups) should be taken before attempting any direct manipulation of the database.
So are there any new features in this new scraper?
YES!!
If you never saw or tested my previous XSLT version, then please read the first post of that thread.
ALL of those features and fixes are in this scraper (... Except Trakt.tv ratings).
Why does it not work on OpenELEC?
By intent or accident OpenELEC does not parse the XSLT used in the new scraper.
This is not a bug in the scraper. XSLT has been available to scrapers since Kodi v13 (Gotham). This is an issue with OpenELEC, and should be brought up with it's maintainers.
LibreELEC by all reports works just fine.
This is too much work! Where can I get the v1.8.5 scraper?
Please understand that the v1.8.5 scraper for The TVDB will soon become obsolete and non-functional.
To continue scraping with The TVDB after the v1 API goes offline you will need to update to v2.0.x. There is no workaround for this.
You are strongly urged to make adjustments now, during this grace period.