Duplicate posts since upgrade

I did some more research, my – final – findings:

1. the meta_value rssmi_source_link is not inserted into the database if the multi importer is run by the Cron job (manager) –> is causing the duplicates, because the search always fails.

2. In the previous release the plugin also checked on duplicates by titles, but this code had been ‘deactived’ (comment // ) so therefore this check is no longer working.

3. Some RSS feeds are not processed. I don’t know why. Is it because the limits of Number of Entries per Feed to Fetch and Number of Total Post Entries per Fetch are reached? (15 feeds, I set this to 100 / 800)

4. the meta_value rssmi_source_link is inserted into the database if the multi importer is run manually (through the admin interface).

1+2 Make the plugin to create duplicates. This mechanisme might by failing for a longer time, but in the past the duplicate check on the title prevented duplicate posts … IMHO … !!

I removed the comments for the duplicate check on titles and will report on any (new) findings within a few days.

Cheers,
Koen

Hi Koen
I had to take off the check for the titles because the php function was causing problems on some versions…if you want, try this file and see if it solves the duplicate problem.

https://www.dropbox.com/s/slomcuvoue0dvzs/import_posts.php

Hi Allen, thanks for the update.
I will upload the new import_posts.php today and let you know in a few days.

btw: my own ‘hacking’ resolved the duplicate issue, but I’ve got this feeling that updates are missing now (did not really check on that, but I used to a have a 10-15 new posts a day, now only got 2), so I rather use your update!

Sorry, duplicates again…

Examples of the rss feeds causings duplicates:

modellspur_de.rss

modellspur_nl.rss

Do you want any additional logging ?

I can’t reproduce this problem…I’m using the most recent version of the plugin, and I get no duplicates no matter how hard I try.

please read this to see how the plugin works to prevent duplicates..

https://www.wprssimporter.com/faqs/im-getting-double-posts-with-the-feed-to-post/

Allen thanks for trying. I replaced your updated version with my own (where the comments for the title check are removed).

You might update your faq, because it says:

… and whether the title of the post is in the post table …

And as far I know, the plug in does not do that any more (unless someone removes the comment-tags // ?? )

ok..try this file and see if this helps…I put another check to see if there is a duplicate

https://www.dropbox.com/s/slomcuvoue0dvzs/import_posts.php

let me know if this works for you.

Allen, I installed the file and the import ran every hour (cron job). The link was not stored in the database (this is query I used):

SELECT *
FROM  <code>ssft_postmeta</code>
WHERE meta_key =  "rssmi_source_link"
AND meta_value =  "https://www.modellspur.de/?DE1403417476"

So I got duplicates (every hour).

Then I started the import manually, and the rssmi_source_link is inserted into the database!

I don’t know anything about the WP cron jobs, but could there be an issue with connecting to the database?

well, the cron file hits the overall function, that should have no different effect than clicking the fetch link (manually) since they do the exact same operation. Are you on a production server or on a development server (like running on your personal computer)?

ok..just tested this with just the cron and had no problem having the link stored in the database…again, not sure what kind of server you are on and how this might be affecting what is happening to you.

Hi Allen, I’m on a production server. Might (another plugin) something lock the table needed for registration of the rssmi_source_link ?

My cron looks very clean – The feedtopost and multi event are the only ones running every hour.

Jun 22, 2014 @ 17:46 (1403459204) Once Hourly wp_rss_multi_event
Jun 22, 2014 @ 18:35 (1403462134) Once Hourly wp_rss_multi_event_feedtopost
Jun 22, 2014 @ 19:34 (1403465640) Twice Daily wp_maybe_auto_update
Jun 22, 2014 @ 23:19 (1403479184) Twice Daily wp_version_check
Jun 22, 2014 @ 23:19 (1403479184) Twice Daily wp_update_plugins
Jun 22, 2014 @ 23:19 (1403479184) Twice Daily wp_update_themes
Jun 23, 2014 @ 4:55 (1403499305) Once Daily wp_scheduled_auto_draft_delete
Jun 23, 2014 @ 6:59 (1403506762) Once Daily updraft_backup_database
Jun 23, 2014 @ 11:24 (1403522667) Once Daily wp_scheduled_delete
Jun 24, 2014 @ 6:59 (1403593162) Once Weekly updraft_backup

At what point do you insert the rssmi_source_link ? Maybe I can add some additional logging to see if it fails (for whatever reason).

Thanks.
Koen

this is the line that inserts the post

$post_id = wp_insert_post($post);

yes, see if another plugin you are running is blocking my plugin from adding the post…I’ve heard the All in One SEO is messing up my plugin, but haven’t had a chance to test this out.

what does this line of code do?

if(add_post_meta($post_id, 'rssmi_source_link', $thisLink)!=false)

Is this the one that adds the meta tag for the rssmi_source_link (in the WP documentation on wp_insert_post I can’t find an update on the post_meta). So I assume that this is the line of code actually adding the post meta data? So in my case it fails (if executed through the cron … )

that line of code says this…..if adding the link to the post meta file does not fail, then go on to do more stuff..hmm, there is no obvious reason why it should fail just because cron triggered the function. I’ll need to see if somebody else has run into this problem (because I cannot reproduce this on my server where is clearly does not fail).

Here is something else….instead of using the cron, try hitting this page:

https://{your url}/wp-cron.php?doing_cron

and doing this will manually fire off the cron scheduler…something is going weird with the WP cron scheduler and I’m finding that hitting the scheduler manually solves the problem. If it does you can use an outside cron scheduler – read more about that here:

https://www.wprssimporter.com/faqs/how-to-have-more-control-over-scheduling-of-feteching-feeds/