Twitter import processes last 530 tweets only

Hi, sorry, I just realized I never replied to this ??

Are you seeing any errors in your server’s error logs? My guess is that either you actually do only have that many “real tweets” (minus retweets and replies), or that it’s hitting some sort of memory limit or other problem when handling that last tweet.

Hi,

Thanks for getting back to me.

I just gave it another try and I still have the same issue. This time though it imported 540 tweets before stopping, so it looks like it’s not about actual number of tweets – it’s about the data itself: it seems to fail to process specific tweet or page/batch of tweets (i.e. it goes back only to some specific tweet/page and then it stops/fails).

Here’s what I observed while watching it importing data: it says “processing X tweets in this batch” and then, once done, says “processing next batch in X seconds…” and goes in loop, right? So, number of items to process becomes lower and lower on each iteration and then, at some point it just says “0 items in this batch” and then it stops b/c I’d guess it thinks that there’s no more data to process.

However, once again, I am 100% confident there’s more data to import: I even downloaded my Twitter Archive to double confirm and it indeed has all the data I’d expect to get imported.

Also, error log does not contain any errors related to the importer. It only had one error related to other plugin and I even uninstalled that plugin to make sure it’s not causing issues with the importer, but still got the same result.

Let me know if you have any clues.

Thanks,
Kion

If you look in your archive, can you locate/figure out where it’s stopping the import, based on the oldest post that it’s created in WordPress? Based on what you’ve said above, as a hunch — is there perhaps a series of a lot of retweets and/or replies (and no normal tweets) right after that? I wonder if maybe it’s hitting a series of them, importing none of them, then incorrectly thinking there’s nothing left to import.

That’s what my initial suspicion was, but no, that’s not the case: I actually don’t have a lot of retweets/replies in my timeline in general and there’s definitely a whole bunch of regular tweets before and after the tweet it stops on.

It feels more like offset handling issue to me, i.e. offset either remains the same or isn’t shifted enough after processing each subsequent batch of tweets, so next page data contains less and less matching tweets (i.e. not retweets/replies) and eventually contains 0 matching tweets, so it stops.

I didn’t spend a lot of time digging in your code (which looks pretty nice, btw!), neither am I a PHP expert, but here’s a piece that could be problematic, if get_posts returns only records matching specific filter:

const POSTS_PER_BATCH  = 50;
...
$found = get_posts(  array(
'posts_per_page' => static::POSTS_PER_BATCH,
'offset'         => $offset,
...
}
update_option( 'keyring_batch_processing_offset', $offset += count( $found ) );

What if it loads 50 tweets, but only 10 of them match the filter? Offset would be increased by 10, not 50, so next batch would load 10 new records, but 40 same ones, no? Again, I might be completely wrong here as I’m not aware of implementation details (I only spent like 15 minutes looking into your code; i.e. get_posts might not be returning filtered records even, in which case my assumption here is completely wrong), but if what I just said is indeed the case, then there might be an issue. And if that’s indeed what it is, wouldn’t it get fixed by simply always increasing offset by POSTS_PER_BATCH instead of number of records matching the filter?

I believe I have one kind-of-proof for this theory: if I choose to import ALL tweets – i.e. without skipping retweets/replies – it actually works perfectly and imports entire timeline (over 2600 tweets). The problem only occurs when I choose to NOT import retweets/replies (which is what I need for my case).

I hope this is helpful. Let me know once you got a chance to look at it. I’d love to use your awesome plugin for my WordPress installation b/c it’s exactly what I was looking for and I think it’s brilliant, except this one issue, which stops me from using it.

In either case – thanks a lot for your time and keep up the great job with the plugin!

~ Kion

Hey Beau,

I also have one idea, which I think would be great to have in general (would add a lot of flexibility to the tool), but would also solve (or rather work around) the reported issue:

* introduce “Only display tweets of the following type” setting (checkbox) with a couple of checkbox sub-options:
* Regular (i.e. not Retweets/Replies)
* Retweets
* Replies

(Well, I guess you might need to introduce similar setting for each social network importer you provide)

See what I’m saying here? This way you could actually backup entire timeline, but only display what you want on your site, e.g. omit retweets/replies (but still being able to turn them on at any time!).

Alternatively, I guess, post fetching by type could be delegated to some other 3rd-party plugin and all you’d need to do would only be adding some sort of “post type” property to each of the imported posts, e.g. “regular”, “reply”, “retweet”, etc., so they could be fetched by type. I’m not a WordPress expert though, so I’m not sure if that’s feasible at all.

This would not only add great flexibility to the tool (I personally would love to have a backup of my entire feed accessible through wordpress admin console, but I don’t want to display replies/retweets on the public site), it would also solve the reported issue b/c I’d then simply import all tweets (it works fine that way) and then simply configure which of those (by type) need to be shown on public site.

P.S. One could argue that I could simply use CSS/JS to detect and hide posts of specific type (e.g. retweets/replies) and while this is true, it’s not the most elegant solution b/c it won’t work well with pagination, obviously.

~ Kion