Unable to resume indexing

You’ll have to tweak this query until you find the right count, which will tell you what posts are missing here.

Not sure how this can be done.

If WPSOLR is not storing any flag against each post to mark it as indexed/not indexed, I guess there is no way to do this. I believe this is a serious bug.

Also, please note that SOLR will be used in such situations only, where the document size is huge, else there is no need for it in normal WP sites. So you need to have an elegant mechanism to determine posts indexed and not, without relying on just the last indexed date.

From time to time, you will have to re-index all documents. For instance in major lucene upgrades. In that situation, you’ll hit a wall, which is the time necessary to send all your posts to Solr. Whatever the mechanism used to remember posts indexed/not indexed.

Using post_modified is strictly equivalent to storing a state for each index, but does not require a database modification and extra space, and extra indexes or extra joins.

I just want to make you aware that you’ll have to reindex your 7M posts in the future.

And no, the main advantage (for most users) with Solr is not it’s scalability, but it’s efficiency at returning the right results (by customizing schema.xml): stop words, multi-languages, synonyms …

Your situation is very specific, very rare, and means you’ll be stuck sooner or later with weeks of re-indexing.
You’re in a big data use case, and for that you would need big data tools, like a hadoop cluster to load your database (sqoop) in hdfs, split it and index it in // in Solr.

Back to the query now:
how many docs the query returns ?
how many docs the query returns if you remove the post_modified condition ?

how many docs the query returns ?
>> The query returns 734433 docs

how many docs the query returns if you remove the post_modified condition ?
>> Removing the post_modified condition, the query returns 7262480 docs

and how many docs in Solr (/select?q=*%3A*) ?

and how many docs in Solr (/select?q=*%3A*) ?
>> 3903847

It means post ids do not follow post_modified.

You’ll need to switch back to the old WPSOLR version, and apply the same workaround, but this time with the last indexed post id rather than the last indexed post_modified.

It means post ids do not follow post_modified.

You’ll need to switch back to the old WPSOLR version, and apply the same workaround, but this time with the last indexed post id rather than the last indexed post_modified.

It means post ids do not follow post_modified.

You’ll need to switch back to the old WPSOLR version, and apply the same workaround, but this time with the last indexed post id rather than the last indexed post_modified.

Please elaborate how this is to be done with previous version.

The change in incremental algorithm (which uses post_modified rather the post id) appeared in WPSOLR 1.7

If your last good version was > 1.7, you should have nothing to do.
Before that, the indexing process was not incremental, it used to re-index all docs each time you launched the indexing process.

So, if I downgrade the version to 1.7 will it solve my problem and indexing all remaining documents?
Please confirm.

Still waiting for an answer on this????