WPSolr Not Indexing All Posts; Stops at 6289

  • Resolved Mike Costanzo

    (@mikecostanzo)


    8 years, 1 month ago

    Hello there,

    I am attempting to index all of my site’s Posts and Pages (both of which are checked in step 2.2 of the Setup process). Before I run the index, WPSolr tells me that I have 8230 documents, which is accurate:

    A total of 0 documents are currently in your index "A Book And A Solr" | 8230 document(s) remain to be indexed

    However, the indexing process stops part-way through at 6289 documents, and WPSOlr gets the impression that there are no more Posts to index. This is the message I get after indexing:

    A total of 6289 documents are currently in your index "A Book And A Solr" | 0 document(s) remain to be indexed

    I tried emptying the index and re-running it, only to get the same result multiple times. I also disabled all other plugins, but that did not help either. (I am running WPSolr 13.7, and am using a GotoSolr subscription to host my Solr instance.)

    Any idea what might be going on here? Thanks in advance for any light you can shed on this one!

Viewing 9 replies - 1 through 9 (of 9 total)
  • Plugin Author WPSolr free

    (@wpsolr)

    Hello,

    Can you identify which type of documents are missing ?

    Thread Starter Mike Costanzo

    (@mikecostanzo)

    They are most likely all Posts; only 4 of the 8230 documents are Pages. I am not indexing custom fields, categories, or comments, just Posts and Pages.

    My original search also had “Index Post Excerpt” and “Expand Shortcodes” enabled, but the problem persists even when these are not checked.

    Any idea what might be going on here?

    Plugin Author WPSolr free

    (@wpsolr)

    Do you index any attachments ?

    Thread Starter Mike Costanzo

    (@mikecostanzo)

    Nope, no attachments were included.

    Plugin Author WPSolr free

    (@wpsolr)

    Did you exclude some posts from the index, either in 2.2 exclusion list, or in wpsolr metabox ?

    Thread Starter Mike Costanzo

    (@mikecostanzo)

    No Posts have been excluded, either in the 2.2 Exclusions list or in the individual Post metabox. If they had, I would expect that such exclusions would be reflected in the initial count of documents to be indexed.

    Plugin Author WPSolr free

    (@wpsolr)

    No, in fact, I delivered a new version to fix that exclusions.

    Thread Starter Mike Costanzo

    (@mikecostanzo)

    I saw that, and I updated to the latest version (13.8). This did not seem to help.

    I also tried changing the number of documents per commit, and in doing so I noticed something very peculiar: there seems to be a proportional relationship between the number of documents per commit and documents indexed:

    100 documents per commit: 6297 documents indexed
    50 documents per commit: 6081 documents indexed
    20 documents per commit: 5968 documents indexed

    So, I did an experiment and increased the number of documents per commit dramatically:

    200 documents per commit: 6515 documents indexed
    1000 documents per commit: 7717 documents indexed
    1500 documents per commit: 8104 documents indexed
    2000 documents per commit: 8237 documents indexed (Success!!!)

    And as you can see, my final attempt was successful in synching all of my Posts and Pages. I have no idea why increasing the number of documents per commit seems to have worked, but it has clearly done the trick. Feel free to mark this issue as resolved, and to use this information for future bug fixes as needed.

    Thanks so much for your timely help, and for the plugin! I very much appreciate authors who care about quality and about helping their users. Have a great day!

    Plugin Author WPSolr free

    (@wpsolr)

    That’s very strange indeed.

    Thanks for the feedback. Could be useful for other users.

Viewing 9 replies - 1 through 9 (of 9 total)
  • The topic ‘WPSolr Not Indexing All Posts; Stops at 6289’ is closed to new replies.