Any suggestion?
]]>I’m looking for a plugin (or developer) that can scrape content like for example (Octolooks) that can put the scraped content into a WP theme (Houzez) that uses array types.
Is there something out there, or can someone develop this for me?
Regards, Robert
]]>I’m currently starting an affiliate site thru WordPress. I’m not looking to make it a typical blog styled site (only writing reviews on products and expecting people to purchase thru that link etc (although there will be a blog at some point)).
I’m probably 75% familiar with WordPress + Woocommerce and can work with it but how would I go about creating each product by scraping it from the affiliate/retailer’s site? My niche includes thousands of different products from different brands and retailers like lyst.com. I’d like users to be able sort, categorize and search products. I would also need the products to automatically update for new sales according to each retailer.
Is this possible or do you think lyst.com did it manually with a custom in-house algorithm? Could be possible as they have $60m from the Louis Vuitton Group.
TIA
Title + KEYWORD
The image you want to output appears.
I was wondering if there is any security to prevent web scraping via the WP REST API. It seems you can grab any data you want, however you like.
And if so, any documentation that talks about setting up a secure connection between apps and the site? Like Credentials..etc.
Thanks.
J.
]]>Content scraping can happen at small levels also. This affects the server as well as our standing in search engines. With this plugin, my content is safe and sound on my own website which will help my own search engine rankings and not someone else’s. And the author is very supportive. Wish him the best.
]]>I do not own these sites and the sites do not even have a sitemap and when I tried importing from a XML generator tool I used, the posts came out all messed up or erroneous. So instead I used this web scraper plugin from the wordpress plugin repository and all was working out well.
When I started scraping at first, the posts got scraped really fast at around 100 posts/10 min. But as the total post number kept increasing, the scraping speed got slower.
After completing 10,000 posts, the speed reduced to 200 posts/hour which is still doable. But now that I have posted 25,000 posts, the scraping speed is down to 200 posts/day!
I still have about 20,000 posts left to add and it is not possible for me to continue at such slow pace.
I created a fresh second site on my localhost and tried scraping to see if it was my WAMP that has slowed things down. But no, it has not. Newer sites scrape just as fast as the main site did at the beginning.
As an alternate solution, I tried creating posts on a second site and importing from its XML to my main site but importing through the XML file is just as slow and the posts often mess up the featured images. Importing content through XML on any site other than the main site takes less than a few minutes for me otherwise. So I am pretty sure it is only the main site I’m working on that has slowed down immensely
I tried using WP-optimize to optimize the tables and database for the main site but that has not helped at all. I’ve tried deleting cache using Ctrl + F5 but that has not helped either.
I did expect the site to get slower as I added more content but not this slow. And it seems only adding new posts, whether it be through scraping or importing is affected. The dashboard, site, navigation and everything else even though a bit is working alright. So if someone could help me figure out how I can make adding new content through scraping just as fast as it was the beginning or even half as fast, I would be very thankful.
]]>