How to scrape date from Quora? Infinite scroll.

I used the instructions from the web scraping tutorial and can scrape the first 20 questions on a quora topic - https://www.quora.com/topic/Marketing/all_questions --- But there is an infinite scroll at the bottom. The URL does not change. How do I overcome this and get more than 20 questions for a topic? Any suggestions will greatly help. Thanks! Arjun

Posted over 1 year ago by arjun
Posted over 1 year ago by arjun

Any suggestion for this? Been stuck on this for 2 days now!

1
Posted over 1 year ago by Alex Yang

The approach used in the web scraping tutorial is useful for scraping HTML. Have you checked whether you can see content beyond the first 20 questions in the HTML source code? If not, you won't be able to use the same techniques.

1
Posted over 1 year ago by arjun

Hi Alex.. The questions below the first 20 show up in the HTML source code after some time. It's hidden. Been reading about this online - and trying to use Watir (https://github.com/watir/watir-rails) to read the ajax code. Maybe a future tutorial for Baserails - scraping pages that have infitinite scrolling :)

1