Web Scraper Course - How to Scrape through a list of URLs

So I've used the scraping course to create a list of listings from AirBnB. How do i use that list to scrape each of those listings? I.e. i want the scrape script to load up the CSV file with URLs, and then visit each one to get the info

Posted about 4 years ago by Albert Pak
Posted about 4 years ago by Alex Yang

What have you tried so far? You should try breaking the problem down into smaller parts. First see if you can load a CSV file. Then try reading the URL data from it. Next, store the URL data into an array variable and integrate it with your scraping script.

1
Posted about 4 years ago by Albert Pak

That's actually what I'm trying now - I'm loading up the CSV of just URLs and then looping through them. The issues that Im running into, as I've pointed out in another post, is scraping through ReactJS parts - looks like AirBnB redid some code and now they have ReactJS parts in their individual listings, and Nokogiri doesn't see it upon first page load.

I've tried using PhantomJS / Watir to load the pages, but still didn't work out.

1
Posted about 4 years ago by Alex Yang

Which elements on the Airbnb page are you having trouble scraping?

1
Posted about 4 years ago by Albert Pak

Im having problem getting to info in "The Space", "Amenities" and "Prices" areas of individual listings.

Here's an example: http://postimg.org/image/qyvpnj9p9/

1
Posted about 4 years ago by Alex Yang

I haven't scraped elements loaded from JavaScript before. I can't say for sure that Nokogiri is even capable of doing this, and it's possible you may need to look at other tools/approaches. Sorry I can't be more helpful here.

1