Website Scraping

Hi Everyone

 

Has anyone ever used Domo and its connectors to web scrape a website? Jut wondering if any of the connectors have the ability

 

Thanks

 

LB

Comments

  • I haven't experimented to see if a connector exists that does that but it sounds like something that's better suited for scripting out in a programming language and sending the results to Domo via the Domo SDK.

     

    Example: in Python use Selenium or Beautiful Soup to scrape the contents of a site and then format it as a CSV to upload to Domo. 

     

    Maybe other people have different use cases that might answer it better but I'm not sure any connector will do that. Do you have an example of a site you would be scraping?

  • Hi

     

    Thanks for the response. Agreed Python is the way or Sceaming Frog, but a connector would be easier

     

    THx

  • I too am currently working on a project involving creating a card with info from scraped websites. I do not believe Domo has the capability to do this since it requires some scripting, but I do have some good resources/code I've been working on for basic scraping if you run into any issues! 

  • ahackett
    ahackett ⚪️

    I'm looking into this, did anyone come up with a solution?

  • GrantSmith
    GrantSmith Indiana 🔴

    @ahackett

    I'd recommend using a scripting language like Python (with the beautiful soup package). You could possibly utilize an ETL with some regular expressions but there's much more flexibility and power with a scripting language than you would have with Domo.