Database of company names + websites?
from friendly_ghost@beehaw.org to programming@programming.dev on 13 Sep 2024 20:11
https://beehaw.org/post/16040635

I realize this is an absolutely unhinged thing to ask for: Can anyone direct me to a large database of company names with websites (just those two columns), ideally 100K rows or more, that I can access for free? Basically this database if it had another column with the company name: github.com/cygenta/top10million (but doesn’t need to be anywhere near 10 million)

#programming

threaded - newest

verstra@programming.dev on 13 Sep 2024 20:27 next collapse

Maybe search for this on kaggle? Or scrape Wikipedia?

henfredemars@infosec.pub on 13 Sep 2024 21:16 next collapse

You don’t even need to scrape Wikipedia. Simply download all of Wikipedia text only and you could match on articles. It’s only like 20 GB or even less for certain database dumps.

dracs@programming.dev on 14 Sep 2024 03:37 next collapse

Probably don’t need to scrape it. Just query WikiData for it

wikidata.org

friendly_ghost@beehaw.org on 15 Sep 2024 14:33 collapse

Thank you for recommending kaggle! I found just what I needed there (actually way more data than I needed!). Here’s what I went with, if it’s useful to anyone else: www.kaggle.com/…/bigpicture-company-dataset

degen@midwest.social on 13 Sep 2024 22:00 next collapse

Could take a subset and do ICANN lookups. Not sure about doing that at scale though.

MxRemy@piefed.social on 13 Sep 2024 22:06 collapse

A lot of public libraries offer access to ReferenceUSA through your library card. I vaguely remember that queries are pretty customizable on there, and exportable to various formats. Despite the generic name, it's specifically for businesses. Would that work?

friendly_ghost@beehaw.org on 15 Sep 2024 14:40 collapse

Thank you for this suggestion, I had never heard of it! I was able to access it via my library, and easily searched up companies with websites, but then it made me manually click which results to download (wouldn’t let me export the results as one large file). Keeping it bookmarked though, seems like a great resource for other use cases.