8000 GitHub - giavinh79/lego-data-collection: A repo for gathering data in case it may later be necessary for our Lego RL environment. Progress on hold until I know whether this is necessary.
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

A repo for gathering data in case it may later be necessary for our Lego RL environment. Progress on hold until I know whether this is necessary.

Notifications You must be signed in to change notification settings

giavinh79/lego-data-collection

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

6 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Lego Data Collection

Background

Web-scraping and API hitting scripts written in Node.js to extract Lego part data and output corresponding JSON files.

(1) Script for converting a Lego parts CSV file into a JSON file with image and dimensions data.

(2) Script for reading in DAT files and using the embedded name data to web-scrape for corresponding dimensions online.

Set Up

Go to root directory and do 'npm install'.

Make a .env file and add your own Rebrickable API key value: REBRICKABLE_API_KEY={some value}

Running

In the root directory, run 'node index.js' in the terminal. Follow CL instructions.

Notes

For development, do 'nodemon index.js' instead for hot reload on save.

The API call to Rebrickable may fail due to their rate limiter (will be in the logs): in this case increase the delay function or split the CSV file into smaller chunks.

About

A repo for gathering data in case it may later be necessary for our Lego RL environment. Progress on hold until I know whether this is necessary.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published
0