Open4

google auth & spreadsheets download as csv with postman

bibidevbibidev
  1. create project in gcp
  2. enable google drive api link
  3. create OAuth consent window. [link] (https://console.cloud.google.com/apis/credentials/consent)
    • external user
    • Google Drive API (auth/drive.readonly)
  4. create OAuth 2.0 client id. link
  5. install Postman
  6. Postman setting OAuth2
  7. request spread sheet
bibidevbibidev

refresh token

curl -X POST -d "grant_type=refresh_token" \
 -d "refresh_token={your_refresh_token}" \
 -d "client_id={your_client_id}" \
 -d "client_secret={your_client_secret"} \
 -H "Content-Type: application/x-www-form-urlencoded" \
 https://oauth2.googleapis.com/token

in scrapy

crawl 時に settingsを渡すとよいかも・・・

ScrapingHub利用時は、クレデンシャル類は、DBなどで管理し、ScrpingHubのAPI に渡すのが楽かも

# in spider class attributes
custom_settings = {
  "ROBOTS_OBEY": False,
}

# in start_requests
yield scrapy.Request(
  url="https://oauth2.googleapis.com/token",  
  body="grant_type=refresh_token&refresh_token=...&client_id=...&client_secret=...",
  headers={"Content-Type": "application/x-www-form-urlencoded"},
  method="POST",
  callback=self.parse
)

# in parse method
response = json.loads(response.text)
response["access_token"]