python - Importing a large CSV from Cloud Storage into App Engine Datastore -

i have large csv file, on order of 1 gb big, , want create entities datastore, 1 entity per row.

that csv file residing in google cloud storage. there clean way this? examples can find online seem rely on having csv file locally, or don't scale well. ideally there's streaming api lets me read in small enough pieces cloud storage make update calls datastore, haven't been able find that.

the buffer receive when open gcs file streaming buffer, can pickled. gcs not support iterator protocol read lines of csv. have write own wrapper. like:

with gcs.open('/app_default_bucket/csv/example.csv', 'r') f:         csv_reader = csv.reader(iter(f.readline, ''))         row in csv_reader:             logging.info(' - '.join(row)) 

if familiair blobstore can use read large csv's gcs using blobstore.create_gs_key( "/gs" + <gcs_file_name_here>). example here