I’ve previously written about grabbing Google Streetview images given a particular address. For a different project I sampled images running along an entire street, so figured I would share that code. It is a bit more complicated though, because when you base it off an address you do not need to worry about drawing the same image twice. So I will walk through an example.
So first we will import the necessary libraries we are using, then will globally define your user key and the download folder you want to save the streetview images into.
#Upfront stuff you need
import urllib, os, json
key = "&key=" + "!!!!!!!!!!!!!YourAPIHere!!!!!!!!!!!!!!!!"
DownLoc = r'!!!!!!!!!!!YourFileLocationHere!!!!!!!!!!!!!!' Second are a few functions. The first, MetaParse, grabs the date (Month and Year) and pano_id from a particular street view image. Because if you submit just a slightly different set of lat-lon, google will just download the same image again. To prevent that, we do a sort of memoization, where we grab the meta-data first, stuff it in a global list PrevImage. Then if you have already downloaded that image once, the second GetStreetLL function will not download it again, as it checks the PrevImage list. If you are doing a ton of images you may limit the size of PrevImage to a certain amount, but it is no problem doing a few thousand images as is. (With a free account you can IIRC get 25,000 images in a day, but the meta-queries count against that as well.)
def MetaParse(MetaUrl):
response = urllib.urlopen(MetaUrl)
jsonRaw = response.read()
jsonData = json.loads(jsonRaw)
#return jsonData
if jsonData['status'] == "OK":
if 'date' in jsonData:
return (jsonData['date'],jsonData['pano_id']) #sometimes it does not have a date!
else:
return (None,jsonData['pano_id'])
else:
return (None,None)
PrevImage = [] #Global list that has previous images sampled, memoization kindof
def GetStreetLL(Lat,Lon,Head,File,SaveLoc):
base = r"https://maps.googleapis.com/maps/api/streetview"
size = r"?size=1200x800&fov=60&location="
end = str(Lat) + "," + str(Lon) + "&heading=" + str(Head) + key
MyUrl = base + size + end
fi = File + ".jpg"
MetaUrl = base + r"/metadata" + size + end
#print MyUrl, MetaUrl #can check out image in browser to adjust size, fov to needs
met_lis = list(MetaParse(MetaUrl)) #does not grab image if no date
if (met_lis[1],Head) not in PrevImage and met_lis[0] is not None: #PrevImage is global list
urllib.urlretrieve(MyUrl, os.path.join(SaveLoc,fi))
met_lis.append(fi)
PrevImage.append((met_lis[1],Head)) #append new Pano ID to list of images
else:
met_lis.append(None)
return met_lis Now we are ready to download images running along an entire street. To get the necessary coordinates and header information I worked it out in a GIS. Using a street centerline file I regularly sampled along the streets. Based on those sample points then you can calculate a local trajectory of the street, and then based on that trajectory turn the camera how you want it. Most social science folks I imagine want it to look at the sidewalk, so then you will calculate 90 degrees to the orientation of the street.
Using trial and error I found that spacing the samples around 40 feet apart tended to get a new image. I have the pixel size and fov parameters to the streetview api hard set in the function, but you could easily amend the function to take those as arguments as well.
So next I have an example list of tuples with lat-lon’s and orientation. Then I just loop over those sample locations and draw the images. Here I also have another list image_list, that contains what I save the images too, as well as saves the pano-id and the date meta data.
DataList = [(40.7036043470179800,-74.0143908501053400,97.00),
(40.7037139540670900,-74.0143727485309500,97.00),
(40.7038235569946140,-74.0143546472568100,97.00),
(40.7039329592712600,-74.0143365794219800,97.00),
(40.7040422704154500,-74.0143185262956300,97.00),
(40.7041517813782500,-74.0143004403322000,97.00),
(40.7042611636045350,-74.0142823755611700,97.00),
(40.7043707615693800,-74.0142642750708300,97.00)]
image_list = [] #to stuff the resulting meta-data for images
ct = 0
for i in DataList:
ct += 1
fi = "Image_" + str(ct)
temp = GetStreetLL(Lat=i[0],Lon=i[1],Head=i[2],File=fi,SaveLoc=DownLoc)
if temp[2] is not None:
image_list.append(temp)I have posted the entire python code snippet here. If you want to see the end result, you can check out the photo album. Below is one example image out of the 8 in that street segment, but when viewing the whole album you can see how it runs along the entire street.
Still one of the limitations of this is that there is no easy way to draw older images that I can tell — doing this approach you just get the most recent image. You need to know the pano-id to query older images. Preferably the meta data json should contain multiple entries, but that is not the case. Let me know if there is a way to amend this to grab older imagery or imagery over time. Here is a great example from Kyle Walker showing changes over time in Detroit.
Andrew Wheeler 
Simon Ghislain
/ May 12, 2018Hello again,
I just left a comment on another of your post, but I guess this one is closer to the bojective i’m trying to achieve…
Btw i have again same kind of error when i run the code (entire python code snippet):
AttributeError: module ‘urllib’ has no attribute ‘urlopen’
I’m completely new to Google map API (never used it), but there seem to be a bunch of parameters that can be accessed from a itineray:
https://developers.google.com/maps/documentation/directions/intro#Waypoints
(with MAPS Direction API, sorry it’s french version of the site)
Do you think the process of collecting all points of the road (as well as directions –to be looking ahead on the road, not sideways–) can be done only with the API, without ArcGIS?
Oh god, I really don’t know how to put all this together…I even don’t know yet how I can use that Google Maps API, I never used JSON i don’t know what is it yet 😀
Anyway, thank you again, i’m very glad to discover someone has already done something similar to what i wish to do…
Going back to analyse your code…
Hoping to hear from you soon!
apwheele
/ May 12, 2018It sounds like you have most of the ingredients. Using the directions API you can get JSON for the route. At that point you just need to write code to interpolate along the path at regular intervals. That will be some work though to figure that out with the spherical coordinates. It might be easier to project the coordinates and use shapely, http://shapely.readthedocs.io/en/stable/manual.html.
To generate the route foregoing the Google directions API you might want to check out OSMnx, http://geoffboeing.com/2016/11/osmnx-python-street-networks/.
Simon Ghislain
/ May 12, 2018Thank you for your answer…
I found someone doing exactly that, but in Java:
https://stackoverflow.com/questions/47784512/plotting-coordinates-on-route-in-gmap-google-maps-android-api
I would like to do it in Python, as I already “took in hands” your code and it’s working…
I installed googlemaps python API…but i don’t know where to start…
could you give me an example of like 5 lines of codes to use googlemaps API to generate itinerary from A to B, and how to access the .routes, .legs, .steps, etc…information?
Again, thank you so much for any help, you save me
apwheele
/ May 12, 2018I don’t have code to do your exact task, so you will need to strike out some on your own. The order of tasks will be something like:
– use begin & end locations to query a route via the google directions api.
– take that route in spherical coordinates, project it into any suitable local projection (can use pyproj)
– use the shapely library to interpolate points at regular intervals along the route
– calculate the local orientation of the road from those interpolated points
– reproject the interpolated points back to spherical coordinates
– grab the streetview images at those interpolated points given your preferred orientation of the camera (which you can figure out via knowing the direction of the road)
– stitch the resulting images into a GIF
You should add a small inset map that shows a point flowing along the route where the images are for as well ;).
Pretty sure that is doable, but will take some work.
Simon Ghislain
/ May 12, 2018Thank you very much, I will try that, and I keep you updated on my full code once it is working!
Hope i’ll be back soon…
Thank you again
Simon Ghislain
/ May 12, 2018I DID IT!!
I F***** MANAGED TO DO IT!
(except orientation sector)
Here is my brute code:
import json
import urllib.request
import urllib.parse
import os
from pyproj import Proj
import math
myloc = r”C:\Gmap\01″
key = “&key=” + “”
def distance_cart(p1, p2):
return( math.sqrt((p1[0]-p2[0])*(p1[0]-p2[0]) + (p1[1]-p2[1])*(p1[1]-p2[1])))
def GetStreet(Add,num, SaveLoc):
base = “https://maps.googleapis.com/maps/api/streetview?size=1200×800&location=”
MyUrl = base + urllib.parse.quote_plus(Add) + “&heading=” + str(00.00) + key #added url encoding
fi = str(num) + “.jpg”
urllib.request.urlretrieve(MyUrl, os.path.join(SaveLoc,fi))
def decode_polyline(polyline_str):
index, lat, lng = 0, 0, 0
coordinates = []
changes = {‘latitude’: 0, ‘longitude’: 0}
while index < len(polyline_str):
for unit in ['latitude', 'longitude']:
shift, result = 0, 0
while True:
byte = ord(polyline_str[index]) – 63
index+=1
result |= (byte & 0x1f) <= 0x20:
break
if (result & 1):
changes[unit] = ~(result >> 1)
else:
changes[unit] = (result >> 1)
lat += changes[‘latitude’]
lng += changes[‘longitude’]
coordinates.append((lat / 100000.0, lng / 100000.0))
return coordinates
start = “Ottignies, Belgique”
finish = “Namur, Belgique”
url = ‘http://maps.googleapis.com/maps/api/directions/json?%s’ % urllib.parse.urlencode((
(‘origin’, start),
(‘destination’, finish)
))
ur = urllib.request.urlopen(url)
result = json.load(ur)
coord=[]
for i in range (0, len (result[‘routes’][0][‘legs’][0][‘steps’])):
points1 = result[‘routes’][0][‘legs’][0][‘steps’][i][‘polyline’][‘points’]
if (points1 is not None):
coord.append(decode_polyline(points1))
cc=[]
cce=[]
for i in range (0, len(coord)):
for j in range(0, len(coord[i])):
cc.append(coord[i][j])
lat=coord[i][j][0]
long=coord[i][j][1]
coord_f = str(lat) + “,” + str(long)
p = Proj(proj=’utm’, zone=10, ellps=’WGS84′) # use kwargs
x, y = p(lat, long)
cce.append([x,y])
mess= ‘x=%9.3f’ % cce[i][0]
#print(mess)
#GetStreet(Add=coord_f,SaveLoc=myloc)
print(len(cc))
print(len(cce))
res=[]
p0=cce[0]
res.append(p0)
temp=0
prev=p0
distance_max=500
for i in range (1, len(cce)):
temp = temp + distance_cart(cce[i], prev)
if (temp>distance_max):
res.append(cce[i])
temp=0
prev=cce[i]
print(len(res))
final=[]
for i in range (0, len(res)):
p = Proj(proj=’utm’, zone=10, ellps=’WGS84′)
lat, long = p(res[i][0], res[i][1], inverse=True)
coord_f = str(lat) + “,” + str(long)
GetStreet(Add=coord_f, num=i, SaveLoc=myloc)
It saves all the images..it’s working perfectly! withtout shapely : a simply sqrt to compute distances…
Simon Ghislain
/ May 12, 2018But it seems i’ve been banned now because i didnt use the key while devugging…
How can I know I’m banned?
At run, I get a cascade of error starting from urlretrieve() to
request.py”, line 650, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden
Error 403 forbidden?
I worked 2min ago, i changed nothing… :-s
apwheele
/ May 12, 2018Nice, yeah just set up a new api key, make sure it works, and then you should be good to go. Sometimes with the google APIs it will let me get ~100 requests before it bans me, but some of the api’s don’t work at all without a key.
Simon Ghislain
/ May 12, 2018Yeah definitely looks like i’m banned…
I tried with a valid key (i know it’s valid because i made a request using googlemaps module and i worked)
The line triggering the errors is in GetStreet() function:
urllib.request.urlretrieve(MyUrl, os.path.join(SaveLoc,fi))
It Triggers all the way to HTTP Error 403: Forbidden
How long generally are you banned?
Are there methods to “reset” the ~100 requests quota without key?
I have several API keys in Google api console (credentials tab)…without restrictions and enabled for API Directions (but not for Google street view…and i can’t find it)
Again, thank you so much for your help, I would never have been able to achieve that without you…
Simon Ghislain
/ May 12, 2018Oh…sorry…nevermind my last message, I just found the Street View API, i just had to get an API key for that particular API, and now it works again 🙂
Simon Ghislain
/ May 12, 2018Here is my final workingg code (with correct orientation+saving all images into a .gif and a .mp5 video):
import json
import urllib.request
import urllib.parse
import googlemaps
import os
from pyproj import Proj
import math
import imageio
import sys
import moviepy.editor as mp
myloc = r”C:\Gmap\01″
key = “&key=” + “API_KEY”
res_dir=[]
filesn=[]
def distance_cart(p1, p2):
return( math.sqrt((p1[0]-p2[0])*(p1[0]-p2[0]) + (p1[1]-p2[1])*(p1[1]-p2[1])))
def dir_cart(p1, p2):
if p1[0]-p2[0] == 0:
return math.degrees(math.pi/2.0)
else:
return( math.degrees(math.atan((p1[1]-p2[1]) / (p1[0]-p2[0]))))
def create_gif(filenames, duration):
images = []
for filename in filenames:
images.append(imageio.imread(filename))
output_file = ‘Gif-1.gif’
imageio.mimsave(output_file, images, duration=duration)
def GetStreet(Add,num, SaveLoc):
base = “https://maps.googleapis.com/maps/api/streetview?size=1200×800&location=”
MyUrl = base + urllib.parse.quote_plus(Add) + “&heading=” + str(res_dir[num]+180.00) + key
print(“processing ” + str(num) + “//” + str(len(res_dir)))
fi = str(num) + “.jpg”
filesn.append(str(os.path.join(SaveLoc, fi)))
urllib.request.urlretrieve(MyUrl, os.path.join(SaveLoc,fi))
def decode_polyline(polyline_str):
index, lat, lng = 0, 0, 0
coordinates = []
changes = {‘latitude’: 0, ‘longitude’: 0}
while index < len(polyline_str):
for unit in ['latitude', 'longitude']:
shift, result = 0, 0
while True:
byte = ord(polyline_str[index]) – 63
index+=1
result |= (byte & 0x1f) <= 0x20:
break
if (result & 1):
changes[unit] = ~(result >> 1)
else:
changes[unit] = (result >> 1)
lat += changes[‘latitude’]
lng += changes[‘longitude’]
coordinates.append((lat / 100000.0, lng / 100000.0))
return coordinates
itin=[]
#create a list of checkpoints for the entire itinerary:
itin.append(“Bruxelles, Belgique”)
itin.append(“Ottignies, Belgique”)
itin.append(“Namur, Belgique”)
cc=[]
cce=[]
print(“len itin: ” + str(len(itin)))
for k in range(1, len(itin)):
url = ‘http://maps.googleapis.com/maps/api/directions/json?%s’ % urllib.parse.urlencode((
(‘origin’, itin[k-1]),
(‘destination’, itin[k])
))
ur = urllib.request.urlopen(url)
result = json.load(ur)
coord=[]
for i in range (0, len (result[‘routes’][0][‘legs’][0][‘steps’])):
points1 = result[‘routes’][0][‘legs’][0][‘steps’][i][‘polyline’][‘points’]
if (points1 is not None):
coord.append(decode_polyline(points1))
for i in range (0, len(coord)):
for j in range(0, len(coord[i])):
cc.append(coord[i][j])
lat=coord[i][j][0]
long=coord[i][j][1]
p = Proj(proj=’utm’, zone=10, ellps=’WGS84′) # use kwargs
x, y = p(lat, long)
cce.append([x,y])
print(“len cce ” + str(k) + “:” + str(len(cce)))
print(“len cce tot: ” + str(len(cce)))
res=[]
p0=cce[0]
res.append(p0)
temp=0
prev=p0
distance_max=250
for i in range (1, len(cce)):
temp = temp + distance_cart(cce[i], prev)
if (temp>distance_max):
res.append(cce[i])
res_dir.append(dir_cart(cce[i], prev))
temp=0
prev=cce[i]
res.append(cce[len(cce)-1])
res_dir.append(dir_cart(cce[len(cce)-1], prev))
print(len(res))
print(“resdir: ” + str(len(res_dir)))
for i in range (0, len(res)-1):
#for i in range(0, 10):
p = Proj(proj=’utm’, zone=10, ellps=’WGS84′)
lat, long = p(res[i][0], res[i][1], inverse=True)
coord_f = str(lat) + “,” + str(long)
GetStreet(Add=coord_f, num=i, SaveLoc=myloc)
create_gif(filesn, .2)
clip = mp.VideoFileClip(“Gif-1.gif”)
clip.write_videofile(“gifvideo.mp4”)
Oswaldo Reategui
/ May 13, 2020Hi dear Simon,
Could you please provide me the python file for your project? I tried to replicate this code but a lot of errors popped up.
Thank you.
Valerio De Luca
/ January 4, 2019NameError: global name ‘mid’ is not defined
hieund198
/ May 13, 2019So sorry, I don’t know the place “mid” in your code define
MyUrl = base + mid + end
fi = File + “.jpg”
MetaUrl = base + r”/metadata” + size + end
Could you tell me what’s the “mid”?
Thanks
apwheele
/ May 13, 2019You are right, that is a typo. “Mid” should be “size”, I will update the code snippet.
hieund198
/ May 13, 2019Thank you, I got run your code. It’s helpful for me
Fatin Abdilah
/ January 6, 2021Dear Andrew, I’m so glad I found this post. I’m unsure if you’re familiar with Treepedia (https://github.com/mittrees/Treepedia_Public). The second script of Treepedia calls for the Google Street API to collect metadatas containing panoIDs of point locations (shp) (created in its first script) to be fed into its third and final script. However, since Treepedia was created before Google updated their API urls, I’m having trouble calling the API. Could you please take a quick look at the script. I’m at my wits end. I suspect the problem is the urlAddress line.
# This function is used to collect the metadata of the GSV panoramas based on the sample point shapefile
# Copyright(C) Xiaojiang Li, Ian Seiferling, Marwa Abdulhai, Senseable City Lab, MIT
def GSVpanoMetadataCollector(samplesFeatureClass,num,ouputTextFolder):
”’
This function is used to call the Google API url to collect the metadata of
Google Street View Panoramas. The input of the function is the shpfile of the create sample site, the output
is the generate panoinfo matrics stored in the text file
Parameters:
samplesFeatureClass: the shapefile of the create sample sites
num: the number of sites proced every time
ouputTextFolder: the output folder for the panoinfo
”’
import urllib.request
import google_streetview.api
import xmltodict
import io
import ogr, osr
import time
import os,os.path
import json
if not os.path.exists(ouputTextFolder):
os.makedirs(ouputTextFolder)
driver = ogr.GetDriverByName(‘ESRI Shapefile’)
# change the projection of shapefile to the WGS84
dataset = driver.Open(samplesFeatureClass)
layer = dataset.GetLayer()
sourceProj = layer.GetSpatialRef()
targetProj = osr.SpatialReference()
targetProj.ImportFromEPSG(4326)
transform = osr.CoordinateTransformation(sourceProj, targetProj)
# loop all the features in the featureclass
feature = layer.GetNextFeature()
featureNum = layer.GetFeatureCount()
batch = featureNum//num
for b in range(batch):
# for each batch process num GSV site
start = b*num
end = (b+1)*num
if end > featureNum:
end = featureNum
ouputTextFile = ‘Pnt_start%s_end%s.txt’%(start,end)
ouputGSVinfoFile = os.path.join(ouputTextFolder,ouputTextFile)
# skip over those existing txt files
if os.path.exists(ouputGSVinfoFile):
continue
time.sleep(1)
with open(ouputGSVinfoFile, ‘w’) as panoInfoText:
# process num feature each time
for i in range(start, end):
feature = layer.GetFeature(i)
geom = feature.GetGeometryRef()
# trasform the current projection of input shapefile to WGS84
#WGS84 is Earth centered, earth fixed terrestrial ref system
geom.Transform(transform)
lon = geom.GetX()
lat = geom.GetY()
key = r’????????????????’ #Input Your Key here
# get the meta data of panoramas
# previous link by the code authors: http://maps.google.com/cbk?output=xml&ll=%s,%s
urlAddress = r’https://maps.googleapis.com/maps/api/streetview’%(lat,lon)
time.sleep(0.05)
# the output result of the meta data is a xml object
metaDataxml = urllib2.urlopen(urlAddress)
metaData = metaDataxml.read()
data = xmltodict.parse(metaData)
# in case there is not panorama in the site, therefore, continue
if data[‘panorama’]==None:
continue
else:
panoInfo = data[‘panorama’][‘data_properties’]
# get the meta data of the panorama
panoDate = panoInfo.items()[4][1]
panoId = panoInfo.items()[5][1]
panoLat = panoInfo.items()[8][1]
panoLon = panoInfo.items()[9][1]
# print ‘The coordinate (%s,%s), panoId is: %s, panoDate is: %s’%(panoLon,panoLat,panoId, panoDate)
# lineTxt = ‘panoID: %s panoDate: %s longitude: %s latitude: %s\n’%(panoId, panoDate, panoLon, panoLat)
# panoInfoText.write(lineTxt)
print(‘The coordinate (%s,%s), panoId is: %s, panoDate is: %s’ % (pano_lon, pano_lat, pano_id,
pano_date))
line_txt = ‘panoID: %s panoDate: %s longitude: %s latitude: %s\n’ % (pano_id, pano_date, pano_lon,
pano_lat)
panoInfoText.write(line_txt)
panoInfoText.close()
# ————Main Function ——————-
if __name__ == “__main__”:
import os, os.path
root = ‘C:\\Users\\fatin\\AppData\Local\Programs\Python\Python36\Scripts\Treepedia\Data’
inputShp = os.path.join(root,’GeorgeTown20.shp’)
outputTxt = root
GSVpanoMetadataCollector(inputShp,1000,outputTxt)
Thanks so much.
apwheele
/ January 6, 2021This will take alittle bit of work, but should be doable. (When pasting comments don’t paste your key! I have deleted it out.)
You need to get the panoid from the metadata api, https://developers.google.com/maps/documentation/streetview/metadata?hl=en, so the url line should be :
urlAddress = f’https://maps.googleapis.com/maps/api/streetview/metadata?location={lat},{lon}&key={your_key}’
This api only returns results in json, so you will need to update the code the parses the xml to parse the json like my post. After that it should be OK with the newer API.
Fatin Abdilah
/ January 6, 2021Thanks Andrew for the help. I tried to parse the json, unfortunately the log said:
Traceback (most recent call last):
File “C:\Users\fatin\AppData\Local\Programs\Python\Python36\Scripts\Treepedia\metadataCollector1.py”, line 124, in
GSVpanoMetadataCollector(inputShp,1000,outputTxt)
File “C:\Users\fatin\AppData\Local\Programs\Python\Python36\Scripts\Treepedia\metadataCollector1.py”, line 77, in GSVpanoMetadataCollector
urlAddress = r’https://maps.googleapis.com/maps/api/streetview/metadata?location={lat},{lon}&key=**’ % (lat,lon)
TypeError: not all arguments converted during string formatting
**what I edited**
time.sleep(0.05)
# the output result of the meta data is a xml object
response = urllib2.urlopen(urlAddress)
jsonRaw = response.read()
jsonData = json.loads(jsonRaw)
if jsonData[‘status’] == “OK”:
if ‘date’ in jsonData:
return (jsonData[‘date’],jsonData[‘pano_id’]) #sometimes it does not have a date!
else:
return (None,jsonData[‘pano_id’])
else:
return (None,None)
# in case there is not panorama in the site, therefore, continue
if jsonData[‘panorama’]==None:
continue
else:
panoInfo = jsonData[‘panorama’][‘data_properties’]
*****
I’m obviously new to python so I’m kind of lost at the moment. Am I going the right direction though?
apwheele
/ January 6, 2021So for this line:
urlAddress = r’https://maps.googleapis.com/maps/api/streetview/metadata?location={lat},{lon}&key=**’ % (lat,lon)
That will not work, as you need to pass in the key now directly. It won’t return any results without a key. Your code still just pipes in the lat/lon, and does not include the key. So I use a formatted string (your not using an old V2.? version of python are you?)
urlAddress = f’https://maps.googleapis.com/maps/api/streetview/metadata?location={lat},{lon}&key={key}’
If you are using an older version, it will need to be something like:
url_base = r’https://maps.googleapis.com/maps/api/streetview/metadata?location=’
urlAddress = url_base + str(lat) + ‘,’ + str(lon) + ‘&key=’ + key
or use some other type of string substitution, see
https://andrewpwheeler.com/2015/02/20/string-substitution-in-python/
or
https://andrewpwheeler.com/2015/02/20/string-substitution-in-python/
Fatin Abdilah
/ January 6, 2021Dear Andrew,
Yes I did include the API key, but for the sake of commenting, I changed it to the asterisks. I should’ve noted that sorry bout that. I’m currently using python 3.6 but I’ve yet to try (and tweak it for) python 2.7.
Like I mentioned the log reported:
*****
Traceback (most recent call last):
File “C:\Users\fatin\AppData\Local\Programs\Python\Python36\Scripts\Treepedia\metadataCollector1.py”, line 124, in
GSVpanoMetadataCollector(inputShp,1000,outputTxt)
File “C:\Users\fatin\AppData\Local\Programs\Python\Python36\Scripts\Treepedia\metadataCollector1.py”, line 77, in GSVpanoMetadataCollector
urlAddress = r’https://maps.googleapis.com/maps/api/streetview/metadata?location={lat},{lon}&key=**’ % (lat,lon)
TypeError: not all arguments converted during string formatting
*****
Did I write the json parse codes correctly?
**what I edited**
time.sleep(0.05)
# the output result of the meta data is a xml object
response = urllib2.urlopen(urlAddress)
jsonRaw = response.read()
jsonData = json.loads(jsonRaw)
if jsonData[‘status’] == “OK”:
if ‘date’ in jsonData:
return (jsonData[‘date’],jsonData[‘pano_id’]) #sometimes it does not have a date!
else:
return (None,jsonData[‘pano_id’])
else:
return (None,None)
# in case there is not panorama in the site, therefore, continue
if jsonData[‘panorama’]==None:
continue
else:
panoInfo = jsonData[‘panorama’][‘data_properties’]
*****
apwheele
/ January 6, 2021That error has to do with the line:
urlAddress = r’https://maps.googleapis.com/maps/api/streetview/metadata?location={lat},{lon}&key=**’ % (lat,lon)
Which is wrong as written. Again I think it is easier to just do f string substitution (since you are on python 3 it is available):
urlAddress = f’https://maps.googleapis.com/maps/api/streetview/metadata?location={lat},{lon}&key=**’
You should have enough though with my post to figure out the Json on your own, it is a pretty simple return nested dictionary, and it looks like you have the error condition as well.
apwheele
/ January 6, 2021You may also be interested in this post, https://andrewpwheeler.com/2020/10/24/using-the-google-vision-and-streetview-api-to-explore-hotspots/. I use the Google Vision API to get labels for items in the image, which includes several green space related items.
Fatin Abdilah
/ January 7, 2021Dear Andrew,
I managed to get an updated version of the script. However, I haven’t been able to execute it correctly using bash. The author mentioned an example to call the inputshp, outdir, key variable:
$ python metadataCollector.py –inputshp=’C:/Users/senseablecity/Dropbox (MIT)/ResearchProj/treepedia/cities-proj/Oakland/OaklandSlowStreets/SlowStreets_points/SS_20m.shp’ –outdir=’C:/Users/senseablecity/Dropbox (MIT)/ResearchProj/treepedia/cities-proj/Oakland/OaklandSlowStreets/SlowStreets_points’ –key=’***’
However the log mentioned that it couldn’t recognise my C:/Users/… directory. Could you show me the correct way to call data for the argparse with path directories?
**** the updated script****
def GSVpanoMetadataCollector(samplesFeatureClass,num,ouputTextFolder,key):
”’
This function is used to call the Google API url to collect the metadata of
Google Street View Panoramas. The input of the function is the shpfile of the create sample site, the output
is the generate panoinfo matrics stored in the text file
Parameters:
samplesFeatureClass: the shapefile of the create sample sites
num: the number of sites proced every time
ouputTextFolder: the output folder for the panoinfo
”’
import urllib,urllib2
#import urllib.request
#import google_streetview.api
import xmltodict
import ogr, osr
import time
import os,os.path
import sys
import json
if not os.path.exists(ouputTextFolder):
os.makedirs(ouputTextFolder)
driver = ogr.GetDriverByName(‘ESRI Shapefile’)
# change the projection of shapefile to the WGS84
dataset = driver.Open(samplesFeatureClass)
layer = dataset.GetLayer()
#sourceProj = layer.GetSpatialRef()
#targetProj = osr.SpatialReference()
#targetProj.ImportFromEPSG(4326)
#transform = osr.CoordinateTransformation(sourceProj, targetProj)
# loop all the features in the featureclass
feature = layer.GetNextFeature()
featureNum = layer.GetFeatureCount()
#batch = featureNum//num
batch = int(featureNum/num + 0.5)
print(batch)
for b in range(batch):
# for each batch process num GSV site
start = b*num
geom.Transform(transform)
lon = geom.GetX()
lat = geom.GetY()
#get the metadata of panoramas
urlAddress = r’https://maps.googleapis.com/maps/api/streetview/metadata?size=600×300&location=%s,%s&heading=-45&pitch=42&fov=110&key=%s’%(lon, lat, key)
time.sleep(0.1)
time.sleep(0.05)
#the output result of the metadata is a xml object
#using different url reading method in python 2 and python 3
if sys.version_info[0] ==2:
request = urllib.request.Request(urlAddress)
metaData = urllib.request.urlopen(request).read()
data = json.loads(metaData)
panoDate = data[‘date’]
panoId = data[‘pano_id’]
panoLat = data[‘location’][‘lat’]
panoLon = data[‘location’][‘lng’]
lineTxt = ‘panoID: %s panoDate: %s longitude: %s latitude: %s\n’%(panoId, panoDate, panoLon, panoLat)
panoInfoText.write(lineTxt)
panoInfoText.close()
# ————Main Function ——————-
if __name__ == “__main__”:
import os, os.path
import sys
import argparse
parser = argparse.ArgumentParser(
description = “parameters”
)
parser.add_argument(
“–inputshp”,
required = True,
type = str,
help = “the path of the shapefile”
)
parser.add_argument(
“–outdir”,
default = “mosaic-mrt.tif”,
help = “the output dir for the meta text file”,
type = str,
)
parser.add_argument(
“–key”,
default = “”,
help = “Google Street View key”,
type = str,
)
args = parser.parse_args()
inputshp = args.inputshp
outdir = args.outdir
key = args.key
print(inputshp, outdir, key)
GSVpanoMetadataCollector(inputshp, 1000, outdir, key)
****end of script***
apwheele
/ January 7, 2021In the command line don’t do the equal signs:
python script.py –inputshp ‘C:/Users/whatever’
Your script it doesn’t look like to me it actually writes to the results to the specified output directory.
eddie
/ October 11, 2021I have same issue in treepedia.
Thanks to you, I’ve figured out what the cause is, but haven’t been able to solve it yet.
did you ever solve…??