Drawing Google Streetview images down an entire street using python

I’ve previously written about grabbing Google Streetview images given a particular address. For a different project I sampled images running along an entire street, so figured I would share that code. It is a bit more complicated though, because when you base it off an address you do not need to worry about drawing the same image twice. So I will walk through an example.

So first we will import the necessary libraries we are using, then will globally define your user key and the download folder you want to save the streetview images into.

#Upfront stuff you need
import urllib, os, json
key = "&key=" + "!!!!!!!!!!!!!YourAPIHere!!!!!!!!!!!!!!!!"
DownLoc = r'!!!!!!!!!!!YourFileLocationHere!!!!!!!!!!!!!!'  

Second are a few functions. The first, MetaParse, grabs the date (Month and Year) and pano_id from a particular street view image. Because if you submit just a slightly different set of lat-lon, google will just download the same image again. To prevent that, we do a sort of memoization, where we grab the meta-data first, stuff it in a global list PrevImage. Then if you have already downloaded that image once, the second GetStreetLL function will not download it again, as it checks the PrevImage list. If you are doing a ton of images you may limit the size of PrevImage to a certain amount, but it is no problem doing a few thousand images as is. (With a free account you can IIRC get 25,000 images in a day, but the meta-queries count against that as well.)

def MetaParse(MetaUrl):
    response = urllib.urlopen(MetaUrl)
    jsonRaw = response.read()
    jsonData = json.loads(jsonRaw)
    #return jsonData
    if jsonData['status'] == "OK":
        if 'date' in jsonData:
            return (jsonData['date'],jsonData['pano_id']) #sometimes it does not have a date!
        else:
            return (None,jsonData['pano_id'])
    else:
        return (None,None)

PrevImage = [] #Global list that has previous images sampled, memoization kindof        
        
def GetStreetLL(Lat,Lon,Head,File,SaveLoc):
    base = r"https://maps.googleapis.com/maps/api/streetview"
    size = r"?size=1200x800&fov=60&location="
    end = str(Lat) + "," + str(Lon) + "&heading=" + str(Head) + key
    MyUrl = base + size + end
    fi = File + ".jpg"
    MetaUrl = base + r"/metadata" + size + end
    #print MyUrl, MetaUrl #can check out image in browser to adjust size, fov to needs
    met_lis = list(MetaParse(MetaUrl))                           #does not grab image if no date
    if (met_lis[1],Head) not in PrevImage and met_lis[0] is not None:   #PrevImage is global list
        urllib.urlretrieve(MyUrl, os.path.join(SaveLoc,fi))
        met_lis.append(fi)
        PrevImage.append((met_lis[1],Head)) #append new Pano ID to list of images
    else:
        met_lis.append(None)
    return met_lis  

Now we are ready to download images running along an entire street. To get the necessary coordinates and header information I worked it out in a GIS. Using a street centerline file I regularly sampled along the streets. Based on those sample points then you can calculate a local trajectory of the street, and then based on that trajectory turn the camera how you want it. Most social science folks I imagine want it to look at the sidewalk, so then you will calculate 90 degrees to the orientation of the street.

Using trial and error I found that spacing the samples around 40 feet apart tended to get a new image. I have the pixel size and fov parameters to the streetview api hard set in the function, but you could easily amend the function to take those as arguments as well.

So next I have an example list of tuples with lat-lon’s and orientation. Then I just loop over those sample locations and draw the images. Here I also have another list image_list, that contains what I save the images too, as well as saves the pano-id and the date meta data.

DataList = [(40.7036043470179800,-74.0143908501053400,97.00),
            (40.7037139540670900,-74.0143727485309500,97.00),
            (40.7038235569946140,-74.0143546472568100,97.00),
            (40.7039329592712600,-74.0143365794219800,97.00),
            (40.7040422704154500,-74.0143185262956300,97.00),
            (40.7041517813782500,-74.0143004403322000,97.00),
            (40.7042611636045350,-74.0142823755611700,97.00),
            (40.7043707615693800,-74.0142642750708300,97.00)]

    
image_list = [] #to stuff the resulting meta-data for images
ct = 0
for i in DataList:
    ct += 1
    fi = "Image_" + str(ct)
    temp = GetStreetLL(Lat=i[0],Lon=i[1],Head=i[2],File=fi,SaveLoc=DownLoc)
    if temp[2] is not None:
        image_list.append(temp)

I have posted the entire python code snippet here. If you want to see the end result, you can check out the photo album. Below is one example image out of the 8 in that street segment, but when viewing the whole album you can see how it runs along the entire street.

Still one of the limitations of this is that there is no easy way to draw older images that I can tell — doing this approach you just get the most recent image. You need to know the pano-id to query older images. Preferably the meta data json should contain multiple entries, but that is not the case. Let me know if there is a way to amend this to grab older imagery or imagery over time. Here is a great example from Kyle Walker showing changes over time in Detroit.

Leave a comment

28 Comments

  1. Simon Ghislain

     /  May 12, 2018

    Hello again,

    I just left a comment on another of your post, but I guess this one is closer to the bojective i’m trying to achieve…

    Btw i have again same kind of error when i run the code (entire python code snippet):
    AttributeError: module ‘urllib’ has no attribute ‘urlopen’

    I’m completely new to Google map API (never used it), but there seem to be a bunch of parameters that can be accessed from a itineray:
    https://developers.google.com/maps/documentation/directions/intro#Waypoints

    (with MAPS Direction API, sorry it’s french version of the site)

    Do you think the process of collecting all points of the road (as well as directions –to be looking ahead on the road, not sideways–) can be done only with the API, without ArcGIS?

    Oh god, I really don’t know how to put all this together…I even don’t know yet how I can use that Google Maps API, I never used JSON i don’t know what is it yet 😀

    Anyway, thank you again, i’m very glad to discover someone has already done something similar to what i wish to do…

    Going back to analyse your code…
    Hoping to hear from you soon!

    Reply
    • It sounds like you have most of the ingredients. Using the directions API you can get JSON for the route. At that point you just need to write code to interpolate along the path at regular intervals. That will be some work though to figure that out with the spherical coordinates. It might be easier to project the coordinates and use shapely, http://shapely.readthedocs.io/en/stable/manual.html.

      To generate the route foregoing the Google directions API you might want to check out OSMnx, http://geoffboeing.com/2016/11/osmnx-python-street-networks/.

      Reply
      • Simon Ghislain

         /  May 12, 2018

        Thank you for your answer…
        I found someone doing exactly that, but in Java:
        https://stackoverflow.com/questions/47784512/plotting-coordinates-on-route-in-gmap-google-maps-android-api

        I would like to do it in Python, as I already “took in hands” your code and it’s working…
        I installed googlemaps python API…but i don’t know where to start…

        could you give me an example of like 5 lines of codes to use googlemaps API to generate itinerary from A to B, and how to access the .routes, .legs, .steps, etc…information?

        Again, thank you so much for any help, you save me

      • I don’t have code to do your exact task, so you will need to strike out some on your own. The order of tasks will be something like:

        – use begin & end locations to query a route via the google directions api.
        – take that route in spherical coordinates, project it into any suitable local projection (can use pyproj)
        – use the shapely library to interpolate points at regular intervals along the route
        – calculate the local orientation of the road from those interpolated points
        – reproject the interpolated points back to spherical coordinates
        – grab the streetview images at those interpolated points given your preferred orientation of the camera (which you can figure out via knowing the direction of the road)
        – stitch the resulting images into a GIF

        You should add a small inset map that shows a point flowing along the route where the images are for as well ;).

        Pretty sure that is doable, but will take some work.

      • Simon Ghislain

         /  May 12, 2018

        Thank you very much, I will try that, and I keep you updated on my full code once it is working!
        Hope i’ll be back soon…
        Thank you again

      • Simon Ghislain

         /  May 12, 2018

        I DID IT!!
        I F***** MANAGED TO DO IT!
        (except orientation sector)

        Here is my brute code:

        import json
        import urllib.request
        import urllib.parse
        import os
        from pyproj import Proj
        import math
        myloc = r”C:\Gmap\01″
        key = “&key=” + “”

        def distance_cart(p1, p2):
        return( math.sqrt((p1[0]-p2[0])*(p1[0]-p2[0]) + (p1[1]-p2[1])*(p1[1]-p2[1])))

        def GetStreet(Add,num, SaveLoc):
        base = “https://maps.googleapis.com/maps/api/streetview?size=1200×800&location=”
        MyUrl = base + urllib.parse.quote_plus(Add) + “&heading=” + str(00.00) + key #added url encoding
        fi = str(num) + “.jpg”
        urllib.request.urlretrieve(MyUrl, os.path.join(SaveLoc,fi))

        def decode_polyline(polyline_str):
        index, lat, lng = 0, 0, 0
        coordinates = []
        changes = {‘latitude’: 0, ‘longitude’: 0}

        while index < len(polyline_str):
        for unit in ['latitude', 'longitude']:
        shift, result = 0, 0

        while True:
        byte = ord(polyline_str[index]) – 63
        index+=1
        result |= (byte & 0x1f) <= 0x20:
        break

        if (result & 1):
        changes[unit] = ~(result >> 1)
        else:
        changes[unit] = (result >> 1)

        lat += changes[‘latitude’]
        lng += changes[‘longitude’]

        coordinates.append((lat / 100000.0, lng / 100000.0))

        return coordinates

        start = “Ottignies, Belgique”
        finish = “Namur, Belgique”

        url = ‘http://maps.googleapis.com/maps/api/directions/json?%s’ % urllib.parse.urlencode((
        (‘origin’, start),
        (‘destination’, finish)
        ))
        ur = urllib.request.urlopen(url)
        result = json.load(ur)
        coord=[]
        for i in range (0, len (result[‘routes’][0][‘legs’][0][‘steps’])):
        points1 = result[‘routes’][0][‘legs’][0][‘steps’][i][‘polyline’][‘points’]
        if (points1 is not None):
        coord.append(decode_polyline(points1))

        cc=[]
        cce=[]
        for i in range (0, len(coord)):
        for j in range(0, len(coord[i])):
        cc.append(coord[i][j])
        lat=coord[i][j][0]
        long=coord[i][j][1]
        coord_f = str(lat) + “,” + str(long)

        p = Proj(proj=’utm’, zone=10, ellps=’WGS84′) # use kwargs
        x, y = p(lat, long)
        cce.append([x,y])

        mess= ‘x=%9.3f’ % cce[i][0]

        #print(mess)

        #GetStreet(Add=coord_f,SaveLoc=myloc)

        print(len(cc))
        print(len(cce))

        res=[]
        p0=cce[0]
        res.append(p0)
        temp=0
        prev=p0
        distance_max=500
        for i in range (1, len(cce)):
        temp = temp + distance_cart(cce[i], prev)
        if (temp>distance_max):
        res.append(cce[i])
        temp=0
        prev=cce[i]

        print(len(res))

        final=[]
        for i in range (0, len(res)):
        p = Proj(proj=’utm’, zone=10, ellps=’WGS84′)
        lat, long = p(res[i][0], res[i][1], inverse=True)
        coord_f = str(lat) + “,” + str(long)
        GetStreet(Add=coord_f, num=i, SaveLoc=myloc)

        It saves all the images..it’s working perfectly! withtout shapely : a simply sqrt to compute distances…

  2. Simon Ghislain

     /  May 12, 2018

    But it seems i’ve been banned now because i didnt use the key while devugging…
    How can I know I’m banned?

    At run, I get a cascade of error starting from urlretrieve() to
    request.py”, line 650, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
    urllib.error.HTTPError: HTTP Error 403: Forbidden

    Error 403 forbidden?
    I worked 2min ago, i changed nothing… :-s

    Reply
    • Nice, yeah just set up a new api key, make sure it works, and then you should be good to go. Sometimes with the google APIs it will let me get ~100 requests before it bans me, but some of the api’s don’t work at all without a key.

      Reply
      • Simon Ghislain

         /  May 12, 2018

        Yeah definitely looks like i’m banned…
        I tried with a valid key (i know it’s valid because i made a request using googlemaps module and i worked)

        The line triggering the errors is in GetStreet() function:
        urllib.request.urlretrieve(MyUrl, os.path.join(SaveLoc,fi))

        It Triggers all the way to HTTP Error 403: Forbidden

        How long generally are you banned?
        Are there methods to “reset” the ~100 requests quota without key?

        I have several API keys in Google api console (credentials tab)…without restrictions and enabled for API Directions (but not for Google street view…and i can’t find it)

        Again, thank you so much for your help, I would never have been able to achieve that without you…

  3. Simon Ghislain

     /  May 12, 2018

    Oh…sorry…nevermind my last message, I just found the Street View API, i just had to get an API key for that particular API, and now it works again 🙂

    Reply
  4. Simon Ghislain

     /  May 12, 2018

    Here is my final workingg code (with correct orientation+saving all images into a .gif and a .mp5 video):

    import json
    import urllib.request
    import urllib.parse
    import googlemaps
    import os
    from pyproj import Proj
    import math
    import imageio
    import sys
    import moviepy.editor as mp

    myloc = r”C:\Gmap\01″
    key = “&key=” + “API_KEY”
    res_dir=[]
    filesn=[]

    def distance_cart(p1, p2):
    return( math.sqrt((p1[0]-p2[0])*(p1[0]-p2[0]) + (p1[1]-p2[1])*(p1[1]-p2[1])))

    def dir_cart(p1, p2):
    if p1[0]-p2[0] == 0:
    return math.degrees(math.pi/2.0)
    else:
    return( math.degrees(math.atan((p1[1]-p2[1]) / (p1[0]-p2[0]))))

    def create_gif(filenames, duration):
    images = []
    for filename in filenames:
    images.append(imageio.imread(filename))
    output_file = ‘Gif-1.gif’
    imageio.mimsave(output_file, images, duration=duration)

    def GetStreet(Add,num, SaveLoc):
    base = “https://maps.googleapis.com/maps/api/streetview?size=1200×800&location=”
    MyUrl = base + urllib.parse.quote_plus(Add) + “&heading=” + str(res_dir[num]+180.00) + key
    print(“processing ” + str(num) + “//” + str(len(res_dir)))
    fi = str(num) + “.jpg”
    filesn.append(str(os.path.join(SaveLoc, fi)))
    urllib.request.urlretrieve(MyUrl, os.path.join(SaveLoc,fi))

    def decode_polyline(polyline_str):
    index, lat, lng = 0, 0, 0
    coordinates = []
    changes = {‘latitude’: 0, ‘longitude’: 0}

    while index < len(polyline_str):
    for unit in ['latitude', 'longitude']:
    shift, result = 0, 0

    while True:
    byte = ord(polyline_str[index]) – 63
    index+=1
    result |= (byte & 0x1f) <= 0x20:
    break

    if (result & 1):
    changes[unit] = ~(result >> 1)
    else:
    changes[unit] = (result >> 1)

    lat += changes[‘latitude’]
    lng += changes[‘longitude’]

    coordinates.append((lat / 100000.0, lng / 100000.0))

    return coordinates

    itin=[]
    #create a list of checkpoints for the entire itinerary:
    itin.append(“Bruxelles, Belgique”)
    itin.append(“Ottignies, Belgique”)
    itin.append(“Namur, Belgique”)

    cc=[]
    cce=[]
    print(“len itin: ” + str(len(itin)))

    for k in range(1, len(itin)):

    url = ‘http://maps.googleapis.com/maps/api/directions/json?%s’ % urllib.parse.urlencode((
    (‘origin’, itin[k-1]),
    (‘destination’, itin[k])
    ))
    ur = urllib.request.urlopen(url)
    result = json.load(ur)
    coord=[]
    for i in range (0, len (result[‘routes’][0][‘legs’][0][‘steps’])):
    points1 = result[‘routes’][0][‘legs’][0][‘steps’][i][‘polyline’][‘points’]
    if (points1 is not None):
    coord.append(decode_polyline(points1))

    for i in range (0, len(coord)):
    for j in range(0, len(coord[i])):
    cc.append(coord[i][j])
    lat=coord[i][j][0]
    long=coord[i][j][1]

    p = Proj(proj=’utm’, zone=10, ellps=’WGS84′) # use kwargs
    x, y = p(lat, long)
    cce.append([x,y])

    print(“len cce ” + str(k) + “:” + str(len(cce)))
    print(“len cce tot: ” + str(len(cce)))
    res=[]

    p0=cce[0]
    res.append(p0)
    temp=0
    prev=p0
    distance_max=250
    for i in range (1, len(cce)):
    temp = temp + distance_cart(cce[i], prev)
    if (temp>distance_max):
    res.append(cce[i])
    res_dir.append(dir_cart(cce[i], prev))
    temp=0
    prev=cce[i]
    res.append(cce[len(cce)-1])
    res_dir.append(dir_cart(cce[len(cce)-1], prev))
    print(len(res))
    print(“resdir: ” + str(len(res_dir)))

    for i in range (0, len(res)-1):
    #for i in range(0, 10):
    p = Proj(proj=’utm’, zone=10, ellps=’WGS84′)
    lat, long = p(res[i][0], res[i][1], inverse=True)
    coord_f = str(lat) + “,” + str(long)
    GetStreet(Add=coord_f, num=i, SaveLoc=myloc)

    create_gif(filesn, .2)

    clip = mp.VideoFileClip(“Gif-1.gif”)
    clip.write_videofile(“gifvideo.mp4”)

    Reply
    • Oswaldo Reategui

       /  May 13, 2020

      Hi dear Simon,
      Could you please provide me the python file for your project? I tried to replicate this code but a lot of errors popped up.
      Thank you.

      Reply
  5. Valerio De Luca

     /  January 4, 2019

    NameError: global name ‘mid’ is not defined

    Reply
  6. So sorry, I don’t know the place “mid” in your code define
    MyUrl = base + mid + end
    fi = File + “.jpg”
    MetaUrl = base + r”/metadata” + size + end

    Could you tell me what’s the “mid”?

    Thanks

    Reply
  7. Fatin Abdilah

     /  January 6, 2021

    Dear Andrew, I’m so glad I found this post. I’m unsure if you’re familiar with Treepedia (https://github.com/mittrees/Treepedia_Public). The second script of Treepedia calls for the Google Street API to collect metadatas containing panoIDs of point locations (shp) (created in its first script) to be fed into its third and final script. However, since Treepedia was created before Google updated their API urls, I’m having trouble calling the API. Could you please take a quick look at the script. I’m at my wits end. I suspect the problem is the urlAddress line.

    # This function is used to collect the metadata of the GSV panoramas based on the sample point shapefile

    # Copyright(C) Xiaojiang Li, Ian Seiferling, Marwa Abdulhai, Senseable City Lab, MIT

    def GSVpanoMetadataCollector(samplesFeatureClass,num,ouputTextFolder):
    ”’
    This function is used to call the Google API url to collect the metadata of
    Google Street View Panoramas. The input of the function is the shpfile of the create sample site, the output
    is the generate panoinfo matrics stored in the text file

    Parameters:
    samplesFeatureClass: the shapefile of the create sample sites
    num: the number of sites proced every time
    ouputTextFolder: the output folder for the panoinfo

    ”’

    import urllib.request
    import google_streetview.api
    import xmltodict
    import io
    import ogr, osr
    import time
    import os,os.path
    import json

    if not os.path.exists(ouputTextFolder):
    os.makedirs(ouputTextFolder)

    driver = ogr.GetDriverByName(‘ESRI Shapefile’)

    # change the projection of shapefile to the WGS84
    dataset = driver.Open(samplesFeatureClass)
    layer = dataset.GetLayer()

    sourceProj = layer.GetSpatialRef()
    targetProj = osr.SpatialReference()
    targetProj.ImportFromEPSG(4326)
    transform = osr.CoordinateTransformation(sourceProj, targetProj)

    # loop all the features in the featureclass
    feature = layer.GetNextFeature()
    featureNum = layer.GetFeatureCount()
    batch = featureNum//num

    for b in range(batch):
    # for each batch process num GSV site
    start = b*num
    end = (b+1)*num
    if end > featureNum:
    end = featureNum

    ouputTextFile = ‘Pnt_start%s_end%s.txt’%(start,end)
    ouputGSVinfoFile = os.path.join(ouputTextFolder,ouputTextFile)

    # skip over those existing txt files
    if os.path.exists(ouputGSVinfoFile):
    continue

    time.sleep(1)

    with open(ouputGSVinfoFile, ‘w’) as panoInfoText:
    # process num feature each time
    for i in range(start, end):
    feature = layer.GetFeature(i)
    geom = feature.GetGeometryRef()

    # trasform the current projection of input shapefile to WGS84
    #WGS84 is Earth centered, earth fixed terrestrial ref system
    geom.Transform(transform)
    lon = geom.GetX()
    lat = geom.GetY()
    key = r’????????????????’ #Input Your Key here

    # get the meta data of panoramas
    # previous link by the code authors: http://maps.google.com/cbk?output=xml&ll=%s,%s
    urlAddress = r’https://maps.googleapis.com/maps/api/streetview’%(lat,lon)

    time.sleep(0.05)
    # the output result of the meta data is a xml object
    metaDataxml = urllib2.urlopen(urlAddress)
    metaData = metaDataxml.read()

    data = xmltodict.parse(metaData)

    # in case there is not panorama in the site, therefore, continue
    if data[‘panorama’]==None:
    continue
    else:
    panoInfo = data[‘panorama’][‘data_properties’]

    # get the meta data of the panorama
    panoDate = panoInfo.items()[4][1]
    panoId = panoInfo.items()[5][1]
    panoLat = panoInfo.items()[8][1]
    panoLon = panoInfo.items()[9][1]

    # print ‘The coordinate (%s,%s), panoId is: %s, panoDate is: %s’%(panoLon,panoLat,panoId, panoDate)
    # lineTxt = ‘panoID: %s panoDate: %s longitude: %s latitude: %s\n’%(panoId, panoDate, panoLon, panoLat)
    # panoInfoText.write(lineTxt)
    print(‘The coordinate (%s,%s), panoId is: %s, panoDate is: %s’ % (pano_lon, pano_lat, pano_id,
    pano_date))
    line_txt = ‘panoID: %s panoDate: %s longitude: %s latitude: %s\n’ % (pano_id, pano_date, pano_lon,
    pano_lat)
    panoInfoText.write(line_txt)

    panoInfoText.close()

    # ————Main Function ——————-
    if __name__ == “__main__”:
    import os, os.path

    root = ‘C:\\Users\\fatin\\AppData\Local\Programs\Python\Python36\Scripts\Treepedia\Data’
    inputShp = os.path.join(root,’GeorgeTown20.shp’)
    outputTxt = root

    GSVpanoMetadataCollector(inputShp,1000,outputTxt)

    Thanks so much.

    Reply
    • This will take alittle bit of work, but should be doable. (When pasting comments don’t paste your key! I have deleted it out.)

      You need to get the panoid from the metadata api, https://developers.google.com/maps/documentation/streetview/metadata?hl=en, so the url line should be :

      urlAddress = f’https://maps.googleapis.com/maps/api/streetview/metadata?location={lat},{lon}&key={your_key}’

      This api only returns results in json, so you will need to update the code the parses the xml to parse the json like my post. After that it should be OK with the newer API.

      Reply
      • Fatin Abdilah

         /  January 6, 2021

        Thanks Andrew for the help. I tried to parse the json, unfortunately the log said:

        Traceback (most recent call last):
        File “C:\Users\fatin\AppData\Local\Programs\Python\Python36\Scripts\Treepedia\metadataCollector1.py”, line 124, in
        GSVpanoMetadataCollector(inputShp,1000,outputTxt)
        File “C:\Users\fatin\AppData\Local\Programs\Python\Python36\Scripts\Treepedia\metadataCollector1.py”, line 77, in GSVpanoMetadataCollector
        urlAddress = r’https://maps.googleapis.com/maps/api/streetview/metadata?location={lat},{lon}&key=**’ % (lat,lon)
        TypeError: not all arguments converted during string formatting

        **what I edited**

        time.sleep(0.05)
        # the output result of the meta data is a xml object
        response = urllib2.urlopen(urlAddress)
        jsonRaw = response.read()
        jsonData = json.loads(jsonRaw)
        if jsonData[‘status’] == “OK”:
        if ‘date’ in jsonData:
        return (jsonData[‘date’],jsonData[‘pano_id’]) #sometimes it does not have a date!
        else:
        return (None,jsonData[‘pano_id’])
        else:
        return (None,None)

        # in case there is not panorama in the site, therefore, continue
        if jsonData[‘panorama’]==None:
        continue
        else:
        panoInfo = jsonData[‘panorama’][‘data_properties’]

        *****

        I’m obviously new to python so I’m kind of lost at the moment. Am I going the right direction though?

      • So for this line:

        urlAddress = r’https://maps.googleapis.com/maps/api/streetview/metadata?location={lat},{lon}&key=**’ % (lat,lon)

        That will not work, as you need to pass in the key now directly. It won’t return any results without a key. Your code still just pipes in the lat/lon, and does not include the key. So I use a formatted string (your not using an old V2.? version of python are you?)

        urlAddress = f’https://maps.googleapis.com/maps/api/streetview/metadata?location={lat},{lon}&key={key}’

        If you are using an older version, it will need to be something like:

        url_base = r’https://maps.googleapis.com/maps/api/streetview/metadata?location=’
        urlAddress = url_base + str(lat) + ‘,’ + str(lon) + ‘&key=’ + key

        or use some other type of string substitution, see

        https://andrewpwheeler.com/2015/02/20/string-substitution-in-python/

        or

        https://andrewpwheeler.com/2015/02/20/string-substitution-in-python/

      • Fatin Abdilah

         /  January 6, 2021

        Dear Andrew,

        Yes I did include the API key, but for the sake of commenting, I changed it to the asterisks. I should’ve noted that sorry bout that. I’m currently using python 3.6 but I’ve yet to try (and tweak it for) python 2.7.

        Like I mentioned the log reported:
        *****
        Traceback (most recent call last):
        File “C:\Users\fatin\AppData\Local\Programs\Python\Python36\Scripts\Treepedia\metadataCollector1.py”, line 124, in
        GSVpanoMetadataCollector(inputShp,1000,outputTxt)
        File “C:\Users\fatin\AppData\Local\Programs\Python\Python36\Scripts\Treepedia\metadataCollector1.py”, line 77, in GSVpanoMetadataCollector
        urlAddress = r’https://maps.googleapis.com/maps/api/streetview/metadata?location={lat},{lon}&key=**’ % (lat,lon)
        TypeError: not all arguments converted during string formatting

        *****
        Did I write the json parse codes correctly?

        **what I edited**

        time.sleep(0.05)
        # the output result of the meta data is a xml object
        response = urllib2.urlopen(urlAddress)
        jsonRaw = response.read()
        jsonData = json.loads(jsonRaw)
        if jsonData[‘status’] == “OK”:
        if ‘date’ in jsonData:
        return (jsonData[‘date’],jsonData[‘pano_id’]) #sometimes it does not have a date!
        else:
        return (None,jsonData[‘pano_id’])
        else:
        return (None,None)

        # in case there is not panorama in the site, therefore, continue
        if jsonData[‘panorama’]==None:
        continue
        else:
        panoInfo = jsonData[‘panorama’][‘data_properties’]

        *****

      • That error has to do with the line:

        urlAddress = r’https://maps.googleapis.com/maps/api/streetview/metadata?location={lat},{lon}&key=**’ % (lat,lon)

        Which is wrong as written. Again I think it is easier to just do f string substitution (since you are on python 3 it is available):

        urlAddress = f’https://maps.googleapis.com/maps/api/streetview/metadata?location={lat},{lon}&key=**’

        You should have enough though with my post to figure out the Json on your own, it is a pretty simple return nested dictionary, and it looks like you have the error condition as well.

    • You may also be interested in this post, https://andrewpwheeler.com/2020/10/24/using-the-google-vision-and-streetview-api-to-explore-hotspots/. I use the Google Vision API to get labels for items in the image, which includes several green space related items.

      Reply
      • Fatin Abdilah

         /  January 7, 2021

        Dear Andrew,

        I managed to get an updated version of the script. However, I haven’t been able to execute it correctly using bash. The author mentioned an example to call the inputshp, outdir, key variable:

        $ python metadataCollector.py –inputshp=’C:/Users/senseablecity/Dropbox (MIT)/ResearchProj/treepedia/cities-proj/Oakland/OaklandSlowStreets/SlowStreets_points/SS_20m.shp’ –outdir=’C:/Users/senseablecity/Dropbox (MIT)/ResearchProj/treepedia/cities-proj/Oakland/OaklandSlowStreets/SlowStreets_points’ –key=’***’

        However the log mentioned that it couldn’t recognise my C:/Users/… directory. Could you show me the correct way to call data for the argparse with path directories?

        **** the updated script****

        def GSVpanoMetadataCollector(samplesFeatureClass,num,ouputTextFolder,key):
        ”’
        This function is used to call the Google API url to collect the metadata of
        Google Street View Panoramas. The input of the function is the shpfile of the create sample site, the output
        is the generate panoinfo matrics stored in the text file

        Parameters:
        samplesFeatureClass: the shapefile of the create sample sites
        num: the number of sites proced every time
        ouputTextFolder: the output folder for the panoinfo

        ”’
        import urllib,urllib2
        #import urllib.request
        #import google_streetview.api
        import xmltodict
        import ogr, osr
        import time
        import os,os.path
        import sys
        import json

        if not os.path.exists(ouputTextFolder):
        os.makedirs(ouputTextFolder)

        driver = ogr.GetDriverByName(‘ESRI Shapefile’)

        # change the projection of shapefile to the WGS84
        dataset = driver.Open(samplesFeatureClass)
        layer = dataset.GetLayer()

        #sourceProj = layer.GetSpatialRef()
        #targetProj = osr.SpatialReference()
        #targetProj.ImportFromEPSG(4326)
        #transform = osr.CoordinateTransformation(sourceProj, targetProj)

        # loop all the features in the featureclass
        feature = layer.GetNextFeature()
        featureNum = layer.GetFeatureCount()
        #batch = featureNum//num
        batch = int(featureNum/num + 0.5)

        print(batch)
        for b in range(batch):
        # for each batch process num GSV site
        start = b*num
        geom.Transform(transform)
        lon = geom.GetX()
        lat = geom.GetY()

        #get the metadata of panoramas
        urlAddress = r’https://maps.googleapis.com/maps/api/streetview/metadata?size=600×300&location=%s,%s&heading=-45&pitch=42&fov=110&key=%s’%(lon, lat, key)
        time.sleep(0.1)

        time.sleep(0.05)
        #the output result of the metadata is a xml object
        #using different url reading method in python 2 and python 3
        if sys.version_info[0] ==2:
        request = urllib.request.Request(urlAddress)
        metaData = urllib.request.urlopen(request).read()

        data = json.loads(metaData)
        panoDate = data[‘date’]
        panoId = data[‘pano_id’]
        panoLat = data[‘location’][‘lat’]
        panoLon = data[‘location’][‘lng’]

        lineTxt = ‘panoID: %s panoDate: %s longitude: %s latitude: %s\n’%(panoId, panoDate, panoLon, panoLat)
        panoInfoText.write(lineTxt)

        panoInfoText.close()

        # ————Main Function ——————-
        if __name__ == “__main__”:
        import os, os.path
        import sys
        import argparse

        parser = argparse.ArgumentParser(
        description = “parameters”
        )

        parser.add_argument(
        “–inputshp”,
        required = True,
        type = str,
        help = “the path of the shapefile”
        )

        parser.add_argument(
        “–outdir”,
        default = “mosaic-mrt.tif”,
        help = “the output dir for the meta text file”,
        type = str,
        )

        parser.add_argument(
        “–key”,
        default = “”,
        help = “Google Street View key”,
        type = str,
        )

        args = parser.parse_args()
        inputshp = args.inputshp
        outdir = args.outdir
        key = args.key

        print(inputshp, outdir, key)
        GSVpanoMetadataCollector(inputshp, 1000, outdir, key)

        ****end of script***

      • In the command line don’t do the equal signs:

        python script.py –inputshp ‘C:/Users/whatever’

        Your script it doesn’t look like to me it actually writes to the results to the specified output directory.

  8. eddie

     /  October 11, 2021

    I have same issue in treepedia.
    Thanks to you, I’ve figured out what the cause is, but haven’t been able to solve it yet.
    did you ever solve…??

    Reply
  1. Using the Google Vision and Streetview API to Explore Hotspots | Andrew Wheeler
  2. Using google places API in criminology research? | Andrew Wheeler

Leave a Reply to hieund198 Cancel reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: