Little Fixes

Monday, November 25, 2024

Update: shared configuration files between client and server

Often the first step, in the sequence of developing a webapp, is to create a central description, i.e. a config file, which can act as a spine, as you add features and parameterize them.

It needs to be available to both your server code (in my current case Python 3.11 with Flask on Google App Engine), and the Javascript in any HTML file. If there are genuine and serious dials to be turned in the config file, both the client and server will typically be involved.

I wrote about this for an earlier, deprecated Google App Engine stack, here.

So this is an update.

app.yaml

runtime: python311

app_engine_apis:
  - url_fetch

handlers:
  - url: /my_config.yaml
    static_files: my_config.yaml
    upload: my_config.yaml
    secure: always

  - url: /.*
    script: auto
    secure: always
    redirect_http_response_code: 301

requirements.txt

flask
pyyaml
appengine-python-standard>=1.0.0
werkzeug==2.2.2

main.py

import yaml
from flask import Flask, Response, render_template, request

app = Flask(__name__)

# Load the YAML file at startup
with open('my_config.yaml', 'r') as file:
    shared_data = yaml.safe_load(file)

@app.route('/')
def index():
        return render_template('index.html')

@app.route('/api/shared-data')
def shared_data_api():
        yaml_data = yaml.dump(shared_data)
        return Response(yaml_data, mimetype='application/x-yaml')

if __name__ == '__main__':
    app.run()

templates/index.html

<!DOCTYPE html>
<html>
<head>
  <title>shared YAML example</title>
</head>
<body>
  <h1>shared YAML example</h1>
  <div id="output"></div>

  <!-- Include js-yaml library -->
  <script src="https://cdn.jsdelivr.net/npm/js-yaml@4/dist/js-yaml.min.js"></script>

  <script>
    // Fetch the YAML file directly
    fetch('/my_config.yaml')
      .then(response => response.text())
      .then(yamlText => {
        const data = jsyaml.load(yamlText);
        console.log('Data from YAML file:', data);
      })
      .catch(error => console.error('Error fetching YAML file:', error));

    // Fetch the YAML data from API endpoint
    fetch('/api/shared-data')
      .then(response => response.text())
      .then(yamlText => {
        const data = jsyaml.load(yamlText);
        console.log('Data from API endpoint:', data);
        const outputDiv = document.getElementById('output');
        outputDiv.textContent = JSON.stringify(data, null, 2);
      })
      .catch(error => console.error('Error fetching YAML from API:', error));
  </script>
</body>
</html>

my_config.yaml

application: google
        - this
        - could
        - be
        - anything

Tuesday, July 02, 2024

Morphology and Software Development

A new, popular library or framework isn't necessarily better than all the old ones. That's a corollary to this: success doesn't necessarily indicate the best quality. Typically, the new platform technology simply won: on a lumpy playing field against other technologies, perhaps through association with a performant platform, or via momentum from a community, a sponsor, or a monopoly. When Google Cloud Platform moved from webapp2 to Flask, it wasn't justified on scientific or engineering or psychological grounds. There was no attempt to make a case: Flask simply became more popular, and Google didn't properly fund webapp2. At the same time, for similar reasons, we had to move from Python 2 to 3, and from datastore db to ndb.

Let me demonstrate two ways in which webapp2 was better. These are morphological observations, or more precisely psychological observations about morphology. That is, I'm talking about the shapes we perceive when we look at our code, and the consequences for our minds. It's really the same approach we take to, for example, understanding how we feel when we walk down a beautiful street, or enter a pleasant town square. Those have a positive effect on our mental life for sometimes very simple reasons: a safer pedestrian environment, more interesting things to see at a human-scale, trees, shade, water, a sense that people care, and a sense of peace. This kind of analysis is pretty common in the UX and design world, but for some reason, it's barely discussed in the world of software development environments. The "user" experience gets a lot of attention. The "developer as user" experience: not so much.

The first thing lost with webapp2 was the route table. These exist in plenty of other programming environments, but now it's forgotten in Google's serverless python environment. It used to look like this:

# handler blocks

# handler block for People
Class People(webapp2.RequestHandler):
def get(self, string):
# code
def post(self):
#code
def people_get_function():
# code
def people_set_function():
# code
def other_people_functions_etc():
...
#lots of other handler blocks
...
# route table
app = webapp2.WSGIApplication([('/people/(.*)', People),
('/neighborhoods/(.*)', Neighborhoods),
('/cities/(.*)', Cities),
... )])

The shape of this code lends itself to clusters of functionality. But equally important:

Do you need to look up how you handle people? Look at the route table.

It's like the table of contents in a book. It lets you find the chapter about people, neighborhoods, cities, etc. ... no matter how idiosyncratically you mapped the world onto your program.

Instead, the approximate equivalent in flask looks like this:

@app.route('/people')

def people_response():

# code

@app.route('/neighborhood')

def neighborhood_response():

# code

@app.route('/city')

def city_response():

# code

On first blush, this puts everything, even the external API presentation or URL endpoints, in the same place as the functionality. But it's atomizing. In a complex application, you'll now be looking through all of your code for the route, which is a conceptual pointer to an idea, whose name you might not even remember, and it could be in a large file or many small ones. That was something you didn't need to do before. You simply needed to look through your route table, which is small, to re-orient yourself in your appplication, like an index or overview. The orienting qualities of such a global shape are obvious, but obviously not well-enough appreciated. In Flask now, you're forced to read everything to find what you want. Did you separate everything into different files? Then you'd have to read all the file names, which are ordered alphabetically, which is no order at all when you can't remember what word you're looking for. It's an unecessary tax on memory. When you have many such applications to build and maintain, you'd have no easy way to "get back up to speed" about your own code, which you might not have looked at for months. Because you can no longer lean on a simple, helpful shape.

Here's a second advantage webapp2 has over Flask. Flask has no built-in class model for the primary network commitment of every server-side application: to build a response to a request.

In webapp2, there was a "self". In this self are all the values that came from the internet. This is also where, in the course of the application, you'll put all of the resources that you'll pass back in your response. This was a simple idea: one part of your server app could focus on handling cookies and authentication, another could focus on stored values to make some sub-domain of the application work, a third would provide the text or image content, etc.

In flask, to get this same functionality, you need to try to find the appropriate place to allocate a response object: but where? So you need an init that gets called everywhere, and flags to see what you've done already ... it's a mess.

This is also morphological ... flask scatters and brings forward a task that was a background assumption in webapp2.

There are many good, humane ideas that disappear in computing. Typically, they are rediscovered in some form in the future, since these good ideas, at least the morphological-psychological ones (morpho-psychological?), are based on what is comfortable for human perception and cognition.

But it would be nice if software developers across the board would advocate a bit harder for their "UX" comfort. This is one topic of a seminar I host at the Building Beauty school of architecture, which we call Beautiful Software. Computer people can study both nature and beauty, to improve understanding of their innate, natural sensitivities. It would change everything if they did: not only their code.

Monday, July 01, 2024

Google Datastore URL-safe keys: incompatibilities from db to ndb inaccessible to LLMs

This is a story about accidental, unnecessary platform migration incompatibilities, which turned into problems undiscovered by LLMs with access to the relevant corpus.

On Google App Engine, in Python 2.7 using webapp2 and db, I had to do this sort of thing, to get a list of the URL-safe keys for a set of entities:

return_string = ""
cities = db.GqlQuery("SELECT * FROM City ORDER BY name ASC")
for c in cities:
return_string = return_string + str(c.key()) + ","
return return_string

Now, moving to python 3, with flask, and ndb:

return_string = ""
with client.context():
cities = City.query().order(City.name).fetch()
for c in cities:
return_string = return_string +
c.key.to_legacy_urlsafe(location_prefix="s~").decode('utf-8') + ","
return return_string

... and I'm sure this won't be the end of the saga.

Datastore, which is now the datastore mode of the firestore database at google, is accessed now through a library called ndb. But when you look at the database viewer on Google Console, the UUID that represents the data entity -- which is called the URL-safe key or reference -- is not the same as the value you extract normally using the programmatic interface of ndb. The numbers are just different. For the same entity.

I was quite surprised that neither Google's AI (Gemini) nor OpenAI's ChatGPT 4o could make heads, nor tails, of this problem. The code is Google's, and Google keeps it on Github, along with a long conversation thread about the problem by someone poking the datastore team until the "legacy" method was implemented. But Microsoft bought GitHub in order to feed all of this sort of information to OpenAI. So, why could it not solve the above problem?

Partly, it's because there was no real documentation of the problem: just a polite complaint that turned into a conversation which would be incomprehensible to an LLM without the context of actually building an application around this feature/bug, and having it fail. So, LLMs are still bound to have trouble from a lack of interaction with their fellow machines, and the lack of human narration for that interaction when it IS allowed.

In the meantime, their vast capacity for reading technical material cannot solve difficult problems like this for us, where the expectation (that a UUID would look the same no matter which library is looking at it) is just an obvious assumption, but not really described by anybody, hence inaccessible to an LLM.

Saturday, April 06, 2024

Python 2.7 -> 3.8 on App Engine. Ordered checklist with an unfolding stub.

The context here is a kind of "resource pump" webapp, a common enough migration of early static web sites, during the mid-web-2.0-era (say 2008), to Python 2 with WSGI on Google App Engine. The simple python program does very little, allowing the app.yaml file to define the work, serving folders full of static files.

So, how do we move these sites to App Engine with Python 3.8 and Flask 2?

Here's an ordered checklist, that is, a set of instructions with ordered dependencies.

That's also typical of unfolding sequences in software, but in the latter, there tends to be more creativity and judgment involved ... and the steps are not instructions ... instead they're helpful issues to consider at that moment.

Most of the steps below simply need to happen, in that order. So they are instructions.

There's only one somewhat creative step here (5) ... yet it highlights the point where more steps might be written and inserted, to serve a wider range of migrations.

I assume this is a directory with git source control.

If so:
git commit -a -m 'starting migration from python 2.7'
If not:
git init
git add app.yaml
git commit -a -m 'starting source control'
mkdir templates
mv index.html templates
and git add templates/index.html
add requirements.txt
and git add requirements.txt
add main.py (if there's a route table in the WSGI version, move the routes to flask. This is the one creative task. It's kind of a stub: this is where all the creative tasks in a sequence would go, to serve a greater range of programs.)
and git add main.py
change the app.yaml head
change the app.yaml tail
and git commit -m 'first python 3.8 changes'
(if it makes sense, set up the local test environment)
(if it makes sense, run gunicorn & test) Note that gunicorn does not use app.yaml, so your mileage may vary, in using this local test environment. If you don't need to debug the server-side python, see deployment test steps 12-14
(if you created a virtual environment, add the <project env> directory (see below) to .gcloudignore)
gcloud app deploy --project <migrating site> --no-promote
Go to (cloud console->app engine -> versions), find the new version, launch and test
Is the test good? Select the new version and click “migrate traffic”.

If you want to setup a local test environment (again, useful if there's more server code to test):

virtualenv -p python3.8.2 <project env>
source ./<project env>/bin/activate
pip install -r requirements.txt
(or
pip install gunicorn
pip install flask
pip install google-cloud-datastore
pip list
)
gunicorn -b :8080 main:app
(test in browser at localhost:8080)
^c
deactivate

old app.yaml head:

runtime: python27
api_version: 1
threadsafe: false

new app.yaml head:

runtime: python38
app_engine_apis: true

old app.yaml tail:

- url: /.*
  script: <migrating site>.app
  secure: always
  redirect_http_response_code: 301

new app.yaml tail:

- url: /.*
  script: auto
  secure: always
  redirect_http_response_code: 301

new requirements.txt:

Flask==2.2.2
google-cloud-datastore==2.7.0
appengine-python-standard>=1.0.0
google-auth==2.17.1
google-auth-oauthlib==1.0.0
google-auth-httplib2==0.1.0
werkzeug==2.2.2

And here's the one creative step in this checklist's sequence: migrating the route table. It's only a stub for further creative-and-judged unfolding steps, if one is migrating server-side application logic:

new main.py:

from flask import Flask, render_template, request
app = Flask(__name__)

@app.route('/')
@app.route('/endpoin_one')
@app.route('/endpoint_two')
def root():
    # NB: index.html must be in /templates
    return render_template('index.html')
if __name__ == '__main__':
    app.run()

old <migrating site>.py:

# universal index.html delivery
# in python27 as a service
# on Google App Engine
import cgi
import os
import webapp2
from google.appengine.ext.webapp import template
from google.appengine.api import users
class MainPage(webapp2.RequestHandler):
    def get(self):
        template_values = {
            }
        path = os.path.join(os.path.dirname(__file__), 'index.html')
        self.response.out.write(template.render(path, template_values))
app = webapp2.WSGIApplication(
                                     [('/', MainPage)
                                     ,('/endpoint_one',MainPage)
                                     ,('/endpoint_two',MainPage)
                                      ],
                                     debug=True)

Friday, March 15, 2024

Broken or erratic or unreliable visual editor (emacs, vim, etc.) over ssh in Mac terminal?

This is a rather specific problem.

But I couldn't find mention of it anywhere.

The MacOS terminal implementation has difficulty when the window, or more precisely the amount of data stored in the terminal process, gets very large ...

... say you've been using it for days, and lots of output has scrolled up, but you haven't opened a new terminal window ... maybe because you want to look at what you've done already. You could export it, but then you'd have to think about where to put that exported data.

Mac terminal tends to be a bit greedy of RAM, and if you have lots of these terminal windows, managing lots of projects, you may see some performance degradation.

... but, also, you may see some actual problems.

For example, if one of these terminals is connected to a remote host using ssh, and you start to use a visual editor (say, emacs, vi, or vim) on the remote machine, the editor might start to make errors, and become essentially unusable. The remote editor has expectations for terminal text control signals, and Mac terminal, unable to work fast enough or buffer signals reliably, because of the large volume of text in the current process, simply fails to keep sync.

So, you're not crazy. Not everything in computing is deterministic, especially when networking is involved. (And yes, Apple could and should fix this problem.)

For now, simply close the remote host, export the terminal text, close the terminal, and open a new one.

Wednesday, November 22, 2023

Fixing software updates by playing with hardware

The "Epson Perfection V600 Photo" is a fine scanner, but the software often fails it, and the user.

This is true after all the most recent updates have been applied, regardless of which scanning app is used.

The error is:

"Unable to send data. Check the connection to the scanner and try again. E583-B318"

It reports this error even while visibly communicating with the scanner, with various whirring and clicking as evidence.

Not so helpfully, the error prevents scanning from taking place, after which the scanning app terminates.

With a bit of oddball tinkering, I found a "fix". I'll use the transparency scanner as an example.

If you open the lid of the scanner, and try to scan, you get this error:

"Remove the document mat from the scanner."

But this is not a fatal error. Close the scanner lid, click "ok", and try your scan.

For me (hopefully others) the scanning then works continually, until I unplug the computer from the scanner, or quit the scanning app.

I'll let the reader draw their own conclusions about the level of investment Epson makes to assure software quality ...

Friday, October 20, 2023

gmail broken with arbitrary, invisible, forced short autowrap

... or that's how it looked to me.

Gmail compose took a long input line, which normally would autowrap within whatever box it was viewed, but instead created a hard, short autowrap, in the background, where I could not see it, and did not want it.

Obviously I'd accidental changed a setting. But I scoured the settings, and couldn't see an option that fit the problem.

There's another set of settings, weirdly not referenced or linked in the main settings. These are in the compose window. And they don't apply until the next time you open the compose window.

Those settings are under the three dots to the right of the text tools, and the culprit was "plain text mode".

Now as someone who used email decades before there was email formatting, I was a little irked by the assertion implied by this setting's name.

If it was 'plain text' why not just treat the input the way it will be received? Why create an arbitrarily short autowrap of the input text, which will alway look wrong? This is because it's not previewed, that is, it's not WYSIWYG. It turns a potentially useful option into one that would only be useful for sending emails to very primitive small-screen devices, with no option to use plain text in a way that's under control of the sender.

So, a broken UX in gmail. Which usually is more careful about its features.

Sunday, October 01, 2023

Werkzeug ... and who is responsible for code stability?

Don't you love it when you haven't deployed for a few days, and you change something insignificant, and then your deployed app crashes, because of something far outside your purview?

The python Werkzeug WSGI library was just updated to 3.0. This caused Python Flask 2.2.2 web apps running on Google Cloud's App Engine to automatically update. Which, if you use one of the utilities, url_quote, you get this error:

ImportError: cannot import name 'url_quote' from 'werkzeug.urls'

So, yes, I might have caught this by updating and running it first in my local environment. But Google Cloud could have caught this too, creating a stable environment for incremental deployment.

The fix is to add this line to your requirements.txt file:

werkzeug==2.2.2

This reminds me of the whole unnecessary forced move to Flask, with its mixed bag of improvements and problems. It should be possible to run a webapp of any age (at least with configurations since 2008) in Google App Engine. Why all the unnecessary updating, crashing, and subsequent compulsory code obsolescence? What happened to backwards compatability? If it was still an observed principle, it would be easier now than ever. Why the insistence on forcing programmers to chase after the latest thing?

Saturday, June 17, 2023

Error messages, Google Maps, PinElement, AdvancedMarkerElement, MapId

I often try using a new feature by adding the most local code possible, that is, right where I need it. It fails, of course. The UX for developers is very rarely a priority. When it fails, I then fix one problem at a time, step-by-step, slowly getting the feature to do its job, by working my way outwards from that first bit of code, and seeing what kinds of errors I get. I want to see whether the errors point me to the next-most general problem or, instead, obscure what's actually going on. As everyone knows, it's usually the latter.

I have a hope that someday error messages will become genuinely useful, and not just strings that we need to search for in Google, in the hope that someone like me has explained the error message, and how to make the necessary repairs.

In this case, quite a lot of time could have been saved if the Google Maps API (which is loaded) had simply caught the exceptions and told me "do (1), then (2), then (3) ..."

Ah well. Developer advocates clearly have no power inside such giant technocracies.

The goal: I wanted a custom marker. The 'free' way to do this is by adding an 'image' name-value pair in a marker:

const bluepin = "/bluepin.png";
...
 marker = new google.maps.Marker({position,map,icon: bluepin});

But there's a new set of advanced marker features. The documentation didn't match my use case (my markers are created in a listener), so I started with:

const pinBackgroundBlue = new PinElement(
    {background: "blue"});
    ...
marker = new AdvancedMarkerElement(
    {map,position: position,
     content: pinBackgroundBlue.element});

... and in the console, Javascript told me that pinElement was not found. Fair enough.

In the head element I added an import of the appropriate Google Maps libraries:

<script async defer src="https://maps.googleapis.com/maps/api/js
key=YOUR_GOOGLE_MAPS_KEY
&v=beta
&libraries=visualization,marker
&callback=initMap">
</script>

Now, in the function initMap itself, I didn't change the map declaration.

The map is global, however.

In the listener, I added the recommended import mappings:

// Request needed libraries. const { Map } = await google.maps.importLibrary("maps"); const { AdvancedMarkerElement, PinElement } = await google.maps.importLibrary( "marker" );
But since this was a listener, javascript gave me an error. The enclosing function wasn't async, so it could not await.

So I did this, which worked:

const { Map } = google.maps.importLibrary("maps");
const { AdvancedMarkerElement, PinElement } = google.maps.importLibrary("marker");

Now it didn't complain about await / async.

But still: "PinElement not found".

Maybe the namespace is still isolated? I really don't know how much "importLibrary" is supposed to do. So I tried:

const pinBackgroundBlue = new window.google.maps.marker.PinElement(
{background: "blue"});

Now it found PinElement!

But it said "PinElement is not a constructor".

Well ... that's just silly. Of course it is.

So, I read the documentation, watched the Google Maps videos ...

And the only thing that could be missing was something that was new to me:

A "MapId".

This is for elite maps, I suppose, since Google requires that the privilege of a generated MapId is added to your billing account. I don't know what the charge is now, or later.

But if you just want to try it, add the following to the name-value pairs in your map declaration:

mapId: 'DEMO_MAP_ID'

And now PinElement is a constructor! And your custom pin will appear on the map.

Sunday, April 23, 2023

Deleting a Google Cloud Platform domain mapping under Google App Engine

Google's current documentation on this topic doesn't match the actual user interface, at least for Google App Engine custom domain mappings.

Say you want to free up a domain that you've used on a different project on GCP or GAE.

When you try to use the domain, you get an error like:

error: [domain] is already mapped to a project

To solve this, use the command line (assuming you have the developer tools and so use gcloud regularly to deploy your apps) and do the following:

gcloud auth login

This will open a browser window for you to sign in with your Google account.

Set the project in the gcloud tool to the old project where the domain is currently mapped:

gcloud config set project PROJECT_ID

Replace PROJECT_ID with the actual Project ID of the old project.

List the domain mappings for the old project:

gcloud app domain-mappings list

Locate the domains you want to remove from the list, and then run the following command to delete the domain mapping:

gcloud app domain-mappings delete DOMAIN_NAME

... replacing DOMAIN_NAME with the actual domain you want to remove.

After a bit of waiting, you'll be able use the domain again in a custom mapping.

Thursday, April 20, 2023

Simplest Google App Engine Python 3.8 Flask web app with Google Identity

Tested as of this post, with Google's latest approach to sign-in ... as far as I know. With the latest python 3.8 and flask resources (as vanilla as possible ... as far as I know).

The proper sequence (skip anything you've done already):

1. sign up for Google App Engine

2. put down a credit card

3. download developer tools

4. create an app

5. enable all the APIs and Services you plan to use

(e.g., Firestore's Datastore interface, in the case below).

6. go to API Credentials on Google Cloud Console

7. add an OAuth 2.0 Client ID

8. authorize your app for javascript origins (you'll use it eventually) but most importantly for this app, authorize the redirect URIs for:

https://app project name.appspot.com

https://app project name.appspot.com/oauth2callback

(and any domains you map to that application)

9. Everything in red on this post needs to be replaced with your own values.

This app loads index.html when not signed in, offers a sign-in link, and when the user has signed in with google, index.html includes a home page with the user info we have access to.

app.yaml

runtime: python38
app_engine_apis: true

env_variables:

CLIENT_ID: 'your client ID copied from your google cloud console'
CLIENT_SECRET: 'your client secret copied from your google cloud console'
SECRET_KEY: "the super secret key you make up"

instance_class: F1

requirements.txt

Flask==2.2.2

google-cloud-datastore==2.7.0

appengine-python-standard>=1.0.0

google-auth==2.17.1

google-auth-oauthlib==1.0.0

google-auth-httplib2==0.1.0

werkzeug==2.2.2

main.py

import os
import requests
from flask import Flask, render_template, request, redirect, url_for, session
from google.oauth2.credentials import Credentials
from google_auth_oauthlib.flow import Flow
from google.cloud import datastore

app = Flask(__name__)

SECRET_KEY = os.environ.get("SECRET_KEY", "the super secret key you make up")
app.secret_key = SECRET_KEY

CLIENT_ID = os.environ.get('CLIENT_ID')
CLIENT_SECRET = os.environ.get('CLIENT_SECRET')
REDIRECT_URI = "https://app project name.appspot.com/oauth2callback"

@app.route('/')
def index():

if 'userinfo' in session:

return render_template("home.html", userinfo=session['userinfo'])

return render_template("index.html")

@app.route('/login')
def login():

flow = Flow.from_client_config(

{

"web": {

"client_id": CLIENT_ID,
"client_secret": CLIENT_SECRET,
"redirect_uris": [REDIRECT_URI],
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://accounts.google.com/o/oauth2/token",

}

},

scopes=[

"https://www.googleapis.com/auth/userinfo.email",
"https://www.googleapis.com/auth/userinfo.profile",
"openid",

],

)

flow.redirect_uri = REDIRECT_URI
authorization_url, _ = flow.authorization_url(prompt="consent")
return redirect(authorization_url)

@app.route('/sign_out')
def sign_out():
session.pop('userinfo', None)
return redirect(url_for('index'))

@app.route('/oauth2callback')

def oauth2callback():

flow = Flow.from_client_config(

{
"web": {

"client_id": CLIENT_ID,
"client_secret": CLIENT_SECRET,
"redirect_uris": [REDIRECT_URI],
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://accounts.google.com/o/oauth2/token",

}

},

scopes=[

"https://www.googleapis.com/auth/userinfo.email",
"https://www.googleapis.com/auth/userinfo.profile",
"openid",

],

state=request.args.get("state"),
)

flow.redirect_uri = REDIRECT_URI
flow.fetch_token(code=request.args.get("code"))
credentials = flow.credentials
userinfo = get_user_info(credentials)
session['userinfo'] = userinfo
return redirect(url_for('index'))

def get_user_info(credentials):

headers = {

"Authorization": f"Bearer {credentials.token}"

}
response = requests.get("https://www.googleapis.com/oauth2/v2/userinfo", headers=headers)
userinfo = response.json()
return userinfo

if __name__ == "__main__":

app.run(debug=True)

templates/home.html

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>your home page title</title>
</head>
<body>
<h1>Welcome, {{ userinfo['name'] }}</h1>
<h2>Email: {{ userinfo['email'] }}</h2>
<img src="{{ userinfo['picture'] }}" alt="Profile picture" width="100" height="100">

</html>

templates/index.html

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>application title</title>
<script src="https://apis.google.com/js/platform.js" async defer></script>
<meta name="google-signin-client_id" content="{{ CLIENT_ID }}">
</head>
<body>
<h1>Welcome to this web app</h1>
<a href="{{ url_for('login') }}">Sign in with Google</a>
</body>
</html>

Thursday, January 26, 2023

Datastore preservation: migrating Python 2.7 to Python 3 on Google App Engine

Let's say you deploy a production web application on Google App Engine, using a Python 2.7 runtime. That's getting quite old today, and there are increasing numbers of incompatibilities you need to cope with as a result. What to do?

It might seem daunting, and risky, to migrate to the rather different development environment of Google App Engine with the runtime of Python 3.x.

But if done properly, one worry, or risk, can be avoided. The data in datastore. If you have a great deal of it, it will be preserved during this migration, without the complications of an ETL process. If you translate your environment and webapp properly, the data doesn't go anywhere.

See for yourself.

Here are two webapp sequences, for the same simple three-tier application.

The first is a Python 2.7 application.

The second is a Python 3.x application.

It's worth creating this project youself, so you can feel confident in the migration.

After this, when working on the migration of your production application, it's important to take precautions -- such as duplication of the codebase, and copying the datastore instance -- to allow rollback.

But you probably won't need it.

--------------

Sequence 1

A simple three-tier application on Google App Engine with the webapp2 framework and the python 2.7 runtime. We assume you've created your project and enabled billing, in your Google Cloud Console.

create index.html

<html>

<head>

</head>

<body>

Visits {{visits}}

</div>

</body>

</html>

create <project id>.py

import cgi

import os

import webapp2

from google.appengine.ext.webapp import template

from google.appengine.api import users

from google.appengine.ext import db

class Visit(db.Model):

visitor = db.StringProperty()

timestamp = db.DateTimeProperty(auto_now_add=True)

def store_visit(remote_addr, user_agent):

Visit(visitor='{}: {}'.format(remote_addr, user_agent)).put()

def fetch_visits():

return str(Visit.all().count())

class MainPage(webapp2.RequestHandler):

def get(self):
store_visit(self.request.remote_addr, self.request.user_agent)
visits = fetch_visits();
template_values = { 'visits':visits}
path = os.path.join(os.path.dirname(__file__), 'index.html')
self.response.out.write(template.render(path, template_values))

app = webapp2.WSGIApplication(

[('/', MainPage)],

debug=True)

create app.yaml

runtime: python27

api_version: 1

threadsafe: false

handlers:

- url: /.*
script: <project id>.app
secure: always
redirect_http_response_code: 301

run "python --version"

Python 2.7.4

test your app locally with:

dev_appserver.py --log_level debug .

deploy with:

gcloud app deploy --project <project id>

Visit the live page a few times, and find your datastore entities from Google Cloud Console.

---------------------------------------------

Sequence 2

A simple three-tier application on Google App Engine with the flask framework and the python 3.8 runtime.

On your machine:

virtualenv -p python3.8.2 <project env>

source ./<project env>/bin/activate

pip install gunicorn

pip install flask

pip install google-cloud-datastore

pip list

Create from the list results, include these two versions in a file called "requirements.txt":

Flask==2.2.2

google-cloud-datastore==2.13.2

[if you do this repeatedly, in different local virtual environments, you end up running "pip install -r requirements.txt" often]

create "app.yaml"

runtime: python38

create "main.py"

from datetime import datetime, timezone

import json

import time

import google.auth

from flask import Flask, render_template, request

from google.cloud import datastore

app = Flask(__name__)

ds_client = datastore.Client()

def store_visit(remote_addr, user_agent):

entity = datastore.Entity(key=ds_client.key('Visit'))

entity.update({

'timestamp': datetime.now(timezone.utc),
'visitor': '{}: {}'.format(remote_addr, user_agent),

})

ds_client.put(entity)

def fetch_visits():

'get total visits'

query = ds_client.query(kind="Visit")

return len(list(query.fetch()))

@app.route('/')

def root():

store_visit(request.remote_addr, request.user_agent)
visits = fetch_visits()

# NB: index.html must be in /templates
return render_template('index.html',visits=visits)

if __name__ == '__main__':

app.run()

create "templates/index.html":

<!doctype html>

<html>

<head>

</head>

<body>

<p>{{ visits }} visits</p>

</div>

</body>

</html>

set project with:

gcloud config set project <project id>

(Note this needs to be run whenever you switch to working on a different project locally, to switch the datastore your local development environment is connected to, that is, to the project's datastore.)

set credentials

gcloud auth application-default login

(This will launch a browser -- log in, and 'allow.' Note this will connect your local environment with the remote datastore. If you want a local datastore, you need to use the datastore emulator).

run locally with:

gunicorn -b :8080 main:app

deploy with:

gcloud app deploy --project <project id>

deactivate

Check your database on google cloud console. It's still there!