GCP Fundamentals :: Core Infrastructure - Summary Notes

⇒ Cloud Source Repositories Concepts

Lots of GCP customers use Git to store and manage their source code trees by running their own Git instances or using a hosted Git provider
Running ones own Git instance is great because it gives total control. Using a hosted Git provider is great because it's less work
Cloud Source Repositories is a third option to keep code private to a GCP project and use IAM permissions to protect it, but not have to maintain ones own the Git instance
Cloud Source Repositories provides Git version control to support a team's development of any application or service, including those that run on App Engine, Compute Engine, and Kubernetes Engine
With Cloud Source Repositories, one can have any number of private Git repositories, which allows them to organize the code associated with their GCP project in whatever way works best for them
Cloud Source Repositories also contains a source viewer so that one can browse and view repository files from within the GCP console

⇒ Cloud Functions Concepts

Many applications contain event-driven parts. For example, maybe an application that lets users upload images. Whenever that happens, there maybe a need to process that image in various ways: convert it to a standard image format, thumbnail into various sizes, and store each in a repository. It would be great if one could write a single purpose function that did the necessary image manipulations and then arrange for it to automatically run whenever a new image gets uploaded. That is exactly what Cloud Functions does
With Cloud Functions one does not have to worry about servers or compute resources or runtime binaries or scaling. Just write the code in JavaScript for a Node.js environment that GCP provides and then configure when it should fire. There is no need for to pay for servers either. Just pay whenever the functions run in 100 millisecond intervals
To setup Cloud Functions, choose events that we care about. For each event type, tell Cloud Functions we are interested in it. These declarations are called Triggers. Then attach JavaScript functions to the Triggers . From then on, the functions will respond whenever the events happen

⇒ Deployment Manager Concepts

Setting up an environment in GCP can entail many steps: setting up compute network and storage resources, and keeping track of their configurations. One can do it all by hand if one desires to, taking an imperative approach. The same goes for change the environment from the old state to the new. To clone the environment, one has to repeat all those commands again. This is a lot of work. It is more efficient to use a Template. That means a specification of what the environment should look like. It is declarative rather than imperative. Using the Deployment Manager one can do just that
Deployment Manager is an Infrastructure Management Service that automates the creation and management of the GCP resources for the user
To use the Deployment Manager, one needs to create a template file using either the YAML markup language or Python that describes what the components of the environment should look like. Then provide the template to the Deployment Manager, which figures out and does the actions needed to create the environment the template describes. To change the environment, edit the template and then tell Deployment Manager to update the environment to match the change

Sample deployment template to create a GCE VM:

mydeploy.yaml

resources:
- name: my-vm
  type: compute.v1.instance
  properties:
    zone: us-central1-f
    machineType: zones/us-central1-f/machineTypes/n1-standard-1
    metadata:
      items:
      - key: startup-script
        value: "apt-get update"
    disks:
    - deviceName: boot
      type: PERSISTENT
      boot: true
      autoDelete: true
      initializeParams:
        sourceImage: https://www.googleapis.com/compute/v1/projects/debian-cloud/global/images/debian-9-stretch-v20180806
    networkInterfaces:
    - network: https://www.googleapis.com/compute/v1/projects/qwiklabs-gcp-4825df6c2b6443ce/global/networks/default
      accessConfigs:
      - name: External NAT
        type: ONE_TO_ONE_NAT

Create a VM from the mydeploy.yaml template using the Deployment Manager:

gcloud deployment-manager deployments create my-first-depl --config mydeploy.yaml

Deployment template with modifications to the GCE VM:

mydeploy.yaml

resources:
- name: my-vm
  type: compute.v1.instance
  properties:
    zone: us-central1-f
    machineType: zones/us-central1-f/machineTypes/n1-standard-1
    metadata:
      items:
      - key: startup-script
        value: "apt-get update; apt-get install nginx-light -y"
    disks:
    - deviceName: boot
      type: PERSISTENT
      boot: true
      autoDelete: true
      initializeParams:
        sourceImage: https://www.googleapis.com/compute/v1/projects/debian-cloud/global/images/debian-9-stretch-v20180806
    networkInterfaces:
    - network: https://www.googleapis.com/compute/v1/projects/qwiklabs-gcp-4825df6c2b6443ce/global/networks/default
      accessConfigs:
      - name: External NAT
        type: ONE_TO_ONE_NAT

Update the running VM using the modified mydeploy.yaml template using the Deployment Manager:

gcloud deployment-manager deployments update my-first-depl --config mydeploy.yaml

⇒ Stackdriver Concepts

Stackdriver is GCP's tool for monitoring, logging and diagnostics
Stackdriver gives one access to many different kinds of signals from the virtual machines, containers, middleware and application tier, logs, metrics and traces. It gives one insight into the application's health, performance and availability
The core components of Stackdriver are: Monitoring, Logging, Trace, Error Reporting and Debugging
Stackdriver Monitoring checks the endpoints of web applications and other Internet accessible services running on the cloud environment. One can configure uptime checks associated with URLs, groups or resources such as Instances and load balancers. One can set up alerts on interesting criteria, like when health check results or uptimes fall into levels that need action. One can use Monitoring with a lot of popular notification tools. And one can create dashboards to help visualize the state of the application
Stackdriver Logging lets one view logs from the applications as well as filter and search on them. Logging also lets one define metrics, based on log contents that are incorporated into dashboards and alerts. One can also export logs to BigQuery, Cloud Storage and Cloud PubSub
Stackdriver Error Reporting tracks and groups the errors in the cloud applications. And it notifies the user when new errors are detected
With Stackdriver Trace, one can sample the latency of app engine applications and report per-URL statistics
Stackdriver Debugger connects the applications production data to the source code. So one can inspect the state of the application at any code location in production. That means one can view the application stage without adding logging statements
Stackdriver Debugger works best when the application source code is available, such as in Cloud Source repositories. Although it can be in other repositories too
The following illustrates the various capabilities of Stackdriver:

Stackdriver Capabilities

Apache Hadoop is an open source framework for big data. It is based on the Map-Reduce programming model which Google invented and published. The Map-Reduce model at its simplest means that one function called the Map function runs in parallel with a massive dataset to produce intermediate results. And another function called the Reduce function builds a final result set based on all those intermediate results
Cloud Dataproc is a fast, easy, managed way to run Hadoop, Spark, Hive, and Pig on Google Cloud Platform. All one has to do is request a Hadoop cluster. It will be built in 90 seconds or less, on top of Compute Engine virtual machines whose number and type the user controls. The cluster can be scales up or down based on the need for more or less processing power while your cluster is running
Running these jobs in Cloud Dataproc allows users to only pay for hardware resources used during the life of the cluster they create. Although the rate for pricing is based on the hour, Cloud Dataproc is billed by the second. Cloud Dataproc clusters are billed in one-second clock-time increments, subject to a one minute minimum billing. Once the user is done with the use of the cluster, they can delete it, and billing stops. This is much more agile use of resources than on-premise hardware assets
Users can also save money by telling Cloud Dataproc to use preemptible Compute Engine instances for their batch processing. The users have to make sure that their jobs can be restarted cleanly, if they are terminated, and can get a significant break in the cost of the instances
Cloud Dataproc is useful for data mining and analysis with datasets of known sizes

Cloud Dataflow is particularly a good choice when data shows up in real time or it is of unpredictable size or rate
Cloud Dataflow is both a unified programming model and a managed service and it lets one develop and execute a big range of data processing patterns: extract, transform, and load batch computation and continuous computation
One can use Cloud Dataflow to build data pipelines that work for both batch and streaming data
Cloud Dataflow fully automates the management of whatever processing resources are required. Cloud Dataflow frees one from operational tasks like resource management and performance optimization
In the following example, Cloud Dataflow pipeline reads data from a Big Query table (source), applies various processing steps such as map operations and some reduce operations (transformations), and writes its output to a cloud storage (sink)
Each step in the Cloud Dataflow pipeline is elastically scaled. There is no need to launch and manage a cluster. Instead, the service provides all resources on demand. It has automated and optimized worked partitioning built in, which can dynamically rebalance lagging work. That reduces the need to worry about hotkeys (situations where disproportionately large chunks of the input get mapped to the same cluster)
Cloud Dataflow is a general purpose ETL tool and its use case as a data analysis engine comes in handy in things like fraud detection and financial services, IoT analytics and manufacturing, healthcare and logistics and click stream, point of sale and segmentation analysis in retail
The following illustration depicts a typical Cloud Dataflow pipeline:

Dataflow Pipeline

BigQuery is Google's fully-managed, petabyte-scale, low-cost analytics data warehouse using which one can do ad-hoc SQL queries on a massive data set
Because there's no infrastructure to manage, one can focus on analyzing data to find meaningful insights, use familiar SQL and take advantage of our pay-as-you-go model
One can get data into BigQuery from Cloud Storage or Cloud Datastore, or stream it into BigQuery at up to 100,000 rows per second
Using BigQuery one can run super-fast SQL queries against multiple terabytes of data in seconds using the processing power of Google's infrastructure
In addition to SQL queries, one can easily read and write data in BigQuery via Cloud Dataflow, Hadoop, and Spark
BigQuery offers free monthly quotas, allows one to scale seamlessly, and has a 99.9 percent service level agreement
BigQuery lets one specify the region where data will be kept. For example, if a user wants to keep data in Europe, they don't have to go set up a cluster in Europe. They just specify the EU location where they create their data set
Because BigQuery separates storage and computation, users pay for their data storage separately from queries. That means, users pay for queries only when they are actually running. They have full control over who has access to the data stored in BigQuery, including sharing data sets with people in different projects
If a user X shares some data sets, it won't impact their cost or performance. People they share with pay for their own queries, not the user X
Long-term storage pricing is an automatic discount for data residing in BigQuery for extended periods of time. When the age of the data reaches 90 days in BigQuery, Google will automatically drop the price of storage

Cloud Pub/Sub is a messaging service that allows one to work with events in real time
Cloud Pub/Sub meant to serve as a simple, reliable, scalable foundation for stream analytics
Cloud Pub/Sub allows independent applications to send and receive messages. That way they're decoupled, so they scale independently
The Pub in Pub/Sub is short for publishers and Sub is short for subscribers
Cloud Pub/Sub is designed to provide at least once delivery at low latency which means that there is a small chance some messages might be delivered more than once. So, this has to be kept in mind when designing applications
Cloud Pub/Sub offers on-demand scalability to one million messages per second and beyond. One just needs to choose the quota they want
Cloud Pub/Sub is an important building block for applications where data arrives at high and unpredictable rates, like Internet of Things systems
For analyzing streaming data, Cloud Dataflow is a natural pairing with Cloud Pub/Sub
Cloud Pub/Sub can be configured for subscribers to receive messages on a push or pull basis. In other words, subscribers can get notified when new messages arrive for them or they can check for new messages at intervals

Cloud Datalab lets one create and maintain web-based notebooks containing Python code that can be run interactively and view the results
Cloud Datalab runs in a Compute Engine VM and one gets started by specifying the virtual machine type and what GCP region it should run in
Cloud Datalab presents an interactive Python environment that's ready to use. And it orchestrates multiple GCP services automatically, so the users can focus on exploring their data
Cloud Datalab users only pay for the resources they use. There is no additional charge for Cloud Datalab itself. It's integrated with BigQuery, Compute Engine, and Cloud Storage, so accessing their data doesn't run into authentication hassles
Cloud Datalab users can visualize their data with Google Charts or matplotlib

Machine Learning is one branch of the field of artificial intelligence. It is a way of solving problems without explicitly coding the solution. Instead, human coders build systems that improve themselves over time through repeated exposure to sample data, which is called training data
Cloud Machine Learning Platform provides modern machine learning services with pre-trained models and a platform to generate custom tailored models
TensorFlow is an open source software library that is exceptionally well suited for machine learning applications like neural networks. It was developed by Google their internal use and then open sourced so that the world could benefit
One can run TensorFlow wherever they like but GCP is an ideal place for it because machine learning models need lots of on-demand compute resources and lots of training data. TensorFlow can also take advantage of Tensor Processing Units (TPUs), which are hardware devices designed to accelerate machine learning workloads with TensorFlow. GCP makes them available in the cloud with Compute Engine virtual machines. Each cloud TPU provides up to 180 teraflops of performance. And because users pay for only what they use, there's no upfront capital investment required
Cloud Machine Learning Engine lets users easily build machine learning models that work on any type of data of any size. It can take any TensorFlow model and perform large-scale training on a managed cluster
Machine Learning fall into two categories, depending on whether the data they work on is Structured or Unstructured. Based on structured data, one can use ML for various kinds of classification and regression tasks like customer churn analysis, product diagnostics and forecasting. It can be the heart of a recommendation engine for content personalization and cross-sells and up-sells. One can use ML to detect anomalies, as in fraud detection, sensor diagnostics or log metrics. Based on unstructured data, one can use ML for image analytics such as identifying damaged shipment, identifying styles and flagging content. One can do text analytics too, like a call center, blog analysis, language identification, topic classification and sentiment analysis
The following illustration depicts the high-level view of the Machine Learning Platform :

Machine Learning Platform

The Cloud Vision API enables developers to understand the content of an image. It quickly classifies images into thousands of categories - sailboat, lion, Eiffel Tower - detects individual objects within images, and finds and reads printed words contained within images. One can use it to build metadata on your image catalog, moderate offensive content or even do image sentiment analysis
The Cloud Speech API enables developers to convert audio to text. Because we have an increasingly global user base, the API recognizes over 80 languages and variants. One can transcribe the text of users, dictating in an applications' microphone, enable command and control through voice or transcribe audio files
The Cloud Natural Language API offers a variety of natural language understanding technologies to developers. It can do syntax analysis, breaking down sentences supplied by our users into tokens, identify the nouns, verbs, adjectives, and other parts of speech and figure out the relationships among the words. It can do entity recognition. In other words, it can parse text and flag mentions of people, organizations, locations, events, products, and media. It can understand the overall sentiment expressed in a block of text. It has these capabilities in multiple language, including English, Spanish, and Japanese
The Cloud Translation API provides a simple, programmatic interface for translating an arbitrary string into a supported language. When we don't know the source language, the API can detect it
The Cloud Video Intelligence API lets one annotate videos in a variety of formats. It helps one identify key entities - that is, nouns - within their video and when they occur. One can use it to make video content searchable and discoverable