How to build unified Batch & Streaming Pipelines with Apache Beam and Dataflow

How to build unified Batch & Streaming
Pipelines with Apache Beam and Dataflow 1
Welcome to ServerlessToronto.org

Serverless Evolution (since FaaS started)
2
Serverless is New Agile & Mindset
#1 We started as Back-
end FaaS (Serverless)
Developers who enjoyed
“gluing” other people’s
APIs and Managed
Services)
#3 We're obsessed by
creating business value
(meaningful MVPs,
Products), focusing on
Outcomes/Impact –
NOT Outputs
#2 We build bridges
between Serverless
Community (“Dev leg”),
and Front-end, Voice-First
& UX folks (“UX leg”)
#4 Achieve agility NOT by
“sprinting” faster (like in
Scrum) but working smarter
(by using bigger building
blocks and less Ops)

Disconnect between IT & Business needs
3
Our group became dedicated to reducing the
Businesses & IT Gap!
Technology is not the point => We are here to create Value
Adopting Serverless Mindset allowed us to shift the focus from “pimping up our
cars” (infrastructure/code), towards “driving” (the business) forward.
≠

Upcoming ServerlessToronto.org Meetups?
4
We’ll remain online in 2022. We’re working on:
• “What, Why, Who & How of CDPs” w/ SEGMENT
• Joe Emison – his new book coming soon!
• Lak’s 2nd Ed book “DS on the GCP” out on May 3
• AWS Cloud: “Event-driven integration patterns”
Focusing more on Data Engineering, Modern Data
Stack, Agility, Leadership and helping Startups, see:
• http://youtube.serverlesstoronto.org/
Your presentations regardless of how big or small ☺
Please rate us on Meetup & tell others about #ServerlessTO user group

Knowledge Sponsor
1. Go to www.manning.com
2. Select *any* e-Book, Video course, or liveProject you want!
3. Add it to your shopping cart (no more than 1 item in the cart)
4. Raffle winners will send me the emails (used in Manning portal),
5. So the publisher can move it to your Dashboard – as if purchased.
Fill out the Survey to win: bit.ly/slsto

©Google Inc. or its aﬃliates. All rights reserved. Do not distribute.
GCP Data
BigQuery
Cloud
Dataﬂow
Cloud
Dataproc
Cloud
Datalab
Cloud
Pub/Sub
Genomics

Everyone.
Who Wants Real-time Data?

Mobile
Devices
Tens of Thousands Events/sec
Tens of Billions Events/month
Hundreds of Billions Events/year
The Lambda Model

Mobile
Devices
Apache Beam
+
A Uniﬁed Model
or
or

Mobile
Devices
Cloud Pub/Sub Cloud Dataﬂow BigQuery
A Uniﬁed Model on Google Cloud Platform

Compute and Storage
Unbounded
Bounded
Resource Management
Resource Auto-scaler
Dynamic Work
Rebalancer
Work Scheduler
Monitoring
Log Collection
Graph Optimization
Auto-Healing
Intelligent Watermarking
S
O
U
R
C
E
S
I
N
K
What is Cloud Dataﬂow?

Here’s a simple graphic showing how Dataflow can integrate and transform data from two sources.
One discrete job
Endless
incoming data
Cloud
Dataflow
What is Cloud Dataflow?

Deploy
Schedule & Monitor
Autoscaling mid-job
Fully-managed and auto-conﬁgured
Auto graph-optimized for best execution path
Dynamic Work Rebalancing mid-job
1
2
3
4
Why Use Cloud Dataﬂow?

Autoscaling mid-job
1
2
3
4
C D
C+D
C D
C+D
A GBK + B
A+ GBK + B

800 RPS 1200 RPS 5000 RPS 50 RPS
*means 100% cluster utilization by deﬁnition
Autoscaling mid-job
1
2
3
4

Autoscaling mid-job
1
2
3
4
100 mins. 65 mins.
vs.

Start off with 3 workers,
things are looking okay
10 minutes
3 days
Re-estimation shows there’s
orders of magnitude more work:
need 100 workers!
Idle
You have 100 workers
but you don’t have 100 pieces of work!
...and that’s really the most important part
Autoscaling at Work

Now scaling up (and down) is no big deal!
Add workers
Work distributes itself
Job starts with 3 workers, scales up to 1000.
When all work is done, scale down
Autoscaling + dynamic rebalancing
Waves of splitting
Upscaling cycles
and VM startup

End-user's pipeline
Libraries: transforms, sources/sinks etc.
Language-speciﬁc SDK
Beam model (ParDo, GBK, Windowing…)
Runner
Execution environment
Java ...
Python
The “Stack”

pipeline
.apply(PubsubIO.Read.named("read from PubSub")
.topic(String.format("projects/%s/topics/%s",
options.getSourceProject(), options.getSourceTopic()))
.timestampLabel("ts")
.withCoder(TableRowJsonCoder.of()))
.apply("window 1s",
Window.into(FixedWindows.of(Duration.standardSeconds(1))))
.apply("mark rides", MapElements.via(new MarkRides()))
.apply("count similar", Count.perKey())
.apply("format rides", MapElements.via(new TransformRides()))
.apply(PubsubIO.Write.named("WriteToPubsub")
.topic(String.format("projects/%s/topics/%s",
options.getSinkProject(), options.getSinkTopic()))
.withCoder(TableRowJsonCoder.of()));
Read from Pubsub
Window of 1 second
Create KV pairs
Count them by key
Format for output
Write to Pubsub

Using Dataﬂow Templates
Launching a Simple Pipeline
Ingest
Cloud
Pub/Sub
Pipelines
Cloud
Dataﬂow
Analytics
BigQuery

Pub/Sub to BigQuery
Dataflow templates let you stage
your job’s artifacts in Google
Cloud Storage.
Launch template jobs via REST
API, or Cloud Console.

public static void main(String[] args) {
…
Pipeline p = Pipeline.create(options);
p.apply(TextIO.Read.named("ReadLines")
.from(options.getInputFile()))
.apply(new CountWords())
.apply(ParDo.of(new FormatAsTextFn()))
.apply(TextIO.Write.named("WriteCounts")
.to(options.getOutput()));
p.run();
}

For further reading/watching
NYC Taxi Tycoon Codelab
Google-provided Dataﬂow Templates
The World Beyond Batch - Streaming 101
What is the watermark heuristic for PubsubIO on GCP?
Spotify’s Event Delivery Pipeline (Part 1 of 3)

www.ServerlessToronto.org
Reducing the gap between IT and Business needs

How to build unified Batch & Streaming Pipelines with Apache Beam and Dataflow

Recommended

More Related Content

Similar to How to build unified Batch & Streaming Pipelines with Apache Beam and Dataflow (20)

More from Daniel Zivkovic (20)

Recently uploaded (20)

How to build unified Batch & Streaming Pipelines with Apache Beam and Dataflow