The Zero-ETL Approach: Enhancing Data Agility and Insight
0 likes778 views
In the ever-evolving landscape of data management, Zero-ETL is an approach that is reshaping how businesses handle and integrate their data. This webinar explores Zero-ETL, a paradigm shift from the traditional Extract, Transform, Load (ETL) process, offering a more streamlined, efficient, and real-time data integration method. We will begin with an introduction to the concept of Zero-ETL, including how it allows direct access to data in its native environment and real-time data transformation, providing up-to-date information with significantly reduced data redundancy. Next, we'll take you through several demonstrations showing how Zero-ETL can deliver real-time data and enable the free movement of data between systems. We will also discuss the various tools that support all aspects of Zero-ETL, providing attendees with an understanding of how they can adopt this innovative approach in their organizations. Lastly, the session will conclude with an interactive Q&A segment, allowing participants to gain deeper insights into how Zero-ETL can be tailored to their specific business needs and how they can get started today. Join us to discover how Zero-ETL can elevate your organization's data strategy.
More Related Content
Similar to The Zero-ETL Approach: Enhancing Data Agility and Insight (20)
More from Safe Software (20)
Recently uploaded (20)
The Zero-ETL Approach: Enhancing Data Agility and Insight
- 1. The Zero-ETL Approach: Enhancing Data Agility and Insights
- 2. Christian Berger Solutions Specialist Stewart Harper VP of Product Development Don Murray Co-Founder Meet our Team
- 3. Welcome to Livestorm. A few ways to engage with us during the webinar: Audio issues? Click this for 4 simple troubleshooting steps.
- 4. How to download slides 1. Hover over the slide deck in the webinar room 2. Click this button
- 5. Agenda 1 What is ETL, and Reverse ETL? 2 What about Spatial Data and ETL? 3 What is Zero-ETL? 4 Real Time Data Services and the last data mile 5 Connecting Applications in Real-time 6 High Velocity Data Streams 7 Coming Soon: Data Virtualization 8 Conclusion, Resources, Next Steps, QA Agenda
- 6. Enterprise Integration. Getting data to where it adds the most value, when it is needed. Extract, Transform, and Load (ETL) is one way to describe the technology to do this.
- 7. 1 What is ETL, and Reverse ETL?
- 8. On-Premises ETL = Extract data from systems, Transform, and Load into Data stores.
- 9. ETL is critical for businesses, combining data from multiple sources, to provide a consistent and unified view for reporting and analysis.
- 10. On-Premises ETL = Extract data from systems, Transform, and Load into Data stores. Reverse ETL = Extract data from data stores, Transform, and Load into systems.
- 11. Reverse ETL operationalizes insights, making the data actionable in workflows and tools that support customers, manage operations, and drive business growth.
- 12. Any type of ETL is ● getting and combining data, ● from where it is, ● to where you need it, ● when you need it, ● how you need it! This data movement is what the FME Platform is all about.
- 13. 2 What about Spatial Data and ETL?
- 14. The world we live in is Spatial. How many ways have you used spatial today without even thinking about it? Yet most businesses don’t use it to gain new insights!
- 15. Spatial Data is critical to get from digital optimization to digital transformation. ● Spatial data reveals new relationships and insights to make better decisions. ● Without spatial data you miss opportunities, allocate resources poorly, and fall behind your competitors. ● Spatial Data is critical for all types of ETL and data movement.
- 16. Unrivalled Data Type support Spatial Data is Everywhere
- 19. On-Premises Zero-ETL ● Direct Access to current data and services. ● Real-time Synchronization of data across systems. ● Flexibility and Agility: Data flows across all systems and data stores in Real-Time. Real-Time Data Movement
- 20. Zero-ETL delivers real-time data management and analysis so organizations can respond quickly to needs and opportunities.
- 21. One platform, two technologies FME Form FME Flow Data Movement and transformations (“ETL”) workflows are built here. Brings life to FME Form workflows FME Flow Hosted Safe Software managed FME Flow fme.safe.com/platform FME Enterprise Integration Platform Safe & FME
- 22. FME Flow delivers Zero-ETL
- 23. 4 Real-Time data services and the last data mile
- 24. Poll or Chat Storm Question Goes Here. This will help grab people’s attention and get everyone engaged right away.
- 25. 1 2 3 Unlimited, Managed Users - User/Role permissions - Authentication Services (AD, SAML) Flexible Engine Deployment - Deploy Engines Anywhere - CPU-Based Pricing - Remote Engines Advanced Engine Management - Queues & Job Routing - Active Periods - Engine Assignment
- 26. Data Streaming Workspace Flow Apps User uploads file to FME Flow Workspace App
- 27. Data Streaming Workspace Flow Apps User uploads file to FME Flow Workspace App FME Flow Engines DataValidation.fmw executes on next available engine Invalid Feature HTML Report Submission ID and Timestamp
- 28. Data Streaming Workspace Flow Apps User uploads file to FME Flow Workspace App Job Queue DataValidation.fmw routed to a specified Queue Invalid Feature HTML Report Submission ID and Timestamp
- 29. Data Streaming Workspace Flow Apps Invalid Feature HTML Report Submission ID and Timestamp
- 31. Data Download Workspace Flow Apps Job Queue DataDownload.fmw routed to a specified Queue User Selects Area of Interest to Retrieve Feature(s) from Job output provided to client as downloadable .zip file
- 32. Demo
- 34. Customer Story City of Henderson Project Digital Plan Submission Solution FME provides self-serve CAD data upload, validation, and integration into GIS. Results ● Reduced data integration time by 75%. ● Eliminated the risk of manual error. ● Removed duplicate work. ● Faster processing of submissions. ● Ensured CAD data validation. “We chose FME for this project because the interface is easy to use and allows us to provide an adaptable solution to enhance interaction with the City’s partners.” - Erika Provost, City of Henderson
- 36. FME Flow Automations Connect different systems so actions in one system update another ● Powerful trigger mechanism via webhooks, data arrival, database triggers, etc. ● Build multi-step complex workflows ● Take advantage of true parallelism. ● Full orchestration control via split merge blocks. ● Reduce errors and save time by replacing manual tasks with automation.
- 37. Uni-Directional Integration Architecture Linear data flow used to repair, validate, and replicate data from system A to system B ● Triggers can vary from files being received to schedules and webhooks. Dynamic data flow(s) used to integrate multiple systems by synchronizing changes between 2 (or more) systems. ● Typically driven by an action being performed in either system for near real time processing via webhooks or topics/notifications. Bi | Multi-Directional
- 38. SFTP Dropbox System Event Azure Event Grid JMS UDP HTTP Request WebSocket FTP Logger SQS/SNS SMTP SQS/SNS Filter Merge S3 Bucket Network Triggers and Notifications
- 39. FME Flow Automation Triggered (Webhook Received from Survey123) User submits Survey Integrate AGOL and Cityworks via REST API Published FME Workspace (.fmw)
- 40. FME Flow Automation Triggered Record inserted, updated, or deleted Record inserted, updated, or deleted Published FME Workspace executes and synchronizes changes to between ArcGIS Online Features & Cityworks Work Orders
- 41. FME Flow Automation Triggered Record inserted, updated, or deleted Record inserted, updated, or deleted Published FME Workspace executes and synchronizes changes to features from Cityworks, Salesforce, and ArcGIS Online
- 42. Demo
- 44. Customer Story City of Naperville Project Automating time-consuming processes for their Electric, Public Works, Finance, and Water departments. Solution Integrating FME with Cityworks to automate service requests and work orders. Results ● 3300+ service requests and work orders automated, saving hours of manual effort. ● Three staff with specialized coding skills free to focus on high-value tasks rather than manual work ● Increased collaboration and stronger relationships between departments “With FME, we were suddenly saying “yes!” to other departments and implementing requests quickly. Now, we’re viewed differently in our interactions. FME makes it easy for us to improve our relationships with other departments by helping improve their business processes.” - Danielle DuBose, Database Developer, City of Naperville
- 46. Let’s start with the data ● Bounded data is finite and has a discrete beginning and end. It is associated with batch processing. ● Unbounded data is infinite, having no discrete beginning or end. It is associated with stream processing. ● FME handles both!
- 47. Use Cases 1. Business Applications - Essential for driving real-time business intelligence and process automation. e.g. Customer orders, airline reservations, insurance claims, bank transactions, telco call detail records (CDR). 2. Digital Information - Captures the dynamics of online engagement and system health. e.g. clickstreams, logs, market data, email. 3. Internet of Things (IoT) - Capture the vast and growing network of connected devices, translating real-time data from the physical world into actionable intelligence. e.g RFID, telemetry SCADA, GPS sensors, smart meters, smart city infrastructure.
- 48. Supported IoT & Messaging Protocols
- 49. Simple Streaming Pattern: Filter by Attribute Reduce data volumes by saving only what you are interested in.
- 50. FME Flow Streams Process data continuously with low latency.
- 52. Data Virtualization Create Open API compliant APIs on top of all FME Data Types. Real-time data (Zero-ETL) to deliver direct data access via web API. Effectively making any data cloud native.
- 53. Data Virtualization Value Agility ● Unified view isolates applications from underlying systems. Cost Savings ● Simplify infrastructure. Save time and resources on data integration. Improved Decision Making ● Real-time access to enterprise applications used by decision makers. Enhanced Data Security and Compliance ● Data access via a secure API improves data security and governance. Increased Productivity ● Frees IT to focus on strategic projects; business applications access data directly.
- 54. Data Virtualization Connects Anything Caching. We don’t want to run any job unless needed. Security. controlling who and type of access they have. Monitoring Who is using what how much. Rate limiting, etc. FME Flow
- 55. FME Flow 2024.x
- 56. FME Form Workbench - Data Virtualization
- 57. 8 Conclusion
- 58. 29+ 27K+ 128 190 20K+ years of solving data challenges FME Community members countries with FME customers organizations worldwide global partners with FME services 30+ 29K+ 128 140+ 25K+ years of solving data challenges FME Community members countries with FME customers organizations worldwide global partners with FME services 200K+ users worldwide Safe & FME
- 59. Resources
- 60. Get our Ebook Spatial Data for the Enterprise fme.ly/gzc Guided learning experiences at your fingertips academy.safe.com FME Academy Resources Check out how-to’s & demos in the knowledge base community.safe.com /s/knowledge-base Knowledge Base Webinars Upcoming & on-demand webinars safe.com/webinars
- 61. Next Steps
- 62. We’d love to help you get started. Get in touch with us at [email protected] Experience the FME Accelerator Contact Us A world where data is not just a commodity but a catalyst for real change. fme.safe.com/accelerator Next Steps
- 63. ClaimYour Community Badge & Dive into the new Community! ● Get community badges for watching webinars ● fme.ly/RewardCodes ● Today’s code: Join the Community today!
- 64. Q&A
- 65. ThankYou Recap of Next Steps 1 Join the FME Community 2 Contact us 3 Experience the FME Accelerator Please fill out our webinar survey