Data Virtualization: Bringing the Power of FME to Any Application
0 likes314 views
Imagine building web applications or dashboards on top of all your systems. With FME’s new Data Virtualization feature, you can deliver the full CRUD (create, read, update, and delete) capabilities on top of all your data that exploit the full power of FME’s all data, any AI capabilities. Data Virtualization enables you to build OpenAPI compliant API endpoints using FME Form’s no-code development platform. In this webinar, you’ll see how easy it is to turn complex data into real-time, usable REST API based services. We’ll walk through a real example of building a map-based app using FME’s Data Virtualization, and show you how to get started in your own environment – no dev team required. What you’ll take away: -How to build live applications and dashboards with federated data -Ways to control what’s exposed: filter, transform, and secure responses -How to scale access with caching, asynchronous web call support, with API endpoint level security. -Where this fits in your stack: from web apps, to AI, to automation Whether you’re building internal tools, public portals, or powering automation – this webinar is your starting point to real-time data delivery.
More Related Content
Similar to Data Virtualization: Bringing the Power of FME to Any Application (20)
More from Safe Software (20)
Recently uploaded (20)
Data Virtualization: Bringing the Power of FME to Any Application
- 1. This is Where We Will Write the Title for our Webinar, Option 1
- 2. Sanae Mendoza Customer Solutions Specialist Meet the Presenters Don Murray CEO
- 3. Welcome to Livestorm. A few ways to engage with us during the webinar: Audio issues? Click this for 4 simple troubleshooting steps.
- 4. How to download slides 1. Hover over the slide deck in the webinar room 2. Click this button
- 5. Agenda 1 Introduction to Data Virtualization 2 Data Virtualization 101 3 Create a Data Virtualization API 4 Create a Data Virtualization Endpoint 5 Authoring an Endpoint Workflow 6 Building Better Endpoints 7 Enriching Endpoints 8 Conclusion 9 Resources & Next Steps 10 Q&A Agenda
- 7. FME’s Data virtualization empowers your enterprise to deliver real-time, code-free access to trusted data across systems. No duplication, no delays.
- 8. What is Data Virtualization? ● A method to access and query data in real-time, without moving or copying it. ● Think: unified views across multiple sources, instantly.
- 9. Common Frustrations Organizations Face Without Data Virtualization ● Siloed systems and inconsistent data ● Time-consuming data movement and ETL ● Redundant storage costs ● Delayed insights due to refresh cycles ● Complex integrations with multiple tools Do any of these feel familiar?
- 10. You don’t need to wrangle data manually or wait on an IT team. With FME’s Data Virtualization, you can connect, transform, and access what you need: on demand.
- 11. Simplifying Data Access Live Data Access Simplified Data Views Flexible Connections Grow with your needs Deliver real-time data through secure APIs. Make complex systems easy to connect to and use. Link systems without heavy upgrades or rework. Add new data sources and users without slowing down.
- 12. The only All-Data, Any-AI Platform. FME Form FME Flow All Data workflows are built here. Brings life to FME Form workflows FME Flow Hosted SaaS version of FME Flow fme.safe.com/platform FME Enterprise Integration Platform Safe & FME FME Realize Experience your data where it matters most. In context
- 13. With 500+ supported data types in FME. Unrivalled Data Support GIS CAD Database XML Raster 3D BIM Web Point Cloud Cloud Big Data IOT Gaming BI Indoor Mapping AR/VR Generative AI Cloud Native Tabular
- 15. Breaking Down Data Virtualization in FME 🧠 In Theory ● Data virtualization means managing access, not making copies ● No data movement ● Just a layer that connects to what already exists 🛠 In Practice (with FME) ● FME builds REST APIs from your live data ● That layer becomes an API ● Other tools can request exactly what they need—live, filtered, structured
- 16. Breaking Down Data Virtualization in FME 💡 Virtualized at the backend, accessed via REST APIs, delivered through FME — no custom code required.
- 18. How does it compare to Webhook URLs and FME Flow REST API? Webhook URLs ● Basic event triggers ● GET/POST only ● Limited input/output ● No schema or docs ● Not a full API ● No customization FME Flow REST API ● FME Flow management focused ● Limited integration ● Limited input/output ● No customization Data Virtualization APIs ● Integration focused ● Full data access ● Full CRUD ● Full customization ● Docs + schema support ● Secure, user-friendly
- 19. Where Data Virtualization Starts… and Stops ❌ Not Read-Only Supports full CRUD operations. ❌ Not Just for Enterprises Scales from small teams to large organizations. ❌ Not an API Gateway Provides real-time data access, not traffic control or authentication. ❌ Not a Data Warehouse Replacement Delivers live views without storing or duplicating data.
- 20. Scenario 2: Legacy Systems Blocking Integration Problem: Critical data lives in old systems that are hard to connect to, slowing down projects and raising costs. Solution: Use FME Data Virtualization to access legacy data directly—no migration needed. Value: Fast, low-cost integration that keeps systems running without disruption. What can I do with FME Data Virtualization?
- 21. Scenario 3: AI Tools Need Better Data Access Problem: Teams want to use AI tools, but connecting them to existing data workflows is slow and expensive. Solution: Expose FME workflows as virtual endpoints that AI can query in real time. Value: Smarter decisions, faster answers, and AI that actually fits your data context. What can I do with FME Data Virtualization?
- 22. Scenario 1: Department Reports to Integrated Dashboards Problem: Teams are emailing Excel reports weekly. Data is copied manually into dashboards, leading to delays and mistakes. Solution: Use Data Virtualization to expose those Excel files as live, queryable data sources—no more manual steps. Value: Updates are instant, errors drop, and managers get accurate dashboards without waiting. What can I do with FME Data Virtualization?
- 23. Flash Demo
- 25. 1. Real-time dashboard with region + capacity filters 2. Live API calls backed by Data Virtualization 3. Fully documented in Swagger 4. Powered by a simple FME workspace (Excel + GeoPackage) Demo: Emergency Facilities Dashboard
- 26. 3 Creating a Data Virtualization API
- 27. Data Virtualization in FME Flow ● API Configuration ○ API Details ○ Schemas ○ Endpoint Metadata ○ Security ○ Workspace Generation Right click and select ‘replace image’ to add an example image from your project.
- 28. Data Virtualization in FME Form ● Endpoint behaviour ○ Data sources ○ Transformations ○ Analysis ○ Format API response Right click and select ‘replace image’ to add an example image from your project.
- 29. FME Flow API Configuration Schemas Endpoint Metadata Security Workspace Generation FME Workbench Endpoint behaviour Data sources Transformations Analysis Generate API response Endpoint Authoring
- 30. OpenAPI Specification (OAS) A standardized way to describe REST APIs in a machine-readable format (JSON) Defines endpoints, methods, request/response formats, and authentication Import and Export APIS to FME Flow Swagger UI API Documentation and interactive testing.
- 31. Testing with API Clients ● OpenAPICaller ● HTTPCaller ● Swagger Documentation (Web) ● Browser ● Postman
- 32. From Import… ● Import an OpenAPI (OAS) file (JSON) ● Auto-generates structure and methods ● Less manual setup ● Useful for migrating existing APIs From Scratch… ● Manually define API structure ● Full control over endpoints and formats ● Ideal for custom APIs ● No existing documentation required Creating a Data Virtualization API
- 33. API Security ● Secure by endpoint or API-wide ● Role- or user-based access ● Managed separately from FME Flow permissions ● Works with external systems ● Supports Basic Auth and API Tokens
- 34. Demo
- 35. Demo: Creating a Data Virtualization API
- 37. 1. Imported OpenAPI Specification file (EnvironData.json) to create an API 2. Toured the Data VIrtualization Interface in FME Flow 3. Exported the EnvironData API OpenAPI file to use in the FME Workbench and Postman Demo: Creating a Data Virtualization API
- 39. What is a Data Virtualization API Endpoint? ● Defines how users access your data ● Customize requests, responses, parameters, status codes, headers, schemas, security, and metadata ● Supports both static and dynamic responses ● Defined in FME Flow, authored in FME Workbench
- 40. Defining Endpoint Metadata in Flow
- 41. Defining Endpoint Metadata in Flow
- 42. Manual vs Workspace Responses Manual Workspace Response Always the same Changes based on inputs or data state Backend No workspace Workspace powers response Data source None (static) Connected to live data Use Case Metadata, service status Queries, analysis, live data
- 43. Workspace Responses ● Powered by an FME workspace ● Response changes based on inputs and live data ● Supports querying, transformation, and analysis ● Ideal for dynamic, real-time APIs ● Flexible functionality: parameters, richer methods, multiple status codes, etc.
- 44. Inside a Workspace Response
- 48. Defining Endpoint Metadata in FME Flow
- 49. Defining Endpoint Metadata in FME Flow
- 50. Schemas ● Define the structure of requests and responses ● Improve API usability, validation, and reliability ● Help developers understand what to send and expect Two main types: ○ Request Body Schemas (for POST requests) ○ Response Body Schemas (linked to status codes)
- 54. Workspace Generation and Management Right click and select ‘replace image’ to add an example image from your project.
- 56. Workspace Generation and Management Right click and select ‘replace image’ to add an example image from your project.
- 57. Reader Represents the API Request ● Creates a feature type for every endpoint ● Attributes: Endpoint path, parameters, body Writer Represents the API Response ● Always a single feature type ● Attributes: Response status code, headers, and body
- 58. Building Blocks of a Data Virtualization Workspace
- 59. Better, Faster, Stronger APIs Purpose Tools Connecting to external data sources and joining datasets FeatureReader, FeatureWriter, DatabaseJoiner, InlineQuerier, SQLCreator, SQLExecutor, HTTPCaller, AttributeValueMapper Formatting response bodies and parsing request bodies JSONTemplater, JSONFragmenter, JSONFlattener Creating status codes and headers AttributeCreator, AttributeManager Validating and filtering request parameters Tester, TestFilter, AttributeFilter, AttributeValidator Fast authoring Save attributes as parameter presets
- 60. Templates Input templates simulate API requests ● Simulate API requests with sample data ● Include method, path, headers, body, and parameters ● Save requests to reuse for testing ● Use in FME Form to test logic, validate responses, and debug locally Output templates represent API responses ● Defined schema, data types, status codes, content types, headers, etc.
- 61. Demo
- 62. Demo: Creating an Endpoint with a Workspace Response
- 64. 1. Created the GET /facilities endpoint 2. Generated an endpoint workspace 3. Authored an endpoint workspace a. Combined data sources: Excel & GIS b. Formatted a JSON response c. Created status codes 4. Republished to FME Flow Demo: Creating an Endpoint with a Workspace Response
- 66. Best Practices ● Request parameters ● Routing for status codes ● Error catching ● Built-in data validation + cleaning
- 67. Request Parameters Endpoint parameters allow users to refine requests and customize responses, improving the flexibility, efficiency, and overall usability of the API. Type Use Case Example Path Identify a specific resource /users/{userId} Query Filter, sort, or control response content /wildfires?region=west&year=2023 Headers Provide metadata or authentication Authorization: Bearer {token}
- 68. Designing Workspaces for Parameters Design workspaces to check for parameter input and apply filters conditionally for better performance and flexibility. Type Tools Pros & Cons Reader Parameters WHERE Clauses, SELECT, Spatial Filters Very efficient - filter before loading Querying Transformers InlineQuerier, SQLExecutor, DatabaseJoiner Very efficient - filter & join before loading Filter Transformers Tester, Sampler Less efficient - filter after loading
- 69. Workflow: Updating Endpoint Metadata
- 70. Workflow: Updating Endpoint Metadata
- 71. Demo
- 72. Demo: Adding Parameters to an Endpoint
- 73. Demo: Adding Parameters to an Endpoint
- 74. Demo
- 75. 1. In FME Flow, updated GET /facilities endpoint to have path and query parameters (path.region, query.min_capacity) 2. In FME Form, updated the facilities.fmw workflow to filter the Excel and Geopackage data. 3. Republished to FME Flow Demo: Adding Parameters to an Endpoint
- 78. Files and Resources ● Accept file uploads (images, documents, datasets) ● Use multipart/form-data in requests ● Return files like PDFs, reports, or shapefiles ● Set content type and filename with response headers
- 79. Caching ● Faster Responses: Store common results to avoid repeated processing ● Reduced Load: Less strain on backend systems and databases ● Improved Scalability: Serve more users with fewer resources Limitations: Cache stored for path parameters, not query, headers, or bodies.
- 80. Job Expiry ● Prevents Stale Requests: Stops old or abandoned jobs from consuming resources ● Handles Timeouts Gracefully: Ensures long-running jobs don’t block the system ● Improves System Stability: Automatically cleans up jobs stuck in queues or processing
- 81. Asynchronous Processing ● Handles Long-Running Tasks: Processes jobs without blocking the client ● Improves User Experience: Clients can check back later or get notified when complete ● Boosts API Resilience: Prevents timeouts and reduces strain on Flow
- 82. Asynchronous Processing { "requestID": {request-id}, "statusUrl": "/api/status/{request-id}", "resultUrl": ”/api/result/{request-id}" } GET GET
- 83. When things go wrong: Debugging Tips Custom vs. System Status Codes ● 500: Job failure or other server side ● 401: Authentication issue ● 404: path not found ● 403: Unauthorized ● 415: Unsupported media type - Flow only supports application/json, multipart/mixed, and multipart/form-data. Job Log files ● Check request parameters System Log Files ● Need permissions
- 84. 8 Conclusion
- 85. Summary Data virtualization delivers a real-time, unified view of your data.
- 86. 30+ 30K+ 128 140+ 25K+ years of solving data challenges FME Community members countries with FME customers organizations worldwide global partners with FME services 200K+ users worldwide 200K+ users worldwide
- 87. 8 Resources
- 88. Data Virtualization Resources Learn ● Tutorial: Getting Started with Data Virtualization Action ● Download the newest release 2025.1 Beta! ● Share your workflows with us!
- 89. Get our Ebook Spatial Data for the Enterprise fme.ly/gzc Guided learning experiences at your fingertips academy.safe.com FME Academy Resources Check out how-to’s & demos in the knowledge base support.safe.com Knowledge Base Webinars Upcoming & on-demand webinars safe.com/webinars
- 90. 9 Next Steps
- 91. We’d love to help you get started. Get in touch with us at [email protected] Experience the FME Accelerator Contact Us A world where data is not just a commodity but a catalyst for real change. fme.safe.com/accelerator Next Steps
- 92. ClaimYour Community Badge & Dive into the new Community! ● Get community badges for watching webinars ● community.safe.com ● Today’s code: 18H2MG Join the Community today! Next Steps
- 93. 10 Q&A
- 94. ThankYou Recap of Next Steps 1 Follow us on LinkedIn! 2 Contact us 3 Experience the FME Accelerator Please fill out our webinar survey