The World Isn’t Flat—Why Your Data Pipeline Shouldn’t Be Either

“Data is the new oil, but only if it’s refined, contextualized, and given depth.”— Adapted from Clive Humby’s original metaphor

The Depth of Data: A Deep Dive into 3D Data Engineering

Introduction: Beyond Flatland

In the earliest days of computing, data was flat—stored in tables, columns, and CSVs. But the world is not flat, and as our systems matured, so did our data. The rise of 3D data represents more than an ability to store geometry—it’s the maturation of how we model the world in space, time, and context.

From LiDAR scans that reconstruct cities to knowledge graphs that represent people over time, 3D data isn’t just about physical dimensions. It’s about depth—in structure, in meaning, and in use.

What is 3D Data?

At its core, 3D data captures and represents multi-dimensional structures—often physical, but increasingly also semantic and temporal. There are two broad lenses through which we can understand 3D data:

1. Spatial 3D Data
   
   • Point Clouds from LiDAR scanners or stereo vision
   • Meshes formed by connecting vertices into surfaces
   • Voxels, or volumetric pixels, used in simulations and medical imaging
   • 3D Tiles used for rendering massive datasets in web-based viewers
   • Scene Graphs for organizing spatial relationships in simulations and games
2. Semantic 3D Data
   
   • Data that combines attributes (e.g., skillsets, properties)
   • Signals that change over time (e.g., career movement, system metrics)
   • Context that gives those attributes meaning (e.g., team structure, business phase, ecosystem)

In either case, the key is modeling not just what something is, but where, when, and in what context it exists.

Historical Context: From Maps to Models

3D data has evolved from primitive beginnings. Civil engineers in the 20th century pioneered CAD systems to model infrastructure in 3D. NASA and defense agencies pushed further with terrain mapping using radar and LiDAR.

By the 2000s, LiDAR moved into civilian use—urban planning, agriculture, and autonomous vehicles. Meanwhile, the entertainment and gaming industries revolutionized how we think about and interact with 3D environments.

Today, that same logic—of capturing real-world complexity—is now being applied to non-spatial domains, from modeling people in talent intelligence platforms to understanding supply chains and organizational graphs.

How to Create 3D Data

For Physical 3D Data

1. Capture:
   
   • LiDAR scanners for point cloud generation
   • Photogrammetry for modeling from images
   • Depth cameras, stereo vision, and drones
   • Mobile LiDAR, increasingly available in smartphones
2. Process:
   
   • Noise reduction and alignment
   • Meshing and surface reconstruction
   • Semantic segmentation and annotation
   • Compression and streaming for real-time rendering

For Semantic 3D Data

1. Ingest:
   
   • Scrape structured and unstructured data
   • Parse historical logs, metadata, and time-stamped records
2. Model:
   
   • Build time-aware knowledge graphs
   • Apply semantic enrichment and relationship mapping
   • Use contextual embedding and machine learning for inference
3. Query:
   
   • Enable graph traversals, temporal slicing, and contextual search
   • Integrate into UIs for semantic exploration or visualization

Data Engineering for 3D Data

Whether geometric or semantic, 3D data challenges traditional pipelines:

• Volume: Gigabytes or terabytes per scan or corpus
• Structure: Complex schemas with nesting and relationships
• Latency: Demands for real-time or near-real-time responsiveness
• Storage: Hybrid data stores—combining blobs, columnar tables, and spatial indexes
• Querying: Requires graph databases, vector search, spatial-temporal indexing

Reference Architecture for 3D Data Pipelines

Business Use Cases

What Good Looks Like

• Data is Richly Annotated: Not just points or keywords, but labeled with relationships, categories, and metadata.
• Downsampling and Indexing: Large datasets are processed efficiently for real-time use.
• Searchable & Explorable: Users can search for patterns or profiles contextually, with spatial or temporal slicing.
• Visualization-Ready: Whether in AR, 3D viewers, or dashboards, the data must be actionable and interpretable.

What Bad Looks Like

• Flat data pretending to be 3D: Without temporal context or relationships, it’s just noise.
• Lack of interoperability: Using custom formats or schemas with no standardization across teams or systems.
• No abstraction layers: Engineers forced to wrangle raw point clouds or unstructured text without APIs or clean data services.
• No lifecycle management: Massive datasets without versioning, retention policies, or update pipelines.

Where It’s Going

• Real-time AI on the Edge: Models trained on 3D environments making split-second decisions in physical or semantic contexts.
• Neural 3D Rendering: NeRFs and other AI methods replacing traditional mesh modeling.
• Web-Scale 3D: Technologies like glTF and 3D Tiles powering real-time, immersive web experiences.
• Unified Graphs: Combining spatial, temporal, and semantic data in single queryable systems.
• Synthetic Data: Generating photorealistic, temporally consistent 3D scenes for ML training or simulation.

Wrapping up…

3D data is no longer confined to geospatial systems or gaming engines. It’s a lens—spatial or semantic—for modeling the real complexity of the world. For data engineers, it demands new tools, new thinking, and new patterns. Whether you’re simulating a city or mapping someone’s career over time, 3D data brings nuance, fidelity, and realism to how we build, understand, and decide.

And just like the world it represents—it’s only going to get deeper.