What is the best storage architecture for manufacturing data?

Once a manufacturer has more than one data source worth analyzing together — machine signals, quality records, ERP data, maintenance logs — a data lake often becomes the natural place to put it all before deciding exactly how it will be used. This article covers how to actually get shop floor data into one: the architecture involved, the ingestion patterns that fit machine data specifically, and the mistakes that turn a data lake into an unusable "data swamp." For choosing a specific platform, our roundup of free industrial data lake and analytics platforms covers the tool landscape; this article is about the how, regardless of which platform you pick.

Contents:

1. What a data lake is, and why manufacturers use one
2. Data lake vs. data warehouse vs. time-series database
3. A typical architecture
4. Ingestion patterns for machine data
5. Schema-on-read vs. schema-on-write
6. Practical steps to get started
7. Common pitfalls
8. Frequently asked questions
9. Conclusion

What a data lake is, and why manufacturers use one

A data lake is a centralized repository that stores data in its raw, native format — structured, semi-structured, or unstructured — without requiring it to be transformed into a fixed schema before being stored. For manufacturers, this matters because shop floor data rarely lives in one shape: machine telemetry, quality inspection records, maintenance logs, and ERP data all have different structures, and a data lake lets all of it land in one place before deciding how each piece will eventually be modeled and queried, rather than forcing a rigid structure upfront that may not fit every source.

Data lake vs. data warehouse vs. time-series database

These three terms get used loosely, but they serve different purposes:

• A data lake stores raw data in its original format, prioritizing flexibility and the ability to ingest anything without upfront schema design.
• A data warehouse stores data that's already been structured and modeled for a specific analytical purpose, typically after passing through some transformation process.
• A time-series database is optimized specifically for timestamped, sequential data like machine telemetry, with query performance tuned for time-range operations rather than general-purpose analytics. We compare this option against traditional relational storage directly in SQL vs. time-series database for industrial data, and cover the underlying concept in our time-series data primer.

These aren't mutually exclusive: a common architecture uses a data lake as the raw landing zone for everything, a time-series database for machine telemetry specifically where query performance on time-range data matters most, and a data warehouse for cleaned, modeled data destined for business reporting.

A typical architecture

• Sources. Machine data (via the connection methods covered elsewhere on this blog), quality systems, ERP/MES data, and any other relevant source.
• Ingestion layer. The pipeline that moves data from each source into the lake, often involving some transformation along the way — covered in more detail in our piece on ETL pipelines for machine data.
• Raw zone. Data lands here largely as-is, preserving the original format for traceability and in case transformation logic needs to be revisited later.
• Processed/curated zone. Cleaned, validated, and often reorganized data that's ready for analysis, built from the raw zone rather than replacing it.
• Consumption layer. BI tools, custom analytics, or machine learning workflows that query the processed zone (and sometimes the raw zone directly for specialized needs).

Ingestion patterns for machine data

• Batch ingestion moves data into the lake on a schedule — hourly, daily — suited to data where near-real-time availability isn't critical, such as historical trend data for periodic reporting.
• Streaming ingestion moves data continuously as it's generated, more appropriate for use cases needing fresher data, often built on top of a message-based pattern like MQTT or a similar streaming mechanism feeding into the lake's ingestion pipeline.
• File formats matter for downstream performance. Columnar formats like Parquet are commonly used for the processed zone because they support efficient querying of large datasets; raw JSON or CSV is often fine for the initial raw zone where fidelity to the source matters more than query speed.

Schema-on-read vs. schema-on-write

A defining feature of data lakes is "schema-on-read" — data is stored without a rigid predefined structure, and the schema is applied at query time by whatever tool is reading it, rather than enforced when the data is written. This is more flexible than the "schema-on-write" approach of traditional databases, since it accommodates data sources with different or evolving structures without requiring the destination to be redesigned every time a new field appears. The trade-off is that without discipline, "flexible" can become "inconsistent" — which is exactly the failure mode covered in the pitfalls section below, and part of why deliberate schema design still matters, covered specifically in our piece on designing a machine data schema for analytics.

Practical steps to get started

• Start with a clear use case, not "collect everything." Identify the specific analysis or reporting goal that justifies bringing shop floor data into a lake before building the pipeline.
• Choose an ingestion method matching data urgency. Batch for periodic reporting needs, streaming for anything closer to real-time.
• Preserve raw data separately from processed data. Keeping the original, untransformed data available means transformation logic can be revisited or corrected without needing to re-collect from source.
• Document what's in the lake as you go. A data catalog, even a simple one, prevents the lake from becoming a collection of undocumented files nobody remembers the meaning of.
• Build governance in from the start. Access control, retention policy, and data quality checks are much easier to establish early than to retrofit after months of unmanaged ingestion; our piece on manufacturing data governance covers this in more depth.

Common pitfalls

• Turning the lake into a "data swamp." Ingesting data with no documentation, no schema conventions, and no governance eventually makes the lake unusable, since nobody can reliably find or trust what's stored there.
• No clear ownership. Without someone responsible for the lake's structure and quality, it tends to accumulate inconsistency over time as different teams add data their own way.
• Ingesting raw machine signals without validation. As covered in our piece on shop floor data quality, bad data at the source doesn't improve by being moved into a lake — it just gets bad data stored at scale.
• Skipping the processed/curated zone. Expecting every consumer to work directly with raw data pushes transformation logic into every downstream tool separately, rather than doing it once centrally.

Frequently asked questions

Do I need a data lake if I only have machine telemetry, with no other data sources?

Not necessarily. If machine data is the only source and it's primarily used for real-time or near-real-time monitoring, a dedicated time-series database is often a simpler, better-fitting choice than a general-purpose data lake. Data lakes earn their value most clearly when combining multiple, differently-structured data sources.

Can a data lake replace a monitoring platform's own storage?

Generally not as a direct replacement — monitoring platforms usually have storage optimized for their own dashboards and calculations. A data lake more typically sits alongside, receiving a copy or export of the data for broader, cross-source analysis.

How is data quality maintained in a data lake if there's no enforced schema?

Through deliberate validation built into the ingestion pipeline and the processed/curated zone specifically, rather than relying on the lake's flexibility to somehow self-correct. Schema-on-read flexibility applies to structure, not to whether the underlying data is accurate.

Is a data lake overkill for a smaller manufacturing operation?

It can be, particularly if the actual need is straightforward reporting on a single data source. A data lake tends to justify its complexity once there are multiple data sources with different structures that genuinely need to be combined for analysis.

Conclusion

A data lake is most useful once shop floor data needs to sit alongside other, differently-structured business data for combined analysis — not as a default destination for every machine signal regardless of use case. Getting it right depends less on the specific platform chosen and more on deliberate ingestion design, a clear separation between raw and processed data, and governance built in from the start rather than retrofitted after the lake has already become difficult to navigate.

Related articles:

• Top Free Industrial Data Lake & Analytics Platforms
• ETL Pipelines for Machine Data
• SQL vs. Time-Series Database for Industrial Data
• Data Quality on the Shop Floor: Common Pitfalls
• MDCplus Machine Connectivity & Integrations

About MDCplus

Our key features are real-time machine monitoring for swift issue resolution, power consumption tracking to promote sustainability, computerized maintenance management to reduce downtime, and vibration diagnostics for predictive maintenance. MDCplus's solutions are tailored for diverse industries, including aerospace, automotive, precision machining, and heavy industry. By delivering actionable insights and fostering seamless integration, we empower manufacturers to boost Overall Equipment Effectiveness (OEE), reduce operational costs, and achieve sustainable growth along with future planning.